SUNDAY,
NOVEMBER 27, 2005
Enough about me...
POSTED BY DOUG AT 7:19 AM
In my first post here, I invited other
Xooglers to join me in reminiscing about life back in the day. Ron Garret has
accepted that invitation. Ron was the lead engineer on the first release of
AdWords, and the experience affected him so fundamentally that even his name changed.
It was great having Ron on the staff for
many reasons, not the least of which was that he nicely filled out our
hyperbole portfolio. We'd been talking about how smart our technology was and
after Ron joined us from the Jet Propulsion Lab, we could truthfully say that,
"Yes, Google is so complex that we have both a brain surgeon
and a rocket scientist working on it."
Welcome Ron...
1
COMMENTS:
Ron
said...
Thanks Doug!
I'll try to get a first post up later today.
SUNDAY,
NOVEMBER 27, 2005
Hello, world
POSTED BY RON AT 1:13 PM
The first post is always the hardest.
I've been debating with myself whether or
not to write about my Google experience at all. It seems like such a
self-indulgent thing to do, and, while in retrospect it has certainly turned
out quite well, there were some bumps along the way (to put it mildly) and I
did some things that I'm not altogether proud of.
On the other hand, I've always enjoyed
being on the receiving end of a good inside scoop, so now that I'm in a
position to share an experience that others seem to care about I feel like I
ought to do so. Besides, they say confession is good for the soul.
A very brief summary to set the stage: I
worked as a robotics and artificial intelligence researcher at the Jet
Propulsion Lab from 1988 until 2000 when I went to work for Google as a
software engineer. I was there for just over a year. I worked on two main
projects, the first release of AdWords, and a little known widget called the
Translation Console. If you go to Google's
preferences page you will see that Google is available in well over
100 languages, including Klingon and Elmer Fudd. (There used to be a Swedish
Chef option, but that seems to have succumbed to political correctness.) All
those translations are provided by volunteers around the world. The interface
they use to do the translating is the translation console (officially called
the Google In Your Language Program).
As far as I can tell it hasn't changed much since I wrote it. I'm pretty proud
of that.
I guess the #1 FAQ for people who have
left Google is why did you leave. My main reason for leaving was that I was
commuting from Los Angeles. I'd fly up on Southwest early Monday morning, fly
back on Thursday evening, and telecommute on Fridays and weekends. That regimen
was pretty stressful even under the best of circumstances, but when 9/11
happened it became completely untenable. I had already given my notice before
9/11, but I don't think I could have stayed on after that even if I had wanted
to. But I'm getting ahead of myself.
As I've said, my story is not entirely
rosy, so I want to make a sort of blanket disclaimer, and to do that I have to indulge
in a little bit of tooting of my own horn. I have a Ph.D. in computer science.
I was a senior scientist at JPL when I went to Google, and when I returned to
JPL I was promoted to Principal, the highest rung on their technical career
ladder. (Actually, it turns out there are higher rungs, but their existence in
not publicly known.) I am generally considered to be a pretty bright guy.
I am tooting my horn to put the following
assessment in perspective: at Googe, if I were to rate people on general smartness
I would have put myself in the bottom 25%. It was pretty much the first time in my life that I found myself
not at the top of the intellectual pecking order. It was not an easy adjustment
for me. But I'm getting ahead of myself again.
I wanted to say this up front because
since the IPO there has been a steady chorus of criticism along the lines of,
"Google has little real value and it's only a matter of time before it all
comes crashing down like a bad flashback to February 2000." In my humble
but better-informed-than-most opinion this is all sour grapes. Google is a
valuable company because the people who built it are incredibly smart and they
work incredibly hard. I feel priveleged to have been a (small) part of it.
There are many valid criticisms of Google (and I expect I'll be making some of
my own), but that they have built little of real value is not among them.
I'm not saying this because I want to kiss
up to Sergey (as one commentor suggested might be the motivation for one of
Doug's postings). I have no need to kiss up to anyone any more (and, though I
don't have any firsthand knowledge, I strongly suspect that Doug doesn't
either). I'm saying it because Google has taken a lot of bashing Some of it was
well deserved IMO (like when Google
blacklisted CNet for doing a story that included personal
information on CEO Eric Schmidt which was obtained by doing a Google search),
but most of it was (and is) not, and I just wanted to stand up and say so.
With that out of the way, I'll get on with
the story.
9
COMMENTS:
http://search-engines-web.com/
said...
...
if I were to rate people on general smartness I would have put myself in the
bottom 25%.
Hmmm,...how
are you defining & rating Intelligence ???????
What is of grave concern is - if everyone there is so-o esoteric, and
SEARCH has now become a "common persons" resource, Can they really
empathize and understand the needs and habits of the so-called less extreme
intelligent members of the Human Race who dominate their customers (and the
world)?
The interview where you were asked for your SAT and GPA scores makes
one wonder if they are only focussing a very LIMITED portion of Human
Intelligence.
Homo Sapiens are so varied and verstile, hopefully the channels
of communication will open up to include and understand the masses in what is
becoming a universal source of information retrieval!
Hopefully the masses
will ALSO be allowed to contribute to Google's evolution ;-)
Ron
said...
Hmmm,...how
are you defining & rating Intelligence ?
A good
question, and I'm actually planning a whole post to answer it.
Ron
said...
Oh, I guess I
should mention too that they didn't ask me my GPA or my SAT score. (I suspect
they reserved that for the non-technical folks.)
Milly said...
Ron, the Swedish
Chef lives! It's under "Bork, bork, bork!" in that prefs page
dropdown. It even had some utility for a year or so: selecting that language
brought back the old blue and white tabs (and less tricky Groups) design. But
they've fixed that now (hey, some security holes take 'em longer) :
www.imilly.com/google-cookie.htm#bork
Once
there's enough of you xooglers gathered, maybe you could do an exit reasons
survey. I wonder if many would cite the growth from jetski to supertanker as a
reason?
In July Marissa Mayer said "We are
planning on releasing some alternatives in the next one to two months that make
our current captchas more compatible with screen readers ...".
Yet
here we are with most Google services, Blogger comments included, still using
evil visually-oriented captchas: www.w3.org/TR/turingtest.
Apparently
it takes time for a supertanker to change course ...
Milly said...
Oh, and Doug -
how about an RSS feed for the comments?
Anonymous said...
I'm looking
forward to seeing the comments on intelligence as I am very interested in how
it manifests itself in terms of how Google makes decisions as well as how well
the technology works.
Here is an example. A long time ago, some Google
employees claimed that it was not possible for their index to be spammed
because their algorithms were powerful enough to defeat any spam attempts. I
think they did not take into account the economics of the Internet -- in
particular, that web sites (and therefore links) could be built rather cheaply.
What I want to understand is what aspect of intelligence does reaching this
type of conclusion fall into and to what extent it factors in the hiring
process.
Disclaimers: I have been turned down for employment by Google. No
sour grapes -- I realize it is like getting into Harvard; some very intelligent
people don't get in. Also, I used to work for another search engine that ran
into trouble and got sold.
Steve
said...
What
is of grave concern is - if everyone there is so-o esoteric, and SEARCH has now
become a "common persons" resource, Can they really empathize and
understand the needs and habits of the so-called less extreme intelligent
members of the Human Race who dominate their customers (and the world)?
Well, maybe
not..
http://www.infoworld.com/article/05/12/06/HNgoogleuserstudy_1.html
cayblood
said...
On the other
hand, there are so many obvious problems out there that don't necessarily
require UI skills that they can just work on those and hire a few good UI
people to write the interface to them.
Anonymous said...
I also worked
at Google and I understand the feeling of being in the bottom 25%. Everyone
there was smart, but I didn't feel that the top 25% were that smart, and I
didn't feel that the management was that smart either.
Basically, it felt like
a graduate research group. Everyone was good or they wouldn't be there. There
was a definite pecking order based on sucking up to the prof.
SUNDAY,
NOVEMBER 27, 2005
In the beginning...
POSTED BY RON AT 3:34 PM
The memory of my first encounter with
Google is seared into my memory like JFK's assassination is supposed to be
seared into the minds of many baby boomers.
As an aside, It's always seemed odd to me
that the baby boom generation is supposed to include anyone born through 1964.
Technically I'm a boomer, but JFK was long dead when I entered first grade. On
the other hand, I do clearly remember what the world was like before there was
Google, before there was the World Wide Web, and even before there were
personal computers. I am a member of the last generation to know what that
world was like. God, do I feel old now.
But I digress.
I was reading a usenet newsgroup (I'll
tell you which one later -- it bears on a peculiar personality trait of mine
that has some bearing on my Google experience later on) when someone answered a
particularly obscure question and followed up with "Thank God for
Google" or some such comment. (When Google acquired Deja and brought their
database of old usenet postings on line I went looking for the post that
started the whole thing and couldn't find it. This was just the first of many Twilight-Zonish
(or X-Filesish for you younger readers) events that happened to me of the
course of the next few years.)
Google? What's Google, I wondered. So I
pulled up a web browser and took a wild guess (which was as good a way as any
of finding things on the web in those days): www.google.com. Oh, it's a search
engine, kind of like Alta Vista. But, holy shit, it's fast! And it has this
uncanny way of putting just what you're looking for right at the top of the
results list.
To understand what happened next I have to
give you a little background about my professional life up to that point. I had
moved to California from Virginia in 1988 along with my Ph.D. thesis advisor, David Miller, who had just been offered a job at
JPL to do research for the Mars Rover program. Over the course of the next
twelve years my career had a lot of ups and downs, and I had a lot of
opportunities to leave JPL, but somehow the opportunities always seemed to come
along at the "up" times when life seemed good and I wasn't much
inclined to rip up my roots, which were growing deeper as the years went by.
But the day I found Google just happened
to be during a "down" time.
My first thought was, "How the hell
do they do that?"
Alta Vista was astonishing enough in its day, but this took speed and accuracy
to a whole new level. I had always been idly curious about how Alta Vista
worked, and now I just had to know. In a fit of what Alan Greenspan would have
called "irrational exuberance", I dashed off a resume.
Google got back to me with astonishing
speed. It was early 2000, the dotcom bubble was just reaching its peak, it was
a seller's market when it came to any kind of technical talent, and I looked
pretty good on paper. If memory serves, it was only about two or three hours
before my phone rang. A week later I was flying up for an interview.
I don't have nearly as many colorful
interview stories to tell as Doug did. I met with half a dozen people. (Neither
Larry nor Sergey interviewed me.) They grilled me on the usual things --
caches, hash tables, virtual functions, etc. It was a pretty standard technical
interview as best I can recall, with a few Googley twists (how would you write
a program that could identify news sites on the web?) I guess I must have
hornswoggled them pretty good because they made me an offer. (That may sound
like a bit of self-deprecating humor, but it isn't. The truth is I really
wasn't qualified for the job. But that didn't become apparent until later.)
So now I'm in a pickle. On the one hand
I've got this job offer and an opportunity to learn how this cool technology
works, and maybe even make a few bucks on the stock options (though that was
never the main motivation for going. It was pretty clear even in early 2000 that
the internet bubble was gong to burst sooner or later, and besides, how was a
search engine ever going to make money?) More to the point, I was worried that
if I didn't get away from JPL now I never would, and I didn't really want to
retire without ever having experienced anything but working for one
organization.
On the other hand, I had a pretty cushy
situation. My seniority at JPL made it possible for me to work on pretty much
anything I wanted to. I was well paid (by my standards at the time). My job was
(or seemed) secure. My wife and I had just bought a nice new house and gotten a dog
and a cat.
(The Southern California real estate market was just starting to pull out of a
slump and we were able to buy for what now seems dirt cheap.) To take this job
we would either have to move or I would be doing the commute from hell. Neither
of those prospects seemed very appealing.
I made my decision while visiting some
friends in upstate New York. One of my friends had just had her father die
unexpectedly. He was an orthopedic surgeon. Very wealthy, or so it seemed to me
at the time. Full of life. Commuted from his farm in Virginia to his job in his
own Bell Jet Ranger.
(I remember he flew in one night while I was visiting my friend on the farm. He
was a real regular down-to-earth guy. Asked me if I'd like to join him tomorrow
-- he was going to the Pittsburg Steeler's summer training camp. I'm not that
much of a football fan, but it was pretty damn cool anyway. There were fans
everywhere wondering who the hell we were that we got to go into all the VIP
areas.)
One day he just keeled over while skiing.
Heart attack. He was dead before they got him off the mountain.
It was during that reunion that I had an
epiphany: life is short, and I had everything I ever wanted: a nice house, a
secure job that I (mostly) enjoyed, why in the world would I want to put myself
through hell just to hang out at some dotcom that would probably be bankrupt in
a year or two?
When we got back from New York I called
Google and told them that I had decided to decline their offer.
13
COMMENTS:
Loughlan
said...
Must have been
a tricky decision. I wonder if knowing what you had known now, would you have
still done the same thing?
I realise it is probably a faux pas to post here.
But I wanted to mention what a wonderful piece of writing this is.
Good luck
with whatever you choose to do in life.
Kindest Regards
Loughlan Burnett
Anonymous said...
Fascinating!
Leah
said...
Hi. Please
don't remove the anonymous option for posting comments -- it allows for more
interesting people to feel comfortable posting. I see that you've already
reserved the right to remove comments; I think that's just, but please do so
sparingly. Just my HO.
Great blog, btw. The exaggerated lovefest that people
have had with Google gets tiresome.
Lucian said...
I'm kinda
facing the exact same situation, though probably not at the same scale or level
as you.
The thrill of working on something everyone uses excites me to no end.
I've a pretty cushy job at the moment; pretty inconsequential, but cushy.
I
wonder about family life should I decide to take on the new, more exciting job.
Whether I should just coast, and whether I'll regret that I never took the
plunge. That is the question.
Anonymous said...
Does anyone at
Google ever stop to think that 99.99% of the company's revenue is the result of
a cold hard reality-
The majority of websurfers out there don't have the
experience to understand that those adsense ads or google search result ads
aren't organic links?
There was a poll done by the BBC about 6 months ago that
showed 68% of surfers were unable to point out what links on a page were ads
and which were part of the content.
Banner ads had a huge clickthrough rate
when they were introduced, then plummeted to almost nothing after a few years
when even the newest of the newbies realized they were just advertisements.
Text ads right now seem to be enjoying the same "novelty" in that
plenty of people arent aware they are ads either.
Since almost every website
using adsense matches the colors and fonts and background of the ad panel to
look like the rest of the content, and google's engine puts ads that seem
related to the rest of the text on the page, is it any surprise that the
clickthrough rate is so high? 10 percent of Google's revenue comes from AOL
users, who have a clickthrough rate that's more than 5x higher any other
demographic on the net. That should tell you something.
Anyway, I guess my
question is, how long until even the AOL crowd catches on and clickthroughs
plummet?
Btw- why does it say ads by "Goooooooogle" ? Are you trying
to be cute or is there some reason dealing with spiders that you want to be able
to identify casual mention of Google from adsense ad mentions?
Doug
said...
I'll let Ron
address the issue of ad clickthroughs, though it seems pretty evident at this
point that the difference between banners and Google ads is relevance to the
user. While there's undoubtedly confusion in some instances about what's an ad
and what is not, the utility of the ads is unmistakable.
I can also state
without hesitation that I worked very closely with Google's partner team to
ensure that search partners followed our guidelines for clearly identifying
what was a sponsored link and what was a search result. I'll likely post about
those wars in the future.
As to "Ads by Goooooogle," we needed a way
to make Google ads unique and thus protectable under the law. This was not a
universally beloved way of doing that, but it seemed to be the best of the
options available.
Ron
said...
I'll
let Ron address the issue of ad clickthroughs
I'm not quite
sure what I'm supposed to address here. At least two issues were raised: 1) the
prediction that text ad clickthroughs are going to plummet the way banner ad
clickthrough did, and 2) the possibility that people don't know that the ads
are ads.
I don't have much to say about issue #2. The ads are pretty clearly
labelled, but I will not dispute that there are some pretty dim bulbs out
there.
As to issue #1 I'll defer to Carl Sagan who observed that prophecy is a
lost art.
Jorg Brown
said...
Turned 'em
down? Heh. For exactly the same reasons, I stayed as far away from dotcoms as I
could. Especially in 2000, when some of the stupider ventures had already
disappeared.
I eventually interviewed at Google in 2004, but only after two
friends of mine went, and I could grill them on whether or not it was really a
place they liked to work. Since Google had just gone IPO, and the stock had
shot from 85 to 120, I figured the easy money was already gone from stock
options. But my career wasn't going anywhere at work, and I didn't have to
move, so why not?
I still want to know what Google told you when you said no.
For me in 1998, I told Microsoft no, and they responded with a better offer.
Given the job market in 2000, I'm surprised that Google didn't do the same to
you.
Anonymous said...
IMO, banner ad
clickthroughs fell because they were (are) intrusive to the search engine
user's experience. Intrusive ads seem to work on sites like ESPN (at least, I
have not heard that they feel the need to abandon them). It would be
interesting to learn what % of their regular users also use Google.
I think at
some point the rate of "legitimate" clickthroughs will fall off (if
it hasn't already) as the medium matures and users become more selective in
their surfing and buying behavior.
Red said...
I think the
defining difference between Google ads and the more 'traditional' forms is
utility, as Doug suggests. Google ads can be so darned interesting that you
want to click on some, especially if you're searching hurriedly for product or
services. I find myself scanning the page, then automatically scanning the
Google ads alongside if I haven't found what I'm looking for. It's an
interesting twist on the ad form.
Stefan said...
"Some pretty dim bulbs"?
http://www.pewinternet.org/PPF/r/146/report_display.asp
Anonymous said...
WHAT IS THE
CONTINUATION OF THE STORY?????
DID U WORK FOR GOOGLE OR NOT?
nick
said...
great piece of
writing!
sorryabout your friend...
i'm wondering uif you could include some
pieces of writing for your spanish-speaking fans... specially the ones from
mexico, like me...
Kudos and good luck in life.
SUNDAY,
NOVEMBER 27, 2005
The wormgear turns
POSTED BY RON AT 5:07 PM
Ron sez:
It ocurred to me that by putting up my
story piecemeal like this most people are gong to be reading it in reverse order.
Hm. I wonder if there's a way to reorder postings -- or if people are so used
to this that they tend to read blogs backwards anyway. Wow, the things you
start thinking of.
Anyway...this is part three of my ongoing
saga of my year at Google.
Of all the random shit that has affected
my life surely none has had more impact than the fact that Google would not
take no for an answer. They asked if they could fly me up again to try to
change my mind. And they wanted me to bring my wife along as well. Well, heck,
a free dinner for two, how could I turn that down?
The person assigned to win me over was Urs Hoelzle, at the time VP of
Engineering. Three things impressed me about him. First, he told a really good
story about how Google was planning to make money (one that turned out to be
pretty much right on the money, so to speak). Second, his technical background
was in programming languges, which was a big plus for me for reasons that I
will explain shortly. And third, he had a really huge dog
that he brought to work with him. (Yoshka, the Leonberger, has since become the
stuff of legend.)
(Another interesting bit of trivia about
Urs: he always wore red socks. Something to do with him being Swiss, I believe,
but I never fully grokked it.)
To fully appreciate the horrible irony of
what happened next, I have to tell you about this peculiar affectation (or
perhaps the word I'm lookin for is affliction?) of mine: I am a fan of Lisp (the programming language, not the
speech impediment).
For those of you reading this who are not
programmers I have to digress even further and explain a little about
programming langauges. Programmers are craftsmen, and craftsmen can sometimes get a little weird about their tools.
It's funny to watch Tim Allen wax rhapsodic about the relative merits of the
Binford model 2330 frabnobulator because, like all good comedy, it contains
more than a grain of truth. Programmers are craftsmen, and programming
languages are their primary tools, and so it is not uncommon (understatement of
the day) to see spirited
discussion of the relative merits of this or that language.
One of the reasons that I had chosen to
stay safely cloistered in the ivory tower of JPL for over a decade was that my
favorite tool/language had definitively lost the language wars. I got into Lisp
in the 80's because it was the language used for artificial intelligence (AI)
work, which was the field I wanted to get into. Unfortunately, AI failed to deliver
on lofty promises, and so it fell out of
favor with its (mostly government) sponsors. As a result, industry
abandoned Lisp in favor of C, and later, C++, Java and Perl.
I don't want to rehash the language wars
here. Suffice it to say that I attributed a significant portion of my own professional success to my
use of Lisp over the years, and I was loath to give it up. But the industry had
moved definitively away from Lisp and towards C++ and, in the Web world, Java.
I found both of these languages very difficult and frustrating to master, and
that was a large factor in my decision to stay at JPL during most of the dotcom
boom. Why should I submit myself to all that pain when I had a perfectly good job
where I could do as I pleased?
But in Urs Hoeltzle I saw a beacon of
hope. I had done my homework on Urs and found that he had made a major portion
of his career in the
development of a language called Smalltalk, which was very similar to Lisp in
many important ways. In particular, it was not a mainstream language, so I
thought that in Urs I would find a kindred spirit, an engineer who knew what it
was like to work on and perhaps even fall in love with a programming language
that was not part of the mainstream.
My wife and I have different recollections
about what happened next. I have very strong memories of my telling her about
starting to change my mind about Google, and her responding that she wasn't too
enthusiastic about the prospect, but that if it was what I wanted to do that
she would support me. She insists that she never said any such thing.
It seemed like a good time to leave JPL.
Things were changing there and I was beginning to get somewhat marginalized on
the project I was working on. (That's a whole 'nuther long story. If you're
interested you can read it here.
Be aware that it contains some spoilers for the Google story, so if you're
hanging on to the edge of your seat waiting for the next installment here at
xooglers (yeah, right) you might want to give it a miss.) So I called Google
and we set a start date in June of 2000, and gave notice at JPL.
To my astonishment, people started coming
out of the woodwork to express their shock and dismay that I was leaving JPL.
Several people who I was sure would be happy to see me leave actually asked me
to reconsider. It did my ego a lot of good to hear it, but it was too late. I
was off to Mountain View.
13
COMMENTS:
Anonymous said...
But Erann, in
the end you didn't get to use Lisp (or Scheme, I presume). That's too bad! Is
it still the case that a lisper has no chance at Google? They have Norvig
there. Inquiring minds want to know.
Anonymous said...
Ron,
Sorry --
I got Erann from your JPL/Lisp/Google piece, and didn't see yet you'd changed
your name.
I'm really curious about languages at Google. E.g. is ML or CAML
used? As I'm sure you know, there is neat and powerful software that runs in
those languages. And Haskell too, I guess.
Thanks for any info -- and aren't
you afraid of getting sued by Google? I heard they are paranoid -- but not
psycho like Intel.
TraderEyal
said...
Hey Doug, Cool
blog. I really enjoy reading your story. As for reading the blog backwards, you
can consider putting a 'sticky' on top by way of a future dated post
recommending readers to start from the oldest post..
All the best
TraderEyal
Ron
said...
Is it
still the case that a lisper has no chance at Google? They have Norvig there.
Peter Norvig came to Google from NASA's Ames Research Center to be
Director of Search Quality. Peter was well known in the Lisp community, having
written am excellent textbook on the topic
but he, to my dismay, was already abandoning Lisp in favor of Python and Java long before he came
to Google.
As far as whether a Lisp programmer has "no chance" at
Google (by which I assume you mean no chance of being allowed to use Lisp), I
have no idea. It's been nearly five years since I left; perhaps things have
changed. But I wouldn't count on it. BTW, it is not at all clear to me in
retrospect that using Java was not the right thing to do from the company's
point of view. More on that in a later post.
aren't
you afraid of getting sued by Google?
No. Fear that the
First Amendment no longer applies is a self-fulfilling prophecy.
Philipp
said...
Yep, I'm
currently reading the last few days' posts backwards... but once you're up to
speed you can read the most recent post everyday and that means it'll be in
chronologically correct order. That being said, I believe there's an option
over at blogger to order things chronologically, but I'd rather turn this into
a book later on (via Lulu.com or BlogBinders.com or even better, a publisher).
Phil
said...
Nat Friedman's blog has the first page with
newest-first, and all previous months in chronological order. It's a bit
disorienting at first, but it makes a whole lot more sense. I haven't gotten
around to doing that on my own blog, but I really should.
David said...
I actually
prefer having the blog with most recent posts first. Otherwise when I came, I'd
have to page through old entries.
And yes, I do start at the bottom and work
my way up. ;)
Charles
said...
I hope you
don't mind but I created a new RSS feed which will spill out the posts in the
opposite order (one per day), ie. the oldest posts first then moving forward.
Details here:
http://www.surfarama.com/?p=266
It sort of serializes the
story. I'll be adding new posts they come.
Doug
said...
Hey, thanks!
Neither Ron nor I know anything about RSS, and we had it way down the list of
things to figure out. And to prove I'm clueless, let me ask you this: can I
just add a link under the Atom feed link on Xooglers to the page you've
set up? Is that what people would expect if it the link were labeled "RSS
feed"?
Charles
said...
Just for
everyone elses interest, I have sent code to Doug to display the serialized RSS
feed link (as opposed to using the link above. Reason being it requires some
javascript to generate unique urls so it can deliver feed content in a
serialized fashion.
Or you can visit my blog to subscribe here.
Anonymous said...
Reading the
blog backwards on a computer screen ain't that big of a deal. However, I don't
read it on a screen, I actually print it out every other day (in order to read
it at home or/and on the bus), and this is where things start getting really complicated. So yes, I would love to be
able to re-sort the entries prior to printing.
Ah, and by the way: this is a
darn good blog, let me tellya that.
Isaack
said...
I was
recommended your blog from another blogger. Once i got here, I saw that I had
to take some time getting it from beginning, so I had the same browser with your
page on opened for two days before I started it... now, an hour ago. I've
gotten to to this point and really enjoying myself!
With the backwards
posting... I just started reading cronologically, scrolled all the way down and
started from there.
Anonymous said...
Ron, another
admirer of Heinlein. Hopefully science fiction is not dismissed as BS at Google
SUNDAY,
NOVEMBER 27, 2005
Day One
POSTED BY RON AT 11:34 PM
Ron sez:
My first day at Google went like this:
5:30 AM, wake up.
6:00 AM drive to Burbank airport.
6:30 AM stand in line at the Southwest
airlines gate to get one of those now defunct plastic boarding passes
7:30 AM board the plane
9:00 AM or so land in San Jose
9:30 AM get the rental car
10:00 roll up to the Googleplex
It was a murderous commute. Four hours,
which I eventually whittled down to three after a year of optimizing. I went
through various permutations of cabs, rental cars, having my wife drive me to
and from the airport (that didn't go over well at all) before I finally settled
into something that vaguely resembled a routine. I ultimately ended up driving
two different cars back and forth between LA and San Jose, and on a couple of
occasions I did my commute in a Cessna 182RG (I'm a pilot) which was just too
cool for words.
But getting to San Jose wasn't nearly as
bad as trying to get back home. Originally I took Monday to be my telecommute
day so that I could be at Google on Friday afternoons for the weekly TGIF
meeting. TGIF is, as far as I know, still a Google tradition where everyone
gets together to hear the latest developments from upper management (which back
then meant hearing it directly from Larry and Sergey. Maybe it still does, I
don't know.)
Trying to get on a Southwest flight at the
San Jose airport on a Friday evening in mid-2000 was a serious nightmare. Heck,
just trying to get to the airport
was a nightmare. (I remember on one occasion the traffic was so gridlocked that
I could not get to the rental car return, which back in those days was still in
the main parking structure. I just left the car in a remote parking lot and
walked from there to the terminal to make my flight. I guess the rental car
company eventually found the car because only the normal rental charges ever
showed up on my credit card.)
I found a place to stay, renting a room
from Susan Wojcicki (now Google's director of product management) and her
husband Dennis. Really cool people, and coincidentally, their house is also the
one where Google first set up shop, so I really felt like I was in the bosom of
history.
In the back of my mind there was one
niggling little worry: I had neglected to ask Urs what I would actually be
doing. Having decided to embark on this adventure I guess I decided it didn't
matter. I was going to do something new and different, and, more to the point,
I was going to learn how this incredible search technology worked. And I was
going to knock Urs's red socks off with all the cool ideas I had for new
features.
Things started off really well. The work
environment at Google was everything that it has since been reputed to be. I
learned to ride a unicycle while I was there, and my pool game improved
considerably. My office had a spectacular view.
Chef Charlie's cooking was delicious. (I still make some the recipes he gave
me, but I can never get them to come out nearly as good as he did. I hear he's
planning to open up his own restaurant soon. I'm looking forward to patronizing
it.) The conference rooms were all color coded, except for the main one which
was called the Lucky Lounge. Meetings, in stark contrast to my experience at
JPL, started on time. It was all very refreshing and energizing. I was brimming
with enthusiasm.
So I was a little disappointed when I
found out on day 1 that I had been assigned to the ads group. But that
disappointment turned to dismay when I learned what my assignment was to be: I
was the lead engineer on a new advertising system code named
"adstoo", what eventually became AdWords. That part wasn't so bad.
The bad part was, this was going to be the inaugural Java project at Google.
Google, which had until now been a Java-free zone (which was one of the reasons
I took the job) was going for Java in a big way, and I, the consummate Java
hater, was supposed to be its chief evangelist.
Just peachy.
On top of that there was trouble at home.
My wife was having a really tough time trying to keep everything together back
on the home front, what with the new house and four-legged kids and all. Our
finances were starting to look a little rocky because I had taken a pay cut and
taken on extra expenses for weekly plane tickets, rent, and a second car.
(Google did give me a travel allowance, one of the factors in my decision to
take the job, but my salary and travel allowance together were still less than
my old salary had been at JPL.)
And then there were the stock options.
Yes, stock options can make you rich, but
getting to that point can be pretty damned annoying. When you sign on to a
company that has stock options as part of the compensation plan you get an option grant up front, but you don't actually vest any of those options until you've been at
the company for a year. After that you vest a few more shares every month, but
if you quit -- or get fired -- after eleven months and twenty nine days you're
screwed.
Notwithstanding all that, you can actually
choose to excercise your options up front even though they aren't vested, and
it's often wise to do so because if you wait and the stock price goes up then
when you do exercise the options then the difference between what you paid and
what they were worth at the time counts as income for purposes of the
alternative minimum tax despite the fact that you might be
restricted from selling your shares because, for example,
your company hadn't gone public yet. A lot of people got badly burned during
the dotcom crash because they exercised their options when they couldn't sell
the stock, generating a huge amount of AMT income on which they owed taxes.
Then they watched helplessly as the stock price crashed, often to levels below
what they had paid for the stock. So not only did they lose some or all of the
actual cash they had paid to exercise the options, they also owed huge tax
bills on paper "profits" that not only had they never actually
earned, they could not have earned even if they wanted to because their stock was restricted. It was
horribly unfair.
On the other hand, if you vest in your
options but don't exercise them and then leave the company then the options expire
and become worthless. Since I wasn't really planning to stay at Google for the
long haul I figured it would be wise to exercise early to avoid the AMT trap.
But to do that I had to come up with the cash to actually pay for the stock.
And to add insult to injury, the day I joined Google they announced that they
had finalized their deal with Yahoo, and the board raised the stock price by a
factor of nearly 5! So not only did I have to come up with the cash, I had to
come up with five times as much cash as I was expecting to have to come up
with! My wife was not pleased.
In retrospect of course it all turned out
OK, but at the time it was nerve-wracking. The dotcom crash was gathering
steam, and Google's long-term prospects were far from clear. Still, growth was
steady and there was electricity in the air, so I was fairly optimistic, even
at the time. But writing that check was painful.
All in all, it was a stressful week.
7
COMMENTS:
Uri L. said...
A great post!
thanks for sharing.
What a manic commuting routine...
Kiltak
said...
And I tought
that 3 hours of car each day was bad enough. Personally, travel time is killing
me, so taking the plane each week to go to work would be out of the question.
Especially with a demanding wife and a baby.
You were very courageous my
friend ;)
Tuor
said...
tuor sez:
the
doug / ron thing is not working very well, and its sort of confusing to read
things the way they are (though things are, of course, extremely intersting,
otherwise I wouldn't be here) As one of the chief complainers on blogger, I
note that MovableType has had a category function for the longest time now, so
that you can only view things by category (e.g. click on "Ron" link
and see only Ron's posts). Maybe you guys should go off Google (or companies
that have been bought by Google, though I suppose its pretty much the same
thing) and try some open-source fun!
Of course, despite my complaining I'm
still here using blogger, so I'm not sure what right I have to make
suggestions. Take it all with a grain of salt, if you will.
Anonymous said...
Thanks for
sharing all those "ancient" moments of Google with us.
"TGIF
is, as far as I know, still a Google tradition where everyone gets together to
hear the latest developments from upper management (which back then meant
hearing it directly from Larry and Sergey. Maybe it still does, I don't
know.)"
Yes, it still does. :-)
Anonymous said...
Surprised you
didn't take Caltrain or the light rail to get to/from SJC.
Taco
said...
iTs a great post!
you should change the way the comments are viewed. allowing comments on a pop
up window make me read both, all other post and comments because the main page
takes a hwile to load because the text is too long!
Chris Marino
said...
Ron, not
surprised that you were at Google only a year. As important as AdWords is, I
don't think it can provide the ongoing technical challenge that is necessary to
keep the best people engaged.
MONDAY,
NOVEMBER 28, 2005
Into the breach
POSTED BY RON AT 12:36 PM
Ron sez:
I dove into the adstoo project with as
much enthusiasm as I could muster, which I'm ashamed to say wasn't much. The
situation was exacerbated by the fact that we had no Java development
infrastructure. We were writing bareback, so to speak. We had no debugger. We
were using JSP, but had no editor support for JSP syntax. (That turned into a
real debugging nightmare. It could take many tens of minutes to find a single
typographical error because the only indication that there was a problem was
that the output just looked all wrong, but the actual problem could be very far
away from the apparent source of the problem.)
Fortunately for me, I was assigned a
junior engineer to work with/for me, and he actually knew what he was doing.
While I struggled to learn the Java libraries and debugging techniques (I knew
the basic language, but I had never done any serious development in it before)
this guy just took the bull by the horns and pretty much just wrote the whole
damn thing in a matter of weeks. I sometimes pull this old joke out of the
dustbin, that in the ancient tradition of senior-junior relationships, he did
all the work and I took all the credit.
That's not quite true. I did end up
writing the credit card billing and accounting system, which is a nontrivial
thing to get right. Fortunately for me, just before coming to Google I had
taken some time to study computer security and cryptography, so I was actually
well prepared for that particular task. Back in those days internal security
was more or less nonexistent. All the engineers had root access to all of the
servers. But I believe in planning ahead, and I anticipated the day when Google
was not going to be a small company any more, and so I designed the billing
system to be secure against even a dishonest employee with root access (which
is not such an easy thing to do). I have no idea if they are still using my
system, but if they are then I'd feel pretty confident that my credit card
number was not going to get stolen.
But on the whole I was struggling, not
just technically, but personally as well. The situation was exacerbated by my
manager, who thought that the answer to my falling behind schedule was to start
micromanaging me. That just made things worse. Much worse. I had been a senior
scientist at a national research lab, essentially the equivalent of a tenured
professor (but without the teaching responsibilities). I was used to being my
own boss more or less, and I really resented being asked to make detailed lists
of everything was going to do every single day. After all, Google had come
asking me to join them, and so I felt I ought to be accorded
more respect. But on the other hand I had to admit that I was not really
performing, and so all the micromanagement was in some sense justified. I
responded, I'm ashamed to admit, with some passive-aggressive head games, and
so the situation deteriorated rapidly to the point where my manager and I were
barely on speaking terms.
Things were made worse by the fact that I
had been assigned an office mate who was also new to Google, and who was not
part of the ads group. Most of the other ads group members were sharing offices
(or cubicles) with other ads group members, and so I felt I wasn't really part
of the club. On top of that, I was away from home and didn't really have a life
up there in Northern California. The stress mounted. I started to get paranoid
that I would get fired before reaching the one-year mark. I started
experiencing stress-related health problems, some of which are still with me
today. On more than one occasion I came that close to quitting. To this day I have no idea why I didn't.
It was about this time that I had my one
and only meeting with Larry Page. It was to discuss the progress of the adstoo
project and to set a launch date. My manager was there along with a couple of
other people (including Doug I think). Things went smoothly until Larry
suggested changing the way billing was handled. I don't remember the details,
but my response was that this would be significant work. No one challenged me,
but I found out later that the reaction of people in the room was something
along the lines of, "Is he crazy? This ought to be a trivial change."
This little incident turned out to have very far ranging repurcussions later,
but that will have to wait for the next blog entry.
Somehow we actually managed to launch
AdWords on schedule, in September of 2000. It still seems like a bloody
miracle. Most of the credit goes to Jeremy, Ed and Schwim. It could not have
been done without them.
I can still remember watching the very
first ad roll in. It was for a company called Lively Lobsters. Two months ago,
after five years of intending to do so, I finally bought myself a little toy
stuffed lobster to commemorate the occasion. (Update on 12/9/2005: It appears
that Lively Lobsters has gone out of business. There's some irony for you.)
About two weeks later all hell broke
loose.
13
COMMENTS:
Philipp
said...
You got me
really curious now...
Zoli Erdos
said...
Ron, this
should be illagal! Like in soap operas, at the most exiting line...stay tuned
.. ahhhhhhhh :-)))
http://search-engines-web.com/
said...
One bit of
information that would really be helpful to clarify
Who came up with the idea
of SPONSOR LINKS
The (two, now three) listings ABOVE the SERPs with the
pastel background color..
Was there any concern/debate about ethics.
Whose
idea was it to create the ADWORDS BOXES -then years later take them away.
The Google Sandbox (suggestion) tool -
why did it NOT include numbers - like the inventory.overture tool?
How did the
transition from Pay per View - to pay per click happen?
AP
said...
It appears that
the boys at google didn't realize Ron's talent and lost out eh?
John
said...
Man, I am
hooked.. very interesting.. keep going...:-)
Anonymous said...
I have a
question about the 20% time to pursue one's own interests. Must the project be
a totally different project or may someone just do more work on their current
projects?
netron
said...
aah man - this
is like one of those cliffhangers that you see in soap operas...
compulsive
reading!
Anonymous said...
Ron,
The
lisper in me wonders why you didn't write a lisp program to generate your Java
program -- at least then, you'd get to use all your lisp tools/skills, and the
stuff would be syntactically correct (as long as it didn't "blow out"
the girly-man Java development tools).
But then again, you'd probably have
only gotten into big trouble for that sort of smartass behavior.
Also, I
sympathize with your AMT/options/salary woes. A lot of people can't deal well
with making monetary sacrifices for risky businesses -- it keeps them up at
night.
If you own risky securities, you can always sell them and buy
less-risky ones (but perhaps get whacked by the 365-day capital gains tax) --
but if the risky securities come as part of your job, you are really stuck. And
you got screwd with the 5x revaluation! What a shame!
Anonymous said...
Your blog is
interesting. The Google persona I read about on tech sites (being super
friendly and laid back) seems to be at odds with your experiences. There is a
lot more business than engineering involved in your work there.
-Ivan
Kevin
said...
I liked the
lobster part
markg
said...
Doug, When do
we get our Xooglers Tee Shirts. Love the stories can't wait until you start
telling about the Google Ski trips to Squaw Valley!!
Ryan said...
Ron,
Here is
some more irony for ya... I founded Lively Lobsters in 1998 and was the guy who
placed the first AdWords ad you speak of. I am not sure if you are aware, but a
nice young lady named Allegra emailed me requesting a high-res version of my
company logo for use on some t-shirts being made for the engineers to
celebrate. (I never received one... I have asked around and heard mixed news on
whether they were ever made, and if they were, I would love to have one now!)
You correctly stated that Lively Lobsters went out of business, but I am
reviving it this fall. The only reason I shut it down in the first place was
because I became so involved with other profitable AdWords advertising that it
no longer made sense to focus on Lively Lobsters. These other AdWords ventures
have kept me a faithful advertiser ever since, and I am one of Google's top
spenders all time, having purchased literally many millions of dollars worth of
clicks through my various companies. The power of AdWords itself is responsible
for my success, and those very fruits are being used to resurrect Lively
Lobsters. In a nutshell... your system (AdWords) made Lively Lobsters, then me,
a success and now that success is coming back to revive Lively Lobsters! I am
not sure if this is interesting to you or anyone else, but I thought I'd share
my story when I came across this blog via a Google search for "Lively
Lobsters" while satisfying my curiosity about references to Lively
Lobsters still floating around. Thanks for everything you did for Google and
AdWords... I owe ya some long-overdue lobsters. Get in touch with me and they
are yours!
Ron
said...
Hi Ryan,
Wow,
what a great story! Glad to hear everything worked out so well. If you are
spending millions on AdWords then you must be doing well for yourself indeed.
Congratulations! And glad to hear the Lively Lobsters is rising from the
broiler. Maybe I can return the favor by placing the first order on the new
site :-)
I owe ya some long-overdue lobsters.
Get in touch with me and they are yours!
Thanks! We'd gladly
take you up on your offer, but we don't know how to reach you! Would you please
send a clue to xooglers at gmail? Doug has some things he wants to ask you
privately as well.
MONDAY,
NOVEMBER 28, 2005
The billing disaster
POSTED BY RON AT 7:48 PM
The AdWords launch went fairly smoothly,
and I spent most of the next two weeks just monitoring the system, fixing
miscellaneous bugs, and answering emails from users. (Yes, I was front-line
AdWords support for the first month or so.)
The billing system that I had written ran
as a cron job (for you non-programmers, that means that it ran automatically on
a set schedule) and the output scrolled by in a window on my screen. Everything
was working so well I didn't really pay much attention to it any more, until
out of the corner of my eye I noticed that something didn't look quite right.
I pulled up the biller window and saw that
a whole bunch of credit card charges were being declined one after another. The
reason was immediately obvious: the amounts being charged were outrageous, tens
of thousands, hundreds of thousands, millions of dollars. Basically random
numbers, most of which no doubt exceeded people's credit limits by orders of
magnitude.
But a few didn't. Some charges, for
hundreds or thousands of dollars, were getting through. Either way it was bad.
For the charges that weren't getting through the biller was automatically
shutting down the accounts, suspending all their ads, and sending out nasty
emails telling people that their credit cards had been rejected.
I got a sick feeling in the pit of my
stomach, killed the biller, and started trying to figure out what the fsck was
going on. (For you non-programmers out there, that's a little geek insider
joke. Fsck is a unix command. It's short for File System ChecK.)
It quickly became evident that the root
cause of the problem was some database corruption. The ad servers which
actually served up the the ads would keep track of how many times a particular
ad had been served and periodically dump those counts into a database. The
biller would then come along and periodically collect all those counts, roll
them up into an invoice, and bill the credit cards. The database was filled
with entries containing essentially random numbers. No one had a clue how they
got there.
I began the process of manually going
through the database to clean up the bad entries, roll back the erroneous
transactions, and send out apologetic emails to all the people who had been
affected. Fortunately, there weren't a huge number of users back then, and I
had caught the problem early enough that only a small number of them were
affected. Still, it took several days to finally clean up the mess.
Now, it's a complete no-brainer that when
something like that happens you add some code to detect the problem if it ever
happens again, especially when you don't know why the problem happened in the first place. But I
didn't. It's probably the single biggest professional mistake I've ever made.
In my defense I can only say that I was under a lot of stress (more than I even
realized at the time), but that's no excuse. I dropped the ball. And it was
just pure dumb luck that the consequences were not more severe. If the problem
had waited a year to crop up instead of a couple of weeks, or if I hadn't just
happened to be there watching the biller window (both times!) when the problem
cropped up Google could have had a serious public relations problem on its
hands. As it happened, only a few dozen people were affected and we were able
to undo the damage fairly easily.
You can probably guess what happened next.
Yep. One week later. Same problem. This time I added a sanity check to the
billing code and kicked myself black and blue for not thinking to do it
earlier. At least the cleanup went a little faster this time because by now I
had a lot of practice in what to do.
Stress.
And we still didn't know where the random
numbers were coming from despite the fact that everyone on the ads team was
trying to figure it out.
6
COMMENTS:
Anonymous said...
There's no need
to apologise for your geekiness -- as if we don't all know what chron and fsck
are.
I'd save the apologies and explanations for the "Steven Levy"
style book. I'd guess that most know to go look at things like this for definitions.
Jan
said...
There are still
problems with creditcards being declined without any reason - perhaps its all
your fault :)
PS: support-team still can't explain the problem
Philipp
said...
Uhm, just to
contrast the first comment, yeah thanks for letting us known what Fsck means
right here and now without us having to jump around websites too much :)
Who
says you can't read a Steven Levy book online? Plus, I've got a suspicion this
will turn into a book one day or another, and then it will come in handy to not
rely on links for crucial points :)
Greg Linden
said...
I'm curious,
what database were you using at that time?
Anonymous said...
The problem is
still there , they are supposed to bill every other day or so but sometimes
they will let it accumulate and the bill the card - But many times the
transaction get declined (as many cards have a 10k limit for a single
transaction)
Anonymous said...
PS. fsck is
file system consistency checker.
WEDNESDAY,
NOVEMBER 30, 2005
Racing to the finish
POSTED BY RON AT 8:26 PM
Ron sez... oh wait, don't need that any
more.
OK, time to wrap up this little soap
opera.
The problem turned out to be something
called a race condition, which is one of the most pernicious and difficult kinds of bugs to
find. (Those of you who are technically savvy can skip to the end.)
Most modern server code is multi-threaded, which means that it does more
than one computation at once. This is important because computers do more than
just compute. They also store and retrieve information from hard disks, which
are much, much slower than the computers. Every time the computer has to access
the disk things come to a screeching halt. To give you some idea, most modern
computers run at clock speed measured in gigahertz, or billions of cycles per
second. The fastest hard disks have seek times (that is, the time it takes the
drive to move the read/write head into the proper position) of several
milliseconds. So a computer can perform tens of millions of computations in the
time it takes a hard disk just to get into position to read or write data.
In order to keep things from bogging down,
when one computation has to access the disk, it suspends itself, and another
computation takes over. This way, one computer sort of "pretends"
that it is really multiple computers all running at the same time, even though
in reality what is happening is that one computer is just time-slicing lots of
simultaneous computations.
The ad server, the machine that actually
served up ads in response to search terms, ran multi-threaded code written in
C++, which is more or less the industry standard nowadays for high-performance
applications. C++ is byzantine, one of the most complex programming languages
ever invented. I've been studying C++ off and on for ten years and I'm still
far from being an expert. Its designers didn't really set out to make it that
complicated, it just sort of accreted more and more cruft over the years until
it turned into this hulking behemoth.
C++ has a lot of features, but one feature
that it lacks that Lisp and Java have is automatic memory management. Lisp and
Java (and most other modern programming langauges) use a technique called garbage collection to automatically figure out when
a piece of memory is no longer being used and put it back in the pool of
available memory. In C++ you have to do this manually.
Memory management in multi-threaded
applications is one of the biggest challenges C++ programmers face. It's a
nightmare. All kinds of techniques and protocols have been developed to help
make the task easier, but none of them work very well. At the very least they
all require a certain discipline on the part of the programmer that is very
difficult to maintain. And for complex pieces of code that are being worked on
by more than one person it is very, very hard to get it right.
What happened, it turned out, was this:
the ad server kept a count of all the ads that it served, which it periodically
wrote out to the database. (For those of you wondering what database we were
using, it was MySQL, which leads to another story, but that will have to wait
for another post.) It also had a feature where, if it was shut down for any
reason, it would write out the final served ads count before it actually quit.
The ad counts were stored in a block of memory that was stack allocated by one
thread. The final ad counts were written out by code running in a different
thread. So when the ad server was shut down, the first thread would exit and
free up the memory holding the ad counts, which would then be reused by some
other process, which would write essentially random data there. In the
meantime, the thread writing out the final ad counts would still be reading
that memory. This is why it's called a race condition, because the two threads
were racing each other, with the ad-count-writer trying to finish before the
main thread freed up the memory it was using to get those counts. And because
the ad-count-writer was writing those counts to a database, which is to say, to
disk, it always lost the race.
Now, here is the supreme irony: remember the
meeting with Larry where he wanted to make a change to the billing model that I
said would be hard and everyone else in the room thought would be easy? The bug
was introduced when the ad server code was changed to accommodate that new
billing model. On top of that, this kind of bug is actually impossible to introduce except in a language with manual memory
management like C++. In a language with automatic memory management like Java
or Lisp the system automatically notices that the memory is still in use and
prevent it from being reused until all threads were actually done with it.
By the time this bug was found and fixed
(by Ed) I was a mental wreck, and well on my way to becoming a physical wreck
as well. My relationship with my wife was beginning to strain. My manager and I
were barely on speaking terms. And I was getting a crick in my neck from the
chip I was carrying around on my shoulder from feeling that I had been
vindicated in my assessment of the potential difficulties of changing the
billing model.
So I went to my manager and offered to
resign from the ads group. To my utter astonishment, she did not accept.
18
COMMENTS:
Anonymous said...
Nice story. But
I wish you did not end most of your posts with dramatic phrases like 'utter
astonishment'. It is a really sophomoric style for a narrative.In fact, it
reeks of the Dan Brown style that I totally detest.
Anonymous said...
Dude, it's
either a nice story or not. Stop nit-picking to show off and just enjoy!
justin flavin said...
yeah - stop nit
picking. just sit back and enjoy the cliff hangers.
Anonymous said...
Actually, I
doubt Ron is exaggerating at all when he says "utter astonishment."
Lispers
are perfectionists -- and they drive lesser programmers nuts with their
well-deserved superior attitude.
It doesn't surprise me that after Ron screwed
up ("failed") because they made him use a lousy language, he felt
disgusted with himself and the job, and figured that they wanted him
out.
Indeed, if he'd been working with a lisper and the guy made an error with
similar ramifications on a NASA mission, Ron would probably be disgusted with
the loser.
But most normal programmers use lousy tools, and they tolerate
errors more than the perfectionists. They were probably just psyched that Ron's
stuff worked and was making them some real money.
But at no point did they
ever decide that they needed to start using Lisp instead of their second-rate Fortran-derived
languages. [Ron -- you can see I've suffered at the hands of management too.]
Anonymous said...
Just curios
whether it is on Microsoft or Linux.
Anonymous said...
I'm surprised
Larry and the others didn't realize this was a difficult problem requiring a
careful design, testing, etc.
Brinke
said...
great story,
but i totally have a headache after reding the C++ nightmare part. Time for a,
uh, nap.
Adi Oltean
said...
Nice story.
Multithreaded
programming is not for everyone. Finding bugs in MT code, or designing some new
MT code require experience, a certain state of mind, and an ability to analyze
methodically a large set of possible cases. Whenever I have to write C++/C# code
that uses shared data structures by two or more threads, I am 10x more careful
than usual.
I can certainly say that it takes years to become somewhat
familiar with multithreaded programming, especially in things like finding
common MT bugs, writing efficient MT code that won't trash the L1 cache or
introduce false sharing in multi-proc systems, dealing in a defensive way with
shared data (life-management, ownership issues of global objects on the heap,
etc), and even more advanced topics like lock convoy issues, lock-free MT
programming, etc.
And, no, don't even think that a high-level language like
Java is going to automatically solve all your problems here. You can have can
be lots of suble MT bugs in Java/C# code as well.
Anonymous said...
Ugh, the
nigthmare of race conditions. The story gives me shivers just thinking about
when I had to track down similar bugs.
Anonymous said...
Reminds me of my
RA during my MS for some reason... Great posts, I'm addicted, keep it up.
Anonymous said...
Definitely a
great post. Carefully choosing language/platform is very difficult in business
environment. I had an internship at of the giant software companies this
summer, and was quite surprised by some of the executive decision made by
managers. Essentially I had to design a test framework for a system that set up
a certain environment for the production code. I wanted to do it in C# or perl,
but everyone on the team favored C++ (with no STL libraries!!). As a result, I
spent 3/4 of my time writing string manipulation functions, implementing
various storage containers, and looking for memory leaks. At the end of the
internship I got most of the things to work, but my life literally went to
hell.
Anonymous said...
Definitely a
great post. Carefully choosing language/platform is very difficult in business
environment. I had an internship at of the giant software companies this
summer, and was quite surprised by some of the executive decision made by
managers. Essentially I had to design a test framework for a system that set up
a certain environment for the production code. I wanted to do it in C# or perl,
but everyone on the team favored C++ (with no STL libraries!!). As a result, I
spent 3/4 of my time writing string manipulation functions, implementing
various storage containers, and looking for memory leaks. At the end of the
internship I got most of the things to work, but my life literally went to
hell.
Anonymous said...
Dude, you
worked for a woman?!!! Do they have those in geekdom?
roschler
said...
It is nice that
Java has automatic memory management and for many, many apps it is a better
choice for fast development without memory management headaches.
But it's lack
of C++ style destructor brings in a host of problems of its own, especially
when it comes to managing lockable resources. Destructors and the intelligent
use of smart pointers (or shared pointers) can go a long way to making resource
management much easier. Last I checked Java's finalizer wasn't much help here
and in fact introduces problems of its own.
I agree that C++, especially when
people try to turn STL into a borderline functional programming language, can
be a nightmare. If someone could create a language with automatic memory
management and C++ style destructors, that would be great.
Multi-threaded
programming will always be a huge precarious chore until the Einstein of
operating systems builds something we haven't seen yet into the next big O/S.
Anonymous said...
use PERL
saurav said...
This kind of
reminds me of the pain i went thru writing a multithreaded external sorter for
a dual cpu machine.
Race conditions in multithreaded programs can be
nightmarish. I wonder if anyone knows of a modelchecker
for C++ which can allow
us to express and validate multithreaded protocols to avoid such problems.
Joshua Smith
said...
"But at no
point did they ever decide that they needed to start using Lisp instead of
their second-rate Fortran-derived languages."
Of course, C++ is not
Fortran-derived. It's ALGOL-derived: ALGOL begat Pascal begat C begat
C++.
ALGOL is fundamentally different from Fortran in that ALGOL was the first
computer language defined in BNF.
Anonymous said...
"I wonder
if anyone knows of a
modelchecker for C++ which can
allow us to express and
validate
multithreaded protocols to avoid
such problems."
I think
Coverity Prevent can
detect some of the problems
discussed in this
thread
coverity.com/concurrency/index.html
THURSDAY,
DECEMBER 01, 2005
Interlude
POSTED BY RON AT 10:07 AM
I'm going to be away for a couple of days
so no new posts from me before the weekend. I did want to say before I went,
though, that it may seem that I am, as Doug put it, a disgruntled ex-employee.
I know it's not apparent from the story so far, but I am in fact quite gruntled
(now there's a word for you), and not just because everything turned out well
in the end. Even at the time (well, more towards the end) I thought that my
experience at Google was on balance very positive, though it won't be apparent
why until I get a little further along in the story.
I wanted to say this up front because I
got an email today from my old manager, the one I've been kind of harshing on
in previous posts, and that made me realize that I may have some people worried
that I'm writing all this because I have an axe to grind. I don't. It's just
coming out negative now because it started out that way, and because, frankly,
negative experiences make much more interesting stories. I'll have more to say
about that later too.
Thanks for bearing with me. Hasta luego!
10
COMMENTS:
http://search-engines-web.com/
said...
It is vital
that this information be publicized - not only for historical research, but so
that Managers and Administrators can discover what imperfections exists, and
tweak the atmosphere.
Most would be less likely to risk offending some
hierarchy by confronting them while you are a subordinate.
Whan a company is
constantly seeking the creme' de la creme' of Engineers and Programmers - they
have an obligation to be as concerned with the emotions and frustrations of
this elite workforce - as much as they are concerned with perfecting the Search
Engine Softwares.
From a historical perspective, it will be another piece of
the puzzle in the history of one of the most globaly influential companies in
the history of the World Wide Web.
The motives do
not automatically invalidate the validity of these concerns being expressed,
and archived for future generations.
John Vanhara
said...
I cannot even
imagine someone would say that you are disgruntled ex-employee. It doesn't feel
like that at all. Maybe that's because I am European, but I am used to much
harsher criticism and it's still considered OK. Nothing is perfect in this
world. Everybody is doing mistakes. Bad stuff and good stuff happens all the
time. So there is no reason to be too careful about how to explain what
happened. I have very positive feeling from the way you explain everything. It
is really good story.
wfzimmerman
said...
P.G.
Wodehouse's Bertie Wooster may have been the first modern figure to use
"gruntled" as a verb. You are following in heroic footsteps.
Anonymous said...
The flood is
over,
the land is dry,
so why do you wear,
such big balls to write all this
stuff.
amar said...
just wanted to
let you guys know that i'm enjoying reading your recollections. you don't come
across as disgruntled employees at all. looking forward to hearing more.
KenP
said...
"Do no
Harm."
Interesting post that isn't on topic but what the heck...
Seems
the bad guys over in Iraq have a GPS and a laptop with Google Earth with them
most of the time. They know where they are and they know the land's appearance
they are attacking.
--from a young Marine just back from Iraq
On the good side
the laptop often provides a lot of good intel.
Anonymous said...
"I could
see that, if not actually disgruntled, he was far from being gruntled."
--
P. G. Wodehouse
I think Bertie Wooster was referring to D'Arcy CheeseWright
:-)
Purnell
said...
That's so very
true exspecially in America negative stories gains the most publicity and
attention.
doug in Seattle said...
Choices,
decisions, consequences. If one consequence is a slam on /., I don't do it.
r_wolfcastle
said...
Whan
a company is constantly seeking the creme' de la creme' of Engineers and Programmers
- they have an obligation to be as concerned with the emotions and frustrations
of this elite workforce - as much as they are concerned with perfecting the
Search Engine Softwares.
'Tis true. And as someone who
spent a combined 16+ years at Bell Labs and Adobe, both places where you
couldn't swing a dead cat without hitting a PhD, I can tell you that managing
herds of brilliant people is an art. Furthermore, although the best
practitioners of that art are usually quite bright, they are not necessarily
brilliant PhDs themselves. And that is something that I am not sure Google has yet been forced to grok, but
they will have to eventually.
Google does have several distinct advantages in
this arena, all of which Doug has pointed out. Among those are their
egalitarian idea culture and their continual striving for excellence while
doing no evil. All these things make even the worst prima donna geniuses
happier, more productive, and much easier to manage. Google also seems to
decidedly not be a place
that hires the cream of the crop, burns them out before they are 30, and throws
them away. That would be Oracle or, back in the day, Texas Instruments.
SUNDAY,
DECEMBER 04, 2005
Just do it!
POSTED BY RON AT 5:17 PM
Doug wrote:
The upside of this philosophy is that
Google did many things quickly, most of which turned out to be positive. The
downside is that Google sometimes did things with unforeseen effects and
individuals occasionally misinterpreted exactly how much power they possessed.
The tradeoff appears to have been worth it.
This reminds me of two anecdotes.
I didn't really appreciate it at the time,
having never experienced anything different, but except for having to do it in
Java and some basic functional requirements, I pretty much had absolute
technical freedom to design and implement AdWords however I saw fit. I wasn't
really able to take full advantage of this freedom with all the problems I was
having, but no one ever second-guessed any of the decisions I made within the constraints
that were given to me up front.
The second story is something that
happened in my (one) meeting with Larry Page to discuss AdWords. The issue came
up about whether or not ads should be screened before appearing on the site.
There was almost universal consensus that they should. The fear was that
neo-nazis or some such group would post an offensive ad and Google would get
bad PR for publishing it. The lone dissenter in the room was Larry. He was
adamant that he wanted to give users the instant gratification of seeing their
ads live right away. And being the boss his view prevailed. But we all left the
room thinking he was making a big mistake.
Turns out Larry was right and we were all
wrong. The instant liveness turned out to be a major feature distinguishing
AdWords from the competition, and the neo-nazis never came. Most organizations
worry about the negative consequences of the things they do. Google, thanks in
no small measure to Larry and Sergey's leadership, tends to worry more about
missing out on the potential benefits of things they don't do. (Some of this approach was visible to
the public for the first time during the IPO.) The net result is that they do
more. Occasionally they hit a bump in the PR road (like during the IPO, or when
GMail launched), but on balance, as Doug says, the tradeoff seems to have been
worth it.
(There's an interesting book
by the same person who wrote "The Tipping Point", which examines the
benefits of not over-analyzing decisions. I'm about halfway through it. So far
it's been a worthwhile read.)
22
COMMENTS:
Noam Samuel
said...
How analytical
is the book?
http://search-engines-web.com/
said...
Spontaneouty
was indeed a likely factor in AdWords early success during its "Pay Per Impression" days.
What is not
clear is..
"why did they NOT use some keyword
filtering software technology - to decrease the
probability that these potentially objectionable words would get online?
it
would be interesting to elaborate on whose idea it was to ultimately CHANGE to
the current "Pay Per Click" model :?
Anonymous said...
I liked the
book. Malcolm Gladwell came and gave a tech talk a few months ago and I was
lucky enough to snatch a copy while he was there. Very inspirtional guy.
Anonymous said...
Can any of you
guys talk about the new compensation item called the "Google Unit"
?
Is it like a stock grant that vests over time?
Why did they start issuing
those instead of regular options or grants?
Robert Carnegie
rja.carnegie@excite.com said...
Is this the
same technology that decorated news stories about the Asian tsunami with Google
ads for surfing equipment, until someone figured out how to stop that?
Genuinely. I saw 'em.
charlie_zoo
said...
I just found
your site Doug - really interesting stuff - and I wonder if any of you out
there are UK-based Xooglers? Can you show yourselves? I'd be interested in
having a conversation on a professional basis, either here or at Zoo Time.
Matisse Enzer
said...
I've often
wondered how conciously Google set about creating the huge real-time auction
system that drives adwords.
By now it must be one of largest auction systems
in the world. Can you say something about the the various degrees of concious
planning and accident that went into it?
Technicator
said...
Google's so
different internally that it's probably why their stock is doing so well. I've
covered that on my website and it's always so bullish. I give them credit. Is
it true that Google has more pHDs hired than a typical clinic?
Anonymous said...
why does
everyone think that a PHD is a magic bullet to making infinite profits?
if so,
half the faculties at universities around the country would be millionaires.
they
arent.
all a PHD proves is that you can follow instructions.
R2K
said...
Isnt is a
conflict of interest for you to host this on a google blog?
R2K..Bathrooms..Rocketry
Ddot the King
said...
Oh ok
cool....good day.
Anonymous said...
well..ok, had
to comment. OK i am dumb founded on this sunject...what exactly is adwords? I
am lost..ahumm..sorry, clue me in?
Anonymous said...
lol..OOPS not
sunject...,haha subject*
Andrew Goodman
said...
I got a lot out
of Blink. Beyond your post, which is great to learn from: we get into debates
with clients about how much to analyze campaign data, etc.
Although Gladwell
probably doesn't have all that much first-hand knowledge, his journalistic
account touches a chord: experienced professionals see the benefit of reaching
decisions based on "quick analysis" or "thin slicing"
rather than a full rundown of all relevant data. The more your job involves
seeing the same complex data patterns every day, the more impatient you become
with the comprehensive approach.
This is not the same as acting on
instinct.
Anyway...
I am not entirely convinced this relates to Larry's
conviction on the instant ads.
Our impression has always been that
"instant on" was and continues to this day to be a wonderful
differentiator for AdWords. Larry is a marketer par excellence (without knowing
it?). People still assume that "instant on" is a big unique feature
of AdWords, even if this isn't entirely true and isn't particularly important.
David Towers
said...
Doug,
This
blog is a goldmine, thank you so much. I'm currently writing my dissertation on
the organisational culture of Google (I'm an undergraduate in International
business commerce). Your posts have been so helpful in helping me understanding
things from inside Google.
Also, as I will be applying to work with Google in
Paris it makes it even more interesting to hear about these things.
Please
keep the posts coming, thanks again,
David Towers
Anonymous said...
That
"someone" would be Malcolm Gladwell, author of "The Tipping
Point" and "Blink," and a frequent contributor to prestigious
magazines. But that's a small aside. Although Google has prevailed with
AdSense, I wish it wouldn't run ads for paper mills. It doesn't need the money
and it encourages scummy behavior.
Anonymous said...
Re:
overanalyzing decisions, fair enough, but let's not forget that a lot of
companies fell apart because they didn't analyze their decisions closely
enough.
doug in Seattle said...
Sounds to me
like the whole premise of Blink is:
Go with your gut instinct
Or as is often
dismissed "woman's intuition".
Anonymous said...
No neo-nazis?
The way I found out the ads weren't screened was when my search returned one
for a *child porn* site. Naturally I contacted Google about it immediately. I
didn't save my original email to them, but here's their
response. The ad was removed, but reappeared a couple of times
before they apparently decided to block it somehow...
Anonymous said...
Offensive ads
were definitely a real problem in the early days of Adwords. I understand the
marketing reasons for instant approval, but, it let some really nasty stuff get
through onto Google result pages. Always they would be "unapproved"
(no one had reviewed them yet), but, not infrequently, unapproved ads would run
for many hours and generate lots of impressions.
Bob Aman
said...
I'd have
probably just used a bayesian filter to identify bad vs. good ads. If the
filter let it through, it'd run until someone complained, at which point the
filter could just be retrained with the new information. If it failed the
filter, it would sit there and wait for approval or denial.
R2K
said...
Wow several
clicks from this very old post. I wonder what happened?
FRIDAY,
DECEMBER 09, 2005
They say it's darkest before the dawn
POSTED BY RON AT 1:37 PM
The billing bug was the only major problem
we ever had with AdWords. It actually astonished me how few major disasters
there were given that the potential for them was so high. The Google system is
mind-bogglingly complex and it grew up very fast, which is usually a recipe for
building a huge collection of intractable bugs (look at all the problems
Microsoft has with Windows and IE). But it didn't happen. In fact, other than
the billing disaster, the worst problem I can recall that was visible to users
was that they once pushed a version of the home page that caused it to appear
without a logo. Other than that things seemed to go very smoothly. Before
coming to Google I would have given long odds against that being possible for a
development of this magnitude done in C++. I was beginning to have serious
doubts about my professional worldview.
A lot of the credit goes to three of the
most brilliant hackers I have ever known, Jeff and Sanjay who shared an office
so I dubbed them the dynamic duo, and Craig Silverstein, director of technology
(I'm making an exception to the xooglers policy of referring to people only by
their first names because Craig has already been outed in the press). I didn't
have many occasions to interact with them since I was working on things that
weren't very technologically demanding, but on a few occasions I did go to ask
their advice about one thing or another and they always blew me away with the
sheer quantity of abstruse technical information that they had stashed away in
their heads -- to say nothing of their ability to type fast. Craig especially
was really something to see. He used this funky ergonomic keyboard
and when he typed it sounded like one of those ratcheting noisemakers you spin
over your head at new years. I once came to ask him a question that I thought
would take me half an hour to figure out on my own and he replied, "Well,
let's find out" and in less than a minute he whipped up a piece of code
that answered my question. My jaw was on the floor.
With the billing problem fixed, AdWords
settled into a nice steady growth pattern, and everyone involved got an
engraved lava lamp. We hired some people to man the customer support lines and
do ad screening, and I was left to relax a bit and build some internal tools
for monitoring the system's performance. Google was a very open company in
those days, and the results were visible for all to see, so anyone could pull
up a web page on the internal server and see just how much money AdWords had
made on any given day. It was very exciting to watch the numbers go up. And up.
And up.
It was becoming clear that Google was on a
serious roll, and one of the things that would soon have to be done was to
internationalize AdWords so that it could serve non-English-speaking audiences.
This presented a major headache. Because AdWords had been written in such a
hurry, there was English text scattered all over the system. A lot of it was in
HTML pages (actually JSP pages, which is like HTML with embedded Java code),
and a lot of it (like error messages) was embedded in the Java code. Just
finding all the English text would be a significant challenge.
Here again I hit a major frustration as a
result of being forced to use Java and JSP. Just before I had come to Google I
had written a bunch of infrastructure to write web servers in Lisp. One of the
cool features of Lisp is that you can very easily change the way the language
is parsed, so finding all the English text is easy. You can more or less have
the system do it for you automatically. But in Java (or any other existing
language for that matter) you can't do that. The way that the system processes
programs is carved in stone and you can't change it. The best you can do is
write another parser for your program and use that to do whatever automatic processing
you want to do. But that's a lot of work. In Lisp it's literally five lines of
code (maybe ten for a really industrial-strength version).
But, of course, that avenue was now closed
to us, and because we had to deal not only with Java code but also with JSP, we
pretty much had to do it manually. Fortunately for me I was able to weasel out
of doing this incredibly tedious work, which was fortunate because if I had
been forced to do it at that point in my life I very well might have thrown
myself off the Golden Gate Bridge.
It was becoming very clear to me that I
was not a good fit for the ads group. There was actually some very interesting
work going on there. A fellow named Eric, who had hired on just a few weeks
after I did, had come up with a brilliant idea for increasing clickthrough
rates. It was so brilliant that it didn't even make me slap myself on the
forehead and say, "Why didn't I think of that?" because I knew that I
never could have thought of it. I didn't have the mathematical background. I
was really beginning to have serious doubts about my ability to contribute
anything meaningful in this company. But on the other hand I had already
invested four months, and it was becoming clear that Google was not going to be
a casualty of the dotcom crash any time soon. Besides, I had offered to resign
from the ads group and my manager refused, so I figured that put the ball in
her court to figure out what to do with me.
I was assigned a series of small projects
and I pretty much screwed them all up. As this is not one of the proudest
episodes in my life I hope you will forgive me if I don't go into too much
detail (actually I think I've repressed a lot of the details). I can remember
two incidents in particular: Jane (my manager) had asked for some statistics to
be produced. She, unlike many managers, was actually quite a competent hacker
in her own right (despite (or perhaps because of) the fact that her degree was
in philosophy) and had taken a preliminary cut at the problem, but the results
were not what she wanted. She handed the project over to me. It was just on the
heels of a series of screwups on my part and I really wanted to redeem myself,
so I threw myself into the work and got it done in an impressively short amount
of time. Too short as it turned out. I sent her the results I had produced and
didn't even notice until she pointed it out that they were identical to the
(incorrect) results she had produced herself. I really wanted to crawl into a
hole.
That was pretty much how November and
December went. Anyone who watched me during those two months would have
justifiably concluded that I was the world's biggest moron. I was beginning to
believe it myself. I have no idea why Google didn't fire me. And given that they
didn't fire me, I have no idea why they didn't try harder to help me find a
place where I fit into the company. It still baffles me to this day. For a
company that does so many things right, and in particular that takes such pains
to make sure its technical employees are happy, why they chose to dump me in
purgatory and leave me there is a great mystery.
UPDATE 12/11/2005: I'm putting this here
rather than in the comments section because blogger won't let you put a
<pre> tag in a comment so you can't format code properly.
Some anonymous coward wrote:
So it's trivial to write a piece of
code that walks a program, replacing string constants with function calls that
pull the correct string for the current language out of a database (for
instance).
It's actually even easier than that. I'd
do it with a read macro, something like this:
(defun my-string-reader (stream char)
(let* ( (s (ccl::read-string stream char))
(translations (rest (assoc s *translations* :test 'string-equal))) )
(if translations
`(or (getf ',translations
*current-language*) ,s)
s)))
(set-macro-character #\"
#'my-string-reader)
Here's an example of how it would work:
(defvar *translations*
'(("Hello" :spanish "Hola" :german "Guten tag")
("Goodbye"
:spanish "Adios" :german "Auf wiedersehen")))
(defvar *current-language* :english)
(defun test ()
(print "Hello")
(print "Goodbye")
(print "This string should not be translated")
(values))
? (test)
"Hello"
"Goodbye"
"This string should not be
translated"
? (let ( (*current-language* :spanish) )
(test))
"Hola"
"Adios"
"This string should not be
translated"
? (let ( (*current-language* :german) )
(test))
"Guten tag"
"Auf wiedersehen"
"This string should not be translated"
This is actual working code, and it took
me about fifteen minutes to write.
19
COMMENTS:
Derrick
said...
Ron, I
completely admire you for your ability to write about this. You are a better
man than I to be able to share what sounds like a really wrenching time in this
forum.
I think you've said that it turned out OK. I'm really cheering for you!
Anonymous said...
Your honesty is
an asset. Google's willingness to allow failure is their asset. Both of you
will work out fine on this planet of dishonesty and required perfection.
Anonymous said...
My guess is
that they assumed you would figure out something useful to do...
Anonymous said...
xxxx
Anonymous said...
This sort of
gybes with my experience as a recent interview candidate. They never really
explained what the job was, although I knew roughly the role. In retrospect, I
believe this is because they expect smart people will find and essentially
define the job, and if the current one isn't working out, then its just a
matter of time before self-adjustment occurs ;-)
I am also learning that
there's a pattern of low-balling candidates on pay. I think there are several
motivations for this:
1. The total benefits package has to be taken into
consideration, and the benefits are substantial, when compared to other companies.
2.
It's like heaven. Once you're in, the living is good. And if you're smart,
you'll only excel and advance.
3. They can get away with it!
Ron
said...
My
guess is that they assumed you would figure out something useful to do...
Maybe. I actually did try to take the initiative several times. Twice I
was shot down (remember, I tried to resign from the ads group so that I could
go somewhere else, but I wasn't allowed to quit), and the other times my
proposals were ignored (maybe I should have pushed harder, but I was getting
more than a little gunshy by then). This was not a systemic problem, because
other people were getting their ideas off the ground (like Eric with his
clickthrough enhancement idea). I don't know, maybe all my ideas just sucked,
but then I'm back to wondering why they didn't just fire me.
amit Dixit said...
First of all,
that’s very honest of you...What I would like to suggest is that, you should
try to learn more about yr self (your -+) rather than google in first
place....Always remember to start from grass root level of a problem and use
your common sense to look for its solution.....and when you put forward your
idea in front of your boss/colleges make sure you give ample justification for
it. Question your own ideas, in your mind before putting it forward to avoid
criticism. Above all always present yourself in such a way that you are looking
for solutions, rather than proving your intellect and try to involve
every1....as I think you must be surrounded by the mastermind of their
field…………………………..And Maaan you are da best ….Cheeer up NOW!!
Brian Slesinsky
said...
Could you say a
little more about how you would do i18n in Lisp? If I understand this
correctly, it sounds like some kind of structured search-and-replace, which
some Java IDE's do have now. (But not every string needs to be
internationalized, so it seems like there'd still be some cleanup.)
Anonymous said...
Programs are
data in Lisp; or, more clearly: the structures built by the parser are part of
the language specification, and are quite simple and easy to understand
besides. So it's trivial to write a piece of code that walks a program,
replacing string constants with function calls that pull the correct string for
the current language out of a database (for instance). In fact, with a Lisp
macro, this can be made to happen transparently as part of compilation, so you
never have to see those calls at all.
Anonymous said...
So, what was
the great idea to help with clickthroughs!?!
Ron
said...
This
post has been removed by a blog administrator.
Ron
said...
So
it's trivial to write a piece of code that walks a program, replacing string
constants with function calls that pull the correct string for the current
language out of a database (for instance).
I've posted a
response to this in the main body of the post (along with an explanation of why
I put it there).
Anonymous said...
Re: Lisp read
macro. Very cool, but what if "Hello" happens to be a command sent to
an internal server via RPC? When you change languages, your RPC fails. It seems
dangerous to mark ALL strings to be translated automatically. I think it'd be
better to explicitly mark strings that are for "external" use to
distinguish them from internal strings (logging, RPC, etc.).
DIE
said...
fascinating,
the lisp references reminded me of the yahoo store articles. I wish more
webbased companies would look at lisp as an alternative you but tend to be
stonewalled with zealots. Sysadmins preach Perl, and managers tout Microsoft
dotnet.
I can work magic in any language but I'd could do so much more if
given the right tool.
Word to the wise!!
Math isn't much of a problem man, I
was one of the ones that didn't sale the math books from college. When in doubt
I use them as reference. Stats books are cheap. peace out and keep it real yo
Matisse Enzer
said...
OK, I need to
get around to learning Lisp :-)
I sympathize very much with what you describe
about being sort of "abandoned" an Google - I've had that happen to
me, and for my particular personality, it is very tough. Very cool of you to
post about it so thoughtfully.
Ron said...
what
if "Hello" happens to be a command sent to an internal server via
RPC?
Then I'd suggest calling that code without rebinding
*current-language*.
Radha
said...
Hi Ron,
Sorry
for taking off the veil of Jeff and Sanjay. I believe they are Jeffrey Dean and
Sanjay Ghemawat. I read their papers on MapReduce and GFS.. Truly
astounding..
One trend I noted is that a lot of early systems folks inside
Google came from DEC(later Compaq , now HP labs) Western Reasearch lab in Palo
Alto. Is it possible to shed some light on how Google(or Sergey/Larry) could have
convinced these great minds to work for Google. Or was it Carly Fiorina that
really drove them away??
Joshua Daniel
Franklin said...
That example
looks a lot like how it's done in C with the poorly-named gettext.
Anonymous said...
Python has some
limited introspection abilities, but it would still be difficult to output
"fixed" programs.
I am familiar with scheme, it was used in
S&IoCP, great book, used for MIT freshmen to teach programming concepts.
However, I've never run across a program where I thought "this is a
perfect reason to use scheme!". So in reality, I've never used it. Which
is to say, I've hardly ever needed to manipulate source code.
FRIDAY,
DECEMBER 09, 2005
Let's get a real database
POSTED BY RON AT 3:09 PM
This story is a bit of a tangent, but I
think it's worth telling because there are some useful lessons to be learned.
AdWords was built using the MySQL
database, which is open-source and therefore available for free. It is by now
also nearly as full-featured as the best commercial databases, but back in 2000
this was not the case. MySQL was quite a capable system, but missing a few
(what some would consider basic) features. These missing features were
obviously not a show-stopper, as we managed to get AdWords to work without
them, but in a few cases it did take some extra programming to work around one
of these missing features. On the plus side, MySQL was fast and reliable and,
as I have already noted, free.
After AdWords launched, Jane, the ads
group manager, decided that now would be a good time to switch over to a
"real" database. "Real" is one of those words that Doug
ought to add to his list of words.
It means "expensive". Many managers seem to have this idea that it is
invariably true that you get what you pay for, and that therefore nothing that
is available for free can possibly be any good. Using MySQL was acceptable as
an expedient to get things up and running quickly and with a minimal of capital
outlay, but now that things were settling down it was time to recognize that
this was really, fundamentally, a mistake, and it should be fixed sooner rather
than later.
The flip side of this philosophy is the
one more commonly espoused by engineers, which is nicely summed up by the old
aphorism, "If it ain't broke, don't fix it." Yes, MySQL was missing
some features, but it wasn't broke(n). We had spirited debates in ads group
meetings over what to do.
We finally decided to go with a commercial
database (I won't say which one) over the objections of a number of engineers,
including myself. To ease the transition it was decided to convert AdWords over
to the new system first, and to do the main ads system later. It was a project
on a par with the internationalzation effort in terms of the tedious work
required to comb over nearly all of the AdWords code and change all of the
database queries. (Databases are supposed to all be compatible with one
another, but in reality they pretty much aren't.)
To make a long story short, it was an
unmitigated disaster. The new system was slower than molasses in February. Some
heroic optimization efforts eventually produced acceptable performance, but it
was never as good as the old MySQL-based system had been. For a long time we
were stuck with the worst of all possible worlds, with the two ads systems
running on two different databases. It was still that way when I left Google in
October of 2001, but I have heard through the grapevine that they eventually
went back to MySQL. (Since then, MySQL has added many of the features that had
been missing at the time.)
The moral of the story is that sometimes,
and in particular with free software, you get more than what you pay for. There
are a lot of companies out there paying dearly for commercial databases (and
operating systems for that matter). As far as I'm concerned they might as well
be flushing that money down the toilet. Actually, they might be better off. We
certainly would have been.
As an aside, there is a raging debate in
the hacker community about the overall economic merit of the open source model.
(Making money producing free software is quite a challenge.) I am not taking sides
in that debate here. All I am saying is that from
the end user's point of view free software is
often much better than the producers of commercial software would like people
to think.
69
COMMENTS:
James A.
Warholic said...
I am actually
somewhat surprised that there was a push by some at Google to pay for another
program in which the original was working quite well to begin with. After all,
I thought the main search servers were built with Open Source Software such as Python. Anyways, I have thoroughly enjoyed
reading the posts back from the beginning. I look forward to the next segments.
Jonathan Ellis
said...
holy crap, you
ran adwords on a transactionless database?
you have balls. and you're lucky as
hell nothing (else) went wrong.
Ron
said...
holy
crap, you ran adwords on a transactionless database?
Yep.
you have balls. and you're lucky as
hell nothing (else) went wrong.
Nope. Luck had
nothing to do with it. If you don't have transactions you just roll your own.
It's actually not hard at all.
Anonymous said...
Aren't there
issues with the GPL? PostgreSQL for example offers a BSD license, which is why
many commercial users prefer it to MySQL.
Rob
said...
This neatly
encapsulates my employer's experiences (and I was one of the tech guys who
encouraged the company to move to *cough*
Oracle *cough*!). You can get an elephant to tapdance,
but the results will never be graceful. Asking one to do so in your living room
is twice a mistake.
Anonymous said...
For the life of
me I don't understand why more folks don't use Postgres. The BSD license seems
to be the way to go. Similarly, I can't comprehend why Google uses a Linux
kernel when the BSD kernel is completely free of any GPL b.s.
Ron
said...
I
can't comprehend why Google uses a Linux kernel when the BSD kernel is
completely free of any GPL b.s.
Google isn't
distributing software based on GPL code, so the GPL "b.s." doesn't
come into play.
seo black & white said...
Btw, MySQL
supports transaction rather well since they added InnoDB storage engine. In the
last three years it works for me like a charm...
Steve
said...
This story says
a lot about the culture at Google--especially in those early days. While LAMP
(er non commercial platform based) architectures are today given the respect
they deserve--it was a much more aggressive thing to do back then (and still be
taken seriously). I recently left a moldy old company doing bank automation
systems which collapsed under the weight of WebSphere, DB/2 and Oracle.
Unfortunately it takes an engineering culture to cut through the hype and
distill the issues to the essence of a particular tool or system's
purpose--then to select the right tool for the job. Google's culture embraces
this and is a fundamental reason for their success.
Anonymous said...
In 1999 I gave
a presentation to the bankers taking MP3.com public (July 1999, $370m offering).
We used MySQL (wish I'd used Postgres and today, that's all I use) and the
bankers were agast when I told them we did not use Oracle. Today your bankers
would be agast if you did use Oracle in your startup. MP3.com also funded Hans
Reiser to add journaling to Reiserfs so as to reduce dependencies on VxFS. An
estimated savings of $8m over five years. Open source definately has its place
and you've got to give back to the community to keep it fresh and moving
forward.
Anonymous said...
People who
don't like the GPL must really like the reach around Microsoft and Apple give
you when they are fucking you for your code.
Anonymous said...
A former
employer of mine got the "let's get a real language" fever. They
decided to ditch Python for Java because of a whole bunch of pointy-haired boss
reasons. The mentality seems the same as that of the "real database"
story here...why does management so often disregard the opinions of the people
that have actually been in the trenches doing the work? Why do they decide to
throw away or overhaul working systems because of some marketing gobbledygook?
This
is an apect of business I really don't understand. Even when the decision
precipatates horrible consequences (unbearably slow systems, unmanageable
hulking codebases), some folks just can't recognize that they made a mistake by
not listening to the engineers.
Jonathan Ellis
said...
"If you
don't have transactions you just roll your own. It's actually not hard at
all."
You're scaring me.
Jonathan Ellis
said...
To elaborate:
some transaction benefits can be simulated "easily," such as logging
each update to make sure Bad Things don't happen when you, say, charge a
customer and the db goes down before the ad gets inserted. Although if single
updates aren't even atomic -- and if your db doesn't support transactions, they
probably aren't, which is the case for MySQL's in non-innodb tables -- you're
still in trouble if you're running something like "update users set
balance = balance - delta" when it goes down.
The harder ones, though,
such as isolation, I've never seen done outside the database, and I wouldn't
trust the code of anyone who claimed to do it. Much safer to use a database
that's already debugged their code many times over.
Mike Hillyer
said...
Jonathan Ellis
said...
holy crap, you ran adwords on a
transactionless database?
you have balls. and you're lucky as hell nothing
(else) went wrong.
Actually, MySQL has had ACID
transactions for many years now, including savepoints and four levels of
transaction isolation. Best get current before criticizing.
Anonymous said...
Erm, the
article is about yr 2000. MySQL got transactions about 2001 if i'm correct, but
don't really count on that :) However, we can presume that MySQL had no
transactions back then :D
Anonymous said...
Single MySQL
updates have been atomic for a looong time (1999 at worst), maybe always. With
those you can roll your own transactions.
It's funny when people trot out
disaster scenarios that derive from premises that are incorrect. "Well,
yeah, if not for Gravity, things might be different like you said..."
Jonathan Ellis
said...
MySQL does
claim that updates are atomic even on MyISAM tables, but I've seen many people
report that this is only true if nothing goes wrong, i.e., power loss, simple
crashing, etc. Two such people have commented on the relevant doc page:
http://dev.mysql.com/doc/refman/5.0/en/ansi-diff-transactions.html
I'm no
expert on MySQL source but I've seen enough non-idiots report on MySQL
corruption that I find toeing-the-party-line claims to the contrary difficult
to believe.
In any case, my main point stands: atomic updates alone aren't
going to help you improvise isolation support, etc., if your db doesn't have a
concept of transactions.
Hank said...
Actually Ron,
"rolling your own" is not a smart way to go. Trying to emulate
transactions in your app means that you don't actually have the guarentee of a
transaction, since your app and the DB are seperate, and one or the other can
go down in the middle of a pseudo-transaction, leaving you with inconsistant
data.
Using such a crippled and broken database is even more crazy when you
consider that it corrupts data (intentionally) and that there was a free
alternative without these problems, and that handles many concurrent queries
better.
And anon, they can't use BSD, they're from Stanford.
Anonymous said...
MySQL is rock
solid.
Online Degree
said...
We have been
using MySQL on our site (www.degreeme.com)
since its inception and have never had a single problem. Impressed? Just
kidding... I know that isn't at all impressive :) But I am still a huge
supporter of MySQL although I also have heard of instances where MySQL has had
corruption problems. Hopefully the new version 5 has fixed that.
Anonymous said...
i love this
story... you do get what you pay for... of course how you pay is relative...
because some of the best things in life are free... fun stuff
Anonymous said...
I'd dare to
guess that the reason they had such a big problem porting over to some other
database was that their code base was full of "hand rolled
transactions", including a huge number of calls to "lock
table".
If they had gome with a database with transaction support from
the get go, like PostgreSQL, the porting work would have been much simpler.
The
fact that MySQL makes you code a lot of non-standard stuff which makes it
difficult to port the code over to other databases is being presented as a good
thing in this article. If the author had opened his eyes a bit he'd learn to
use something else from the get go the next time.
And yes, PostgreSQL is a
"real database", and it's not expensive, it's free.
Ron
said...
Actually
Ron, "rolling your own" is not a smart way to go.
What can I say? It worked for us. Worked for this guy too.
Anonymous said...
As some
previous people said, I would be surprised if the problems porting to the
'other' database from MySQL where really any different from the problems
developers generally get when porting from database A to B. Without a really
good understanding of how things work in a given DBMS, especially locking and
transaction logging, it is very difficult to get something to work as well when
ported. I would wager than even the high calibre googleplex folk would find it
a challenge to get that experience quickly. Tom Kyte's Oracle books contain
numerous stories which illustrate this point from his experiences of going from
Sybase to SQL Server to Oracle.
sf said...
It's not
"free" vs "proprietary". It's some features against other
features and tradeoffs against other tradeoffs. MySQL has the single unique
feature of being a speed demon amongst the databases. The tradeoffs are quite
dire in the domain of functionality, and need to use specific MySQLisms instead
of the "high way" to have things running fast, not slower than all
the rest. Other databases are more feature-packed, but they are simply slower.
Sometimes WAY slower. The change of the database should have considered this.
Samuel Cochran
said...
It is quite
possible to make money from Open Source software. Rather than charging for
software, one must charge for services. Installation and configuration of
software, customisation of software and solutions, etc. can all be marketed
rather than software. This is my approach.
By doing this one can contribute
any changes and improvments to the Open Source community whilst providing
solutions for companies, and make a living, too. :)
Anonymous said...
I went through
similar issues at a biotech startup. I was the only programmer there for the
the first 1.5 years and had built everything with a postgres back end. I caught
a lot of heat resisting managements wish to convert to Oracle, but persist I
did, because there was no technical justification for the effort. I left for a
much better opportunity six months later and my successor apparently got the
job by promising the conversion. After 1.5 man-years, they still did not have
the system working as well as it had been. Eventually the company was sold for
pennies on the dollar.
Anonymous said...
You have a team
of engineers that are resisting the move to another database system pushed by
management.
These engineers are all happy with their current database, which is
probably the one they use any chance they get, so they are really in tune with
how it works.
The new database has to be used and tuned in a completely
different way, requiring its own years of experience to master.
Of course the
second solution using the new database is going to be worse, how could it not?
It says nothing about the database products tbemselves.
Anonymous said...
postgres
advocates are delusional -- millions of mysql users have no time for
crashgreslow.
Back in 2000, postgres didn't support the LIMIT keyword. I got
hoodwinked by the FUD they spread about MySQL and tried using it for three
projects: each time I got crashes and data corruption with small data sets. You
could find very few sites that were using Postgres on the web, and the
operators of many of those sites regretted that choice.
Somewhere around
postgres 7.4, postgres became stable enough that you actually ~could~ use it in
a production system. Still, it's pretty common for people to try a few queries
on Postgres and mysql, notice that some very common queries run 60 times faster
on mysql, and come to an obvious conclusion.
People who push the "no
transactions" FUD also forget that transaction support often reduces the
reliability of applications -- yes, it's the stupidity of the application
programmers to blame, but an organization I've worked for spent $250,000 on
some commercial junkware which doesn't do correct error handling with
transactions, and this leads to strange, glitchy, sporadic problems... And
these guys aren't using an "off brand database", but they're using
Oracle.
The truth is that it's not the end of the world if you mess up a row
or two in most databases: if you consider all the other ways an e-commerce site
can screw up because of user error, human error in fulfillment, problems in
shipping, other programming errors, losing a row because of the lack of
transaction support is a minor problem... As the CEO of one company told me,
"if we mess up one order in six months, we can just let the customer have
it for free" -- that's a lot cheaper than the cost of a DBA and the extra
hardware to run Oracle, never mind the cost to license Oracle.
Anonymous said...
What I don't
understand is why modern huge database applications are written using the
database engines that run on top of some universal OS? I mean, what else can be
on that server except the database? So the ideal modern database should be the
OS, possibly even without the file system as we know it. He hard-drive itself
can be structured to suit the database needs so one would never have to rewrite
any files, just the segment of the disk that was assigned to the particular
data. The improvement is speed and reliability would be enormous.
Lauri said...
Ron, how much
need did you have for transactions when working AdSense anyway?
It looks to
me like it is mostly INSERT operations and lots of them. They get written to
separate servers based on the user hash and the balance changes get calculated
at X intervals offline.
Am I close? Or did you have some other setup.
Anonymous said...
I'm not a
developer, i'm a financier so please excuse my ignorance. I found this post
quite interesting -- where would you suggest I go to read the best open source
debates? I'm interested in what the technology community pitches as the pro's
and con's. Thanks.
Ron
said...
Ron,
how much need did you have for transactions when working AdSense anyway?
Well, that depends on what you mean. In some sense we had no need for
them at all, obviously, because we built the system without them. If you want
to know how many times we had to write extra code that we wouldn't have had to
write had we had transactions, it's hard for me to recall exactly (it's been
five years) but it wasn't very much. It certainly was not a major issue. And,
like I said, in those situations where you need real transactions (fairly rare
in most real world applications I believe), emulating them yourself isn't
particularly hard.
Of course, it's all a moot point now that MySQL has
transactions.
Anonymous said...
The MySQL doc
page says an update might not update ALL of the rows if the thread is killed
during the update. That still does not stop single-row updates from working, as
mentioned. Furthermore, if you are using your own transactions, you are writing
some extra code to make sure multi-row updates work as well.
Matisse Enzer
said...
One way of
setting up MySQL these days is to do all the INSERT/UPDATE/DELETE to an InnoDb
table and then to have a pool of "query slaves" where the table is
MyISAM.
The query-slaves are behind a load-balancer and are used by
applications that need to do SELECTs, and this way they never have to wait for
the locks that the INSERT/UPDATE/DELETE create.
The Master can be a MyISAM
table if transaction-safety isn't as important as write-speed - INSERT into a
MyISAM table is very very fast.
Sugree
Phatanapherom said...
Did they
develop any lock server to simulate transaction in global scale?
Anonymous said...
All I
am saying is that from the end user's point of view free software is often much
better than the producers of commercial software would like people to think.
"End user" being the developer or the manager who actually
gets to pick and pay the product?
Anonymous said...
People prefer
MySQL over PostgreSQL for the same reason people prefer GNOME over KDE: Because
the developers are arrogant bastards, and the supporters act like dicks.
Jayson Vantuyl
said...
In all
seriousness, I don't see what the GPL vs BSD comments are about.
Google
developed an app on MySQL on Linux. The GPL didn't come into play unless they
tweaked and then distributed a modified Linux kernel. Even then, they CAN STILL
LEGALLY SELL IT!
The only thing that someone else can't do is improve the
source without providing a little quid pro quo. This is not unreasonable, it's
a LICENSE choice. For the small developer, it's a damn sensible one as
well.
That said, I prefer Postgres for reasons absolutely unassociated with
the license. I feel it implements much "purer" SQL than MySQL. I like
its administration better. I like how it plans queries better.
As for rolling
your own transactions, you should have covered this if you took a CS degree.
Between lockless techniques and a simple test_and_set routine, rolling your own
locks (and thus everything Atomicity, Concurrency, and Isolation) is really
childs play. As for Durability, a log table (or fsync'd file even) is really
not that difficult either.
Anonymous said...
Ron, I don't
see anything at all about "that guy" trying to create safe
transactions on top of year 2000 mysql. He says very clearly that he didn't use
a database at all, he used files. Because he not using tables and rows accessed
through SQL, he can ensure that his updates are completely atomic. You can't do
that with year 2000 mysql. There is a very big difference.
And Jayson, I
realize its easy to say you can make safe transactions, but its not easy to do.
The only way to make it safe is to rely on a log, and if a query fails, take
the database down and restore it from a known good point, then replay the log.
This is not transactions, its recovery. There is simply no way with year 2000
mysql to run a bunch of queries in a single atomic transaction and guarentee
that they either all succeed, or they all do nothing.
Arqueaopterix said...
They were right
in that MySQL is a toy database. It only got popularity because Slashdot uses
it.
However, unless you're using Oracle, PostgreSQL is better than your
database.
Ron
said...
There
is simply no way with year 2000 mysql to run a bunch of queries in a single
atomic transaction and guarentee that they either all succeed, or they all do
nothing.
That is true, but irrelevant. What is
important is not to guarantee that they do nothing, but merely to guarantee
that whatever they do they do not affect the results of subseqent queries
unless they all succeed. That is possible, even with y2k MySQL. Figuring out
how is left as an exercise.
Rob
said...
What
I don't understand is why modern huge database applications are written using
the database engines that run on top of some universal OS? I mean, what else
can be on that server except the database? So the ideal modern database should
be the OS, possibly even without the file system as we know it. He hard-drive
itself can be structured to suit the database needs so one would never have to
rewrite any files, just the segment of the disk that was assigned to the
particular data. The improvement is speed and reliability would be enormous.
This exists presently. It is called Microsoft SQL Server (presumably,
as usual for MS products, its hooks dig deep beneath the OS into the kernel).
It is of necessity a security nightmare, as are all MS products.
You have a team of engineers that are resisting the move to another
database system pushed by management.
These engineers are all happy with their
current database, which is probably the one they use any chance they get, so
they are really in tune with how it works.
The new database has to be used and
tuned in a completely different way, requiring its own years of experience to
master.
Of course the second solution using the new database is going to be
worse, how could it not? It says nothing about the database products
tbemselves.
Certainly there are elements of this,
but then, if the DB requires an army of expensive high priests to keep it
running smoothly, why bother? The design flaw of failing to self-tune becomes a
business feature in that the customer now has to sign up for hours and hours of
expensive consultant hours. I don't see this as a positive.
r_wolfcastle
said...
I have worked
at 2 companies that rolled their own transaction logic in the application and
used MySQL MyISAM tables -- one starting in 1999 and the other starting in
2001. Anyone remember AvantGo? That entire thing was a single MySQL instance
with a load balancer and multiple query servers.
At the second job, the one
starting in 2001, as jack-of-all-trades I got put in charge of database stuff.
We wrote our own layer between the application logic and the database for ease
of porting. Our app was essentially 100% database-centric, but we were able to
port from MySQL to MS SQL Server in a couple of weeks because of the way we had
written the layer. It turned out that none of our customers wanted MS SQL
Server (actually, they didn't care what we used as long as it worked and it was
cheap), so we let support for that lapse and just stuck with MySQL.
If you are
selling a product that is substantially smaller in scope than, say, a
comprehensive SAP solution or e-commerce infrastructure for eBay or Amazon, you
cannot use Oracle for economic reasons. You can't use the customer's existing
Oracle installation for your app, because that installation is already
highly-tuned by expensive folks for SAP or CRM or whatever. They won't let you
and your app and its processing and bandwidth requirements anywhere near it. So to use Oracle the customer has to
have an Oracle instance just for your app, and it is pretty damned difficult to
sell a customer an $80,000 product when you have to say, "Oh, and by the
way, you'll need to fork over about $30,000 for Oracle and hire another admin
to tune and babysit it." That is where open source databases like MySQL
really shine.
Finally, someone said that INSERT into MyISAM tables is very
fast, and that is true. But there are special circumstances (and they're not that rare) in which you can go even faster
using LOAD DATA INFILE. MySQL claims is "up to 20 times faster" (last
time I looked) than using MyISAM INSERTs, but which in practice is actually 5-8
times faster in apps I've measured.
Anonymous said...
This has just got to be a wind-up, right?
BTW Oracle
bought up InnoDB a while back. You stick to your, um, MySQL
disaster-in-waiting. I'll stick with PostGres, or Firebird rather than a toy.
Tyler
said...
Just wanted to
chime in and comment on some of the "Oracle is expensive" comments.
There are really 4 versions of Oracle to consider:
- Enterprise Edition = $1x
-
Standard Edition = $.38x
- Standard Edition One = $.13x
- Express Edition
= FREE! (just released this month)
The .38 and .13 numbers are from the
current price list.
I use x, since government pricing is typically MUCH lower
than what you see on our price list.
Most people have seen the price for EE,
but don't know about the other versions. MS has a VERY similar pricing
structure.
Full Disclosure: I am an Oracle employee.
Anonymous said...
What a sad
commentary about how little so many in the IT industry know about data.
I've
read that mysql was free, but commercial databases are expensive. Then the
author admits that they had to write their own transaction logic. Where did
this free labor come from?
I've read that "it's not the end of the world
if you mess up a row or two in most databases". The issue is that the
results are *unpredictable*. You don't know what will break when you violate
your contract between components.
I've read that mysql is a speed demon. Ah,
right. With no parallelism, no partitioning capability, inability to properly
optimize queries with 4 joins, no memory tuning. Please. The only queries that
it runs fast fall into a very small niche of read-only retrievels of highly
selective data (<1%) via btree index. That's good - but only covers a subset
of possibilites that you should be looking at.
I've read that mysql has all
the features you really need. Back in 2000 it didn't support transactions,
pk/fk constraints, views, unions, subselects, triggers, stored procedures, etc.
And it still doesn't support query parallelism or partitioning. If you're
writing database apps without any of these features you might as well stick to
MS Access.
I've read that they wrote their own transaction logic (since mysql
didn't support transactions well) and it worked fine. Really? How do you really
know? How did you empiracly measure data accuracy? Hmmm, you didn't? Really, I
wonder what kinds of problems (loss of accuracy of adsense) were caused by data
quality problems. Note: I've never encountered a database that didn't have data
quality problems, the lack of constraints and transactions almost guarantees
their widespread existance - whether or not it was shoved in the face of a
programmer.
I've read that conversion from mysql to the other database was
tough. As though that was caused by the other database. FYI, mysql is the least
ANSI-SQL compatible database in the market (especially so in 2000). Having to
roll your own transactions, and getting by with mysql deliberately converting
invalid data arbitrarily to work and then going to a standard solution would of
course, be a nightmare.
Undeniably, oracle was (and still is) overpriced.
Though it isn't tough to get discounts if you know what you're doing. DB2 and
Informix are both cheaper and easier to work with, though they weren't VC
darlings in 2000. And undeniably, you can write working software with mysql.
But there are compatibility problems, data quality problems, and scalability
problems. Maybe it's unfair to assume that this is obvious to junior
programmers unfamiliar with database technology. Especially given the current
hype around mysql.
Anonymous said...
I've been using
MySQL since 2000 to implement a community website (bicycle touring journals,
now about 800 journals and over 60,000 pictures). The site gets somewhere in
the region of 20-50,000 page requests (150,000 - 300,000 http reqs) per day. So
it's not exactly a super busy site, but it does have quite a few active users.
I've had the site up and running 24/7 (with some breaks for moves and hardware
failure) for about six years now. Not once have I had any integrity or dataloss
problems caused by MySQL. It's been really rock solid. All these people who say
that using MySQL for anything significant is a "disaster in waiting"
are simply blindly repeating religious dogma. That's what it is - religious.
They believe that MySQL is crap, so no matter what other people demonstrate to
the contrary (that it, uh, actually works, rather well in fact) they will still
continue to twist reality around to suit their existing viewpoint. I don't
really understand the outright hatred that exists for MySQL in the minds of
some people. It's been working for me for six years now... how much evidence do
you need before you accept that it really does work well? And as for the data
integrity issues, lack of transactions etc... all I can say is this: Despite
all the wailing and gnashing of teeth, the fact is that fixing data problems is
really not all that hard. It's very occasional, and actually I haven't even
seen any problems that were caused by MySQL itself - the biggest problems were
caused by user error (mine, generally). This involves stuff like dropping the
wrong table or field, or mistakenly deleting entries wholesale. Sorry, but
arguing that "the database should stop you from making that kind of
mistake" isn't valid at all. Any database that lets you change data will
have the potential for user error, and that's the biggest source of data loss
in my experience. Only on a couple of occasions has MySQL itself had a problem
- and then it was just a corrupted index file, which was easily fixed. I'll
take that kind of rock solid stability any day. Sure, MySQL could be better - I
have wished for a while that it was able to utilize more than one index in a
query. I think that is starting to happen. I'll be using MySQL for a while yet,
I think.
Neil Gunton
said...
Sorry, for some
reason the post preview system reverted my identity on the previous msg to
"anonymous" even though I had previously selected "other"
along with my home page. Anyway, the community website I was referring to is crazyguyonabike.
b7j0c
said...
This
post has been removed by a blog administrator.
Anonymous said...
Rob
said:
What I don't understand is why modern huge database applications are
written using the database engines that run on top of some universal OS? I
mean, what else can be on that server except the database? So the ideal modern
database should be the OS, possibly even without the file system as we know it.
He hard-drive itself can be structured to suit the database needs so one would
never have to rewrite any files, just the segment of the disk that was assigned
to the particular data. The improvement is speed and reliability would be
enormous.
The ability to create database
"files" on raw disk slices/partitions has existed in Oracle for as
long as I can remember (at least 13 years). This was a highly recommended way
of installing Oracle some years back but modern Oracle documentation now downplays
this method in favour of cooked file systems.
Aminorex said...
I've been
running dozens of websites offering commercial services, all backed by mysql,
since 2001. I can attest that it is good enough and fast enough and plenty
plenty cheap enough for everything I've ever needed in a database system. It
scales on demand, with no incremental cost. Unless you have a very specialized need, anything else should draw the
intense ire of your CFO, and maybe a shareholder lawsuit.
Anonymous said...
Someone asked
where the hatred of MySQL comes from? It started with the... I want to call
them lies... that they continually repeated in their presentations, community
websites, and documentation; like how transactions are not important, that ACID
doesn't really matter, and that being strict about data integrity and features
like foreign keys and views were only things that "purists" cared
about. And the hatred grew as people watched them implement all these things
over the last 5 years.
Sure you can make MySQL work, you can make GOTO work
as well, but that doesn't change that 90%+ of MySQL users would be better
served by choosing a different database.
Paul Reilly said...
At the end of
the day, MySQL WAS adopted by Google. OK, so additional transactional
development was required but it worked, and more importantly, it worked more
efficiently that the commercial counterpart! I see this attitude towards the
adoption of commercial software Vs. the adoption of Open Source software.
Here
within the corporate culture, where I work, I often see the same pro-commercial
attitude. The key point here is that by adopting the commercial alternative, it
provides a scape goat surely every corporate top director who answers to
shareholders, surely needs to protect himself by having someone to sue in the
event of a mission critical failure, a fine motivation to adopt a commercial
database.
Anonymous said...
A few thoughts
about recent posts:
1. Why the hatred towards mysql? I don't think there is
hatred, I think a few issues have really raised people's hackles:
- hype vs
reality
- history of company for spreading worst-practices (don't use
transactions or foreign keys, etc)
As proof of the above - note that you almost
never see so many criticisms of Postgresql - a database that is both freer and
more standard than mysql.
2. It works great on my hobby website, it should be
good enough for everyone: Ah, no. Go do a dozen business critical database
projects - and for god's sake, use a few other products besides mysql to get a
perspective. *Then* you're in a much better position to talk about mysql vs the
rest of the options. In the meanwhile, you really have zero credibility on the
subject of databases.
3. I've used it for years on my site and have never seen
any data quality problems, therefore that just isn't a big deal: Really? How do
you know you don't have data quality problems? Are you explicitly testing for
them? Almost nobody does - they take it for granted that things are working
right. The fact that they are only looking at a report with 95% of the data
instead of 100% is quite difficult for them to know!
4. mysql is good enough
for everyone, it's just the out of touch purists that don't like it: the people
who say this are typically those with very little experience with databases,
perhaps only with mysql. And they are complaining about the people with 10+
years of experience across a half-dozen products. These aren't purists -
they're just experienced professionals.
5. I've been running dozens of mysql
websites offering commercial services. Anything else is just a specialized
application: if you're just supporting e-commerce packages for smallish
companies at your isp you've touched on about 1% of what people use databases
for. You haven't yet handled large volumes, 100% failover environments,
reporting, complex queries, etc. It's great if it worked for you, but you
really probably wouldn't notice the data corruption caused by silent data
truncation/conversion, orphaned data allowed by lack of fk constraints, etc.
Sigh.
The proponents of mysql really need to broaden their perspective by trying out
a few other products and developing a few applications that have higher
standards (not just performance, but also reliability & quality).
Additional recommendation: work with people who have been doing this for a
while and who can show you the many ways that data quality problems occur, how
they can have unpredictable impacts on systems, and how to best prevent them.
Anonymous said...
Sigh. The
ignorance and lack of objectivity from commercial hucksters is disheartening.
The commercial bigotry persists. The truth is many of those closed minds have
their careers tied to a commercial product and have little or no knowledge of
alternative products or solutions. The folk pushing commercial products need to
ask themselves what features of a commercial product are 'really' needed in the
majority of real world applications and STOP preaching vendor product marketing
jargon and inventing FUD of open source.
Neil Gunton
said...
I thought I'd
come back here to see what other people had posted since my prior comment. And,
of course, there it is... the typical attitude of the PostgreSQL pusher, which
can roughly be summarized as this: "You don't know what a database is,
even if you think you've been using it for years to do non-trivial tasks, you
don't really know what a real database is. There are certain things that any
database MUST do, and if it doesn't do them then it is NOT a database and you
should run along now and run your little hobby website. Leave the REAL database
stuff to the professionals who know what they're doing". Wow! Talk about
arrogance, not to mention obnoxious. It really shouldn't surprise anybody if I
actually avoid PostgreSQL because of people like this. Also, I do hear from the
odd person that actually tried shifting over to something like PostgreSQL that
it was *very* slow. That little fact gets conveniently swept under the carpet
by the fundamentalists. If you look at any discussion involving MySQL vs
PostgreSQL over on slashdot (and it seems that any discussion that has to do
with MySQL will bring out the PostgreSQL fanboys), then you'll see comments
tucked away in the oblivion of "unmoderated hell" which talk about
how difficult or slow PostgreSQL is in reality. I have learned to take the Open
Source zealots with a huge pinch of salt these days. What you tend to see are
comments that just keep repeating the same thing over and over - PostgreSQL is
the only thing anybody should be using (even though it's actually still really
slow)... or Mozilla/Firefox is the best thing since sliced bread (even though
it was basically unusable on my old 450 MHz workstation, it was so slow, and
moreover has had major bugs unfixed in the rendering engine for literally
years)... seems like the whole Open Source crowd has a tendency toward the
religious in terms of closed thinking. Something to do with a mixture of
"us vs them" tribal attitude, with an added dose of dogma and hatred
of anything that threatens their cosy little world view. You'll notice that
these people won't acknowledge that MySQL is pretty good for many applications
(the majority of database applications, actually). In fact, many of them won't
even acknowledge that MySQL deserves to be called an RDBMS. Any post praising
or supporting MySQL is rebuffed with a link to a "Gotchas" page
listing irrelevant or out of date bugs or anomalies. Any post defending MySQL
is picked apart, point by point - you can tell a zealot when they think you are
*totally* wrong on *every* point, each of which has to be individually addressed,
simply to show you just how *wrong* you are. No middle ground, no no no. Just
"mysql totally sucks, PostgreSQL is reasonable and sane, anybody who says
otherwise is delusional". Whatever. I'll just keep developing my
"hobby site" quite happily, scaling it up as needed using MySQL. It
works, thanks MySQL! Oh, and for the record: I'm sure PostgreSQL works just
fine too. I'm sure you can tune it just so, so that it is ok speedwise. But a
note for the PG fanboys: You don't win people over with overbearing, arrogant
posts that say anybody who thinks that you can do quite a lot with MySQL is
somehow less professional or just a hobbyist. I'm sorry to go on, but this
kinda gets on my nerves after a while. It's like swatting annoying flies -
every time someone mentions MySQL, these fanatics show up claiming that it
can't possibly be used for anything. To which I reply: GET A LIFE MATE!
Thanks.
GoodBytes
said...
I am coming a
little late in the debate...
I wanted to congratulate you on this wonderful article
and two excellent points:
"with free
software, you get more than what you pay for",
and
"from the end user's point of view free
software is often much better than the producers of commercial software would
like people to think".
I have used both MySQL and
PostgreSQL, and have good opinion about them both. They have their good and bad
sides, but you get to learn them in time and find workarounds. What's best,
they are free and they are constantly improving with new versions.
Also, it is
good to mention that not everyone needs a database for heavy-lifting
applications. For simpler tasks, having one of these databases is
priceless.
Thanks.
Anonymous said...
I think someone
made a comment a bit like that one. I've used both MySQL and Oracle and I must
say that i'd rather work with an Oracle db that with MySQL. It has many
advantages like having the "data integrity logic" directly in the
database. This is priceless when two or more apps are accessing/writing to the
database.
But, i would never use Oracle for an high number of concurrent
sessions. By high number I mean as low as 1000. IMHO, Oracle for a complex
and/or high data volume database and where transactions matters; MySQL for a
simple but heavely queried database. Like the frontend-backend thing.
I still
believe that you can achieve the same performance with Oracle or with MySQL.
But with Oracle you need the knowledge and the resources to do so. And why
would you need to put your hand in your pocket, MySQL is there to do just that
(the performance thing). But I fear that while on the road of adding more and
more "basic" features, MySQL will release a light version of
itself.
js
uri
said...
Just wondering,
when you talk about the "AdWords program" is this the one external
customers work with, or something used for internal processing of ads etc.?
Thanks
for the insightful story :)
Andrei Lopatenko
said...
Hard to
believe.
I certainly agree that MySQL may outperform untuned Oracle (sorry, I
talk ONLY about database I am truly experienced with fort many years) for
transaction-free database under 10 million records. I believe that AdWords,
AdSense needs much large database to operate.
Tuning of Oracle to real
environment which depends on both hardware and software - RA memory, HDD,
network, operating system, RAD is hard and time consuming but results maybe
surpising. Performanse may increase scores times comparing to
"default" untuned configuration. Proper programming like more
functionality to PL/SQL then to Python may improve functionality up to 10 times
in data intensive applications even for plain select statements without any
database update functionality.
I do understand that select processing is
99.9999% of time of functionality required from Google AdSense database, but
remaining 0.0001% is not less important for such applications.
Besides
performance issues -- what is about fault recovery which includes transaction
loging, backup and restore. If hardware and software fails, could Google
restore it to the point just before fail or some users, customers, will loss
their money, settngs etc
What is about updates? MySQL is pretty efficient for
SELECT, but not for updates especially for updates of indexed tables. How many
update transactions (new AdWord customers? users? settings changes? clicks?)
per day ar eperformed? Can MySQL handle them? If it can handle them will it
handle in one year? Are updates done in parallel?
Again, let mt emphasize a
point about perfortmance. It is hard to believe that good Oracle engineer can
not tune Oracle to be faster then MySQL in case if you have a database more
then 10 million tuples.
I am not again Open source and I do not sell Oracle
Database :). I like Open Source, but my around 12-year experience proves that
for large databases of mission critical data Open Source databases can not
compete with Oracle.
I do not say anything about MS SQL or IBM DB2 for only
the reason that I do not know how to tune them or how efficient they are for
large databases.
Andrei Lopatenko
said...
But, i would
never use Oracle for an high number of concurrent sessions. By high number I
mean as low as 1000.
What do you mean "concurrent sessions"?
User
connections? You definitly do not need to use 1000 connection to run
applications like AdSense
1000 parallel. My experience - Oracle is pretty
efficient for parallel queries, especially if you try to apply cluster
architecturw
about PostgresQL. It is reaLLy slow
Anonymous said...
"To
elaborate: some transaction benefits can be simulated "easily," such
as logging each update to make sure Bad Things don't happen when you, say,
charge a customer and the db goes down before the ad gets inserted. Although if
single updates aren't even atomic -- and if your db doesn't support
transactions, they probably aren't, which is the case for MySQL's in non-innodb
tables -- you're still in trouble if you're running something like "update
users set balance = balance - delta" when it goes down.
The harder ones,
though, such as isolation, I've never seen done outside the database, and I
wouldn't trust the code of anyone who claimed to do it. Much safer to use a
database that's already debugged their code many times over."
....
It's
all code that makes it all work in the database. If the base goes out at the
wrong time, no matter how many hundreds of safeguards happen, you're going to
have a loss of data. All this advanced junk to prevent it is pretty
meaningless, because all it does is move the point where a failure will cause a
loss to a different spot in the program.
Administrator
said...
Thanks for your
blog...
http://best-casino-online.t35.com
Best Online Casino
Anonymous said...
Hmm,
After
reading the mixed opinions, and very stern beliefs I found it rather difficult
to not post a response.
All in all the artical touts the success of Mysql Over
a commercial database. This onto itself is a benchmark
for success. I have been
involved in a large number of commercial projects, including one of the worlds
largest
databases which happens to run on Mysql "EdgeHealth".
I have
worked side by side with developers and dba's who have scratched their head at
how well Mysql performs.
Oracle, DB2, Postgres all seem to perform rather well
until you compair them to Mysql. Although I do often enjoy
the ability to
extend postges with perl modules, and create objects for oracle in java.. The
speed and reliability of
Mysql cannot be denyed.
Consider the following, Cox,
Google, IBM, Sun, Oracle, Microsoft, and a hoard of other companies
deploy
Mysql as thier database of choice for web and e-commerce applications,
it speaks volumes when a vendor
does not trust thier own products for
production enviorments.
Mysql Postgres and Oracle will be around for years to
come because they are good products. As for which database is the best..
thats
just a matter of oppion. For the moment Mysql is the king of spead, and step
child of features. My experiance says of the
3 Mysql wins out in deployment for
the reasons, 1 performance, 2 scalability, 3 cost.
Don't forget products
evovlve or die, so if all 3 of these products are still around in the next
couple of years, I would expect them
to evolve in the dirrection of what they
don't have now.
Oracle and Postgres simply lack through put, Mysql lacks
external scripting langauges.
All of these products offer transactions,
clustering, and high avalibility.
If Mysql ever adopts external scripting
languages, will it still keep its speed? Only time will tell.
Anonymous said...
Re: Database as
OS
You are absolutely correct, the database server typically runs nothing
else, and using the file system to store tables rather than a raw partition is
inappropriate. The file system is tuned for small files; in the 1-8kB range.
That's just what Unix normally holds.
OTOH, by keeping the databases as normal
files, backup and manipulation don't require new tools; you just use mv and cp,
like you would any other file (with the database turned off, natch).
Anonymous said...
Speaking of
SQL, I heard that no database actually implements the full feature set that was
defined back in the 1970s.
I personally tried to write a python layer that
implemented views for MySQL. It was quite a bit of work and I never finished
it.
Does anyone else have their sense of aesthetics offended when they see SQL
embedded in another language? Reminds me of inline assembler or something.
There has to be a better way. I tried to invent an OO layer over MySQL in the
views project, but it was very ambitious.
Anonymous said...
"Real"
simply doesn't mean expensive in the database case. PostgreSQL and Firebird are
both just as free as MySQL, but are actually robust and reliable.
SATURDAY,
DECEMBER 10, 2005
Oops!
POSTED BY RON AT 8:57 AM
I accidentally posted a political rant on
xooglers that I had meant to post to my personal blog.
I have a BlogThis! button in my bookmarks, and I forgot that now that I have
posting access to more than one blog I have to actually tell it which one I
want to use. Sorry. It won't happen again.
(Interesting how this happened the morning
after I wrote a xooglers enry about how I kept screwing up at Google.)
3
COMMENTS:
Tuor
said...
actually, this
may be a bug in blogger and not your own fault. I have observed it myself using
the "blogthis" feature with multiple blogs. Sometimes it posts to my
primary blog even when I have selected another blog in the select option.
I
have probably filled out a bug report for this, but I have never heard back
from blogger.com staff about any bug reports I have filed, so I don't know if
they look at them.
(TIP: I have noticed that if the BlogThis window doesn't
*blink* briefly, it has not updated and will post on the default blog instead
of the blog you have selected)
13TonGimp
said...
I used to have
tons of blogger related errors, so I switched to wordpress.
I have fewer
problems now....
Anonymous said...
My sites work
fine
SATURDAY,
DECEMBER 10, 2005
Good things
POSTED BY RON AT 9:07 AM
Since my last posts painted a pretty
gloomy picture I thought I'd devote a post to some of the good things that were
happening at the time.
Despite all the stress I actually looked
forward to my time at the 'plex. I didn't really have a life up there so I
could really immerse myself in work, which I actually enjoy. I am an engineer
at heart. I get a real kick out of watching something that I've built work. And
the 'plex was the perfect environment, filled with smart, friendly people,
state-of-the-art equipement, and half a dozen different kinds of coffee
machines. (I actually had some interesting conversations about the finer points
of espresso making, and at one point someone even brought in a book that was
entirely devoted to the topic.) I learned a lot, more I think in my one year
there than in the previous five.
As I said, I didn't really have a life, so
I almost always stayed late (except when I felt like making the odd pilgrimage
to Fry's). Charlie served dinner every evening around 5:00 or so, and there
were always about twenty or thirty people who showed up for that. Late at night
when there weren't people running around I'd try to learn to ride Sergey's
unicycle (I'm pleased to report that neither I nor any of my colleagues
sustained any permanent injuries), or work off some stress by riding one of the
Zappys
around the 'plex at full speed. (Nowadays they have Segways!)
One week I got to go on the annual Google
ski trip. The year before it had been only a couple of dozen people. The year I
went we had 150 or so, and filled three or four busses. It was a really strange
week. I flew up from LA, got on a bus, rode to Tahoe, went skiing, got bussed
back to Mountain View, and got back on a plane to LA. And I got paid for it!
But the best part of being at Google was the
sense of being part of history. At the time no one had any idea how big Google
would get, but I had been a computer geek all my life, and the Silicon Valley
was always the Promised Land. I had wanted to be a part of it ever since I
touched my first computer, a Sol-20.
I had done a couple of short stints as a co-op employee at IBM (the Cottle road
plant), and visited a lot (my first wife's family lived in Los Gatos), but this
was the first time I was really a part of it. I even got a certain perverse thrill out of the insane
commute. I would look around at my fellow passengers and think to myself that
these were the movers and shakers. There were always Cool Things going on in
the Valley, and these were the people making them happen. And I was one of
them. I was finally living my lifelong dream. It felt great.
In the main, it felt great a lot more than
it felt crappy. But the crappy parts make much better stories.
21
COMMENTS:
Tavarius J.
Taylor said...
Wow, so you
really had no idea of the sucecss to come for Goog? By the way I really enjoyed
your posts.
Anonymous said...
Hi,
I'm a
mother of a Googler. I just wanted to say that I love your blog. Keep it up!
http://search-engines-web.com/
said...
HERE IS
ANOTHER "GOOD THING"
Just been added to
Slashdot today - your will get thousands of visitors during the next
hours
http://slashdot.org/article.pl?sid=05/12/10/2125204&tid=217&tid=95
Anonymous said...
I think its an
awesome right. Interesting perspective but I disagree that all computer geeks
dream of working in Silicon Valley. These days with the cost of living I think
that is less true then ever. Especially since computers are so mainstream now
and companies are moving about the country like other industries.
Anonymous said...
Keep up the
great blogging! I love these stories!
Anonymous said...
This place
reminds me of my time at Disney. Good luck with the blog. ^_^
Anonymous said...
This blog is
shit. You complain too much. Get over it, move on with your life.
Anonymous said...
your post is
shit, you criticize too much. Stop criticizing other peoples blogs and get on
with your life.
Anonymous said...
Why do people
leave Google? It seems that some people after just one year. I would stay for
at least 10. Are you working contractually?
sickboy
said...
These are an
awesome inside view of google, I have always wanted to work for a company like
google, kinda helps you see through the stuff everyone talks about to the reality.
http://redsoxunixgeek.blogspot.com
Anonymous said...
Psst...Getting
a job at Google isn't so hard if you don't mind working as a contractor for
awhile and thinking outside of the valley. Trust me!
Anonymous said...
I worked for
the IBM on Cottle Rd for most of 2000. When were you there?
Ron
said...
I
worked for the IBM on Cottle Rd for most of 2000. When were you there?
The summers of 1983 and 1984.
Anonymous said...
"equipEment"?
Oh
my, Google hires illiterate people. yuk.
Anonymous said...
In general, and
not a comment about the poster. But why does every web hack call themselves an
engineer? I remember having to take a state PE cert then be required to work in
the field for several years before taking the engineering exam. Fields had to
be recognized engineering fields. Maybe the word has been regulated *joke* to
the same professions as rocket scientist and brain surgeon.
Anonymous said...
this is very
funny
Google
BADSENSE!
http://www.worldbankinginfo.com/credit.htm
http://img506.imageshack.us/img506/1133/badsense9ga.jpg
Luke said...
I am a
professional barista by day so of course I have to ask: which machines did you
like using best and which book were you handed?
Thanks for the interesting
read!
Anonymous said...
Ron,
You are a
true Lisper! Thanks for the awesome read-macro! That's awesome.
I hope you can
find work where you can hack lisp and be happy.
And thanks for putting the
transactions-must-be-in-the-db fool in his place. As if a DB guy, working in
C/C++, is going to do manage the concurrency better than you guys. That Paul
Graham reference was awesome!
Orodemniades said...
I'm merely a
fellow blogger (with a BA in Eng Lit!) but this is without doubt one of the
best blogs I've come across.
Funny, I feel like I'm a part of history merely
reading this blog.
Anonymous said...
Hey Doug and
Ron,
Just wanted to ;et you know that I found your blog on SlashDot Google
feed.
I use gmail, Adwords, Adsence, Site Map and a few others. My home page
has been google for years.
I have been using the Personalization feature and
have added your Blog to 'Create a section'. However if you guys have any
friends on this project tell them there is an anyoying bug. Everytime the page
updates it forces 'always on top' to the browser I have it in. Seeing I am an
avid user of the right click and open in new window this always seems to happen
when I am typing in a form or something and screws me up. I am almost to the
point of just going back to the old google home page but I figured someone
would figure this out sooner or later.
Lastly, I work in the internet industry
and work smack dab in the middle of Marketing and R&D so I can relate to
both of your stories. I also always wondered how the "big boys" did
things and it is nice to see my shop has a lot of the same things going on and
the grass isn't greener...
SJ
said...
Nice post. I
would like wokr for Google or bettre yet start the next Google. I am tryign to
see if I can pick any lessons from here.
THURSDAY,
DECEMBER 15, 2005
What he said
POSTED BY RON AT 12:06 AM
Funny, I was finally getting around to
writing a sorry-I've-been-away-it's-these-damned-holidays-will-probably-be-gone-a-while-longer
entry but Doug beat me to it. I'm probably going to be out of commission until
Jan 1.
I'd like to send a great big thank you to
all of you who have left supportive comments. Aw heck, it's Christmas -- thanks
to everyone whether you've been supportive or not. Just knowing that someone is
reading all this drivel that I'm writing makes it all worthwhile.
I wish you all a happy holiday season and
a healthy and peaceful new year. Merry Christmas, happy Hanukka, splendid
Solstice, stupendous Saturnalia, kick-ass Kwanzaa, a fun fabulous fortuitous
and fortune-filled Festivus,
and from the better-late-than-never department, a delightful Diwali.
Now stop surfing the net and go drink some
egg nog!
42
COMMENTS:
AG said...
Excellent start
guys. You both blog in an entertaining way which keeps me coming back for
more.
Please keep up the good work as you have a lot of fans now.
AG
la lara
said...
Then call when
you're back
Matthew
said...
Ya Salams!
here
here, I too have been irrisitably drawn to return to glean more & more
details about the conciousnesses involved in the big G juggernaut..... !
&
don't forget 'eid al adha' for all Muslims just after Xmas!
God Bless!
MT
Dave
said...
and a happy
Agnostica.
Chrissie
said...
I hope you have
a happy holidays as well.
Anonymous said...
No shout out to
all us Muslims and Eid ul-Adha? Sure its in early January this year but still
its almost the holiday season!
Kiran
said...
been going thro
ur posts for some time now. its been interesting.
and abt Diwali it happened
in Nov itself but anyways thanks and wish u all a very happy X'mas and New
year.
Kiran
jeffysspot
said...
yah .....what
you said
Anonymous said...
Thanks for
great blogg.
Ron
said...
No
shout out to all us Muslims and Eid ul-Adha?
An
unintentional oversight. My bad.
An excellent Eid ul-Adha to all!
Noctis
said...
This
post has been removed by a blog administrator.
Sports Bettor
said...
Keep writing.
Happy Holidays.
Anonymous said...
i'll be
watching my RSS =)
have a good holiday!
Samer
said...
great stuff!
Happy holidays...
Sam
said...
Nice job...
wonder why you call your blog, Xooglers? But I am just starting to type my own
ideas and stuff on mine...
omm
said...
I am one of
your big fans too. Merry Christmas to you too and thanks for the nice thoughts
:-)
Archosidf
said...
one of the most
fascinating things I've read on the web in the last months
Dunzo
said...
I'll stop
drinking egg nog and start surfind the web right now.
Anonymous said...
Here is a blurb
from the 12/20 WSJ,
"Mr. Kordestani oversaw an important shift in
advertising models at Google in 2003. Google moved from charging advertisers
based on how many consumers saw their ads to its model of billing them only for
users who clicked on their ads. That involved combining several sales databases
-- an initiative known as "Project Drano" -- which Mr. Schmidt says
"nearly killed" Google. Phasing out the old ad system entailed
changing the incentive and commission structure for Google's sales staff -- a
treacherous task, which Mr. Kordestani supervised."
Any juicy tidbits
about this project or Mr K?
Anonymous said...
Happy Holidays
regardless what you celebrate!
Anonymous said...
I'm curios
about the relationship between Engineering and Product Managment. Was it
cooperative? How was the PM group viewed internally? Did they fall under the
70/20/10 rule? What did they spend their time doing? How big was the team?
JK & PK
said...
Dude, you
deserve some eggnog (you can have mine too, i think it tastes like crap)
Anonymous said...
Diwali is
already over in November in India .
Happy X mas to you.
Rufus the Great
said...
I love your
blog! Please check out mine, http://placeofrufus.blogspot.com/
Jennas Blog
said...
I'm new to
blogging, and internet business. So far, I've found google to be difficult
and Yahoo much more user friendly.
chuck
said...
IT'S OFF TO
EGGNOG I GO...
Stenar
said...
rafiq said...
The New Google
doodle page has an error on it.
http://www.rafiq.za.net/2005/12/24/google-holiday-doodle-page-has-an-error-on-it/
rafiq
said...
Wow! that was
fast. Luckily I took a screenshot as proof (available at my blog)
underpanter
said...
you are hot,
wanna link to or from my site?
localudal
said...
Merry
Googlemas!
Even as the blog is dedicated to the Google past, here's something
for the ominous Google future (just one excerpt from my blog
http://goolocalizations.blogspot.com):
GOOGLEVISION
One Google opening for
i18n specialist requires a candidate to possess a "vision". You want
a vision, here you go.
In 1993-1994 I worked on Nostradamus TV script
translation, and it was me who said that the scene of a global war of West
against Islam must be dated 2001, not 2000, as faulty translation of Nostradamus'
quatrains suggested. Check it out in Mosfilm archives, or in some Hollywood
script hangars.
Here's what happens in the next decade from now:
1. Google
buys Australia, Starbucks, Moffett Field, Antarctica, some parts of California
-- from a weakening grip of Governator. Google finances the digging of the
tunnel and the pipeline between Eurasian Empire of Czar Putin I and Alaskan
Dominion.
2. On Moffett Field, Google builds the Babelfish Tower out of
millions of nanocarbon tethers delivering food, water, replacement nuts and
bolts to ISS by dozen molecules at a time. Ancient flotilla of space shuttles
is grounded and converted into Starbucks restaurants.
3. Google buys CNN,
FoxNews, replaces their field force by 360 deg Webcams put on every phone pole
in US and in major critical points around the globe. The new service is called
Google News, and its content, zoom, direction and choice of commenting blogs
can be tailored by user's preferences. Hollywood movies can be watched in raw
footage, or pre-director's cuts. Effects, actors and scripts can be edited by a
viewer.
4. Cash is virtually abolished and used only in illegitimate trade of
drugs, firearms, in presidential campains and in live prostitution.
Numismatists grumble over cash substitute called AdCents: an ancient $2 bill is
valued at 200 AdCents at GoogleBay: 1 AdCent is equal to 1 million of AdSense
clicks. As the result, Google Cache replaces Fort Knox. Google's motto 'Don't
Be Evil' now reads: 'Don't Be!" Accordingly, the final solution to private
information and its theft is brought about: except for the name, approximate
date of birth, and fingerprints there's nothing could be found on any
individual. Legacy data like account numbers, PINs, SSN, Apt./House Numbers,
etc., are known only to Google Index and corresponding agencies.
5. Google
trades McDonalds' back to Canada for Québec. Resulting huge reserves of frying
oil, with Persian Gulf's oil being almost finished, are used in mix with corn
schnapps to fuel 4-wheeled segways, or SUVs, as in Segway Utility Vehicle. Ford
leads the pack of auto manufacturers with a nostalgically smelling Segway-T
model capable of 1 mpg.
6. Googlers (see section below) are routinely shot off
the skies over China and Northern Korea by ancient DoS ACKACKers. Google
retaliates by using Mandarin over the Continental China. Meanwhile, with
Google's backing Al Gore, the old Internet inventor, buys EU Internet back from
European rebels. A new resulting Internet is called Google Index.
7. Google
buys LOC, NYPL, SJPL, Stanford Library, and copyright debates cease with an
acceptance of Copygoogle Law. Google buys Wikipedia and Britannica, publishes
the resulting Brikiwikipedia in 100 set in Katull paper volumes of 5000 pages
each. To enhance readability of such a tome, Googlese it is printed in (see
below) has lost many useless letters and signs like 'c', 'j', 'q', 'v', 'w',
'y', 'z', '$', comma, colon, semicolon, ellipsis, dashes, etc. Sites insisting
on using old, or so called correct English, are progressively excluded from
Google Index.
8. Google launches Goobble, a space telescope armed with
hurricane battling laser. First fire test was partially successful: the Omega
hurricane was dissipated over Cancún, Transtexas, which was severely fried in
the process. Goops!
9. Google's AdSense program is challenged by AbSence
ideology by Teoma, the most used search engine of 2015. Teoma's success is
based on abolishing spamming and sponsored links in its search results.
10. At
Googleplex and next to Vint Cerf's, Google unveils a bronze monument of me, the
best Google Borshcht Chef whose after-borshcht burp renders a better
"Goooooooogle" sound out of majority of contest jury for 10th year in
a row.
Shaun
said...
Awesome blog :)
"Happy
HOlidays"
Shaun - ohpunk.blogspot.com
JR
said...
Merry 2 miutes
afta christmas!!!
comment to me at hairston.blogspot.com
JR
said...
u guyz no how
to make a good blog!
Rufus the Great
said...
jr,
I cannot
understand you. And with my superior intellect, you would think I could
understand anyone. What is wrong with you?
Mel Logan
said...
happy holidays,
and thanks for the good reading material.
Miyagi said...
Great blog
dude!
Anonymous said...
Egg Nog
rocks
Signature:
Get $25 free on first deposit at the world's largest poker
room with 70,000+ players
http://www.partypoker.com use code: shefa
todd
said...
wow you have a
lot of fans i just read this blog. i like it. i'm a fan of courtesy. your a
writer thats cool i want to be a writer. i noticed you did the whole naming of
every holiday to be politicaly correct. just say happy holidays. or merry
christmas. reading your blog i can tell your smart. i didn't get bored reading
your blog like i do with others. you kept me interested.
frank white
said...
hey hey now. i
like your blog. please explain to a google browser search user what xoogler is
all about? what does the word mean? is there a site? maybe a program? hey kiran
where are u from dude? you need to wear slightly loose brand of jeans. those
balls need to breath bro. by the way i can see that you have exactly 2 dimes
and 4 pennies in your front right pant pocket
JC
said...
You left out Agnostica!
Harry
said...
And may the
force be with you :p
MONDAY,
DECEMBER 26, 2005
We interrupt this vacation...
POSTED BY RON AT 4:13 PM
... to bring you an important message.
I had a little down time today and so I
decided to hook up my laptop and check my email.
Big mistake.
It seems that Xooglers has attracted the
attention of all sorts of people who think that because I used to work at
Google I now know Everything there is to know about Anything. I try to be
helpful when I can, but listen up folks: it's been nearly five years since I
worked there. Everything I know about the current state of the company I know
from reading the papers (or Bob Cringely).
I cannot offer you any particular insight about how to get a job at Google, or
how to prepare for an interview, or how to get better tech support for your
blogger account.
But as long as I've gone to the trouble of
getting on my soap box, here's a bit of advice I can offer: when sending a
cold-call email, a little common courtesy can go a long way. Introduce
yourself. (A simple "Hello, my name is..." is fine.) Tell the person
you are writing to how you found them. Just because you found someone through
(say) their blog doesn't mean everyone has. Tell them a little bit about
yourself and what you are doing that leads you to ask whatever it is that you
are asking. And finally, you can get a surprising amount of leverage out of the
odd "please" and "thank you." (But don't overdo it. Being
demanding isn't cool, but neither is groveling.)
And for God's sake, never ever send a
cold-call email from a mail account with a self-aggrandizing alias like
"Lord and Master of the Known Universe." You may just as well tattoo
the word "dweeb" on your forehead if you're going to do [expletive deleted] like
that.
We now return you to our regularly
scheduled intra-holiday lull.
34
COMMENTS:
Anonymous said...
More eggnog!
Captured
Intrigue said...
Hahaha....
Signed,
Princess
of The Dark Realm Of Pixies That Hunt People Who Worked For Google But Are
Expected To Know It All Still.
That is like asking somebody who used to work
for McDonald's 7 years ago what this week's prices are for a chicken nugget
meal.
SJ
said...
Hey you, get me
a job at Google ok. And make it fast.
:))
Happy holidays!!
NN
said...
Hi,
i'm am
errr... a student of business. i read you blog because it gives a lot of
insight on how google works/worked.
i don't remember where i came from.
but i
come here often :D
-me
PS: can please make my professors stop talkign about
google!
Pasamio the Geek said...
Hello, my name
is Pasamio. I found your blog when it was on slashdot and had a slow and gentle
read through the posts that the two of you have made here.
I'm a geek and I'm
wondering if calling myself 'Pasamio the Geek' is too dweeb like.
I await
your answers! :p
On a more serious note, very interesting historic read from a
time that I often wish I was a part of, but didn't get the chance. Season Greetings
guys, thanks for providing more information for the world to digest!
Anonymous said...
/haha
can u
teach how hack? source code plz.
send to lordofall@hotmail.com
Anonymous said...
umm ok why is
whereever I go, google is there!! ?? !! ah
Anonymous said...
HOW CAN I GET A
JOB AT GOOGLE??? I WANT A JOB AT GOOGLE> GET ME ONE NOW> hahahahaha
Anonymous said...
A little
courtesy goes a long way. Great Blog...Thanks
Signature:
Get $25 free on first
deposit at the world's largest poker room with 70,000+
players
http://www.partypoker.com use code: shefa
Dr. Ægustüs Ŗiviŋiallli
said...
What a glorious
place the World Wide Internet is! In this vast "web" of information,
we find expertise in the most mundane of areas, such as searching for jobs! Oh,
the joy of this cornucopia of knowledge!
Talha Masood
said...
:)
interesting
thank
You!
Anonymous said...
Can you blog
about the growth currently at Google? Your thoughts? Why does this company
function as a institution rather a business? Is Google the replica of Stanford
but in Search form?
Trish
said...
Hello lord and
master my name is nobody, any idea how many hits you get on google when you
type in buffalo tack glass and why?
khaled
said...
SJ
me
too!!!!!!
Steve said...
Surely if you
were the best and the brightest and worked at el Goog you knew everything at
that time, and therefore still know everything now? And I DEMAND you tell me
EVERYTHING! (As I understand, CAPITALISING some WORDS is likely to get you a
FASTER response).
Writer on Board
said...
Trish says her
name is Nobody. I'm Nobody too. Will the real NOBODY please stand up. What
happens when you type NOBODY into GOOGLE?
Tell me Jason.
Tell me Mr.
Google-Man, Mr. Googl-ator, Senor Goog-alito! Tell me!
Sorry. I'm having a
really hard day. Good Blog.
-Jason Nobody
Silyth
said...
Sometimes I
think a hint is lost on others since you state clearly your response on some
things and people just come back asking the same things. Oh well, how tedious
it must be for you.
As you mentioned before introductions are nice so...hello
I'm Silyth. It's a pleasure reading about what you want type. Now I want to get
back to reading and I wish you a good holiday.
Thanks for the good
read!
Silyth
http://silyth.blogspot.com
W.
said...
I am a Berkeley
educated Social Scientist. I can tell you how to get a job at Google. I have
cold emailed Google in the past to evaluate how they respond to various
personalities. As well, I have been inside the mind of some of their stars.
LOL
Ask yourself if you want to have some guy who is socially dysfuctional say
things like "shit" to you because he got some false confidence after
his equity in the company grew. I would rather work as a park ranger with
people that actually are not so insecure.
Anonymous said...
I don't mind
reading interesting stuff from someone even if they are potentially socially
dysfunctional or use words like "shit". Not to imply that the former
is the case with xooglers, I can't see why someone would even care.
submergent
said...
I know what you
mean. Once people find out I once bagged groceries for a couple years out of
high schools, it's non stop questions about spacial relationships and weight
ratios. It's a curse.
Sarah Khokhar
said...
My neighbours
think I can solve all their artistic needs just cuz I learned most of the Adobe
stuff in class. All I really know is how to crop an image.
samuel
said...
I kept reading,
waiting for the guy with the problem with a word. That shit is fucking weak! It
was actually kind of a let down when I did finally read it. I was ready for
moralizing and I got some dimwit that ran out of real things to bitch about.
So
is it true that you know the HR guy at Google? email me at
drunkanddisorderly.com Dude, totally hook me up!
speed_demon
said...
Hi there,
No
offense, but how do you feel when google went IPO?
Man, working at google must
be great. It must have felt like "walking among the giants"!
Happy
New Year!
The Gradster
said...
This is really
stupid...
http://toypalace.blogspot.com/
go there!!!!!!
remcgregor
said...
Just wanted to
comment on what a great Blog you have here. Google as a company has always
fascinated me. Looking forward to when you guys are back and blogging again! :)
IanShorten
said...
Google is a
sweet company. I've used google for so fucking long and so naturally when Gmail
comes out I flipped. Wish they would make a win 98 compatible google talk.
Anonymous said...
Isn't blogger
owned by google?
The Urban
Kunoichi said...
Hello there,
I
began to read your blog since it was in the featured list at blogger.com.
Great blog!
But I have to admit, I get as much enjoyment reading your
comments as I do from your own words!
Thanks for the entertainment!
Anonymous said...
To the
"Berkley educated Social Scientist" who could probably tell everyone
a little bit about everything: A really big ego is often a sign of insecurity.
So, you seem to have earned your false confidence at Berkley.
You probably
couldn't fit your ego into a park ranger's shirt.
Kimberly
said...
Lord and
Master,
Please please please please please please please please please please
please please please please please please ...ignore this post.
Thank you,
thank you thank you thank you thank you thank you thank you thank you thank you
thank you.
b said...
please can u
give us ignorant folks some fast fast search advice.thank u in advance
Anonymous said...
Jokes.
Harsha
said...
That is funny..
Adbloggers.com
said...
I love this
blog. I check it often for updates. Keep blogging!
Ron
Wilson
www.adbloggers.com
WEDNESDAY,
JANUARY 04, 2006
We are experiencing technical difficulties
POSTED BY RON AT 9:47 AM
Happy new year everyone!
If you've been watching my personal blog
you will have seen that I've actually been back for a while now. The reason
I've been quiet on xooglers is that Google legal has asked to talk to me about
it (no big deal I am assured) and we've been having trouble making contact.
[UPDATE: I was hoping this would just
pass, but I guess I'd better clarify before it spins wildly out of control:
Google has not asked me to stop blogging. They have not even remotely hinted
anything of the sort. All they have done is ask to talk to me, and I have
entirely on my own initiative stopped writing Google stories as a courtesy to
them until we're able to make contact and I hear what they have to say. So I'm
sorry to disappoint all you conspiracy theorists looking for a juicy tidbit,
but there is nothing more sinister going on here than a game of phone tag.]
[UPDATE2: We finally made contact. The conversation
lasted all of five minutes, and the message was basically, "We think
Xooglers is great, but please be careful not to reveal any confidential
information, and if you have any doubts about what might be confidential please
feel free to call us and ask." No big deal. So I should be picking up the
story again later today.]
So in the interim I thought I'd tell a
story that Google can't possibly object to because it happened in 1997, a year
before there was a Google.
Doubtless what fascinates many people
about Google is the vast amount of money it has generated. Larry and Sergey are
now billionaires. They've bought a 767 (which is not quite the splurge it might at
first glance appear to be). What is it like to have all that money?
Well, I have no idea what it is like to
have that kind of money,
though I can report from firsthand experience that having some money is awfully
nice. But it's no panacea. Money can let you eliminate some of life's
headaches, but it comes with some of its own. Don't get me wrong, I definitely
place these in the category of "nice problems to have", but they are
nonetheless problems.
My most direct encounter with
"that" kind of money came in 1997 when I attended a technical
conference in Santa Barbara. Because SB is pretty close to where I live and I'm
a pilot, I decided to fly myself there. At the time I was flying an old Cessna 182RG, which is a
pre-dark-days four-seater with retractable gear (hence the "RG". No,
it's just a coincidence that my new initials are RG. The plane I'm flying now
is a Cirrus SR22, which has
fixed gear.)
(As long as I'm digressing I guess I should
explain that the "dark days" were the period in the late eighties to
early nineties when no small planes were being manufactured in the U.S. because
of liability concerns. At the time, common law held that an airplane
manufacturer was liable for manufacturing defects in the airframe no matter how
old the airplane was. In 1994 Congress passed the General Aviation
Revitalization Act which limits the liability of aircraft and parts to 17 years
after the date of manufacture. 17 years still seems like a long time to me, but
apparently it's enough to let you make a business case, and since then the U.S.
aircraft industry has undergone a renaissance.)
Anyway, it turned out that I was not the
only one who had flown their own plane to the conference. There was a very
highly ranked executive of a major software company there as well. I won't say
who it was or what company he was with. I'll just call him "Chuck." I
had actually met Chuck several years earlier when I interviewed for a job
(which I didn't get) so I re-introduced myself and we started chatting. It
turned out that both our planes were parked at the same place at Santa Barbara
airport, so I offered to give him a ride to the airport when the conference was
over, an offer which he declined.
So the conference ends, I get in my little
rental car and start to drive out of the parking lot when here comes Chuck
running after me waving his arms. It turns out that his ride to the airport
hasn't shown up and can he take me up on my offer after all? Sure, I say, hop
on in, which he does. He then goes on to make himself comfortable by taking off
his shoes and putting his feet up on the dash. OK, whatever.
We get to the airport and we both walk
into the FBO. My
plane is off in the boonies, but his is parked right out front. It's a Falcon 2000,
pretty much top of the line in its day (about $30M worth of plane I believe).
The red carpet is rolled out (literally) and a little army of crew people are
running around getting the thing ready. I pick my jaw up off the tarmac and
wish Chuck a pleasant journey.
I walked over to my plane, which seemed
very, er, cozy at this point, and started preflighting. In the distance I can
hear the Falcon spooling up its engines, and then spooling them down again.
Then once more, up and down, and again. And again.
Hm, that's odd, I think. Why don't they
just leave?
Then they shut the engines down and open
the door. Hm. Doesn't look good.
I go back inside the FBO and there's Chuck
sitting on the couch looking very unhappy. "Disaster," he says to me.
One of the engines has a broken part. They can't take off. And he has to be in
Europe in eighteen hours for a wedding.
Since he had no where to go, Chuck invited
me for a tour of his plane. It was a pretty sweet ride. Overstuffed leather
upholstery, a nicely stocked bar, I think there was even a shower in back. But
the part that really got my attention was in front: it was the first time I'd
ever seen a glass cockpit
up close.
Being in a plane like that without being
able to go flying is kind of like being in a candy store without being able to
eat anything, so I hung out for a few minutes, then thanked him for the tour,
and took my leave. I offered to fly him to LAX where he could catch a
commercial flight to Europe but he declined, which was probably the right thing
to do. Flying in to LAX is theoretically possible in a small plane (I've
actually flown to JFK in a four-seater once), but it's not something you'd want
to do lightly, and at the time I was probably not experienced enough to do it
right.
So I flew home. It was a perfectly clear
night, and the city lights were spectacular. I marveled at the fact that I was
going home while behind me a man worth many billions of dollars was stuck at
the airport like an ordinary shmoe.
Postscript: thinking back on it now, I
have no idea why he didn't just call a charter company and arrange for another
plane to come pick him up. For a few hundred thousand (which should have been
chump change by this guy's standards) you can have your pick of the fleet at
your beck and call. Maybe if I ever run into Chuck again I'll have a chance to
ask him.
22
COMMENTS:
EmailHosting.com
said...
What is a
xoogler?
http://search-engines-web.com/
said...
That is sad
that there are potential concerns about the blog express by Google legal - they
must understand the effect Google is and has had on the planet - and Google
history should be more than press releases and PR speeches given at
keynotes....Good Luck
By the way, read Matts blog about the new BigDaddy
Penguin
said...
are Xooglers ,
former employees of google?
Anonymous said...
I wonder - was
that Larry Ellison, or Paul Allen ? Hard to think of many people with that kind
of money and interest.
tby
said...
In my
experience, people that have achieved wealth (as opposed to being born into it)
sweat the little things. $30M plane, no problem. $12K for last-minute
first-class airfare to Europe, problem.
I knew a guy that sold the company he
co-founded and walked away with nine figures. He got the Ferrari, the sailboat,
a small island with a log cabin of McMansion proportions in Canada. He made his
wife go back to dial-up because the ISDN bill was a couple hundred bucks one
month.
Anonymous said...
the glegal
stuff is lame! this blog has a following. are they stalling on purpose? anyway,
i say give them another few days to say whatever they need to say, and then
start writing again.
Meme chose
said...
Ha! My friends,
who owned two planes, used to say: "If you really have more than enough
time, take your own plane".
Anonymous said...
I hope the
"little talk" with Google isn't going to disrupt this blog.
Keep it
up, I love reading it :)
TF
rafiq
said...
A few more clicks and reading would tell you what xooglers are.
The Xooglers blog is one of the best blogs I've ever come accross. Legal
Shmegal, I think we should have the right to know what the guys over at the big
G are up to seeing that they know what we are up to if we use their services,
gmail, personalised homapges blah blah. Who know our lives online are are up
to. Welcome back & Keep it up.
Instead of asking I will google for your
personal blog or just check this one if it is somehwere here.
off topic: If
friends or family ever use you notebook/workstation/tablet or pc be sure to be
logged into your google account :) I wonder if there is a study comparing your
personality to how & what you search for?
Anonymous said...
My guess would
be Charles Simonyi
Tim said...
Yeah, I'm going
for Charles Simonyi too. Doubt it was Paul Allen - I've met both and Charles is
more sociable than Paul. It didn't sound like Paul. Well, he said something,
for a start :)
Anonymous said...
Keep up the
great posts! Please keep us updated on the legal stuff. We may not be lawyers,
but I am guessing your dedicated readers can help make some noise if you need
it.
TwistedNoggin
said...
The little
planes are SO much more fun to ride in. I wish I could pilot.
Sorry about the
Google complications. Hope that all works out well. :)
Anonymous said...
Interesting
that you'd choose to call me "Chuck". Heh.
Ron
said...
Interesting
that you'd choose to call me "Chuck".
Well, I didn't want to have to keep referring to you as "this
person" so I had to call you something, and I figured you wouldn't want me
to use your real name. If you prefer I used a different name contact me by
email and let me know and I'll change it.
(For the record, I have no reason to
believe that this particular anonymous really is Chuck, but you never know.)
Ron
said...
This
post has been removed by a blog administrator.
Ron
said...
This post
has been removed by a blog administrator.
Ron
said...
OK, I can hear
the conspiracy nuts going to work on the two deleted comments above. What
happened was that I was trying to respond to twistednoggin who wrote "I
wish I could [be a] pilot" by saying but for some reason the link
disappeared and the result looked really weird. Hopefully this more verbose
version will work...
Ron
said...
Wow, something
really seems to be broken here!
Anonymous said...
Are you
implying that the owner of the plane was a Google employee? And that the fact
that the plane did not work is your way of suggesting that Google does not
work? You better be careful there Mr. Blogger.
Bernoulli
said...
Kind of off
topic but...... does anyone know about the new IPv6 and do you think Google is
looking at this? Heres the link that made me curious.
http://www.imakenews.com/signal/e_article000497585.cfm?x=b11,0,w
Anonymous said...
I do have the
best guess of who Chuck is. It was Elon Musk former paypal's largest
shareholder right? I know he owns a falcon 2000. His spacex rocket's name is
also falcon.. He is not a billionaire though, I think
THURSDAY,
JANUARY 05, 2006
Lost in translations
POSTED BY RON AT 9:38 PM
In December of 2000 I had my first
performance review. I don't really remember it all that well. Both my manager
and I were keenly aware that on the one hand I wasn't performing very well, but
on the other hand the project that I was nominally in charge of had launched on
time and was a huge success. AdWords was starting to bring in a significant
portion of the company's revenue.
There was something else had started to
notice in the weeks leading up to the performance review: I wasn't being asked
to do interviews any more. Interviewing was widely viewed as a chore. It was
time-consuming and disruptive, but the company was growing, and someone had to vet the new hires. Personally I
enjoyed doing interviews, probably because they afforded be a break from
writing Java code. No one ever said anything to me about it, but after about
September or October I noticed that I wasn't being put on the interview roster
any more.
I remember only two significant things
from that first review. First, I didn't get a raise, which is customary (or at
least so I'm told) unless you've really screwed up. I suppose that was fair
enough; I would be the first to admit that I'd really screwed up. But the
second thing floored me: I was being kicked out of the ads group.
I was furious, though I tried hard to
cling to the tattered remains of my professionalism and not to show it. Three
months earlier I had offered to resign from the ads group and my offer had been
refused, and now they were kicking me out.
I spent that winter vacation in a deep
funk not sure at all what to do. I was getting so many mixed signals. On the
one hand I had twice tried to walk away from the situation with some honor
intact and twice I'd been refused. But on the other hand I was getting so many
signals that the company really wanted to be rid of me: no raise, no
interviews, and to top it off, getting kicked ignominiously out of the group.
I cannot for the life of me remember where
I was between Christmas and New Years of 2000.
I remember my wife and I having an earnest
discussion about whether I should resign. The situation seemed to be getting
untenable, but I'd already invested six months, and it was starting to look as
if Google was going to do reasonably well, and that the stock might actually be
worth something some day. I didn't really want to walk away from that. So in
January of 2001 I once again got up at 5 AM to catch the 7:25 Southwest flight
from Burbank to San Jose.
Oddly, one of the things I remember most
clearly from this time was this big patch of impatiens that were
growing in a flower bed in one of the parking lots adjacent to the Googleplex.
I had hooked up with a ridesharing partner who was doing the same commute I was
(you'd be amazed how many people do insane commutes in California!) and he
would drop me off at the corner of Shorline Boulevard and Amphitheater Parkway
and I would walk the rest of the way to the 'plex. It's a beautiful area, right
next to a wildlife preserve.
Imaptiens are annnuals, and I thought that
they were supposed to die, and yet here was this batch of flowers just growing
like gangbusters. The colors were brilliant. And as the winter wore on they
just seemed to grow and grow and grow. It's amazing how a little color can
cheer you up.
To mark the start of the new year and to
really put the cap on my ignominy, the manager with whom I had had all the
friction went on maternity leave shortly after the new year (as best I can
recall -- I could be a little off on the timing of events here. My memory of
this period is not the best.) She was replaced with a fellow named Ross, with
whom I ended up getting along much better. At about the same time there was
this big reorganization of the engineering department, and I ended up -- wait
for it -- back in the ads group! Or at least I was reporting to Ross along with
all the rest of the old ads group members. I wasn't actually working on ads
stuff any more.
My new project was the Translation Console. At
the time, Google was available in about a dozen languages. The translations
were provided by a professional translating service which was very expensive
and (I was told) not very reliable. The translations were also not particularly
good. The solution was typically audacious: build an interface to allow
Google's users to translate the site contents into their own languages for
free. The result would be higher quality translations in more languages (over
100 at last count, including Klingon!) for free.
The problem with this approach is, how do
you do quality control? If nobody at the company happens to speak, say,
Serbo-Croatian, how can you tell if the ostensible Serbo-Croatian translation
of "I'm feeling lucky" really means "I'm feeling lucky"
instead of "Visit my hot new porn site"?
Again, Google audaciousness: let the users
do the quality control too. The translation interface allows you to not only
translate new content, but also to correct errors that you find in already
translated content. As long as there are more honest translators than there are
people trying to introduce spam, the spam will quickly get corrected away. Just
as the Google search engine computes a "Page rank" based on how many links
a page has pointing to it, the translation console computes a "translator
rank" based on how many translations you've submitted and how many of your
translations have been corrected by other translators. It works like a charm.
To my knowledge, there has not been even a single instance of translation spam
that has ever gone live onto the Google site.
I wish I could take credit for all this
brilliance but I can't. I wrote the code, but the idea was conceived by Marissa
Meyer (whose name I feel safe mentioning because her role in Google has been
well documented in the press) and Deb (whose last name I'll withhold because
her's hasn't). Marissa did most of the design, and I did the actual
implementation. It was not the most challenging thing I've ever done, but at least
I got to do it in Python instead of Java. If you can't have Lisp, Python is the
next best thing. I also got to do some hacking on the Google Web Server code so
that it could use the new translations, which was very cool and a little bit
scary, because if you screw that up you will definitely hear about it.
It was probably good for me to have
something a little less demanding to do at that point in my life. I got to
decompress a little, and I had some free time to explore other parts of the
company. In particular, I got to know people in the research group, which is
where I thought I probably would be the best fit, but it soon became clear to
me that I was way out of my league here. The sad fact of the matter is that my
own field, autonomous robots, is (or at least was -- things may have changed)
rather badly screwed up, and there was an awful lot of bogosity that passed for
scientific research. (I actually published a paper about this at one
point, which went pretty much unnoticed.) But the people in the Google research
group were actually doing real stuff. I would have had to work for months,
maybe years, to catch up to them, and it was becoming more and more obvious
that my days at Google were numbered, so it seemed pointless to try.
I set my sights on my one-year anniversary
in June when I would finally vest my first (and only) batch of stock options,
gritted my teeth, and settled into the routine.
I can hardly remember any of it any more.
21
COMMENTS:
Anonymous said...
MORE ABOUT
GOOGLE :
Unofficial Google Blog
Google news
Steve
said...
You surely had
a lot more strength than I did. I aimed for doing at least two years with my
previous employer, but I just could not hang on. I made it to my 18th month
with the company, and although I promised my peers that I would try my best to
stick around, I just couldn't.
The people who were working there did not drive
me away from the company. It was the environment. I honestly could not give a
feasible prediction on what I would be doing there over the next five years,
and if it was me doing the same thing I was doing at the time, that future is
simply unacceptable.
I am not even sure if I still have any stock options, but
obviously the number of stock options and their potential value could not keep
me from leaving. The promised quantity of options was also low enough not to create incentive for me
to push for the advancement of the company.
So I left.
Veky
said...
Yes, your
translation console worked pretty well. It's interesting that you mention
"I'm feeling lucky" translation to Serbo-Croatian, because there was
some controversy about it (in Croatian at least -- Serbian and Croatian are two
"different" languages now). However, it wasn't about spam, but about
clumsy translations. I guess you English-spaeking people don't really get that
feeling: when something just stabs you in the eye as a bad translation. Yes, I
know that "I'm Feeling Lucky" is very hard to translate (there was
even a whole entry in your FAQ about that particular phrase, IIRC), but I
thought I had a translation that sounded better -- at least, it evaded problems
with gender and cases, which are so common in automatic translation to
Croatian.
However, I couldn't communicate it to Google well. The problem in
your ranking scheme, as I perceived, was that for many "small"
languages, one person had translated mostly everything in the beginning, and if
just one his/her (you see? gender problem:) translation was bad (and he/she was
convinced it was good), there wasn't much you could do -- because his/her
"rank" was too high compared to your one.
But, I'd like to tell it
was overall a pleasant experience (much the same as your work at Google -- a human
mostly talks about the bad side of things). And it was surely much better than
what we have today (e.g. for Gmail) -- where you can't even enter the circle of
translators except if you are "chosen". :-(
Danny
said...
There was at
least one translation problem I know of that, if not spam, certainly wasn't
meant to help the program. The search button used a slang word in Malta that
means "penis in a can." More on that
here:
http://blog.searchenginewatch.com/blog/050412-160229 cover
Greg Linden
said...
Minor
correction. "Marissa Meyer" should be Marissa Mayer, I suspect.
Anonymous said...
You write:
"I got to know people in the research group..." and "...the
people in the Google research group were actually doing real stuff." This
is what gets me about Google. Four years ago I talked to some senior folks at
Google, about working there. I wanted to do research. I was told, in uneqivocal
terms, that "Google does not do Research and Development.. it does
Engineering and Development". One senior fellow told me that Google does
not hire researchers, it hires engineers. That was enough for me to hear, and
so I have never bothered applying for a job.
So what is this "research
group" you're talking about? Development, yes. Engineering, yes. But
research? Hadn't I been told that "Google doesn't do that"?
Pshaw.
Google makes no sense.
Bernardo
said...
Hey. I've been
reading this blog with much interest since it got started. After working at
Google for about 9 months, I quit in November (on the 17th, the first day of
this blog, coincidentally). While the people I worked with were really awesome,
there were things about how Google is run that I could not stand. (No, I will
not talk about those things - I'm too afraid of Google's Legal dept, despite
(or maybe because of) having done some work with them). Because of my
less-than-great experience working there, I am naturally curious for any
information about how Google evolved from a small start-up to a secretive,
not-too-flexible, not-too-friendly large company. Hence my interest in your
blog. Beyond this, I should say you guys' writing is very good indeed - the
blog would be almost as interesting if it were about some other company. So
keep it up! (And Ron, through this blog I discovered your personal blog. Your
views on lots of things are very similar to mine (least of which is the wonder
of flight), so I'm enjoying that a lot).
The reason I finally wrote just now
was because I did some work with Google involving translation, so I thought I'd
throw in my two cents. Most of that work involved revising the Portuguese
translations made by that "very expensive and not very reliable"
service of translators. The translations were indeed sub-optimal. I had trouble
believing that someone was being paid a lot of money to translate this poorly,
inconsistently, and uncreatively. (When I started working at Google, Portuguese
was moving from being a Console-translated language to an officially-translated
language, but instead of having everything re-translated, we inherited some
material from the Translation Console days, which was also (predictably) not
expertly translated, but (surprisingly) not much worse than the professionally
translated material).
I was also one of the few people doing end-user support
in Portuguese (and actually the only person doing end-user support in
Portuguese for some products), so I was the first one to hear complaints from
users about our bad translations. There were so many examples of poor
translation, we (the Brazilian folks at Google) could not catch all of them.
But we did try.
I liked Veky's comment about how "you English-speaking
people don't really get that feeling: when something just stabs you in the eye
as a bad translation". I think enough of the world speaks good-enough
English that, when something needs to be translated into English, it's usually
done pretty well (which is not so true of other languages) so English speakers
just aren't exposed to really bad translation (with a few humorous exceptions,
compiled in a handful of websites like Engrish.com). English vocabulary is also
flexible enough that, if a slightly-wrong word is used, it doesn't look too
bad. But I would CRINGE looking at some of the Portuguese material on Google's
pages and products. I would then write somewhat angry emails to the people in
charge of this material, saying "This is TERRIBLE. You're sending a
message that says 'International users don't really matter to us'." I knew
that they were busy, and that having the Portuguese material be perfect was not
their highest priority, but I had to give them a hard time. =]
However, later
in Veky's comment, there's a part about how "you can't even enter the
circle of translators except if you are 'chosen'." I disagree. Well, the
REAL reason that I'm writing is in the defense of the people in charge of the
international side of the products. When I wrote them an email pointing out a
mistake in translation, it usually got fixed. So it would be incorrect to paint
an image of a Google that is too lazy, too skeptical, or for some other reason
too reluctant to correct its language mistakes. International users matter a
lot to Google. If it doesn't look like it, then this is just because there is
just TOO MUCH to be translated and revised, and the few people at Google who
speak (insert your not-super-common language here - like Portuguese) have
things to do other than revise mountains of dubiously-translated material.
Until that material gets revised, it's better to have that poor translation on
our products and pages than no translation at all, right? But my point is, if
you see a mistake, write Google about it (using the "contact us" on
the website of the product/service where the mistake is), and they'll usually
do something about it. As more and more international people are hired, and as
more and more material is revised and updated, I am sure Google's
foreign-language material will eventually be extremely well written. That's the
current tendency anyways.
Google is working very hard right now to be as
international as they can be - doing things like being sensitive to each
region's culture while adapting a product to new regions and languages. The
responsibility to be sensitive to local language and culture is not one Google
takes lightly - there are a lot of very smart people dedicated to this.
However, they are only human and have a LOT of work to do.
While I have some
perspective into how the company operates, what it wants, how it makes
decisions, etc, I would not share those insights publicly (I congratulate Ron
and Doug on their courage). So I don't think I'd be comfortable sharing any
more than I just did. Besides, almost all the work I did at Google was not very
interesting at all. So don't bother asking me about it...
All right. Keep up
this great blog!
Bernardo
said...
Ah, one more
thing. Ron, you mentioned that the Translation Console languages numbered at
"over 100 at last count, including Klingon!". But I don't think a
user can ADD a new language to the console - only the Googlers in charge of the
console can. I heard that Klingon, Elmer Fudd, Bork Bork Bork (Swedish Chef),
Pig Latin, etc, were added while the console was being tested so that the
people doing the testing and the debugging could understand the
"languages" they were working on. I'm not sure whether or not this is
true. (Is it?)
Some of the languages on the console are obscure enough that it
would be hard for me to imagine they were suggested by the Googlers in charge
of the console. Guarani, for example, is one of the languages spoken by Native
Brazilian tribes. I wonder if a Brazilian person wrote in, saying they were
willing to translate everything into Guarani, at which point the Googlers in
charge of the console said "Why not" and added it. That to me seems
to be the most likely explanation. (While working at end-user support, I once
received an email from someone offering to translate the Google pages and
interface into Qenya (Tolkien's Elvish language). I see this has not been
implemented, probably because most computers do not have the Tengwar alphabet
language pack...
localudal
said...
You say 'To my
knowledge, there has not been even a single instance of translation spam that
has ever gone live onto the Google site.'
Oh really? Who presented you with
that 'priceless' knowledge? My estimate (see
http://goolocalizations.blogspot.com, sections 'Goolocalizations' and 'Googlese')
there's tons of 'translational spam' and 'junk translations' in Russian,
Ukarainian, Japanese, Simplified Chinese. Someone with inflated 'page rank' and
on a local competition assignment produced translational frauds, so, say,
Google in Russian is just a silly joke compared to native Yandex.ru. Simplified
Chinese of Google by the same scheme is either laughable, or illiterate,
compared to native Baidu.com. Now, just for a second: Chinese market is no way
Serbian, or Croatian, to believe so easily to any crook 'experts'.
This is the
price of un-lingustic approach to translations and localizations, so even
outstanding Python, Java, and whatever Lisp knowledges can't repair the damage
done.
Besides, who said that professional (OK, maybe expensive, but never so
costly as the loss of Chinese market) translators did poor job? Your miserable
page rank of their Web site, lack of links to and from them? Who are you
kidding with that?
Measurements of text qualities based on authors'
claims/appearances/page ranks is a dangerous illusion. Say, I presume that page
rank of the Google.com itself goes through the roof, right? In your scheme, it
makes next to impossible then to really copyedit the junk American English
used, say, in originals of AdSense/AdWords descriptions.
Anonymous said...
all your base
are belong to us
http://search-engines-web.com/
said...
These scenarios
appear to have been a result Poor Communication and lack of individual
attention....
There HAD to have been a place at Google for you - but perhaps
the Management style was just too solipsistic, perhaps everyone was just too
focussed on SELF to see the entire perspective of things...
The poor
communicationm, may have also been a result of being too individually self
focussed...
It takes a superhuman to be Extremely technically skilled and also
be a people-oriented generalist Manager....
This excerpt is really
confusing...
I remember only two significant things from that first review. First,
I didn't get a raise, which is customary (or at least so I'm told) unless
you've really screwed up.
Ron
said...
I
heard that Klingon, Elmer Fudd, Bork Bork Bork (Swedish Chef), Pig Latin, etc,
were added while the console was being tested so that the people doing the
testing and the debugging could understand the "languages" they were
working on. I'm not sure whether or not this is true. (Is it?)
It certainly helped with testing, but that was not the main reason
those "languages" were added to the list. They were always intended
to go live, and indeed they were among the earliest languages to go live.
Some of the languages on the console are obscure enough that it would
be hard for me to imagine they were suggested by the Googlers in charge of the
console.
As best I can recall, we got a list of
languages from some official-seeming website somewhere (maybe the UN or
something like that?) I really don't remember. I just know that the list came
from outside Google.
So what is this "research
group" you're talking about?
[Shrug] maybe they
disbanded it. Don't forget this was five years ago. Or maybe they want to
discourage academic-style researchers who think that publishing papers is a
good measure of productivity. I don't know. All I know is that when I was
there, there was a group called the research group and they did researchy kinds
of things.
This excerpt is really confusing...
What I meant was this: Companies generally don't like to fire people
except in the most egregious of circumstances because that exposes them to
potential wrongful termination lawsuits. Instead they send subtle signals that
you aren't working out and hope that you leave on your own initiative. Not
giving someone a raise, even a small one, at their first performance review is
one such signal.
Anonymous said...
Google has
hired a number of "research scientists," many of them belonging to
the research team. (Try googling for "research scientist at google"
with the quotes.) It's an impressive set of people.
Anonymous said...
Oh, I know Google has hired a number people who are
top research scientists. It's just that they don't call them researchers. They
call them engineers.
Check out the Google Jobs listing: Where is the
"Research Department" on this list?
Google Jobs
I
see no research department. I see an Engineering department, yes. But no
research.
Maybe there really is a research department, too, but they're just
not hiring right now? Is that why the "research lab" doesn't appear
on the list? Maybe they've solved all the tough information retrieval problems
with all the researchers they already have, and don't need any more? Uh
huh.
Look, admittedly this is a minor point in the grand scheme of things. But
it's one that gets under my skin. It's mild schizophrenia on Google's part.. to
continually talk about how they only hire engineers, and then also talk about
their fantastic research lab. Just call it for what it is, is all I'm saying.
If a researcher has to do more coding at Google than in a comparable position
at AT&T, IBM, or academia, just make that part of the job requirements. But
you're still hiring a researcher, not an engineer.
Anonymous said...
Whoever posted
that it would take $100 million to keep Mountain View's Slater Elementary open
didn't know what he was talking about. I have this from a VERY reliable source:
The cost of keeping Slater opened is estimated by the savings from
reduced expenditures that would otherwise exist in keeping the school open.
This savings in reduced expenditures is approximately $360,000. Additionally,
the cost of keeping Slater open is the loss of the potential revenue of leasing
the site. The potential revenue of leasing the site is estimated at $500,000.
Therefore, the savings in expenditures and the estimated increase in revenue
adds up to $830,000. This “cost” is ongoing; meaning, the district would need
to find this amount not just for the 2006-07, but for every year thereafter.
Surely that is within reason and even easily doable by a small group of
Googlers and/or Google corporate right?
Who is the right person to ask about
this?
Anonymous said...
Ah, sorry, I'm
the anonymous guy above, and I think I need to eat my words. A little bit more
scratching on this particular itch led me to find, buried in a non-immediately
obvious location in the engineering department, the following opening for "Research
Scientist"
So I guess it does exist. I do stand
corrected.
However, I also stand by the fact that some top Google tech folks
I've spoken with really have said "Google does not do research and
development; it does engineering and development". I still wish they would
call it for what it is. But I suppose that's just part of the whole Google
corporate personality. Anyway, back to this fascinating blog :-)
The Irrational
Investor said...
What about the
"bio-info-nano" initiative? That's engineering too? It actually
sounds scary.
Greg said...
Regarding the
"research" department at Google: back when Ron was at Google, there
was a group of people we called "Research". These days, I don't know
that most people think of there being an official research group any more, but
researchers are still employed at Google (at least in the Mountain View and New
York offices.) There are actually a couple of different groups that these
people belong to (including, for example, the Systems Lab, which is where a lot
of the Bell Labs people ended up.) And at least a few of the people I know from
research have moved on (for example, the director of engineering in New York
used to be in the research group.)
As far as the comment that Google doesn't
hire researchers, only engineers, I believe that you could have been told that.
But to clarify it a bit, Google doesn't want people who come up with ideas and
then don't implement them: a researcher has to also implement at least a
prototype of their system. So Google didn't want people who weren't willing to
actually build something, and that's all that's meant by "engineering and
development as opposed to research and development".
Maybe the research
the anonymous poster posted about above was not concrete enough. Or maybe
he/she was talking to the wrong people. But there certainly was a research
group at Google, and without it, we wouldn't be seeing Froogle or Google News,
among other contributions that might be less visible.
Steve
said...
I think this is
a great blog. I'm very interested in the inner workings of Google, and somewhat
angry with them right now since they kicked me out of Adsense because my
friends and family clicked on my ads without my knowing it. Next time I will
keep my mouth shut. But I digress...
It has been interesting to see Google
rise from a little-known search engine to some kind of Internet...
conglomerate, for lack of a better term... I don't really know what to call
them now. Keep up the good work guys.
gonights.blogspot.com
Anonymous said...
Greetings
again.. I'm the anonymous "research" guy from above. I'd posted three
times in a single day, so I decided to give it a rest for a few days, and let
the majority of the readership move on, lest I come off as a troll.
Anyway,
in response to both Ron and Greg, above, I very much understood at the time,
and still understand today, that Google is not interested in hiring someone who
thinks productivity is measured in the number of papers published.
My own
area of research is multimedia information retrieval. In order to evaluate the
research, I have had to build, over the years, a number of functioning
prototype systems. With no system, there is no evaluation. With no evaluation,
there are no results. With no results, you really don't have any research. So
the prototyping component has always been a part of the research. That wasn't
an issue.
So again, my main agitation was just that Google really just called
everyone "engineers", and everything they did
"engineering", even though I clearly knew that Google had hired
researchers.
If that were the only issue, I wouldn't be posting a comment again.
However, in the past two days my mind has really started to change and clarify
on this whole matter. Something clicked in me, when Google released Video. In
just the past two days, I have begun to think that Google really does not do
research. That they really only spend their efforts on engineering. Even with
that 20% time. Let me explain.
See, when I talked to these senior Google
managers back in 2002, the thing they told me about the research/engineering
schedule is that the idea-to-prototyping schedule lasts 5 months. (I've since
hear the same thing repeated from other sources, so I don't think I'm giving
away any NDA here by talking about the 5-month cycle.)
In 5 months, Google
expects you to come up with a new idea, work the kinks out of your idea, and
then implement the idea in a working prototype, so carefully crafted that it
could go live on the Google main page one or two months later, after being
passed off to the hardcore coders.
So whether or not Googles calls them
"researchers" or "engineers", they basically only have a
month or two to really be able to come up with new ideas, before you have to
start hacking away, debugging, etc. to come up with this
"soon-to-go-live" prototype.
I think this is too little time to
really think up and do the exploratory data analysis that you need to do, to
come up with really great new ideas. Real research is not just about the
prototyping. Real research is about having the time to do the exploration and
bounce all the ideas around, to come up with something good. If you're worried
about going live in four months, you just don't have -time- to do real
research. You will be forever stuck in this shallow mode of slightly
incremental improvement, but never a real breakthrough.
So I've been thinking
about this in the context of the new Google Video store that was released just
a few days ago. I've watched as the blogosphere has greeted the offering with a
yawn. Everything from "We Sat Around Waiting For Google Video And All We Got
Was This?" to "Google and CBS release embarrassment of a video
store".
And it makes me sad.
I was there, in 2002, with my multimedia
information retrieval experience, ready to buckle down and get some real ideas
flowing, get some new, working notions going about how to index, retrieve, and
present/display multimedia data. I was ready, with all that Google idealism, to
help change the world.
But even if I had applied to Google (which I didn't)
and gotten an offer, it wouldn't have worked. If the expectation was that all
the "research" someone in my position was doing had to go live to the
Google main site every half year, then any research at all that was
multimedia-related would not have been allowed... because Google Video itself
didn't go live until just a few days ago. There would have been no place on the
Google page to integrate those working prototypes, back in 2002, in 2003, 2004,
and 2005.
If Google did research, -really- did research, the kind that
requires a dedication window larger than 5 months, they might actually have a
video offering that wasn't so universally greeted as lackluster.
And search,
which I presume means all kinds of search, video included, is at the core of
Google's mission statement. That is where the research had been needed. Not in
developing interfaces for Google Talk. Not in incremental text search hacks
such as letting you type in a Fedex tracking number. But in really developing
the ideas (research) that will allow us to overcome the semantic gap in
multimedia search.
And it makes me sad because what attracted me to Google in
the first place, not just as a researcher but as a user, was this wonderful
dedicated focus on search. Now, I feel like nothing has really happened in that area (esp
multimedia search), when it should have been researched for years now. Maybe it
has been. I just get the feeling it wasn't.
Maybe there is a lot going on
under the scenes. But.. with all that, where is my ability to do relevance
feedback? This is a well-known, extremely useful IR technique. Why do we get
GTalk before we get relevance feedback mechanisms, for example?
So Google.. if
anyone is actually reading this post by this point.. you're a black hole. If
you really are doing wonderful stuff, you gotta let us know somehow. If you're
doing all this fantastic research, you gotta help us users see it, so that we
get excited about it. As an outside observer, I see what you've been offering
in the last 2+ years, and I don't see that you've really got anything that has
pushed information retrieval forward.
Eric Gometz said...
I read the
comment above about Guarani being an "obscure language" spoken only
by the native tribes of Brazil. Actually, it is the second official language of
the country of Paraguay. In addition to Spanish, it is spoken by over 80% of
all the people there. I am sure that the 6.3 million Paraguayans are happy that
Guarani was included by the Googlers in charge of the console. Just FYI.
THURSDAY,
JANUARY 12, 2006
All your T-shirt are belong to us
POSTED BY RON AT 12:21 AM
Worldplay wrote:
Google: All your searches are belong to
us!
which reminded me of something. It was at
the height of the "All your
base are belong to us" madness (and if you think that was weird
you should take a look at this)
when Doug held a little seminar to explain branding and marketing to us
technodweebs over in engineering. The seminar was advertised with an incentive:
everyone who attended would get "a special bonus prize" or something
like that. The prize turned out to be a Google T-shirt with the slogan
"All your search are belong to us" on the back. (Note that "search"
has to be singular to remain true to the spirit of the thing.) We were
admonished not to wear them in public.
Hm, I probably still have mine somewhere.
I wonder how much I could get for it on eBay? ;-)
9
COMMENTS:
Cole Blak Pascal
said...
I dun have a
lot of cash but I'd give you my bike for that shirt. Seriously.
Anonymous said...
I think the ???
guy should get a shirt just for that comment.
Ryan
said...
Ron doesn't
like when someone doesn't speak or write correctly. I think that
"dun" disqualifies you.
Ryan
Michael
said...
The "All
your base" phenomenon was certainly intriguing. My favorite sighting was
in the index of Scott Meyers' Effective STL. It's on the second page of the index,
between "algorithms" and "allocations". I think the indexer
just slipped it in at the last minute.
Doug
said...
Yeah, that was
one of the first non-standard logo shirts we made. I was surprised by the
turnout for that Marketing 101 session and we quickly went through the
allocated amount, pissing off some engineers who had been unable to attend. The
whole point of the seminar was the power of marketing and having a collectible
premium was intended to be part of the lesson. It worked pretty well. Btw, the
first non-standard logo shirt was the one announcing the Yahoo deal. Susan put
them together and they were given out at the party celebrating the signing.
Jerry Yang was there, I think, but I was off on a family trip and missed the
big event. I did manage to get a shirt though and have it stashed away
somewhere with the rest of my collection. I'm off on another family trip, but
if I get ambitious when I return, I'll post photos of all the t-shirts we made
in the first few years. I tried to keep at least one of each. If nothing else,
it should help future traders in Google logo gear establish provenance...
Anonymous said...
Your 'you
should take a look at this' link is one of the funniest things I have ever
read. A cure for depression, indeed!
I rather suspect it would actually work,
too.
Ron
said...
I
rather suspect it would actually work, too.
It worked for
me. (Reading the book, that is. I was laughing so hard that I couldn't be
depressed!) ;-)
Wordplay
said...
You're kidding
me.
You actually used that dumb phrase on a T-shirt?
I want royalties baby!
sarahintampa.com
said...
Thanks a lot
for the Amazon link. Fabulous. Now Amazon thinks I like books about my *^#@.
Ugh.
THURSDAY,
JANUARY 12, 2006
2001 in a nutshell
POSTED BY RON AT 11:00 AM
After I finished the translation console I
was quietly (and, as far as I can recall, unofficially) transferred back to the
ads group. Like I said, my memory of this time (early 2001) is pretty fuzzy,
and I seem to remember that official groups had been abolished by then, but I
went back through my old notebooks and from February through June I was working
on day-to-day operations of AdWords, with an occasional tweak on the Translation
Console (though it never really needed a whole lot of intervention).
Google was growing at a frantic pace even
as the dotcom world was melting down all around us. We were hiring like crazy
and space started to be a real problem. People were being rearranged and
stuffed three and four to an office or cubicle. By early summer it was, for the
first time, pretty much impossible to know everyone else who worked at the
company. Google was not a small company any more. The transformation had
happened with breathtaking speed.
It was a very exciting time for the
company, but a pretty dull time for me, and to be honest I was not unhappy to
have a little dull spell after my first six months. I had settled into a nice
comfortable if not particularly challenging groove, and I was worrying less
about making it to my one-year anniversary when I could finally stop this crazy
commute.
Urs Hoelzle, the guy who had hired me who
had been VP of engineering, stepped down (or maybe it was up?) from that
position and became the first "Google Fellow". He was replaced by
Wayne Rosing, a Silicon Valley legend who, among other things, led the (alas,
ill-fated) development of the Apple Lisa. In March, Eric Schmidt was hired as
CEO, and for a couple of months his office was two doors down from mine. (I
think he actually was sharing an office himself for a while, space was that
tight.) I never got to know either Wayne or Eric very well because they were
extremely busy, and I knew I was going to be leaving the company soon and didn't
really want to waste their time.
June finally rolled around and I had my
second performance review the week after my one-year anniversary, where I told
Ross I was giving my two weeks notice. Once more, to my great surprise, Ross
seemed dismayed. It seemed to me that the company had been sending me pretty
clear signals that it wanted me out, and by then I was only too happy to
oblige. But no. They wanted me to stay, despite the fact that at least by my
own self-assessment I was contributing very little by then. I shrugged my
shoulders and said, OK, if you want me to stay I'll stay, but I can't do the
commute any more. You have to let me work from LA.
They agreed.
Well, actually we compromised. Instead of
coming up to Mountain View four days a week I'd only have to come up one day a
week and telecommute the other four, which seemed reasonable. Some things
really require face to face contact. Even my wife was OK with the new
arrangement since it meant I would no longer be away from home overnight.
I was pretty sure that the new arrangment
wouldn't last long because Wayne Rosing was an big fan of colocation, even if
it meant stuffing people into cubicles like sardines (which was pretty much
what was happening by then). They moved my desk (which consisted of a door on
top of two yellow sawhorses) out into the hallway, which suited me fine. I
actualy had more privacy out in the hallway than I would have had in an office
at that point. At least I could lean back in my chair without physically
bumping into another person.
In early June I was also contacted by the
high-level manager at JPL who had made me promise to come back there after a
year asking me to make good on that promise. I had some reservations about
going back. Things had not been going terribly well for me there before I left
(that's a whole 'nuther story which I might write about some day) and things
didn't seem to have changed a whole lot while I had been away. Since I still
had the Google job I felt like I had a little leverage, so I told him I'd come
back if I got a promotion.
I have to digress a bit here and explain a
little about the technical career ladder at JPL. There were at the time four
rungs on that ladder: Associate, Staff, Senior and Principal. Climbing the
first three rungs was not particularly difficult (I was a Senior when I left)
but getting promoted to Principal was a Big Deal because once you became a
Principal you automatically got the ultimate JPL fringe benefit: on-lab
parking.
JPL is at once the smallest and largest
NASA center. It has the largest number of employees and the smallest physical
space. Parking has been a serious problem pretty much throughout the lab's
history. Only the privileged few (Principals, very senior managers, and people
who have worked there for twenty-five years or more) get to park on-lab. The
rest have to park in two remote lots and hike in to work or take a shuttle bus.
Even among on-lab parkers there is a
pecking order. The parking lots are color-coded. At the bottom of the hierarchy
is the green parking pass, which lets you park in the least desirable spaces.
Then there is the blue lot, which is centrally located next to the main plaza.
Then there are the legendary red and silver passes, which give you access to
the only covered parking spaces in all of JPL. There are only about forty of
these space in total. Less than one person out of a hundred gets to park there.
The upshot of all this is that getting
promoted to Principal is really really hard. It's the equivalent of getting
tenure at a university or making partner at a law firm. You have to be approved
by multiple committees, and finally, by the JPL Executive Council, and the
number of slots is extremely limited. Many highly qualified people get passed
over year after year.
My insisting on getting my Principal
promotion as a condition of returning had the effect of delaying JPL's ability
to make me an offer as it shepherded the paperwork through the system, which
suited me just fine as my Google situation had gotten fairly cushy, at least for
the time being, and I was more than happy to wait and vest in a few more stock
options.
You'd think that working from home would
be an ideal situation, but the fact of the matter is that it gets pretty
lonely. Face time is really important to building relationships, and email is a
very poor substitute. June turned into July and July turned into August. There
was one delay after another at JPL. Funding was tight. They wanted to promote
me, but couldn't find the money to pay my salary. And Google was starting to
hint that my days there were, finally, numbered. I was beginning to worry that
the return to JPL could fall through and I'd end up unemployed, which would be
financially disastrous. Coming up with the cash to pay for exercising the stock
options on top of my commuting expenses had left us with virtually no cash
cushion. We were living paycheck to paycheck.
Then, at about 8 AM Pacific time on
September 11, 2001, I got this email:
To: ...@google.com
Subject: Is NY Alive?
In case you aren't watching... one
trade center collapsed.
The pentagon is on fire.
turn on your tv.
WTF? I turned on the TV. I stared at it
for a while in utter disbelief at what I was seeing, then went downstairs where
my wife was making coffee.
She could tell just by looking at me that
something was up. "What's wrong?" she asked.
I said, "It's going to be a bad
day."
15
COMMENTS:
Anonymous said...
Ron, Did you
ever find out why they rejected
your resignation (twice)?
Ron
said...
Did
you ever find out why they rejected your resignation (twice)?
No, but stay tuned. The story is not quite done yet.
http://search-engines-web.com/
said...
There are
statements being made constantly about "feeling" that days were
numbered - but there is no illustration of exactly Why those feeling were valid
and what "specifically they were based on - besides not getting certain
anticpated reactions - were your expectations too high about how explicitely
you should be reinforced that you were a valuable commodity.
If nothing was
adversely said - it could have just been everyone being - justifiably -
extremely busy with themselves to constantly reinforce the more advanced people.
It may have been
assumed that advanced people are not in need of great reassurance.
If better
quality people were available - wouldn't they have hired and use them - with
all the quality resumes they were getting???
~~~~~~~~~~~~~~~~~~~~~~
Could there
also be posts about any initial concerns about Click Fraud becoming a problem -
and what proactively was done to help prevent it in AdWords - and any Politics?
Haridas Dave
said...
Ron, yours posts
are very engrossing, hope you write more than your normal quota for a day.
Waiting eagerly for whats next !!!
Ralf
said...
Just wanted to
stop by and say: I read this all the time and i really like it. Keep up the
good work! Now, back to my code....
Isaack
said...
What a
cliffhanger, you left us with!
Ben
said...
I kept reading,
and the moment I looked at the scrollbar and realize it's coming to an end, I
screamed internally that I wanted more. Keep writing.
I'd also be interested
to see how Google's been hinting to you that your days at google are numbered.
what kinds of hints are they?
Ron
said...
I'd
also be interested to see how Google's been hinting to you that your days at
google are numbered. what kinds of hints are they?
Before June the main concrete indications were (as I mentioned in
earlier posts) that I wasn't asked to do interviews any more, and that I wasn't
given the customary token raise at my first performance review. On top of that
was my own personal sense that I wasn't performing well or contributing much.
After AdWords, most of the coding that needed to be done was in C++, and I just
wasn't very proficient in it at the time. (To prepare for the performance
review we had to fill out a form that included questions like, "List three
areas where you need to improve." My answer for my second performance
review was "1. C++ 2. C++ 3. C++".)
I don't know, maybe I was just
being paranoid, and everything really was hunky dory. I'll probably never
know.
After June, they were explicitly saying: Wayne wants everyone
co-located, and so if you can't be here we will have to let you go before long,
or words to that effect.
But like I said, stay tuned, there's more to the
story.
Ron
said...
Oh, two more
things:
1. Thanks to everyone for the kind words!
2. There was one other
reason I thought my days were numbered as June approached: I knew I could no
longer do the commute after June, and I wasn't expecting Google to allow me to
telecommute full time because I knew that Wayne wanted everyone colocated.
(That's why they were stuffing people in to the 'plex like sardines.)
Anonymous said...
I always want
to pat you on the back and tell you keep your chin up and things will get
better. So far as your ending of this entry, we all remember where we were that
day. I look forward to your future posts. Both of you have a comfortable style.
Anonymous said...
Surprised you
didn't just try to find some temp housing. During your time at Google apartment
rent fell and there were many vacancies not far from the 'plex.
These posts
have really been an eye-opener. It's good to see someone being frank and honest
about what goes on within the company.
Anonymous said...
hopefully, and
more than likely googlers@google.com isnt a real distribution list. Otherwise,
well - you can imagine
Anonymous said...
hopefully,
and more than likely googlers@google.com isnt a real distribution list.
Otherwise, well - you can imagine
Hey Anonymous, that
address sounds like the real deal, or something close to it. I'd wager that
most new tech companies have a list like that. But you should know that mailing
list software for internal lists is usually set up to only allow subscribers to
post. (As a side note, at the company where I worked at the time, we had an
internal IRC channel that everyone used. As people logged in, we'd tell them
personally...)
Kunal
said...
Hey - you work
at JPL? Finally, a person to sympathize with about on-lab parking. We should do
lunch sometime.
Anonymous said...
But
you should know that mailing list software for internal lists is usually set up
to only allow subscribers to post.
Yes - but any one
can fake a from address. I guess either way it would be easy to spam the entire
googlers group if that is the case.
FRIDAY,
JANUARY 13, 2006
The real surreal life
POSTED BY RON AT 1:22 PM
9/11 was the start of the strangest five
years of my life. It ranged from subtly weird to overtly bizarre. Example: at
about 10 AM, an hour or so after I'd first turned on the TV and the magnitude
of what was happening was beginning to sink in, one of our next door neighbors
called -- I swear to God this is true -- to complain that our cat was
defecating in his flower beds. Much later we learned that this person was
having serious problems with chemical dependency, but at the time we had no
idea.
Traumatic events weren't new to me. In
February of 1991 I came home one night and surprised a burglar who took a
pot-shot at me with a .38 on his way out the window. The bullet missed my left
knee by about a foot, punched through one of the interior walls, ricocheted off
a heating duct, and embedded itself in the living room wall. The odd thing
about that kind of experience, I have found, is that it takes a long time --
like days or weeks -- to become consciously aware of the extent to which you've
been traumatized. Immediately after it happened my heart was beating pretty
fast, but I didn't feel panicked or anything like that. I crawled over to the
phone and called 9-1-1, told the cops what had happened, etc. etc. The panic
came later, with the nightmares. It was several years before I could really get
a good night's sleep again. But the really insidious effect of post-traumatic
stress was that it really shortened my fuse. I had never been particularly easy
to get along with, but after that night and for a very long time I would say
things to people that in my more lucid moments I would be amazed at. It was
almost as if I -- or at least a piece of me -- had turned into a different
person over which the rest of me had no control, although at the time I just
felt like me. It's a very weird and difficult to describe sensation. Jekyll and
Hyde were not entirely a product of Robert Louis Stevenson's imagination.
9/11 did that to me again, and again I
wasn't aware of it. (And in retrospect I expect that's what happened to my neighbor
too, although he probably helped himself along with a couple of pills.) The
thing I did that I regret most was arguing to remove the Arabic translations
from the Google site. Fortunately, cooler heads prevailed and the translations
weren't touched, but there were a couple of hours when taking them down really
seemed to me like the right thing to do.
The days immediately following 9/11 were
oddly peaceful in LA. There were no planes in the air and less traffic than
usual on the roads, and the weather was gorgeous. I like to think that this
might have been a preview of what life will be like in a few decades when the
world really starts to run out of oil, but that's a topic for another day.
A few days after 9/11 my wife and I got an
invitation from a wealthy acquaintance of ours to spend the day at their house.
Actually, calling it a house doesn't even begin to do it justice. It's an
estate. I have been there several times and only ever seen about half of it. He
was a bond trader and the markets were all closed, so he had some time on his
hands. I can remember hanging out in the jacuzzi with his trophy wife (who has
since left him -- money really can't buy happiness) thinking to myself how
surreal this situation was. Two thousand miles away in New York people were
digging themselves out of the rubble and here I was sitting in a hot tub with
two gorgeous women drinking a glass of very expensive wine. It seemed horribly
self-indulgent on the one hand, but on the other I didn't see how sitting at
home moping was going to help anyone in New York.
(Fast-forward briefly to June of last
year. We were vacationing in Europe, and landed at London Heathrow about an
hour after the subway bombs went off. We were really struck by how little
overall impact it seemed to have on the city. Our hotel was in central London,
and when we drove in to town it was deserted, but by evening the busses were
running again and things seemed pretty much back to normal by the following
day. I don't know whether it's rational or rationalizing, but it does seem to
me that the best way to thumb our noses at the terrorists is to live our lives.
It worked for London.)
When they finally reopened the airports I
did a few more commutes, but it was completely untenable. What before had been
merely horribly inconvenient was now a serious nightmare. Fortunately, I was
flying out of Burbank instead of LAX, so things could have been even worse. I
shudder to think.
I don't know how long Google might have
kept me on if 9/11 hadn't happened, but it was obviously the last little nudge
that Google needed to finally let me go. My tenure there ended at the end of
September. At the same time the process of getting me re-hired at JPL came to a
dead stop. October rolled around with no word, and for the first time in my
life I was unemployed.
I don't want to be overly dramatic about
it because JPL did come through just a few weeks later, but let me tell you, if
you've never been out of work, count your blessings (and if you have, I know
how you feel). I had been working more or less continuously since high school
(I got my first job programming computers when I was a sophomore) and I never
in a million years imagined that I would end up on the dole. (I actually went
to the umemployment office and picked up the forms to file for unemployment
benefits, but I'm happy to report that I never turned them in.) It felt as if
life had become a game of musical chairs and I was in the wrong place when the
music stopped.
I finally got the word that my promotion
had been approved. I breathed about a dozen sighs of relief before I got up the
nerve to ask The Question, the one that a year and half earlier was the
beginning of the end of my Google career: what would I be working on?
The answer came back: they wanted me to
work on -- wait for it! -- search engines!
Aaarrrgghhh!
Fortunately, working on search engines at
JPL doesn't mean the same thing as working on search engines at Google. Working
on search engines at Google means you're working on a search engine. Working on
search engines at JPL means you're working on procuring a search engine, and that was actually a topic I knew something about:
call Google, place an order. (They had just come out with their search
appliance.) You don't want to know how many of your tax dollars went to pay me
to help shepherd purchase orders through JPL's procurement system.
Which brings me to one of the very few
bits of juicy Google inside information that I have to share. (I'm able to
share it because I learned it as a Google customer and not a Google employee.
Furthermore, Google has been aware of this for four years now, and if they
haven't fixed this problem by now then they deserve to be outed.)
We got one of the very first Google Search
Appliances to evaluate. As everyone by now surely knows, the thing works by
crawling your organization's intranet just like www.google.com does. But
because this was an evaluation copy we would have to send it back to Google
when we were done.
To address the obvious security problem
(there were files available on the JPL intranet that contained sensitive
information) the search appliance had a self-destruct feature that could be
invoked to erase all the files on the device's internal hard drive. The problem
was that self-destruct was a little too thorough. It erased everything,
including the operating system, making the device unbootable. On top of that,
the box was physically sealed with rivets. This made it impossible for us to
verify that the information had in fact been erased.
I had the very devil of a time explaining
to Google why this was a problem and why we could not just take their word for
it that the self-destruct feature did what they said it did. We could not send
the box back without actually verifying that the data had really been erased.
That meant either getting root access to the machine, or physically opening the
box and extracting the hard drive. Google suggested that we run a magnet over
the box, and it took me a long time to explain to them why that wasn't good
enough. (Just in case you don't understand the problem, it's this: it is very
easy to make it appear that the data is gone without actually erasing it. For
example, just erasing the boot sector on the drive will make the disk
unbootable without actually erasing any of the data on the disk. Erasing the
directory will make it appear that the files are gone to a casual inspection,
but in fact all the data is still there.)
Google didn't want to give us root access
because they didn't want us to be able to access their sensitive information (like their code). The solution seemed obvious to
me: self-destruct mode should erase everything, including the code, but leave
the machine bootable with no root password. That they didn't do this indicated
to me that Google might be starting to lose some of their technical edge.
We finally reached the following
compromise: they would send a technician down who would witness our running
self-destruct mode. After this the tech would open up the box, take out the
hard drive and leave it with us, and take the rest of the appliance back up to
Mountain View. This seemed fine.
The tech arrived and together he and I
pushed the Big Red Button, as it were. After launching self-destruct mode the
machine put up a screen saying "Self-destruct in progress..." or
words to that effect, with the colorful Google trademark bouncing ball
animation showing that us that the machine was still alive.
Half and hour later the machine was still
self-destructing. I asked the tech how we would know when it was done. He said
that it should have already finished, and he didn't know what was going on.
After another fifteen minutes or so he
pulled the plug on the machine. He plugged it back in, and it booted right up.
So much for self-destruct mode.
By now over an hour had gone by and the
tech and I were both starting to feel a little impatient, so the tech logged in
as root and typed "rm -rf /*'. (For those of you who don't know unix, that
is the command that erases all the files -- except that it doesn't really erase
them, it just erases the directory.) When that was done he opened up the box,
took out the hard drive, and handed it to me.
All the data, including all of Google's
code, was still on it. But we had already spent nearly two hours on this (or
several weeks depending on how you count) and I really didn't want to drag it
out any longer, so I just thanked him and walked him and the box back to the
visitor center.
But I was starting to have some real
doubts about the future viability of the company.
41
COMMENTS:
Anonymous said...
Delightful,
thanks Ron.
Anonymous said...
well... what
did you do with the code??? ;)
Larry + Sergey said...
Another killer
cliffhanger. Damed if this isn't the best blog on the whole stinking net.
Anonymous said...
dd if=/dev/zero
of=/dev/hda
Or better yet, plug the drive into another system running knoppix
and do a few alternating writes with 0, 1, and rand.
Is anything stopping a
customer from buying a unit and disassembling it?
Andrew Hitchcock
said...
the
search appliance had a self-destruct feature
Oh :( . I
read this line and was hoping it was a physical self-destruct. Perhaps in the
next version...
Dov said...
Amazing blog. I
think it should be made into a thriller movie : )
Regarding this latest post
about Google losing its edge, isn't a small incident like Google's refusal to
change their appliance erase specifications a little too trivial to back up a
"foreshadowing" statement like that?
Thanks for your story.
http://search-engines-web.com/
said...
A colleague
walking to a 9 to 5 job in the World Trade Center at
8:43 a.m., on that 70
degree, bright, sunny Septemeber Morning -
(just one
block away from the office 17 minutes, I'll grab a cup of coffee to go)
then, all the pedestrians suddenly looking up the sky at that Low
Cruising, Noisy Jet along lower Manhattan - minutes later crashing into the
first Trade tower, people screaming and running out and all along the
avenue...
then ultimately, dozens of office workers jumping out of windows
many floors up with no other hope,
then the second crash ...
then the
buildings collapsing...
the effect was a realization of how
easily life and hard work can suddenly disappear, and the need to savor life's
minutes and family more.
Anonymous said...
Hmmm, quite a
bind with the hard disk. Did anyone think about taking the hard disk out to the
JPL carpark and smashing in with a hammer? Hard to recover data from that!
Martin said...
"Damed if
this isn't the best blog on the whole stinking net."
Sure is. Keep those
updates coming!
Anonymous said...
If Google
already agreed that JPL keeps the HD, why did you guys insist on the the
self-destruct? You could just take out the HD, send the HD-less appliance back
to Google, and destroy the HD in whatever methods JPL deems fit without tying
up a Google tech.
Anonymous said...
hey, anonymous
above me:
did you read the thing about google wanting to be sure not to give
the harddisk with their secret code to the customer, hence the sealing of the
psysical box?
Eater
said...
writing zeros
with dd isn't really sufficent. 'shred -v /dev/{s,h}da' would be more
appropriate, writing multiple patterns of pseudo-random data to the disk.
increase the iterations with '-n' to satisfy your level of paranoia :)
pBay
said...
Nice blog guys,
continue.
Matt Bland said...
Why don't they
use two drives. One locked inside the machine with their code on it. And
another removable drive with your search data on it. Then all you need to do is
remove the data drive and send the application box back. It's completely
obvious to me that would be the solution for the next revision. Or
alternatively put a SATA, USB, FireWire or SCSI port on the machine for using
an external storage device for the search data as an option in the setup stage.
Anonymous said...
The
non-functioning self-distruct and shoddy tech work aside, this is a very
interesting problem.
It seems like most evaluation appliance boxes would have
a similar delimma. How do you protect your data and give the customer
verifiable control over their data?
The dual hard drive idea makes some sense,
but how does the customer know that you haven't pulled some of their data to the
other drive?
Putting your code on a ROM might work, but then you can't easily
patch it and you would constrained by size.
If your appliance is in customer
hands, you can never guarantee 100% secrecy. Even if you decrypt it as you run,
at some point the code is unencrypted. All you can do is increase the
difficulty level.
Brandon said...
Sounds to me
like this's a time when a physical self-destruct would be helpful. Something
which would physically destroy the HD and its contents, if not the entire
machine.
Anonymous said...
Realistically
from your perspective keeping the harddrive is the only solution available to
you.
Let's say their self-destruct feature followed the procedure you
advocate, and keeps a root login. How do you know their OS hasn't been modified
to report to you that the data has been deleted when, in fact, it still
exists?
Nothing!
You either have to physically keep the harddrive, trust
Google, or remove the harddrive, put it in a computer you trust and wipe it
from there.
Doug
Ron
said...
why
did you guys insist on the the self-destruct?
We didn't.
That was Google's insistence.
Ron
said...
How
do you know their OS hasn't been modified to report to you that the data has
been deleted when, in fact, it still exists?
You always
have to have a baseline level of trust or you quickly fall down a rabbit hole
of paranoia. In the case of the self-destruct feature we had actual evidence up
front that at least some people at Google "didn't get it", and that
it was therefore plausible that the feature was not properly implemented,
particularly since really erasing a hard drive properly is a decidedly
non-trivial thing to do.
To modify the OS in the way you suggest would have
required a significant covert effort on Google's part, one which, if discovered
(which sooner or later it likely would have been), would cause significant
damage to the company. At the very least it would have made a hash of their
"don't be evil" slogan. So it would be a significant risk with very
little potential benefit. It seemed safe to assume that Google would not put
special effort into doing something that stupid. (I'll grant, though, that this
is not always a safe assumption, c.f. the Sony rootkit.)
As an aside, I really
believe that Google does do its best to live by "don't be evil". I
think Larry and Sergey are idealists who in their heart of hearts really want
to change the world for the better. (By way of contrast, I think Bill Gates is
a ruthless businessman who just wants to maximize Microsoft's power, influence,
and profit by whatever means he can get away with.) So I trust Google at least
to try to do the right thing, but not necessarily always to succeed.
Is anything stopping a customer from buying a unit and disassembling
it?
No, but this was an evaluation unit and did not belong
to us. (We actually offered to buy it, but it was already an out of date model
and Google didn't want to have to support it.) We later bought a box and
someone at JPL did take it apart after the software license expired and it
stopped working. It's just a generic rack-mount PC as far as we could tell.
Rob L. said...
I
think Bill Gates is a ruthless businessman who just wants to maximize
Microsoft's power, influence, and profit by whatever means he can get away with
A common enough viewpoint, Ron, but in light of that what are your
thoughts on the billions of dollars the Gateses putting towards world health
through their foundation? The man is becoming or already is the biggest
philanthropist in the history of humankind, isn't he? Is that nothing but a PR
stunt?
Is it nothing compared to what we're going to see from Larry + Sergey?
Ron
said...
what
are your thoughts on the billions of dollars the Gateses putting towards world
health through their foundation?
I think it's
completely irrelevant to the issue of whether or not Microsoft is trustworthy.
I trust Google not because they are doing good things with their money, but
because they made their money (and continue to make their money) by playing
straight. Microsoft made their money playing crooked, and no amount of
generosity can paper over that fact. (And I think that to the extent they're
playing straight now it's not because they think it's the right thing to do but
because they think they have no choice. I think they'd play crooked again in a
minute if they thought they had half a chance of getting away with it.)
I also
think that a person's generosity ought to be measured not by how much they give
but by how much they have left after they give. A billion dollars is a lot of
money, but if your net worth is tens of billions then giving away a billion
here and there doesn't really call for tremendous sacrifice. To be fair, Bill
is still a pretty young guy and I'm sure he's not done yet, but I'm not quite
ready to nominate him for sainthood just because the numbers on the checks he
writes are big enough to make most people's eyes bug out.
SM Guy
said...
The part about
the being unemployed hit home.
Between Dec of 2002 and Sep of 2004 I was out
of a job for a total of 18 months. I have a BS and MS in Computer Science (but
I never wrote code for money, my expertise was elsewhere). 18 months is a mind
boggling time to be out of work for someone that never even imagined he could
be without a job. I still am shocked when I look at that number: 18 months.
I
ran a small software company and we had to essentially shut down. So I had to
fired my brother, cousin and 10 other employees (oh, that was fun) and tell 10
investors, their stock was worthless. And I got to watch every single penny I
had to my name (in the high six figures) evaporate overnight because I had all
my skin in the game.
Then I spent 8 months looking for a job. Got hired. My
wife and I thought we could actually start to live a real life. Boy were we
mistaken. 3 months later I was laid off from a well paying job and no severance
pay. 10 months of sickening, mind-numbing agony followed. I sent out over 3,500
resumes over those two periods of being laid off (I am SERIOUS. three thousand
five hundred). I contacted everyone I ever met (want to talk about eating every
morsel of pride you ever had). It became utterly surreal. How many resumes can
you actually send before it becomes ridiculous (and they were well written,
with great cover letters)?
I had no Google stock or any other for that matter
I could hope would pay off someday. I had cashed in every single investment,
down to my 401K's to pay the bills at home. It got to the point that I couldn't
even dream about buying myself a cup of coffee at Starbucks, I would feel too
ashamed to spend that much on coffee. Every day unemployed meant walking around
with a lump in my throat and a sickening burn in my belly wondering what the
hell was wrong.
So even though Google was a rough ride, thank your lucky stars
you have a nice job at JPL and got a year plus of options vested at Google.
Don't get me wrong I am certain you earned it and deserve every bit of success
you have. Just say a little prayer tonight thanking whatever god you pray to,
if any. It ain't so bad my Xoogler friends, it ain't so bad.
