(Tedious disclaimer: my opinion only, not speaking for anybody else. I'm an SRE at Google.)
We expect and accept a high false-negative rate. Our interview process is optimised for zero false-positives at the cost of many false-negatives. This is a deliberate choice. So yes, I would expect to see a significant rate of rejections of people who are clearly qualified.
The sort of people that we want to hire are likely to come back for another try anyway, and the long-term effect of this process seems to be doing what it was supposed to.
I've often heard this, and it seems a poor excuse for a terrible interview process/terrible hiring figures. Why do you not try to improve your interview process to lower the false-negative rate? Doing so in an intelligent way need not raise the false-positives rate (unless you somehow believe that the interview process is already perfected by Google...)
They definitely try their best to give every candidate as fair a shot as possible, but there's always going to be error. It becomes a question of what type of error hurts more, accidentally rejecting someone who's qualified or accepting someone who isn't. They reasonably believe the latter is significantly more dangerous.
I am not allowed to share our data with you about how well the process is performing.
But I would like to point out that of the two of us, only the one who doesn't know is suggesting that we have "terrible hiring figures" or a "terrible interview process".
Disclaimer (or, using less of a humblebrag, "my claim"): I'm a former Google engineering manager.
I was hired when getting into Google was arguably more difficult than it is now (2007). During my years at Google, I conducted hundreds of interviews and managed dozens of engineers at Google. I left of my own accord, in case there's any temptation to question that.
In my personal experience, the high false-negative rate in exchange for hiring only the best people is a myth that perdures from the early times. It's also a great morale booster for those who need to feel part of a select elite.
I had the pleasure to work with many exceptional people at Google and I learned more from them that I could have ever dreamt. As the organization grew to tens of thousands of engineers, I've also worked with many who are definitely not the cream of the crop. Same for management (proof: they hired me). And sadly saw very talented people rejected for stupid reasons because they just couldn't, or didn't want to fit into a mold.
Of course, you'll point out, this is just anecdotal data, and you have internal, non-shareable data that proves you right and we just have to believe you because you're currently an SRE at Google.
My point is that bringing a tired, cold argument about false negatives sounds elitist and inconsiderate. It's obvious that discrimination has taken place that has affected a fellow engineer and human being. I would be amazed to be helped at an Apple Store by such a talented individual. I would love to chat with him about his past work and share war stories of he had some time.
Your comment and posterior response smack me of elitism and lack of sympathy and tact.
Since you're posting from a throwaway account (149 days old with no prior comments): please email me your old @google.com username. Mine is the obvious one. I'll happily add a confirmation to this thread when I get it, it only takes a moment to check.
Yes, there are many things I just can't share, and the only data point you can get in that area is my own opinion. How much value you place on that is up to you; I'm giving you the only thing I can. If that is of no value to you, you're free to discount it. I freely acknowledge that I can't prove you wrong. The only alternative I have is to say nothing at all, which is what I usually do. If you would prefer to have no input from people like me at all, by all means say so.
> It's obvious that discrimination has taken place that has affected a fellow engineer and human being.
I would like to make it clear that I am responding only to the comment I responded to, which raised a very specific question that I could answer. I do not feel that I have any basis to comment on the original article; please do not associate what I am saying with that.
> Since you're posting from a throwaway account (149 days old with no prior comments)
I just didn't find it necessary to comment on anything until I saw your comment yesterday. And, honestly, I don't see any reason why I should disclose my identity to you.
I don't know how long you've been at Google, and I don't know what culture you're drinking from. At my Google we tried to listen carefully and respect opinions and arguments without regard to who issued the opinion. Doubting a person's background just because they are presenting conflicting arguments wasn't part of the Google I worked at.
It's ok to be proud of one's company, but it's dangerous to bask on reflected glory and put oneself above others based on that. Ego is a reason killer.
I didn't downvote, but I'll take a shot at answering. Your post had a touch of condescension, sort of: "well I work at Google and you don't, so I obviously know better than you ...".
You generally don't know the history of the people you're interacting with here on HN, so it's not always appropriate to act so smug about where you work.
That thread is amazing. My favorite part is the inventors of Tarsnap and Dropbox basically saying to each other, "oh hey, I'm working on the same thing as you!"
That's a bit of a mischaracterization. Drew said that he was working on the same thing as me; I didn't respond, because my response would have been "no, you're not; I'm building a secure backup tool" and I didn't want to antagonize people any further.
I am also a googler, and the attitude you had definitely felt weird to me. Since you asked for a proof before from another googler / xgoogler, here is mine:
echo "Your Manager Name Here" > file; shasum file
b8de53741d9e64711d4f47dd3a409230fc242fac
I made 30+ interviews before i got bored, for the reasons explained by ludable@. That's a personal choice, but i didn't buy much into asking useless and weird data structure questions. The attitude interviewers had wrt the "false negative" was also really artificial, and also felt elitist. You could really feel that when interviewers were thinking of/discussing new questions, and you could feel how "proud" they were when they found something not even remotely relevant to their day to day job.
I have been at google more than you have, that might be a factor in my thoughts, too.
We expect and accept a high false-negative rate. Our commenting system is optimised for zero false-positives at the cost of many false-negatives. This is a deliberate choice. So yes, I would expect to see a significant dropout of commenters who are clearly qualified.
The sort of people that we want to comment are likely to come back for another try anyway, and the long-term effect of this process seems to be doing what it was supposed to.
No, but you need to be aware when you are defending your employer that you are obviously a biased participant in the conversation, and people will generally look at you as such. (Not disclosing where you work isn't a good idea either, it's blatantly obvious and easy to identify biased discussion members.)
And you therefore need to focus on things you can clearly demonstrate and/or prove. Which is good advice for all discussion, really, but especially when people are going to be evaluating your commentary with more skepticism.
If you know someone is wrong due to confidential data, there's a couple ways of tackling that. Saying you know they're wrong because of info they can't see is generally not a productive way to correct someone, because nobody can really be sure you're being honest, or see what you mean. You will change no opinions with that approach. However, you may be able to use public data to at least demonstrate that someone is probably wrong.
Quality of candidates and success of hiring practices tends to be pretty subjective, mind you, and it's going to be hard to provide hard evidence of it. Long term success of a company would probably be the best metric. (Of course, which Google has in spades.) Though the counter to that would be many of their more recent stumbles that show that trend may be coming to an end.
As an additional question, apart from the rest of my comment: You've said you're an SRE. Do you have a hand in hiring at Google, perhaps for a team under your management?
> over 77% – or just over $52 billion – came from Google’s own websites.
That includes gmail, youtube, etc, not just search.
A further breakdown would be interesting though, AFAIK the still pay mozilla a bucket load to use google as the default search, so I'd say search alone was a significant portion of this.
Not "some portion". As I told you, that first part is all ad revenue.
From the article:
- 75 billion revenue total
- 52 billion "AdWords and Search Advertising"
- 15 billion AdSense
So yes - the vast majority came from the ad business:
15+52=67 billion (90%) from the ad business, that leaves just 8 billion from all other businesses.
"came from Google’s own websites" - that was traffic, it still was ad revenue! With that ad revenue they created free content websites like Youtube. But what pays for Youtube are Google's ads. It is ad revenue. Creating a huge free video site over the years when you have billions to subsidize it is not a sign of superior business intelligence in my opinion.
Downvoting is a form of feedback. If you are participating, then hopefully you are open to the idea that people could give you negative feedback. You can still participate, but if you keep participating in the same way, then I'm guessing you'll continue to get downvoted.
Oh, I can play this game too: Why don't we make software without bugs? if you do software in an intelligent way there's no need to create bugs.
Of course everyone strives for better processes —at the end of the day, that means more revenue— is just that those problems seem easier to solve from the outside.
As an extreme example: NASA has done a pretty good job of making software without bugs [1].
Consider these stats : the last three versions of the program — each
420,000 lines long — had just one error each. The last 11 versions
of this software had a total of 17 errors. Commercial programs of
equivalent complexity would have 5,000 errors.
I wouldn't be surprised if NASA had a different definition of "error" than a typical "bug" in commercial software. My video player plays every single mpeg/avi/mkv I through at it. Yet it has dozens if not hundreds of serious bugs. I just doubt those bugs fall under the NASA's definition of "error".
I don't know if it's possible for anyone to answer this question but: does any organisation have a hiring process that's 5000:1 better than the average equivalent, like NASA's 5000:1 software defect rate?
This explanation summarizes why tech companies lack diversity. Sampling bias at its best. This logic is a smart persons way of saying you only want to hire people similar to those you've already hired.
All I am free to say about this is that we look hard at diversity issues in hiring and put a lot of effort into eliminating them, and that I personally believe we do a better job of stamping this out than any other company I have worked for in my career. I suspect, but cannot prove, that our process gives better diversity results than most of the ideas which are "popular" on HN at present.
I do some work on this personally. I'm just not allowed to discuss the details at present.
(The reasons why I'm not allowed to discuss this are due to tedious bureaucracy, not anything interesting. I have tried to get that changed, but it would require more effort than I am willing to expend on a problem that will go away in time.)
I perform a lot of data analysis and separately I've been involved in hiring decisions. I've seen two common issues: 1) humans make decisions based on statistically insignificant sample sizes 2) we are incapable of identifying our biases without data.
Even when large organization do look at the data, they are reluctant to share it, even internally. I applaud your efforts and hope are tracking the results.
It shouldn't be very hard to do well in diversity, however you choose to define it.
For instance, say you notice your workforce is strongly biased against one population group, like female software engineers [1].
In that case:
a) Set a target for the proportion of female software engineers you want to hire.
b) Stop hiring male engineers at the point where hiring more would make you miss your female-engineer target.
c) Keep hiring only female engineers until you hit your target.
In principle, that might cause some concern among male engineers who could feel discriminated against.
In practice however, google and all other tech companies are already employing that process, except they do so informally (one hopes) and the groups they hire for are not the ones usually included in "diversity"- for instance, according to [1] 72% of tech workers at google are male, vs 50% ish in the general population.
Additionally, when it comes to google specifically, recruiters are supposed to actively go after "the best", so it shouldn't be a problem for that company in particular to go after the best female people.
In fact, that google targets its hires and yet it ends up with a strong bias towards a specific kind of engineer is a very good example of how not to do diversity.
>>> when it comes to google specifically, recruiters are supposed to actively go after "the best", so it shouldn't be a problem for that company in particular to go after the best female people.
The problem with your idea is that it's illegal and a law suit waiting to happen.
You simply can't hire someone because of their sex.
>> The problem with your idea is that it's illegal and a law suit waiting to happen.
Also btw- "my idea" is not mine at all. It's the method of gender (and also race etc) quotas, already utilised in some countries.
Those are also countries that have very strict regulations about gender (etc) discrimination so it's perfectly possible to have legislation that hinders discrimination and supports equal treatment at the same time.
Not deliberately. They simply hire the best person for the job at a particular job opening. For whatever reason, the majority is of a particular gender.
Yes, deliberately. The fact there's inequality in the distribution, when there is no good reason for it is evidence of some sort of bias in the selection process.
>> For whatever reason, the majority is of a particular gender.
For the reason that there is inequality between the two genders in society. You don't need to grasp for metaphysical explanations when you already know the two genders are treated differently in employment (as in many other matters).
>>> Yes, deliberately. The fact there's inequality in the distribution, when there is no good reason for it is evidence of some sort of bias in the selection process.
If it's unconscious bias, it's not deliberate. And it's much more than just selection bias, it's also a cultural/social issue. It happens way before someone enters the workforce.
>>> For the reason that there is inequality between the two genders in society.
The bias might be unconscious, but the tendency to avoid hiring people from a certain group is entirely conscious. The person subject to the bias will find a way to rationalise and therefore justify their tendency to hire fewer of that group- "in my experience, most A candidates are not fit for the role" or "maybe group A is just not very good at this role because of human evolution" etc.
>> It happens way before someone enters the workforce.
It's 2016. That excuse -the whole "it's society's fault"- died sometime last century, when society's attention was drawn to the issues of gender inequality for the first time. We all know there are issues, we all know to keep an eye out for them- those of use who care about that sort of thing anyway.
As to the tech companies in particular, they don't have much of an excuse because the lack of diversity in their workplaces has often been pointed out.
>>> The bias might be unconscious, but the tendency to avoid hiring people from a certain group is entirely conscious
That statement doesn't make sense. Avoiding to hire people from a certain group is bias by definition. How can it be unconscious and conscious at the same time?
>>> It's 2016. That excuse -the whole "it's society's fault"- died sometime last century
What are you talking about, there's absolutely a cultural and societal element here.
>>> those of use who care about that sort of thing anyway
Oh get off your high horse. Some of us just don't recommend breaking the law to fix this.
Your solution to tech diversity is overly simplistic and unhelpful. Not to mention sexist and ILLEGAL.
At best, it would solve equality of outcome, but not equality of opportunity. Women might still be discriminated against during interviews, but "that's OK because we filled our quota for this year" ?! How is that fixing the problem?
Just curious, would you ask other industries to do the same and implement these quota requirements? Would you impose that sewage workers reach 50% gender parity?
Do you mean to say that diversity of ideas is promoted by hiring the same kind of dude in the same kind of role all the time?
Besides, "diversity of ideas" is not the point. There is no such thing in the industry anyway (see "Uber for X" or "Facebook for Y" etc). The point of what I suggest above, essentialy imposing a quota to "help" companies hire more of the kind of people they currently don't, is to give more opportunities to those groups, not to help companies increase their bottom line.
Because increasing companies' bottom line is a goal of companies, not a goal of society. Society must ensure that all its members are treated fairly and have equal opportunities (an opinion, of course and you may disagree). Whether company Z makes a few more bucks or not is of no consequence, as long as the whole community prospers together.
And there's no way for the whole community to prosper together when half of it (at best) is traditionally excluded from the best paid jobs.
Edit: also, from my POV as a soft eng, the idea of using only half of your resources (in this case, people) because you are unwilling to challenge some centuries-old tradition that says the other half is not as good, is bloody stupid and very, very inefficient. It's up there with forcing left-handed kids to learn to write with their right hand because the left one is "bad". Or something.
Why are you optimizing for zero false-positives? That's what I'd expect from an early startup, where every employee is critical, not from a large company.
My assumption would be that Google has a continuous stream of high-quality applicants to choose from, so they don't really lose much with false negatives, because they will simply end up hiring a different qualified candidate for the position. Every false positive on the other hand brings down the overall quality of their workforce.
Part of the company culture is that we value always being able to expect that everybody you encounter is a strong engineer who will do sensible things when presented with data.
You never go into an encounter with a new person or team being unsure of whether they're going to be difficult. You never have to avoid dealing with "that guy". You get to trust everybody that you meet.
It doesn't just improve overall quality, it makes it a better place to work. Good engineers are happier in this environment, and there is pretty near universal agreement from people who have experienced the results that this is a thing worth preserving.
There are plenty of things about the hiring process which get enthusiastic internal debate, criticism, and data-driven analysis. This is not one of them. This is a thing which we really like.
(Full disclosure: I have gone through the interview process twice, failed the first time, passed the second.)
You can get all the same benefits by firing people who don't work out. Optimizing for zero false positives mean you miss a lot of people who would have been great but were rejected in the name of zero false-positives. Startups can't handle having someone bad be hired early on, because even if they're fired later they've still done a lot of damage. Big companies don't have to worry about that.
The point is that google is trying to hire "only the best". Let's say that "the best" are 1% of applicants (to make it simple).
Now, imagine that google's interview process, optimised to reduce the false positive rate [1] to 0% as it purportedly is, rejects 10% of applicants that should be hired (i.e. it has a 10% "false negative rate").
How would you guarantee that this rejected 10% does not include the 1% that are "the best"? You can't find out because you've already ditched them, so you can't exactly compare them to the ones you hired. You can find out which of the ones you hired are "the best" but only compared to your other hires. There's no guarantee that you don't end up hiring mediocre people, just by consistently failing to hire the actual best every time.
How likely is it that you'll ditch the 1% by chance? If you consistently reject 10% of candidates you actually should hire, then it's one out of ten, I'd say.
So it depends on how high is google's "false negative rate". If it's as high as 50% they may well end up rejecting half of the people they're trying to hire. The google SRE user above mentions "many false-negatives". That sounds like worse than 50%.
So, to answer your question with another question: what happens if you consistently miss most of the group you are trying to hire, week after week?
_______________________
[1] Normally people look at true positive rate and true negative rate, the former being the proportion of all positive results that are correct, and accordingly for the latter. "False positive" is just the complement of "true positive".
Also, note that a process may have a high TPR and high TNR at the same time, so a high TNR on its own is no guarantee of a good-quality process.
I say that false positives in recruitment are quite bad for any company, not just Google. But I don't agree on how Google (and others) go about preventing false positives.
If Google is so worried about false positives, why don't they just ask much harder questions in interviews ?
If the hard questions are solved, Google should proceed for recruitment and reject otherwise. No candidate feels puzzled by a rejection, since they know they failed too.
I'd say the questions used by Google can be solved (or expect to be solved) by most of the CS grads coming out of top 50 world's universities and also various coding competitions/ top coder etc. The fact is that the output of those universities far exceeds what Google wants to recruit in a year. Yet Google complains on not finding enough talent !
Please learn to convert the false negatives to clear cut negatives while also preventing false positives.
Why does Google set up multi-layered committees post interview to gloss over the interview discussions (actual interviewers not included) and then take decisions based on some fuzzy undisclosed logic ?
Thanks for the response. From the replies, it is evident that this subject touches some raw nerves!
I wanted to run that small experiment as a way of showing that the shortage of engineers is overstated (a common refrain in the H1-B visa debates). I am not sure if Google is one of the companies that says they can't find qualified engineers. Many peer companies to Google do say that. Having a very high false negative rate is something that should be taken into account with the current visa debates.
>> Our interview process is optimised for zero false-positives at the cost of many false-negatives.
Why are false positives worse than false negatives? Optimising for either exclusively sounds like a very good way to fill up with mediocre people: either you miss too many of the best (what you do), or you hit too many of the worst (what you try to avoid doing).
Of course, there's no hard-and-fast rule about the quality of a hire. You can hope to have zero false-positives, but you can't really count on it.
What you should aim to optimise is the ratio of the people you really wanted to hire over the people you actually hired. This would allow you to improve the quality of your hires over time so that it approaches the high point of some measure of goodness.
Realistically speaking, that's the best result you can expect to achieve. Any tactic that purports to give a better outcome (zero false negatives? Really?) should be regarded with suspicion.
That just makes Google sound risk averse. Given some of their business models and decisions over the last 5 years, I don't think that's the case. Given the quality of those products over the last 5 years, I might be inclined to question the nature and/or quality of the engineers being hired. Or maybe it's further up the ladder?
We expect and accept a high false-negative rate. Our interview process is optimised for zero false-positives at the cost of many false-negatives. This is a deliberate choice. So yes, I would expect to see a significant rate of rejections of people who are clearly qualified.
The sort of people that we want to hire are likely to come back for another try anyway, and the long-term effect of this process seems to be doing what it was supposed to.