I'm going to temporarily put aside what this guy did (which is really bad, but people with bad intent aren't common), to discuss what this tells us about Google (which is about The System, and cause for larger concern).
If anybody from Google can (anonymously if necessary) step in and answer questions, it'd be great.
* Different gmail accounts. Google knows they're all you.
In the original Gawker story, this caught my eye:
"...pulled up the person's email account...[and] a list of other Gmail addresses that the friend had registered but didn't think were linked to their main account—within seconds"
Keeping separate Gmail accounts is how many protect against "Google knows everything about me." In fact on Google's "What Google knows about you" page, it never crosses accounts (unless you've manually connected them).
This story basically tells us the "What we know about your account" page is a bit misleading.
Of course most folks in IT know it's a bit naive to think one could never figure out that different gmail accounts are related. But it was interesting that Google pretty formally knows the relationship, but doesn't tell you right where it should.
* SREs, and their level of access.
It's not so much that I care is a specific group has lots of access. I care that not that many groups do in total. This story makes me concerned that actually many groups have lots of access. Despite the "elite navy seal" vibe presented in the Gawker story about SREs, I'm now thinking that many, many teams have this kind of access. (Previous to this story, I was led to believe that SREs were quite low level (not in importance. but in nature of responsibilities. Very performance oriented, having little reason to have access to an individual user's data.).
Please feel free to jump in and correct this, Google peeps. It would make me feel better.
* What this does for SaaS and web apps in general
I love Google Docs and sincerely believe that most web apps that allow across-the-net collaboration are good for us. And are preferable to The Old Way.
I want people to TRUST their stuff to Google (and Github and Amazon, etc).
I hate security FUDers who love to derail conversations of great possibility with some far out scenario, "Can my enemy see my Google Docs?!?!"
I'm way less worried about a few creeps who work at Google (they work everywhere...) and more concerned about laissez-faire access processes.
"...pulled up the person's email account...[and] a list of other Gmail addresses that the friend had registered but didn't think were linked to their main account—within seconds"
This surprised me too. In the absence of any further comment from Google, I'd be very interested to see some journalists doing some investigation here.
Assuming that this is real and not mis-reporting or user error, I'm guessing Google links using either their google.com cookie, IP address and/or browser identification. Any of those methods have potential for errors (in particular they mean you should never share a browser with another person in case your account ends up linked. That seems.... extreme..)
Not a journalistic investigation, but a while back I decided that I need a new gmail account. I signed it up using some different credentials, including a different name. However, I did not mention my old gmail account in the entire sign-up process. After I activated my new account, I got an email on my old account referencing the new address I just signed up for and a verification code "in case something happens."
From this it seems to me like this is not a deliberate maneuver to deceive, but rather just an oversight.
I have a second gmail account (in the olden days that's what they recommended on http://code.google.com/apis/gadgets/docs/publish.html - I see now that they have switched to using filters), and this didn't happen to me.
Are you sure you didn't use the first gmail account to send an invitation? Because in that case it does add the address to both accounts address books.
Giving people responsible for performance tuning access to user data is almost inevitable if you run anything remotely like unix. After all, it may require tweaking kernel parameters, changing init sctripts or reinstalling applications compiled with different optimizer settings. All of these operations require root, which is an all or nothing affair in unix. (You can do some tricks with /etc/sudoers to restrict what else people who need root for a specific task can do as root, but writing a sudoers file that allows serious performance tuning and is still airtight might prove a challenge.)
This has in fact been criticized as a major design flaw of the unix authorization model. To prevent problems like these you need a more fine grained authorization model that separates access to the system and its configuration from access to the data, so that even if I can replace an application with something else, I still cannot make it write data to a location of my choice. This would reduce the number of people who I need to absolutely trust from everybody who may need to change some minor configuration file to the handful of people managing the security kernel of the OS.
Systems like that are possible, but not easy to implement and mange correctly and sometimes inconvenient to use, so peole go the easy way and assume that everyone that is a member of a group mentioned in /etc/sudoers is trustworthy or at least lack the criminal energy to bypass the restrictions sudo can enforce. Most of the time, this works.
As a former systems person, I thought of all the ways I could divine that an account was related. IP address of course came to mind, but I dismissed it, because so many networks use NAT, etc.
(Especially at high schools, there must be so many accounts with the same IP. But I get that the student goes home at night...)
So you're basically saying this SRE made an educated guess. It does make me feel better about that part :D
Much of my professional life is evangelizing Google Apps. I know and advocate the party line about security. And I have experienced the customer support staff who have access when helping me troubleshoot. (I'm glad they have access.)
I guess I just assumed/heard that SRE's weren't involved in troubleshooting user stuff, as they were in deep disk i/o, networking, performance stuff. (Translation: was hoping they couldn't access user stuff.)
As for internal auditing, it would be cool if there was an alert any time user data was accessed that didn't have a corresponding trouble ticket.
Grandparent deleted their post, but, assuming they just said "they correlated across IP" -- it's actually easy to guess, since the snoop would be able to detect and ignore a NAT IP -- it would be accessing thousands of accounts.
AFAIK Google's SRE's are more fancy sysadmins than performance engineers. Sysadmins often have the keys to the kingdom, though one would hope a sysadmin on project X only has the keys to project X.
Working at big banks and credit card processors I can tell this sort of insider abuse happens everywhere. And firing a bunch of people instantly for breaching policy actually does work as a good dis-incentive. Whether this raises the question due to the amount of data the Google knows about everyone of us could the impact of any potential insider abuse be magnified is a something for debate.
From all reports, it seems that this Google employee accessed data which he knew he had no authorization to access. That sounds like a textbook case of computer crime -- why hasn't he been arrested yet?
He did have authorization to access it. He didn't have authorization to abuse it in the way that he did. It's all kinds of icky but what do you think was so obviously criminal about it that you expect him to be in jail?
Unless Google's policies were written by bungling idiot, Barksdale would have authorization to access information required for him to fulfill his duties. The information he accessed very obviously goes well beyond such authorization.
Well. We can get into a silly discussion about what 'authorization' means, I just don't understand why you find this so obviously criminal. Remember these idiots -
Harass maybe. Coerce, I don't really see that. There's a fairly wide gap between being a massive dick and outright criminality. I have no trouble imagining that an ambitious prosecutor might find something to hang on him but I was wondering what 'textbook case', shoved-into-back-of-a-police-cruiser crime cperciva was seeing here.
- reading private communications of people he knew in real life
- retrieving IM conversations and publicizing them without consent or knowledge of the user
- re-inserting himself in contact lists after being removed to stop the harassment
- seeking real life contact with people whose private information he had retrieved
Let's hope there wasn't more.
Each and every one of those would be a firing offense, the fact that this is beyond just snooping and led to irl contact between this guy and the people (kids?) he was stalking means the situation is more than out of control.
Mistakes happen, but such a series is not a mistake any more.
A position of such trust requires a more than ethical behavior and good oversight. Google failed in the second.
A good sysadmin has 'sysadmin blindness', even when you're looking at user data to do your job, you are not going to read the emails that sit in those inboxes unless you are specifically directed to do so by the owner of the data.
For the rest those files might as well contain random bits.
Again, behaviour that's outrageous and reprehensible is not necessarily criminal. I am well aware he did some pretty nasty stuff. I'm not talking about any of the things you're bringing up - I just found it interesting someone saw the this as an obvious crime worthy of immediate arrest. So I asked why. That's all.
Doesn't some of that stuff fall along the lines of stalking? If the term "authorized" can be defined properly in this context, isn't there a branch of cybercrime that deals with "unauthorized" access to or use of information? Is it possible for the victims to use DMCA here? Finally, in that his victims were minors, are there extra protections afforded them, especially with the violation of the intent for users of the service to block? How about trespass? Some of what this guy did is akin to forceful entry into your house in order to press his agenda, whatever it is.
I am not a lawyer, but I can imagine at least asking for someone to look within these categories. I would personally feel very violated if I had some stranger even commenting about an email that was not directed towards them ... and even more violated if that stranger manipulated the system to get beyond whatever walls I threw up.
I fully agree with you. I further wouldn't mind laws covering it. But I'm not sure there are any laws in the USA that do cover it. If not then the legal issues that are available are that he can be fired. Which he was.
(Edit: Note that I work at Google as an SRE. This limits how much I can say in this discussion.)
This is the huge problem we are still facing with computer crime; because the law is still quite vague and unresolved. Juries have a tendency to not really "get" what has happened and so figure it's not all that bad etc.
It seems rotten but the best chance you would have with this is in getting the Jury excited about the stalking aspect and the contact with children.
Technically he has broken the law (at least, I think he has). Proving it though, along with mens rea (intent) is an absolute minefield and it would be torn apart by a decent lawyer. This is why most computer criminals are currently prosecuted for other crimes (in this case, probably the stalking offences).
That might be problematic but there is no indication at all that google did anything to even contact the justice department, let alone that they filed charges.
Assuming California law, I'd say there a fair number of potential violations available for prosecution with a focus on Penal code section 502 http://www.privacyprotection.ca.gov/privacy_laws.htm
Though I am no lawyer, I'm sure a civil case could also be made. Will be interesting to see what actions the affected parties take.
Google likely wanted to keep this quiet, but if they'd turned this over to the FBI, I'm sure there would have been no trouble finding laws he'd broken.
Again, it's the distinction between accessing something you're not authorized to access and accessing something you are authorized to access but shouldn't.
If I break into google and poop on the floor of their server room, I can be arrested, but if one of their sysadmins gets in with their key and then poops on the floor of the server room, they can only be fired.
First let me complain about the rule that says I cannot reply to a direct descendant post. But I'm apparently allowed to reply to you here. Anyway, IANAL, but a google search turned up this, which looks relevant:
Regardless of how Google is playing this in the press, the question people need to be asking isn't about the rogue employee. It is: "what are the controls being put in place to prevent SRE's from accessing sensitive data inside Google apps, and what specific forms of information is Google considering sensitive for those purposes, and is there a class of employee at Google that is expected to be exempt from these controls?"
We dismissed David Barksdale for breaking Google’s strict internal privacy policies. We carefully control the number of employees who have access to our systems, and we regularly upgrade our security controls–for example, we are significantly increasing the amount of time we spend auditing our logs to ensure those controls are effective. That said, a limited number of people will always need to access these systems if we are to operate them properly–which is why we take any breach so seriously.
I would assume that the logs he is talking about are logs of accesses made by Google employees to data covered by the privacy policy.
(Disclaimer, I am an SRE at Google. I do not speak for Google.)
You left out most of the credentials that were listed in that ad. And I don't think that anyone believes that every SRE who has the paper credentials gets hired.
Also most SREs don't get access to the same things that this guy did. (What you get access to depends on what you're working on.)
The rest of the credentials were skills-based, except for that last one, which suggested that some management skills would be a nice-to-have. Do you take my point about how "SRE" isn't a very strong answer here?
I think you don't know what you are talking about here.
It is like someone seeing an ad for entrepreneurs that says, "Willing to work. Willing to take risks. Strong computer skills a significant plus"." And then concluding that the bar to being a successful entrepreneur is very low so they should be dismissed as a group.
Becoming an SRE is much, much harder than just having the credentials you listed. Being an SRE generally does not give you full access to everything at Google. I never met this one, so I don't know what his role was or why he was given that level of access. But that access really isn't something that just gets handed out to people off the street.
The fact that you found that ad, and that Google screwed up this particular case, doesn't say that Google doesn't limit who gets access to sensitive data.
I think you're extrapolating too much out of my comments. I'm saying that "SRE is an important job" doesn't answer the concern. I'm not surprised that Google has controls beyond "you're an SRE, you can do whatever you want" --- in fact, I'd be shocked if they didn't. But it sure sounds that way from the story that just broke yesterday.
SRE at Google is a position of high trust and engineering seniority. There will always be a need for a certain (very few) people to have access to sensitive user data -- that's why this breach is so embarrassing for Google. The most we can hope for, at this class of employee, is better auditing of their access logs. The engineering exec quoted in the article promises they will perform more detailed audits.
The most we can hope for, at this class of employee, is better auditing of their access logs.
And better selection of the people put into that position?
Is it indicative of the downside of Silicon Valley's youth culture that you can wind up in a position of "high trust and engineering seniority" at the age of 27? You don't see many 27-year-olds in other ultra-high-trust professions, do you?
I think you expect too much of the press. You see, asking this question makes a lot of sense, but it doesn't make a whole lot of money. The press will run with whatever gets ratings (or sells papers if you roll that way).
At first glance, google comes off pretty good on how they dealt with this, but you have to wonder how come a single engineer has access to google voice and google mail and IM data of end users. SRE's as these employees are labeled (site reliability engineers) are 'highly experienced engineers who can be trusted'.
It goes beyond just snooping too, apparently this guy changed end-user settings which had specifically made to lock him out, and spent a lot of time and effort to use his position at google to achieve real world effects with the people he was snooping on.
This guy has a serious case of sysadmin god complex and while I'm really not sure if it is ok for him to be exposed with name and picture I hope he'll never be in a position of such responsibility again, and I hope that google will perform better oversight of the people that have access like this.
The only thing that got the ball rolling here was the parents of some of the kids alerting google.
I got the impression that SRE basically have low level access to the storage stack. So wouldn't be subject to most of the normal application level logging that I would assume would red flag this behaviour pretty fast.
The only way to get around this is to have someone audit all their actions constantly, which you need someone equally or more familiar with the systems they are working with.
I think that is pretty impossible to implement that level of overview with humans, the best way to go normally is the 'buddy system' so no one can access a system unless they have a 'buddy' with them. Like the military do in nuclear weapon silos.
Access to the low level storage stack would not allow you to query with so much detail and would likely not have an interface that would allow you to modify user settings at will. So he must have used some higher level tools.
For example if an application uses Bigtable, then the key + column names often gives a lot of information about what data is stored there, which if somebody had access to some basic application data they might be able to get at somebodies specific data.
However as you might expect there are many safeguards in place, including ensuring every action is fully and securely authenticated so even low level SREs cannot read application data without a paper trail. This story is pretty surprising to me, and if true this guy is an idiot.
Depends how subversive he was trying to be I guess. I was thinking more around the query layer for bigtable etc. He probably would have known the stack top to bottom.
I am not sure this is a 'solvable' problem. You can mitigate by always working in pairs. But even that just reduces the potential for privacy breaches.
This is pretty serious and the fact it's turned into a story is one of the more damaging things you could expect to see about Google w.r.t. privacy. SREs are indeed very privileged and in many cases have carte blanche on their associated products. It's saddening one of them used his rights for nefarious purposes and broke some of the trust around Google's handling of personal data. I hope this leads to better auditing at least internally - seems like something that better transparency of access would have brought to light earlier.
While it is no guarantee of any change I like that they are not attempting to sweep this under the rug. There are a lot of companies where people have access to a lot of sensitive data. All you can do is screen the employees, limit their access where possible and audit their use of the security.
But then someone needs to audit the auditors. Just before I started here we used to have an employee who would look in the Oracle database used by Lawson to check payroll data. Nobody knew for a long time since he was the UNIX admin and DBA.
Valleywag was blacklisted early on, and Gawker has been too at least as long as Valleywag has been reintegrated into it. Plenty of other domains are blacklisted too: http://infoworld.com has been for some time, http://oppugn.us/ got added fairly recently, http://www.kungfugrippe.com/ got added after a (wholly justified) rant about Scribd (YC S06!).
I think google manned up well on this one. They will always have this problem. At least it seems they have less (that we know off) incidents than the government does. It's pretty incredible how many stories of government employees snooping (even selling it to organize crime) information stored in their databases.
That was my first take as well, but after reading up a bit on it it seems that they tried to make it go away by not charging him, when if you look at the severity of this case they had every reason to.
So they tried to sweep it under the rug by just letting the guy go.
If an employee of mine had ever snooped on end-user data and would have used that data in order to get real-world effects in the lives of those users I'm fairly sure I would have registered a complaint with law enforcement.
Google has their 'image' to be aware of, but in this case just letting the guy go may not be the best way to preserve that image.
Not that I have any information, but it could be that the parties affected did not want to pursue legal action. All we know about the story is a quote from Google and a speculative article from Gawker.
this isn't that surprising. this stuff happens all the time at any company that has that many employees. a person i know who is a software developer at facebook told me that everyone there looks at people's private stuff and reads people's private messages when they want to and you just have to be discrete about it so that no one (i.e. users) notices.
I can tell you that it does NOT typically happen at Google. If you so much as joke about this stuff people will give you negative reactions. I was genuinely shocked to read about this today as I never would have expected any Google employee to be so unethical.
If what you say about the practices at Facebook is even remotely true then it is disgusting and shameful behaviour.
"For evil to flourish, all that is needed is for good people to do nothing." -- Edmund Burke.
Google's mere dismissal of the guy comes across as pretty evil. According to the article, there was a previous instance of malfeasance. If the bad PR behind all these privacy breaches were taken more seriously, Google would probably have to clean up their act and users would benefit as a result.
Wait, are you saying that we should be reacting to the PR (we didn't, he was let go some time ago) or to the act? I would always rather we react appropriately to the act. (disclaimer, work at google, blah blah blah)
I'm impressed with how transparent Google was with handling the issue. Unfortunately, things like this can and will occasionally happen. You can either put up your own Postfix server if you do not like it - or you can thank Google for continuing to provide such a kick ass free service.
As for David Barksdale, good luck to you, you will need it.
I'm a lot more concerned about the data centers full of government employees wiretapping innocent people for no apparent reason. A fair bit came about concerning this a year or two ago, with ex-employees stating that the system was routinely abused for amusement. What's up with that these days?
Just to keep this into perspective, we are reading about this because it is Google. But every system and every relay through which your email passes is a point where somebody with less then well-meaning intentions can read your email. We may be able to somewhat rely on Google to enforce some privacy policy, given publicity pressures, but some danger lies in all the carriers between point A and point B.
It is a shame that PGP only took off in the hardcore user community. If it was made insanely accessible to users -- maybe even transparent -- maybe we could have a better assumption of privacy for our communications (as well as a potential reduction in spam?).
Still kind of alarming, I mean I do personally know some google employees and none would even remotely consider doing anything like this for both philosophical and practical reasons. Either way it's kind of scary that some douche fucker "quality assurance" dweeb had enough access to do this kind of thing.
I think we ought to have some kind of equivalent HIPPA act for ALL data personally identifiable to us, not just in medical contexts. That'd put the fire under googles ass enough to take our privacy seriously. Fuck Eric Schmidt and his "change your name at 18" bullshit. We know who that fucker is right now.
Did you ever see the full quote of the "change your name" stuff?
> Mr. Schmidt is surely right, though, that the questions go far beyond Google. "I don't believe society understands what happens when everything is available, knowable and recorded by everyone all the time," he says. He predicts, apparently seriously, that every young person one day will be entitled automatically to change his or her name on reaching adulthood in order to disown youthful hijinks stored on their friends' social media sites.
Which makes sense to me. Hell, I wish I could delete some videos and/or photos of me on various sites.
Lately Schmidt has been making statements like this though; they are reasonable when complete, but some reporters snip out five words (or, just paraphrase or interpret) and create a news storm. CEOs are supposed to be good at avoiding that sort of thing.
> equivalent HIPPA act for ALL data personally identifiable
We ought to start with getting the same level of laws for voip, IM, and email that phone and mail have. "All PII" is too vague, but those seem like a slam dunk.
Are GV calls all recorded? Is it there somewhere in the TOS? Even if its there, the consent of the other party is required to record calls. Otherwise its a offense. Right?
You hire 20k people over 10 years and make not one mistake? Iknow you are being snarky, but there is no perfect hiring process, no matter what you measure.
Probably going way off deep end here... Sometimes I wonder if someone in the process shouldn't be licensed by the state in the interest of protecting the interests of the public. Just like doctors, lawyers and engineers.
That way, if something goes horribly wrong, someone's ass is more on the line than them just losing their paycheck.
As someone who formally worked in an engineering profession that required state licensing and now working doing the same job Mr. Barksdale did (not for Google, though), I can't see any sort of licensing helping with this sort of problem.
And I think at this point Mr. Barksdale ass is pretty much screwed -- it's unlikely he'll ever get a job doing this sort of work again.
It's a tricky problem. I know to do my job I need root access to everything. I guess at Google scale you could compartmentalize so the same person doesn't have free access across services.
But at some point you just gotta trust your people.
OTOH, perhaps I just don't understand -- what this fellow did is so over the top it's difficult for me to understand why he would do such a thing. It's wrong on so many levels -- it's just not something I can comprehend.
My read on it was that he's a typical Aspergian nerd (of which there are several at Google), and it never occurred to him that what he was doing was not-okay. A lot in the story seems to support that. Why else is he hanging out with high schoolers - who don't even like him? Why does he feel the need to brag about his position at Google and the power it gives him?
Some people are born knowing all the rules to social interaction. But others have to learn them through painful trial and error. A lot of us got that out of the way in middle school, high school, and college, before we were given the responsibility to do anything truly damaging to ourselves and others. Maybe he just had the bad luck to not seriously screw up until he's at an age where everyone will blacklist him for it.
This is a really big deal guys, it means nothing at all how comfy we feel knowing google peeps personally. The fact is there is no regulation or oversight dictating how seriously google needs to take our privacy other than random - easily ignored - blips like this. We need government intervention/oversight to make this stuff go away. The gov is already there unofficially anyway, let's not kid ourselves about that. Google gave that little piece away years ago. No us company with that much personal data would be allowed to exist otherwise. I would know, did gigs at bellatlantic long before "it" happened and uncle sam was and always has been there just the same. More about what they're allowed to officially charge you with in court - don't think they couldn't know either way at some level..though probably what they were doing really was in our national best interest. Not judging, just saying..we need offiial public oversight or be left at the mercy of what the corporation decides to do with our data. The same data that provides the overwhelming majority of revenue via advertisers. It's just sick is all.
Hard call. They offer great free services and have changed the landscape of email, phone, and document communication. Yet they are one entity with all the information of DeJa News, Keyhole Maps, YouTube, DoubleClick, GrandCentral, Gizmo5, DocVerse, and their own email and app offerings. (see http://en.wikipedia.org/wiki/List_of_acquisitions_by_Google)
So yes, it is refreshing to see transparency of "Engineer fired for snooping where they shouldnt". But we keep using them as a service, so it's a hard problem to combat. After all, the price is right. Just costs your privacy.
I have a blackberry hooked up to Google Voice and Mail servers. They know my name, address, all my phone numbers, all my emails, my contact lists, frequency I receive calls on my Google number, text transcription of voicemails. They also can potentially record every call I receive and make with GV.
Considering the benefit I get from just Mail and GV, the datamining is a cost I'm willing to make. I also know if my phone is lost, I dont lose my data. And I can back it up elsewhere.
And I am somewhat happily shocked that they came such forthright that they "fired him for snooping". Most places will only say "They no longer work for the company".
>After all, the price is right. Just costs your privacy.
It is worth pointing out that payable cloud services may also cost you your privacy. It has happened before, various data leaks, stolen passwords, etc.
If anybody from Google can (anonymously if necessary) step in and answer questions, it'd be great.
* Different gmail accounts. Google knows they're all you.
In the original Gawker story, this caught my eye:
"...pulled up the person's email account...[and] a list of other Gmail addresses that the friend had registered but didn't think were linked to their main account—within seconds"
Keeping separate Gmail accounts is how many protect against "Google knows everything about me." In fact on Google's "What Google knows about you" page, it never crosses accounts (unless you've manually connected them). This story basically tells us the "What we know about your account" page is a bit misleading. Of course most folks in IT know it's a bit naive to think one could never figure out that different gmail accounts are related. But it was interesting that Google pretty formally knows the relationship, but doesn't tell you right where it should.
* SREs, and their level of access.
It's not so much that I care is a specific group has lots of access. I care that not that many groups do in total. This story makes me concerned that actually many groups have lots of access. Despite the "elite navy seal" vibe presented in the Gawker story about SREs, I'm now thinking that many, many teams have this kind of access. (Previous to this story, I was led to believe that SREs were quite low level (not in importance. but in nature of responsibilities. Very performance oriented, having little reason to have access to an individual user's data.).
Please feel free to jump in and correct this, Google peeps. It would make me feel better.
* What this does for SaaS and web apps in general
I love Google Docs and sincerely believe that most web apps that allow across-the-net collaboration are good for us. And are preferable to The Old Way. I want people to TRUST their stuff to Google (and Github and Amazon, etc).
I hate security FUDers who love to derail conversations of great possibility with some far out scenario, "Can my enemy see my Google Docs?!?!"
I'm way less worried about a few creeps who work at Google (they work everywhere...) and more concerned about laissez-faire access processes.