I greatly apologize for the mess. Thanks for posting this! We're looking into this right now.
- All FB posts via Buffer are temporarily deleted. They will reappear again once we've enabled the FB app again (which we'll do once we're sure we're not compromised anymore).
- All Tweets are also stopped from sending.
We'll keep you updated from the @Buffer twitter and FB account.
I know that PBKDF1 is deprecated, but using SHA1 is an upgrade to PBKDF1 which was once the standard. I'm not sure if "extremely unsafe" is the right level of worry. You can still do iterations to make the hashing slower, if for some reason you don't do the right thing and use the methods you listed.
Is there a good way to 2 way encrypt something in something like rails where the app needs to use the decrypted token. If you have app logic doing decryption then someone can just look at the source to figure out how to do this. Is the only good solution to use a compiled language?
Compiled programs are as easy to read the source for (at least the relevant bits) as interpreted ones. Still, if you encrypt it and someone gains access only to the database, they wouldn't be able to use the tokens.
Hashes like SHA1 and MD5 are fast. This makes them great for things like verifying file contents... You can hash a lot of data and get your answer very quickly.
This is exactly why they are not particularly well suited to passwords. A brute-force attack (since you're salting your passwords I'm sure!) against them isn't a huge undertaking. Just with the equipment in the computer I'm on right now, I'm looking at generating about 11.5b MD5 hashes per second, or 3.1b SHA1 hashes per second.
At eight characters a-zA-Z0-9, I'm looking at about 5 and a half minutes to brute force every combination with MD5. Under a day for SHA1.
Hashes like bcrypt and scrypt, on the other hand, are designed to be slow. Their complexity factors actually provide means to slow the hashes down even further as hardware becomes faster. Instead of 11.5b/second, you can increase the complexity until you're only able to generate one hash per second... Now it takes you over three and a half centuries to hash what might take one second with MD5.
Even ignoring the possibilities of MD5/SHA1 being 'broken', they're simply too fast to be considered for hashing passwords.
But it appears that almost all user passwords (99.8%) appear in the top 10,000 list [1]. So even a brute-force attack on a slow hash like bcrypt is pretty cheap in the vast majority of cases. So switching from md5 to bcrypt doesn't improve your security much.
According to that one guy with that one list. I acknowledge that the top N passwords are X% of all user passwords. I sincerely question the 99.8% figure. The problem with doing studies like this is while we have some fairly big password dumps we still do not have the universe. Furthermore, there are some number of un-cracked passwords in the dumps we have. Further complicating the situation are password policies which may reject common passwords. It has been many years since we learned that "password" is the most common password and is commonly disallowed.
Therefore 1) it is not useless to not increase your storage security even if Y% of your users use bad passwords as you are protecting 100%-Y% of you users. 2) Y% is probably not 99.8% for you, and if you are worried about it you can take steps to mitigate the problem.
ps. He is ignoring punctuation which is an important detail for actually doing the cracking.
pps. I appreciate the sentiment (users choose shitty passwords) but not the conclusion (so don't bother storing them well). The proper conclusion is use scrypt/bcrypt and increase the work factor. You can take reasonable steps to protect your users and you should.
Yes the figure of 99.8% does seem a little high. After a bit more research it seems Mr Burnett himself can see 'a few flaws' with that figure [1].
Just to clarify: my original point wasn't that you should continue using md5. Rather, it was that bcrypt doesn't improve your security much. Given the problems with the 99.8% figure, it would be better to say, "the extra security that bcrypt provides might be less than you expect".
I wanted to get in touch to apologize for the awful experience we've caused many of you on your weekend. Buffer was hacked around 1 hour ago, and many of you may have experienced spam posts sent from you via Buffer. I can only understand how angry and disappointed you must be right now.
Not everyone who has signed up for Buffer has been affected, but you may want to check on your accounts. We're working hard to fix this problem right now and we're expecting to have everything back to normal shortly.
We're posting continual updates on the Buffer Facebook page and the Buffer Twitter page to keep you in the loop on everything.
The best steps for you to take right now and important information for you:
Remove any postings from your Facebook page or Twitter page that look like spam
Keep an eye on Buffer's Twitter page and Facebook page
Your Buffer passwords are not affected
No billing or payment information was affected or exposed
All Facebook posts sent via Buffer have been temporarily hidden and will reappear once we've resolved this situation
I am incredibly sorry this has happened and affected you and your company. We're working around the clock right now to get this resolved and we'll continue to post updates on Facebook and Twitter.
If you have any questions at all, please respond to this email. Understandably, a lot of people have emailed us, so we might take a short while to get back to everyone, but we will respond to every single email.
Same here, I found out from text messages from friends this morning saying my Facebook had been hacked, but no e-mail from them and the "Delete Account" functionality on their website does nothing.
Wow that's terrible, I guess it means they stored their access tokens with no encryption.
Personally feeling slightly smug now, I actually applied for a job there not too long ago and got a half hearted rejection without any feedback (or the courage to actually reject me). I have direct experience in doing security for a Facebook SPMD - we never got successfully hacked, and yet we ran scenarios like this in our mind all the time.
If the web server is making API calls, encryption could still protect you from some SQL injection attacks, even if the decryption key for all tokens was in the web app code.
With a system of this scale, the web server will probably be making no API calls, as everything might be sent via some distributed system.
There are, anyway, many solutions to store the keys, a key vault that is separate from both database and application/web servers would be ideal.
There's not much to do if an attacker gains control of every machine, so the general idea is to keep every attack vector as isolated as possible.
Remember encryption is not the same as hashing :) With encryption you can reverse the process, as long as you have the encryption key(s). With hashing, you can't.
I realize, but whatever server is making the API calls needs unrestricted access to the key and the encrypted tokens. So if server is compromised, the access tokens are necessarily compromised too, aren't they?
Yeah, same thing. I remember a large retailer I did some pentesting for a long time ago, they kept the encryption key for credit card numbers in the same table as the card numbers. Doh.
Yup, I'll keep using it. By advising users to revoke access they're clearly putting user safety ahead of their corporate metrics. That shows that they have their priorities straight and are worth working with.
True. In the end, they'll lose a LOT of free users who won't bother to reactivate Buffer. A good % of paid users might stay. Hell, that might even be a net benefit - less resources to devote to supporting the free users.
I will certainly use Buffer App again. For the same reason I used Buffer App before this incident. That might be easier because my accounts were not affected. In addition, the Buffer App team seems to be working hard to resolve the incident. Security issues are unpleasant but unavoidable.
Hi Joel, any thing you would recommend other devs who're connecting with facebook or any other social media API to look into? maybe you can share what you guys have learned reg: security and how to do it better from this
The best thing we've learned here is to enable a setting Facebook has called "Require AppSecret Proof for Server API calls". They actually have a lot of great security features which we've not been making use of.
I think considering the gravity of the situation, you and your team has handled this nightmare pretty well so far. Any of us who write software could end up in a similar situation - be it caused by a small configuration mistake, a slightly out of date 3rd party library, or a skilled, determined attacker.
When I see a local credit union site written in VBScript get hacked, I don't worry too much because it was probably riddled with SQL-injection bugs that any script kiddie can exploit. However, I know that you guys take data security seriously (you and I Skyped a while ago) and yet this happened. That's what worries devs like me the most.
I think it would be very helpful to us all (including many of your tech-savvy users) if you would write a detailed blog entry describing what actually happened and how. I believe you're encrypting tokens now but how did they get exposed in the first place? Was it a framework issue? Unhandled exceptions? Wrong chmod on a log file? New employee hooking up trojan'ed PC to your internal network? Here's hoping you feel comfortable in sharing.
Buffer is handling this exceptionally well; I'm impressed and it isn't easy. We've caught bugs that lead to this sort of exploitation in the past, so if anyone from Buffer is reading this: we're happy to help you out and offer a few free months of website security scanning while you figure out exactly what happened. Just email me at borski@tinfoilsecurity.com
I'm sorry for the Buffer team, it's not going to be a fun Saturday. The Buffer team is superb, both on product and customer service, so I'm sure things will be sorted out pretty quickly.
I really feel for the Buffer team. They make an awesome product and it's hard to prevent things like this, especially when social media doesn't generally have great security right now.
Though I don't want to be crass, I can't help but mention that I'm working on a project to help minimize the damage from these sorts of things. Would be interested in any feedback. (http://socialsecurity.io)
>Accounting for how much spam actually reaches the inbox,
we estimate that only about 1 in 25,000 people needs to succumb to the temptation to make a
grey-market purchase to make it pro table for spammers to inundate everyone with advertisements
at current levels.
One thing that drives me nuts about that paper format (a LaTEX template, I guess) is that it doesn't have a date. And this happens a lot. If the topic was an abstract mathematical concept, evergreen and timeless, then the date isn't so important. But a paper about spam that is vague on the date? I don't understand. That field is always changing, and I presume the economics are changing, too.
This is rather typical, because when you prepare a paper you don't yet know the publication date, so your submitted/final version can't include any date, but the published version tends to be behind a paywall.
All our recent posts on social media (facebook, twitter) via buffer seem to have been automatically deleted. That's a lot of engaging posts, gone in a poof!
I don't think you can find a service that makes a promise that they will never got hacked or be down. Common, look at what they've done to the situation.
Buffer is a service where you can link your Twitter/Facebook/etc accounts and share them at timed intervals. It "buffers" your updates for you so you can release them on a specific timeline.
Buffer is a great app, particularly for Twitter. I use it regularly across all devices. I don't use it for Facebook though. I haven't observed anything untoward on my Twitter account. I am tempted to temporarily revoke app permissions on Twitter though.
Kudos to Joel, Leo and the team for how they're handling it so far. The speed with which Joel had a comprehensive email update out to every user was impressive. I hope they resolve it soon.
Agreed. Unf. can't delete what I posted - didn't mean the plug in a malicious way. Buffer is a far superior and loved product (I/friends use it). batch.me is a small weekend experiment a few friends and I put together about a year ago. PS: Buffer team is doing an incredible job trying to put out this fire - so props.
my point was being a centralized app like buffer , it will affect all users. But using an individual desktop client will prevent being hacked centrally.
- All FB posts via Buffer are temporarily deleted. They will reappear again once we've enabled the FB app again (which we'll do once we're sure we're not compromised anymore).
- All Tweets are also stopped from sending.
We'll keep you updated from the @Buffer twitter and FB account.
- Leo