I don't expect every developer to be aware of every vulnerability. But I do expect that a financial institution has a specialist somewhere that audits the code before it is sent for testing ("white box"), and then I expect them to have an independent audit team probe for vulnerabilities ("black box").

After the inexperienced developer has had his code rejected for various flaws, he will become quite aware of the obvious ways things like this can go wrong.

Don't get me wrong, I expect that the vast majority of developers wouldn't make this mistake in the first place. But if you aren't a specialist, it is pure hubris to think that you write code that is hardened against all of the attacks out there. And if you have a vulnerability, it really doesn't matter if it's an embarrassingly simple vulnerability or one that requires sophisticated techniques to uncover and exploit. Either way, you're road kill.

High-value targets like banks need two security specialists (the code audit and the penetration test) to accompany the development specialists. That's simple separation of concerns, and it works as well in team organisation as it does in code organisation.

When you engage a software security firm to check out an app, you're trusting that their report is reasonably complete. As a practitioner, this is basically your nightmare scenario. You know in the back of your head that it's always possible you're going to miss things, but a vulnerability this trivial seems like the kind of thing you find on the first day of testing.

I know this isn't what you're talking about, but I hope that an institution that's protecting my money will formally specifying their protocol and not haphazardly putting their app together with "Test Driven Development". But maybe that's how they managed to be this stupid.

Your idea that the input and output paths through a typical banking J2EE app were carefully whiteboarded seems mythologized, based on what I've seen at fisrv companies.

If you are an enterprise dev with a knack for making web apps that are't horribly insecure, and you don't have a phantasmagorically awesome comp package already, more advice: start looking for new gigs. You're underemployed.

Do banks make so little from checking/current accounts that it isn't worth spending a dollar per customer to build a decent solution?

Though there clearly are people who care deeply about bank UX, those people are not normal.

However, it seems like there is relatively little to distinguish between banks, and a web service that truly pleased people might be a good way to get and retain customers, whilst cutting costs (branches and call centres.)

How do you compare them, anyway? It's not like banks give out trial accounts you could use for this purpose.

You'd have to go to the trouble of signing up for a real bank account at each of the banks whose web services you'd like to evaluate. Then spend the time to try them out and compare. So even a cursory inspection of a variety of banks' web services would already be a pretty huge hassle.

Even if you go to all this touble, you're probably not going to know much more than how intuitive a given bank's web interface is, or if they did something glaringly awful in terms of its usability.

You wouldn't know how accessible the interface is under load, or test how well the bank's interface handles the various real-life corner cases you're likely to encounter over time.

And how would you test the interface's security? Is even a reasonably security-aware consumer really going to try breaking in to the bank?

The ironic thing is that most users who use online banking probably do care about the web interface and its security, even if they can't articulate it in so many words. But of them, only a small minority care enough and aware enough of the issues to take the time to comparison shop. And even then they probably won't find out much.

There's an opportunity here for a Consumer Reports style investigation in to the web interfaces of various banks. That would be a valuable service that could save the consumer a lot of time and hassle, and maybe even do things that a lone consumer couldn't: evaluate interface security.

For instance, you write a test that says the user should get a "not authorized" if they try to access another user's account. You run the test, and it fails. You change your code to make it pass. And every time you update your app from then on, you re-run the test and make sure you didn't make that test fail.

Having a design on paper isn't as useful as having one that you can PROVE your code conforms to.

Tests make it possible to refactor your code towards simplicity and clarity, making security errors more clear. If you're writing a banking application, ignoring them is simply foolhardy.

Associating TDD to being haphazard is pretty random. Being hap hazard and TDDing are really in 2 different math spaces. Care not to mix em?

TDD and using your brain coexist quite happily. But if you're not using your brain, then TDD won't save you.

So no, that's not how they managed to be this stupid. They managed it through not using their brains.

When working with a substantial financial organisation, there were strict rules prohibiting my team from communicating with the code audit and other security teams. Going for coffee with one of the internal audit people would get you both fired. The external testing was done by a security consulting firm. Their reports were passed on to me with the identifying details stripped out so that I wouldn't even know who they were.

Talk about "low coupling!"

This creates a boot strapping problem. If you're an organization of magicians then you have a comparatively easy time finding other good magicians. But if you don't have that hsitory and you still have the need you won't be able to know the quality of people you're hiring except on the least granular of scales (lots of failed and low quality projects). This is a big reason behind so much of the problems of software development in "the enterprise". Because people who know how to run a bank or a media company haven't the slightest clue how to run software development, and there generally aren't trusted 3rd parties that people can go to for help.

Worse yet, there are perverse incentives at play. If you can't tell the difference between a good security person and a stunningly mediocre one, you are going to go with the cheapest one who you think is good enough (generally: solid job history, resume fully buzzword compliant, etc.)

Try replacing software development with writing english.

I am going to stop banging on about this eventually, but reading and writing code is something that needs to be understood at all levels of a company before that company gets good at it. If the boss of the company and everyone who reports to him is illiterate, one cannot expect the memos and the policy documents and the amusing posters to be of any decent quality

Perhaps the problem is the "security specialists".

I agree that just about any reasonably skilled developer knows to, at the absolute minimum, include a user-specific session key with every transaction.

But I've also seen "security specialists" get a pass for a variety of bad behaviors. And we've heard of the complete incompetence of HBGary. My hunch is that the secrecy/obscurity that the passed for security in pre-Internet culture is exactly what allows total screw-ups like this to happen in these large organizations. The gold-plated consultants who are friends with the VP wind-up being far less competent than your average/decent developer.

And, the notion that "good" developers turn out uniformly secure code is delusive. We're lucky to have gotten to work with some of the best run, most carefully recruited teams in the industry. You find terrifying things on all these engagements.

My advice, again as a practitioner: if your security firm isn't finding terrifying things, at least on the first go-round with an app, ask very tough questions. Ask if they need an extra week or two to catch up and get real findings. Then ask which of the two of you should pay for those weeks.

We are still finding code execution on a good chunk of our web gigs, even at the "good" companies.

Anyway, I once found myself writing some PHP code to demo a slightly complex SQL injection attack for the class I co-lecture at Northwestern (Network Security and Penetration). This code purposefully had a SQL injection vulnerability in it. It wasn't until the third reading of my own code that I noticed that I mistakenly dropped a CSRF vulnerability in alongside it. CSRF was literally the topic I was teaching next Monday and I put one into my own security code accidentally.

Secure code is so difficult to write that I can't believe that even the best developer writes secure code much of the time. Hell, apparently even I can't write secure PHP when I'm looking straight at it.

If you told me someone at, say, iSec Partners (a firm I like) wrote something I was pentesting, I'd go f'ing nuts trying to find flaws in it. If you told me iSec was reviewing something I wrote, I'd stay up nights thinking of ways to shore it up.

I tried to get ahold of newsmedia, but realized afterwards that the links I was sending did have a session timeout associated, so by the time a reporter clicked a link, they got nothing.

Finally, I managed to get in touch with someone at 'fuckameritech.net' (IIRC) - a consumer watchdog (I hesitate to say 'group' - I think it was just one guy) who said "I'll take care of it". He made some contacts - I think got it to a reporter in Chicago, and that afternoon Ameritech's online bill view and pay was taken down (a wednesday IIRC) and it wasn't brought up again until Monday.

The 'fix' was not much - they were now hashing the account number in some massively long (128 char?) ID instead of just your account number. But it was all still visible in the URL, which was the bigger problem to start with, because it encouraged 'hackers' like me to change my account number by one digit.

I suspect others had noticed this before, tried to contact citi, and couldn't get in touch with anyone who understood what the caller was saying.

Companies need separate 'web vulnerability' hotlines to call/contact to report issues like this - perhaps just hidden in the 'view source' - if you're good enough to find the info, you know what you're doing enough to report a problem. Too low a bar?

We've popularized crime reporting as a social good - mcgruff the crime dog, etc. When will we start taking online safety and security with the same level of seriousness?

As much as I don't like the idea, being able to report issues to a state or federal agency might be a way to go.

Also Facebook has a form for reporting vulns [2] and people are still happy to share their personal info there. I'm sure there are other companies that have "hotlines" but these are just a few I can think of.

I don't think having an avenue for responsible security bug disclosure gives anyone the impression that their data is unsafe.

[1] http://googleonlinesecurity.blogspot.com/2010/11/rewarding-w...

[2] https://www.facebook.com/help/contact.php?show_form=white_ha...

http://idunno.org/archive/2011/06/14/citibank-hacked-ndash-d...

"This was not sophisticated or ingenious, as reported, this was boringly simple. ... OWASP has had Insecure Direct Object references on it’s Top 10 list for years. It’s in the SDL Threat Modeling tool. Any security firm worth its salt checks for this"

Yes, there's a good description of this kind of trivial "hack" in the Open Web Application Security Project Top 10: https://www.owasp.org/index.php/Top_10_2010-A4

As for constantly-changing apps, let me speak against my own direct financial interests here (we have a product coming out that addresses that problem, so I'd like it to be a big one). Many, if not most, of the large financials we work with or have talked to have a fairly strict process for deploying new code, and the process gates on security review. Not deploying unreviewed code comes pretty close to being part of the due care standard at large modern banks. If I had to gamble on this, I'd bet that this specific code did get reviewed.

1.) This app has been sitting around in production and was never tested.

2.) This app was part of the normal testing procedures (which usually means it's tested annually) and somehow this vulnerability was missed in every test.

3.) This vulnerability was not present the last time the application was tested, and somehow this version was deployed before it was signed off on.

I've been around too long in this industry to claim that scenario 1 or 2 are impossible, but knowing the particulars, they seem exceedingly unlikely.

That leaves me to think it was the third scenario, which is still abberant behavior on their part.

I feel bad when I hear about situations like this. As you mentioned in another comment, this is pretty much what we fear the most.

For this specific vulnerability, I find it shocking that even the most rudimentary assessment wouldn't have caught it; but my own personal befuddlement might be biasing me against thinking that (2) is likely.

I don't work for a financial company but I work for one that works for/with them. Their code review is more like: "Hey, did you even test this code in QC? I can see a syntax error." I don't think anyone here actively looks for security problems during review. If it compiles and it's sat in dev/qc for a month (we just assume it's been tested), it's pushed out. I don't think anybody here would recognize XSS if it hit them in the face, and this particular bug ("allow any authenticated user to view any URI matching a given string") sounds suspiciously like a bad ACL rule in their identity/access management servers.

Edit: and the web app would have to not be rejecting access by an invalid user. I can see a single line's test being formatted in a weird way and this getting missed when somebody committed it - after all, if it doesn't cause failure, is there a bug?

"this is a dead simple and common hack and Citi should have seen it and prevented against it. Seriously, this is kindergarten level stuff. Really, really stupid."

On the other hand, so is waving a gun at a teller. That attack has been around for decades and still works a few dozen times a year, because the cost/benefit analysis says that after hardening the banks a little it is easier to just lose a few tens of thousands of dollars every once in a while than it is to give them the Secret Service's attention to physical security.

That is hardly the only systemic vulnerability in the banking system. For example, let's suppose I want to compromise your account number and credentials sufficient to take you for every penny you possess. You know what I need? A check of yours. Any will do. Everything I need to create a demand draft against your account is on every check you have ever written. Every employee of every business you have ever paid by check got the keys to your financial kingdom.

You may not be aware of it, but since those credentials are assumed compromised, the security is in a) catching me when I use the demand draft to suspiciously drain your account and b) failing that, making you whole out of the bank's pocket. The numbers have been crunched: it is vastly, vastly more efficient to treat fraud as a cost of doing business than it is to tighten the screws 100%

The attack surface on software the size and complexity of a bank's is like the Death Star, except any single rivet being out of place will eventually result in this headline.

(Step #1 in tightening the screws would be turn off public facing websites, because inexpert users plus compromised machines means that no banking website will ever be secure, even without coding errors. This will never happen, because the provable cost savings of moving customers to online banking roflstomp over the marginal fraud risk.)

Your subsequent analogy/justification/complaint is only valid if ONE doctored URL were used. My bank, Chase, does in fact implement security against such a "lots of random account numbers" attack: not only must the account match, but the MAC/IP address, browser/cookie, and other under-the-hood identifiers must line up; any mismatch between account number and access tools initiates emailing or texting a verification code to a known address/phone, which then must be submitted to close the loop of verification and, only then, allow access. Not running some kind of "one account per access device" sanity check is insane.

It's not about one rivet being out of place - such vulnerabilities are understandable. It's about having an uncovered vent lead straight to the reactor core - that's stupid.

So I would need different accounts for my personal computer, my laptop, office computer and the computers of my parents? I regularly work on all of those.

This in contrast to the lead story, where some 200,000 accounts were accessed from a very small number of computers clearly not authorized by the account holders - achieved because not even a basic sanity check was performed. Heck, the servers didn't even notice that no login process was performed for the accounts, much less track which devices the account holders tended to use.

The lock on the front door was good, but the only thing that protected the bank until now was that nobody tried the windows & safes on the assumption that they would, in fact, be locked.

That's not weighing costs, that's being criminally negligent.

I agree that banking software is probably complicated. But really, this bug is a beginners mistake and it shoul dhave never happened.

It is something that the original developers should have known about and also something that the company auditing this code should have seen.

There is really no excuse for this.

Your White Whale unfortunately pops it head up everywhere.

Really? I'm sure there is more to it then I imagine but I've used the websites of 3 different banks and their functionality is quite limited, it looks like something you could do in a Rails app in a few weeks.

None of the real payment processing is done by the web app as far as I know, it is just a thin layer on top of a database with the transaction history and can store payment orders for batch processing later on. None of the real payment processing systems would face the internet I imagine, so I think the attack surface would be quite small.

Instead of a user id that is obviously some row in a database, use a UUID and do your db lookups against that. Worst case you have to put in some kind of local UUID -> user id lookup table because you can't manage to change the user record database directly. Now, barring a broken UUID generation scheme, it's virtually impossible to crawl for user accounts. Plus you have a scheme for providing the same guarantee on any other identifiers.

Obviously that doesn't prevent the larger problem of "guessing your user id and munging a url lets me into your account without a login", but it at least makes the guessing part worlds harder for relatively little work.

Not really. To make it so that waving a gun at a teller doesn't work you need to do lots and lots of complicated hard things.

However it's as hard to ensure that person with account A doesn't access account B.

How is it that a bank, of all places, pays money for a web infrastructure, and manages to employ people who don't even think about the most basic of attacks? I've been changing info in URLs since I started using the internet.

I would love to get more information about this breach it sounds to simple to be true.

Those two desires are in direct competition. For example, I bank at Citibank in America precisely because their website will allow me to initiate a US to JPY international wire transfer without my physical presence in the US. That class of activities is just about the most dangerous thing a consumer-grade banking website could allow you to do. (International wire transfers are practically non-reversible. If you are induced to send one to a fraudster or they compromise your online account and send one on your behalf, and the bank doesn't catch on within a few seconds, you're pretty much screwed.)

Accordingly, many banks do not offer online international wire transfers and will laugh in the general direction of adding it to their feature lists, despite it being technically not rocket science.

The rock solid feature that lets me eat on a semi-regular basis is also an attack vector against almost every other HNer with a Citi account, most of whom will never send money overseas. So, what should Citi do? Optimize for security and shut down that feature from their website, or optimize for being able to just wire some money in a rock solid fashion?

Perhaps require a setup process to enable the feature on an account-by-account basis. When you set up your Citi account you could have had another piece of paper work that you signed stating you know the risks of international wire transfers and that you authorize Citi to allow them to be processed through the website. Most customers wouldn't set it up and will have exactly the same experience as they do now, some wouldn't get scammed and customers like you also get to sit pretty.

You can change the registered phone number only by calling the bank and passing a set of identification questions posed by a human operator, so there has to be some significant identity theft to get past it. I don't think that would be particularly hard for a determined and experienced thief though.

I guess it hinges on what the meanings of "fancy features" and "everything should just be rock solid" are. I tend to agree with your earlier comment that for the banks this comes down to a risk assessment and a cost/benefit equation.

In practice what Citi does is neither of these absolute extremes; it just flags transactions over a certain amount (I believe it's generally $5K) and any transactions that the Citi systems deem suspicious. Anything in these categories yields a notification to the account holder, and the transfer has to be confirmed via phone before the funds are released. This is a variation of the "Are you sure you want to do this?" messagebox confirmation in programming.

When your phone has a NFC reader and your bank smartcard can talk to each other to handle it, even better. (Well, higher risk of intrusion because it's a multipurpose device, but way ahead in terms of usability)

profits are all they care about.

Honestly, stuff like this burns me up. Large orgs like this lobbied for crap like PCI compliance standards to 'protect' data, but they don't have to follow their own rules. Seriously, if PCI compliance is mandated for anyone who stores CC info, why the hell isn't citi being shut down for this sort of breach? 'too big to fail'?

Something this egregious should have been caught by PCI compliance checks (if not development) in the first place though.

I doubt the penalty will be anything severe enough - something like "no cc processing and management for citi for 180 days" might make them take this a bit more seriously.

Yeah, Citi's greater punishment will likely be in the form of a weakened reputation then it will be in actual damages paid.

Yes, yes you can.

Listen, PCI compliance is something people like to talk about, but you'd be surprised at the number of companies that don't follow even the bare minimum (unencrypted cards and keeping the CVV). We are talking about large, national corporations (and not Sony).

Besides, who do you think is writing this software? Normal people. There isn't a "Programming for Banks" degree you can get. It's programming. They hire contractors for months/years at a time, and then they are done with them.

=======

The method is seemingly simple, but the fact that the thieves knew to focus on this particular vulnerability marks the Citigroup attack as especially ingenious, security experts said.

=======

Sorry but no, this isn't ingenious - it's really the basics!!!

This is something that should cause the immediate dismissal of the CIO, but sadly, probably won't.

Visit http://www.nytimes.com/2011/06/14/technology/14security.html... instead

Suppose I accidentally stumble upon a gaping security hole in my bank's online service (or any other online service for that matter).

Am I legally obliged to notify them of that security bug? Can I offer the bank my assistance, for hire, in solving the bug without it constituting blackmail? (i.e. I'd be happy to help you solve this at a $300/hr rate)

Sure you can offer to fix it, but since you don't know squat about the system (save for a small flaw at the surface) and they have teams of developers who do, they won't be interested in paying you to fix it. Offering to explain the bug for a fee won't be blackmail unless you threaten to reveal the bug to others if they don't pay up.

Be a decent chap. Send 'em a nice letter explaining the problem. It's your bank, remember, and they're humans like you; work with your service providers to improve the service. Assume you're not the only one who knows about the problem, that someone who also knows isn't as nice as you, and it's YOUR bank balance that is at risk.

If you like their service and feel like telling them then go for it. If you want to try to charge them, then go for it (no it doesn't make you evil to charge for a service). If they don't want to pay you, feel free to say nothing.

Now if it's a mom and pop shop down the street, then yes, please be a good neighbour and help them fix it (although you can still charge for it, but avoid douchebaggery like $300 / hour unless that's your usual rate).

At that point the company is subject to practically anonymous shareholders through many levels of abstraction. Voting is then done based purely on financials and often short term gains, which really puts a company at odds with their customers. So I have no customer loyalty to a company that's publicly traded.

Many private companies can still maintain my loyalty though based purely on their actions if the owner(s) / investors aren't completely disconnected from their customers. That's more of a case by case. The smaller they are, the more likely they care about their customers (there are always exceptions though).

I would never threaten them to release it publicly though - I'd tell them that I'd forward the letter to the relevant government organisation. Where I am in Australia this would be something like the BFSO (Banking and Financial Services Ombudsman) and/or ACCC (Australian Consumer and Competition Commission).

As long as it's cheaper to clean up after the debacle than prevent it in the first place, that's what people will opt for.

If he really was fired and the company made a public stink about how much he screwed up, he may well care.

Many people at that level care about their prestige, and care about being shamed in front of their peers, even when they don't need to worry financially.

With some of the regulations the big missing piece is openness, there is no transparency into it at all. Any audited company should say who audited them and then after some period of time, 180 days maybe, the audit should be made public. The business risk is that customers will leave, in many cases like Playstation Network, customers effectively can't leave, they've already invested in something and there isn't an alternative. In many other cases it's not typically going to be widely publicized. If the customers can't leave, en mass, there is no business pressure for security and without any transparency the regulations will simply be gamed.

Even though it's insanely easy to spot and exploit it's also easy to miss it while coding. But any decent pen-tester will find it. Regardless, unacceptable for a finance company.

  curl http://example.com/user/[1-100]

You can fire the CIO, you can replace the offshore developers with onshore, or vice versa, but experience says it won't matter.

I looked in amazement at googletesting's dependancy graph test suites yesterday, and realised that the playing field is not flat at all.

Reading and writing code is the literacy of the 21st C.

And the end most big companies are like newspapers owned and managed by illiterates.

It does not matter how you rearrange the strucutre or the hierarchy, when the chips are down decisions will be made on what the illiterate management understand is the best way to work. As such it is infinitely unlikely that the decision then will be set up to support what a literate person would decide.

Until a generation of coders grows up, or all illiterate companies go bankrupt, this will merely be one of a myriad of pathologies exhibited by large companies run by the illiterate.

And as for the "hackers", I guess legally this was not even a break-in. At least in Germany, for legally being a break-in, a computer system must be "specially secured with the intention of preventing access". Well, this system wasn't.

...still, I have a hard time believing that it could be true.

To identify a logged in user and give the user access to their private account data, ONLY EVER use a unique and temporal random string. Nothing else. Ever.

Storing that random string in the URL may be done but is more insecure, because it will remain in the browser history. Not good if the user in on a public PC. Better store it in a cookie.

Otherwise - you're absolutely right allowing access to protected records - via id, key, or whatever - should absolutely be checked against the authenticated session upon every request.

Also doesn't say much for the company doing security review, its a basic check. Furthermore to not have a user/onwer id to join on there (no doubt sql back end) is shameful. I mean I can see it now :

select x from accounttable where accountnumber = @val

how about simply :

select x from accounttable where accountnumber = @accno and ownerid = @ownerid

http://www.dailymail.co.uk/news/article-2003393/How-Citigrou...

What this looks like, in the context of all the other serious recent breaches like Sony and the IMF, and from the point of view of someone who's never had to fight this particular battle but knows a little code, is that these corps deployed online apps in the early days when this wasn't a major part of their corporate face. Practices and points of view evolved from an initial environment where there just wasn't as much motivation for criminals to crack apps, because there wouldn't be that much of a market for what they stole. So corps could get away with deploying almost anything, relying on both security through obscurity and security through rarity (breaches were rare due to low profit). People in corporate offices that even knew their corps had these apps would be rare because the prestige of managing these people and apps would be low.

The apps we have today would then be direct descendants of the old insecure apps, and in many cases would be built directly on those old apps. Layers of mud, and you can't change the inside layers because old mud is brittle.

And now the corps are going up against, not people who are merely exploring or looking for bragging rights, but people working for criminal enterprises that, while not having the global scope of banks, are large enough and focused enough to directly challenge the technical power of the banks. And the banks are working with old, dry mud.

Again, grains of salt, but I suspect I'm in the right salt mine.

Disclaimer: I worked in product development making banking software and simple URL hacking was always a standard test.

The experts are certainly part of the problem.

There is probably no concept of linking an authenticated account to a restricted set of bank accounts. Instead, they've probably wired it up to CICS directly to retrieve account details. This is why the Quick Fix appears to be obsfucating the account number in the URL.

Is there a public report anywhere? Aren't companies required to report all privacy breaches?

Etrade's password system requires 6 to 32 characters with at least one number according to the password change instructions. They don't mention punctuation, but if you try to include some, they're deemed to be "invalid characters". Go figure.

This stuff makes me sad that I signed up for etrade:

"Thank you for your message regarding enabling HTTP Strict Transport Security. I have sent a request to our Product Development Team to have this feature added. Due to the high volume of requests, there is no guarantee that this will be implemented." -- etrade representative

"ETRADE does not allow certain characters to be used when establishing a password online. There is no specific ETRADE publication providing information on why certain characters are not allowed, such as special characters used for punctuation, etc. We appreciate your feedback concerning the online passwords. I forwarded this suggestion to the Product Development Team for future consideration and implementation. I can not guarantee when or if this change will be able to be made..." -- another etrade representative

These kind of vulnerabilities in high-profile sites can go unreported for ages, even up to the point that they are common knowledge in certain groups.

It's more like an apartment building with high-tech locks in the front door and apartment doors. But after you unlock the front door, you can unlock any apartment's door with your key! The keys are all identical, only the number printed on the label is different.

That's the minimum that should be done for those accounts which were compromised.