Synopsis: SEC sends clear message to tech people in finance: shut the fuck up if you find something, silently fix it, and sweep the remaining crumbs under the rug, or else your company will be fined millions.
If they did that and the SEC found out, they probably wouldn't stop at just fining you. Which is probably what Citigroup guessed and why, after realising the extent of the issue, they reported it fully within a month
Indeed, Martha Stewart is a prime example of how NOT to deal with the SEC. Turns out she wasn't guilty of insider trading, but she still went to jail for lying to the SEC about the trades in the first place. They take that very seriously.
Directors of companies have legal responsibilities to provide these reports to the government. So instead of the company getting a fine for making a mistake even though you tried to comply, now you are personally choosing not to fulfil your legal obligations as a director. Smart move.
> Directors of companies have legal responsibilities to provide these reports to the government.
What if you don't tell the directors, but just fix the problem you have discovered? Is everyone off the hook then, or does that also amount to breaking the law (in some country)?
Why you're in Operations or Technology in large financial companies, you're confronted with 'could do better' issues all of the time. All of the time. The more issues the greater the legacy of the system you're working on, as in this case, functionality has been altered from the original vanilla system (here, alphanumeric field type replacing a numerical feed type, the the documentation not being updated/references/or probably in an unreferenced document documenting the original system).
Spotting these things is a _fantastic_ way for a VP to get on the fast-track to Director. And it is fantastic, as a Director, to have people on your team actively looking for holes.
Why?
Because Operational Control is a #1 strategic target for all banks. $7million is nothing. At an extremely senior level this is evidence that a culture of transparency and compliance exists in the company, and ammo the next time the SEC or FED express a 'concern'. At a low level it is a Director or VP demonstrating to their boss that they understand the strategic direction, and that under their watch nothing big is going to blow up, nothing $7billion big; something $7million big is nothing, they know their boss knows this and will get a thank you for it being raised.
So you tell the directors. You make a nice PPT and include it in 'initiatives' when a senior visitor comes to visit and gets a de-brief on your department. You make sure it's carefully and clearly explained, so they can explain it to their boss in a nice, pro-active, continuous improvement kind of manner.
A bank which does not operate like this, in the post 2008 era of regulatory punishment for purgery, is an organisation with a very short future.
Source: Work in Operations and Technology in large banks.
And run the risk of losing big, instead of knowing for sure you'll lose small. Can you get a good estimation of the risk? If not, can you (not your company: you) afford the big loss?
Isn't the lesson that you should blow the whistle? You could take home $700k to $2.1 mio, which is not bad for a back office developer. This is a lower bound because I'm sure the fine/reward would have been bigger if they didn't report it themselves.
$7m is a slap on the wrist for the sort of thing we're talking about, and coming clean shouldn't be a get out of jail free card either, which brings with it all sorts of pathological behaviours as well
Would you like someone to actually slap you on the wrist every time you make an innocent mistake in which there are no actual damages and nobody is hurt?
This is not a case where the company failed to transfer money owed to the government. And even in such a case, the appropriate remedy would be actual damages plus interest, where the interest is at some punitive rate.
Because some transactions were innocently concealed, the damage is that the government may have lost some opportunities to catch some people laundering money through those branches of the bank. But that is very indirect. In any case, in those cases they would probably have existing suspicions, right? And they would notice that, oops, that person is using a branch for which we have no data from Citigroup: how come?
The SEC didn't catch this error precisely because they were not investigating any user of those branches for which data were missing.
They only lost the opportunity represented by situations in which the data is the primary source of the initial suspicion of wrong-doing. That is all. As in, something in the numbers raises a red flag, and then they investigate and uncover something.
Well, they have the data now; they could comb through it, right? This 15 year period, or at least most of it, should be well within the statutes of limitations that they could still prosecute cases uncovered by the data.
$7m is basically nothing to someone like Citigroup. If they had tried to cover it up and the information leaked out some other way they could be facing a real fine.
A $7m fine for someone like Citigroup is basically a parking ticket.
These fines are fundamentally stupid: yes, Citi's IT fucked up, but the fault is with the regulators for having such braindead reporting requirements to begin with.
Instead of something sensible like "you trade, you report", the regulators have set up a patchwork of formats, inclusion criteria and target agencies that pretty much ensures that lapses like this occur. Turning Frank-Dodd into workable code is fucking hard: I spent a couple of years pushing that boulder up the slope until I ragequit a couple of months ago.
The regulators should be ingesting data in one place and running analytics on a data lake. The banks are so sick of spending money dealing with the lack of regulatory technical competence that they'll probably happily pony up a couple of billion dollars between them to set up the surveillance system for the Feds; doing it once is cheaper in the long run than repeating the same set of mistakes at every bank on the Street.
The banks are sick of this, but just like the grass that doesn't want to be eaten by the cows, they also depend on this complicated regulation to keep our competitors.
I'm not sure complicated reporting is how banks are maintaining margins. They're diverting billions into complying with these requirements, for very little gain.
And anyone entering their business would have to as well. The large banks are happy to have these regulations in some ways as it makes it hard for midsized banks to grow and compete with them.
> Citigroup failing to send information on 26,810 transactions in over 2,300 such requests.
26810 requests, do you think they made more than $7m on this?
It's about proportional fines. Just because they're a giant company doesn't mean we should find then $1b for forgetting to put a handicapped parking space at one of their offices.
There is a school of thought that punitive fines should be a proportion of company value/earnings rather than an absolute dollar figure to exact an equal amount of discomfort.
I think Finnish speeding fines are a percent of income.
There is a school of thought that punitive fines should be a proportion of company value/earnings rather than an absolute dollar figure to exact an equal amount of discomfort.
In the infamous McDonald's coffee lawsuit, this was actually the motivation behind the initial large damage award. The jury attempted to award punitive damages equal to two days' worth of McDonald's coffee revenue.
(obligatory note here for the many people who have heard false information about that case: the coffee spilled in a car, yes, but the car was motionless, in a parking space, and the person who spilled it was not the driver, and was found partially at fault for the spill; she suffered severe burns requiring hospitalization and skin grafts, which is not generally what one expects from coffee; it was found McDonald's served its coffee significantly hotter than other chains, in a temperature range making burns more likely, and was aware of the fact that it could cause severe burns because this wasn't the first case, and in fact McDonald's was aware of hundreds of cases of burns resulting from its coffee; the initial damage award was significantly reduced by the judge; search for Liebeck v. McDonald's for more details)
Only because the fine is based on your income doesn't mean it has to be draconic. Quite the opposite. They can actually be less draconic than fixed amount fines. If you earn minimum wage a speeding ticket can be a complete disaster. If you are a tech worker you will barely notice the ticket.
25-to-life is indeed excessive, but if some people had 25 year lifespans and others had 2500 year lifespans, "3 strikes and you're 10-25% of your lifespan" is actually quite reasonable.
I think I see what you're getting at: punishments or penalties for offences that are arguably disproportionate to the crime.
On the other hand I think there is a gulf of difference between "3 strikes" laws and punitive damages being awarded against a company for an arguably frivolous lawsuit.
The former will disproportionally target people who are relatively disadvantaged, e.g. people living in poverty or drug addicts, and ruin their lives.
The latter (in this case) targeted a multinational corporation, with the damages being around 0.01% of its annual profit or less. McDonald's can survive that. Even if a person or two in the chain of command get fired, there's a gulf of difference between losing your job and getting locked up for 25 years.
I know you're not talking specifically about mcd coffee suit, but there's nothing arguable about that one; because there's still some measure of belief that it was frivolous here's a quote taken from the wikipedia article[0]:
> Liebeck was taken to the hospital, where it was determined that she had suffered third-degree burns on six percent of her skin and lesser burns over sixteen percent. She remained in the hospital for eight days while she underwent skin grafting. During this period, Liebeck lost 20 pounds (9 kg, nearly 20% of her body weight), reducing her to 83 pounds (38 kg). After the hospital stay, Liebeck needed care for 3 weeks, provided by her daughter. Liebeck suffered permanent disfigurement after the incident and was partially disabled for two years.
What? Absolutely not. I'm just saying that it sounds reasonable to believe that burns increase with temperature, as opposed to being high in a range of temperatures and lower below and above that range. Figure 4 in your first link agrees with this common-sense guess.
The guy said "in a temperature range where burns are more likely". I'm not sure why you're objecting; there are certainly ranges where burns are less likely; that's what the paper I posted is about.
I think it's also pretty clear just from the burn time curve. If it takes you a second or two to notice the heat and move away, then anything above ~155F is going to make a burn much more likely. At 180F, the burn is basically instantaneous. Whereas at 140F, having five seconds to respond gives you a lot of time to move, shake off the liquid, et cetera.
Not to me, so I'd say it's more "your issue" than "the issue". Industry articles on serving temperatures are all about ranges, so I presume he's just talking about those.
But even with your interpretation, it's true. The upper bound is 212F. A cup of steam is not a significant burn risk, that being something like 0.16 ml of water.
Even if fines were in proportion to company value/earnings, they would also need to be in proportion to the violation. Not every violation is equally as egregious.
> I think Finnish speeding fines are a percent of income.
Yes and no. Lesser infractions are static amounts, larger ones are based on day fines, they determine the amount of days -> units for the fine based on the severity of the infraction and the units are used as a multiplier against your daily income for your fine.
I've thought long and hard about this for many years. Punitive damages - eg jail time or financial punishment should be removed from our system except in cases where people are unfit to be in society (like murders' etc).
What should happen - all of the management at Citgroup should have to attend a 5 day training provided by the SEC showing how they can fix the reporting problems they have. How it causes problems to society. How it is dishonest - and how it wastes taxpayers dollars.
I'm betting having that happen to Citigroup 2 or 3 times a year would really make them think about following the rules.
Plus - its positive reinforcement instead of negative (punative) damage.
same goes for Switzerland, but as common sense suggests, there is and should be some threshold based on severity. below is static fine, above it things get more interesting/intense.
some (a lot) people would like to see banks burn, in same way common folks enjoyed public decapitations of ruling classes in french revolution. not judging, hard topic on its own, probably depends on where you are positioned in your life.
For me it's not as much about fining them as it is holding the correct people responsible (not just throwing some junior engineer under the bus as usually happens) and making sure it cannot happen again or gets noticed much more quickly
Large companies have more employees and will have more violations, maybe proportional to their size, whereas a rich person should not have more speeding tickets than a poor person and should arguably pay more per violation.
That may have been true back in the 20th century, but not anymore--Apple and Walmart have market caps on the same order of magnitude, but one employs far more employees than the other. Besides, if you have more $ per employee, that just means each violation will cost more money/damage.
The other thing is with a company the size of citigroup (250,000 employees), it is statically impossible not to have:
- incompetent and/or careless employees and managers
- dishonnest employees
- computer bugs, glitches, clerical errors
If you take down a large corporation every time you find any of these, there will be no corporation left within a year. Just small companies that were statistically lucky to have neither of those that particular year.
Name me a program, any program (other than Hello World) where no bug has never been found!
They had this bug for more than a decade. They should have systems in place to look for these things.
People make mistakes, but these mistakes should be caught before they get into production. And the ones that still make it into production should be hunted.
Should we discuss about all the 15 years old bugs that are found in Windows, Linux and MacOS which are well into production (and many of them critical bugs that affect the core of the product)? Has a software company ever been fined or held liable for bugs in its products? In fact too often, bug fixes are paid updates.
Financial companies are held to extraordinary standards, and in my opinion it's a game they cannot win.
This is not a minor bug, this is a bug that caused data to be misrepresented. If you want to compare it to OS bugs, then you need to look at silent data corruption: how many data corruption bugs have gone undetected in operating systems for 15 years?
You mean a server OS leaking kernel memory to any external connection doing something special with TLS without leaving any audit trail that this happened is a minor bug?
this is unbelievably naive comment... do you work in IT? 15 year old bugs are nothing special, with known ones having workarounds implemented (often buggy), or just some completely new happening on broken data feed, unexpected values etc. the list is endless
Who said anything about being evil? If my car manufacturer has a bug in their manufacturing process and I die in a ball of fire as a result, they weren't being evil (they didn't intentionally kill me), but it doesn't change the fact that they screwed up and are going to have to make amends.
Citigroup screwed up. Now they have to pay a fine. If they get off the hook for free, how is that fair to their competitors, who also had to do this rather tedious reporting?
Citigroup was $7 BILLION for their mortgage fraud. [1] I think that can be considered a "real consequence." This was a far less severe infraction, and the fine was correspondingly much lower.
I was thinking the exact same thing. They should have a really thorough investigation and they need to make sure that NO ONE knew about this bug at any point in the past 15 years.
If Citygroup has a habit of hiring and fostering employees who turn a blind eye and keep their mouths shut, then they should be punished severely.
They didn't lose anything, they were FINED $7m for submitting incorrect reports. With the number of mergers that they have gone through it's not surprising they have trouble with company wide reporting.
"But in 1998, the company started using alphanumeric branch codes as it expanded its business. Among them were the codes 10B, 10C and so on, which the system treated as being within the excluded range, and so their transactions were removed from any reports sent to the SEC."
The original title stated said "Citi lost $7m after legit transactions mistaken for test data". That was what the Register used too, but it's very misleading.
I think the most impressive part is that apparently the same system and code had been running for 15 years as the number of transactions reported on had increased exponentially
Good point. Just under 5 transactions a day really. I think the point that's really annoyed the sec is the number of years they have to go back and correct now that they have the right data.
i think the bad kind of impressive would be replacing a system that's had thousands of man-years of real world user testing without a really really really critical reason.
I'd have to disagree on that part. The ATMs that the company I work for run Windows. We are the largest manufacturer, until the other 2 big ones finish merging.
Besides, C is 44 years old and used on a ton of stuff.
I took this in the UK two years ago: https://4z2.de/atm_windows.jpg - I don't know enough about how the various versions of Windows look but maybe this helps.
You need to trust the bank's network security more than you trust the ATM itself.
There were some high profile breaches at some retailers in the past couple years that exploited some 0days. How can you defend your POS/ATM against a 0day if the retailer/bank has bad network security practices?
In the US, we have private leased lines to connect with some of the major banks. But the banks most likely use something else to connect to the ATMs.
Most attacks are physically breaking into the cash safe but there were some attacks a couple years ago where people were plugging into the USB port and getting money or something.
Part of responsibility here is on SEC too IMO. When designing that "blueprint" format and data exchange protocols, they didn't implement mechanisms to verify correctness and completeness of data received.
Important to note that they only left these transactions out of report data sent to the SEC. It's not like they were not honoring the transactions or that some people were missing money.
I have done bluesheet reporting code for another large bank and it is one of the most tedious things you can ever do in software. It's a report of all the trades that the bank does and the SEC can come back and ask for historical data at any point of time.
So there's a huge database with feeds coming in from multiple trading systems. Usually these feeds have to be enriched with the right account numbers and so on. This needs lookups to other reference data systems, which frequently are changing because of changing regulations, growing businesses etc. I am not surprised this bug remained undetected for more than 15 years. The guys who coded this initially are probably long gone and nobody did knowledge transition of the fact that some account ranges are not test accounts, even though they look like test accounts.
Totally unrelated, but I am not surprised. I once logged into my Citi credit card account and was granted access to another user's account. Certain places were off limits but I was able to view a lot of details. Pretty scary! I never heard back after reporting the issue.
I did grab a few screens and contacted the only available "support" link I could find easily on their site, but after a few weeks with no response I eventually deleted the files.
Arbitrary short and easy to remember string presumed to be unlikely to generate false positives when searched for. TODO can generate false positives if your system actually deals with todo items, etc.
I suspect the bugs like this come about because of a patch not because of the original development. Devs and dev teams tend to get sloppy after the first push and budgets tend to shrink dramatically.
One time I found a bug that was running for a few years and the result of it was the company was under reporting by millions of dollars per quarter (the running total was near to $100mm, and im sure it crossed it after my contract ended).
This was VERY well tested software in the beginning (one of the best test suites i've seen actually) and audited up to high heaven. The problem started when the patches rolled in and those, are not tested anywhere near as much.
function should_not_exclude_10B_10C(x) {
if ("089" < x < "100") {
console.log('excluding ' + x + ' in report.')
} else {
console.log(x + ' is normal.')
}
}
I've never understood how these fines are meant to benefit anyone. If no one is affected, then why is there a fine? Who is this money going to for damages to be repaid?
Also, why are you not allowed to use real data for testing purposes?
The SEC was not impressed and said in a statement announcing the fine that the "failure to discover the coding error and to produce the missing data for many years potentially impacted numerous Commission investigations."
There is mention of trading data which suggests to me that it might be to do with traders employed by Citigroup ie staff members whose job it is to use a block of the bank's money or outside investor's money to buy and sell shares, commodities and pretty much anything where they believe they can buy and sell to turn a profit.
