> Bam. Bam. Bam. Dummy trade signals that were supposed to stay within the company’s electronic systems broke loose and slammed into computers at the New York Stock Exchange’s options markets. So many orders crashed through that by 8:44 a.m., safeguards within Goldman Sachs sprang into action, severing the connection between the company and the exchanges.
Anyone whose built a trading system lives with this type of fear on a daily basis.
Write your risk system first and your algo second.
> The cause? A coder had mistakenly programmed a router to send placeholder bids as live orders. If not for the good graces of the options exchanges, the bank would have lost $500 million, according to the U.S. Securities and Exchange Commission. Cancellations and price adjustments reduced that to $38 million
And this is at the top of my things that are unfair in the markets. Knight Capital wasn't bailed out by the exchanges for the same thing and it bankrupted them. Goldman with its alumni every where.... was.
Imagine coming back from lunch, you start to unwrap your sandwich at your desk. Then, your boss walks by and throws your sandwich across the room and explains while you were out your bug caused 440 million dollars of erroneous trades - and the company now no longer exists.
Exactly. You don't just do code reviews and run tests to catch bugs. You also do code reviews and run tests so that, when something does slip through, it's no one person's fault.
The fact that this is even a thing is the problem. Is this industry creating anything of actual value or is this just a giant MMORPG where the object is to grind long enough to get a house in the Hamptons?
Should capital markets work? Should people be allowed to dynamically allocate capital to the most attractive companies?
The argument tends to be that making a decision about buying and selling in microseconds is not providing value, but generally everyone agrees that it should be possible at some level (by minute? by hour? by day?). There are a bunch of arguments here but broadly speaking I don't see how they change the underlying game. Make good decisions on where to allocate capital, make money as the market price caches up with your prediction.
I'd accept that answer if you added, "AND is just a giant MMORPG where the object is to grind long enough to get a house in the Hamptons". Both can be true. I think the original poster's criticism is valid.
In particular, since one of the objects is to grind long enough that the rules of the game don't apply to you, see 'Goldman Sachs', I think the original poster's criticism is valid.
There is no referee. The only goal is to give game winners more money, and if they screw up, give them their money back and then give them more money. It justifies things like investments in index funds (or identifying those who are already competing outside the rules and rewarding them by investing more money in them since they cannot fail), but it's gone well beyond the rational functioning of a system, much less a self-regulating system with useful purposes.
It's Calvinball. Go ahead and enjoy it, but it's not right.
One of the scariest moments for me was when I wrote a poker playing bot back in ~2005 and placed most of my months salary in its trading account before switching it on. Even though I knew I had written money management routines, tested them, and the program did sensible funds allocation and risk-of-ruin calculations, it was still nerve wracking to think that an unnoticed coding error could have wiped out my earnings entirely. (Spoiler - it didn't, fortunately!)
That's a bit of a mischaracterization I think. Exchanges have well-defined rules for clearly erroneous executions, and these are not the same in the equities (Knight) and options (Goldman) markets. If anyone else had made the options trades that Goldman did that day, their trades would have been busted as well.
> Exchanges have well-defined rules for clearly erroneous executions,
I'd disagree with this statement.
it's true that each exchange does have its own rules and that they do set some guidelines but as far as I know they all have rule that state that they carry final veto power over what trade do and don't get broken.
For instance from Nasdaq
> Nasdaq is providing general guidance on when transactions may be deemed erroneous under Rule 11890. This guidance is not a mandate of the rule but is information on how Nasdaq generally applies the terms of the rule (at Nasdaq2019s discretion) to determine whether an execution is clearly erroneous.
Yeah I really don't understand why Goldman should get a pass like this. They fucked up, they should lose money. Or better yet, errors like this should be reversed or safeguarded against better.
Honestly, this stuff just shows us how rigged the system is. This isn't the first time it has happened. Flash crashes caused by bad algorithms lost a bunch of money that was reversed as well. IMO, it is horrible to reverse this. They took the risk and got to keep the reward, but as soon as they fail, they get bailed out every single time.
> I really don't understand why Goldman should get a pass like this.
They didn't. They cancelled a lot of the trades before they were hit and a whole bunch were filtered out at the exchange end because they were way off market. They lost $38mm from what wasn't caught by either.
In other words, they fucked up and lost money. And it's generally agreed that safeguards needed to be better. That was a big part of Mifid II.
Different situations so I doubt it has anything to do with GS nepotism.
Knight couldn’t cancel their bad trades because most weren’t clearly erroneous, based on exchange rules. If the market was bid 20.01 offered at 20.02, Knight’s test program was buying 20.02 and selling 20.01 over and over. Those prices weren’t far from the prevailing market, they just executed stupidly and very quickly.
Goldman was selling $10 options for $0.01, so they met the exchange rules for breaking erroneous trades since they were so far from fair value.
If I were an evil Machiavellian exchange, I’d much rather stick Goldman with a nasty error than Knight. At the end of the day, exchanges need transaction volume to make a profit. Goldman is an 800lb Gorilla with huge capital reserves and diverse business lines, so they’d still come back to trade after licking their wounds. Knight almost went bankrupt.
> The cause? A coder had mistakenly programmed a router to send placeholder bids as live orders. If not for the good graces of the options exchanges, the bank would have lost $500 million, according to the U.S. Securities and Exchange Commission. Cancellations and price adjustments reduced that to $38 million
It basically outlines a cascade of failure in controls, bad configuration defaults, and poor SDLC. In particular:
>In addition, the firm’s operation and management of its electronic “circuit
breakers” did not effectively block the erroneous orders sent on August 20. These circuit
breakers existed to prevent erroneous orders by halting all message traffic to the exchanges once
that traffic had exceeded a certain rate. However, on August 20, the firm’s control personnel
repeatedly lifted the circuit breakers blocks between 8:44 a.m. and 9:32 a.m., thereby permitting
additional erroneous orders to be sent to the exchanges. Before lifting the circuit breaker blocks,
the control personnel did not obtain authorization from the responsible technology employees, as
required under written firm policies.
>The firm’s policies relating to the manual “lifting” of those circuit breakers were
not disseminated to or fully understood by the employees responsible for deciding when the circuit
breakers should be lifted, and, prior to August 20, 2013, GSCO personnel had lifted circuit breaker
blocks shortly after learning of the block and while still investigating the cause of the circuit
breaker trip.
From what I remember, this circuit breaker was notorious for raising so many false positives that control personnel just got used to lifting it without thinking.
That’s really a terrible place to put people in. From what this says, the SEC put part of the blame on the employees who lifted the circuit breakers without approval, and also state that employees didn’t know that they needed approval?
Before lifting the circuit breaker blocks, the control personnel did not obtain authorization from the responsible technology employees, as required under written firm policies.
The firm’s policies relating to the manual “lifting” of those circuit breakers were not disseminated...
> By the time the trades were blocked for the last time, less than an hour after they began, Goldman Sachs executed orders to sell more than 1.5 million options contracts for $1. The cause? A coder had mistakenly programmed a router to send placeholder bids as live orders. If not for the good graces of the options exchanges, the bank would have lost $500 million, according to the U.S. Securities and Exchange Commission. Cancellations and price adjustments reduced that to $38 million.
It sure does seem like there are two sets of rules on Wall Street. I doubt any small trader would be able to reduce their losses 90+% after errantly submitting a bunch of live options orders.
There are N^2 sets of rules. At the end of the day you have two counterparties engaged in voluntary exchange. Everything is negotiable, anything is possible -- you just have to ask or pay. What people don't seem to grasp about modern finance is how mind boggingly complex and dynamic it really is. The idea that a tier one investment bank and a small trader should be held to the same "rules" is so wondrously silly... it's hard to describe but there ought to be a word for it.
The only thing really annoying here is that all this wackiness is backstopped by the Federal government. Nobody has any doubt that at the end of the day they will step in to save GS if everything really goes sideways. And that's probably a good thing. But it's very strange that the big banks, which are essentially quasi-state actors at this point, get to keep so much of their profits when the public is bearing so much of the risk.
This is the real difference between China and America: in China the government does everything in its power to make sure its state-owned enterprises succeed and then it takes its pound of flesh. In America the government does everything in its power to make sure its state-owned enterprises succeed and then it gives them a huge tax cut to make the shareholders that much richer. It's a bold move, let's see how it plays out.
>The idea that a tier one investment bank and a small trader should be held to the same "rules" is so wondrously silly... it's hard to describe but there ought to be a word for it.
Just like the idea that the aristocracy should be accountable to the same laws as commoners, right?
No but if GS asks a counterparty (with whom they have billions in other contracts) to cancel contracts as a result of a system error, the counterparty will probably agree (if they don't lose as a result) knowing that GS will next time do the same. With a small party, this could just be an attempt to undo a bad deal.
I think what the poster meant is that deals are just contracts and can be amended if both parties agree. For large banks with longstanding relationships, that is naturally much easier than for a trader that no one knows.
Yes, and that's a good scaling rule. the person with $50 dollars is likely clueless and stumbling around trying to figure out what the rules are. The person with $50 billion can do serious damage when they screw up.
The rules in place are exactly backwards. the $50 dollar team is held to a high standard, with no affordances. the $50 billion dollar team is part of the club, so enforcers look the other way when they screw up.
It is not backwards at all. Corporations are made up of a ton of people, there's no one person you can lay blame on. Some guy fucks up and costs a firm half a bil with a computer error, that's not something you want to just allow to happen.
A dude losing $50? Give me a break, his risk is his own. Consider his $50 a small price to pay for learning how things work. And let me remind you, it is a very small price.
"Corporations are made up of a ton of people, there's no one person you can lay blame on. Some guy fucks up and costs a firm half a bil with a computer error, that's not something you want to just allow to happen."
That's exactly how the big corporations hold the whole country hostage. When they do things well they get to keep all the profits and distribute them but when things go bad it's suddenly nobody's fault and the rules have to be changed. Since 2008 I am of the strong opinion that if someone in a corporation makes a big mistake we should let them go under without hesitation. Otherwise they can always blackmail us into being bailed out or working under a different set of rules if they make a mistake.
I love this. This is the ultimate wisdom of crowds hack.
Have a thousand people throw in a thousand bucks, each one selects an option play to purchase. Randomly select the purchase. If the play is profitable, keep the money. if it's a loss get the trade reverted because 'a lot of people were involved in the process.' I can make the process as convoluted an necessary to meet your 'no one person you can lay blame on' rule.
I think that's a stupid rule, because it's so easy to hack.
The more money staked on a trade, the higher the threshold should be to roll it back. More people mean more chances to catch the error, not the other way around.
edit
also, the corporation itself is the person that takes the blame. if it can't manage itself, it shouldn't manage your money.
That pretty clearly wouldn't work. A necessary (but not sufficient) condition for breaking a trade typically involves language like "clearly erroneous". Selling an option that is trading in the $100s for $1 (the case here) is probably in this class; trading at the market and then having it move against you is completely different.
How do you know the player isn't up to something intentional, for instance triggering some kind of 'outside investors suddenly acquire all of a stock or bond that we think is going to tank in the next five minutes'?
Are stock market players not allowed to divest extremely suddenly, to dump their property at fire sale prices in order to get rid of it at what would be a paper loss, all the while knowing the consequences of holding would be worse?
What if they're dumping equity in a company that will be known to have committed terrorist acts? I'm not sure if I buy 'clearly erroneous'. I also don't buy that the scale of the error was really threatening to Goldman Sachs.
And the rules defining what is 'clearly erroneous' and what is not is intentionally made vague because arbitrage of this clause makes the exchanges money.
They stand as the arbiter of who will gain money and who will lose. There basically is a series of cases 'erroring parties vs Goldman' and 'erroring Goldman vs other parties' and whether Goldman will lose money is decided by an Exchange (hint: they rule in Goldman favor every time, making an analogy with forced arbitration even more proper)
....what? Your logic is truly frightening. Are you saying this is a regularly occurring event and that they deliberately did this? Do you have any idea of the scale and complexity and risk of the code they have deployed? This shit understandably happens. There is no "more people means mistakes don't happen" in any organization on the planet.
There is no "whether or not who should morally be able to roll back a trade". There is a "hey, we are a customer of your business, we do a lot of business together and I make you a lot of money. We had a once in a blue moon mistake in our billions of lines of code, can you help us out. Other banks are watching and there are plenty of other exchanges to do business with"
> Are you saying this is a regularly occurring event and that they deliberately did this?
Nah.
> Do you have any idea of the scale and complexity and risk of the code they have deployed?
No. But apparently neither do they.
> ... I make you a lot of money. ...
With basically every other risky thing people and corporations encounter, the response is "go buy insurance". Usually they're told that by finance guys.
Every exchange that I’m aware of has a rule allowing them to roll back clearly erroneous transactions. “Clearly erroneous” isn’t always clearly defined, but it would obviously apply in the GS case (huge volume executed at $1).
He wasn’t live trading, he was a programmer, like any other dude here in HN, and he wrote something incorrectly. A bad trade went through. Is he to blame? Who knows, no one else caught his mistake and maybe he actually coded what he thought was asked for, or someone told him something wrong.
For instance, the effect was that Goldman lost money, while some other guys made money - what's the harm here ?
Moreover, the trade being busted, the guy that initially made the money, probably found himself in a very uncomfortable position, since chances are he already covered his risk and hedged the lucky trades. So overall, he lost money just because Goldman was able to force the rules in their favor.
Unfortunately, markets are rigged, just as most things -> the bigger you are, the more influence on the rules and how they are applied you have.
You're not contradicting the objection, you're just outlining another mechanic by which it happens.
"The New England Patriots reached ten Super Bowls, therefore the foot out of bounds just before scoring a touchdown was clearly a mistake and out of the ordinary. Therefore, award the Patriots the point, on the grounds that they usually don't make mistakes like that. And don't check the pressure in the football please, there have been ten Super Bowls where that wasn't apparently a problem…"
(I wonder who I'm insulting more, the Pats or Goldman Sachs :D )
Loving the analogy :D but (and excuse my ignorance), but in NFL those rules don't allow that, where as it appears the markets do have these mechanisms to allow it?
They do allow it, but that's precisely what people are complaining is unfair. They are literally too big to follow the same rules as the small traders.
One upside is such environments help nurture future technologies as they will happily pay millions for something just a bit faster than what they already have and more so before others have it. Hence it is avenues of business like this that have help feed certain industries over the years, from networking to FPGA's. Another avenue of financing technology which we all know is the military.
As for the whole stock market being driven faster and faster in precision from seconds to nanoseconds and picoseconds, I've always questioned the need to drive towards a faster processing and if the stock market only updated once every minute, it would sure help in curtailing many ineeded aspects that are creeping into stock trading.
I personally feel we need to change the way exchanges operate. Too much emphasis exists on speed of execution. We must consider alternative auction formats where speed of execution is not the only criteria for matching an order, as it gives rise to masses of spam generated by a thundering herd of traders as they all try and fulfill the same opportunity.
In no particular order, their solution doesn't seem to deal with:
- It doesn't account for the arbitrage types that seem to bother people the most: cross-exchange arb, locality arb, regnms arb or payment for order flow.
- If the discrete time chunks are very short, it doesn't seem to solve latency arb much.
- If the discrete time chunks are very long, it defaults to pro-rata matching. We have pro-rata products already, they are not kind to the little guy.
- Most of the really weird edge cases in exchanges happen due to so called 'exotic' order types, which are really just ways to get the exchange to atomically do something for you. In the discrete auction world I'd imagine you'd have more need for complex instructions not less.
Finally, I think that the problem they are attempting to solve is fairly low priority to solve. We are operating in the lowest cost trading environment of all time. It could hardly be more fair when it comes to order execution. To throw all that out seems dumb, when there are much bigger fish to fry (such as our government bailing out one set of traders but not others).
Just wanted to say, I really like the design of your blog and your writing style. Also, there is a minor typo in the last paragraph of the IOC/DI article.
For one thing, it claims to eliminate the speed arms race by setting a sufficiently long gap between auctions, but depends on a bizarre assumption that there are 2 speeds at which information can get to the market, each at a particular cost. I suspect this would be better modeled as continuous, which would imply that tuning the auction gap merely moves the clearing price for additional information speed.
It also doesn't cover the fact that it changes common exchange revenue streams; the necessary replacements would surely affect the market. Similarly, a lot of the more complex order types would no longer make sense, which would again have market effects.
A lot of this stuff is hard to figure out, though, and trying it in the real world isn't be an obviously bad idea.
Why aren’t these firms insuring against this risk? Seems like any other risk that can be managed - pay premiums and so that a third party would fund any erroneous losses.
Because they don't have to? They don't suffer from the risk...
Clearing frequently act like a form of insurance for smaller firms, up to and including asking for killswitches in your trading engines and audits of your risk procedures.
But big firms clear themselves so don't have that. Theoretically the internal risk team is responsible for that but those teams are frequently undergunned.
Cleared themselves. I'd be shocked if any company in the world would act as counterparty to a trade that hedged 'operational risk at Knight scale'.
One of the thing many of us in the industry at the time commented on, was how little was done to bail Knight out. Other than the 'oligarchy' argument that states that the old timers hated them (they did) the argument I subscribed to was, they weren't systematically important. Their entire function in the market could be taken over quickly by someone else with little disruption and largely they were the only ones that lost money on that day (not really but to an approximate).
"If you just hurt yourself, nobody cares. If hurting yourself hurts others, then other people care." That's pretty cold-blooded and brutal, but it does make some sense...
Except it wasn't the machines that caused the issue, it was a bug (and lack of following the controls in place) caused by Lucas Renick, of Goldman Sachs, who was fired on the spot and remained unemployed (though still on the Goldman payroll, in order to avoid a lawsuit) for many months until he joined Bank of America and later Morgan Stanley
Places like D.E. Shaw, Two Sigma, Jane Street, Renaissance, Citadel, etc... were doing this long before any of these banks. To call it innovation is a bit of a misnomer in my opinion, more of a catching up with the times really.
Efinancialcareers is a good site to peruse job ads and see what people are looking for. The banks themselves all have job postings on their sites. Once you have the main qualifications you can apply directly or contact a financial recruiter.
Anyone whose built a trading system lives with this type of fear on a daily basis.
Write your risk system first and your algo second.
> The cause? A coder had mistakenly programmed a router to send placeholder bids as live orders. If not for the good graces of the options exchanges, the bank would have lost $500 million, according to the U.S. Securities and Exchange Commission. Cancellations and price adjustments reduced that to $38 million
And this is at the top of my things that are unfair in the markets. Knight Capital wasn't bailed out by the exchanges for the same thing and it bankrupted them. Goldman with its alumni every where.... was.