This kind of attitude comes up on HN a lot. They have lots of internal tooling and jobs that have to be done, especially at their scale.
WhatsApp with 50 employees is an anomaly. In reality, companies need lots more employees just to get by. Not all industries are equal when it comes to responsibilities and programmers are the most ignorant but arrogant class, thinking that all you need is software. Spolsky covered this 2 decades ago in his “Developer Abstraction Layer” article.
Not surprisingly, software engineers think that the only thing going on with a company like Uber is software engineering. Which is way more complex and requires way more people than it is frequently given credit for. (Uber isn't just an app; it's a bunch of apps, supporting a 2-way marketplace with complex dynamics and many millions of participants.)
But the vast majority of Uber's employees are not software engineers. Running a physical service in ~70 countries is /hard/ and requires a lot of people. Support and operations eat up a lot of the headcount, and can't be scaled like software. Similarly you need a lot of accountants and lawyers to operate globally, along with all of the infrastructure to support a global workforce.
They're doing complicated things, but is that headcount in proportion to it? For comparison, Apple's HQ headcount is under 20k, and entire state level governments for small to mid-sized states are in the low tens of thousands of employees. We all know human things generate lots of work to employ lots of people, but the scale still feels a bit off, even if no one can point to exactly why.
It doesn't feel off to me. As others have discussed in considerable detail, operating in a lot of countries worldwide is a lot more complex than it intuitively seems to be, and the complexity scales superlinearly.
I suspect they might be a few percent overstaffed, but I don't think that hurts a company focusing on growth over all else, and I don't think it's even all that uncommon for such companies. It wouldn't surprise me to see a round or two of layoffs once that growth tops out, but it would surprise me if they added up to more than 10% of headcount.
I work for the a huge financial institution, its the largest exchange by volume, we operate in multiple markets all across the globe and have our trading platforms distributed across many locations such as NYC, London and Tokyo in datacenters we own, manage and operate, our matching engine matches more daily trades than there are daily Tweets all with a guaranteed sub 50 microseconds latency with 99th percentile sitting around 20 microseconds.
We have about 4500 (up to 2 years ago it was around 2500 a bunch came in when we acquired a company nearly our size, it will probably shrink to about 3500-3000 over the next few years) employees, and about 1000 of them are engineers who develop and support a few 100’s of applications.
We have to comply to dozens of different regulations in dozens of regulatory regimes, so it’s unlikely that Twitter needs more compliance and lawyer than we do.
We are considered critical infrastructure, if we have a hiccup the world’s financial markets will be having a stroke. If we go down or worse that’s not some lost ad revenue but bilious to trillions on the line.
It never ceases to amaze me how easily the attitudes of finance folk so easily leak out. It seems performing a worst-case O(log N) lookup and spitting out the resulting 100 bytes over UDP counts as harder than Twitter, parent comment clearly has no idea how much work Twitter needs to perform for every tweet (URL previews much?)
Meanwhile, could have simply said you work for NYSE.
I don’t work for a stock exchange, stocks are not particularly high volume nor particularly that important for the market as many other instruments and asset classes.
Your comment shows you clearly have no idea how modern QDM and ODM markets work if you think that sending 100 bytes over UDP is all there is or even what we do, we don’t even use UDP...
An exchange like that by definition is quite focused on tight non functionals with slim functionals, aka it does a small number of things very well. Vs Uber which does a very wide set of things at a high level of consumer polish.
In my mental model of the two problems to solve, Uber is directly consumer facing to
Millions of humans and that alone creates so much work in engineering that doesn’t exist when you can craft an api with sub 50 ms latency consumed by thousands of other machines.. the humans create the complexity. Solving for their problems is why you need way more people.
Order matching, order execution, credit and limit controls, reporting, multiple APIs and interfaces for order entry, market data and many other things supporting everything from legacy FIX clients coming over leased lines to modern restful web APIs and SBE over websockets, identity management with highly complicated RBAC model for traders and firms (who can trade what on behalf of whom under which conditions), highly complex and robust logging especially for regulatory requirements, maintaining alignment with regulations across the globe...
And if you think normal end users are picky you haven’t met traders that still want to use a 25 year old keyboard with non-standard ASCII codes that’s barely recognized as an HID device on your new electron based trading frontend and if you can’t accommodate for that they will leave.
So you have millions of clients all of which are adversarial and you have to maintain market fairness and integrity to ensure no manipulation is possible with trillions of dollars on the line that can be wiped off in a heartbeat and that is before you start thinking about nation state actors who would love to be able to cause major economic damage to the US, Europe and other major western economies.
Yeah the complexity is insane, millions of users is surprising tho I thought it would more be an aggregation of larger agents.
I guess unicorns optimize for delivery capacity not cost
> If we go down or worse that’s not some lost ad revenue but bilious to trillions on the line.
What products are exclusively traded on your exchange? If NASDAQ or NYSE goes down for half a day it’s just an article on WSJ at best. They shut down every night and for entire days on holidays. What financial exchange truly has a meaningful impact when it goes down?
Equities aren’t a real economy choke point. They are also, as you observe, federated.
A more critical and delicate financial artery is the tri-party repo marker. If BoNY Mellon or JPMorgan’s repo desks went offline, it would almost certainly shut down large swathes of the American economy, prompt a credit crisis and merit urgent Federal Reserve and possibly Congressional intervention.
But what is the timeline here? If JPM goes down for 2 hours it will cause a congressional intervention? That’s honestly really hard to believe and it borders in delusions of grandeur. Congress intervenes after days at a minimum of an ongoing crisis.
My point is that your strange Internet dick measuring contest of micros to match up an order book is only one tiny aspect of engineering. Your uptime numbers have lax requirements, your geographic distribution is severely limited, and your client counts (as in clients directly accessing your software) are orders of magnitude smaller than the Twitter/Uber you compare yourself to.
Operating a financial exchange it not trivial by any means, but to compare your challenges to a massive public-facing service is apples to oranges. There are likely more adversarial clients connected to twitter’s services at any given time than the peak concurrent connections to any given exchange.
> * If JPM goes down for 2 hours it will cause a congressional intervention?*
It will create a crisis meriting Congressional intervention. A hiccup in the tri-party repo market automatically sends, within twenty minutes, a large fraction of the American financial system into default within twelve hours from almost anywhere on the clock.
> That’s honestly really hard to believe and it borders in delusions of grandeur
Are you saying JPM going down means there is no alternate provider and everyone defaults in 12 hours or if the entire tri-party repo market fails? Those are not even remotely the same thing.
I would be shocked to learn that Exxon mobile will go bankrupt in half a day because one bank has an outage. It would speak volumes to how piss-poor financial networks are architected.
More realistically, JPM goes down and within an hour all of its large repo clients have already called BNY or whatever and are ready to go. Billion dollar businesses have contingency plans for headquarters being taken out by a bomb, you better believe they have a plan for their repo provider taking a shit.
> Lovely
Do you not see how ridiculous it is to suggest that the entire functioning US economy hinges on a single service run by a single company being available 24/7? If it were true, it would be a bigger terrorist target than the Whitehouse.
Apparently the entire US economy is dependent on the uptime of the work of a few engineers at one company. Better hope that information doesn’t become public or else they will be prime targets for coercion, bribes, nation state level manipulation, etc. These engineers are literally more important than Congress to keep alive and working.
> Are you saying JPM going down means there is no alternate provider and everyone defaults in 12 hours
Yes. Look up how tri-party repos work [1] and then look up the Fed’s commentary on their fragility.
One of the three legs is either JPMorgan or BoNY. These are short-term but critical loans. If one of those two fails or goes offline, which practically means failure since others can’t verify the other two legs’ liquidity, large swathes of the rates, credit, futures, regular repo and stock loan markets go offline which quite literally will lead to, maybe not Exxon, but the likes GE having payments failures. You can’t “hot swap” JPMorgan to BoNY after the contract has been initiated. It would be like saying it doesn’t matter if your brokerage fails, just call someone else. Sure, for your next trade that’s fine, but in the meantime the failed broker has your assets.
> Better hope that information doesn’t become public or else they will be prime targets for coercion, bribes, nation state level manipulation, etc.
Friend is a senior IT guy at BoNY. They are regularly in touch with the Fed and FBI. It’s a known vulnerability, and there is constant scholarship and policy work on nationalising or reforming the tri-party repo market. But it’s never failed, and it’s profitable work for the two champions, so for now there are higher priorities for legislators.
Also, there are loads of these centralised pressure points in our system. In every modern financial system. Cede & Co., the Fedwire system, ADP, et cetera.
Yep, repo and swaps are huge, our euro repo market alone is in around quarter of trillion a day..
People don’t understand what happens when some things go down, and how it would affect them.
Futures are critical to you being able to buy milk eggs and bread at the store day after day and not having the price fluctuating on you by 100’s of % or worse those items simply being unavailable because the price either spiked too high or completely crashed so the supply is nonexistent.
Same goes for many other commodities. A cock up in the futures market can and will send ripples through the entire commodities market which will have a direct impact on the world’s supply chain.
> It lasted half an hour, caused a trillion dollars to be wiped off the US economy and was followed by investigations and a congressional hearing.
A bunch of things:
- that wasn’t an outage, that was actual trades taking place at significantly decreases prices. Not even close to the same category. Those same exchanges involved in that crash literally go offline every evening and on holidays/weekends and nobody cares. In fact, if the exchanges had gone completely offline during that time period, it wouldn’t have even made the news because there wouldn’t have been a place for those trades to occur.
- it didn’t cause a trillion dollars to get wiped off the economy any more than my friend transferring a share of Apple to me for $0.50 causes a trillion dollars to get wiped off of the economy.
- nobody gives a shit about a “congressional hearing”. Do you realize how many congressional hearings tech companies have been pulled in front of now? A hearing != action and if you think otherwise, there has been significantly more “congressional action” over tech companies in the last couple of years than financial companies.
> If you think Twitter has a problem with adversarial clients connected to its network I think you don’t understand the modern financial market.
Show me the public IP address of NYSE or whatever major financial exchange you want that I can connect to from any IP in the world. If exchanges were open to the world, they would get DDoSed out of existence in a heartbeat.
Derivatives and futures a lot of them are exclusive contracts not available on other exchanges, if we go down during trading hours we take the market with us.
Where did you get the 100 number from? I think Nintendo is an interesting company for comparison here but I do find it hard to believe they only had 100 employees as recently as 2011.
Lyft had minuscule global presence. Uber has enormous global presence, it is a top 2 player in India , not to mention it's presence in Europe and other parts of the world.
The sheer number of languages they have provide user support and staff for driver management in these two markets alone is enormous.
Lyft has only US and Canada operations. Vast majority of Uber employees will be in customer service and operations. It it no way comparable to GitHub and Nvidia
the two largest employers are both the number one players in their market, being the number 1 player in a market may require that you invest for "first mover advantage". This would lead to a lot of failed/re-tooled initiatives over time.
Being number 2 often just means copying the number 1 player in a more capital efficient manner, product descriptions become "X did it this way, customers want it this way". In a winner takes all market, being the first mover matters a lot.
Also, diminishing returns. In many markets you can "get to the table" with a small crew, but to actually win the customer you'll need bodies almost entirely dedicated to it. And it's that sort of margin, multiplied over and over, that makes one company the undisputed n.1.
Does it have to be one or the other? This is an equally ignorant response. 21k is a lot, and to hand-wave any questioning of this as "ignorant" and "arrogant" is... those things. Lots of companies have bloat, and this is especially apparent if you've also worked somewhere that's competently managed. Recently someone was telling me about a shoe auction company employing hundreds of engineers (sorry, I forget the exact figure, want to say 300-600). I don't care how much traffic you're doing or how many shoes there are, that is a ridiculously unnecessary number.
This is sort-of hand-wavey. The truth is there are many companies with order-of-magnitude too-many employees (and software eats them). And there's some that aren't.
There's no way to know which one uber is without exactly breaking down those numbers a bit more. But call me unconvinced for one.
I don't have the experience to back my opinion, but my gut agrees.
Intuitively, it would seem to me that hiring past a certain threshold increases need to hire more engineers rather than quenching it — that is, the complexity resulting from changes made to the projects to keep such a high number of engineers productive approaches or even outstrips innate complexity and becomes unsustainable, demanding ever more engineers to keep it all afloat. Following this, a company that isn't keeping an eye out for net productive decreases as a result of hiring could end up with far more engineers than they actually need.
But I've always been an individual contributor, so my perspective may be limited.
they also haven't made a single dime in what, 11-ish years? Writing software is fun, but usually companies do it to earn money. If they were building the death star it would be one thing, but they're making a taxi app. The entire company feels like a jobs program for software engineers run by Softbank instead of the government
or you turn out to be the next MoviePass because selling a dollar for 75 cents isn't a business. I would not make the mistake and compare everything that grows fast to Amazon or Netflix. The former has logistic centres and half of the servers that run the internet, the latter a ton of content. It's not clear to me what Uber actually owns. They even sold off their self driving unit.
Don’t get so caught up with what a company “owns”, what matters is what they do. Owning something isn’t a guarantee that it won’t become a worthless asset later.
Also MoviePass was at a good start by selling tickets at a loss, what killed them was that they couldn’t leg into the profit generating phase of their strategy and burnt out.
I have a deep suspicion that this is mostly about risk management. With just 50 employees, what you have is essentially a board of directors that have enormous leverage over you.
> With just 50 employees, what you have is essentially a board of directors that have enormous leverage over you.
How true this is. I've seen what happens when you have a small number of very wealthy engineers in senior leadership roles in a small to midsize tech company that has gone public. They make it as difficult as possible to hire senior people from industry that's needed to fill critical skill gaps within the engineering org. Then all of the talented mid-level folks who have no internal mobility jump ship for Facebook.
But it's not "in reality". It's "with the mindsets and experience and architecture and tooling that are prevalent in the industry"...that's what takes more.
Any given project might not require large numbers people, the headcount driver might the process.
Where you have control, would recommend rethinking the model to look more like Whatsapp than to dismiss as an anomaly.
It would be handy to understand how many of those 21,000 employees are local operations vs centralised ops (like development).
I'm sure it takes more developers to build and support Uber than you might initially guess, but I'm also sure that Uber just has more employees than they need "because they're successful".
And all the software to support and enable users to make use of it... as well as a ton of cutting edge research in the fields of IC design and manufacturing, computer graphics and artificial intelligence.
I would guess a sizable fraction is regional policy, legal and marketing teams. The whole point of Uber is to operate everywhere and you simply can't run all of that in Silicon Valley without access to local knowledge. Not to mention the incredible bloat that comes from building in market-specific features across the whole planet.
WhatsApp with 50 employees is an anomaly. In reality, companies need lots more employees just to get by. Not all industries are equal when it comes to responsibilities and programmers are the most ignorant but arrogant class, thinking that all you need is software. Spolsky covered this 2 decades ago in his “Developer Abstraction Layer” article.