The story is a lot more complicated and nuanced than the headlines you read in publications but I will say that Fast was full of really talented folks who I'd be happy to work with again. For me, a big lesson I took away from this experience is the perils of an overly positive fully remote culture. It's very easy for leadership to hide things from people when information doesn't easily spread across different organizations

I didn't get as much exposure to the rest of the company as parent comment, but I'll say I saw a weird air of complacency and assumed success.

I first ignored my instincts that something was wrong, but as I saw more of how the company operated, I became convinced that the single largest problem the company had was the culture. On the surface, I agree with the parent comment, things were definitely positive, but rather than toxic, I think it was more of a naive one, like most of these people had never worked at a startup before, and didn't understand that it's always easy to snatch defeat from the jaws of victory, and that if they didn't understand what they were doing and how it affected the company then probably nobody else did either. It felt like everyone was very smart, but somewhat foolish (myself included).

That being said, I enjoyed my time there, and definitely learned a lot, especially organizationally. The people are solid and I think Affirm is going to get a pretty sweet deal.

But if you are an early employee with equity, you are not just an engineer. You are, figuratively, a shareholder. One of the reasons they give you stock options as opposed to actual shares is that shareholders have a ton of rights against companies, and can e.g. sue for investor fraud, vote the board out, get quarterly reports with financial & performance statements therein backed by federal law and the threat of investor fraud lawsuits. You, on the other hand, are given the financial exposure to the company's performance without any of the attached rights. And they will happily lie to you in order to keep you happy & get you to project enough positivity to hide flaws & keep outside investment flowing until it all collapses.

So as an employee, you should be trying to claw back some of what being a shareholder would get you, given you are so heavily exposed. I've worked at companies where there is a monthly meeting, nearly everyone present, where the entire company's business position, prospects and challenges are discussed openly. The employees (largely) didn't even have equity. Any cracks (like an imminent complete collapse in the next three days) would show very early on, but there were none, because the employees were empowered to do something about any challenges.

So, yes, there are warning signs, and your response isn't a binary choice between staying and leaving. I would like people to come away with a better model of this than "if they are hiring exclusively from big tech, that's a red flag, get out of there". You can demand more transparency if you know what transparency looks like. If you are content to sit at your desk and happily churn out React components, satisfied with management sending enough emojis at you via slack, then you can't protect yourself at all.

You want to be at the company where the CFO gets up at the monthly all hands and goes over the numbers in gory detail and doesn't hold anything back. You do not want to be at the cargo cult company where they get up and cheerlead and never talk about numbers. It took me a shockingly long time to figure this out.

In the end the stock gains have all been in the periods where I didn't work at startups but instead worked at public companies. Go figure.

"Toxic positivity" and ISOs/rights:

ISOs nominally give you "ownership" in the company, an extra incentive to make sure it succeeds. But you lose them within some time period after you leave (say, are fired), and there's no way to sell them during that time unless the company has gone public. So this thing that's supposed to make you an "owner" just makes you a hostage. It does not motivate you to take risks on behalf of the company, and it does not motivate you to speak the truth.

If they couldn't take it from you, then you'd speak the truth.

Double-trigger RSUs are AFAIK just a way to avoid paying taxes on illiquid assets. The reason to shift from options to RSUs at all is (I think) that once the company's FMV gets high enough, the strike price and taxes due for the options will be so high that employees won't be able to afford paying them if they leave the company.

I thought it was common knowledge that fb started the double trigger thing to avoid this limit, and I heard something similar when a startup I was in converted options to RSUs.

HR would tell us we were extraordinary, we would have world-class catering for lunch every day, and yet we had no customers, no contracts, and no money except the investors'.

It was a sobering realisation on the damage that can happen when you are submerged in toxic positivity. You forget that companies still need to earn actual money, that it's not a party every day.

I'd like to point out that this is not the case for every Silicon Valley start-up: the three start-ups I worked at (CollabNet, Aeluros, Ardatech) were incredibly frugal with money (exception: salaries were competitive).

At one of the companies, the CEO and I took a trip to Safeway to buy soft drinks for customers. When we got back, I walked around the office telling everyone the soft drinks were _not_ for them, only for customers.

At another company, I had to make a compelling presentation to the CTO to buy a $1.5k printer. "The Linux workstations need a PostScript printer!" We bought the printer, but he didn't like parting with the money.

Being frugal doesn't guarantee success: my CollabNet stock was washed out in later rounds, Aeluros had a fair-to-middling exit to Broadcom, and Ardatech had a good exit to Google. But it's good to keep an eye on the money.

The lesson should be buy whatever the hell people want, just stop buying people until you need them. They're the expensive bit.

3 great devs can do about the same work as like 30 average ones. People massively underestimate communication overhead. As soon as you lose the 'startup feel', you lose the speed, and it's gone for good. The transition is probably inevitable, but there's a threshold of revenue you have to reach before you make it. If you don't reach it, you're cactus.

150 devs and 50 sales people to generate $600k yearly revenue is laughable.

Absolutely. We were frugal, but not to the point of self-sabotage. Our biggest expense was payroll, followed by the Cadence/Synopsys EDA software licenses.

So, for example, no catered lunches or breakfasts, but if there was a customer or investor dinner, it would be expensed.

We had good health insurance, but we didn't have gym memberships or (gasp!) an onsite gym.

The workstations were powerful, but we shaved costs by buying "gamer rigs" instead of the more expensive "engineer workstations". We bought prosumer equipment (Netgear) instead of enterprise equipment (Cisco).

If someone needed, say, a Wacom tablet to accommodate a wrist injury, we made that happen.

We had a well-stocked lab, but some of the electrostatic benches we got at a good price secondhand.

Our office was near the train station to accommodate the engineers who commuted from the city, but it wasn't expensive space (all glass & chrome); it was a "B" office. We shared a bathroom with the other tenants. It was a little run-down, but it was clean.

We didn't have offices or cubes, just desks, and the CEO & President sat next to each other in the big room we all shared.

If only they could be bought that easy right when you'd need them. At least in a corporation, the HR will tell you "50 engineers? That's going to take X monts, judging from the hiring rate we had so far."

Fast pitched itself as taking the prudent path compared with e.g. Bolt. The latter went for big clients first. Fast started with small businesses.

There were clearly other issues. But the closing era of cheap capital favoured the bold, and Bolt’s seizure of the fertile low grounds cut off Fast from its growth. Despite Fast taking the orthodox path.

Fast did great job hyping the company on Twitter, likely the founder also was a good storyteller. The old adage is that you raise with a story, team or traction. Sometimes the story actually pans out, like with YC Airbnb is often an example of company that struggled for a long time in the beginning and no-one could really predict how big they could become.

Who? (Also, given an open market, as was the case when the Amazon patent expired in 2017, the company fundraises to scale will beat its more timid competitor. This is basic strategy.)

If you were in a different industry…great execution in the wrong industry is as good as a Ferrari in a swamp.

> all that matters is knowing people

Sounds like you’ve been through some shit. I’m sorry for that.

The point I was making, summarily, is that Bolt ate Fast’s lunch. It went straight for the big customers. Fast wanted to grow towards them incrementally. In this story, if anyone had a fundraising connections advantage, it was Fast—they were Stripe backed. Bolt outmanoeuvred them (and was lucky).

This is unorthodox strategy. Fast played it conservatively. Bolt swung for the fences. Curiously, Bolt landed the home run.

Fast made what was manifestly the wrong decision, both in hindsight and in foresight (I'm happy to post my WhatsApp screencaps from a year and a half ago when I was exploring starting a co in that space). It's not possible to build a hands-off 'platform' like Stripe in that space - integrations are complex, and so integrating a large number of small clients predictably bogged them down till they ran out of money.

(That's not the only reason, I'm sure. It was also a shitty implementation of one-click checkout: most saliently, it wasn't one click. Anyone who actually tried Fast at the time - and I & my prospective cofounder shared the few tiny client websites either of us found like they were rare gems - could see that it was a terrible implementation, in a space where UX is literally the entire game. Many actually did, e.g. this guy from back then: https://chrisfrantz.com/checking-in-on-fast/)

maybe Bolt knew a couple of things Fast did not.

We failed to raise money exactly because we had a bunch of small clients and not enough revenue. It was a prudent decision for the potential investors - but the main failure on our side was not to network enough with our current investors in order to get another friendly round of investment.

We had a competitor that was doing exactly the same thing (with worse metrics) that raised several millions a few months later, so it wasn't too far fetched.

VCs invest in a lot of baseless crap - sometimes the crap turns into gold and sometimes in order to be the chosen crap you need to know the right people.

I detailed a case where a recent deploy I had done had resulted in elevated error rates for some of our clients and how I had solved it with a quick follow up push to prod. The interviewer asked why it wasn't caught in staging and I mentioned that it could have been but as a small company we have to make velocity vs bug tradeoffs and in general it feels like we are currently at a sweet spot. The interviewer was unwilling to explore that space at all, or the concept that different companies might want to land in different places.

Later I was given feedback that I did exceptionally well in the technical interviews but the company didn't want any cowboys on their team. All of that is probably just a long way of confirming the other anecdotes in this thread that from the outside fast seemed somewhere between delivery hostile and overly cautious.

But to be deemed as a cowboy - utterly hilarious, given where they ended up. Pretty cowboy with the investor money instead I reckon?

Fun fact: The 2011 article Cowboys and Pit Crews (https://www.newyorker.com/news/news-desk/cowboys-and-pit-cre...) points out that modern cowboys actually spend a lot of time on communication and checklists.

That’s a gross generalization, but gives you the flavor. I couldn’t do what they do (huge budget, low incrementalism) and would hate the environment. But they can’t do what I do, and when hired to do so, typically fail. Not because they are dumb, or lazy, or foolish, or anything negative — typically they are none of those things. Just haven’t experienced the startup environment.

There's a big difference between the product strategy side (v0 of N different chat products) and the tech side which is that all those products are made of many of the same components arranged differently.

i'd say that if you plan on scaling, you need a bit of both. a lot of tricky scale/process/architecture questions have effectively been "solved" in big orgs, it's useful to have people who can bring this experience

to take a completely random examples: setting up a performance improvement plan, performing incident post-mortems, organising oncall rotas, branching strategies when you have a mix of small customers on standard product and big accounts with customisations etc

Quite likely to happen of the founders / existing hiring managers have no idea themselves how big corp solved these problems.

Meanwhile, good startup engineers are used to learning new technologies quick, can work across the full stack, can own all aspects of their app/feature, work without much direction, never mind a formal spec, and generally are used to solve problems over shipping code.

Often, the key drawback with startup engineers who are available after working at one or two startups, is that they pick bad companies and do not really understand the business end of the equation. However they are still excellent problem solvers.

This is from my experience as a hiring manager who has worked at multiple startups and now works for a large public tech company.

There aren't many good startup engineers with experience at successful startups who are floating around, especially from when the company in question was actually a startup.

edit: i re-read, my comment "failed startup engineer" may have read differently to what i intended. I meant they worked at a failed startup, not that they were a failure. It's largely unfair, but people do seem to think people who worked at a successful company are better than those who worked at a failed company.

For example, a pattern I noticed among the pool of BigTech engineers recruiters would pitch to me was the following. The engineer would join a startup with an immediate promotion in title, and then 18 months after the fact, they would jump back to BigTech to a higher level than the one they had when they left. When you asked what they delivered at the startup, it was clear that they didn't do much or often couldn't correlate the impact of their features to the ROI of the business. Nevertheless, the title upgrade in their resume helped them game the recruiter search and the BigTech ladder.

https://www.teamblind.com/ is full of recommendation for similar strategies that BigTech engineers use to climb the ladder.

Also, the good startup engineers aren't floating around, but you can still poach them. You'll have to give them a big signing bonus to cover the purchase of their options, but it isn't a risk to me when they are sure bets; a recommendation from my network for example.

There seems to be a significant number of tech industry employees still floating around that are convinced that instant financial independence is available for all.

This is a stupid view of motivation and competence.

First, it implies people are only motivated by money. At the first startup I worked at, one of the senior engineers had been through 4 successful exits and was in the business to build things.

Second, competence of the engineering org and individual engineers is orthogonal to successful exits. An engineer can successfully ship many iterations of a product that are just a bad market fit and the company will fail. I’d still take that engineer any day over the typical FAANG cog who’s big project was a 2 year tax billing database migration.

It's a fund raising and public marketing trick.

It means your leadership can't think of a way to make the business work other than to get bigger checks from the same size customer base. "If only all our customers could pay us an extra zero!"

Moving upmarket doesn't work that way. They are totally different companies with purchase requirements and habits that will generate tons of work orthogonal to your core product. Spend a few quarters on compliance accreditations so you can tick boxes, multi-month sales conversations where you aren't even sure if you're just there to pressure the real Belle of their ball, etc.

If you can't do it small, you can't do it.

I've been on companies that did the opposite from going small to big. They started big, very big, with Fortune 50 costumers. That meant they hired a small but very specialized sales team with the right connections very early on.

Because the burn rate was very small (due to needing just a few engineers and employees and a small infrastructure) and the sales cycle took months, we could easily develop a strong product with the requested features, and then it turned out when we moved down to medium and small businesses most of the features were already in place all we needed to focus was on scaling the architecture.

Plus, big costumers pay very good and are loyal as long as you don't cause them headaches.

The only downside of this approach is if we hired the wrong sales people, the company would never take off.

The companies I know that succeeded with that strategy either have great starting relationships with insiders in their first sales, and had contracts signed before committing to a big staff, or got into said companies when they were small: If one of your customers turns into a rocket ship and you are tied to their revenue stream, their growth is your enterprise-sized company.

Enterprise sales might not even be all that profitable when it's all said and done: How many other companies are going to try to bid on their business? With their large size, how good is their leverage to squeeze down the profits of the winning bid.

You might have a relatively unique product, the right connections, and end up being Palantir, but it's a very narrow road with steep cliffs everywhere.

You would have companies where you have existing relationships, they love what you offer, and really want to work with you, but the deal just gets delayed by some other priority or reality of their business. The solution they already have is good enough (even if it’s barely good enough), and sticking with it is not actively putting them out of business for the time being.

It’s just reality… just like VC investing, selling into 100 small companies and growing along with some of them is waaaaay more likely to pay off than landing one whale

I'm currently at a seed-stage startup with a few Fortune 500 companies as users. We're lucky to have a great sales team that can bring that business in. If we're solving a problem for them, they're willing to use us regardless of our size, though they do care about some small particular issues around size.

- It saves all the details needed for 1-click, not only your card (which is just PayPal). So: shipping address, name, possibly even 'pre-done' age or fraud checks, etc.

- It 'maps to' the business's API far better than a browser's autofill can do, which only has some random HTML tags and a helluva lot of guessing. (In other words, users won't have to keep filling in the, like, 4 out of 9 inputs that it didn't understand.) Your advantage is that the merchant itself does the integration work themselves - though ideally your API/SDK will make that as easy as possible.

(Essentially, it's just: "user has a third-party cookie for your site, comparable to SSO, and you charge their saved details and call the merchant's API on the backend [or, for small businesses, a Shopify integration or whatever fresh hell the Fast engineers must've gone through]".)

It's an enormous potential market, though that's also true of the shipping industry and it doesn't guarantee big margins. I'm interested to see what comes of it all. I expect something boring like "Yeah, eventually [company X] got big enough, that's why Apple released the Apple Pay Checkout Fields API for JS".

As a footnote: Amazon has actually offered this as a service for ages, which I know because of the precisely one website I used which happened to offer it. If it weren't for the terrible antediluvian UX and the lack of any evident marketing, it probably wouldn't have been such an open goal for these startups. Here, this is what Fast was trying to sell a shittier version of, without a user base: https://pay.amazon.com/what-is-amazon-pay

The reason this is possible for us is because our tech isn’t mission critical for them.

I doubt the CEOs/CXOs of any of these companies even know we exist.

We’ve just been fortunate that a lot of ops teams in these companies need our thing.

So - that has really changed my paradigm of enterprise sales.

Sometimes, if you’re SAP, you need to get whole of company buy-in. But if you’re SurveyMonkey, you don’t

It definitely is hard to have conversations about what isn't going well, even though it doesn't have to be negative or gloomy. Often the disconnect in results is chalked up to external factors outside of your control, which is sort of depressing because if that is true then you can't do anything about it.

It's like when someone encounters a bug in the stuff they wrote and look every which way they can to blame it on the library, framework, compiler, etc.. - I'd rather have the bug be in code I wrote because then I can fix it easily. You can show someone simple data that shows a strategy isn't working, and then they'll spend a week torturing the data to come up with a contrived counter-explanation.

Ultimately, it is tough because part of doing a startup is remaining positive even in situations where it isn't warranted and there are times where the strategy may not be working out in the short term but needs more time to play out. Being unwilling to ever even entertain the idea that what you're doing isn't working seems like it might be problematic though.

As for theranos and wework - i don't mean those kinds of things. I mean standard non-fraud startups. The always positive culture is there, and at least for folks who take some pride in being rational, it is quite demoralizing.

Fraud is not black and white. "Toxic positivity" can enable shenanigans or conceal problems in an otherwise non-fraudulent startup.

here we go, this is late 90s all over again

The reason late 90's start-ups invested the way they did is because they had no meaningful way to track the effectiveness of their ads, much less the payback. A Super Bowl ad for Pets.com seems fine if you can (more or less) guess that it adds a bunch of customers.

Since then, start-ups have built a bunch of tools to track customer acquisition costs and paybacks in a pretty sophisticated way, and some of the obviously crazier ads dies (or at least become more one-off).

But now we are in a fintech/crypto era and nobody really knows how to calculate paybacks once again, because the business model is so new vs. a traditional ads or commerce business (will this new crypto.com user do $1k of transaction volume every year for the next ten? sure why not?)

As a result of all this, we see crazy 'spaghetti on the wall' style advertising again, usually focused in the crypto/fintech space (See: this year's Super Bowl). I wonder when the music stops.

Remember beanie babies? We call them NFTs today.

Remember HYIP and e-gold? We call them cryptocurrencies now.

The point is that crowd behavior, especially markets driven by envy & greed, lead to the same outcome. every. single. time.

this is why guys like Warren Buffett and Charlie Munger keep making money.

Who cares? Champagne on the Concord, and let's order a bunch of Starfire servers from Sun Microsystems!

I wonder if any doomed startups will order a rack from Oxide Computer. I hope Oxide's primary customer base is established companies that actually need serious efficiency at scale.

At least publicly, Fast never addressed this, and insisted on requiring the user to log into Fast to make a payment even when it was a worse option.

Browsers like Safari are explicitly designed to prevent people from remaining logged into third party sites like Fast. All this was just brushed under the rug and instead it was just a bunch bluster on how Fast was a revolution.

At least Bolt allows payment via ApplePay, and seems more about doing whatever it takes to maximize each seller’s conversion, rather than having an ulterior motive to push its own brand everywhere.

The two of us ditched it when we realised how inhospitable the economics were, but even then we knew Fast's shitty play was bound to fail, and I heard the news from my prospective cofounder sending me a text ("we called it!") when it happened...

You could put that as subtext for pretty much every story you read. ;-)

> Engineers calculated the load Fast had in needing to serve their traffic. The Fast button was rendered less than 500,000 times per day - rarely needed to ever serve more than a few requests per second.

> One of the few warning signs engineers noticed is how Fast spent far more on infrastructure than the scale of the operation would have called for. Engineers sometimes brought up suggestions to scale infra down, and save costs - given there was not much revenue generated.

Sounds like the whole thing could have run on a single cheap VM, perhaps with a second one for redundancy.

I did an interview screen with them at the end of last year for a data engineering position (former coworker was high up on the design side of things and he gave a referral). The first bit of the technical question was just "write a SQL query" and boiled down to doing a group by, count, and limit. The second was something like "what if we needed to return this query from a distributed system without going to the database?" without much context.

As any sane interviewer would do, I asked for context around scale, infrastructure, and what "without going to the database" even meant. The interviewer just reiterated the question. I started talking about different approaches depending on context and all the interviewer did was tell me to program a solution. Turns out all he wanted was a time-windowed hash-map to cache values and the fact that it's a distributed system didn't matter at all.

I think that just about sums up their approach to engineering solutions.

https://github.com/google/guava/wiki/CachesExplained

This is often the problem when hiring too fast. New employees don't have context or direction but generally people try to be productive, so they start inventing work.

Management team is new too, and not built up either direct or handle the bandwidth and they also are lacking the direction. Senior leadership and founders are likely busy hiring more folks and potentially trying to sort through the chaos described before.

IMO, pre-product market fit company shouldn't have 500 employees, maybe not even 10. These things don't become easier with more people, usually everything gets harder and slow. You also shouldn't have massive marketing or sales force before you actually have product to sell and have figured out your sales motion.

It seemed that Fast was just trying to shortcut their way to be the next Stripe by hiring people instead of actually working on the fundamentals like the product and business.

If you have a 500 person org that has operated for years you can probably add 10%-20% new people each year without it breaking too bad. But it can be very hard to build a complete function from 0 to 100 people within a year because there is no infra, culture or understanding how things work. The new people cannot be absorbed/aligned because there is no central gravity so people easily end up pulling different directions and spending their time on internal coordination vs actual useful output.

The parent mentioned 5 teams instead of 2. That's director or higher level decisioning. Which is where you frequently end up having someone far enough away from the actual work that needs to be done, starting early to come up with a roadmap. Given fixed deadline and fixed scope, you lean on your PM triangle knowledge and try to get as much headcount as possible to ensure success, because you don't have enough knowledge yet to really decide at what point adding headcount leads to a negative gain. It's usually less about empire building and more about trying to plan things early rather than growing in response to need.

I've been in this situation, as a manager, being told to hire three teams (two being teams of contractors), pushed back to say I should hire only one (the FTE team) as we didn't have enough work for them, was rebuffed and told I needed to hire (the roadmap said we had the work!), hired, and then had leadership panicking about the fact the teams were idle with nothing to do. After a couple months had enough work we could split it (unnecessarily finely) between them so they could all look active, which lasted about a month before the FTE team just started doing it all, as that was more efficient. I didn't stay there long.

This may be true about Amazon but it is not a given in general. Managers can operate in the grey area where they deliver just enough to justify more headcount "in order to deliver more" where the underlying motivation is increasing headcount. Not atypical in political environments like banks etc.

For your huge hires, 1/3 of the people were hired to be fired in the stack ranking, which of course is a huge morale hit.

The ones that aren't setup to fail lose 20-50% of productivity doing machiavellian backstabbing, defensively or offensively, for promotions other culturally ingrained/highly encouraged organizational sociopathy.

With 50% turnover per year (IT stats, not warehouse worker stats), it'll be hard to sustain development as well.

So it makes sense that Amazon would overhire.

Get out as soon as you can.

Heavy stack ranking basically means for a life event, a worker is very likely to get axed. Any health event, having a child, partner has hard times, or management reorg that puts you with a boss that doesn't like you, and the writing is on the wall.

Companies with a good long term focus should concentrate on acquiring and developing long term workers.

Amazon in particular structures its pay (payoffs come from bonuses and vesting 3-4 years in) in such a way that if one takes a perverse sociopathic view of how they manage workers, that management is incented to rug-pull workers three-four years in so the vesting doesn't happen.

Amazon is DEFINITELY sociopathically and perversely managed to do this.

Once that fundamental disdain of your work force becomes ingrained, then all other organizational abuses (stack ranking, hire for fire, backstabbing) is all fair game.

It starts with stack ranking and the fundamental management sociopathy. Which anyone that looks into accounts of Jeff Bezos becomes apparent of the source.

So why has Amazon been successful? Amazon's advantage in the overall corporate competition landscape was simply that they built an integrated business+IT strategy while every other company treated IT as a cost and annoyance.

But they treat their people like utter shit, and as Amazon transitions from an explosive growth company to a more sustained presence, and from AWS from explosive growth to "utility", the now-huge middle management and workforce devolves from stack ranking and sociopathic disdain into a cesspool where only the wicked survive.

https://en.wikipedia.org/wiki/Toxic_workplace

Unfortunately, Amazon continues to ride a growth momentum, so the toxic workplace will be viewed as a positive. Nothing will change until serious incidents occur.

Of course Amazon is huge, so there are workers who may be in less troubled areas. Just keep in mind that toxic areas of companies that derive from corporate practices will inevitably spill over to the "good" parts. Amazon is acquiring massive amounts of pure sociopaths in their management culture, and those sociopaths will "eat" other idealistic parts of the corporate management.

Corporate america is adapting. Better integration of IT and management is going to come to Amazon's competitors. Amazon's apathy to its product fraud and fake reviews, now stretching into a three year ongoing problem shows that it is faltering. Its lies during its AWS downtimes show a blame culture in action.

My company's strategy is get one extremely solid person across all our verticals (Android / iOS / Backend / Infra) to minimise communication overhead and maximise iteration speed (since we throw out half the code we write anyway).

I would be very hard pressed to hire more than 2-3 people in each role even if we were offered 10s of millions.

but ... but ... that makes sense ...

I have found that having fewer good engineers is much better than lots of bad ones.

That approach is treated as heresy, in today's tech industry.

I kept engineers (really good ones) for decades. They had many "life problems" (like divorce, cancer, etc.) during that time, and I kept them on.

It's entirely possible. I have done it. I ran a team that was all "top-shelfers."

Depending on key roles is something that every company in history has had to do. It’s a risk, but one that has been paying off for hundreds of years, for millions of companies.

There are ways to reduce “bus factor,” yet still maintain high Quality work. The corporation I worked for, managed this, but the price is extreme levels of overhead and rigidity. That has its own risks. I was treated as a “cowboy” in our company, and was considered to be borderline “reckless.” Most folks here, would have considered me to be destructively conservative and risk-averse. I got the job done, though. Our team consistently delivered high-Quality solutions to difficult problems, for decades.

The “easy answer” is to keep expectations and demands on labor low. Keep quality to a minimum. Rely on “black box” dependencies. Don’t use advanced development techniques. Squeeze your coders like lemons. Treat every employee the same as every other one, and control for the lowest common denominator.

That’s not how I worked.

It's OK. The industry is safe from me, and my radical views. It was made abundantly clear, years ago, that no one wants people like me in their company, so I have been forced to work with people that can't afford even crappy programmers.

Poor bastards are living the nightmare.

I feel as if I was a very “human” manager, and that seemed to make the difference.

My employees were loyal to me, not the company. Not an ideal situation, but it WFM.

It has changed a bit, but a large part of it is due to HR policies around salary bands. You can have one person do 10x or 5x and another do 1/3x but their salaries are often max .5 factor apart. Ive been in situations where i'd rather reward (perhaps with a vest period) one person with 3x the salary and just skip all the inter-person communications bs, but HR makes it hard.

We were able to do this at hedge funds, and i'm sure small startups can do this, but as companies grow, HR will generally not allow this.

In the 90's there was a joke that they'd take the engineering headcount and multiply it by 10M (or something), subtract a multiple of management and there's your valuation.

The "shiny toys" in this case are employees and the task of managing them.

As a (future?) lead/manager, having more people on your team bolsters your resume. A lot of people would benefit from the company hiring more too as it generates demand in other parts of the business - HR, IT support, etc (which in turn causes more hiring if you need more people to deal with the increased workload). As a higher-up in the company, it looks better on your resume if it was a "big" company with 100+ employees rather than a scrappy garage startup with 3 people, and likewise for investors.

Basically, to move development velocity faster we "load everything". Seems like a really bad idea on the surface. In practice, we only ran into an issue with a single customer that was literally 1000x as large as the other customers. It took about two weeks by 1 engineer to rework a select few calls that were egregiously slow and we were back at it.

----

In other words, even with some arguably inefficient technical decisions, I've rarely if ever seen performance issues at early stage startups.

If you can make the problem go away by spending more in servers always do it.

For startups, the life and death is product market fit and you only get there by iterating on the business as many times as possible, not making something more efficient.

Put engineers on business iterations.

Massive servers are laughably cheap these days. E.g. 16 cores, 128GB RAM is €112 from Hetzner.

In the meantime have you looked at leaseweb?

https://www.leaseweb.com/dedicated-servers#US

I'm planning on running nimbus web services[0] there and there are a few other smaller providers I have earmarked depending on how brave you are:

https://www.defendhosting.com/usa-unmanaged-servers/

https://cc.delimiter.com/cart/dedicated-servers/

https://greenserver.io

[0]: https://nimbusws.com

Spending time on needless infra (scale you don't need, AI you don't need, infra you don't need, things could have been postgres as in this case, ...) both prevents building useful stuff ($-generating features/experiences/...) and increases operational cost (maintenance, debugging, ...). Both the lost revenue growth and the increased costs are compounding. Imagine a maniac steadily piling on a bit more technical debt every day and eat a bit more of the seed corn every day instead of growing it.

A good phrase for this is "playing house": http://www.paulgraham.com/before.html

More like, sounds like management was trying to get as much money as quickly as they could from their VCs before it all imploded. The engineers aren’t at fault that management didn’t have a real product or vision.

I mean, the engineers held up their side of the bargain. They were hired for webscale, the company got webscale…

Now, if this was a more strategic startup, that had tried to hire pragmatic engineers, that insisted on webscale, I would blame the engineers. However, in this case, I think it’s pretty clear that the engineers did exactly what they were hired to do.

It's reasonable to argue that shelling out the US$800 for a server with 128 gibibytes of RAM (or renting one from a cloud provider for US$150 a month) amounts to buying nifty toys for the engineering department. But if you're paying each engineer US$200k per year in salary and a similar amount in benefits, that's US$190 an hour, so each engineer's workday buys you almost four such servers. Your RAM has a minimum transaction time of about 100 ns, while an SSD is more like 10000 ns. If you have 100+ GiB of data to query, the alternative to buying nifty toys is paying the engineering department to do optimization work that wouldn't be necessary if your data was stored in memory that was 100x faster.

The rough equivalency here is that adding one more engineer to your team costs as much as adding 16 dedicated Xeon servers at Hetzner with 128 GiB of RAM each.

Sometimes the best answer is still a relational database, of course! It's often easier to write your queries in relational terms than by looping over hash tables, though probably not if you're using an ORM, and Postgres is delighted to cache all your data in RAM.

At some point, you may grow beyond being able to run your production environment on a single server, at which point splitting things into tiers is worthwhile. (In some cases you start at that point, for whatever reason, but a lot of web services can serve a million users on a single server. It sure sounds like Fast's service would have worked fine on a single server.)

I've done something similar before and it saved countless hours wasted on configuring databases, writing queries, fighting impedance mismatches. Postgres brings a lot of work. With a file you just need a couple of lines to write your memory out to a file and a couple more to read it back. Simple and effective.

Until it stops scaling, but at that point you've established that your business is successful enough that it warrants the effort of doing something shiny. Postgres has a place in this world, but no need to put the cart before the horse.

Note that this basically works 99% of the time now and is part of a learning process that had the whole thing taking 2 - 5 hours every day for several weeks, down to 2 hours a day for several weeks after that, and now down to the nearly inescapable 20 - 40 minutes a day I have now.

Yeah, not doing that would be my own form of playing with nifty toys.

Programmers love to add the latest tech to their resume then move on to a new position within ~18 months

If/when a developer builds deep knowledge of a problem domain, things like framework-of-the-month may become an unwelcome distraction.

Well the best way to reap the benefits of your experience is to run your own company. Not screwing yourself over with BS tech is a competitive market advantage. However you might need to hire people so you can’t pick something too old.

Working for other people it’ll mostly net you less on call headaches.

I've seen a few startups that seemed to overengineer things (or build custom solutions rather than use something off the shelf) in expectation of massive growth.

I also thought it might have been due to resume driven development and the need to keep engineers engaged so they didn't leave.

I've also seen successful startups with a monolithic rails or PHP application that ran on heroku with next to zero custom architecture components.

My experience has been that companies who will customize integrations for you have poorly designed products. By that, I mean, when we ask for a feature, and they bring on an engineer who does some requirements gathering then comes back with some invisible solution. That's a major redflag.

The success of an integration, in my experience, is directly proportional to the amount of the work I'm able to handle as a client.

I'm feeling a bit called out right now (not an owner but that's my experience, too)

I've seen this repeated on HN, but never saw the alleged flak.

They went from a JSON file, to etcd, to SQLite. etcd seems a little misplaced, but presumably it was already in their infrastructure and they thought they could save time leveraging it. The file-based approach seems appropriate for their particular use-case, though. It's not like it's a Rails app.

I’m still of the opinion that whatever you are storing, if your alternatives are JSON or Sqlite, etcd is a really strange/unconventional choice.

If your boss's boss or peers signed a contract for $1m a year from Oracle, you're going to use Oracle even if Postgres is a better product. And the only way to use Postgres in this situation is if they don't know you're running it. Which puts you in the uncomfortable situation of wanting to say, "But Oracle sucks, look how successful we've been at using Postgres?".

Basically if you don't already know that someone higher up the food chain is comfortable with saying, "I've made a huge mistake", you aren't going to 'help' them learn to do it by pointing out they fucked up. What they're going to hear is that you're asking them to say, "I just wasted a million dollars of company money on something everybody hates," which not only are they not going to do, but they're going to start thinking about how to shoot the messenger.

As many conversations go in a couple of my hobbies and talking to friends and acquaintances about health concerns, the answer often starts with, "first, get a time machine" and ends with, "or learn to live with it," often with some useful compromises in between.

One of the important things to do early in your tenure at a place is to bid various people in the management chain and those who seem to be on a management track to see how open they are to having their minds changed. Because you want to know this when the stakes are low (also the blowback on low stakes things seems to be less problematic). Then you know if or when a boondoggle is in the making whether you can stop it before it starts, or amplify all of the problems and kill it along the way, or whether you should keep your head down and conspire to keep the 'doggle at arm's length with those who agree with you.

What you'd do in that situation is make the best of that $1M contract you're stuck with anyway, while being flexible enough to migrate away from that solution when it actually makes sense to do so. Just because you've made a huge mistake doesn't mean that making a different choice after-the-fact is more sensible. Hindsight is always 20/20.

Otherwise it’s just a sunk cost fallacy.

I absolutely hate waste and was always proud of myself when I found ways to shave a thousand here and there off our AWS bill, but it was a bit depressing when I went through the thought exercise of "What would it be like for the company if I managed to get everything onto the AWS free tier?" And then compared that to what it would be like for the company if we found a way to make 1 or 2% more in revenue.

Early on, the math heavily favors worrying less about infra spend and more about product enhancements that can build revenue.

(Although often reducing the size of your infra for other reasons - usually around complexity for ops or ease of development can also have the side benefit of reducing cost and it seems worthwhile in those cases.)

YES! This is VERY true

I'm curious to learn why, if there's any more you can share. What about vertical scaling, i.e. bigger machines instead of more of them? Can't that go pretty far?

For everyone that will say "well why..."

1. real products need a generally available shipping unit (SKU) when your customers are enterprise / telco types.

2. for item 1, what is the lowest common thing this type of customer would take - a VM they can run on their existing virtualization setups. the reality of K8s, containers in this type of customers at that point was near zero.

3. there was thoughts about a SKU of a HW appliance, ie, beefy server and just toss CPU/RAM in it and ship that. shipping some stupid HW appliance that we controlled would just be a distraction and a waste of money also requiring 2 shipping artifacts, 2 test setups, etc. not to mention the need to sort for support for HW failure (you can contract with Dell for example to have them label their Kit as your appliance).

I don't understand the issues with using a bigger VM, but I have no reason to doubt you.

So in my now close to 30 years of experience later, working on systems expected to run 24/7/365, I can't tell you what percentage of failures and outages were caused by the redundancy/fail-over/etc software and hardware layered into a system, but its definitely a very significant part of the problem. Frequently because its just that a "layer" someone bolted on, even if that layer was some software engineer designing for "web scale". All the edge cases are frequently not completely thought out, and when you hit one its a lot harder to recover, than simply restarting a single service running on a single server.

I’ve had massive problems with fancy (HPE, IIRC, but it wasn’t called HPE then) raid arrays caused by the controller being a POS. Using plain mdraid would have been far more reliable.

Sometimes I wonder if all the effort people put into protocols that run Paxos and Raft could be better spent running a small number of coordinator servers with manual failover. I’m suspicious that, in many use cases, leader elections cause failures more often than the leaders themselves fail.

Sounds like someone didn't understand redundancy...

There has to be a smart reply to that, like: 100% of pilots of single engine planes don’t make it home after a single engine failure. 100% of pilots of dual engine planes do make it home after a single engine failure.

pushes glasses

So the odds are you don't. Keep it very simple, very small. I can't highlight this enough. A few boxes will get you very far. If you ever actually hit the limits, throw a big party to celebrate success and only then start to think about making your infrastructure a bit (just a bit) more complex.

> [sales] signed up a large number of smaller businesses on the platform. [...] However, integrating these smaller businesses was challenging thanks to several customizations needed for each new customer.

The ratio of revenue per each small customer vs the total cost of integration (and very likely on-going maintenance) was probably at least an amber flag somewhere for those who had visibility of it.

But maybe there was on-going hope that they would be able to sign up a big customer and integrate them before they ran out of money?

At the last company I worked at (won't call them a startup, just a 10-year old small business that somehow kept raising funding but has never made a profit) I was astonished to learn from a new CPO that in our tenth year we didn't know if we were making money from a customer.

We had no idea how individual customers used the application, which was very data intensive, or what that was costing us. The information was there, but no-one ever looked. Every sale was considered a success, even if it turned out to cost us 3x in data costs and customer support. Often customer support alone would make a sale into a loss.

We spent a relatively large portion of our revenue running very expensive servers to respond instantly to searches that no-one ever performed. I created detailed monitoring to show this. I talked about them to everyone, including Product and the CEO. No-one disputed those facts. But instead we had several people essentially dedicated to downsizing application servers and deleting unused S3 buckets, trimming three zeros a month when we could have trimmed five.

I have learned not to assume that warnings are visible, that people pay attention to them if they are visible, or that anyone cares.

They're still circling the drain, still raising money (somehow!) and I doubt anything's changed.

The last 7 years of my life as an SRE. It's maddening.

It was the pet project of another team-lead, that no-one else wanted. Group decision making is bizarre.

Cost can be a difficult problem to solve! I've been working on a project for about six months trying to identify just how much it costs to deliver a unit of the thing we sell. There's just so much variability, one customer might have widgets that are 40kb in size and rarely ever run widget analytics workloads, while another customer has 40mb widgets and can't stop looking at them.

I feel for the whole "how do customers use the application" thing too. Product analytics at most places seems to be a concern long after features are developed.

"So how many customers have been using that feature we spent two years and $20MM developing?"

"Good question."

"Soooo, mywittyname, can you figure out how many people at companies with over 200 seats look at dog pictures after opening an email with a 'C' in the title?"

I think what saved us was the unlimited license we had negotiated for the database server we were using (which wasn’t bad technology), expired and then the company was going to charge us through the nose for any new servers.

That triggered a rearchitecture of our system, into micro services that made sense for the kinds of queries, volume, and load we were dealing with. We did have a lot of data, but many of the queries could be satisfied by a key value store, for example, which didn’t require the same amount of hardware resources.

The original project did lead us to products our customers wanted to buy. Then we just had to change the architecture to fit those actual products.

Places I've worked yearly contracts for B2B have been in the $100k-$5M/yr range.

The customers paying $100k/yr barely get any integrations and feature changes... they go for the ride. The customers paying $1M+/yr get features on the double.

If Fast was doing integrations and customization for customers that were paying $1000 or less a year something was very very wrong.

Unfortunately this can lead to the outcome that the company just loses more money.

from the article: "The average volume of Fast customers was low because most customers were from small businesses. And yet, every small business needed custom engineering work to be done, making integration slow. Several engineers mentioned how they did not understand how spending lots of engineering effort for each small client resulting in little revenue would result in building a company that could be worth $12B one day."

I think their primary problem was a huge valuation—with correspondingly huge expectations—but no real revenue. If you're pulling in $600K revenue, but valued at $500M... you're gonna have a bad time.

You're describing most startups here.

The real thing here is they're not that surprising. A lot of companies with their kind of funding use enterprise sales to force say $5-10M ARR, but there's only so much VC money can force for a leaky funnel, broken product, and overall incorrect market + fit. I didn't appreciate this until maybe a year or two ago. Valuation multiples in 50-100X range are super common (and even wackier numbers in seed/a). Think make believe stories like "well with another 12-18mo of growth this really just a bit over a 30X on some future forward revenue multiple...". The cash almost always leads to overspending, and it's highly unlikely the next 2-3 raises won't blow up and everyone goes home. I'm actually super impressed by the Docker team because they've been one of those rare cases of crawling out of that trap, even if with a lot less of the team.

We get job candidates with high competing offers for companies I know to be rotten inside, yet there's only so much I can say. "Our new hires are getting paid from customer revenue and with equity that doesn't have $50M-$500M of investor thumbs on the scales already cutting you out in 95% of the likely scenarios" generally doesn't punch through the kool aid.

I fell for this trick once when I was younger, but never again.

As it turns out, when companies are bleeding talent and struggling to hire they suddenly find ways to pay well above market rate. You can have a fancy title, too!

We had a competitor do something similar at a prior company. I would tell candidates that we can't match their compensation numbers but to come back if things don't work out at the other company. I'd often get a message about 3-4 months later asking if the job was still available

It's hard to tell the difference between "stop the bleeding" title/pay inflation and "aggressively competing for talent" inflation.

Meta is ~$2M/employee. Probably more like $1M once contractors are accounted for? But for a good person, even if the rest of the company has random issues, they can carve out a win.

Fast was like $1K/employee

Consider what that's like for some hypothetical B2B startup BlitzScaler Inc. They're betting on 3x+ growth year-over-year for next 2+ years, 2-3 year sales payback periods, etc, and is probably closer to Fast levels of revenue per employee than Meta's. But hitting $500K is hard, $2M is hard again in different ways, then again to $10M, and again to $20M+ (and increasing range depending if many sales people inflating it). That matters because many aren't there, even unicorns (!). That means, like Uber, they're burning increasing piles of money, except unlike Uber, the revenue is unlikely to be keeping pace. Fundraising buys team, and an artificial sense of fit+growth (kool-aid-drinking team + forced sales that tap out + churn.)

A cockroach team is closer to Amazon style: reinvest ~100% in growth, but don't gamble your employee's stock options.

What could be a legitimate reason for giving silly titles? If it were a startup, it would be a huge red flag for an auditor.

> Sure, come as a Senior! That's not enough? Principal it is!

Do you mean staff (E6, the level above Senior)? Also, no one is getting E6 offers 2 years out of college.

IME, F are the only ones to whip out their laptop at a house party or bar at 10PM on a Friday night because they felt an insatiable to work on some feature request; GMA may do the same when on-call, but I've never seen it otherwise.

I'm personally going the other route: Class of 2019, secured a decently sized FAANG bag (barely worked the whole time while doing it), and now I'm transitioning to a position with even fewer hours of commitment required but also lower salary. Going to focus on things I've come to find more important than money and code.

On my way out of the first startup I worked for, there was an offer to double my pay! Nope!

Don't know if that's the origin, but hey it could be. Without the story, calling someone "cockroach" has unclear sentiments.

In contrast, popular models prone to failure and largely fueled by market phenomena like low inflation rates include ad-based social media ("we'll turn on revenue in 5 years, no, really!"), blitzscaling ("we'll eventually figure it out!"), sales-driven growth ("our federal team will make our $ back in 3-5 years, no really!"), acquihires ("with enough AI talent we'll just sell our staff!").

I like venture capital, such as for deep tech and true growth, but the reality is most tech VC funding is more about financial engineering that assumes most of the portfolio fails. They push for commercial scaling too early and wipe out, which is fine for the rich VC who just needs 1-3 bets to win. Fine for them, but sucks for most founders & employees unless they job hop every 2 years. If we take VC $ again, it'd be on our terms and with a clear payback/growth return.

A great way to kill an otherwise great idea & company is putting VC $ in, which puts a company on overdrive and financially implodes before it has a chance to figure things out. They're addicted and likely can't stop relying on VC without layoffs (or less likely, succeeding), at which point many of the A players leave as well and hard to recover. Instead, if a cockroach raises > $2M (so commercialization phase), that should generate enough sales revenue that they can keep organically growing in 18-24mo later even if a follow-on round doesn't happen. You'll hear such cockroach founders saying they don't even touch the money for months. Numbers-wise, most Series A companies fizzle out, which is because of this unicorn-or-bust structure.. and especially whenever the market is not on a bull tear. With a bit more time, they probably could have figured it out... but they blew it.

Some articles: - Paul Graham's article on this: http://www.paulgraham.com/badeconomy.html . - https://medium.com/swlh/how-to-build-a-cockroach-startup-ins... - https://medium.com/the-mission/be-a-cockroach-not-a-unicorn-...

is this a reference to http://paulgraham.com/guidetoinvestors.html#:~:text=nuclear%...

I think I know what you mean by this, but I'm not sure. Could you elaborate on what this means? Thanks in advance. :)

Money is fungible

> Fast did less than $300K worth of sales and below $6K in revenue on most days from January 2022 to April 2022. There were days with around $2,000 in revenue for Fast.

I'm actually surprised that everyone was receiving daily updates of company revenue.

If you're surrounded by 100s of people at a company known to give high base pay but you're seeing daily revenue numbers in the range of $1K to $6K (they had $600K total revenue in 2021, supposedly) then you have to know that your time is very limited.

I assume they were led to believe that more investment money was just around the corner to keep the business going? With a $10 million monthly burn rate they would have needed a staggering amount of capital to just continue to exist, let alone execute any plans to turn the ship around.