Something I learned relatively late in my career was that really impressive work is the result of iterations in tiny steps. When you see something that makes you go "WOW! I could never do that!" you're seeing the final result of a long journey, and you probably could have taken each small step with a bit of effort.

There's no need for a special title for people who make amazing things. They're not that different to the rest of us. The main thing they have going for them is an environment that lets them shine - time to research and explore, less pressure to do things that aren't moving them towards the end result we see, and fewer externalities that make them lose focus.

I'm not suggesting just anyone could make Linux or Roller Coaster Tycoon, but I am definitely of the opinion that given then right circumstances most people could produce something that other people believe is at that sort of level of complexity.

May we all find those jobs one day.

Its stamina.

No there isn't some higher level of understanding or deep perception. Every one of the all time great programmers had 1 exact thing in common with no exceptions. Stamina. You will not find a single outlier.

They did their craft every day, for many hours, year after year.

So sure, any of us could have taken the individual small steps. But when you look at a whole body of work, you'll see they're cranking through these "small steps" like the Juggernaut running through walls for decades

From "Courage to Create" by Rollo May: "It is said that the novelist Thomas Wolfe, who was one of the highly creative figures of the American scene, that he was a "genius without talent." But he was so creative because he threw himself so completely in the material and the challenge of saying it-he was great because of the intensity of his encounter"

Apologies going philosophical, its the only way to debate such an answer-science has no real answers for this title we call "staff engineer"

- Will Smith

My take is that talent is a potential, without practice it is left unrealised and never turns into skill, nor success.

Then people look at success and say "well person P has natural talent, I don't, so I can't do that", which is very sad, because ultimately it's mostly about a shit ton of work, and dedication, and commitment, and persistence.

It is possible to "press on" in the wrong direction.

Kind of like how virtually all great cooking requires salt, but use of salt goes not guarantee a great dish.

I work in a relatively 'flat hierarchy' so there are few organisational structures and proceedings that keep projects on track. It's only me that keeps the marble on the track, there are no organisational tracks. Once I lose interest or stamina the marble is gone.

Other organisations with regular check-ins, project tracking etc. have a system that at least keeps the marble in place.

So when you’re looking for that role after you think you’ve got that stamina/perseverance/commitment, do remember that maybe you still have to find that right place to work at.

I've seen this happen so many times and every time, I'm surprised that more businesses don't spin off cool internal projects like this into startups. Hashicorp's market cap is roughly 10% of the market cap of the megacorp I used to work for. And right now, my team is working on an internal project to fill the gaps left by various authentication providers we use. But instead of spinning this off into it's own business, they are just kind of waiting around for someone else to develop a better product so they can buy it instead.

It was explained to me that investors generally don't like spinning off internal projects because it can be a distraction, but it does feel like investors are leaving a billion+ dollars on the table. Especially given the absurd P/E multiples software startups enjoy.

To circle this comment back, I think a lot of staff engineers are working on the next generation of cool technology right now. But they might be working at boring old companies, like a massive retailer, and eventually that internal project will get replaced by one built by the engineer who decides to venture on their own and replicate what they were building as a stand-alone product.

If they go off using that money to spin up side-hustles selling engineering tools, they're taking risks with someone else's money to do something they weren't told to do.

Truly dispiriting to be unable to use JSON or Protobuf, because some executive director claims his “standard” is better when it is demonstrably not so. In the end it breeds a culture of mediocrity — management has made it clear that engineering excellence is not a priority. So support the business use case, but do whatever …

> built by the engineer who decides to venture on their own and replicate what they were building as a stand-alone product.

We need to change our contracts.

Why did Terraform end up being the dominant product? Product-market fit. That’s not just engineering, but bringing to market a product at the time when people need and understand it. This includes marketing and educating. Also, Terraform has an ecosystem of prebuilt providers. NPM as a package manager is not as well designed as other systems, but it’s arguably been the most successful. While the internal prop tech you mentioned may be more advanced, it’s not obvious you had the right product people with vision to build out that ecosystem. Most managers and product people at large companies aren’t tech savvy beyond buzzwords

I agree 100%

> I am definitely of the opinion that given then right circumstances most people could produce something that other people believe is at that sort of level of complexity.

This is the "if my grandma had four wheels, she would be a truck" argument and in my experience it is not accurate. A quote from Ratatouille puts it very eloquently: "A great chef could be anyone, but anyone could not be a great chef"

> They're not that different to the rest of us

I beg to differ. Again, with the caveat of "in my experience", they are very different. Maybe not in ways that would make a great Forbes article, like having superhuman IQ, but for example "having grit" or simply "ability to obsess over the right things" is a key ingredient that most people don't have enough of.

I think you are right. Really impressive work is generally the result of a longer journey than you realize. The time and effort put into it is probably underestimated. But my mind immediately tries to think of exceptions (which totally prove the rule). I'm curious if others would agree with this one: Sometimes there are just really great ideas.

Minecraft comes to mind. And I don't mean today's Minecraft. I mean the alpha and beta versions which were wildly popular and essentially just an engine and sandbox world with a tiny fraction of the features today's game has. People have recreated that core game many times with fairly trivial code, which is why I attribute this more to "great idea" than long iterative process that most people cant replicate. And a totally anecdotally and subjective reason (although I've heard others express it too). I found the game immediately nostalgic. It was new but it felt old - like it had existed in some form before but had just now taken shape. That to me seems like a great idea.

Ironically, I find myself questioning if I underestimated how long Notch took to create it. A cursory search says a week for the initial version, but this timeline shows more like 1 to 1.5 years from nothing to alpha: https://minecraft-timeline.github.io/ Even still, the fact that the core game can be recreated without too much effort does suggest that that it was more about being a good idea than being difficult to implement. I don't look at Minecraft and think "wow I could never do this." Instead I look at it and think, "Wow. That was a really great idea."

Although I suppose you could says that great ideas are an iterative process too. But at that point you have to decouple "really impressive stuff" from "I could never do this".

I mean, it had. It was called Infiniminer. Admittedly it had only really existed for about a month before, but it was obvious that that first person voxel modification mechanic was a format that could support some really cool games. It got Notch thinking, for sure.

When Notch posted his first tech test video (archive version: https://www.youtube.com/watch?v=UMpv5kZ9-rE), he wrote:

> This is a very early test of an Infiniminer clone I'm working on. It will have more resource management and materials, if I ever get around to finishing it.

Ironically, he never did get around to finishing it.

As someone that's "recreated the core game" a few different times using a few different techniques (Unity, then Unreal, then OpenGL twice) this just isn't true. Minecraft is very deceptive in the fact that it looks simple, but recreating the core of Minecraft alpha is incredibly time consuming and difficult. Most of the clones you see don't truly recreate the core either, they'll add very simple biome generation, chunking, and some player movement and lighting.

The core of Minecraft alpha consisted of: mobs, "smart" blocks (fences that join to other fences, interactable blocks, animatable blocks etc), water, lava, physics, pathfinding AI, inventory management, crafting system, Redstone (this is actually pretty huge because it's not trivial figuring out how to properly update chunks), multiplayer networking (also not trivial by a long shot), survival mode, biomes and the nether.

Just building the base game with all of that will take around a year of full time dev work, which it did take Notch around that much time. Additionally, making all of these systems good and fast is a whole 'nother story. There are so many optimizations that can be made that add a lot of extra dev time to the mix.

So, sure, creating a cool blocky chunk renderer where you can add and break blocks can be done in like a week. But Minecraft alpha is not a simple endeavor, and I don't think it would have taken off the way it did if Notch just left out all the actual game mechanics.

> I don't look at Minecraft and think "wow I could never do this." Instead I look at it and think, "Wow. That was a really great idea."

And again, I know what you mean by this because I thought the same thing, but this couldn't be further from the truth! It's funny how I wildly misjudge the amount of effort that making a thing takes until I try to make that thing.

And we also have to remember that before you can even begin to create Minecraft you need all the prerequisite knowledge of how the graphics pipeline works and how rendering in general works. But anyways, I just wanted to interject some personal experience here and mention that a lot of the "simple details" cough multiplayer are actually pretty tough to pull off :D

Yes, because they have something to reference. It was like Notch took a level editor and made it into the game itself. That isn't an easy thing to figure out and takes time.

> Sometimes there are just really great ideas.

Ideas are easy and usually it's really stupid ideas that turn out to be great ideas. Who would think that an FPS level editor would be a good idea? Or a text box that anyone can edit on a profile page would be popular (facebook)?

There are not great ideas, just ideas that are executed well.

> Yes, because they have something to reference.

But you couldn't reimplement podman in a few hundred lines of code.

> Ideas are easy

Not all ideas have the same quality. Should be self evident. Nevertheless you should be able to see that I understand ideas are cheap in some sense - that's what the comment about decoupling "this was an iterative process" from "I could never do this." You can transfer an idea effortlessly - it's more about implementing it. But if its it trival to implement clearly more of the brilliance was on the idea side of things.

You don't even need a few hundred: https://github.com/p8952/bocker

And then there's 'dokku' which IIRC, started as a bash version of Heroku.

> Not all ideas have the same quality.

They really do. I've heard all kinds of things in my career, but almost none I would want to dedicate a portion of my life building. Not because they are bad ideas or won't work, but because of the person with the idea or it just didn't interest me. Those people went on to be moderately successful (like hundreds of millions worth) but I'm glad I wasn't on that ride.

He built an infiniminer clone for himself when it was discontinued. It certainly grew into more than that, but that's where it started.

It’s not something anyone could do, but often it’s the product of what many can do.

Clearly that can’t be true. “Amazing” by definition cannot be done by the average person.

Most engineering is this way. Somebody needs to build a thing that isn't too different from other things on a budget and a timeline. Your typical civil engineer isn't designing a new burj khalifa. Most software engineering is about organizing a group of people to build a thing for a customer that isn't too different from other things, on time and on budget. There just simply aren't as many jobs working on wild stuff nobody has ever done before.

RollerCoaster Tycoon is, in my opinion, a great example of something that isn't really engineering. One person did it and the decision to write it all in assembly is almost anti-engineering. It is an impressive feat, but not really comparable.

While I agree it's often a team-sport, being able to create amazing things yourself (i.e. Git, Redis, RollerCoaster Tycoon, Rust and other single-person projects) that others struggle to replicate or contribute too shouldn't mean you're a bad software engineer.

I actually don't know for certain that writing Rollercoaster Tycoon in assembly was the not only possible way of doing it. But in popular consciousness it is usually discussed as a wildly difficult thing rather than the most efficient solution to a problem. That's anti-engineering.

There isn't really a good set of words to break down categorization. But I do think that the skills involved in the large majority of software engineering roles are pretty much entirely separate from the skills involved in writing some exceptionally difficult program.

There aren't a lot of companies that are doing that in software. For example, there are a lot of SaaS businesses doing things that might have interesting engineering problems to solve but those engineering problems usually don't result in creating new products for software engineers to use. Rails is a good example of a byproduct from a SaaS business, React is a byproduct of whatever Meta does.

They are vastly different disciplines. Especially engineering on a large team.

Creating amazing things yourself that others struggle to replicate or contribute to is more in the realm of craftsmanship, in my opinion, and not engineering.

I can appreciate both good engineering and good craftsmanship. And I also often find myself doing (and enjoying) what I would consider crafting and not engineering - the lines can be blurry and as programmers we often end up doing both (sometimes at the same time!). Nothing wrong with it. But it is something that is distinct from engineering, in my opinion.

This is absolutely the case and my main struggle as a SE. In my opinion this is all arbitrary and companies could easily structure themselves to allow engineers to focus on what they do best, instead of adding a ton of overhead with tasks that someone else that is not trained as an engineer could do. The response seems to always be the same and boils down to "you need to add business value and that includes all these other easy but time consuming tasks".

I seriously doubt that is the only way, but I haven't build a large software company so ultimately it is hard for me to say, however I suspect that if an engineer's time is maximized to practice engineering, it would bring a lot more business value in the long run.

Not many devs create the core of what is arguably the best programming language available today as a personal project. However, it's true that what it has become today is now the work of many.

IME, yes there's comparatively far more CRUD roles than platform roles. But bluntly speaking as someone who's been on both sides of the fence, platform roles are not for everyone. For these roles, there is a expectation of quality by very technical stakeholders and this creates some pressures and incentives that don't necessarily exist in orgs that cater to non-technical faceless stakeholders.

In practice, a big challenge is to avoid being someone who is "not hardcore enough" (i.e. incapable of implementing large scale reusability due to lack of ability, tendency to cave to timeline or other pressures, or distaste for "office politics") but also avoid being "too hardcore" and being constantly in the weeds chasing some cool clever idea that might not align w/ overall picture.

One thing that is often overlooked in open source projects with sizable communities is the amount of cat herding you need to do. People have all sorts of ideas for "features and improvements" and sometimes your job as a BDFL may be to say "no" to your most ardent supporters. This also happens a lot for staff level platform work.

You hit the nail on the head. You've heard of a "yesman" (i recognize the sexism in that phrase, but it is the colloquial term), people that simply say yes to everything. I always joke that I am a "no-man". My job is literally to say "no" to probably 12-15 people a day. I say "yes" maybe once a week. It is actually exhausting and frustrating to be in this position.

I can't talk too much about what I do, but I am a staff level platform engineer and run a platform that no one has heard of, and no one has used directly, but it affects most people reading this. Because of how important the security and infrastructure is, my job is to field requests all day long, say "no" to almost all of them, and accept 1-3 things a month that our team actually works on.

Saying "no" so often and being "the bad guy" that no one wants to give ideas to is frustrating and difficult. It is the hardest part of the job, far more than the technical component. I actually discuss it with my therapist nearly weekly, it takes that significant of a toll on me. I enjoy the core of the work, but these externalities of the job wear me thin and I often fantasize of going back to the time when I just showed up in a daily standup, took a ticket, and went away to work on it. Essentially what I am saying is "the grass is always greener", remember that.

Good questions are "how can we X?" And good answers are "here's how: Y"

Then the asker decides if they are willing and able to Y.

1) Decide: is this worth it for the business? NOW?

2) Delegate: ok, we've decided it's worth it NOW. Who is best equipped to execute?

So, that's not a "bad question."

Most of the time, saying "no" also comes with helping the requester de-scope to meet the crux of their obligations, save face on any commitments they won't be able to meet.

Saying "yes" means figuring out what the work will displace.

A significant part of my job as a staff infrastructure engineer was carrying this kind context between planning rituals with various time horizons, ranging from weekly to quarterly to annually.

Occasionally I would drop down into execution mode myself to knock out something particularly gnarly, or set up some pins for someone else to knock down (e.g. promo season).

Application/product developers are seen as less capable and they put their best developers in infrastructure/platform/sdk/libraries.

This is all completely backwards. The application is what generates revenue and what needs the most attention to ensure the tech is a good fit. The needs of the application should be what drives the infrastructure to support it, but instead those companies have infrastructure people call the shots and heavily constrain how all applications should be built.

Thankfully there are sectors where application developers have full freedom on their tech stack in order to satisfy their business requirements and infrastructure is just people maintaining a collection of tooling and services that others can opt in to use or not.

Anywhere there are extremely large or technical company will have stellar ICs. All the FAANG have them (not everyone working there is one, but they are around and you find a lot of them on e.g massive distributed data systems). Not only do you run into stellar staff engineers at these places, but if you work on critical infra at say Google, you run into a couple 50x engineers that defy classification. Any company that produces a world-class database, distributed system, etc. will have very good, very expensive staff engineers, who really are ICs and not managers.

The definition I have generally seen for "staff engineer" is an engineering leadership position without direct reports - I see "manager" as generally meaning "has direct reports".

I think this is important: leadership and management are not the same things!

I'm responsible for the wellbeing of my direct reports. I need to ensure they can grow their careers. I need to know when they're stressed or burnt out or bored. I need to manage projects and be a shit umbrella.

While I am technically strong, I have architects and technical fellows who are WAY stronger than me. I work closely with them to ensure we're adopting the latest best practices and tools. They do much of the system design. They also mentor newer employees, but they don't have direct responsibility for those employee's careers. And they don't have direct responsibility for the success or failure of any given project.

Engineering management is more on the management of people and engineer leadership is more on the management of products/projects/systems.

My manager, skip, and director have all been extremely supportive of this throughout the process, despite it taking a somewhat lengthy period of time and putting me (a senior-almost staff eng on the team) in a position where it's very difficult for me to commit to a significant ownership of big strategic things (which is my job) since I might transfer out at any moment.

Similarly, a few years back, I expressed intent to transfer away from my team to my manager. He basically said "stick around an extra quarter since you're on the cusp of promotion and it'll be easier this way" (and he was correct, my promotion would have been delayed if I transferred first), and I did and got promoted, and ended up staying longer, for various reasons.

Additionally, my managers have all been receptive to my goals in terms of career growth: do I want to lean into management (no!) or into deep technical expertise? Do I want to go for additional promotions, or am I happy to stay in the role I'm in and continue doing the same kind of work?

All of this, to me, reads more like career management than job management.

This is an incredibly common part of the role(s), so my guess is that you have been unlucky in this.

For what it's worth, all the strong managers I've ever know or worked with pay attention this this, and vast majority of companies (again in my experience) made it an explicit part of their role.

I'd go so far as to say as a people manager (as opposed to a project manager, say) you cannot do a great job without considering things like career progression, because if you don't, you don't understand your team well enough to be really effective.

I could be very unlucky, sure, but I've worked at my fair share of places and it's always been the same. Management is there to keep the cats herded and HR is there to make sure nobody fucks with the money. All that matters is that I do my job and that the employer can use the manager to push more work on me under the guise of setting myself up for a promotion or raise that will never, ever come - it never has.

> all the strong managers I've ever know or worked with pay attention this this, and vast majority of companies (again in my experience) made it an explicit part of their role.

All enhancing an employee's career does is ensure that they have leverage to negotiate higher salaries or leaving the company. No company wants this.

> I'd go so far as to say as a people manager (as opposed to a project manager, say) you cannot do a great job without considering things like career progression, because if you don't, you don't understand your team well enough to be really effective.

Sure, you know where strengths and weaknesses are but that doesn't mean you devote company resources to making the employee better suited for leaving the company. You can do things like growth but you're going to come around and put penalties and fees to prevent the employee from leaving voluntarily.

I have never seen any manager or business treat an employee's career as anything but an inconvenience at best nor have I heard of any of my coworkers or friends or family having experienced anything like this. I can only assume that it doesn't exist or that it is so rare that it may as well not exist.

Explicitly, no. Growing talent inside is far less risky than hiring from the outside. Sometimes that means someone will leave, yes - but if the relationship is good they will do it at the right time when they are ready, not when they are miserable. This is win win. Long term net positives here are hard to understate.

> I can only assume that it doesn't exist or that it is so rare that it may as well not exist.

Well I can equally attest your assumption is false. But this is probably just a demonstration that path dependence is a real thing.

All I can say is you have a very negative view of the whole situation, which doesn't match my experience at all, or that of many people I know.

A power culture has zero trust and is high blame. Bureaucratic is process oriented. Generative experiments, tests, learns from failure.

The organization you have never experienced is generative. And coupled with transformational leadership it’s a pleasant environment to work in as either a manager or IC. These orgs do exist and are not rare either.

Leadership is about leading: helping the company make strategic decisions about what problems to solve and how to solve them, mentoring and inspiring and educating others, generally up-leveling the wider organization but without the direct responsibility of managing individuals.

People whose job title is manager might have to lead people too, but they're being held accountable for managing a thing. A project, a product, a budget, a release, a software system. Managing means 'making sure the thing happens'.

What a staff+ engineer is held accountable for is engineering. What gets built and how. That a technical solution meets its requirements and constraints.

In order to do that they might also need to lead people. But they don't have to manage anything.

On the other hand people's sense of who they should really listen to always skews to their manager (the person that can fire and/or promote them, or at least has the most say in the process).

(I originally wrote this with "responsible" in place of "accountable" but that was fuzzier; surely as a staff engineer I'm responsible for more than my direct contributions even if I'm not directly accountable for them.)

Management: you can change their pay grade.

Engineer gets evaluated by the products they contribute to that launch.

This means looking for companies who build and maintain a product, SaaS or not, which they offer to customers, and avoid all sorts of consulting, offshoring, nearshoring, whatevershoring third-parties.

Project/Service companies draw little value from engineer familiarity with the systems they work on. Contracts and projects come and go, and everyone needs to learn new systems, processes, and technologies all the time. Experienced people end up as Project managers because that's the only path to career growth.

Product companies, on the other hand, suffer a lot from engineering turn-over, since they have this one complex system they need to babysit forever, and ever time someone leaves, the accumulated knowledge built on that person leaves with them (likely to a competitor who really benefits from that knowledge as well).

1. Work where no individual component is complex, but there are a ton of moving parts and interlocking dependencies and interacting systems and so forth. A staff+ engineer is needed to drive this, but that job is less about writing the hard code and more about teasing out and resolving the hard interactions, so that kind of system design 'n' people role.

2. Work where there is a self-contained component that is very complex. A staff+ engineer is great to drive this, and may do so by writing the code themselves and reviewing other people's code as a developer.

In practice, many more companies have complex distributed systems than complex contained systems, but certainly both kinds exist.

I manage 2 core teams, one a "platform" team made up of all senior => principal engineers, and they spend a lot of their time shepparding other less experienced teams while trying to answer very tough, vague questions like "what bottlenecks do we predict at 10x scale? What can we opportunistically do today?". No one gets huge, dedicated time blocks to code without some degree of accountability and expectation.

FANG and FANG-model (anyone copying FANG engineering/product model) companies do it well and have the bulk of success cases. Look for engineering-lead companies first, product lead companies second.

A key question when interviewing is "how is the roadmap established?" You want answers where senior leadership establishes goals (strategic or metric), product provides their understanding of opportunities, and engineering decides what to actually do. Even then, there will be the common failure model of EMs doing the roadmapping, not senior (staff+) ICs, but you can tease apart how that works, as it is always going to be a mix.

Absolutely spot on. Making use of Staff+ engineers requires giving them a hard problem that would catapult your company forward if it were solved, and a ton of autonomy in how they go about it. Give them dedicated time to prepare presentations on what they're doing and what they've learned recently. Give them dedicated time to mentor other staff members. Give them a routine, but not a schedule. Stop asking them for so many fucking estimates. Give them access to support staff. And then get your fucking hands off them and let them veritably radiate value to your company.

All of this is, of course, antithetical to whichever imposter-syndrome micro-managing MBA vice president happens to have landed in the chair that gives them authority over these people in their never-ending game of Management Musical Chairs. No fucking way are they going to let some grunt just work on a hard problem without constantly getting in their way, nagging them about pointless deadlines and bullshit estimates, and showing visual progress via good-looking DOM elements so they can take credit for it and get Business Points before their 2-year tenure in that chair is up and they move to Head of Brand Awareness or wherever. No: doing so would reveal that their entire position is literally pointless. Can't have that.

That's because this is what 90% of companies need. Even at big tech or any nationally-known tech brand, I'd bet at least half the work in maintenance, support, incremental improvement, and integrations.

> Even at big tech or any nationally-known tech brand, I'd bet at least half the work in maintenance, support, incremental improvement, and integrations.

That's because they are extremely inefficient. Most people build software badly, most companies expect software to be bad, and they hire a bunch of bullshit jobs to do inefficient things to support that bad software badly. The gap between what software could be vs. what it is is so overwhelmingly massive that it's crushing my soul. These people do stupid CRUD API micro-service bullshit because they're blissfully ignorant of how tremendously amazing their software could be if they fired 90% of their boot-camp dev ops and replaced them with a handful of 10x staff-level developers.

I wish this was true, I really do. But a few staff developers cannot physically handle the volume of work required to replace that many developers.

Staff developers are good at solving complex problems that many other developers combined could not solve. They are not good at writing 10x more LOC than other developers.

Also, this shouldn't be framed as 10x more code but 10x faster velocity. That is what I meant by "[staff developers] are not good at writing 10x more LOC than other developers". Staff devs cannot move 10x faster than average devs on every problem.

Most work is undifferentiated. Staff developers don't add much value over another dev for basic work (Code quality has high diminishing returns) and they might only be 2-3x as fast, if that.

It's possible that many businesses could be sustained as they currently are today with 1/10th of the code and/or 1/10th of the people, but in a growing company that's impossible.

But, maybe it's true that most code covers edge cases; then in my opinion, staff developers should be building tools for the rest of the company to use, and if they are powerful enough, you might be better off by having fewer people employed as programmers and more people writing programs. Obviously a lot has to go right for this sort of thing to happen.

I think service/API companies like Netflix are the most suited to your approach, but even with Netflix leaning into that style of company hard they still have thousands of engineers. There is most definitely some amount of irreducible work for a problem/domain/other.

> Obviously a lot has to go right for this sort of thing to happen

It feels like a very naive approach. When I'm building stuff, the edge cases often require a lot of digging into the surrounding context and working with other teams.

I don't think you can solve this by abstracting it away, because it's still working at the same level of abstraction just in a different part of the codebase.

There is also the fact that every abstraction is leaky...

You're assuming "volume of work" is a constant. I'm saying: 90% of "volume of work" is something that a handful of good developers would make the compiler do instead of humans. It's actually more than 90% over time, because there's a fundamental difference in the "capability growth curve" -- this curve is super-linear with good developers and sub-linear with poor developers, so the ratio between them grows greater over time.

> They are not good at writing 10x more LOC than other developers.

Right, it's quite the opposite: they will write 10x less code than the other developers.

I don't understand what you're saying here. I cannot hand off feature work, rollout planning, context gathering and investigation, etc to the compiler.

> Right, it's quite the opposite: they will write 10x less code than the other developers

At a growing company, that is not helpful... You will not be able to move fast enough.

> It's actually more than 90% over time, because there's a fundamental difference in the "capability growth curve" -- this curve is super-linear with good developers and sub-linear with poor developers, so the ratio between them grows greater over time

This is only true when you are working on complex problems. If I have a lot of tasks like "change the colour of this button" having a few staff developers vs a lot of average developers is far worse.

At almost every single company, there are far more mundane and relatively simple problems than complex problems. Most companies would not get more value from having way less super senior devs. There are just not enough complex problems for them to solve.

Hell, it is often the case that requirements change up the whole program logic based on which country it is being used from. It has a much different complexity than a JIT compiler, but it is a super-human, not-really-compressible complexity nonetheless.

A "regular old CRUD app" doesn't exist. Poor development teams end up translating all requirements into "regular old CRUD apps" because that's easier. Humans are excellent at replacing hard problems with easy ones, regardless of whether solving the easy one is anywhere near as useful.

And yes, they absolutely would implement them faster. Most "regular old CRUD apps" are rife with SELECT N+1 and other simple and crippling performance issues.

> “these systems are building the rails in front of the train”, and it is not even an engineering problem most of the time.

You're assuming we're stuck with trains and rails. In the physical world, we are: if I invent a different kind of train that doesn't need rails, or a completely different way to transport freight, then I've solved about 1% of the problem -- we still need to build out all the infrastructure around it, change all our systems to use it, etc.

If I invent a different kind of process for handling data flow in an application, I've solved somewhere between 40% and 100% of the problem, depending on how 'private' it is, etc. "Building" the thing takes the compiler a few seconds. We might still need to adjust downstream dependencies, but if they're also written well, that should be extremely easy.

> it is a super-human, not-really-compressible complexity

While this definitely exists, my claim is that the intrinsic, Kolmogorov complexity of the problems most existing software solves is somewhere between 10% and 1% of the actual complexity of that software.

Sometimes upstream processes will need to change (usually becoming much simpler). This is usually the sticking point. if the upstream process is the U.S. Code of Federal Regulations, then this isn't likely (without lots of lobbying money to simplify it, which the FAANGs of the world should actually consider a tool in their toolbox, but most of us can't), and we can call the "intrinsic complexity" high.

But more often than not, that complexity is actually coming from the whims of some previous moron vice president who now works somewhere else, and they're conflicting with the whims of the current moron vice president. My whole point here is that the staff-level developers need the authority to tell those people to shove off and implement it in a way that actually makes sense. Yes, I'm aware that will never happen because of the deep toxic dysfunction endemic in modern corporations -- this is my point.

I called them vertical and horizontal, not sure if they have some better name, but the vertical one would correspond to “hard” problems, like some complex lock-free parallel algorithm that had to be made in exactly that way to not have race conditions. Or some of mathematics.

The other dimension would be the essential part of business logic, where the business logic has grown in a haphazard, organic way - but since other services depend on it, it has to work as is, from bug to bug almost. This is more like a biological system — cells have plenty of redundant, vaguely similar pathways handling the same thing, it is chaotic, but if you were to simulate the whole thing, you couldn’t go around it, you would have to implement the whole complexity. For a more human example, history might be such subject. These things are complex in the same way compressing a random noise file is.

A software can have both kinds os complexity, they are different axes — and I argue that CRUD apps are usually of the second kind, most of the former kind of complexity are handled by external services/libraries (e.g. they might enter two addresses to a third party system to get two coordinates, and put it to another service to plan a trip between the two — the app only handles the communication). But these can get arbitrarily complex, and the specification is the program itself. That’s why we continue to run Fortran/Cobol software at certain banking firms.

Well, we know stories behind each of these:

- linux kernel: comp sci Msc student - redis: written by a startup founder (in TCL[1]) - next.js: written by a founder of Vercel (as "Zeit") - rollercoaster tycoon: written by one dude in scottland

None of these people are Staff Engineers, in the sense of this book. You may find a few traveling between big tech companies but mostly as a prestige recruiting / consulting role. Or Torvalds, whose finances I don't really understand -- multiple sources say he gets a million dollar salary from the linux foundation?

You might also argue that the people who put more value on their skills as "hardcore engineer" end up in small firms and startups where equity comp can end up paying them what they're worth if the skills pan out. There was a "why is Google behind openAI even though they published the transformers paper" post recently that hinted that all the authors have left google for greener pastures, which supports this thesis.

[1] https://gist.github.com/antirez/6ca04dd191bdb82aad9fb241013e...

Except Rollercoaster Tycoon, which is more like the work of an artist.

Like, coming up with Linux is a lot like Guglielmo Marconi putting together a working radio transmitter/receiver in his parents' attic.

These things can overlap. Art Fry, inventor of the Post-it note, was in a kind of 'principal engineer' equivalent role at 3M when he came up with it (he was the senior scientist in a lab team).

After you've built up credibility in some interesting tech, other companies would find you based on your work.

This is just my observation, so take it with a grain of salt

In my career I built trust internally and so got interesting work. I talked about that work and so got approached about other jobs, and I’ve been able to move when I’ve wanted to because I’ve kept in contact with some former colleagues.

I would however like to stress that the most important point of this is doing the aork, and doing it well. If you don’t do that then everything else is pointless.

Big tech sucks a lot, and I mean a lot, of the oxygen out the market for these roles.

If you take the total population of engineers, then how many engineers are needed to build TCP/IP stacks, new app network protocols (i.e. QUIC), databases, web frameworks?

The unfortunate truth is that getting deep experience is rare.

Risk aversion is a core culprit; would you trust a tech not supported by a large community? This creates a tremendous market pressure on a handful of players.

This is one of the reasons that I had to work at big tech to get the experience and skills that I wanted. For example, when I was at AWS, I read every COE (cause of error) report for the big services. This helped me tremendously both as an IC and leader.

Now, I hate that I had to bend the knee to big tech...

I've got 2 yoe in high performance c++, working on data processing engines. I'm really tempted to go to the startup... But big tech pays more. And I feel like I could actually learn how to use tools properly before I come back to writing them. Any opinions?

Big tech isn't going anywhere. If you are getting offers from big tech now, you will later as well. I always tell people to start at smaller companies. They are way better when you are young because you will face a wider array of problems, which builds better experience. Big Tech is great when you are older. It is generally more focused, higher paying, better benefits, but can often be less rewarding.

But my point is that, you can almost always find some way into an interesting company such that you may be able to eventually pivot to (or at least periodically work with) a team you're interested in.

The difference is probably that it is harder to get those jobs because everyone wants to work for the flashy exciting companies. I was considered for a job at Docker a long time ago (when I was an early-midlevel) and the recruiter told me that over 1,200 people applied in 5 days. For entry-level/intern positions I assume it is similar or even worse.

Really bad engineers see these environments and complain without doing anything about it, ultimately leaving the project. Slightly better engineers complain about the tools and blaze ahead anyway leaving a mess in their wake. The great ones I've been so lucky to work with pave the way on those fronts at the same time as solving the problem at hand and make it easier for their successors. It's quite beautiful when you see it.

In a larger company now, still working on GPU drivers (AMD, but not the "main" platform), there's a couple of people who might do a bit of this, but feels more like you get specific domain experts rather than anyone designated by the company. The team is notably larger than at PowerVR, and there is a "Manager" who is more involved in doing things like placating customers and prioritizing work and issues, but again completely uninvolved in the actual development.

In none of these examples does the job title have any relation to the differences in their work, both used completely different stacks of terms and difficult to compare cross-company.

But I guess it's all "Hardcore Engineering", and even the newest members of the team are expected to design systems themselves. Hell, straight out of university at PowerVR I was designing and implementing my own stuff, and significant systems not just following an already concrete architecture. But it was a small enough team everyone was involved in feedback, and I did feedback to them in turn. So I guess GPU drivers are still greaybeard "Hardcore Engineering"?

This may be helped by driver development often being pretty well defined, you have a spec (which you may have been involved with designing, but very much a large collaboration), you have hardware architects and another team trying to figure out the next generation might look (are they the "true" architects here?).

If you step back and think about it, it makes sense. The majority of companies are one-trick ponies, maybe following up with a few additional products. Incremental development is the path of least risk once product-to-market fit in a large market with high growth is achieved. As public companies (and late-stage startups or private companies) are incentivized to reduce risk, the majority of the people will be doing incremental work. The growth of the pseudo-technical managerial class in the form of Product Managers, Project Managers, and umpteen levels of mid-level managers further encourages the relative lack of 'hardcore engineering'.

The idea that large companies aren't solving hard problems, that that's only being done at startups, and that large companies are 'one trick ponies' is just incredibly myopic. Have you worked at a large publicly traded tech company?

Sure, in any business a bunch of the systems will be trivial. In larger businesses - especially ones that have grown through acquisition - there's also some scaling-law complexity that means they have to also do a bunch of annoyingly trivial work gluing together lots of duplicative and redundant trivial systems. When you look in from the outside of a large business that mess can definitely obscure the fact that at the heart that stuff is all in support of, usually, several significant and impressive engineering achievements, which support and drive that business forward. And they're probably performing operations that, from the outside, you would hear the name of and think 'well, that doesn't sound very difficult'.

I'm going to give a non-software example, that I think might show a little bit of an analogy for how 'engineering' at big companies can be at the same time operating at a complexity you wouldn't imagine, and yet be accomplishing something you might think from the outside is totally trivial.

Jump to this timestamp in the recent Smarter Everyday video about Kodak's film manufacturing line: https://youtu.be/mrJP82ZZiag?t=1742

See that workshop? Those precision engineered toolheads they call 'hearts' are the punch+die sets that they use to cut sprocket-holes in film. Everything there is bespoke, custom-made, and meets constraints you would never have thought of. It's engineered.

Software engineering in large companies is like that. If you don't think that banks contain software systems as complex and carefully engineered as those hearts? Or that manufacturing control software isn't in the same league? Or that the inventory management systems that drive a large retail business aren't on that level? You just have no idea what engineering is.

The point of the original question isn't whether products as a whole are complex and hard, it's whether an individual engineer will get to do fairly large, complex systems.

Somewhere someone has to have the time to build out the tools, then dig the foundations and lay the scaffolding and the supporting walls. None of which looks like the artists impression and none of which has a web interface you can show.

So Staff engineer is like architect. Holding the nervous customers hand while they spend a fortune to have a large hole in the ground.

You cannot build a bridge by stacking bricks in front of the customers feet one at a time in a nice iterative process. You cannot always "see" progress at the user level.

to me the signature of really good software is when one marginal change delivers outstanding value and the engineers just say "was that it".

They create their own thing from scratch, and turn it into a job.

The first 5 from that list were open source creations that became big because they met a need at the time and lots of people adopted them. Which is one route into creating your own thing.

I've seen senior engineers who have leveraged lots of domain knowledge to build a prototype of something super useful internally within a company that then grows in adoption and size as people recognise it's value. Projects then start to organically cluster around the core that was built solo.

If I were still an IC rather than a talking head on Zoom all-day, my role would map to somewhere between Senior Staff and Principal. I would write microservices from scratch, deploy to the cloud, operate them, and then write all of the clients (iOS/Android/Web) myself. I've been doing this for ~20 years so I have the ability to quickly pick up new languages, frameworks, platforms, technologies, etc.

The current Senior Staff/Principal engineers do projects like decomposing that old miscellaneous database from the original monolithic codebase and implement it across all domains with correct boundaries. Build libraries that all engineers on the team use. Ship V1 of that new product that is very strategically important to the company.

(Send me an email if you are a Staff+ engineer that is looking for something new!)

If I had to take a guess, a lot of these places are close to hardware.

Working with hardware has the additional bonus of often mandating high performance. Any task becomes a lot more exciting when you have microsecond deadlines for controlling an industrial laser.

At the end of the day, you're the voice and the advocate to ensure your end-to-end vision is implemented.

The creators of the software you mentioned are rarely in a role defined as such. If you really want to go the hard-core coding path, the startup scene is the best place to look.

I would define that as "software engineering". What else are any of us doing all day? The compiler's job?

"Here is a vaguely understood problem, that might take 2 hours to solve."

"Here is a vaguely understood problem, that might take 2 years to solve."

Though if you're used to the latter timescale, you might not consider the former as all that "vaguely" understood.

But yes, jobs are very rare. Staff+ ICs tend to like to stay where they work (because great achievements take time), there's only a limited number of roles, and a lot of this is word-of-mouth in a given field.

I'd look either at FAANG - they're large enough to always need staff+ folk - or startups once they have round A. They live or die by having really experienced and productive staff+ folks, and they're large enough that they can offer staff+ interesting challenges.

Why not pre-founding? At that point, any good staff engineer I know is driven enough that they'd be better off just starting their own. In a small startup, the founder is either technical and interested in solving the hard questions, or non-technical and you'll have essentially a CTO role, but without founder credit.

It's not a hard-and-fast rule, there are exceptions to everything - but that's where I'd focus my search if I were to search for that kind of role.

There is NO such thing as hardcore engineer job position that doesn't involve a lot of management. All the good engineers that get jobs are the ones who do management work as well as coding. If you cannot do this, the industry will just close its doors for you.

So bizar. I've been in a few companies with senior "hardcore" engineers heads down in code, without management responsibilities.

> If you cannot do this, the industry will just close its doors for you.

Wait what?

A good engineering company will strive for a mix of seniority within teams of ICs. Some of these seniors may become "staff", which usually means, can talk tech stuff to the brass.

In other words, the Idea is that engineers adds more value to a company if they also do all these management tasks, which implies that engineering is not enough and is not as valuable on its own - something I would contend because as we can see software is capable of amazing things like AI that will soon automate all these management tasks anyway. If companies invested in their engineers doing only engineering, then people could maximize their skills, and reach those new levels of productivity thanks to advanced software. Instead the industry limits engineers, and insists on them hitting a technical ceiling, effectively pushing them in a non-technical direction in favor of hiring younger engineers to do the "grunt work". Software engineering is only grunt work when you don't understand its true potential, and its endless complexity that takes a lifetime to master.

I'm not saying managers are not necessary. It just seems like there is a pyramid structure, and managers are at the top. If you want to be on top of the pyramid you must evolve into a manager and let others do the engineering. I'm willing to grant this is ok in non-software companies, but in software companies it's bizarre and I'm even thinking software engineers should be unionizing and getting some representation.

But HFT jobs might be interesting if you want to do hardcore engineering. I imagine there are a lot of interesting ML jobs.

The reality for other staff engineers here is exactly opposite though - you're not allowed to just be "executing" and your day consists of being stuck in meetings, doc writing, and management-lite. Honestly it's so bureaucratic and so far removed from the actual tech I really got turned off from that career path as I got more visibility into it.

For smaller companies it's probably much more fulfilling though.

I am by no means a rockstar in any particular field of software engineering, but I have worked on a very wide gamut of projects. This enabled me to dissect the task into the constituent parts in an attempt to reduce uncertainty and complexity.

Fwiw, I have no life, severe executive dysfunction that doesn’t allow me to stop thinking about what I am working on, and an abundance of curiosity.

If you prefer system level programming there are jobs out there and plenty of work to do. I dunno if this counts as, "hardcore engineering." To me, engineering, involves many other skills other than writing code: being able to specify systems clearly and, as the occasion demands, formally is a part of engineering; communicating with others; Knowing which trade offs to make depending on a situation, etc.

Hello!

I'm making a roller-coaster tycoon inspired game, in the form of a city builder. I ended up creating my own path finding algorithm that 10x's the performance of A*. I felt this would be useful since pathfinding is one of the major slowdowns of simulation games.

https://twitter.com/YesboxStudios/status/1647224184597102592

End of the day, everyone at Bell Labs was a researcher with a PhD.

Engineering, bringing research to people, trends to CRUD over time.