Money aside, there is a sort of more systematic reason we see physics MS/PhDs bailing to industry.
Though theory groups in general tend to use computational simulations as a tool to complete calculations, groups that develop novel computational methods and techniques tend to be headed by younger, more junior professors. These groups are typically well-funded and do very exciting (trendy? cutting edge?) work with distributed computation, machine learning, neural networks, etc, so they tend to pull quite a few students.
While these computational groups tend to bring in funding and are well-staffed by excited grad students, the junior professors leading them tend to be marginalized by the more traditional, seniority-focused establishment. Which is to say, a new PhD might have a lot of trouble landing a prestigious postdoc because a) their adviser might have been too young to have high name recognition outside their field and b) departments might place limits the amount of staff for these more junior professors/young groups doing exciting computational work. This is, of course, on top of the overall scarcity of jobs in academia.
But there's no such job scarcity in industry-- especially not for stats-smart programmers with years of experience a) wrangling data in python, b) writing fortran that runs on distributed clusters, or c) designing algorithms to solve /approximate hilariously expensive problems. Advisers know this and point some of their students who might thrive more in industry than academia towards that route.
(Anecdote: And of course, as a physicist who builds models/simulations in industry, I can speak a personally a little re: thriving. If you're someone in love with solving disparate problems, you're unlikely to find that in academia. Some of us learn in graduate school that we can't spend our whole lives-- or in my case, more than a few months-- solving one problem. Academia just... didn't seem like something that would be worth fighting for.)
I assume that this will gradually change as there's turnover within physics departments and we get more computational-first professors with seniority (or even in leadership). There are a few departments with better-known professors you can see it happening now. Universities are spinning up incubators and institutes for computational research. Physics departments are just slower to adapt to new developments, and the hierarchy of theorists can have more to do with seniority and internal politics than it does with technology.
I don't think you're necessarily wrong re: difficulty getting prestigious post docs after graduating from a young lab in a newer field, but I think you're way overselling it's importance.
The fundamental issue is the field is not growing (very much). Each professor will graduate 20-50 students over their career, but only one will get their job when they retire (on average).
From someone who might be pursuing a physics degree soon.
> Some of us learn in graduate school that we can't spend our whole lives solving one problem.
Would you please expand on this. I am not sure if you meant that problems are hard enough or what.
The nature of a PhD is to study one sub-discipline long enough to reach the edge of human knowledge, and then expand it. Often this is so difficult that a set of multiple diverse projects is not practical. But not always; in my PhD I worked on both astrobiology and solar cells. It helps to learn versatile methods that apply to a range of problems.
Though theory groups in general tend to use computational simulations as a tool to complete calculations, groups that develop novel computational methods and techniques tend to be headed by younger, more junior professors. These groups are typically well-funded and do very exciting (trendy? cutting edge?) work with distributed computation, machine learning, neural networks, etc, so they tend to pull quite a few students.
While these computational groups tend to bring in funding and are well-staffed by excited grad students, the junior professors leading them tend to be marginalized by the more traditional, seniority-focused establishment. Which is to say, a new PhD might have a lot of trouble landing a prestigious postdoc because a) their adviser might have been too young to have high name recognition outside their field and b) departments might place limits the amount of staff for these more junior professors/young groups doing exciting computational work. This is, of course, on top of the overall scarcity of jobs in academia.
But there's no such job scarcity in industry-- especially not for stats-smart programmers with years of experience a) wrangling data in python, b) writing fortran that runs on distributed clusters, or c) designing algorithms to solve /approximate hilariously expensive problems. Advisers know this and point some of their students who might thrive more in industry than academia towards that route.
(Anecdote: And of course, as a physicist who builds models/simulations in industry, I can speak a personally a little re: thriving. If you're someone in love with solving disparate problems, you're unlikely to find that in academia. Some of us learn in graduate school that we can't spend our whole lives-- or in my case, more than a few months-- solving one problem. Academia just... didn't seem like something that would be worth fighting for.)
I assume that this will gradually change as there's turnover within physics departments and we get more computational-first professors with seniority (or even in leadership). There are a few departments with better-known professors you can see it happening now. Universities are spinning up incubators and institutes for computational research. Physics departments are just slower to adapt to new developments, and the hierarchy of theorists can have more to do with seniority and internal politics than it does with technology.