There's a whole host of concessions and project redesigns that occurred for essentially political reasons.
Just look at the currently proposed route map [1]. It deviates to the east side of the valley because that's where these towns are vs the west side, which is more direct.
Deviating a supposedly high speed route for small towns doesn't make a ton of sense. Not only does it increase the cost and travel time directly, but extra stops slow the overall travel time. This could've just as easily beeen on the west side of the Central Valley and had feeder lines and stations into a smaller number of stations.
Look at any high speed rail route in Europe or China and you'll see fairly limited stops for this reason.
The biggest and easiest win for a high speed rail should've been LA to Las Vegas. It's a shorter distance and through mostly desert and other uninhabited land. Ideally LAX would've been one of these stops but I'm not sure how viable that is. Then you add a spur that goes north to SF so you avoid building through LA county twice, which is going to be one of your most expensive parts.
Instead we have a private company (Brightline) building a LA to Vegas route.
As an aside, Vegas desperately needed to build a subway plus light rail from the airport up the strip. The stupid Teslas in tunnels under the strip was another of those efforts of billionaires proposing and doing projects to derail public transit. Like the Hyperloop.
The central valley is where growth is expected. See growth projections into the middle of the century. It is expected to double in population. Look at the rate of growth already. These "little towns" of 100k people basically double in size every 20 years. Greenfield is where the growth happens because CA urban politicians are against meaningful amounts of infill development. If they built it along the 5 in 50 years when the central valley has over 15 million people, you'd say it was foolish not serving these communities when they had a chance.
Brightline is building a victorville to vegas train. They have no plan to reach LA. Maybe as close as Rancho Cucamonga. In either case no work has been done yet on that project while construction on the HSR is ongoing.