A lot comes down to "perfect storm." At least many parts of California have a long history of controlling growth for a variety of reasons including environmental concerns and people wanting to preserve existing character.

Proposition 13 likely plays at least some role.

People like living in California. For the most part, it's just about the perfect climate whether your idea of a perfect climate is better satisfied by SoCal or NoCal. Lots of natural and other recreation activities of all sorts.

Finally, it's been a job growth magnet across a lot of different areas, most recently in certain types of tech. And, with respect to the Bay Area specifically, much of that growth has been relatively sudden on the scale that sort of thing normally happens.