Those billions are also mostly R&D. It's also from a country external to Taiwan. ASML, the Ductch.
Yes, Obviously there's a lot that goes on in this process, more then "just" silicon. But if you want a summary, then cost expertise and knowhow, eclipses actual process. In a sense it is just "silicon" when measured relative to the ability and knowledge required to transform that material.
> The material is just silicon.
No, a lot of chemicals are required as well.