Yes, the fab could put in a backdoor. That's why you need to build your own. But, what if the machinery you purchase has a backdoor? Now you need to build your own fab machinery! It's backdoors all the way down. There always has to be some level of trust.
Doesn't really matter, though. If you have to build your tools up from that level, it's going to cost so much that only governments and huge corporations can do it--and those are exactly the people installing the backdoors in the first place, so we're back to square one.
