I'm actually not sure this is a question of scale at all? The amount of revenue and the amount of moderation needed both scale with the amount of content.
If anything, YouTube's scale gives it a better chance of being able to address this issue, since there are likely fixed costs here that YouTube can better fund (obviously investing in better automated screening, but also setting up the workflows for human moderation, etc.) I think the only benefit a smaller scale competitor has is that they're less visible and unlikely to be "caught" for hosting content they "shouldn't". But if the rules applied equally to everyone, I think YouTube is strictly at an advantage in being able to implement better human moderation.
If I were designing from scratch, I would probably use a Peertube/Mastodon type federated system.
Each host would responsible for the videos they host - meaning they would have incentive to limit videos uploaded to any given host to an amount they could handle, or only allow trusted uploaders to their host.
Then you would have the federation so that you could allow hosts that you didn't have issues with, while blocking those you did. Eventually, there would evolve safe/block lists - like with email and adblockers - so it wouldn't even require that much effort on the parts of those running the hosts.
If anything, YouTube's scale gives it a better chance of being able to address this issue, since there are likely fixed costs here that YouTube can better fund (obviously investing in better automated screening, but also setting up the workflows for human moderation, etc.) I think the only benefit a smaller scale competitor has is that they're less visible and unlikely to be "caught" for hosting content they "shouldn't". But if the rules applied equally to everyone, I think YouTube is strictly at an advantage in being able to implement better human moderation.