It'll generate one token because it's trained to predict one missing UNK, as I understood it. What is the scaffolding? Do you generate random sentences and iterate repeatedly? And how does that get you whole coherent paragraphs? (Has anyone demonstrated that this actually works with BERT?)
BERT can pretty easily be used to generate text. It's intended to be used as a base model and fine-tuned with an additional model on top. The fine-tuning model could then be trained to generate sentences with the underlying language model powered by BERT.
