These things with <think> and </think> tokens are actually trained using RL, so it's not like GSM8k or something like that where you just train on some reasoning.

It's actually like QuietSTaR but with a focus on a big thought in the beginning and with more sophisticated RL than just REINFORCE (QuietSTaR uses REINFORCE).