What I really hope this helps solve is covering for the huge lag in knowledge cutoff. A recent example is where it went "oh you're using Go 1.23 which doesn't exist so that's clearly the problem in your Dockerfile, let me fix that".
But I'm not keeping my hopes up, I doubt the model has been explicitly fine-tuned to double check its embedded knowledge of these types of facts, and conversely it probably hasn't even been successfully fine-tuned to only search when it truly doesn't know something (i.e. it will probably search in cases where it could've just answered without the search). At least the behavior I'm seeing now from some 15 minutes of testing indicates this, but time will tell.
But I'm not keeping my hopes up, I doubt the model has been explicitly fine-tuned to double check its embedded knowledge of these types of facts, and conversely it probably hasn't even been successfully fine-tuned to only search when it truly doesn't know something (i.e. it will probably search in cases where it could've just answered without the search). At least the behavior I'm seeing now from some 15 minutes of testing indicates this, but time will tell.