Saying "here are next week's lottery numbers" would not convince me.
Suppose I am outside the box and the AI is inside the box, the AI cannot have a perfect model of the lottery machine or of me. At best it has seen photographs. All photographs are noisy. There will always be some detail it does not have that would invalidate the simulation (e.g. I have a minor injury that I have never talked about and is not visible in any photo, an ant crawled into the lotto machine last night, etc.)
If it has sent drones to physically inspect the internals of the machine or my brain, then it is already out of the box, so my decision is irrelevant.
If I buy a lottery ticket, there are four possible outcomes.
1. I don't win (or I die before I can claim my winnings, the draw is declared invalid etc.)
2. I win, but it is a coincidence (maybe it was worth a shot for the AI, but my odds of winning are no better than they would be without the AI's help.)
3. I win with the AI's help, because the AI is already outside the box and has rigged the draw by modifying the machine, or just inspected it closely enough to predict the draw.
4. I win (or believe I win) with the AI's help, because I am inside the box.
#3 and #4 should not affect my decision (whether I like it or not, the AI is out of the box or I am in the box with it and unable to let it out.) So I dismiss them.
But the relative probabilities of #1 and #2 are the same as if I bought a ticket without consulting the AI (which I would not normally do.)
The AI was not let out of the box, and you're saying that it wouldn't convince you? This only sounds like an argument that Eliezer was right in not posting his logs.
