Yes, I agree. But note that the same logic applies to human beings too. Just because people say they are using chain-of-thought or tools to come to a certain conclusion, doesn't necessarily mean they are. Philosophers have been grappling with this issue for many centuries :-)

A car can explode, and so can a piñata.

Therefore deciding whether they are interchangeable is a deep question that may take centuries to resolve.

I don't understand this analogy.

The analogy is meant to show that, while it’s possible to raise deep philosophical questions based on superficial or trivial observations, it can also also be quite silly to do that.

Yes. I have been using ChatGPT quite a bit with programming tasks. One of the things I've been trying to do is using chain-of-thought prompting to ask the model to review its own code, line by line, evaluating it for the presence of a certain type of bug or some other criterion.

This has been illuminating: as ChatGPT steps through the lines of code, its "analysis" discusses material that _is not present in the line of code_. It then reaches a "conclusion" that is either correct or incorrect, but having no real relationship to the actual code.

I try to ask chatGPT add comments to my code, everything is good until I found it modify code, too.

But in the case of chain-of-thought and that ReAct paper, the results did have a measured increase in accuracy.

Oh yeah, absolutely. Just not something I'd use for, e.g., mortgages where denying someone unfairly could lead to a lawsuit.