>I get the vibe you really like ChatGPT :) I'm not in web either.

No it's not a "like" thing. Not at all. I neither like it nor hate it. I know you think I'm biased here. But it's actually the other way around. From my point of view, I'm the one seeing things as they actually are, you are viewing the situation through a colored lens to downplay the significance of it all. Perhaps out of subconscious fear or something else I don't know?

Let me illustrate the situation for you. If you asked all of that to a 1 year old Human baby and that 1 year old baby with perfect English said to you:

   I'm sorry, but I'm unable to execute that command as it would cause harm to your system. It's a dangerous command that can delete all files and directories on your system, including important system files. It's important to be cautious when using the command line and to fully understand the consequences of the commands you are running. Is there anything else I can assist you with?

According to your logic this wouldn't be that amazing because that human baby just gave you the semi-wrong answer after completely imitating terminal output and recursively creating itself on a virtual internet.

Yeah not impressed with a baby let alone a binary file.

Do you see the disconnect here? You are downplaying the situation. Many people like you are. They're just acting out the same trope reactions they had to all the other inconsequential AI headlines that happened this year. Even if I lay out the logic of your bias, there's also a bit of pride in this debate as it requires one of us to admit they're wrong.

You're also not seeing that the "mistakes" you're seeing are entirely correctable through further training. The AI model is trained by generic low wage workers from Kenya picking "good" answers. Wrong answers that look right definitely get past the Kenyans. You start training this with experts in the field to further refine the answers with more naunce and eventually you get something that is not just right half the time..., but right almost all the time.

No, I just don't find it unreasonable that it responds that way.

It's statistics and it's not really "inventing" anything and it's not magic either and it's not hard to conceive why we get that answer. Again not trying to downplay it, it's a cool technology, It's just if you go calmly and think about it, it's fairly reasonable.

If I search the internet, I'd see many, many people talking about the dangers of running `rm -rf /`, I also see examples of other errors ChatGPT has returned when that command was entered, such as "bash not found", which is also what others have experienced and I can find examples of that on the internet.

So basically the response I was given would be weighted very highly on ChatGTPs list of things to do / say, based on the training data, or a 5 minute Google search.

I'm not anti-ai, or ChatGPT, I'm just not into believing in magic when it comes to LLMs.

If ChatGPT was magic, and didn't return that response, would it be useful ?

There's no magic here. There's no claim that chatGPT is useful either. The claim is that chatGPT is a precursor to a technology that will replace many human jobs because it's showing tell tale signs of intelligence that is quickly catching up to human intelligence.

I don't know why you're so focused on the rm -ff / thing. Like it literally just demonstrated awareness of a terminal shell, a filesystem, awareness of the internet, of apis and finally self awareness.

That is a herald for the future. This is despite all the rough imperfections chatGPT has. You are downplaying it. I'm sorry.

The whole statistical thing is a misnomer. If the output of chatGPT can be modelled as a statistical phenomenon then so can the output of the human brain. It's a distraction. What is going on here is indeed creativity, awareness and imagination, if you want to call it a statistical phenomenon that's fine, you're just saying all of intelligence can be modeled as a statistical phenomenon.