I think the reality is that these AI output the "average" of what was in their training set, and people receive it differently depending on if they are below or above this average.

It's a bit like what happens with "illusion of knowledge" or "illusion of understanding". When one knows the topic, one can correct the output of AI. When one doesn't, one tends to forget it can be inaccurate or plain wrong.

They are a useful tool, but not 'incredibly useful'. The simple, repetitive code in this example is what they are good at. It's like 1% of what I do working on products. Writing code isn't even that impressive, the whole job is figuring out exactly what people want.

given that there's no standardized scale of usefulness, is the distinction between "useful" for one person vs "incredibly useful", when nothing concrete has been specified; is that distinction really the important thing here? both of you find it useful. I might go off on a long tangent about how I love my hammer, it's the best, and you'll think I'm ridiculous because it's just a hammer, but at the end of the day, we can both agree that the hammer is doing the job of driving in nails.