When ChatGPT appeared I have been working at a small startup for couple of months that was planning to hire another programmer.
CTO became extremely enthusiastic about ChatGPT and said that the programming would be a dying job and tried to show during presentation how good ChatGPT was and asked it to write a basic code related to our tasks. It produced total garbage that could not be used even as a starting point. CTO tried to prompt it to the needed directions, but it made things worse.
After the presentation I tried to search for the task from the presentation. It turned out there were very few StackOverflow or GitHub entries about it as the topic was rather specialized and ChatGPT tried to average those into the task.
In a month I and another recent hire departed from the company. And a year later the company was hiring programmers again.
Out of curiosity I repeated the task few times with different models all the time resulting in the same garbage.
So my rule of the thumb is that if a task generates a lot of search hit, then perhaps a LLM can average the knowledge into something reasonable. If not, averaging is not possible.
CTO became extremely enthusiastic about ChatGPT and said that the programming would be a dying job and tried to show during presentation how good ChatGPT was and asked it to write a basic code related to our tasks. It produced total garbage that could not be used even as a starting point. CTO tried to prompt it to the needed directions, but it made things worse.
After the presentation I tried to search for the task from the presentation. It turned out there were very few StackOverflow or GitHub entries about it as the topic was rather specialized and ChatGPT tried to average those into the task.
In a month I and another recent hire departed from the company. And a year later the company was hiring programmers again.
Out of curiosity I repeated the task few times with different models all the time resulting in the same garbage.
So my rule of the thumb is that if a task generates a lot of search hit, then perhaps a LLM can average the knowledge into something reasonable. If not, averaging is not possible.