Serious question as someone who has also tried these things out and not found them very useful in the context of working on a large, complex codebase in not python not javascript: when I imagine the amount of time it would take me to select some test cases, copy and paste them, and then think of a todo list or prompt to generate another case, even assuming the output is perfect, I feel like I’m getting close to the amount of time and mental effort it would take me to just write the test. In a way, having to ask in english for what I want in code for me adds an additional barrier: rather than just doing the thing I have to also think of a promptable description. Is that not a problem? Is it just fast enough that it doesn’t matter? What’s the deal?
I mean, for me personally, I am writing out the English TODO list while I am figuring out exactly what changes I need to make. So, the thinking and writing the prompt take up the same unit of time.
And in terms of time saved, if I am just changing string constants, it’s not going to help much. But if I’m restructuring the test to verify things in a different way, then it is helpful. For example, recently I was writing tests for the JSON output of a program, using jq. In this case, it’s pretty easy to describe the tests I want to make in English, but translating that to jq commands is annoying and a bit tedious. But o3-mini-high can do it for me from the English very well.
Annoying to do myself, but easy to describe, is the sweet spot. It is definitely not universally useful, but when it is useful it can save me 5 minutes of tedium here or there, which is quite helpful. I think for a lot of this, you just have to learn over time what works and what doesn't.
Thanks for the reply, that makes sense. jq syntax is one of those things that I’m just familiar enough with to remember what’s possible but not how to do it, so I could definitely see an LLM being useful for that.
Maybe one of my problems is that I tend to jump into writing simple code or tests without actually having the end point clearly in mind. Often that works out pretty well. When it doesn’t, I’ll take a step back and think things through. But when I’m in the midst of it, it feels like being interrupted almost, to go figure out how to say what I want in English.
Will definitely keep toying with it to see where I can find some utility.
That definitely makes a lot of sense. I think if you are coding in a flow state on something, and LLMs interrupt that, then you should avoid them for those cases.
The areas that I've found LLMs work well for are usually small simple tasks I have to do where I would end up Googling something or looking at docs anyway. LLMs have just replaced many of these types of tasks for me. But I continue to learn new areas where they work well, or exceptions where they fail. And new models make it a moving target too.
> I think if you are coding in a flow state on something, and LLMs interrupt that, then you should avoid them for those cases.
Maybe that's why I don't like them. I'm always in a flow state, or reading docs and/or a lot of code to understand something. By the time I'm typing, I already know what exactly to write, and thanks to my vim-fu (and emacs-fu), getting it done is a breeze. Then comes the edit-compile-run, or edit-test cycle, and by then it's mostly tweaks.
I get why someone would generate boilerplate, but most of the time, I don't want the complete version from the get go. Because later changes are more costly, especially if I'm not fully sure of the design. So I want something minimal that's working, then go work on things that are dependent, then get back when I'm sure of what the interface should be. I like working iteratively which then means small edits (unless refactoring). Not battling with a big dump of code for a whole day to get it working.
Yeah, I think it matters a lot what type of work you do. I have to jump between projects a lot that are all in different languages with a lot of codebases I'm not deeply familiar with. So for me, LLMs are really useful to get up-to-speed on the knowledge I need to work on new projects.
If I've got a clear idea of what I want to write, there's no way I'm touching an LLM. I'm just going to smash out the code for exactly what I need. However, often I don't get that luxury as I'll need to learn different file system APIs, different sets of commands, new jargon, different standard libraries for the new languages, new technologies, etc...
It does an ok job with C# but it’s generally outdated code I.e [required] as an annotation rather than as a keyword. Plus it generates some unnecessary constructors occasionally.
Mostly I use it for stupid templates stuff for which it isn’t bad. It’s not the second coming but it definitely speeds you up
