I also work at Google and I agree with the general sentiment that AI completion is not doing engineering per se, simply because writing code is just a small part of engineering.
However in my experience the system is much more powerful than you described. Maybe this is because I'm mostly writing C++ for which there is a much bigger training corpus than JavaScript.
One thing the system is already pretty good at is writing entire short functions from a comment. The trick is not to write:
function getAc...
But instead:
// This function smargls the bleurgh
// by flooming the trux.
function getAc...
This way the completion goes much farther and the quality improves a lot. Essentially, use comments as the prompt to generate large chunks of code, instead of giving minimum context to the system, which limits it to single line completion.
This type of not having to think about the implementation, especially in a language that we've by now well-established can't be written safely by humans (including by Google's own research into Android vulnerabilities if I'm not mistaken), at least with the current level of LLM, worries me the most
Time will tell whether it outputs worse, equal, or better quality than skilled humans, but I'd be very wary of anything it suggests beyond obvious boilerplate (like all the symbols needed in a for loop) or naming things (function name and comment autocompletes like the person above you described)
It isn't something I worry about at all. If it doesn't work and starts creating bugs and horrible code, the best places will adjust to that and it won't be used or will be used more judiciously.
I'll still review code like I always do and prevent bad code from making it into our repo. I don't see why it's my problem to worry about. Why is it yours?
Functional bugs in edge cases are annoying enough, and I seem to run into these regularly as a user, but there's yet another class of people creating edge cases for their own purposes. The nonchalant "if it doesn't work"... I don't know whether that confirms my suspicion that not all developers are aware of (as a first step; let alone control for) the risks
It generates bugs in pretty similar ways. It’s based on human-written code, after all.
Edge cases will usually be the ones to get through. Most developers don’t correctly write tests that exercise the limits of each input (or indeed have time to both unit test every function that way, and integration test to be sure the bigger stories are correctly working). Nothing about ai assist changes any of this.
(If anybody starts doing significant fully unsupervised “ai” coding they would likely pay the price in extreme instability so I’m assuming here that humans still basically read/skim PRs the same as they always have)
As a security consultant, I think I'm aware of security risks all the time, also when I'm developing code just as a hobby in spare time. I can't say that I've come across a lot of stackoverflow code that was unsafe. It happened (like unsafe SVG file upload handling advice) and I know of analyses that find it in spades, but I personally correct the few that I see (got enough stackoverflow rep to downvote, comment, or even edit without the user's approval though I'm not sure I've ever needed that) and the ones found in studies may be in less-popular answers that people don't come across as often because we should be seeing more of them otherwise, both personally and in the customer's code
So that's not to say there is nothing to be concerned about on stackoverflow, just that the risk seems manageable and understood. You also nearly always have to fit it to your own situation anyway. With the custom solutions from generative models, this is all not yet established and you're not having to customise (look at) it further if it made a plausible-looking suggestion
Perhaps this way of coding ends up introducing fewer bugs. Time will tell, but we all know how many wrong answers these things generate in text as well as what they were trained on, giving grounds for worry—while also gathering experience, of course. I'm not saying to not use it at all. It's a balance and something to be aware of
I also can't say that I find it to be thoughtless when I look for answers on stackoverflow. Perhaps as a beginning coder, you might copy bigger bits? Or without knowing what it does? That's not my current experience, though
This is a good idea even outside of Google, with tools like copilot and such.
Often when I don't know exactly what function / sequence of functions I need to achieve a particular outcome, I put in a comment describing what I want to do, and Copilot does the rest. I then remove the comment once I make sure that the generated code actually works.
I find it a lot less flow-breaking than stackoverflow or even asking an LLM.
It doesn't work all of the time, and sometimes you do have to Google still, but for the cases it does work for, it's pretty nice.
Why remove the comment that summarises the intent for humans? The compiler will ignore your comment anyway, so it's only there for the next human who comes along and will help them understand the code
Doesn't need an explanation, but when working in a language I don't know well, I might not remember whether I'm supposed to call orderBy on the query or on the ORM module and pass query as the argument, whether the kwarg is called "field" or "column", whether it wants a string or something like `User.name` as the column expression, how to specify the ordering and so on.
Like he says, the "comment" describes what he wants to do. That's not what humans are interested in. The human already knows "what he wants to do" when they read the code. It's the things like "why did he want to do this in the first place?" that is lacking in the code, and what information is available to add in a comment for the sake of humans.
Remember, LLMs are just compilers for programming languages that just so happen to have a lot of similarities with natural language. The code is not the comment. You still need to comment your code for humans.
> Like he says, the "comment" describes what he wants to do. That's not what humans are interested in.
When I'm maintaining other people's code, or my own after enough time has gone by, I'm very interested in that sort of comment. It gives me a chance to see if the code as written does what the comment says it was intended to do. It's not valuable for most of the code in a project, but is incredibly valuable for certain key parts.
You're right that comments about why things were done the way they were are the most valuable ones, but this kind of comment is in second place in my book.
It's often unnecessarily verbose. If you read a comment and glance at the code that follows, you'll understand what it is supposed to do. But the comment you're giving as an instruction to an LLM usually contains information which will then be duplicated in the generated code.
I see. Might still be good to have a verbose comment than no comment at all, as well as a marker of "this was generated" so (by the age of the code) you have some idea of what quality the LLM was in that year and whether to proofread it once more or not
I tried making a meme some months ago with exactly this idea, but for emails. One person would tell an LLM "answer that I'm fine with either option" and sends a 5 KB email, in response to which the recipient receives it and gets the automatic summary function to tell them (in a good case) "they're happy either way" or (in a bad case) "they don't give a damn". It didn't really work, too complex for meme format as far as my abilities went, but yeah the bad translator effect is something I'm very much expecting from people who use an LLM without disclosing it
If someone is going to use an LLM to send me an email, I'd much rather them just send me the prompt directly. For the LLM message to be useful the prompt would have included all the context and details anyway, I don't need an LLM to make it longer and sound more "professional" or polite.
Not necessarily. Your prompt could include instructions to gather information from your emails and address book to tell your friend about all the relevant contacts you know in the shoe industry.
Wow, I love good, original programming jokes like these, even the ideas of the jokes. I used to browse r/ProgrammerHumor frequently, but it is too repetitive -- mostly recycled memes and there is anything new.
(No need to Orientalize to defamiarize, especially when a huge fraction of the audience is Chinese, so Orientalizing doesn't defamiliarize. Game of Whispers or Telephone works fine.)
Chinese-Americans, at least, call it a game of Telephone, like everyone else in the English-speaking world except for the actual English.
We call it “Telephone” because “Chinese Whispers” not only sounds racist, it is also super confusing. You need a lot of cultural context to understand the particular way in which Chinese whispers would be different from any other set of whispers.
I happened to re-read this, and to be clear, I'm not Chinese-American. the "we" there means "everyone else in the English-speaking world except for the actual English."
I can guarantee you there is more publicly accessible javascript in the world than C++.
Copilot will autocomplete entire functions as well, sometimes without comments or even after just typing "f". It uses your previous edits as context and can assume what you're implementing pretty well.
I can guarantee you that the author was referencing code within Google. That is, their tooling is trained off internal code bases. I am imagining c++ dwarfs javascript.
They're probably talking about Closure Compiler type annotations [0], which never really took off outside Google, but (imo) were pretty great in the days before TypeScript. (Disclosure: Googler)
I find writing code to be almost relaxing plus that's really a tiny fraction of dev work. Not too excited about potential productivity gains based purely on authoring snippets. I find it much more interesting on boosting maintainability, robustness and other quality metrics (not focusing on quality of AI output, actual quality of the code base).
I frequently use copilot and also find that writing comments like you do, to describe what I expect each function/class/etc to do gives superb results, and usually eliminates most of the actual coding work. Obviously it adds significant specification work but that’s not usually a bad thing.
Has anyone made a coding assistant which can do this based off audio which I’m saying out loud while I’m typing (interview/pairing style), so instead of typing the comment I can just say it?
I had some success using this for basic input, but never took it very far. It's meant to be customizable for that sort of thing though: https://talon.wiki/quickstart/getting_started/ (Edit: just the voice input part)
However in my experience the system is much more powerful than you described. Maybe this is because I'm mostly writing C++ for which there is a much bigger training corpus than JavaScript.
One thing the system is already pretty good at is writing entire short functions from a comment. The trick is not to write:
But instead: This way the completion goes much farther and the quality improves a lot. Essentially, use comments as the prompt to generate large chunks of code, instead of giving minimum context to the system, which limits it to single line completion.