> doesn't mean the coding interview is useless or that ChatGPT could actually do the job

Aren't these kind of mutually exclusive, at least directionally? If the interview is meaningful you'd expect it to predict job performance. If it can't predict job performance then it is kind of useless.

I guess you could play some word games here to occupy a middle ground ("the coding interview is kind of useful, it measures something, just not job performance exactly") but I can't think of a formulation where this doesn't sound pretty silly.

It’s possible that being able to pass the interview is indicative of performance in humans but not in LLMs.

Humans think differently from LLMs so it makes sense to interpret the same signal in different ways.

We've been saying for years these interviews are not predictive of job performance. Here's the proof.

Nothing you do in an interview like this resembles day to day work in this field.

For what it's worth, when I ask these kinds of questions (rarely anymore), I'm looking more at how the problem is solved, not what the solution is.

A wrong answer with good thinking is better than a correct answer with no explanation.

Finding a single counterexample does not disprove correlation or predictive ability. A hiring test can have both false positives and false negatives and still be useful.

I don’t think your militant attitude helps them understand any better.

I don't think I had a militant attitude, but I do think saying, "I don't understand..." rather than "I disagree with..." puts a sour note on the entire conversation.

You literally went to their profile and called them out about how they should be able to understand something you’re describing as so easy to understand.

Yeah, what is the problem with that? They engaged dishonestly by claiming they didn't understand something, why should I do anything other than call them on that?

OK — just don’t be surprised when people think you’re being a jerk because you didn’t like the words someone chose. I’d assert you’re acting in bad faith more than the person you responded to.

I just... how is what you're doing here different from what I was saying, other than you're explicitly calling me names?

> engaged dishonestly by claiming [someone used a word they didn’t like] why should I do anything other than call them on that?

Have a great day.

...what? I'm so confused.

It’s really very easy to understand. When someone gives you the same crap back that you just got done giving someone, you don’t like it and act like that shouldn’t happen.

Did I say I didn't "like" (I'd use the word "appreciate") it, or that I didn't think it should happen? If so, could you please highlight where?

I just see, in what you're doing, a wild lack of self awareness. You're criticizing me for doing to someone else a milder version of what you're trying to do to me now; I'm genuinely confused how you can't see that, or how you could possibly stand the hypocrisy if you do understand that.

I feel that you’re being dishonest if you are saying you’re confused or don’t understand my point.

I can't change how you feel.

I was just responding to the ‘water is blue’ level dismissal of a seemingly clear advancement. Sorry it seemed like a sour note— not my intent:)

All good, it's just hard sometimes to understand intent online, maybe I overreacted to what I perceived as bad faith engagement.

Words are hard!

I'll try to phrase it so that even someone who is not a research scientist (?) can understand. I'm not one, whatever that means.

Let's define the interview as useful if the passing candidate can do the job.

Sounds reasonable.

ChatGPT can pass the interview and can't do the job.

The interview is not able to predict the poor working performance of ChatGPT and it's therefore useless.

Some of the companies I worked for hired ex fang people as if it was a mark of quality, but that hasn't always worked out well. There is plenty of people getting out of fangs having just done mediocre work for a big paycheck.

> Let's define the interview as useful if the passing candidate can do the job.

The technical term for this is "construct validity", that the test results are related to something you want to learn about.

> The interview is not able to predict the poor working performance of ChatGPT and it's therefore useless.

This doesn't follow; the interview doesn't need to be able to exclude ChatGPT because ChatGPT doesn't interview for jobs. It's perfectly possible that the same test shows high validity on humans and low validity on ChatGPT.