I'm not doing anything with HN. I expect the server to emit the proper line breaks because that's what it should do, especially if it's going to be made freely available and used by others.

There's no reason that such a simple bug should have made it into or remained in the production source for so long. It's laziness for laziness' sake.

I would think that HN wouldn't really care about someone crawling them as long as they were respectful about it.

Also, as far as writing crawlers/scrapers goes, a header-handling error is one of the least annoying things you can run into.