Seems to me like a lot of these are solved by a dedicated search engine?

I see it in this business all the time how people try to reinvent the wheel and end up writing their own search engine, thinking it's just another small in-house project, but they quickly run into the difficult problems like, well, these.