Depends: which is more precious: network bandwidth, or my development time.
High volume: custom-binary wins, just like you said. Low volume: REST/text wins.
There's something to be said for being able to try out your API in a web browser in 2 minutes. Compression also goes a way in reducing the volume of text, though not the latency of follow up queries that are too finely granular.
Small, sharp, tools swapping text is very unixy. Even if sometimes the response is a binary document :-)
The concept of poking about with HTTP to try and treat it vaguely like a TCP style transport protocol is rather silly.
If you think about how "wordy" HTTP is, along with the standard RESTful verbs, you'll start to think "hmmm, perhaps this isn't the best way."
Websockets and if you must a bit of JSON (binary if done properly is lower latency and much more efficient) should be the way forward.
yes its a bit harder, but you know, its smaller, better, more efficient.