Rust could still have a toolchain like DMD devoted to fast compilation with minimal optimization. It just doesn't, yet (and likely won't for quite some time, since the present advantages of having a single toolchain are fairly significant and Rust doesn't have a formal specification yet).
There are C compilers out there that work like this (like, say, the Plan 9 C compiler) and they rarely ever get used in practice, because GCC and clang -O0 are fast enough. I think making a second compiler backend just to eliminate the IR generation step isn't going to be the solution we want in the long run.
I think they are only fast enough because developers have not been exposed to better. In general, I would like to see more aggressive build modes for CI servers, and less aggressive modes for dev.
Multirust makes working with multiple toolchains on *NIX pretty great, but no formal spec can be a pain. Ruby doesn't have a spec, yet has multiple implementations, so it's not the _end_ of the world...