I use lots of versions of GHC across several architectures, and can't remember the last time I compiled GHC. I think you must be doing it wrong. It's not without its flaws, but `stack` can do a lot of the work for you too.
On some platforms that build from source haskell apps pull ghc as a dependency and build it from source as well. I try to avoid those apps but it is getting harder.