That's been my experience with MacPorts, which does have some binary packages, but much lower coverage than a distribution like Debian, so fallback building from source is common. It can be really annoying when you just want to install some utility and it takes literally 3 or 4 hours to install, because somewhere in the dependency or build-dependency chain is some big and slow-to-compile thing like GTK, X.org, or a newer GCC.