Isn't homebrew specifically supposed to build from source? e.g. the example on the homepage of a recipe is running ./configure && make on wget.
The fact that you installed the XCode CLI tools for that wget example to work when you first installed homebrew because homebrew itself requires it, and you only get Cargo the first time you get a rust dependency seems to be what you're really complaining about.
Homebrew tries to install binaries by default. (They call them bottles)
Building from source happens if a suitable 'bottle' isn't available, or when `--build-from-source` is specified with the install command.
I know cargo is installed only once, but I don't want cargo. I don't build Rust software myself, so I don't want to have it hanging out on my system taking up space purely just so I can have one or two useful programs that were written in Rust and depend on it. I'll just go with some other alternative.
The fact that you installed the XCode CLI tools for that wget example to work when you first installed homebrew because homebrew itself requires it, and you only get Cargo the first time you get a rust dependency seems to be what you're really complaining about.