That approach has always ended badly in my experience because your manual installs will require more care to remove than the Hombrew ones. My usual recommendation is to install anything which supports it using --HEAD which will keep things tracked and make it trivial to either completely uninstall or simply use "brew reinstall" packages after the upstream stabilizes.