It's not specifically the DOM modification but the way those frameworks are designed. With React you have hundreds of dependencies, which have hundreds of dependencies. You lost me right there.
I've looked into Vue.js, and they seem to indicate that it's easier to get up an running than react (using what you describe here). Are you suggesting vue.js and react work exactly the same?
I don't know because I don't use Vue. What you'd be missing if you dropped in the react library and did no compilation at all is the ability to use JSX. I'm guessing vue's argument is their attribute tags still work in their minimal deployment whereas with React you'd be manually wiring up things to the React object functions.