That's pretty much what ember is now. Which is not a bad thing per se, but can be frustrating and was one of the reasons we chose React/Flummox over Ember; plugging in non-Ember specific tools into the build system becomes more difficult vs having a pipeline you have control over!
Any particular reason why you chose to roll your own workflow instead of using ember-cli? (in most use cases I read about, people already had this before ember-cli matured).
I want to use LiveScript, Emblem and Stylus with the build modules of my choosing. Also I don't need any of these code generators. Broccoli feels also like they are running their own stuff just because they can.
I use my own build-chain with Gulp/LiveScript, but I found that some Plugin-Devs mentioned that supporting non-CLI-users was getting harder[0]
Now I have the fear that Ember will become a huge Rails like code generator blob in the future :\