You don't necessarily need to target ES5.

What percentage of your users is on IE11? Are you making money from them? Can you serve only them a compiled bundle?