Eliminating the "ums" would lend an amount of polish disproportionate to the required effort. A friend once told me about a trick she learned from a course in public speaking: every time a student speaker said "um", the entire class would chant "um!" in response. With that kind of instant feedback, speakers quickly learned to pause silently instead of filling the space with sound. You could try this technique with a practice audience, or—if you were feeling really bold—even with a real one.