Actually, if you are in the audience of a conference and there is someone talking in the front, any disturbance to the people around you is at least impolite. And if anyone should leave it's not the one who is silently sitting there and listening.