That seems unfair. He is (or at least was when he was doing it -- he's 96 years old now) a very competent singer and pianist. He's not particularly trying to make a beautiful sound, but there's more to singing than that.
Well, yes, self-deprecating humour is a thing, but there's a difference between Tom Lehrer pretending he can't really sing and someone else claiming he can't really sing.
