Sometimes the IRS is wrong, too, and you just mail them proof. Like if you forget to report your cost basis for a stock sale, they'll send you a letter and say "We're assuming the cost basis is zero, so here's what you owe!" and you just respond back with a letter that says "This is the cost basis, and here's a copy of my brokerage statement." and that's that. I've found dealing with the IRS actually quite straightforward and non-scary. As long as you have your paperwork in order. If you're careless about your records, then you're going to have a bad time.
Exactly this. Is what I meant by them working with you. They aren't playing some devilish game of trying to trick you into paying more. They tell you why they say you owe something and what it would take to get them to change that. And are fine if you need time to find it.
If you are actively trying to avoid the calls, I'm sure that is probably not a good idea. But, that is basically in the "actively trying to commit fraud" territory, at that point. No?