>Sometimes you do have to put on headphones to cut down on the auditory distraction

That's exactly the problem. You are plugging inefficiencies that exist for no reason. Listening to music hampers my work because I focus too much on it. I just want a quiet room.