It’s absurd to claim that white men can’t ever under any circumstances have anything to say about the struggles of other people, but there are certainly people who make such a claim.
As one example, my father, an anthropologist/historian in his 60s who spent his whole career and life befriending, interviewing, translating, and writing about indigenous rural peasants in southern Mexico, and probably knows as much or more about their history and circumstances as anyone alive (in an intellectual sense, if not precisely as lived personal experience, since he was never himself a destitute peasant) was told by a 30-year-old recent Ph.D from a well-off family (who had literally never worked a job outside academia, and spent her entire life living in rich US suburbs) that he had no right to comment on the circumstances of indigenous rural peasants, “the subaltern” to use her term, because he he was a privileged member of the hegemonic class, whereas she could, as a Latina.
I was specifically speaking about people that pontificate about things they have no legitimate claim to have experienced. Being a Latin@ in the United States doesn't legitimize your claim to speak about rural peasants of Mexico (though, if your family has roots there you may have lingering first hand accounts that can color the discussion more than those without).
In fact, historians are one of the people I would consider a source, especially if their words are born from the interactions and studies done from interacting with real people. They make it their life's work to understanding the situation in which people live.
Also, everyone makes mistakes and/or embellishments. I make mistakes. People in their own experiences get caught up in those experiences instead of the truth. Historians get caught up in their own narrative of truthiness instead of truth. Skepticism and an open mind are great tools toward enlightenment.
So, yeah, sometimes white men can be more legitimate than others in their viewpoints. But far too often the voices of the privileged and uninformed drown out the voices of experience and truth. That's the tragedy.
As one example, my father, an anthropologist/historian in his 60s who spent his whole career and life befriending, interviewing, translating, and writing about indigenous rural peasants in southern Mexico, and probably knows as much or more about their history and circumstances as anyone alive (in an intellectual sense, if not precisely as lived personal experience, since he was never himself a destitute peasant) was told by a 30-year-old recent Ph.D from a well-off family (who had literally never worked a job outside academia, and spent her entire life living in rich US suburbs) that he had no right to comment on the circumstances of indigenous rural peasants, “the subaltern” to use her term, because he he was a privileged member of the hegemonic class, whereas she could, as a Latina.