Before people had a firm idea of homosexuality, men did show affection for other men as well.
It went like this: it is ok to hold hands with a man (because no one will think you are gay/because no one thinks anyone is gay) -> it is not ok to hold hands (because people will think you are gay) -> it is ok to hold hands (because it would not be a problem if you were gay)
It went like this: it is ok to hold hands with a man (because no one will think you are gay/because no one thinks anyone is gay) -> it is not ok to hold hands (because people will think you are gay) -> it is ok to hold hands (because it would not be a problem if you were gay)
and the last step is still ongoing...