> if the experiment can't be reproduced to be verified, how does the paper provide more proof than a blanket 'trust me'?
It doesn't, and it is a big "trust me". People review papers based on the merits of the idea and methodology, and then tend to trust the results. Of course, if the results are very "significant" (e.g. cold fusion), then it will be scrutinized more, people will fail to reproduce, and they will harass the author. 99% of papers don't fall in this category, though.
> isn't this letting politics come before science?
Yep. The games at play are often: "How do I write my paper in the most convincing way?" and "As a referee, this paper is hurting the research work I am currently doing. What is the best way to reject this paper?"
The extremely annoying part was I felt I was back to taking literature courses, where I'm graded on very subjective metrics. It was horrible, especially when all my work was extremely objective. However, the publishing system is not incentivized to be that objective.
Simple example: A colleague's paper was rejected because he explained a phenomenon using method A, and the referee complained there was no mention of method B. Method B was the hot topic of the day. Neither method A nor method B had good empirical data to support it - it was almost purely theoretical at that point. But that community was gravitating towards method B, and really did not want to see alternative explanations.
It doesn't, and it is a big "trust me". People review papers based on the merits of the idea and methodology, and then tend to trust the results. Of course, if the results are very "significant" (e.g. cold fusion), then it will be scrutinized more, people will fail to reproduce, and they will harass the author. 99% of papers don't fall in this category, though.
> isn't this letting politics come before science?
Yep. The games at play are often: "How do I write my paper in the most convincing way?" and "As a referee, this paper is hurting the research work I am currently doing. What is the best way to reject this paper?"
The extremely annoying part was I felt I was back to taking literature courses, where I'm graded on very subjective metrics. It was horrible, especially when all my work was extremely objective. However, the publishing system is not incentivized to be that objective.
Simple example: A colleague's paper was rejected because he explained a phenomenon using method A, and the referee complained there was no mention of method B. Method B was the hot topic of the day. Neither method A nor method B had good empirical data to support it - it was almost purely theoretical at that point. But that community was gravitating towards method B, and really did not want to see alternative explanations.