If it is massaged into desired results, then it will be invalidated by facts quite easily. Inversely, obfuscating things is also easy if you just provide the whole package and just say "see, you click on the button and you get the same result, you have proven that it is correct". No providing code means that people will redo their own implementation and come back to you when they will see they don't get the same results.

So, no, no need to invent that academics are all part of this strange crazy evil group. Academics are debating and are being skeptical of their colleagues results all the time, which is already contradictory to your idea that the majority is motivated by frauding.

Occams razor is simply that there are some good reasons why code is not shared, going from laziness to lack of expertise on code design to the fact that code sharing is just not that important (or sometimes plainly bad) for reproducibility, no need to invent that the main reason is fraud.

Ok, that's a bit naive now. The whole "replication crisis" is exactly the term for bad papers not being invalidated "easily". [1]

Beacuse - if you'd been in academia - you'd find out that replicating papers isn't something that will allow you to keep your funding, your job and your path to next title.

And I'm not sure why did you jump to "crazy evil group" - noone is evil, everyone is following their incentives and trying to keep their jobs and secure funding. The incentives are perverse. This willing blindness against perverse incentives (which appears both in US academia and corporate world) is a repeated source of confusion for me - is the idea that people aren't always perfectly honest when protecting their jobs, career success and reputation really so foreign to you?

[1]:https://en.wikipedia.org/wiki/Replication_crisis

That's my point: people here link the replication crisis to "not sharing the code", which is ridiculous. If you just click on a button to run the code written by the other team, you haven't replicated anything. If you review the code, you have replicated "a little bit" but it is still not as good as if you would have recreated the algorithm from scratch independently.

It's very strange to pretend that sharing the code will help the replication crisis, while the replication crisis is about INDEPENDENT REPLICATION, where the experience is redone in an independent way. Sometimes even with a totally perpendicular setup. The closer the setup, the weaker is the replication.

It feels like it's watching the finger who point at the moon: not understanding that replication does not mean "re-running the experiment and reaching the same numbers"

> noone is evil, everyone is following their incentives and trying to keep their jobs and secure funding

Sharing the code has nothing to do with the incentives. I will not loose my funding if I share the code. What you are adding on top of that, is that the scientist is dishonest and does not share because they have cheated in order to get the funding. But this is the part that does not make sense: unless they are already established enough to have enough aura to be believed without proofs, they will lose their funding because the funding is coming from peer committee that will notice that the facts don't match the conclusions.

I'm sure there are people who down-play the fraud in the scientific domain. But pretending that fraud is a good strategy for someone's career and that it is why people will fraud so massively that sharing the code is rare, this is just ignorance of the reality.

I'm sure some people fraud and don't want to share their code. But how do you explain why so many scientists don't share their code? Is that because the whole community is so riddled with cheaters? Including cheaters that happens to present conclusions that keep being proven correct when reproduced? Because yes, there are experiments that have been reproduced and confirmed and yet the code, at the time, was not shared. How do you explain that if the main reason to not share the code is to hide cheating?

I've spent plenty of time of my career doing exactly the type of replication you're talking about and easily the majority of CS papers weren't replicable with the methodology written down on the paper and on dataset that wasn't optimized and preselected by the papers author.

I didn't care about sharing code (it's not common), but independent implementation and comparison of ML and AI algorithms with purpose of independent comparison. So I'm not sure why you're getting so hung up on the code part: majority of papers were describing trash science even in their text in effort to get published and show results.

I'm sorry that the area you are exercising in is rotten and does not have the minimum scientific standard. But please, do not reach conclusion that are blatantly incorrect in areas you don't know.

The problem is not really "academia", it is that, in your area, the academic community is particularly poor. The problem is not really the "replication crisis", it is that, in your area, even before we reach the concept of replication crisis, the work is not even reaching the basic scientific standard.

Oh, I guess it is Occams Razor after all: "It's really strange seeing how many (academic) people will talk themselves into bizarre explanations for a simple phenomenon of widespread results hacking to generate required impact numbers". Occams Razor explanation: so many (academic) people will not talk about the malpractice because so many (academic) people work in an area where these malpractice are exceptional.

But what’s the point of the peer review process if it’s not sifting out poor academic work?

It reads as if your point is talking in circles. “Don’t blame academia when academia doesn’t police itself” is not a strong stance when they are portrayed as doing exactly that. Or, maybe more generously, you have a different definition of academia and it’s role.

I think sharing code can help because it’s part of the method. It wouldn’t be reasonable for omitting aspects of the methodology of a paper under the guise that replication should devise their own independent method. Explicitly sharing methods is the whole point of publication and sharing it is necessary for evaluating its soundness, generalizability, and limitations. izacus is right, a big part of the replication crisis is because there aren’t near as many incentives to replicating work and omitting parts of the method make this worse, not better.

Maybe for the audience here, it is useful to consider that peer review is a bit like scrum. It's a good idea, but it does not mean that everyone who say they do scrum does it properly. And when, in some situation, it does not work, it does not mean that scrum is useless or incorrect.

And, like "scrum", "academia" is just the sum of the actors, including the paper authors. It's even more obvious that peer review is done by other paper authors: you cannot really be a paper author and blame "academia" for not doing a good peer review, because you are one of the person in charge of the peer review yourself.

As for "sharing code is part of the method", it is where I strongly disagree. Reproducibility and complete description allowing reproducibility is part of the method, but keeping enough details blinded (a balance that can be subjective) is also part of the method. So, someone can argue that sharing code is in contradiction with some part of the method. I think one of the misunderstanding is that people cannot understand that "sharing methods" does not require "sharing code".

Again, the "replication crisis" can be amplified by sharing code: people don't replicate the experiment, they just re-run it and then pretend it was replicated. Replicating the experiment means re-proving the results in an independent way, sometimes even with an orthogonal setup (that's why CMS and ATLAS at CERN are using on purpose different technologies and that they are not allowed to share their code). Using the same code is strongly biased.

It seems you are conflating concepts, maybe because you take it personally which it shouldn’t be. The process can be broken, but that doesn’t mean the academic is bad, just that they are part of a broken process. Likewise if a scrum is a broken process, it will lead to bad results. If it isn’t “done properly” then we seem to be saying the same thing: the process isn’t working. As I and others have said, there are some misaligned incentives which can lead to a broken process. Just because it sometimes works doesn’t mean it’s a good process, anymore than a broken clock is still correct twice a day. It varies by discipline, but there seems to be quite a few domains where there is actually more bad publications than good. That signals a bad process.

As others have talked about here, sometimes it becomes impossible to replicate the results. Is it because of some error in the replication process, the data, the practioner, or is the original a sham? It's hard to deduce when there's a lot you can't chase down.

I also think you are applying an overly superficial rationalization as to why sharing code would amplify the replication issue. This is only true if people mindlessly re-run the code. The point of sharing it is so the code can be interrogated to see if there are quality issues. Your same argument could be made for sharing data; if people just blindly accept the data the replication issue would amplify. Yet we know that sharing the data is what led to uncovering some of the biggest issues in replication, and I don’t see many people defending hiding data as a contradiction in the publication process. I suspect it’s for the reasons others have already eluded to in this thread.

I'm not sure what you are saying. The peer review process works relatively well in the large majority of the scientific fields. There are problems but they are pretty anecdotal and are far from counterbalancing the advantages. The previous commenter was blaming the peer review process for "bad incentives that lead to bad science", but that is an incorrect analysis. The bad science in their field is mainly due to the fact that private interest and people with poor scientific culture are getting more easily into this field.

Also, let's not mix up "peer review" or "code sharing" and "bad publication" or "replication crisis".

I know people outside of science don't realise that, but publishing is only a very small element amongst the full science process. Scientists are talking together, exchanging all the time, at conferences, at workshops, ... This idea that a bad publication is fooling the domain experts does not correspond to reality. I can easily find a research paper mill and publish my made-up paper, but this would be 100% ignored by domain experts. Maybe one or two will have a look at the article, just in case, but it is totally wild to think that domain experts just randomly give a lot of credit to random unknown people rather than working with the groups of peers that they know well enough to know they are reliable. So, the percentage of "bad paper" is not a good metric: the percentage of bad papers is not at all representative of the percentage of bad papers that made it to the domain experts.

You seem to not understand the "replication crisis". The replication does not happens because the replicators are bad or the initial authors are cheating. There is a lot of causes, from the fact that science happens to the technology edge and that the technology edge is more tricky to reach, that the number of publications has increased a lot, that there is more and more economical interest trying to bias the system, to the stupid "publish or perish" + "publish only the good result" that everyone in the academic sector agree is stupid but exist because of non-academic people. If you publish scientifically interesting result that says "we have explored this way but found nothing", you have a lot of pressure from the non-academic people who are stupid enough to say that you have wasted money.

You seems to say "I saw a broken clock once, so it means that all clocks are broken and if you pretend it is not the case, it is just because a broken clock is still correct twice a day".

> This is only true if people mindlessly re-run the code. The point of sharing it is so the code can be interrogated to see if there are quality issues.

"Mindlessly re-running the code" is one extreme. "reviewing the code perfectly" is another one. Then there are all the scenario in the middle from "reviewing almost perfectly" to "reviewing superficially but having a false feeling of security". Something very interesting to mention is that in good practices, code review is part of software development, and yet, it does not mean that software have 0 bugs. Sure, it helps, and sharing the code will help too (I've said that already), but the question is "does it help more than the problem it may create". That's my point in this discussion: too many people here just don't understand that sharing the code create biases.

> Yet we know that sharing the data is what led to uncovering some of the biggest issues in replication,

What? What are your example of "replication crisis" where the problem "uncovered" by sharing the data? Do you mix up "replication crisis" and "fraud"? Even for "fraud", sharing the data is not really the solution, people who are caught are just being reckless and they could have easily faked their data in more subtle ways. On top of that, rerunning on the same data does not help if the conclusion is incorrect because of a statistical fluctuation in the data (at 95% confidence level, 5% of the paper can be wrong while they have 0 bugs, the data is indeed telling them that the most sensible conclusion is the one they have reached, and yet these conclusions are incorrect). On the other hand, rerunning on independent data is ALWAYS exposing a fraudster.

> and I don’t see many people defending hiding data as a contradiction in the publication process.

What do you mean? At CERN, sharing the data of your newly published paper with another collaboration is strictly forbidden. Only specific samples are allowed to be shared, after a lengthy approval procedure. But the point is that a paper should provide enough information that you don't need the data to discover if the methodology is sound or not.

>I'm not sure what you are saying.

I'm saying the peer review process is largely broken, both in the quality and quantity of publications. You have taken a somewhat condescending tone a couple times now to indicate you think you are talking to an audience unfamiliar with the peer review process, but you should know that the HN crowd goes far beyond professional coders. I am well aware of the peer review process, and publish and referee papers regularly.

>There are problems but they are pretty anecdotal

This makes me think you may not be familiar with the actual work in this area. It varies, but some domains show the majority (as many as 2/3rds) of studies have replication issues. The replication rates are lowest in complex systems, with 11% in biomedical being the lowest I'm aware of. Other domains have better rates, but not trivial and not anecdotal. Brian Nosek was one of the first that I'm aware of to systematically study this, but there are others. Data Colada focuses on this problem, and even they only talk about the studies that are generally (previously) highly regarded/cited. They don't even bother to raise alarms about the less consequential work they find problems with. So, no, this is not about me extrapolating from seeing "a broken clock once."

>it does not mean that software have 0 bugs

Anyone who regularly works with code knows this. But I think you're misunderstanding the intent of the code. It's not just for the referees, but the people trying to replicate it for their own purposes. As numerous people in this thread have said, replicating can be very hard. Good professors will often assign well-regarded papers to students to show them the results are often impossible to reproduce. Sharing code helps troubleshoot.

>So, the percentage of "bad paper" is not a good metric: the percentage of bad papers is not at all representative of the percentage of bad papers that made it to the domain experts.

This is a unnecessary moving of the goalposts. The thrust of the discussion is about the peer-review and publication process. Remember the title is "one of my papers got declined today" And now you seemingly admit that the publication process is broken, but it doesn't matter because experts won't be fooled. Except we have examples of Nobel laureates making mistakes with data (Daniel Kahneman), or high-caliber researchers sharing their own anecdotes (Tao and Grant) as well as fraudulent publications impacting millions of dollars of subsequent work (Alzheimers). My claim is that a good process should catch both low quality research and outright fraud. Your position is like an assembly line saying they don't have a problem when 70% of their widgets have to be thrown out because people at the end of the line can spot the bad widgets (even when they can't).

>What are your example of "replication crisis" where the problem "uncovered" by sharing the data?

Early examples would be dermatology studies for melanoma where simple bad practices were not followed, like balanced datasets. Or criminal justice studies that amplified racial biases or showed the authors didn't realize the temporal data was sorted by criminal severity. And yes, the most egregious examples are fraud, like the Dan Ariely case. That wasn't found until people went to the data source directly, rather than the researchers. But there are countless examples of p-hacking that could be found by sharing data. If your counter is that these are examples of people cheating recklessly and they could have been more careful, that doesn't make your case that the peer-review process works. It just means it's even worse.

>sharing the data of your newly published paper with another collaboration is strictly forbidden

Yup, and I'm aware of other domains that hide behind the confidentiality of their data as a way to obfuscate bad practices. But, in general, people assume sharing data is a good thing, just like sharing code should be.

>But the point is that a paper should provide enough information that you don't need the data to discover if the methodology is sound or not.

Again (this has been said before) the point in sharing is to aid in troubleshooting. Since we already said replication is hard, people need an ability to understand why the results differed. Is it because the replicator made a mistake? Shenanigans in the data? A bug in the original code? P-hacking? Is the method actually broken? Or is the method not as generalizable as the original authors led the reader to believe? Many of those are impossible to rule out unless the authors share their code and data.

You bring up CERN so consistently that I tend to believe you are looking at this problem through a straw and missing the larger context of rest of the scientific world. Yours reads as a perspective of someone inside a bubble.

I will not answer to everything because what's the point.

Yes, sharing the code can be one way to find bugs, I've said that already. Yes, sharing the code can help bootstrap another team, I've said that already.

What people don't realize is that reproducing from scratch the algorithm is also very very efficient. First, it's arguably a very good way to find bugs: if the other team does not have the exact same number as you, you can pinpoint exactly where you have diverged. When you find the reason, in the large majority of the case, it totally passed through several code reviewer. Reading a code thinking "does it make sense" is not an easy way to find bug, because bugs are usually in place where the code of the original author looked good when read.

And secondly, there is a contradiction in saying "people will study the code intensively" and "people will go faster because they don't have to write the code".

> Remember the title is "one of my papers got declined today"

Have you even read what Tao says? He explains that he himself have rejected papers and has probably generated similar apparently paradoxical situations. His point is NOT that there is a problem with paper publication, it is that paper rejection is not such a big deal.

For the rest, you keep mixing up "peer review", "code sharing", "replication crisis", ... and because of that, your logic just make 0 sense. I say "bad paper that turns out to have errors (involuntary or not) are anecdotal" and you answer "11% of the biomedical publication have replication problem". Then when I ask you to give example where the replication crisis was avoided by sharing the data, you talk about bad papers that turns out to have errors (involuntary or not).

And, yes, I used CERN as an example because 1) I know it well, 2) if what you say is correct, how on hell CERN is not bursting with fire right now? You are pretending that sharing code or sharing data is a good idea and part of good practice. If it is true, how do you explain that CERN forbid it and still is able to generate really good papers. According to you, CERN would even be an exception where replication crisis, bad paper and peer-review problem is almost existent (and therefore I got the wrong idea). But if it is the case, how do you explain that: despite not doing what you pretend will help avoiding those, CERN does BETTER?!

But by the way, at uni, I became very good friend with a lot of people. Some of them scientists in other discipline. We regularly have this kind of discussion because it is interesting to compare our different world. The funny part is that I did not really think of how sharing the code or the data is not such a big deal after (it still can be good, but it's not "the good practice"), I realise it because another person, a chemist, mentioned it.

>What people don't realize is that reproducing from scratch the algorithm is also very very efficient.

This is where we differ. Especially if the author shares neither the data or the code, because you can never truly be sure it's a software bug or a data anomaly or a bad method or outright fraud. So you can end up burning tremendous amounts of time investigating all those avenues. That statement (as well as others about how trivial replication is) makes me think you don't actually try to replicate anything yourself.

>there is a contradiction in saying "people will study the code intensively" and "people will go faster because they don't have to write the code".

I never said "people will go faster" because they don't have to write the code. Maybe you're confusing me with another poster. You were the one who said sharing code is worthless because people can "click on the button and you get the same result". My point, and maybe this is where we differ, is that for the ultimate goal is not to create the exact same results. The goal I'm after is to apply the methodology to something else useful. That's why we share the work. When it doesn't seem to work, I want to go back to the original work to figure out why. The way you talk about the publication process tells me you don't do very much of this. Maybe that's because of your work at CERN is limited in that regard, but when I read interesting research I want to apply it to different data that are relevant to the problems I'm trying to solve. This is the norm outside of those who aren't studying the replication crisis directly.

>I say "bad paper that turns out to have errors (involuntary or not) are anecdotal"

My answer was not conflating peer-review and code sharing and replication (although I do think they are related). My answer was to give you researchers who work in this area because their work shows it is far from anecdotal. My guess is you didn't bother to look it up because you've already made up your mind and can't be bothered.

>I ask you to give example where the replication crisis was avoided by sharing the data, you talk about bad papers that turns out to have errors

Because it's a bad question. A study that is replicated using the same data is "avoiding the replication crisis". Did you really want me to list studies that have been replicated? Go on Kaggle or Figshare or Genbank if you want example of datasets that have been used (and replicated), like CORD-19 or NIH-dbGaP or World Values Survey or any host of other datasets. You can find plenty of published studies that use that data and try to replicate them yourself.

>how on hell CERN is not bursting with fire

The referenced authors talk about how physics is generally the most replicable. This is largely because they have the most controlled experimental setups. Other domains that do much worse in terms of replicability are hampered by messier systems, ethical considerations, etc. that limit the scientific process. In the larger scheme of things, physics is more of an anomaly and not a good basis to extrapolate to the state of affairs for science as a whole. I tend to think you being in a bubble there has caused you to over-extrapolate and have too strong of a conclusion. (You should also review the HN guidelines that urge commenters to avoid using caps for emphasis)

>"sharing the code...but it's not "the good practice""

I'm not sure if you think sharing a single unsourced quip is convincing but, your anecdotal discussion aside, lots of people disagree with you and your chemist friend. Enough so that it's become a more and more common practice (and even requirement in some journals) to share data and code. Maybe that's changed since your time at uni, and probably for the better.

Rolling eyes.

> Especially if the author shares neither the data or the code

What are you talking about. In this example, why do you invent they are not sharing the data? That's the whole point.

> A study that is replicated using the same data is "avoiding the replication crisis"

BULLSHIT. You can build confidence by redoing the experience with the same data, but it is just ONE PART and it is NOT ENOUGH. If there is a statistical fluctuation in the data, both studies will conclude something false.

I have of course reproduced a lot of algorithm myself, without having the code. It's not complicated, the paper explains what you need to do (and please, if your problem is that the paper does not explain, then the problem is not about sharing the code, it's about paper badly explaining).

And again, my argument is "nobody share data" (did you know that some study also shares code? Did you know that I have occasionally shared code? Because, as I've said before, it can be useful), but that "some don't share data and yet are still doing very good, both on performance, on fraud detection or on replication".

For the rest, you are just saying "my anecdotal observations are better than yours".

But meanwhile, even Terence Tao does not say what you pretend he says, so I'm sure you believe people agree with you, but it does not mean they do.

>Rolling eyes.

Please review and adhere to the HN guidelines before replying again.

>why do you invent they are not sharing the data?

Because you advocated that very point. You: "some data is better not to share too" The point in sharing is that I want to interrogate your data/code to see if it's biased or misrepresented or prone to error if it doesn't seem to work for the specialized problem I am trying to apply it to. When you don't share it and your problem doesn't replicate, I'm left wondering "Is it because they have something unique in their dataset that doesn't generalize to my problem?"

>BULLSHIT.

Please review and adhere to the HN guidelines before replying again.

>It's not complicated

You can make this general claim about all papers based on your individual experience? I've already explained why your personal experience is probably not generalizable across all domains.

>you are just saying "my anecdotal observations are better than yours".

No, I'm saying the systematically studied, published, and replicated studies trump your anecdotal claims. I've given you some example authors, if you have an issue with their methods, delineate the problems explicitly rather than sharing weak anecdotes.

> Because you advocated that very point. You: "some data is better not to share too"

SOME data. SOME. You've concluded, incorrectly, that I was pretending that sharing data is not useful all the time, which is not at all what I've said.

> You can make this general claim about all papers based on your individual experience?

What? Do you even understand basic logic? I'm saying that I've observed SOME paper where sharing the code did not help. I'm not saying sharing the code never help (I've said that already). I'm just saying that people usually don't understand the real cause of the problem, and invent that sharing the code will help, while in fact doing other things (for example being more precise in the explanation) will solve the problem without having to pay for the unblinding that sharing the code generate.

Sure, one reason I say that is because of my experience, even if my observations are not at all limited to one field as I've exchanged on the subject with many scientists. But another reason is that when I discuss the subject, the people who overestimate the gain of sharing the code really have difficulties to understand the disadvantages in sharing the code.

Yourself, you seems to not understand what we need for a good replication. Replication is supposed to independently demonstrate, so we build up the confidence in the conclusions. Rerunning with the same data or the same code is not enough, because it does not prove that the conclusions will remain valid if we try with other data or other implementation. When you understand that, only then you understand that sharing the code has a price to pay.

By the way, it will also explain why CERN is doing something that, according to you, has absolutely no reason to exist except for cheating. Of course, if it was the case, intellectually honest scientists would all ask CERN to cancel these policies. They don't, because there are real reasons why scientists may prefer in some case to forbid sharing code and data (not just "I don't do it myself because I'm lazy", but "I don't do it because it's a specific rule, they explicitly say it's a bad thing to do it").

And, sure, maybe it is not everywhere. But it does not matter. It's a counter-example that demonstrates that your hypothesis does not work. If your hypothesis was true, what CERN does would not be possible, it would be obviously a bad move and would be attacked.

> I've given you some example authors, if you have an issue with their methods, delineate the problems explicitly rather than sharing weak anecdotes.

These studies do not conclude that sharing the code is a good solution. None of these studies are in contradiction with what I say.

Of course, from someone who think that saying "some data is better not to share too" and conclude that it means "data is better to never be shared", or that did not understood the point of Tao, I'm sure you are convinced they say that. They just don't.