Hi Peter – I claim these do very often get by peer review (that’s precisely why people engage in this behavior – if it didn’t get through peer review, there would be little point). Additionally, this behavior, I believe, does distort scientific knowledge because upon a cursory reading of these papers (or, worse still, just reading the abstract) they often seem like they showed something they didn’t show (because if they didn’t seem that they showed something they didn’t show, it would be obvious that they showed something of little to no value). In that regard, it is much like when p-hacking occurs (resulting in false positives) because it leads the reader to a false impression about what’s true.

I really became aware of Importance Hacking occurring as I worked on replicating papers. When replicating, we rebuild the study from scratch in all its details, engaging with the exact wording of every question, the precise presentation of materials, etc., which is a much closer investigation than reviewers typically do. It is far easier to spot Importance Hacking when replicating than when reviewing.

Besides, I don’t see how it can both be true that this importance hacking is tricking the reviewers into giving priority to the wrong papers but yet you can see through it to notice it’s happening so much as to threaten science.

We can discover p-hacking that the reviewers couldn’t notice because of replication failures and by looking at statistical properties of many articles. The only way to see this problem is to observe the same information the reviewers saw and second guess them. So how can you distinguish between this and the theory that they do see through it and it’s just that there is a fair bit of disagreement about what’s important/publishable?

Besides, I’m not seeing the alternative. If you try to say that scientists shouldn’t make their work sound as interesting/important/etc.. as possible how could you possibly create an enforceable rule? Whoever was willing to push the line the most would have an advantage.

At least if everyone tries to make their work sound as important as possible then it’s all competing on a fairly level playing field.

What’s troubling about p-hacking is that it creates a false/misleading scientific record. The problem here seems to be, at worst, that we might not promote the people with the most important results but that’s largely a matter of luck anyway.

Even your examples don’t really seem that bad. Maybe grit is really just conscientiousness (I suspect it is). But maybe it’s not and that possibility makes it interesting and even if not it’s still good work. Yah, sometimes a significant result isn’t actually meaningful in the real world but if the scientists reading the paper can’t tell the difference the problem isn’t importance hacking since they’ll have the same issue interpreting their own data.