I was in the middle of a twitter conversation about publishing when it again hit me that everything could be so much better if publications would just be published by publicly funded journals.
The currently hot idea for changing science publishing for the better: ‘pre-reg’ (article in The Guardian). This concept is tailored to give less incentives for overselling your studies. You submit your unfinished project as a kind of preliminary manuscript and it can be registered for future publication, regardless of the outcome. The idea is to base the publication decision on scientific soundness rather than outcome spectacularity. Awesome! But! By reducing one problem we increase another one. And that is the problem of reviewers being asked to make prophecies on whether your work will or will not be of interest. We see this problem already in the infamous grant application process. Funding agencies and publishers alike want to have ‘only the best work’ funded/published, so they would want reviewers to only allow the most predictable work for ‘pre-reg’/funding. This is or could be a big cockblock for explorational studies that aren’t really based on rock solid hypotheses but on a rather general question and assumptions. When competing for the same ressources, then strictly hypothesis driven studies win. To make things worse, reviewers forced to make predictions start taking the applicant’s scientific heritage into account (‘if he was a postdoc in the lab Prof X then his projects can only be excellent’). And the problem I have with this is that if new studies can only be based on older studies and the ‘offspring’ of certain silverbacks is selected for, we facilitate inbreeding of scientific ideas and we miss opportunities for diversity. Michael Eisen once proposed a system that would adress the problem of premature crystal-ball judgement by turning around the funding system from ‘pre-reg’ to funding based on success rate. In other words: publications, citations and progress reports are used for post-hoc evaluation of your work and estimation of how worthy you are of funding.
So, we have two systems: publishing and funding. Both evaluate our work. The one system is doing it before we actually have results, the other one afterwards. Both pose problems and both were proposed to be adressed by turning the process around. I agree in both cases that the existing system is neither good for our reputation in public, nor for the scientific process itself.
In both systems yet another critique of the review process is that reviewers might simply not like the authors. Bands of scientists stick together and dismiss all the work of other scientists in the field, or stall competitors in order to publish their own studies earlier. This is competition at it’s worst: when it leads to fights outside the framework of the scientific method. So there is a demand for transparency in the review process to make sure only scientific and not political aspects are considered and an idea to solve this would be post-publication review. Meaning that one would publish a paper and while it is already visible to the world, reviewers would comment on it in public and the interested reader can reproduce the development of the article.
Finally, there is the pressure to open science to the public. Open access, open data – some researchers want to move away from journals completely and instead go forward to self-publishing to prevent any sort of copyright problems. And there would be neither publication fees nor paywalls: no sinking of public money into private publishers’ pockets anymore. This has become a possibility since the rise of the internet. Everybody can produce professionally edited documents and host them on their webspace. So everybody can be their own publisher. Which other problem would that solve? Well, you can just publish whatever you want, garanteed!
A lot of good things so far, right? So why isn’t my title ‘Go start your open access, open data, post-publication review self-publishing website! Pre-reg totally unnecessary!’? Because there are good things about centralized publishing and institutionalized editing and reviewing of papers. The recent boom of for-profit open access journals shows what happens when the threshold for publishing is lowered to the level of the author’s willingness to publish. I am hinting towards that article on how dinosaurs didn’t go extinct but were ‘traumatized’ by god… in a ‘peer reviewed’ journal on head traumata. I still don’t think the word ‘traumatized’ means what the author thinks it means, unless he pictured god striking the dinosaurs with a big hammer… Thor style. Kinda cool, but also kinda coocoo crazy. I don’t know what went wrong in the process of that journal, I don’t know if they usually do better work or not. But it made me aware of what kind of bullshit people are trying to publish in scientific journals. If we all can put up our own websites, publish our own papers and call it ‘science’ it means that all those crazy people who have no idea what science is, can do that, too. If it is being positively reviewed later, or not, doesn’t change the fact that it is already public and has the science-stamp on it.
The real advantage of journals today, is centralized archiving and quality control. I am not happy about the for-profit nature and the way science is evaluated in that context, but some process needs to be in place to ensure that only science gets labeled ‘science’ and that there is one single place (or a few places) you can go to and you can trust in. This also has the advantage that everybody gets the same chance to be found and read… and what would be even more important in a post-publication review system: being critisized.
So here my not-thought-through idea:
Why don’t the public funding agencies found and sustain independent not-for-profit journals? No publication fees or paywalls would be necessary. Since the public is pretty much the only customer of scientific journals, taxes already pay for all the costs. But we wouldn’t be paying the profit margins of for-profit journals and it would free scientific publishing from economic competition. The author’s willingness to pay the publication fee is no entry criterium anymore. So we have open access down. Obviously, a journal is a central archive, so another point down. What about post-publication peer-review and quality control? In the controlled environment of a journal we can install mechanisms for a a public, post-publication peer review, ensuring for instance that it is appropriately moderated by an editor who then decides to label an article as ‘not reviewed’, ‘under review’, ‘peer accepted’ or ‘peer rejected’, based on the discussion (I always picture traffic lights ;)). What about the whole pre-reg thing? The post-publication review makes pre-reg obsolete. If you publish everything and let the scientists sort it out, everything is pre-reg. Public, transparent review will make non-scientific considerations on what should be published much harder, and noone can stall publication – because it is out there immediately – and no commercial decisions would bias for novel and spectacular findings, so less incentive to rig your data to make it look nicer.
There is definitely more to say and discuss. So, please make yourself a home in the comment area 😉
P.S. I want to recognize Björn Brembs blogs on the topic, who argued pro public publishing before me (and others did before him):
@DennisEckmeier @chrisdc77 @leechbrain @deevybee @tom_hartley As I've been saying for years: https://t.co/xFZPMycFpE http://t.co/47zupLGwVi
— Björn Brembs (@brembs) July 28, 2013
@DennisEckmeier @chrisdc77 @leechbrain @deevybee @tom_hartley or http://t.co/wXYEz9Dy3p http://t.co/U1ROhcH1p7 http://t.co/o1kKiYeNY0
— Björn Brembs (@brembs) July 28, 2013
Government deciding what gets published in scientific journals has to be the worst idea I’ve heard in 2013. Great, let’s have the results published on climate and IQ genetics and every other controversial field where crazies on right and left have axes to grind now be vetted by political appointees, and debated by legislators!
‘Government deciding what gets published in scientific journals has to be the worst idea I’ve heard in 2013.’
I agree, that would be terrible. But that is not my idea. The idea is to publish everything before anybody makes a decision of any sort and have it funded by the public. In other words: the government gives funding to the institution, but can not decide on what is published.
Just want to note that I may have just read the most awesome sentence/use of the word “cockblock” ever:
“This is or could be a big cockblock for explorational studies, who aren’t really based on rock solid hypotheses but on a rather general question and assumptions.”
Thanks for that !!
huh, I am not a native speaker, so I hope it is not too embarrassing.
http://en.wikipedia.org/wiki/Cockblock
oh, yeah I know about that part 😉
“oh, yeah I know about that part ;)”
That’s why I thought it was a very creative use of the word (instead of using “obstacle” for instance) 😛
Regarding the post itself: “What about the whole pre-reg thing? The post-publication review makes pre-reg obsolete. If you publish everything and let the scientists sort it out, everything is pre-reg.”
I am a big fan of post-publication review. However, I think pre-registration helps with things that post-publication review doesn’t help with. If I understood things correctly pre-registration helps with some questionable research practises (like HARKing, p-hacking) and with low power of studies, which would not necessarily be solved by publishing everything.
@ A.S.:
‘pre-registration helps with some questionable research practises (like HARKing, p-hacking) and with low power of studies, which would not necessarily be solved by publishing everything.’
The pre-registration idea is that you are garanteed publication before the study is completed. This is supposed to help take away incentives to try oversell your results and making them look better than they are. A post-publication peer review system would also garantee publication. The transparency of the post-publication review is supposed to ensure the review is scientifically sound.
http://cdn.elsevier.com/promis_misc/PROMISpub_idt_Guidelines_cortex_RR_17_04_2013.pdf
page 4 “power analysis must be based on the lowest available or meaningful estimate of the effect size. The a priori power (1 ) must be 0.9 or higher for all proposed statistical tests.”
So if I understood this correctly, in the Cortex pre-registration model, you would have high power in the published study. In your model (publish everything) that would not necessarily be the case, or am I wrong? (e.g., scientists would/could publish a study with low power)
That’s why I stated “If I understood things correctly pre-registration helps with some questionable research practises (like HARKing, p-hacking) and with low power of studies, which would not necessarily be solved by publishing everything.”
Yes sorry, in post-publication review this can only be demanded post-hoc.
However, a journal can make policies on this without the need of pre-registering. You can simply make it a prequisite for moving an article to ‘peer accepted’, without pre-registering it. This, of course, puts the author in charge to either increase the power by doing more experiments or being more concerned about it beforehand.
This seems like such a simple thing to institute, as well. Research Councils in the UK are throwing huge sums of money at publishers to cover APCs for open access because of a new policy that all government-funded research must be OA. That taxpayer money is going straight into the pockets of Elsevier, Springer, and Wiley-Blackwell, who then charge university libraries (taxpayer-funded) for access to the material. It’s absolutely ludicrous…
I would suggest that each funding agency has its own journal, and that that journal publishes ALL of the work funded by that agency. The creation of new journals is a piece of cake and there is obviously a lot of money to cover it. Simple.
I agree, it should be easy. I started with the same idea that funding agencies should publish all the research they funded. However, after realizing some people are afraid certain political parties would try to influence the science, I am beginning to think that international solutions would be the better. For example a central journal group for research funded by European taxpayers.