Sunday, February 17, 2013

The Future of Peer Review

This week's cover of The Economist.
A year ago, I told you what I think is the future of scientific peer review: Peer review that is conducted independently from the submission of a manuscript to a journal. You would get a report from an institution offering such a service, possibly some already existing publisher, possibly some new institution specifically created for this purpose. This report you could then use together with submission of your paper to a journal, but you could also use it with open access databases. You could even use it in company with your grant proposals if that seems suitable. I call it pre-print peer review.

I argued earlier that, irrespective of what you think about this, it's going to happen. You just have to extrapolate the present situation: There is a lot of anger among scientists about publishers who charge high subscription fees. And while I know some tenured people who simply don't bother with journal publication any more and just upload their papers to the arXiv, most scientists need the approval stamp that a journal publication presently provides: it shows that peer review has taken place. The easiest way to break this dependence on journals is to offer peer review by other means. This will make the peer review process more to the point and more effective.

The benefit of this change over other, more radical, changes that have been proposed is that it stays very close to the present model in that the procedure of peer review itself need not be changed. It's just the provider that changes.

I am thus really excited that the recent issue of Nature reports that one such service exists now and another one is about to be created:
The one that already exists is called Peerage of Science, based in Jyväskylä, Finland. Yeah, right, the Nordic people, they're always a little faster than the rest of the world. Peerage of Science seems to have launched a little more than a year ago, but this is the first time I've heard of it. The one in the making is US based and the project is managed by a guy called Keith Collier.

Of course it's difficult to say whether such a change will catch on. Academia has a large inertia, and it depends a lot on whether people will accept independent reviews. But I am confident, so let me make a prediction, just for the fun of it: In 5 years there will be a dozen of such services, some run by publishers. In ten years, most of peer review will take place this way.

11 comments:

Robin Hanson said...

Your ten year claim sounds pretty unlikely. Care to make a bet?

Sabine Hossenfelder said...

I have two kids, I don't bet.

Phillip Helbig said...

I don't see it happening, especially with an author-pays model. I have always argued that it is good to separate questions about peer review from questions about journal prices. However, with some new type of journal which essentially uses arXiv as a repository, the new journal can do the peer review. I don't see a journal willing to let someone else decide what is acceptable, and if there is no guarantee of publication such external review won't work.

Sabine Hossenfelder said...

Hi Phillip,

If you have a report that documents your manuscript has been peer reviewed and that provides a quality assessment, you don't need to publish your paper in a journal anymore. That's the whole point. The "guarantee of publication" just isn't relevant anymore for the scientists. This means it's then in the publishers interests to convince the authors to publish with them. This makes economically dramatically more sense, because after all they are making money with your work. I agree that author-pay, if it's a fairly high charge, is not a good solution. But I am expecting there will be different models and also institutional subscriptions. Best,

B.

Phillip Helbig said...

If you have a report that documents your manuscript has been peer reviewed and that provides a quality assessment, you don't need to publish your paper in a journal anymore. That's the whole point. The "guarantee of publication" just isn't relevant anymore for the scientists."

This seems even worse. So there will be the paper, on arXiv or whatever, and independent of that some stamp of approval. That depends on the reputation of the stamp. Multiple companies? More confusion.

But the article says "Researchers waiting for their manuscript to emerge from multiple rounds of peer review as it bounces from journal to journal can easily get frustrated at the inefficiencies of the system. Soon they may have another option: paying for a fast, independent peer review that could travel with the paper from one journal to another." That sounds like people don't know where to submit it and /or start at the top and keep submitting it until it is accepted somewhere. That seems to be a much larger source of uncertainty.

Check out Peter Coles's proposal for a non-commercial, peer-reviewed journal for astrophysics. I don't see why peer review can't continue more or less as now (i.e. peers doing it for free). Perhaps the motivation to be quick will increase if people realize that their effort is not putting money into the pockets of publishers. And, of course, if someone works for months on something worthy of publication, there are only a few people competent enough to review it and if it could be checked in an hour or two then it is probably not worth doing anyway, so by definition almost anything worthy of peer review will take a while.

Phillip Helbig said...

"much larger source of uncertainty" --> "much larger source of inefficiency"

Phillip Helbig said...

"I have two kids, I don't bet."

One could argue that having children is a huge bet.

Uncle Al said...

Government outlaws keeping the money you earn lest you breach a monopoly of spending it unwisely. "It's just the provider that changes." Science is now a business too big to fail, not a discipline eliminating failure. Re-sourcing is little change. The past 45 years of quantum gravitation and particle theories are hectares published, not application. Peer review covering a naked Emperor's bottom covers its own bottom.

1) "This is not the solution we are seeking." That is discovery. 2) "There is no precedent for this experiment." "If it were of value, somebody else would have already done it." 3) "It contradicts theory." YES! Attack theory that rattles with experiment that screams. Peer review as practiced enforces staric status quo.

Zephir said...

/*..peer review that is conducted independently from the submission of a manuscript to a journal...*/

Scientists are people and the people are just desperate cheaters, who just want to minimize their effort required for achieving their salary (this is whole driving force of progress, after all). Your proposal goes against already growing trend of retractions, which are mostly based on failed/low quality peer-review. With your approach the printed journals would become effectively full of crap, which would become retracted at the on-line versions of journal later. The peer-review process is already superficial way too enough. I seriously doubt, such a situation would help the publishers of printed journals, not to say about the rest of scientific community.

StrangeRep said...

Hi Bee,
I had a look around the "Peerage of Science" website, but failed to gain a clear understanding of their funding model. It sounds like you can just submit a paper and, if it passes their processes, you then become a "peer" also. But I found their peer-review-or-peer-reviews model interesting, if it can be sustained.

Do you understand their funding model in any more depth than this? If so, can you enlighten us further?

(BTW, I also think that an author-pays model is deeply flawed.)

Cheers.

Sabine Hossenfelder said...

I got a comment by email from Janne-Tuomas Seppänen, one of the founders of Peerage of Science that I mentioned. Seems he had trouble posting, so I'm submitting his comment:

____________________________
"I am one of the founders of Peerage of Science, and the principal designer of the service (and also an ecology postdoc at University of Jyväskylä). I am sorry if the website is not clear enough about the funding model (a complete redesign is coming this month). In the meantime, I am happy to explain:

Peerage of Science is organized as a company. The service is absolutely free for scientists, the business model is to charge a fee from subscribing journals/publishers (just like the service providers of old-style submission systems, like ManuscriptCentral). Journals can purchase a quota, and then can access all peer review processes, and make direct publishing offers to authors. A quota unit is "consumed" only if authors accept a publishing offer, so journals only pay when peer review leads to published article.

Of course a small start-up has little actual sales in the beginning. The early stage of Peerage of Science has been funded by sponsorship agreements with universities. University of Jyväskylä, and University of Eastern Finland, are the first two supporters.

I will be happy answer any questions, and of course we welcome new scientists into Peerage of Science too (provided you can demonstrate a peer-reviewed publication record and identity)."
__________________________