User:Sylvain Ribault/WJS essay

Academics and Wikipedia: the WikiJournal experiment

Introduction

Wikipedia is a wonderful platform for the dissemination of scientific knowledge. It combines a number of powerful features in a way that is not found in other platforms or media:

a vast network of hyperlinks, with no artificial disciplinary boundaries,
a free and open platform,
collaborative editing and version control,
a talk page for each content page.

Wikipedia is already used a lot by students and researchers, in particular for orienting oneself outside one's speciality. And fortunately, guidelines that articles should not be too technical are not strictly enforced.

However, not many academics write in Wikipedia, and the quality of Wikipedia articles on scientific subjects is very unequal. For example, according to the WikiProject Physics, most physics articles that are considered of top importance, are below Good Article quality. Academics devote much time and effort to writing journal articles that often have few (if any) readers, while neglecting Wikipedia, where even obscure technical articles can reach thousands of readers.

A plausible explanation for the dearth of academic editors, is that academic careers are built on journal articles, not on Wikipedia articles. Time spent editing Wikipedia is time lost for one's career. WikiJournals offer a solution to this problem, by publishing Wikipedia articles as bona fide academic publications, whose authors can therefore receive credit. WikiJournals actually aim to be much more than that: they also publish original research (unlike Wikipedia), they disseminate preprints, and they implement recognized best practices in academic publishing, starting with libre open access.

The three WikiJournals (Medicine, Science and Humanities) have existed for a few years, and have published a few dozen articles. Not a long time, not many articles, but already enough for drawing conclusions that were not obvious before (to me at least): conclusions about scientific articles in Wikipedia, reactions of academics to Wikipedia, and the WikiJournal experiment itself.

(These conclusions are based on my experience as a member of the editorial board of the WikiJournal of Science since November 2017, and on data provided by the Editor in Chief, Thomas Shafee.)

WikiJournals: how do they work? what are their priorities?

A journal is made of three elements:

Rules: a set of practices, codified or informal, that define how articles are processed.
People: mainly the journals' editors, but also the authors and reviewers and, in the case of WikiJournals, members of the WikiJournal User Group.
Infrastructure: in the case of WikiJournals, websites hosted by WikiVersity, plus Google Groups mailing lists and Google Forms documents.

It is the rules that matter most, and this is what I will focus on. Let me show that WikiJournals' rules are both very flexible and very conservative: They are very flexible regarding the types of submitted articles and the peer review process:

the three journals taken together have an essentially unlimited scope,
they allow both double blind peer review and fully open peer review (i.e. making reviewers' names public),
they publish Wikipedia-style articles as well as original research,
they allow everyone to participate in peer review and even modify submitted articles.

To be sure, there are some robust features too: reviewers' comments are publicly viewable, and the journals are free to authors and readers. However, the lack of focus on a particular type of content and editorial process lead to convoluted procedures and rules. For example, the ethics statement includes rules on human research and animal research that are irrelevant to encyclopedic review articles, and the submission form asks authors to make half a dozen procedural choices.

On the other hand, the rules are very conservative regarding the core principles of peer review, in particular the roles of editors, reviewers and authors. These roles are taken straight from traditional academic publishing, with minimal adaptation to Wikipedia practices. In particular, WikiJournals require the submission of an article from Wikipedia to be done by an author who did substantial work on it, while acknowledging the other (possibly pseudonymous) authors as "et al". The only manifestation of the blurring and mixing of the roles that characterizes Wikipedia is that editors and reviewers can modify the submitted article like everyone else. This is because the WikiJournals' conservative principles are implemented on top of a MediaWiki infrastructure, i.e. the same platform as Wikipedia, which does not distinguish between the roles.

These conservative principles allow WikiJournals to fully play the institutional game of academic journals: indexing by various services, formal code of conduct, inclusion in databases and/or guilds of open access publishers, compliance with Plan S: this type of issues pretty much dominates the discussion board. A more radical rethinking of authorship and of the editorial process might have limited institutional recognition to indexing by Google Scholar, but that road was not taken.

Therefore, WikiJournals are an awkward combination of academic journals with all their bells and whistles, and Wikipedia-style content and techniques. This makes for clunky infrastructure and procedures, which are probably not scalable to more than a few dozen articles per year. As they now function, the WikiJournals are not a revolution, but an experiment in scientific communication.

Academics' attitude towards Wikipedia and WikiJournals

Since Wikipedia is universally known, and peer review is part of academics' work, it is not particularly difficult to convince academics to review Wikipedia articles. This does not mean it is easy: finding good reviewers is a hard task even for well-established journals with a good reputation.

In my view, two features of the WikiJournals make it a bit more difficult to recruit reviewers:

WikiJournals are not Wikipedia, and a Wikipedia article submitted to a WikiJournal is first cloned to the journal's site on WikiVersity. This can confuse potential reviewers, and they must be prevented from switching from their naturally positive reaction towards the well-appreciated Wikipedia, to the instant rejection of yet another sollicitation from a new and unknown journal.
WikiJournals are broad scope journals, and their current editorial boards are too small to cover all fields and subfields of science: the WikiJournal of Science has 33 editors, we would need thousands. This means that editors can have to work in subfields of which they are not specialists. This makes their judgement less sound, and most crucially this makes it harder to recruit reviewers, as editors have to ask people whom they do not know.

Nevertheless, as far as I know, WikiJournals have been able to find enough good reviewers. The main struggle is rather to get a decent volume of submissions from authors. Of course, if there were already many good Wikipedia articles written by academics, we would not need WikiJournals in the first place. More submissions may come later, after WikiJournals become well-known, and after the newly incentivized authors have time to write in Wikipedia. Still, given the vastness of Wikipedia, it is disappointing that the WikiJournal of Science had only 24 submissions so far.

One natural way to suggest that a Wikipedia article be submitted is via that article's Talk page. We have written a standard message to this effect, and posted it to a few Talk pages. Only a few, because there are not so many Wikipedia articles that seem of good enough quality according to academic criteria. This particular effort did not lead to any submission so far. (We do not even know whether the main contributors of the targeted articles are academics.)

Can Wikipedia articles pass peer review?

I regularly consult Wikipedia articles on topics that do not belong to my field of research, and find them very helpful. And well-known studies show that Wikipedia compares favorably to traditional encyclopedias written by specialists. At the start of the WikiJournal experiment, I expected that peer review of submissions from Wikipedia would only result in minor improvements, and be mainly useful for officially validating the authors' work.

This naive expectation was destined to be shattered by the work on Surface tension, a submission from Wikipedia for which I was one of the editors in charge. This is a Good Article, having passed a formal Wikipedia procedure to that effect, and begin considered as meeting the criteria. Only a small minority of Wikipedia articles are Good Articles, and they tend to be clearer and better written than most academic articles.

It was not easy to organize peer review. There was no specialist of the relevant kind of physics on the WikiJournal's editorial board, and I did not know any. There were even suggestions that the review could be done by a non-specialist physicist -- and to non-specialist physicists on the board, myself included, the article looked quite good. Nevertheless, we managed to find good specialist reviewers. The resulting reports were surprisingly harsh, with one report concluding:

In summary, this article is flawed in so many respects that it will be a disservice to readers not already experts in the subject matter.

We then recommended a major revision of the article, but the authors never undertook it, and the article was declined.

Another submitted Good Article that received harsh reviews is Dyslexia. The bulk of the submissions from Wikipedia that were eventually published are Feature Articles, the highest echelon in the Wikipedia hierarchy. On the other hand, the WikiJournal of Science also published the article on Spaces in mathematics, which is not even a Good Article, but was written by well-known mathematicians and is of high academic quality.

This prompted me to have a more critical look at physics articles rated Good Article, as listed by the WikiProject Physics. These articles typically look good, but who knows how they would be judged by specialist reviewers? Review by academic specialists could probably help improve these articles a lot. But for this to occur, we need committed authors. Failing that, we need the reviewers and/or editors to perform the improvements themselves, but this would bring WikiJournals much further from traditional academic journals than they have gone so far.

To be fair, a Wikipedia article can be very useful even if it falls far short of academic standards. Some reasons for that:

All the links to and from the article are very valuable in themselves.
Reflecting the most recent research is not crucial.
Even rigor and accuracy are not essential to readers who only want to orient themselves in an unfamiliar subject.

And it is a bit artificial for the same article to be subjected to two distinct review processes with different goals and criteria. Academics can certainly bring a lot to Wikipedia, but maybe not by doing peer review in the traditional sense.

Some modest proposals

Here are a few suggestions for WikiJournals, or other initiatives aimed at making academics engage with Wikipedia.

Do not duplicate articles: do all the work on the original Wikipedia article, do not have a WikiVersity version. Aims: reducing complexity, saving work, and not confusing authors and reviewers.
Make authors optional: Wikipedia articles could be submitted by editors, reviewers, or others. There could be a process of nomination and votes to decide which articles to prioritize, possibly coordinated with disciplinary WikiProjects. Authors could also be optional for revising the article, as anybody can do it.
Focus on reviewers and their work: the main outcomes could be the reviews themselves rather than the Wikipedia articles. The collection of the reviews and comments could be published as an article, with its DOI and PDF, and a title of the type "Review of the Wikipedia article on X". Anonymity would no longer be an option for reviewers. If this leads to a version of the Wikipedia article that satisfies the reviewers, that version can be published too. Otherwise, just link the review from Wikipedia for future reference.
Be relaxed about official recognition: being recognized as a full-fledged traditional academic journal is not vital. It is more important to remain experimental long enough for finding what really works.