I have an idea for a website that could improve some well-known difficulties around peer review system and "hidden knowledge" in mathematics. It seems like a low hanging fruit that many people must've thought about before. My question is two-fold:
Has someone already tried this? If not, who in the mathematical community might be interested in creating and maintaining such a project or is working on similar projects?
Idea
A website dedicated to anonymous discussions of mathematical papers by experts.
Motivation 1: Hidden knowledge
Wilhelm Klingenberg's "Lectures on closed geodesics" can be found in every university's math library. One of the main theorems in the book is the following remarkable result, a culmination of decades of work on Morse theory of the loop space by many mathematicians: Every compact Riemannian manifold contains infinitely many prime closed geodesics.
Unfortunately, there is a mistake in the proof. 44 years after the book's publication the statement is still a widely open problem. The reason I know this is because when I was in grad school I mentioned the book to my adviser and my adviser told me about it. If I tried to look for this information online I wouldn't find it (in fact, I still haven't seen it written down anywhere).
This is one of many examples of "hidden knowledge", information that gets passed from adviser to student, but is inaccessible to an outsider. In principle, a new Ramanujan can open arxiv.org and get access to the cutting edge mathematical research. In reality, the hidden knowledge keeps many mathematical fields impenetrable to anyone who is not personally acquainted with one of a handful of experts.
Of course, there is much more to hidden knowledge than "this paper form 40 years ago actually contains a gap". But I feel that the experts' "oral tradition" on papers in the field is at the core of it. Making it common knowledge will be of great benefit to students, mathematicians from smaller universities, those outside of North America and Europe, people from adjacent fields, to the experts themselves and to the mathematical progress.
Motivation 2: Improving peer review
Consider the following situations:
- You are refereeing a paper and get stuck on some minor issue. It will take the author 5 minutes to explain, but a few hours for you to figure it out on your own. But it doesn't quite feel worth initiating formal communication with the author through the editor over this and you don't want to break the veil of anonymity by contacting the author directly.
- You are being asked to referee a paper, but don't have time to referee the whole paper. On the other hand, there is a part of it that is really interesting to you. Telling the editor "yes, but I will only referee Lemma 5.3" seems awkward.
- You are refereeing a paper that is at the intersection of your field and a different field. You would like to discuss it with an expert in the other field to make sure you are not missing anything, but don't know a colleague in that area or feel hesitant revealing that you are a referee for this paper.
These are some of many situations where ability to anonymously discuss a paper with the author and other experts in a forum-like space would be helpful in the refereeing process. But also outside of peer review mathematicians constantly find small errors, fillable gaps and ways to make an old paper more understandable that they would be happy to share with the others. At the same time, they often don't have time to produce carefully polished notes that they would feel comfortable posting on arxiv, or if they post notes on their website they may not be easy to find for anyone else reading the paper. It would be helpful to have one place where such information is collected.
How will it work?
The hope is to continue the glorious tradition of collaborative anonymous mathematics. One implementation can work like this:
Users of the website can create a page dedicated to a paper and post questions and comments about the paper on that page. To register on the website one needs to fill in a form asking for an email and two links to math arxiv papers that have the same email in them (this way registration does not require verification by moderators) and choose their fields of expertise. When a user makes a comment or question only their field/fields are displayed.
Best Answer
I'm the founder of https://papers-gamma.link, an Internet place to discuss scientific articles, mentioned by Matthieu Latapy. I have been supporting this site for 6 years now. I hope that one day it will become popular (in a good sense of the word) and useful for the entire scientific community. As you may imagine, I'm pretty convinced that the idea of public review and public comments is, potentially, a very promising one. My persuasion is less than 6 years ago though, and here's why.
Observing that Papers$^\gamma$ gains its popularity very slowly, I started to think more and more about the scientific review and publishing processes. I'll share with you my current understanding of this subjects, admitting that these issues are more likely to be in the field of sociology rather than mathematics.
The original goal of scientific journals was to inform about and discuss the research currents. But after, this had to make room for other things: archiving and bibliometrics. I'm OK with archiving. But, it seems that the optimization of bibliometric statistics negatively affects the discussion power, the original (!) goal of the scientific journals.
Let me try to illustrate exactly what I mean by "discussion power" of scientific journals. Consider Miller's one-page paper [2], which is an excellent example of conversational mathematics. The paper contains an alternative proof of Miles' results [1] about the characteristic equation of the $k$-th order Fibonacci sequence. Miller's paper can be considered as a response to Miles's paper. Compare this to the modern Internet forums like MathOverflow.
Does any journal, wiki, or another internet portal is attracted to archiving and bibliometrics as time goes forward? Maybe there is a natural or induced drift from "discussion" towards "judgment"? If it is the case, should we consider to create a new journals every, say, 10 or 20 years, to restart discussion processes? Or should we be extremely careful, trying to keep the discussion power of existing scientific journals?
Here are main arguments against the mass acceptance of public review and comments, that I can imagine:
Existing systems works pretty well.
Not all conversations should be made public. Private reviews and conversations have their advantages. People feel more free to commit errors, express misunderstanding, and criticize in private. Good reviews help the authors to polish and publish almost perfect articles.
The more a resource is open, the more it is susceptible to spam.
Anonymous public discussions may be used as a platform for attacking other scientists.
Here is a list of some online resources related to the idea of public reviews and collaborative science in general:
MathSciNet and Zentralblatt MATH, databases with post-publication reviews.
nLab wiki and The $n$-Category Café, collaborative works on Mathematics, Physics and Philosophy.
Polymath Project, a collaborative project that aims to solve important and difficult mathematical problems.
Machine Learning Paper Discussions subreddit.
Atmospheric Chemistry and Physics journal with interactive public peer review process.
CoScience, a service that aims to "recreate scientific communication as a virtuous, open, community-driven process".
F1000Research, a platform covering the life science publications.
PubPeer, a online platform for post-publication peer review. "The site has served as a whistleblowing platform, in that it highlighted shortcomings in several high-profile papers, in some cases leading to retractions and to accusations of scientific fraud", as Wikipedia says.
BibSonomy, a social bookmark service allowing comments.
Selected Papers Network was an open-source project for share and comment scientific articles.
I wonder if certain usenet groups in sci.* hierarchy are still active.
In any case, the source code of Papers$^\gamma$ is open under CC0 Public Domain Dedication. And you are welcome to send me a patch or fork it if you wish so.
To conclude, I think that "Hidden knowledge" will always be here, and the solution to this issue lies not in the technical but rather in the societal dimension. Someone just need to write it down in some searchable place. For instance, in enumerative combinatorics Sloane's The On-Line Encyclopedia of Integer Sequences helps a lot, but we need to watch out and constantly update it.
Conclusion update: for me, both OEIS and MathOverflow are popular because their main purpose is to allow people make a research together and not to judge each other.
Paywalled Biblio
[1] Miles, E. “Generalized Fibonacci numbers and associated matrices”. The American Mathematical Monthly, 67(8), 1960, 745–752
[2] Miller, M. D. "On generalized Fibonacci numbers". The American Mathematical Monthly, 78(10), 1971, 1108–1109.