Last week I attended a symposium geared towards a general science audience where the topic of the day was the role of the microbiome in human health. I would exaggerate only slightly if I said that all of the presenters motivated their own interest in the microbiome by citing the work of Turnbaugh and colleagues from the Gordon lab pointing to differences in diversity and the ratio of Firmicutes and Bacteroidetes among lean and obese individuals. Similar sets of introductory slides probably grace most talks at these types of symposia. Lost in the hubbub about the microbiome and obesity are two very important papers (here and here) - one of which was written by an author on the original study - saying that the effect doesn’t exist when you consider any other dataset. Yet as of my writing, there is nothing on PubPeer or PubMed Commons directing readers of the original paper to these meta-analyses.
I’ll come back to the obesity story in a moment. Two weeks ago I attended another symposium where the keynote speaker was Brian Nosek who briefly mentioned a study he co-authored with Anna Dreber that was posted this Monday on PNAS’s website. In this study, Dreber and colleagues asked other scientists their opinion of the veracity of various studies. But they asked them to put their virtual money where their opinions were by trading their opinions like you would stocks. If they thought the study was good, they would take a long position and if they thought it sucked, they would take a short position. Then Dreber got people to re-do the original studies and see whether they could reproduce the results and whether the market-based approach worked. As is typically the case, they couldn’t reproduce the results and the market-based approach wasn’t perfect, but was pretty good. I’m leaving a lot out, so go read the original or see Ed Yong’s great explanation. The advantage of this type of prediction market and others including corn futures, political elections, and Super Bowl winners is that there will be a date by which we will know the answer. But if you think about the market for say Amazon, we will never really know the “value” of Amazon - there’s only what we think its value is.
Since hearing Nosek talk, I’ve been wracking my brain to think about how we might use a prediction market approach to address the original Turnbaugh study, the general microbiome-obesity hypothesis, and other microbiome-related hypotheses. One problem with the Dreber approach is that it is unlikely that we will get researchers to do exact reproductions of other studies. Although there are ongoing efforts led by Dreber, Nosek, and others, there are always caveats about methods, population structures, and what not. For example, the original Turnbaugh study was done with a form of 454 sequencing that is not long for this world. Anyway, I’m more interested in the broader veracity of a hypothesis than that of a specific study.
One problem with all markets is that they are only efficient when information is widely available. Experts can predict the veracity of a study, because they know the surrounding literature. Dreber’s study also pointed out that individually the scientists weren’t as good at assessing the veracity of a study as they were as a group. People can guess the weight of a heifer, because they kind of know something about weights of things. People can correctly predict who will win an election, because they have a stake in the game - they will vote themselves. But ask people whether differences in diversity are associated with obesity and they are likely to draw a blank stare or repeat the same Turnbaugh paper without knowing about the follow-up meta-analyses. There is a roadblock to information.
Based on all this, here’s a proposal: Let’s take a hypotheses (i.e. statements that are testable, falsifiable, and lead to predictions) about the microbiome. Then let’s accumulate studies that are pro/con/neutral on the topic. Present these papers to other scientists and the public along with an anonymous +1/-1 weighting for the papers. The papers people think the most highly of will appear at the top. In addition, for each paper, there could be a discussion of whether the paper actually supports the hypothesis. There are numerous obesity-microbiome papers out there, some poorly done and some well done. Having a ranking of papers within this area and how well they support or refute a specific hypothesis would be enormously valuable. At the top of the page presenting the hypothesis there would be the ability to long or short the hypothesis. As new papers come out, the discussion would continue. There may be some volatility in the market as a study supporting Turnbaugh comes out or price hardening as yet another study fails to find a relationship between diversity and obesity. I’m sure there are numerous hypotheses we could propose to set up a market for. Because this would not be about carrying out PubPeer-style attacks on individual papers that tend towards the ad hominem it would possible to have a higher level discussion about the science in general. My hope would be to cultivate a more constructive form of post-publication review by having fun discussions of what topics appear valid across the field and to develop a consensus or identify areas that need further research. More broadly we could think about applying this approach to other topics in science like evolution or climate change.
The incentives wouldn’t be financial, but credibility points. So if the value of your portfolio is high, you’ll have more credibility and your comments would be weighted higher by visitors to the site. Just as people might want to know what Warren Buffett’s Berkshire Hathaway is investing in, someone might be interested in what hypotheses their peers are investing in or shorting. You might think Melissa Microbiome is typically pretty sane about microbiome research and look to see where she is positioned on various topics. Alternatively, it could serve as a BS meter for Barney Symbiosis who constantly seems to be over interpreting data.
There are three problems with this idea that keep me from launching it. First, I don’t really have the time to carry this on my own. Second, I really suck at web design, databases, and the other infrastructure that would be needed to pull this off. Third, this has been rattling around in my head without the opportunity to bounce the idea off other people. So, I’m wondering whether there are people out there that think this might be a fun idea to pilot and see where it goes. It might be possible to have a hack-athon where we get together and put together a prototype with two or three hypotheses to see whether the idea might have any traction.
Speaking of prototypes, there is actually a call out for entries to Open Science Prize that was recently announced by the NIH, Wellcome Trust, and Howard Hughes:
The Prize provides funding to encourage and support the prototyping and development of services, tools or platforms that enable open content – including publications, datasets, codes and other research outputs – to be discovered, accessed and re-used in ways that will advance discovery and spark innovation
The competition will be held over two phases. The first will fund up to six prizes of $80,000 each to fund prototype development over an eight to nine-month period. Of these prizes, one will be selected to go on to the second phase and to receive a prize of $230,000. It is hoped that the successful prototypes that don’t make it to the second round will be able to find support through other means. The teams must be international including one researcher from the US and one from another country and the applications are due on February 29, 2016.
If you think you might be interested in this idea let me know. There’s nothing that says I have to be in charge - I don’t claim to have the technical expertise that would be required to launch such a prototype. Even if we didn’t go for the Prize, this might be a fun project to work on in our Spare Time. If the prize money came through it probably wouldn’t really be enough to fully fund the effort so some donated “fun time” might be needed in the long run. Alternatively, if you have an idea for distilling the microbiome literature to help people assess the veracity of various claims, let’s hear it!