False-positive science

The Atlantic reports on the dynamics of yet another group of scientists coming to grips with having wasted time and resources chasing down a dead end. (link) It's a good read but long. Here is the gist of it: Almost 20 years ago, some researchers made a huge splash by claiming to have discovered the "depression gene". The one gene eventually engendered 450 publications, and when counting related genes, over 1,000 publications. A recent large-scale "validation" study is likely to bring down the entire cottage industry - the depression gene is found to have little explanatory power for depression after all. Gene data is an example of a type of Big Data. Big Data can be big in terms of the number of individuals in the dataset, or the number of measurements per individual. Two decades ago, the scale was attained by virtue of more measurements, not more individuals. The original study looked at about 300 or so individuals but each person's genome is vast. The basic analysis is to compare the average depressed individual versus the average not-depressed individual in the sample. The data analyst sifts through large numbers of genes to find one or a few that are highly correlated with having depression. This is a classic fishing expedition, because of the large number of candidate genes, and also because of the large number of ways to define depression. Such an analysis rides on top of a "model" of the world in which a single gene is responsible for depression. Over the years, the scientific community has discovered that this model is wrong. The new model assumes depression is indicated by a large set of genes each contributing a weak effect. This type of structure is very hard to elicit from the typical datasets of the past - those that have numerous measurements on few individuals. Nowadays, we have data on lots of individuals but the sourcing of the data and other problems pose formidable challenges. It's also not clear how to use a model that spreads the blame thinly around a large number of genes for treatment. Science is proceeding as it should - weak theories are overturned with more research. The article laments that it took 20 years to turn the tide, earlier warnings were ignored, the publish-or-perish culture in academia creates perverse incentives, retraction of scientific studies, etc. *** I recently wrote about the challenge of Big Data expanding the variety of measurements here. Also, in writing Numbersense (link), I was concerned that the explosion of data collection causes an avalanche of false-positive science.  

from Big Data, Plainly Spoken (aka Numbers Rule Your World) http://bit.ly/2Wf3unn
via IFTTT