Getting Science Right

Every once in a while, a scientific scandal makes big news.  Someone faked doing an experiment, or grossly misinterpreted their results, or failed to reproduce someone elses important result.  Particularly in medicine, this can have big consequences.

Unfortunately, these scandals are often blown out of proportion, with insinuations that many scientists are dishonest.   At least partly because of this, scientists are paying increasing attention to irreproducible results.  There is a blog, "Retraction Watch" which is devoted entirely to scientific papers that have been formally retracted.  Some common problems are plagiarism (including self-plagiarism) or apparently faked results (particularly manipulated images).  Honest scientific mistakes (i.e. missed minus signs, etc.) also make an appearance, as does occasional subversion of the peer review process.    These problems are real, but it is important to keep them in perspective.  Retraction watch typically posts 1-3 retractions/day, out of hundreds of thousands of scientific papers published each year.  This is a very miniscule percentage.  Although Retraction Watch probably doesn't catch every retraction, they do appear to be very efficient at finding them.

Many of these errors are caught rather quickly, by other scientists.  Most important retractions occur within a year or two.   Pubmed, an online library of medical literature, run by the National Institute of Health recently (2013) opened Pubmed commons, where readers can comment on the scientific literature; suspect images and other visible problems can be (and are) vigorously discussed.  

A bigger problem may be papers that are just not reproducible, for reasons that are not clear.  This is mostly an issue for biology and medicine, fields that deal with complex systems (large molecules, cells, humans), where .   At least according to some reports, like this article in the New Scientist, this is an epidemic problem, affecting a large fraction of published papers.  This track record is a good reason to take the latest medical advice with at least a small grain of salt.  However, even here, the scientific record is generally self-correcting, albeit most slowly.  Science builds on previous results, and you can't build much on a cracked foundation. Darwinian evolution gradually weeds out bad conclusions.

As a more quantitative science, physics suffers less from irreproducibility than biology.  It is far easier to quantify the uncertainties in a neutrino energy measurement than in, for example, unknown contaminants in a reagent used in a biology experiment.    Over the past decades, physics has also taken increasing efforts to eliminate sources of unconscious bias.  In many experiments (IceCube included), most analyses are done in a 'blind' manner, whereby the analyst prepares his analysis using simulated data, and a small fraction of the real data.  Only after the analysis procedure is fixed, and reviewed by the collaboration, is the real data analyses.  This avoids any tendency to zero in on fluctuations in the data (the 'look here' phenomena).  So, when choosing a list of possible neutrino sources to analyze, we won't unconsciously pick one(s) that correspond to upfluctuations in the data.   As a result of these practices, particle and nuclear physics have a pretty good (but not perfect) record with being able to reproduce previous results.