{"id":1457,"date":"2014-12-02T11:00:20","date_gmt":"2014-12-02T02:00:20","guid":{"rendered":"http:\/\/www.lookingatnothing.com\/?p=1457"},"modified":"2015-03-03T12:14:11","modified_gmt":"2015-03-03T03:14:11","slug":"the-dark-side-of-science","status":"publish","type":"post","link":"https:\/\/lookingatnothing.com\/index.php\/archives\/1457","title":{"rendered":"The Dark Side of Science."},"content":{"rendered":"

A little step sideways from small-angle scattering for this week’s post. As you are probably aware by now, I sometimes use the LaN<\/a> weblog to crystallize ideas into something resembling a coherent story. This is needs to be done now, as I am preparing another presentation (due late January), one that covers a tangential topic from my usual repertoire: a little overview of bad tidings in science. <\/p>\n

\"Panel<\/a>
“The opposing side of a conflict you never knew existed”. Reproduced with permission from Dresden Codak, Copyright 2011 Aaron Diaz. Source: http:\/\/dresdencodak.com\/2011\/06\/16\/dark-science-10\/<\/figcaption><\/figure>\n

Let me start by quoting the (slightly flowery) talk abstract:<\/p>\n

The pressures on young scientists are enormous. After years of education, they emerge to find themselves scrambling for jobs in an unforgiving world. Some scientists survive. Some make it out through perseverance and hard work, some are in the right place at the right time, yet others take less illustrious approaches as they discover the system can be gamed.<\/p>\n

The system can be gamed. There are ways to reduce the pressure and follow an easier path towards perceived “success”, and several scientists have been spotted following this road (with unknown numbers still following it, wittingly or unwittingly). Through highlighting such examples in this talk, we can get a glimpse of what risks lie ahead for the pressured and unwary scientist on a slippery slope.<\/p>\n

The reasons for this gameability of science lie in the values of our society. We can point out a few easy culprits. We can point to the enormous pressures to succeed coupled with a skewed understanding of success. To the propensity of administrative staff to elevate metrics beyond their application limits. To the push from the public to make complex science palatable, and demand quick, polarized answers to increasingly intricate questions. To the efforts from progressively obsolete for-profit publishers struggling to highlight their relevance by peddling inadequate impact factors. Examples will be shown, unfortunate details will be brought to the surface.<\/p>\n

This talk has it all. A warning message, cringe-worthy examples, and real-life experience from a young old fart drawing experience from so many more. Science is in trouble, and it is starting to show. Something is rotten in the state of science.<\/p><\/blockquote>\n

Not my field of expertise, as I am sure you would agree. this topic is more often covered by the likes of David Colquhoun<\/a>, Michael Eisen<\/a>, Stephen<\/a> Curry<\/a>, Philip Moriarty<\/a>, and others. However, I am getting interested in this for a variety of reasons:<\/p>\n

    \n
  1. A long-time sufferer of self-doubt and impostor syndrome<\/a> (hi academia!), I obsessively check for evidence of my impact on science. H-index<\/a>, journal impact factors, number of citations to my papers, ResearchGate “score”<\/a>, number of downloads<\/a> of my papers, Twitter followers<\/a>, number of grants (none so far), … the list goes on. But the more often one looks at these numbers and compares them with the numbers of others, the more depressed one gets at their performance. Occasionally, however, I\u00a0 start to wonder if it is really me being bad at science or if there is a problem with the numbers themselves.<\/li>\n
  2. Messages on Twitter are reinforcing the idea that it may just be the numbers (a relieving thought). From these sources, it appears there is a significant over-reliance<\/a> on numbers and numeric evaluation metrics in assigning all kinds of things: funding, job security, prestigious positions, and so on. Such a pressure on having the right numbers naturally leads to cheating, or gaming<\/a>, epitomized by Goodhart’s law<\/a> (but also Campbell’s law<\/a> and Lucas’ critique<\/a>): “When a measure becomes a target, it ceases to be a good measure”.<\/li>\n
  3. The experience and knowledge gained from being involved in the stripy nanoparticle saga<\/a> also leaves me wondering. How on earth can such a vitiated, poorly substantiated version of science lead to such prolific writing in high-impact journals, prestigious academic positions and plentiful grants? Why has it become so incredibly tough to do critical work, why are we at every step feeling hindered<\/a> in actively discussing or calling out such poor work?<\/li>\n<\/ol>\n
    \"Metrics<\/a>
    Metrics of Looking at Nothing.<\/figcaption><\/figure>\n

    And then you start seeing it more and more; epitomized by groups “surfing” the grant waves along popular topics, whose output comprises an impressive amount of very quick but poorly researched papers.<\/p>\n

    This behaviour is encouraged by universities<\/a> and institutes ostensibly having no nobler goal than a higher ranking<\/a> (in whatever obscure system they rank highest), and basing their hiring and firing <\/a>policies on that [ex1<\/a>, ex2<\/a>, ex3<\/a>]. This trend<\/a> towards<\/a> quantification of people<\/a> (typically with a single number), naturally causes worried researchers to try to stay employed by boosting their metric (and can be lethal to those who don’t<\/a>). Whatever kludges are proposed to improve the metrics, you are still trying to encompass an entire, complex academic using simplistic measures. Hands up, all who have seen quantifications similar to the one used by Imperial College London (source and discussion<\/a>):<\/p>\n

    Multiply the impact factor of the journal by the author position weight, and divide by the number of authors. The author position weight is 5 for the first and last author, 3 for the second author, 2 for the third author and 1 for any other position.<\/p><\/blockquote>\n

    My institute uses something similar, but assigns double points if you publish in our institute’s own journal<\/a>. Make of that what you will. None of these, by the by, lets one score points by doing arguably important scientific outreach through blogs or videos, often eschews book contributions, and are thereby actively discourages such activities. Most metrics stay necessarily confined to weighting number of publications, number of citations and grant income in one way or another.<\/p>\n

    \"Alternative<\/a>
    Alternative publishing methods. Graffiti: (possibly) Banksy. Image source unknown.<\/figcaption><\/figure>\n

    Likewise, journals are trying to stay relevant in light of advances in alternative<\/a> science<\/a> dissemination<\/a> outlets<\/a>, are pushing terrible metrics<\/a> (journal impact factors<\/a>, JIFs) and blaming open access<\/a> (um, no<\/a>) for problems that have arisen. Apropos, many problems have arisen due to the increased importance of the metrics they themselves promote. Their push of journal impact factors leads <\/a>or forces<\/a> researchers to waste time<\/a> chasing that metric instead of dissemination for the purpose of science.<\/p>\n

    Just to recap briefly, one problem with the impact factor is that it does not reflect very well that which it is supposed to represent; a very large fraction of papers<\/a> in a given journal are cited less than its JIF (due to the arithmetic mean being a poor statistical measure for the Bradford distribution it tries to encompass<\/a>). Secondly, the JIF is used<\/a> to assess the “quality” of researchers publishing in them (see example quoted earlier). Lastly, the JIF only correlates strongly<\/a> with the amount of retractions, and poorly predicts<\/a> how many citations a paper will receive.<\/p>\n

    Researchers are found<\/a> sexing<\/a> up<\/a> their<\/a> topics, or twisting<\/a> facts<\/a> and<\/a> figures<\/a> just to aim their paper at that higher IF journal. Researchers not publishing when research is done, but when it is time<\/a> for another paper. Publishing duplicate papers<\/a> to boost publication record.<\/p>\n

    Editors and reviewers are found boosting their H-indices or JIFs by asking<\/a> for their own off-topic papers to be cited (since it appears that the H-index is surprisingly well described<\/a> as 0.54 times the square root of your total number of cites). Similar gaming appears through citation rings<\/a>, soliciting<\/a> or accepting<\/a> ghost-authorships <\/a>(more here<\/a>).<\/p>\n

    All this leads to an involuntary shift of priorities<\/a> of the scientist from research first and foremost, to publishing<\/a> and grant-finding first (and research in their spare time).<\/p>\n

    So it seems apparent that science has strayed from its original path. To highlight what that was, skip to 5:33 in this video<\/a> (previously highlighted in this post<\/a>) and watch what GE had to say about it in the 1950’s:<\/p>\n