A little step sideways from small-angle scattering for this week’s post. As you are probably aware by now, I sometimes use the LaN weblog to crystallize ideas into something resembling a coherent story. This is needs to be done now, as I am preparing another presentation (due late January), one that covers a tangential topic from my usual repertoire: a little overview of bad tidings in science.
Let me start by quoting the (slightly flowery) talk abstract:
The pressures on young scientists are enormous. After years of education, they emerge to find themselves scrambling for jobs in an unforgiving world. Some scientists survive. Some make it out through perseverance and hard work, some are in the right place at the right time, yet others take less illustrious approaches as they discover the system can be gamed.
The system can be gamed. There are ways to reduce the pressure and follow an easier path towards perceived “success”, and several scientists have been spotted following this road (with unknown numbers still following it, wittingly or unwittingly). Through highlighting such examples in this talk, we can get a glimpse of what risks lie ahead for the pressured and unwary scientist on a slippery slope.
The reasons for this gameability of science lie in the values of our society. We can point out a few easy culprits. We can point to the enormous pressures to succeed coupled with a skewed understanding of success. To the propensity of administrative staff to elevate metrics beyond their application limits. To the push from the public to make complex science palatable, and demand quick, polarized answers to increasingly intricate questions. To the efforts from progressively obsolete for-profit publishers struggling to highlight their relevance by peddling inadequate impact factors. Examples will be shown, unfortunate details will be brought to the surface.
This talk has it all. A warning message, cringe-worthy examples, and real-life experience from a young old fart drawing experience from so many more. Science is in trouble, and it is starting to show. Something is rotten in the state of science.
Not my field of expertise, as I am sure you would agree. this topic is more often covered by the likes of David Colquhoun, Michael Eisen, Stephen Curry, Philip Moriarty, and others. However, I am getting interested in this for a variety of reasons:
- A long-time sufferer of self-doubt and impostor syndrome (hi academia!), I obsessively check for evidence of my impact on science. H-index, journal impact factors, number of citations to my papers, ResearchGate “score”, number of downloads of my papers, Twitter followers, number of grants (none so far), … the list goes on. But the more often one looks at these numbers and compares them with the numbers of others, the more depressed one gets at their performance. Occasionally, however, I start to wonder if it is really me being bad at science or if there is a problem with the numbers themselves.
- Messages on Twitter are reinforcing the idea that it may just be the numbers (a relieving thought). From these sources, it appears there is a significant over-reliance on numbers and numeric evaluation metrics in assigning all kinds of things: funding, job security, prestigious positions, and so on. Such a pressure on having the right numbers naturally leads to cheating, or gaming, epitomized by Goodhart’s law (but also Campbell’s law and Lucas’ critique): “When a measure becomes a target, it ceases to be a good measure”.
- The experience and knowledge gained from being involved in the stripy nanoparticle saga also leaves me wondering. How on earth can such a vitiated, poorly substantiated version of science lead to such prolific writing in high-impact journals, prestigious academic positions and plentiful grants? Why has it become so incredibly tough to do critical work, why are we at every step feeling hindered in actively discussing or calling out such poor work?
And then you start seeing it more and more; epitomized by groups “surfing” the grant waves along popular topics, whose output comprises an impressive amount of very quick but poorly researched papers.
This behaviour is encouraged by universities and institutes ostensibly having no nobler goal than a higher ranking (in whatever obscure system they rank highest), and basing their hiring and firing policies on that [ex1, ex2, ex3]. This trend towards quantification of people (typically with a single number), naturally causes worried researchers to try to stay employed by boosting their metric (and can be lethal to those who don’t). Whatever kludges are proposed to improve the metrics, you are still trying to encompass an entire, complex academic using simplistic measures. Hands up, all who have seen quantifications similar to the one used by Imperial College London (source and discussion):
Multiply the impact factor of the journal by the author position weight, and divide by the number of authors. The author position weight is 5 for the first and last author, 3 for the second author, 2 for the third author and 1 for any other position.
My institute uses something similar, but assigns double points if you publish in our institute’s own journal. Make of that what you will. None of these, by the by, lets one score points by doing arguably important scientific outreach through blogs or videos, often eschews book contributions, and are thereby actively discourages such activities. Most metrics stay necessarily confined to weighting number of publications, number of citations and grant income in one way or another.
Likewise, journals are trying to stay relevant in light of advances in alternative science dissemination outlets, are pushing terrible metrics (journal impact factors, JIFs) and blaming open access (um, no) for problems that have arisen. Apropos, many problems have arisen due to the increased importance of the metrics they themselves promote. Their push of journal impact factors leads or forces researchers to waste time chasing that metric instead of dissemination for the purpose of science.
Just to recap briefly, one problem with the impact factor is that it does not reflect very well that which it is supposed to represent; a very large fraction of papers in a given journal are cited less than its JIF (due to the arithmetic mean being a poor statistical measure for the Bradford distribution it tries to encompass). Secondly, the JIF is used to assess the “quality” of researchers publishing in them (see example quoted earlier). Lastly, the JIF only correlates strongly with the amount of retractions, and poorly predicts how many citations a paper will receive.
Researchers are found sexing up their topics, or twisting facts and figures just to aim their paper at that higher IF journal. Researchers not publishing when research is done, but when it is time for another paper. Publishing duplicate papers to boost publication record.
Editors and reviewers are found boosting their H-indices or JIFs by asking for their own off-topic papers to be cited (since it appears that the H-index is surprisingly well described as 0.54 times the square root of your total number of cites). Similar gaming appears through citation rings, soliciting or accepting ghost-authorships (more here).
So it seems apparent that science has strayed from its original path. To highlight what that was, skip to 5:33 in this video (previously highlighted in this post) and watch what GE had to say about it in the 1950’s:
The core message for me from that video, is the following:
“one must realize […] that the truly great discoveries of science — the fundamental breakthroughs — have most often resulted from scientific research that had no specific or practical goal. The scientist who is left free to pursue the truths of nature, no matter where his research may lead, and who is free to fail, time and time again, without criticism, may in the end come to a finding. This finding may become valuable immediately, eventually, or not at all.”
Science is the search for the nature of things. And sure, some scientists are more successful at finding things than others may be. But science is the search, not the result. It is the process of exploring old and new ideas as rigorously and critically as necessary, and then to document the findings of that exploration. The key to quality in science is not the publication but the scientific attitude and rigor, and it is that which is most difficult to quantify.
We can get a sense of the degree of rigor and attitude of the scientist through personal communication and by careful study of their notes and protocols. What we cannot do is assess these qualities through proxies of the journals they may eventually publish in, through the number of citations their works garner, or by counting the money they spent each year. We should also consider whether it is even worthwhile to spend the time and money on assessments of researchers. Sure, it is a slight waste of money to support an unfortunate researcher, but can you really deign to foretell the unfortunates from those on the verge of “success”?
We seem to be taking another, altogether more dangerous approach: we are on the path to redefine “good science” in terms more easily quantified and judged by bean counters. We are moving to a bastardized version where its quality is synonymous to retweets, citations (essentially a more official retweet), impact factors and self-reinforcing grant expenditure. This way, science is rapidly becoming a caricature of itself, not dissimilar from the version portrayed by Aaron Diaz. If we let science be redefined in these terms, then yes, it can be quantified, assessed, evaluated, ranked. And you better make sure your metric is up to scratch.
But that is not the science I signed up for.