Scientific criticism of psychology research is good. Bullying is bad. Between these two truisms lie a world of cases and a sea of disagreement. Look at the different perspectives described and shown in Susan Dominus’ article on Amy Cuddy, and Daniel Engber’s take, and Simine Vazire’s, and Andrew Gelman’s.
Talking about a “tone problem,” as in previous years, hasn’t worked. Worrying vaguely about overall tone seems petty. Everyone justifies their own approach. Few are repenting. The question needs to be sharpened: why do critics say things that seem reasonable to them, and unacceptably personal to some audiences?
In principle, it’s easy to insist on avoiding the ad hominem: “tackle the ball, not the player.” In practice, scientific criticism and testing have many reasons to veer into gray areas that leave the referees confused about where the ball ends and the foot begins. Here are some of them.
1. Who to critique? I see triage, you see resentment.
Asking for data, replicating studies–these are time-consuming activities. Checking everyone’s research is impossible. There are far more people producing than critiquing research. Since the start of the movement there has been a quest, backstage and quiet, to come up with rules governing whom to train the telescope on. Until this quest returns an answer, choices are going to be subjective, ill-defined, seemingly unfair.
Criticism is going to focus on results that are highly publicized. One, because they come to mind easily. Two, because the critic sees a special duty to make sure high-profile results are justified. But to the criticized, this focus can seem unfair, maybe even motivated by resentment of their success (or Nietzsche’s ressentiment). Naturally, it’s unlikely that a critic who investigates only high-profile results will have the same level of recognition as the results’ authors.
I endorse this authentic Mustache Fred quote.
Criticism is also going to focus on results that are surprising and implausible. Indeed, 5-15 years ago, there was considerable overlap between being high-profile and being surprising. “Big behavioral results from small input tweaks” was a formula for success in journals that prized novelty. Whether or not you think it’s fair to go after those examples might relate to what you think of the prior probability that a hypothesis is true. That probability, by definition, is lower the more implausible the effect is.
- People who think prior probabilities are necessary in order to interpret positive evidence will agree that extraordinary claims require extraordinary evidence. This includes Bayesians, but also frequentists who understand that a significant p-value by itself is not the chance that you have made a type I error; you need priors to interpret the p-value as a “truth statistic” (see Ioannidis, 2005 and this widget).
- If you don’t get this point of statistics, you will probably lean toward seeing an equal-tolerance policy as more fair. Surely, all findings should be treated alike, no matter how supported they are by precedent and theory. By this standard, focusing on surprising findings looks like at least a scientific-theoretical bias, if not a personal one. You may wonder why the boring, obvious results don’t get the same level of scrutiny.
2. Are your results your career? Maybe they shouldn’t be, but they are.
In addressing bad incentives in science, I sometimes hear that the success of a scientist’s career should not depend on whether their research gives hypothesis-confirming results. But this has never been the case in science and is unlikely ever to be. If a scientist has a wrong idea that is later shot down by more evidence, their stature is diminished. After all, if being right is not important to the personal reputation of a scientist, what incentive is there to be right?
Let’s not be naive. Science careers have been increasingly precarious and competitive, making it hard for a scientist to “merely live” with null or nullified results. Tenure, grant and hiring standards at the highest levels make it necessary for a scientist to be personally identified with a substantial, positive, and influential line of research. Criticism that leads to doubt about published work is a career threat, like it or not. Under these conditions, the scientific is inherently personal.
3. What to do in the face of impunity?
When impersonal, procedural justice fails to deliver results, there is a strong pull to deliver personal, frontier justice. Now when it comes to scientific error in psychology, or worse yet misconduct, who exactly is the law? Universities are often vested in maintaining the reputations of their good grant-getters (but, as with free speech issues, happy to throw temporary faculty under the bus.) Not all researchers belong to professional organizations that enforce an ethics code. Funders take some interest in research misconduct, but not all research is funded. Journal editors have leverage with retractions, corrections, and expressions of concern, but this is itself a form of sheriff law, with very different results depending on the editor’s opinion, and very few investigative powers on its own.
Having to rely on sheriff law, mob law, or vigilante law, frankly sucks. There’s very little stopping this kind of justice from being capricious, unaccountable, unjust. Batman, Dirty Harry and the Bad Lieutenant are all in the same game. Historically, sociologically, what makes this kind of justice go away is the establishment of reliable, accountable institutions that police integrity. Institutions that are empowered not just to find smoking guns pointing to misconduct, but to close a raftload of loopholes that allow evidence to be concealed or destroyed with impunity.
Until that happens, justice is stuck in the Middle Ages, with judicial duels involving vested persons rather than impartial institutions. With nobody responsible for bringing clear charges, accusations confuse sloppiness, overstatement, and outright fraud, just as Batman punishes serial killers and truant kids alike. Even posse members get the blues about this, sometimes.
Criticize, with civility
If criticism in today’s science, for all these reasons, is going to have personal resonance, the rule “avoid anything that seems like a personal attack” is not going to cut it.
(I know some of you think it is wrong to put any community standards on what can be said. This discussion is not for you, and we are all now free to call you someone who really gets off on being an asshole. Shh! Don’t complain.)
For the rest of us, I propose that civil discourse means we treat other scientists as people of good character who are gravely mistaken. Even when we suspect that they are not people of good character. Especially when we suspect this; otherwise we are merely being nice to people we like.
Assuming good character means, concretely, this:
- Confine your public observations to judging their words and behaviors.
- Don’t try to get inside their head; avoid claims like “He must really want to tear down science” or “She just loves to pull off these tricks.”
- Don’t try to paint their character with a label. People are uncannily good at making inferences from actions. “Con artist,” “shady,” “zealot,” all go over the top. They inflame, when your evidence should be enough.
- Do all this even when you are itching to land them a zinger, even when you want everyone to rise up in arms behind your call. You have great freedom in highlighting and arranging the facts; that freedom can be challenged, and answered rationally. Let the case speak for itself.
Approaching Significance 
This sounds reasonable and well mannered but there are some important issues ignored. I do think that anyone can make mistakes and in most cases criticism should be about making corrections, ideally this should be a collaborative effort but people often perceive any criticism as personal. However a great deal of research has effects that are in no way neutral, there are huge sums of money wasted, which could have been used to support valuable work. While its true funding agencies bear some responsibility but when it becomes clear that the researchers haven’t even bothered to do any reading, use discredited theoretical frameworks or are announcing discovery’s that everyone else has known about for decades, a certain about of anger is understandable.
Then there is the way in which make it as difficult as possible to validate their findings, refusing to release their data even when the research was publicly funded,in many cases it is the researchers themselves that resort to personal attacks on their critics and in fact some of the defensive strategies can add even more issues involving the ethics and even the law. The history of the PACE trail in the UK provides an excellent example of how these things can get totally out of hand.
Then there is the case of medical research, I was recently reading about the effects of bad research on a specific drug, the commentary described the many thousands that may have died and compared it to a modern day genocide.
While these things may not be representative they have had a huge influence on the public and political attitudes towards science and scientists. The relative freedom from control and oversight has effectively been lost and this is as a direct result of science in general failing in keeping its own house in order. Criticism has always been at the very core of science and yet its critics who find themselves the target of other scientists being labelled as sexist or bullies and some scientists simply refusing to engage with the concerns of the public. Scientists when compared to other workers have had an extraordinarily privileged position in society, something which many fail to recognize, this has been effectively thrown away. In the last few years we have even seen funding bodies simply ignoring the usual control mechanisms and directly invoking the criminal law. Only recently this would have been incredibly rare and yet currently there are a number of scientists serving prison sentences, I wonder if they think of this as bullying.
While its nice to be nice, science is seeing massive changes and the views of scientists themselves seen as unreliable and at time irrelevant. I think these things are self inflicted, and in fact continue to be, I’m not sure whether the views in this article, reasonable as they are will help, maybe the focus should be on promoting criticism so it returns to being the norm this way scientists should expect it and be prepared to engage with it.
i appreciate this balanced consideration.