Category: politics

Fly less, give more

Russ Poldrack writes that he will be flying less:

I travel a lot – I have almost 1.3 million lifetime miles on United Airlines, and in the last few years have regularly flown over 100,000 miles per year. This travel has definitely helped advance my scientific career, and has been in many ways deeply fulfilling and enlightening. However, the toll has been weighing on me and Miles’ article really pushed me over the edge towards action. I used the Myclimate.org carbon footprint calculator to compute the environmental impact of my flights just for the first half of 2019, and it was mind-boggling: more than 23 tons of CO2. For comparison, my entire household’s yearly carbon footprint (estimated using https://www3.epa.gov/carbon-footprint-calculator/) is just over 10 tons!

For these reasons, I am committing to eliminate (to the greatest degree possible) academic air travel for the foreseeable future. That means no air travel for talks, conferences, or meetings — instead participating by telepresence whenever possible.

I’m sympathetic to the sentiment, and have considered reducing my own travel on a couple of occasions. So far, I’ve decided not to. It’s not that I disagree with Russ’s reasoning; I’m very much on board with the motivation for cutting travel, and I think we should all try to do our part to try and help avert, or at least mitigate, the looming climate crisis. The question for me is how to best go about that. While I haven’t entirely ruled out cutting down on flying in future, it’s not something I’m terribly eager to do. Travel is one of the most rewarding and fulfilling parts of my job, and I’m loathe to stop flying unless there are no other ways to achieve the same or better outcomes.

Fortunately, there are other things one can do to try to keep the planet nice and habitable for all of us. For my part, I’ve decided that, rather than cutting back on travel, I’m going to give some money to charity. And in the next ~4,000 words, I’m going to try to convince you to consider doing the same (at least the giving to charity part; I’m not going to try to argue you out of not flying).

Now, I know what you’re thinking at this point. You’re probably thinking, what does he mean, ‘charity’? Is he talking about carbon offsets? He’s talking about carbon offsets, isn’t he! This dude is about to write a 4,000-word essay telling me I should buy carbon offsets! Like I don’t already know they’re a gigantic scam. No thanks.

Congratulations, my friend—you’re right. I do mean carbon offsets.

Well, kind of.

What I’ll really try to convince you is that, while carbon offsets are actually a perfectly reasonable thing for a person to purchase, the idea of "offsetting" one’s lifestyle choices is probably not the most helpful way to think about the more general fight against climate change. But carbon offsets as they’re usually described—i.e., as promissory notes you can purchase from organizations that claim to suck up a certain amount of carbon out of the atmosphere by planting trees or engaging in other similarly hippie activities—are a good place to start, because pretty much everyone’s heard of them. So let me start by rebutting what I see as the two (or really three) most common arguments against offsets, and in the process, it’ll hopefully become clear why offsets are a bit of a red herring that’s best understood as just a special case of a much more general principle. The general principle being, if you want to save yourself a bunch of reading, do a little bit of research, and then give as much as you comfortably can.

Offsets as indulgences

Carbon offsets are frequently criticized on the grounds that they’re either (a) ineffective, or (b) morally questionable—amounting to a form of modern day indulgence. I can’t claim any particular expertise in either climate science or catholicism, but I can say that nothing I’ve read has convinced me of either argument.

Let’s take the indulgence argument first. Superficially, it may seem like carbon offsets are just a way for well-off people to buy their way out of their sins and into heaven (or across the Atlantic—whichever comes first). But, as David Roberts observed in an older Grist piece, there are some fairly important differences between the two things that make the analogy… not such a great one:

If there really were such a thing as sin, and there was a finite amount of it in the world, and it was the aggregate amount of sin that mattered rather than any individual’s contribution, and indulgences really did reduce aggregate sin, then indulgences would have been a perfectly sensible idea.

Roberts’s point is that when someone opts to buy carbon offsets before they get on a plane, the world still benefits from any resulting reduction in carbon release—it’s not like the money simply vanishes into the church’s coffers, never to be seen again, while the newly guilt-relieved traveler gets to go on their merry way. Maybe it feels like people are just taking the easy way out, but so what? There are plenty of other situations in which people opt to give someone else money in order to save themselves some time and effort—or for some other arbitrarily unsavory reason—and we don’t get all moralistic and say y’know, it’s not real parenting if you pay for a babysitter to watch your kids while you’re out, or how dare you donate money to cancer charities just so people see you as a good person. We all have imperfect motives for doing many of the things we do, but if your moral orientation is even slightly utilitarian, you should be able to decouple the motive for performing an action from the anticipated consequences of that action.

As a prominent example, none of us know what thought process went through Bill and Melinda Gates’s heads in the lead-up to their decision to donate the vast majority of their wealth to the Gates Foundation. But suppose it was something like "we want the world to remember us as good people" rather than "we want the world to be a better place", would anyone seriously argue that the Gateses shouldn’t have donated their wealth?

You can certainly argue that it’s better to do the right thing for the right reason than the right thing for the wrong reason, but to the extent that one views climate change as a battle for the survival of humanity (or some large subset of it), it seems pretty counterproductive to only admit soldiers into one’s army if they appear to have completely altruistic motives for taking up arms.

The argument from uncertainty

Then there’s the criticism that carbon offsets are ineffective. I think there are actually two variants of this argument—one from uncertainty, and one from inefficacy. The argument from uncertainty is that there’s just too much uncertainty associated with offset programs. That is, many people are understandably worried that when they donate their money to tree planting or cookstove-purchasing programs, they can’t know for sure that their investment will actually lead to a reduction in global carbon emissions, whereas when they reduce their air travel, they at least know that they’ve saved at least one ticket’s worth of emissions.

Now, it’s obviously true that offsets can be ineffective—if you give a charity some money to reduce carbon, and that charity proceeds to blow all your money on advertising, squirrels it away in an executive’s offshore account, or plants a bunch of trees that barely suck up any carbon, then sure, you have a problem. But the fact that it’s possible to waste money giving to a particular cause doesn’t mean it’s inevitable. If it did, nobody would ever donate money to any charity, because huge inefficiences are rampant. Similarly, there would be no basis for funding clean energy subsidies or emission-reducing technologies, seeing as the net long-term benefit of most such investments is virtually impossible to predict at the outset. Requiring certainty, or anything close to it, when seeking to do something good for the world is a solid recipe for doing almost nothing at all. Uncertainty about the consequences of our actions is just a fact of life, and there’s no reason to impose a higher standard here than in other areas.

Conversely, as intuitively appealing as the idea may be, trying to cut carbon by reducing one’s travel is itself very far from a sure thing. It would be nice to think that if the net carbon contribution of a given flight is estimated at X tons of CO2 per person, then the effect of a random person not getting on that flight is to reduce global CO2 levels by roughly X tons. But it doesn’t work quite that way. For one thing, it’s not as if abstaining from air travel instantly decreases the amount of carbon entering the atmosphere. Near term, the plane you would have traveled on is still going to take off, whether you’re on it or not. So if you decide to stay home, your action doesn’t actually benefit the environment in any way until such time as it (in concert with others’ actions) influences the broader air travel industry.

Will your actions eventually have a positive impact on the air travel industry? I don’t know. Probably. It seems reasonable to suppose that if a bunch of academics decide to stop flying, eventually, fewer planes will take off than otherwise would. What’s much less clear, though, is how many fewer. Will the effective CO2 savings be anywhere near the nominal figure that people like to float when estimating the impact of air travel—e.g., roughly 2 tons for a one-way transatlantic flight in economy? This I also don’t know, but it’s plausible to suppose they won’t. The reason is that your purchasing decisions don’t unfold in a vacuum. When an academic decides not to fly, United Airlines doesn’t say, "oh, I guess we have one less customer now." Instead, the airline—or really, its automated pricing system—says "I guess I’ll leave this low fare open a little bit longer". At a certain price point, the lower price will presumably induce someone to fly who otherwise wouldn’t.

Obviously, price elasticity has its limits. it may well be that, in the long term, the airlines can’t compensate for the drop in demand while staying solvent, and academics and other forward-thinking types get to take credit for saving the world. That’s possible. Alternatively, maybe it’s actually quite easy for airlines to create new, less conscientious, air travelers by lowering prices a little bit, and so the only real product of choosing to stay home is that you develop a bad case of FOMO while your friends are all out having fun learning new things at the conference. Which of these scenarios (or anything in between) happen to be true depends on a number of strong assumptions that, in general, I don’t think most academics, or even economists, have a solid grasp on (I certainly don’t pretend to).

To be clear, I’m not suggesting that the net impact of not flying is negative (that would surprise me), or that academics shouldn’t cut their air travel. I’m simply observing that there’s massive uncertainty about the effects of pretty much anything one could do to try fight climate change. This doesn’t mean we should give up and do nothing (if uncertainty about the future was a reason not to do things, most of us would never leave our house in the morning), but it does mean that perhaps naive cause-and-effect intuitions of the if-I-get-on-fewer-planes-the-world-will-have-less-CO2 variety are not the best guide to effective action.

The argument from inefficacy

The other variant of the argument is a bit stronger, and is about inefficacy rather than uncertainty: here, the idea is not just that we can’t be sure that offsetting works; it’s that we actually have positive evidence that offset programs don’t do what they claim. In support of this argument, people like to point to articles like this, or this, or this—all of which make the case that many organizations that nominally offer to take one’s money and use it to pull some carbon out of the environment (or prevent it from being released) are just not cost-effective.

For what it’s worth, I find many of these articles pretty convincing, and, for the sake of argument, I’m happy to take what many of them say about specific mechanisms of putative carbon reduction as gospel truth. The thing is, the conclusion they support is not that trying to reduce carbon through charitable giving doesn’t work; it’s that it’s easy to waste your money by giving to the wrong organization. This doesn’t mean you have to put your pocketbook away and go home; it just means you might have to invest a bit of time researching the options before you can feel comfortable that there’s a reasonable (again, not a certain!) chance that your donation will achieve its intended purpose.

This observation shouldn’t be terribly troubling to most people. Most of us are already willing to spend some time researching options online before we buy, say, a television; there’s no reason why we shouldn’t expect to do the same thing when trying to use our money to help mitigate environmental disaster. Yet, in conversation, when I’ve asked my academic friends who express cynicism about the value of offsets how much time they’ve actually spent researching the issue, the answer is almost invariably "none" or "not much". I think this is a bit of an odd response from smart people with fancy degrees who I know spend much of their waking life thinking deeply about complex issues. Academics, more than most other folks, should be well aware of the dangers of boiling down a big question like "what’s the best way to fight climate change by spending money?" to a simplistic assertion like "nothing; it can’t be done." But the fact that this kind of response is so common does suggest to me that maybe we should be skeptical of the reflexive complaint that charitable giving can’t mitigate carbon emissions.

Crucially, we don’t have to stop at a FUD-like statement like nobody really knows what helps, so in principle, carbon offsets could be just as effective as not flying. No, I think it’s trivial to demonstrate essentially from first principles that there must be many cost-effective ways to offset one’s emissions.

The argument here is simple: much of what governments and NGOs do to fight climate change isn’t about directly changing individual human beings’ consumption behaviors, but about pro-actively implementing policies or introducing technologies that indirectly affect those behaviors, or minimize their impacts. Make a list of broad strategies, and you find things like:

  • Develop, incentivize and deploy clean energy sources.
  • Introduce laws and regulations that encourage carbon emission reduction (e.g., via forest preservation, congestion pricing, consumption taxes, etc.).
  • Offer financial incentives for farmers, loggers, and other traditional industrial sources of carbon to develop alternative income streams.
  • Fund public awareness campaigns to encourage individual lifestyle changes.
  • Fund research into blue-sky technologies that efficiently pull carbon out of the atmosphere and safely sequester it.

You can probably go on like this for a long time.

Now, some of the items on this list may be hard to pursue effectively unless you’re a government. But in most of these cases, there’s already a healthy ecosystem of NGOs working to make the world a better place. And there’s zero reason to think that it’s just flatly impossible for any of these organizations to be more effective than whatever benefit you think the environment derives from people getting on fewer planes.

On the contrary: it requires very little imagination to see how, say, a charity staffed by lawyers who help third-world governments draft and lobby for critical environment laws might have an environmental impact measured in billions of dollars, even if its budget is only in the millions. Or, if science is your thing, to believe that publicly-funded researchers working on clean energy do occasionally succeed at developing technologies that, when deployed at scale, provide societal returns many times the cost of the original research.

Once you frame it this way—and I honestly don’t know how one would argue against this way of looking at things—it seems pretty clear that blanket statements like "carbon offsets don’t work" are kind of dumb—or at least, intellectually lazy. If what you mean by "carbon offsets don’t work" is the much narrower claim that most tree-planting campaigns aren’t cost-effective, then sure, maybe that’s true. My impression is that many environmental economists would be happy to agree with you. But that narrow statement has almost no bearing on the question of whether or not you can cost-effectively offset the emissions you’d produce by flying. If somebody offered you credible evidence that their organization could reduce enough carbon emissions to offset your transatlantic flight for the princely sum of $10, I hope you wouldn’t respond by saying well, I read your brochure, and I buy all the evidence you presented, but it said nothing about trees anywhere, so I’m afraid I’m going to have to reject your offer and stay home.

The fact of the matter is that there are thousands, if not tens of thousands, of non-profit organizations currently working to fight climate change. They’re working on the problem in many different ways: via policy efforts, technology development, reforestation, awareness-raising, and any number of other avenues. Some of these organizations are undoubtedly fraudulent, bad at what they do, or otherwise a waste of your money. But it’s inconceivable to think that there aren’t some charities out there—and probably a large number, in absolute terms—that are very effective at what they do, and certainly far more effective than whatever a very high-flying individual can achieve by staying off the runways and saving a couple dozen tons of CO2 per year. And you don’t even need there to be a large number of such organizations; you just need to find one of them.

Do you really find it so hard to believe that there are such organizations out there? And that there are also quite a few people whose day job is identifying those organizations, precisely so that people like you and I can come along and give them money?

I don’t.

So how should you spend your money?

Supposing you find the above plausible, you might be thinking, okay, fine, maybe offsetting does work, as long as you’re smart about how you do it—now please tell me who to make a check out to so I can keep drinking terrible hotel coffee and going to poster sessions that make me want to claw my eyes out.

Well I hate to disappoint you, but I’m not entirely comfortable telling you what you should do with your money (I mean, if you insist on an answer, I’ll probably tell you to give it to me). What I can do is tell you is what I’ve done with mine.

A few months ago, I set aside an evening and spent a few hours reading up on various climate-focused initiatives, Then I ended up donating money to the Clean Air Task Force and the Coalition for Rainforest Nations. Both of these are policy-focused organizations; they don’t plant tree saplings or buy anyone a clean-burning stove. They fight climate change by attempting to influence policy in ways that promote, respectively, clean air in the United States, and preservation of the world’s rain forests. They are also, not coincidentally, the two organizations strongly recommended by Founders Pledge—an organization dedicated to identifying effective ways for technology founders (but really, pretty much anyone) to spend their money for the benefit of society.

My decision to give to these organizations was motivated largely by this Founders Pledge report, which I think compellingly argues that these organizations likely offer a much better return on one’s investment than most others. The report estimates a cost of $0.02 – $0.72 per ton of CO2 release averted when donating to the Coalition for Rainforest Nations (the cost is somewhat higher for the Clean Air Task Force). For reference, typical estimates suggest that a single one-way economy-class transatlantic plane ticket introduces perhaps 2 – 3 tons of CO2 to the atmosphere. So, even at the conservative end of Founders Pledge’s "realistic" estimate, you’d need to give CfRN only around $2 to offset that cost. Personally, I’m a skeptical kind of person, so I don’t take such estimates at face value. When I see this kind of number, I immediately multiply it by a factor of 10, because I know how the winner’s curse works. In this case, that still leaves you with an estimate of $20/ton—a number I’m perfectly happy with personally, and that seems to me quite manageable for almost anybody who can afford to get on a transatlantic flight in the first place.

Am I sure that my donations to the above organizations will ultimately do the environment some good? No.

Do I feel confident that these are the best charities out there? Of course not. It’s hard to imagine that they could be, given the sheer number of organizations in this space. But again, certainty is a wildly unrealistic desideratum here. What I am satisfied with is that I’ve done my due diligence, and that in my estimation, I’ve identified a plausibly effective mechanism though which I can do a very tiny bit of good for the world (well, two mechanisms—the other one is this blog post I’m writing, which will hopefully convince at least one other person to take similar action).

I’m not suggesting that anyone else has to draw the same conclusions I have, or donate to the same organizations. Your mileage will probably vary. If, after doing some research, you decide that in your estimation, not flying still makes the most sense, great. And if you decide that actually none of this climate stuff is likely to help, and instead, you’re going to give your money to charities that work on AI alignment or malaria, great. But at the very least, I hope it’s clear there’s really no basis for simply dismissing, out of hand, the notion that one can effectively help reduce atmospheric CO2—on a very, very tiny scale, obviously—via financial means, rather than solely through lifestyle changes.

Why stop at offsets?

So far I’ve argued that donating your money to climate-focused organizations (done thoughtfully) is a perfectly acceptable alternative to cutting back on travel, if your goal is to ultimately reduce atmospheric carbon. If you want to calculate the amount of money you’d need to give to the organization of your choice in order to offset the carbon that your travel (or, more generally, lifestyle) introduces every year, and give exactly that much, great.

But I want to go a bit further than that. What I really want to suggest is that if you’re very concerned about the environment, donating your money can actually be a much better thing to do than just minimizing your own footprint.

The major advantage of charitable giving, unlike travel reduction, or really any kind of lifestyle change, is that there’s a much higher ceiling on what you can accomplish. When you try to fight global warming by avoiding travel, the best you can do is eliminate all of your own personal travel. That may not be trivial, and I think it’s certainly worth doing if your perceived alternative is doing nothing at all. Still, there’s always going to be a hard limit on your contribution. It’s not like you can remove arbitrarily large quantities of carbon from the environment by somehow, say, negatively traveling.

By contrast, when you give money, you don’t have to stop at just offsetting your own carbon production; in principle, you can pay to offset other people’s production too. If you have some discretionary income, and believe that climate change really is an existential threat to the human species (or some large subset of it), then on on some level it seems a bit strange to say, "I just want to make sure I personally don’t produce more carbon than the average human being living in my part of the world; beyond that, it’s other people’s problem." If you believe that climate change presents an existential threat to your descendants, or at least to their quality of life, and you can afford to do more than just reduce your own carbon footprint, why not use more of your resources to try and minimize the collective impact of humanity’s past poor environmental decisions? I’m not saying anyone has a moral obligation to do that; I don’t think they do. But it doesn’t seem like a crazy thing to do, if you have some money to spare.

You can still fly less!

Before I go, let me circle around to where I started. I want to emphasize that nothing I’ve said here is intended as criticism of what Russ Poldrack wrote, or of the anti-flying movement more generally. Quite the opposite: I think Russ Poldrack is a goddamn hero (and not just for his position on this issue). If not for Russ’s post, and subsequent discussions on social media, I doubt I would have been sufficiently motivated to put my own money where my mouth is on this issue, let alone to write this post (as an otherwise fairly selfish person, I’m not ashamed to say that I wrote this post in part to force myself to give a serious chunk of cash to charity—public commitment is a powerful thing!). So I’m very much on board with the initiative: other things equal, I think cutting back on one’s air travel is a good thing to do. All I’m saying here is that there are other ways one can do one’s part in the fight against climate change that don’t require giving up air travel—and that, if anything, have the potential to exert far greater (though admittedly still tiny in the grand scheme of things) impact.

It also goes without saying that the two approaches are not mutually exclusive. On the contrary, the best-case scenario is that most people cut their air travel and give money to organizations working to mitigate climate change. But since nobody is perfect, everything is commensurable, and people have different preferences and resource constraints, I take it for granted that most people (me included) aren’t going to do both, and I think that’s okay. It seems perfectly reasonable to me to feel okay about your relationship with the environment so long as you’re doing something. I respect people who opt to do their part by cutting down on their air travel. But I’m not going to feel guilty for continuing to fly around the world fairly regularly, because I think I’m doing my part too.

I hate open science

Now that I’ve got your attention: what I hate—and maybe dislike is a better term than hate—isn’t the open science community, or open science initiatives, or open science practices, or open scientists… it’s the term. I fundamentally dislike the term open science. For the last few years, I’ve deliberately tried to avoid using it. I don’t call myself an open scientist, I don’t advocate publicly for open science (per se), and when people use the term around me, I often make a point of asking them to clarify what they mean.

This isn’t just a personal idiosyncracy of mine in a chalk-on-chalkboard sense; I think at this point in time there are good reasons to think the continued use of the term is counterproductive, and we should try to avoid it in most contexts. Let me explain.

It’s ambiguous

At SIPS 2019 last week (SIPS is the Society for Improvement of Psychological Science), I had a brief chat with a British post-undergrad student who was interested in applying to graduate programs in the United States. He asked me what kind of open science community there was at my home institution (the University of Texas at Austin). When I started to reply, I realized that I actually had no idea what question the student was asking me, because I didn’t know his background well enough to provide the appropriate context. What exactly did he mean by “open science”? The term is now used so widely, and in so many different ways, that the student could plausibly have been asking me about any of the following things, either alone or in combination:

  • Reproducibility. Do people [at UT-Austin] value the ability to reproduce, computationally and/or experimentally, the scientific methods used to produce a given result? More concretely, do they conduct their analyses programmatically, rather than using GUIs? Do they practice formal version control? Are there opportunities to learn these kinds of computational skills?
  • Accessibility. Do people believe in making their scientific data, materials, results, papers, etc. publicly, freely, and easily available? Do they work hard to ensure that other scientists, funders, and the taxpaying public can easily get access to what scientists produce?
  • Incentive alignment. Are there people actively working to align individual incentives and communal incentives, so that what benefits an individual scientist also benefits the community at large? Do they pursue local policies meant to promote some of the other practices one might call part of “open science”?
  • Openness of opinion. Do people feel comfortable openly critiquing one another? Is there a culture of discussing (possibly trenchant) problems openly, without defensiveness? Do people take discussion on social media and post-publication review forums seriously?
  • Diversity. Do people value and encourage the participation in science of people from a wide variety of ethnicities, genders, skills, personalities, socioeconomic strata, etc.? Do they make efforts to welcome others into science, invest effort and resources to help them succeed, and accommodate their needs?
  • Metascience and informatics. Are people thinking about the nature of science itself, and reflecting on what it takes to promote a healthy and productive scientific enterprise? Are they developing systematic tools or procedures for better understanding the scientific process, or the work in specific scientific domains?

This is not meant to be a comprehensive list; I have no doubt there are other items one could add (e.g., transparency, collaborativeness, etc.). The point is that open science is, at this point, a very big tent. It contains people who harbor a lot of different values and engage in many different activities. While some of these values and activities may tend to co-occur within people who call themselves open scientists, many don’t. There is, for instance, no particular reason why someone interested in popularizing reproducible science methods should also be very interested in promoting diversity in science. I’m not saying there aren’t people who want to do both (of course there are); empirically, there might even be a modest positive correlation—I don’t know. But they clearly don’t have to go together, and plenty of people are far more invested in one than in the other.

Further, as in any other enterprise, if you monomaniacally push a single value hard enough, then at a certain point, tensions will arise even between values that would ordinarily co-exist peacefully if each given only partial priority. For example, if you think that doing reproducible science well requires a non-negotiable commitment to doing all your analyses programmatically, and maintaining all your code under public version control, then you’re implicitly condoning a certain reduction in diversity within science, because you insist on having only people with a certain set of skills take part in science, and people from some backgrounds are more likely than others (at least at present) to have those skills. Conversely, if diversity in science is the thing you value most, then you need to accept that you’re effectively downgrading the importance of many of the other values listed above in the research process, because any skill or ability you might use to select or promote people in science is necessarily going to reduce (in expectation) the role of other dimensions in the selection process.

This would be a fairly banal and inconsequential observation if we lived in a world where everyone who claimed membership in the open science community shared more or less the same values. But we clearly don’t. In highlighting the ambiguity of the term open science, I’m not just saying hey, just so you know, there are a lot of different activities people call open science; I’m saying that, at this point in time, there are a few fairly distinct sub-communities of people that all identify closely with the term open science and use it prominently to describe themselves or their work, but that actually have fairly different value systems and priorities.

Basically, we’re now at the point where, when someone says they’re an open scientist, it’s hard to know what they actually mean.

It wasn’t always this way; I think ten or even five years ago, if you described yourself as an open scientist, people would have identified you primarily with the movement to open up access to scientific resources and promote greater transparency in the research process. This is still roughly the first thing you find on the Wikipedia entry for Open Science:

Open science is the movement to make scientific research (including publications, data, physical samples, and software) and its dissemination accessible to all levels of an inquiring society, amateur or professional. Open science is transparent and accessible knowledge that is shared and developed through collaborative networks. It encompasses practices such as publishing open research, campaigning for open access, encouraging scientists to practice open notebook science, and generally making it easier to publish and communicate scientific knowledge.

That was a fine definition once upon a time, and it still works well for one part of the open science community. But as a general, context-free definition, I don’t think it flies any more. Open science is now much broader than the above suggests.

It’s bad politics

You might say, okay, but so what if open science is an ambiguous term; why can’t that be resolved by just having people ask for clarification? Well, obviously, to some degree it can. My response to the SIPS student was basically a long and winding one that involved a lot of conditioning on different definitions. That’s inefficient, but hopefully the student still got the information he wanted out of it, and I can live with a bit of inefficiency.

The bigger problem though, is that at this point in time, open science isn’t just a descriptive label for a set of activities scientists often engage in; for many people, it’s become an identity. And, whatever you think the value of open science is as an extensional label for a fairly heterogeneous set of activities, I think it makes for terrible identity politics.

There are two reasons for this. First, turning open science from a descriptive label into a full-blown identity risks turning off a lot of scientists who are either already engaged in what one might otherwise call “best practices”, or who are very receptive to learning such practices, but are more interested in getting their science done than in discussing the abstract merits of those practices or promoting their use to others. If you walk into a room and say, in the next three hours, I’m going to teach you version control, and there’s a good chance this could really help your research, probably quite a few people will be interested. If, on the other hand, you walk into the room and say, let me tell you how open science is going to revolutionize your research, and then proceed to either mention things that a sophisticated audience already knows, or blitz a naive audience with 20 different practices that you describe as all being part of open science, the reception is probably going to be frostier.

If your goal is to get people to implement good practices in their research—and I think that’s an excellent goal!—then it’s not so clear that much is gained by talking about open science as a movement, philosophy, culture, or even community (though I do think there are some advantages to the latter). It may be more effective to figure out who your audience is, what some of the low-hanging fruit are, and focus on those. Implying that there’s an all-or-none commitment—i.e., one is either an open scientist or not, and to be one, you have to buy into a whole bunch of practices and commitments—is often counterproductive.

The second problem with treating open science as a movement or identity is that the diversity of definitions and values I mentioned above almost inevitably leads to serious rifts within the broad open science community—i.e., between groups of people who would have little or no beef with one another if not for the mere fact that they all happen to identify as open scientists. If you spend any amount of time on social media following people whose biography includes the phrases “open science” or “open scientist”, you’ll probably know what I’m talking about. At a rough estimate, I’d guess that these days maybe 10 – 20% of tweets I see in my feed containing the words “open science” are part of some ongoing argument between people about what open science is, or who is and isn’t an open scientist, or what’s wrong with open science or open scientists—and not with substantive practices or applications at all.

I think it’s fair to say that most (though not all) of these arguments are, at root, about deep-seated differences in the kinds of values I mentioned earlier. People care about different things. Some people care deeply about making sure that studies can be accurately reproduced, and only secondarily or tertiarily about the diversity of the people producing those studies. Other people have the opposite priorities. Both groups of people (and there are of course many others) tend to think their particular value system properly captures what open science is (or should be) all about, and that the movement or community is being perverted or destroyed by some other group of people who, while perhaps well-intentioned (and sometimes even this modicum of charity is hard to find), just don’t have their heads screwed on quite straight.

This is not a new or special thing. Any time a large group of people with diverse values and interests find themselves all forced to sit under a single tent for a long period of time, divisions—and consequently, animosity—will eventually arise. If you’re forced to share limited resources or audience attention with a group of people who claim they fill the same role in society that you do, but who you disagree with on some important issues, odds are you’re going to experience conflict at some point.

Now, in some domains, these kinds of conflicts are truly unavoidable: the factors that introduce intra-group competition for resources, prestige, or attention are structural, and resolving them without ruining things for everyone is very difficult. In politics, for example, one’s nominal affiliation with a political party is legitimately kind of a big deal. In the United States, if a splinter group of disgruntled Republican politicians were to leave their party and start a “New Republican” party, they might achieve greater ideological purity and improve their internal social relations, but the new party’s members would also lose nearly all of their influence and power pretty much overnight. The same is, of course, true for disgruntled Democrats. The Nash equilibrium is, presently, for everyone to stay stuck in the same dysfunctional two-party system.

Open science, by contrast, doesn’t really have this problem. Or at least, it doesn’t have to have this problem. There’s an easy way out of the acrimony: people can just decide to deprecate vague, unhelpful terms like “open science” in favor of more informative and less controversial ones. I don’t think anything terrible is going to happen if someone who previously described themselves as an “open scientist” starts avoiding that term and instead opts to self-describe using more specific language. As I noted above, I speak from personal experience here (if you’re the kind of person who’s more swayed by personal anecdotes than by my ironclad, impregnable arguments). Five years ago, my talks and papers were liberally sprinkled with the term “open science”. For the last two or three years, I’ve largely avoided the term—and when I do use it, it’s often to make the same point I’m making here. E.g.,:

For the most part, I think I’ve succeeded in eliminating open science from my discourse in favor of more specific terms like reproducibility, transparency, diversity, etc. Which term I use depends on the context. I haven’t, so far, found myself missing the term “open”, and I don’t think I’ve lost brownie points in any club for not using it more often. I do, on the other hand, feel very confident that (a) I’ve managed to waste fewer people’s time by having to follow up vague initial statements about “open” things with more detailed clarifications, and (b) I get sucked into way fewer pointless Twitter arguments about what open science is really about (though admittedly the number is still not quite zero).

The prescription

So here’s my simple prescription for people who either identify as open scientists, or use the term on a regular basis: Every time you want to use the term open science—in your biography, talk abstracts, papers, tweets, conversation, or whatever else—pause and ask yourself if there’s another term you could substitute that would decrease ambiguity and avoid triggering never-ending terminological arguments. I’m not saying that the answer will always be yes. If you’re confident that the people you’re talking to have the same definition of open science as you, or you really do believe that nobody should ever call themselves an open scientist unless they use git, then godspeed—open science away. But I suspect that for most uses, there won’t be any such problem. In most instances, “open science” can be seamlessly replaced with something like “reproducibility”, “transparency”, “data sharing”, “being welcoming”, and so on. It’s a low-effort move, and the main effect of making the switch is that other people will have a clearer understanding of what you mean, and may be less inclined to argue with you about it.

Postscript

Some folks on twitter were concerned that this post makes it sound as if I’m passing off prior work and ideas as my own (particularly as relates to the role of diversity in open science). So let me explicitly state here that I don’t think any of the ideas expressed in this post are original to me in any way. I’ve heard most (if not all) expressed many times by many people in many contexts, and this post just represents my effort to distill them into a clear summary of my views.

There is no “tone” problem in psychology

Much ink has been spilled in the last week or so over the so-called “tone” problem in psychology, and what to do about it. I speak here, of course, of the now infamous (and as-yet unpublished) APS Observer column by APS Past President Susan Fiske, in which she argues rather strenuously that psychology is in danger of falling prey to “mob rule” due to the proliferation of online criticism generated by “self-appointed destructo-critics” who “ignore ethical rules of conduct.”

Plenty of people have already weighed in on the topic (my favorite summary is Andrew Gelman’s take), and to be honest, I don’t really have (m)any new thoughts to offer. But since that’s never stopped me before, I will now proceed to throw those thoughts at you anyway, just for good measure.

Since I’m verbose but not inconsiderate, I’ll summarize my main points way up here, so you don’t have to read 6,500 more words just to decide that you disagree with me. Basically, I argue the following points:

  1. There is nothing wrong with the general tone of our discourse in psychology at the moment.
  2. Even if there was something wrong with the tone of our discourse, it would be deeply counterproductive to waste our time talking about it in vague general terms.
  3. Fear of having one’s scientific findings torn apart by others is not unusual or pathological; it’s actually a completely normal–and healthy–feeling for a scientist.
  4. Appeals to fairness are not worth taking seriously unless the argument is pitched at the level of the entire scientific community, rather than just the sub-community one happens to belong to.
  5. When other scientists do things we don’t like, it’s pointless and counterproductive to question their motives.

There, that’s about as much of being brief and to the point as I can handle. From here on out, it’s all adjective soup, mixed metaphor, and an occasional literary allusion*.

1. There is no tone problem

Much of the recent discussion over how psychologists should be talking to one another simply takes it for granted that there’s some deep problem with the tone of our scientific discourse. Personally, I don’t think there is (and on the off-chance we’re doing this by vote count, neither do Andrew Gelman, Chris Chambers, Sam Schwarzkopf, or NeuroAnaTody). At the very least, I haven’t seen any good evidence for it. As far as I can tell, all of the complaints about tone thus far have been based exclusively on either (a) a handful of rather over-the-top individual examples of bad behavior, or (b) vague but unsupported allegations that certain abusive practices are actually quite common. Neither of these constitutes a satisfactory argument, in my view. The former isn’t useful because anecdotes are just that. I imagine many people can easily bring to mind several instances of what seem like unwarranted attacks on social media. For example, perhaps you don’t like the way James Coyne sometimes calls out people he disagrees with:

Or maybe you don’t appreciate Dan Gilbert calling a large group of researchers with little in common except their efforts to replicate one or more studies as “shameless little bullies”:

I don’t doubt that statements like these can and do offend some people, and I think people who are offended should certainly feel free to publicly raise their concerns (ideally by directly responding to the authors of such remarks). Still, such cases are the exception, not the norm, and academic psychologists should appreciate better than most people the dangers of over-generalizing from individual cases. Nobody should labor under any misapprehension that it’s possible to have a field made up of thousands of researchers all going about their daily business without some small subset of people publicly being assholes to one another. Achieving zero instances of bad behavior cannot be a sane goal for our field (or any other field). When Dan Gilbert called replicators “second-stringers” and “shameless little bullies,” it did not follow that all social psychologists above the age of 45 are reactionary jackasses. For that matter, it didn’t even follow that Gilbert is a jerk. The correct attributions in such cases–until such time as our list of notable examples grows many times larger than it presently is–are that (a) reasonable people sometimes say unreasonable things they later regret, or (b) some people are just not reasonable, and are best ignored. There is no reason to invent a general tone problem where none exists.

The other main argument for the existence of a “tone” problem—and one that’s prominently on display in Fiske’s op-ed—is the gossipy everyone-knows-this-stuff-is-happening kind of argument. You could be excused for reading Fiske’s op-ed and coming away thinking that verbal abuse is a rampant problem in psychology. Consider just one paragraph (but the rest of it reads much the same):

Only what’s crashing are people. These unmoderated attacks create collateral damage to targets’ careers and well being, with no accountability for the bullies. Our colleagues at all career stages are leaving the field because of the sheer adversarial viciousness. I have heard from graduate students opting out of academia, assistant professors afraid to come up for tenure, mid-career people wondering how to protect their labs, and senior faculty retiring early, all because of methodological terrorism. I am not naming names because ad hominem smear tactics are already damaging our field. Instead, I am describing a dangerous minority trend that has an outsized impact and a chilling effect on scientific discourse.

I will be the first to admit that it sounds very ominous, all this talk of people crashing, unmoderated attacks with no accountability, and people leaving the field. But before you panic, you might want to consider an alternative paragraph that, at least from where I’m sitting, Fiske could just as easily have written:

Only what’s crashing are people. The proliferation of flashy, statistically incompetent findings creates collateral damage to targets’ careers and well being, with no accountability for the people who produce such dreck. Our colleagues at all career stages are leaving the field due to the sheer atrocity of its standards. I have heard from graduate students opting out of academia, assistant professors suffering from depression, mid-career people wondering how to sustain their research, and senior faculty retiring early, all because of their dismay at common methodological practices. I am not naming names because ad hominem smear tactics are already damaging our field. Instead, I am describing a dangerous trend that has an outsized impact and a chilling effect on scientific progress.

Or if you don’t like that one, maybe this one is more your speed:

Only what’s crashing are our students. These unmoderated attacks on students by their faculty advisors create collateral damage to our students, with no accountability for the bullies. Our students at all stages of graduate school are leaving the field because of the sheer adversarial viciousness. I have heard from graduate students who work 90-hour weeks, are afraid to have children at this stage of their careers, or have fled grad school, all out of fear of being terrorized by their advisors. I am not naming names because ad hominem smear tactics are already damaging our field. Instead, I am describing a dangerous trend that has an outsized impact and a chilling effect on scientific progress.

If you don’t like that one either, feel free to crib the general structure and play fill in the blank with your favorite issue. It could be low salaries, unreasonable publication expectations, or excessively high teaching loads; whatever you like. The formula is simple: first, you find a few people with (perfectly legitimate) concerns about some aspect of their professional environment; then you just have to (1) recount those stories in horrified tones, (2) leave out any mention of exactly how many people you’re talking about, (3) provide no concrete details that would allow anyone to see any other side to the story, and (4) not-so-subtly imply that all hell will break loose if this problem isn’t addressed some time real soon.

Note that what makes Fiske’s description unproductive and incendiary here is not that we have any reason to doubt the existence of the (anonymous) cases she alludes to. I have no doubt that Fiske does in fact hear regularly from students who have decided to leave academia because they feel unfairly targeted. But the thing is, it’s also an indisputable fact that many (in absolute terms) students leave academia because they have trouble getting along with their advisors, because they’re fed up with the low methodological standards in the field, or because they don’t like the long, unstructured hours that science requires.

The problem is not that Fiske is being untruthful; it’s that she’s short-circuiting the typical process of data- and reason-based argument by throwing lots of colorful anecdotes and emotional appeals at us. No indication is provided in her piece—or in my creative adaptations—as to whether the scenarios described are at all typical. How often, we should be asking ourselves, does it actually happen that people opt out of academia, or avoid seeking tenure, because of legitimate concerns about being unfairly criticized by their colleagues? How often do people leave the field because our standards are so terrible? Just how many psychology faculty are really such terrible advisors that their students regularly quit? If the answer to all of these questions is “extremely rarely”–or if there is reason to believe that in many cases, the story is not nearly as simple as the way Fiske is making it sound–then we don’t have systematic problems that deserves our collective attention; at worst, we have isolated cases of people behaving badly. Unfortunately, the latter is a malady that universally afflicts every large group or organization, and as far as I know, there is no known cure.

From where I’m sitting, there is no evidence of an epidemic of interpersonal cruelty in psychology. There has undeniably been a rapid increase in open, critical commentary online; but as Chris Chambers, Andrew Gelman, and others have noted, this is much better understood as a welcome democratization of scientific discourse that levels the playing field and devalues the role of (hard-earned) status than some kind of verbal war to the pain between rival psychological ideologies.

2. Three reasons why complaining about tone is a waste of time

Suppose you disagree with my argument above (which is totally cool—please let me know why in the comments below!) and insist that there clearly is a problem with the tone of our discourse. What then? Well, in that case, I would still respectfully suggest that if your plan for dealing with this problem is to complain about it in general terms, the way Fiske does—meaning, without ever pointing to specific examples or explaining exactly what you mean by “critiques of such personal ferocity” or “ad hominem smear tactics”—then you’re probably just wasting your time. Actually, it’s worse than that: not only are you wasting your own time, but you’re probably also going to end up pouring more fuel on the very fire you claim to be trying to put out (and indeed, this is exactly what Fiske’s op-ed seems to have accomplished).

I think there are at least three good reasons to believe that spending one’s time arguing over tone in abstract terms is a generally bad idea. Since I appear to have nothing but time, and you appear to still be reading this, I’ll discuss each of them in great gory detail.

The engine-on-fire view of science

First, unlike in many other domains of life, in science, the validity or truth value of a particular viewpoint is independent of the tone with which that viewpoint is being expressed. We can perhaps distinguish between two ways of thinking about what it means to do science. One approach is what we might call the negotiation model of science. On this model, when two people disagree over some substantive scientific issue, what they’re doing is trying to find a compromise position that’s palatable to both parties. If you say your finding is robust, and I say it’s totally p-hacked, then our goal is to iterate until we end up in a position that we both find acceptable. This doesn’t necessarily mean that the position we end up with must be an intermediate position (e.g., “okay, you only p-hacked a tiny bit”); it’s possible that I’ll end up entirely withdrawing my criticism, or that you’ll admit to grave error and retract your study. The point is just that the goal is, at least implicitly, to arrive at some consensual agreement between parties regarding our original disagreement.

If one views science through this kind of negotiation lens, concerns about tone make perfect sense. After all, in almost any other context when you find yourself negotiating with someone, it’s a generally bad idea to start calling them names or insulting their mother. If you’re hawking your goods at a market, it’s probably safe to assume that every prospective buyer has other options–they can buy whatever it is they need from some other place, and they don’t have to negotiate specifically with you if they don’t like the way you talk to them. So you watch what you say. And if everyone manages to get along without hurling insults, it’s possible you might even successfully close a deal, and go home one rug lighter and a few Euros richer.

Unfortunately, the negotiation model isn’t a good way to think about science, because in science, the validity of one’s views does not change in any way depending on whether one is dispositionally friendly, or perpetually acts like a raging asshole. A better way to think about science is in terms of what we might call, with great nuance and sophistication, the “engine-on-fire” model. This model can be understood as follows. Suppose you get hungry while driving a long distance, and pull into a convenience store to buy some snacks. Just as you’re opening the door to the store, some guy yells out behind you, “hey, asshole, your engine’s on fire!” He then continues to stand around and berate you while you call for emergency services and frantically run around looking for a fire extinguisher–all without ever lifting a finger to help you.

Two points about this story should be obvious. First, the guy who alerted you to your burning engine is very likely a raging asshole. And second, the fact that he’s a raging asshole doesn’t absolve you in any way from taking steps to put out your flaming engine. It may absolve you from saying thank you to him after the fact, but his unpleasant demeanor unfortunately doesn’t mean you can just choose to look the other way out of spite, and calmly head inside to buy your teriyaki beef jerky as the flames outside engulf your vehicle.

For better or worse, scientific disagreements are more like the engine-on-fire scenario than the negotiation scenario. Superficially, it may seem that two people with a scientific disagreement are in a process of negotiation. But a crucial difference is that if one person inexplicably decides to start yelling at the other–even as they continue to toss out methodological or theoretical criticisms (“only a buffoon of a scientist could fail to model stimulus as a random factor in this design!”)–their criticisms don’t become any less true in virtue of their tone. This doesn’t mean that tone is irrelevant and should be ignored, of course; if a critic calls you names while criticizing your work, it’s perfectly reasonable for you to object to the tone they’re using, and ask that they avoid personal attacks. Unfortunately, you can’t compel them be nice to you, and the fact remains that if your critic decides to keep yelling at you, you still have a professional obligation to address the substance of their arguments, no matter how repellent you find their tone. If you don’t respond at all–either by explaining why the concern is invalid, or by adjusting your methodological procedures in some way–then there are now two scientific assholes in the world.

Distinguishing a bad case of the jerks from a bad case of the feels isn’t always easy

Much of the discussion over tone thus far has taken, as its starting point, people’s hurt feelings. Feelings deserve to be taken seriously; scientists are human beings, and the fact that the merit of a scientific argument is indepedendent of the tone used to convey it doesn’t mean we should run roughshod over people’s emotions. The important point to note, though, is that the opposite point also holds: the fact that someone might be upset by someone else’s conduct doesn’t automatically mean that the other party is under any obligation–or even expectation–to change their behavior. Sometimes people are upset for understandable reasons that nevertheless do not imply that anyone else did anything wrong.

Daniel Lakens recently pointed this problem out in a nice blog post. The fundamental point is that it’s often impossible for scientists to cleanly separate substantive intellectual issues from personal reputation and ego, because it’s simply a fact that one’s intellectual output is, to varying extents, a reflection of one’s abilities as a scientist. Meaning, if I consistently put out work that’s heavily criticized by other researchers, there is a point at which that criticism does in fact begin to impugn my general ability as a scientist–even if the criticism is completely legitimate, impersonal, and never strays from substantive discussion of the intellectual issues.

Examples of this aren’t hard to find in psychology. To take just one widely-cited example: among the best-replicated findings in behavioral genetics (and indeed, all of psychology) is the finding that most traits show high heritability (typically on the order of 50%) and little influence of shared environment (typically close to 0%). In other words, an enormous amount of evidence suggests that parents have minimal influence on how their children will eventually turn out, independently of the genes they pass on. Given such knowledge, the scientifically honest thing to do, it would seem, is to assume that most child-parent behavioral correlations are largely driven by heritable factors rather than by parenting. Nevertheless, a large fraction of the developmental literature consists of researchers conducting purely correlational studies and drawing strong conclusions about the causal influence of parenting on children’s behavior on the basis of observed child-parent correlations.

If you think I’m exaggerating, consider the latest issue of Psychological Science, where we find a report of a purely longitudinal study (no randomized experiment, and no behavioral genetic component) that claims to find evidence of “a positive link between more nurturing family environments in childhood and greater security of attachment to spouses more than 60 years later.” The findings, we’re told in the abstract, “…underscore the far-reaching influence of childhood environment on well-being in adulthood.” The fact that 50 years of behavioral genetics studies have conclusively demonstrated that all, or nearly all, of this purported parenting influence is actually accounted for by genetic factors does not seem to deter the authors. The terms “heritable” or “genetic” do not show up anywhere in the article, and no consideration at all is given to the possibility that the putative effect of warm parental environment is at least partly (and quite possibly wholly) spurious. And there are literally thousands of other papers just like this one in the developmental literature–many of them continually published in some of our most prestigious journals.

Now, an important question arises: how is a behavioral geneticist supposed to profesionally interact with a developmental scientist who appears to willfully ignore the demonstrably small influence of parenting, even after it is repeatedly pointed out to him? Is the geneticist supposed to simply smile and nod at the developmentalist and say, “that’s nice, you’re probably right about how important attachment styles are, because after all, you’re a nice person to talk to, and I want to keep inviting you to my dinner parties”? Or should she instead point out—repeatedly, if need be—the critical flaw in purely correlational designs that precludes any serious causal conclusions about parenting? And if she does the latter—always in a perfectly civil tone, mind you—how can that sentiment possibly be expressed in a way that both (a) is taken seriously enough by the target of criticism to effect a meaningful change in behavior, and (b) doesn’t seriously injure the target’s feelings?

This example highlights two important points, I think. First, when we’re being criticized, it can be very difficult to determine whether our critics are being unreasonable jerks, or are instead quite calmly saying things that we just don’t want to hear. As such, it’s a good idea to give our critics the benefit of the doubt, and assume they have fundamentally good intentions, even if our gut response is to retaliate as if they’re trying to cast our firstborn child into a giant lake of fire.

Second, unfortunate as it may be, being a nice person and being a good scientist are often in fundamental tension with one another–and virtually all scientists are frequently forced to choose which of the two they want to prioritize. I’m not saying you can’t be both a nice person and a good scientist on average. Of course you can. I’m just saying that there are a huge number of individual situations in which you can’t be both at the same time. If you ever find yourself at a talk given by one of the authors of the Psychological Science paper I mention above, you will have a choice between (a) saying nothing to the speaker during the question period (a “nice” action that hurts nobody’s feelings, but impedes scientific progress), and (b) pointing out that the chief conclusion expressed during the talk simply does not follow from any of the evidence presented (a “mean” action that will probably hurt the speaker’s feelings, but also serves to brings a critical scientific flaw to the attention of other scientists in the audience).

Now, one could potentially mount a reasonable argument in favor of being either a nice person, or a good scientist. I’m not going to argue that the appropriate thing to do is to always to put science ahead of people’s feelings. Sometimes there can be good reasons to privilege the latter. But I don’t think we should pretend that the tension between good science and good personal relationships doesn’t exist. My own view, for what it’s worth, is that people who want to do science for a living should accept that they are going to be regularly and frequently criticized, and that hurt feelings and wounded egos are part and parcel of being cognitively limited agents with deep emotions who spend their time trying to understand something incredibly difficult. This doesn’t mean that it’s okay to yell at people or call them idiots in public–it isn’t, and we should work hard collectively to prevent such behavior. But it does mean that at some point in one’s scientific career–and probably at many, many points–one may have the distinctly unpleasant experience of another scientist saying “I think the kind of work you do is fundamentally not capable of answering the questions you’re asking,” or, “there’s a critical flaw in your entire research program.” In such cases, it’s understandable if one’s feelings are hurt. But hurt feelings don’t in any way excuse one from engaging seriously with the content of the criticism. Listening to people tell us we’re wrong is part of the mantle we assume when we decide to become scientists; if we only want to talk to other people when they agree with us, there are plenty of other good ways we can spend our lives.

Who’s actually listening?

The last reason that complaining about the general tone of discourse seems inadvisable is that it’s not clear who’s actually listening. I mean, obviously plenty of people are watching the current controversy unfold in the hold on, let me get some popcorn sense. But the real question is, who do we think is going to read Fiske’s commentary, or any other commentary like it, and think, you know what–I see now that I’ve been a total jerk until now, and I’m going to stop? I suspect that if we were to catalogue all the cases that Fiske thinks of as instances of “ad hominem smear tactics” or “public shaming and blaming”, and then ask the perpetrators for their side of the story, we would probably get a very different take on things. I imagine that in the vast majority of cases, what people like Fiske see as behavior that’s completely beyond the pale would be seen by the alleged perpetrators as harsh but perfectly reasonable criticism–and apologies or promises to behave better in future would probably not flow very freely.

Note that I’m emphatically not suggesting that the actions in question are always defensible. I’m not passing any judgment on anyone’s behavior at all. I have no trouble believing that in some of the cases Fiske alludes to, there are probably legitimate and serious causes for concern. But the problem is, I see no reason to think that in cases where someone really is being an asshole, they’re likely to stop being an asshole just because Fiske wrote an op-ed complaining about tone in general terms. For example, I personally don’t think Andrew Gelman’s criticism of Cuddy, Norton, or Fiske has been at all inappropriate; but supposing you do think it’s inappropriate, do you really think Gelman is going to stop vigorously criticizing research he disagrees with just because Fiske wrote a column calling for civility?

We therefore find ourselves in a rather unfortunate situation: Fiske’s appeal is likely to elicit both heartfelt nods of approval from anyone who feels they’ve ever been personally attacked by a “methodological terrorist”, and shrieks of indignation and moral outrage from anyone who feels Fiske is mistaking their legitimate criticism for personal abuse. What it’s not likely to elicit much of is serious self-reflection or change in behavior—if for no other reason that it doesn’t describe any behavior in sufficient detail that anyone could actually think, “oh, yes, I see how that could be perceived as a personal attack.” In trying to avoid “damaging our field” by naming names, Fiske has, ironically, ended up writing a deeply divisive piece that appears to have only fanned the flames. I don’t think this is an accident; it seems to me like the inevitable fate of any general call for civility of this kind that fails to actually define or give examples of the behavior that is supposed to be so offensive.

The moral of the story is, if you’re going to complain about “critiques of such personal ferocity and relentless frequency that they resemble a denial-of-service attack” (and you absolutely should, if you think you have a legitimate case!), then you need to point to concrete behaviors that people can consider, evaluate, and learn from, and not just throw out vague allusions to “public shaming and blaming”, “ignoring ethical rules of conduct”, and “attacking the person and not the work”.

3. Fear of criticism is important—and healthy

Accusations of actual bullying are not the only concern raised by Fiske and other traditionalists. One of the other recurring themes that have come up in various commentaries on the tone of our current discourse is a fear of future criticism–and in particular, of being unfairly “targeted” for attack. In her column, Fiske writes that targets “often seem to be chosen for scientifically irrelevant reasons: their contrary opinions, professional prominence, or career-stage vulnerability.” On its face, this concern seems reasonable: surely it would be a bit unseemly for researchers to go running around gunning for each another purely to satisfy their petty personal vendettas. Science is supposed to be about the pursuit of truth, not vengeance!

Unfortunately, there is, so far as I can see, no possible way to enforce an injunction against pettiness or malicious intent. Nor should we want to try, because that would require a rather active form of thought policing. After all, who gets to decide what was in my head when I set out to replicate someone else’s study? Do we really want editors or reviewers passing judgment on whether an author’s motives for conducting a study were pure–and using that as a basis to discount the actual findings reported by the study? Does that really seem to Fiske like a good way to improve the tone of scientific discourse?

For better or worse, researchers do not–and cannot–have any right not to fear being “targeted” by other scientists–no matter what the motives in question may be. To the contrary, I would argue that a healthy fear of others’ (possibly motivated) negative evaluations is a largely beneficial influence on the quality of our science. Personally, I feel a not-insubstantial amount of fear almost any time I contemplate the way something I’ve written will be received by others (including these very words–as I’m writing them!). I frequently ask myself what I myself would say if I were reading a particular sentence or paragraph in someone else’s paper. And if the answer is “I would criticize it, for the following reasons…”, then I change or remove the offending statement(s) until I have no further criticisms. I have no doubt that it would do great things for my productivity if I allowed myself to publish papers as if they were only going to be read by friendly, well-intentioned colleagues. But then the quality of my papers would also decrease considerably. So instead, I try to write papers as if I expect them to be read by a death panel with a 90% kill quota. It admittedly makes writing less fun, but I also think it makes the end product much better. (The same principle also applies when seeking critical feedback on one’s work from others: if you only ever ask friendly, pleasant collaborators for their opinion on your papers, you shouldn’t be surprised if anonymous reviewers who have no reason to pull their punches later take a somewhat dimmer view.)

4. Fairness is in the eye of the beholder

Another common target of appeal in arguments about tone is fairness. We find fairness appeals implicitly in Fiske’s op-ed (presumably it’s a bad thing if some people switch careers because of fear of being bullied), and explicitly in a number of other commentaries. The most common appeal is to the negative career consequences of being (allegedly) unfairly criticized or bullied. The criticism doesn’t just impact on one’s scientific findings (goes the argument); it also makes it less likely that one will secure a tenure-track position, promotion, raise, or speaking invitations. Simone Schnall went so far as to suggest that the public criticism surrounding a well-publicized failure to replicate one of her studies made her feel like “a criminal suspect who has no right to a defense and there is no way to win.”

Now, I’m not going to try to pretend that Fiske, Schnall, and others are wrong about the general conclusion they draw. I see no reason to deny Schnall’s premise that her career has suffered as a result of the replication failure (though I would also argue that the bulk of that damage is likely attributable to the way she chose to respond to that replication failure, rather than to the actual finding itself). But the critical point here is, the fact that Schnall and others have suffered as a result of others’ replication failures and methodological criticisms is not in and of itself any kind of argument against those replication efforts and criticisms. No researcher has a right to lead a successful career untroubled and unencumbered by any serious questioning of their findings. Nor do early-career researchers like Alec Beall, whose paper suggesting that fertile women are more likely to wear red shirts was severely criticized by Andrew Gelman and others. It is lamentably true that incisive public criticism may injure the reputation and job prospects of those whose work has been criticized. And it’s also true that this can be quite unfair, in the sense that there is generally no particular reason why these particular people should be criticized and suffer for it, while other people with very similar bodies of work go unscathed, and secure plum jobs or promotions.

But here’s the thing: what doesn’t seem fair at the level of one individual is often perfectly fair–or at least, unavoidable–at the level of an entirely community. As soon as one zooms out from any one individual, and instead surveys the field of psychology as a whole, it becomes clear that the job and reputation markets are, to a first approximation, a zero-sum game. As Gelman and many other people have noted, for every person who doesn’t get a job because their paper was criticized by a “replicator”, there could be three other candidates who didn’t get jobs because their much more methodologically rigorous work took too long to publish and/or couldn’t stack up in flashiness to the PR-grabbing work that did win the job lottery. At an individual level, neither of these outcomes is “fair”. But then, very little in the world of professional success–in any field–is fair; almost every major professional outcome, good or bad, is influenced by an enormous amount of luck, and I would argue that it is delusional to pretend otherwise.

At root, I think the question we should ask ourselves, when something good or bad happens, is not: is it fair that I got treated [better|worse] than the way that other person over there was treated? Instead, it should be: does the distribution of individual outcomes we’re seeing align well with what maximizes the benefit to our community as a whole? Personally, I find it very difficult to see trenchant public criticism of work that one perceives as sub-par as a bad thing–even as I recognize that it may seem deeply unfair to the people whose work is the target of that criticism. The reason for this is that an obvious consequence of an increasing norm towards open, public criticism of people’s work is that the quality of our work will, collectively, improve. There should be no doubt that this shift will entail a redistribution of resources: the winners and losers under the new norm will be different from the winners and losers under the old norm. But that observation provides no basis for clinging to the old norm. Researchers who don’t like where things are currently headed cannot simply throw out complaints about being “unfairly targeted” by critics; instead, they need to articulate principled arguments for why a norm of open, public scientific criticism would be bad for science as a whole–and not just bad for them personally.

5. Everyone but me is biased!

The same logic that applies to complaints about being unfairly targeted also applies, I think, to complaints about critics’ nefarious motives or unconscious biases. To her credit, Fiske largely avoids imputing negative intent to her perceived adversaries–even as she calls them all kinds of fun names. Other commentators, however, have been less restrained–for example, suggesting that “there’s a lot of stuff going on where there’s now people making their careers out of trying to take down other people’s careers”, or that replicators “seem bent on disproving other researchers’ results by failing to replicate”. I find these kinds of statements uncompelling and, frankly, unseemly. The reason they’re unseemly is not that they’re wrong. Actually, they’re probably right. I don’t doubt that, despite what many reformers say, some of them are, at least some of the time, indeed motivated by personal grudges, a desire to bring down colleagues of whom they’re envious, and so on and so forth.

But the thing is, those motives are completely irrelevant to the evaluation of the studies and critiques that these people produce. The very obvious reason why the presence of bias on the part of a critic cannot be grounds to discount an study is that critics are not the only people with biases. Indeed, applying such a standard uniformly would mean that nobody’s finding should ever be taken seriously. Let’s consider just a few of the incentives that could lead a researcher conducting novel research, and who dreams of publishing their findings in the hallowed pages of, say, Psychological Science, to cut a few corners and end up producing some less-than-reliable findings:

  • Increased productivity: It’s less work to collect small convenience samples than large, representative ones.
  • More compelling results: Statistically significant results generated in small samples are typically more impressive-looking than one’s obtained from very large samples, due to sampling error and selection bias.
  • Simple stories: The more one probes a particular finding, the greater the likelihood that one will identify some problem that questions the validity of the results, or adds nuance and complexity to an otherwise simple story. And “mixed” findings are harder to publish.

All of these benefits, of course, feed directly into better prospects for fame, fortune, jobs, and promotions. So the idea that a finding published in one of our journals should be considered bias-free because it happened to come first, while a subsequent criticism or replication of that finding should be discounted because of personal motives or other biases is, frankly, delusional. Biases are everywhere; everyone has them. While this doesn’t mean that we should ignore them, it does mean that we should either (a) call all biases out equally–which is generally impossible, or at the very least extremely impractical–or (b) accept that doing so is not productive, and that the best way to eliminate bias over the long term is to pit everyone’s biases against each other and let logical argument and empirical data decide who’s right. Put differently, if you’re going to complain that Jane Doe is clearly motivated to destroy your cherished finding in order to make a name for herself, you should probably preface such an accusation with the admission that you obviously had plenty of motivation to cut corners when you produced the finding in the first place, since you knew it would help you make a name for yourself. Asymmetric appeals that require one to believe that bias exists in only one group of people simply don’t deserve to be taken seriously.

Personally, I would suggest that we adopt a standard policy of simply not talking about other people’s motivations or biases. If you can find evidence of someone’s bias in the methods they used or the analyses they conducted, then great–you can go ahead and point out the perceived flaws. That’s just being a good scientist. But if you can’t, then what was in your (perceived) adversary’s head when she produced her findings is quite irrelevant to scientific discourse–unless you think it would be okay for your critics to discount your work on the grounds that you clearly had all kinds of incentives to cheat.

Conclusions

Uh, no. No conclusions this time–this post is already long enough as is. And anyway, I already posted all of my conclusions way back at the beginning. So you can scroll all the way up there if you want to read them again. Instead, I’m going to try to improve your mood a tiny bit (if not the tone of the debate) by leaving you with this happy little painting automagically generated by Bot Ross:


* I lied! There were no literary allusions.

in praise of self-policing

It’s IRB week over at The Hardest Science; Sanjay has an excellent series of posts (1, 2, 3) discussing some proposed federal rule changes to the way IRBs oversee research. The short of it is that the proposed changes are mostly good news for people who do minimal risk-type research with human subjects (i.e., stuff that doesn’t involve poking people with needles); if the changes pass as written, most of us will no longer have to file any documents with our IRBs before running our studies. We’ll just put in a short note saying we’ve determined that our studies are excused from review, and then we can start collecting data right away. It’ll work something like this*:

This doesn’t mean federal oversight of human subjects research will cease, of course. There will still be guidelines we all have to follow. But instead of making researchers jump through flaming hoops preemptively, enforcement will take place on an ad-hoc basis and via random audits. For the most part, the important decisions will be left to investigators rather than IRBs. For more details, see Sanjay’s excellent breakdown.

I also agree with Sanjay’s sentiment in his latest post that this is the right way to do things; researchers should police themselves, rather than employing an entire staff of people whose jobs it is to tell researchers how to safely and ethically do their research. In principle, the idea of having trained IRB analysts go over every study sounds nice; the problem is that it takes a very long time, generates a lot of extra work for everyone, and perhaps most problematically, sets up all sorts of perverse incentives. Namely, IRB analysts have an incentive to be pedantic (since they rarely lose their jobs if they ask for too much detail, but could be liable if they give too much leeway and something bad happens), and investigators have an incentive to off-load their conscience onto the IRB rather than actually having to think about the impact of their experiment on subjects. I catch myself doing this more often than I’d like, and I’m not really happy about it. (For instance, I recently found myself telling someone it was okay for them to present gruesome pictures to subjects “because the IRB doesn’t mind that”, and not because I thought the psychological impact was negligible. I gave myself twenty lashes for that one**.) I suspect that, aside from saving everyone a good deal of time and effort, placing the responsibility of doing research on researchers’ shoulders would actually lead them to give more, and not less, consideration to ethical issues.

Anyway, it remains to be seen whether the proposed rules actually pass in their current form. One of the interesting features of the situation is that IRBs may now perversely actually have an incentive to fight against these rules going into effect, since they’d almost certainly need to lay off staff if we move to a system where most studies are entirely excused from review. I don’t really think that this will be much of an issue, and on balance I’m sure university administrations recognize how much IRBs slow down research; but it still can’t hurt for those of us who do research with human subjects to stick our heads past the Department of Health and Human Service’s doors and affirm that excusing most non-invasive human subjects research from review is the right thing to do.

* I know, I know. I managed to go two whole years on this blog without a single lolcat appearance, and now I throw it all away for this. Sorry.

** With a feather duster.