WhatIf: Software for Evaluating Counterfactuals

Heather Stoll, Gary King, Langche Zeng


What would happen if pigs could fly? The first known attempt to answer this question was in 1909 by J.T.C. Moore-Brabazon, who earlier the same year was the first British pilot to fly in Britain. On the left is Moore-Brabazon in his personal French-built Voisin aero plane. On the right is a pig in a wicker basket behind a sign that says "I am the first pig to fly."

Inferences about counterfactuals are essential for prediction, answering "what if" questions, and estimating causal effects. However, when the counterfactuals posed are too far from the data at hand, conclusions drawn from well-specified statistical analyses become based largely on speculation hidden in convenient modeling assumptions that few would be willing to defend. Unfortunately, standard statistical approaches assume the veracity of the model rather than revealing the degree of model-dependence, which makes this problem hard to detect. WhatIf offers easy-to-apply methods to evaluate counterfactuals that do not require sensitivity testing over specified classes of models. If an analysis fails the tests offered here, then we know that substantive inferences will be sensitive to at least some modeling choices that are not based on empirical evidence, no matter what method of inference one chooses to use. WhatIf is also used to identify the areas of common support in causal inference. It is implemented in MatchIt and can easily process Zelig output objects so that counterfactuals can be evaluated, prior to computing quantities of interest, with only one additional command.

WhatIf implements the methods for evaluating counterfactuals discussed in Gary King and Langche Zeng, 2006, "The Dangers of Extreme Counterfactuals," Political Analysis 14 (2): 131-159; and Gary King and Langche Zeng, 2007, "When Can History Be Our Guide? The Pitfalls of Counterfactual Inference," International Studies Quarterly 51 (March): 183-210.