Foreign Policy Evaluation: Challenges — and Solutions — for Causal Identification and Policy Relevance
Abstract: International relations scholars increasingly engage in, what is commonly referred to as, program or policy evaluation. Leveraging experimental and quasi-experimental tools for causal identification, this research seeks to understand the effect that some policy intervention has on outcomes of interest. The appeal of this sort of research is undeniable, but scholars of international relations should pause before jumping headfirst into foreign policy evaluation for two reasons: foreign policy effects are both particularly difficult to identify; and, even when well-identified, of limited use to foreign policymakers.
First, I argue that typical features of foreign policy and international relations are likely to violate all three of the assumptions necessary for observational analyses of foreign policy effects to have a causal interpretation. The problem arises because foreign policy responses generally combine multiple instruments of statecraft into strategies that are crafted for the specific circumstances that states confront and because actors in international relations have the opportunity to learn from and adapt their behavior in response to past policy implementations. In this context, meeting the assumptions that allow for a causal interpretation of relationships may be impossible or demand research designs that are not regularly employed.
Second, I argue that average treatment effects -- the finding that comes out of most policy evaluation -- are of little use to foreign policymakers, who generally have the luxury of personalizing specific policy responses to each of the situations they confront. While average treatment effects tell us about the overall effect of a treatment on a group as a whole -- information that is useful when policymakers must implement a single policy over many units -- they tell us nothing about the likely effect of a treatment on a specific unit. In the context of personalized policymaking, average treatment effects do not provide foreign policymakers with the information that they need.
After diagnosing these problems, I propose new research designs for producing causally-identified research on foreign policy effects and for producing research that is truly relevant to foreign policymaking.