Technical critique of Brookings Institution's FamilyScape sex and contraception simulator

[fnr]

The Brookings Institution is a well-respected policy think-tank that is often cited by executive agencies. A group of its researchers have developed a simulation model, FamilyScape, which predicts pregnancy and abortion rates for U.S. women. Policy analyses based on this model have been cited by the U.S. Department of Health and Human Services’ (DHHS) Title X family planning program as evidence for its effectiveness.

I plan to offer a systematic technical review of this model in coming weeks, particularly its Architectural Overview. This model is an “agent-based” simulator, which includes rules of behavior for each simulated actor (“agent”) in the model.

My primary basis of critique is that the agent behavior model is exclusively based on short-term behavioral responses to the immediate social and political environment. The model apparently includes no reference to behavioral-economic studies of abortion and contraception policy changes, for example, Klick and Stratmann’s 2003 study on state-level gonorrhea prevalence rates following the liberalization of abortion laws. I would also like to evaluate whether the model is able to simulate the lack of response in population-wide pregnancy rates associated with the introduction of emergency contraceptive pills. These studies suggest a longer-term behavioral response to the contraceptive and abortion policy environment, which FamilyScape appears not to address. Any simulation that does not account for how policy affects behavior is not a suitable model for policy analysis.

In the mean time, I wanted to share this model with the CAF forum, and see if anyone else would like to help with the technical review of the model. Computer scientists, engineers, epidemiologists, actuaries, economists, physicians, risk assessors, statisticians, mathematicians, decision theorists, physicists, biomedical scientists, quantitative ecologists, and others with specialized training would be especially helpful in offering a critique of this model.

 

[fnr]

I will point out that the overall Brookings project of which FamilyScape is a part, the Social Genome Project, has indicated a willingness to consider the types of studies I mentioned, which tend to use methods like difference-in-difference and other observational (vs. experimental) methods. For example, in the Social Genome Project overview document:

We have relied on randomized controlled trials or quasi-natural experimental estimates, whenever possible, and given great attention to data issues. We also rely, as much as possible, on external empirical work that is based on a good research design or identification strategy in answering any question. Studies that have used instrumental variables, regression discontinuity methods, differences-in-differences analysis, fixed effects, and the results of randomized trials will be consulted. Estimates based on a credible research design are always to be favored over those based on a more naïve approach. (See especially Angrist and Pischke in the Journal of Economic Perspectives, Spring 2010.) But we also agree with Jim Heckman that it is possible –and often desirable –to combine the two approaches. (See “Building Bridges between Structural and Program Evaluation Approaches to Evaluating Policy,” Journal of Economic Literature, June 2010.) That is, one can use the best estimates from the evaluation literature or from a quasi-experimental study and combine them with a parsimonious structural model in which one doesn’t attempt to identify every parameter. The key is to be able to predict a baseline outcome with a set of right-hand side variables and to include a good parameter estimate for the change in outcomes associated with on e’s policy variable. Note that the emphasis is on getting the policy-induced change in an outcome right and using experimental or quasi-experimental evidence for this purpose. This will always be our preferred approach. Although we have done some modeling within life stages, we have moved away from a focus on within-stage analysis and are making estimating the transition probabilities between life stages in combination with a policy-induced treatment effect the top priority for the future.