Social Pressure and Self-Selection in Experimental Policy Evaluation

A leading technique uses randomized experimentation to evaluate the impact of policy in- terventions. Applications of this technique often measure the impact of an intervention on a self-selected sample. In this paper, I propose that social pressure from the experimenter is a determinant of these selection decisions. I test this hypothesis in the context of a study on a free LED lighting program. Recruitment takes place door-to-door in the suburbs of Chicago. A day before their recruitment visit, each household is informed about the study with a flyer on their doorknob. I isolate the role of social pressure by varying whether households can select out of the study by checking an opt-out box on their flyer. My main finding is that the LED lighting program causes a 15 percent reduction in evening energy use in the sample recruited with an opt-out flyer, whereas no energy savings are observed amongst households recruited with a baseline flyer. Social pressure appears to be the driver of these disparate effects, as the opt-out flyer causes a 13 percent reduction in households answering their door and, conditional on answering, causes a 52 percent increase in selection into the study. These results have important implications for the way evaluation experiments with self-selected samples are conducted, reported, and modeled.