How do you know a program is working

The Byerly Foundation is in the community building business and we do this by providing grants for programs that the Byerly Board hopes will make a significant difference in helping Hartsville become an even better place to live and one of the best places to live -- anywhere. That is a stretch goal, by almost any defihttp://www.blogger.com/img/blank.gifnition.

One of the major problems the Foundation has and nearly all Foundations and community development types of programs is how to know if programs work. Normally the groups implementing the programs will set their goals and list the measures they want to use to measure effectiveness. Then you assess the program based on the approved metrics. Sometimes this gets you the information you want and sometimes it doesn't.
More-Than-Good-Intentions
One thing that those working with Foundations and other funders do is continue to explore ways to measure effectiveness. This week the Stanford Innovation Review has a review of a book that discusses how programs might be measured. Just a few minutes ago I ordered this book from Burry Bookstore -- More Than Good Intentions: How a New Economics Is Helping to Solve Global Poverty by Dean Karlan & Jacob Appel

According to the reviewer, Kevin Starr, "Karlan and Appel believe that understanding what works for poverty alleviation programs boils down to one deceptively simple question: “How did people’s lives change with the program, compared with how they would have changed without it?” The primary—but not only—tool that Karlan et al. use to answer that question is the randomized controlled trial (RCT). In an RCT, a pool of subjects is randomly divided into intervention and control groups; the former gets the interventions and the latter does not. The two groups are fundamentally alike—both are measured before and after, and the impact is the difference between what happened to the intervention group and to the control group. RCTs are not new. The novel element here is the systematic and creative application of RCTs to test poverty solutions in the real world."

This is an interesting concept because when you are working on social interventions, it is difficult to keep something that is working away from control groups and often times you have try designing your research project to somehow give those randomized out a chance for being in. While I have not discussed this fully with Dr. Eve Puffer I believe this is a tactic she has used in a major innovation project fighting the spread of HIV/AIDS in Muhuru Bay, Kenya. The Byerly Foundation is currently funding a major project with the Hartsville Middle School called Foxy Leaders that aims to take students at real risk of not finishing school and giving them an even stronger foundation than they might get in regular classes. I am not sure how this could be an RCT. That is why it is going to be interesting to get that book and discover what they authors have found in their research.