As I've been preparing to end my term here and leave Mozambique, I've become more interested in figuring out how to measure the effectiveness of our project. Part of this interest is because we need to do this as part of the work, but also I'm just curious to know the answer to the question did I even help at all?
Measuring success sounds easy, but I've spent a lot of brain power on it for a long time now and I've realized that it ain't. You would think you could just ask people if their lives have improved because of the project, but how do you measure how much better or in what ways is it better? Or even for how many people is life better?
If you built a dam in a community and people began using the water stored in it to wash their dishes, to wash their clothes, to irrigate gardens, to give to their livestock to drink, and to drink themselves, then how do you measure that? I mean, where do you even begin looking for the kind of data you need to estimate this impact at all? And what if it doesn't rain at all during the year you go out to take measurements?
This stuff makes you sympathize with No Child Left Behind, because that stuff is probably way harder than our work.
One of the impacts of our project is to reduce hunger in the area. Ending hunger is a great goal, but it turns out to be a surprisingly difficult thing to quantify. I've looked around to see how other organizations mark their progress towards ending hunger. I've written about the United Nations before and their MDGs, and fortunately they have an MDG that clearly states to reduce by half the proportion of the population suffering from hunger. So you would think we could just copy what they do, right? Nope. They measure hunger in two ways, both of which are crazy difficult. The first way is to look at the Body Mass Index of children under five. So that means going to some representative sample of children, measuring their height and weight and comparing that to some standard value. The second way is to look at the food imported and produced within the country, the population, and the inequality within that country and from that calculate an amount of calories available to the population. Neither of which we are planning to do anytime soon.
The only lessons we can take away from the UN is that we will need to use survey questionnaires and that they will be a lot of work. These surveys actually do what I said we couldn't quite do earlier - ask people how their lives are better (my bad). The difference is that we need to ask simple things like "how many vegetable plants are you growing - let's count them" or "how many minutes do you walk to get to your water source" and repeat those questions for everyone in the project.
Simplifying the questions is the hard part. There's this term used in economics called externalities. An externality is the cost or benefit of an act to someone that wasn't initially taken into account when the act was done. An example of a negative externality is the pollution you emit into the air when you drive your car to the market. The increased pollution you just added to the air is a cost that someone else has to pay (asthma) for your combustion.
Positive externalities to our project include the clothes washing, dish washing, and bathing that occur there. People clearly benefit from the dams in those ways, but since they are tangential to the main point of our project, namely water and food security. Therefore we don't ask things like "where do you wash your dishes" or "how much clothes do you wash at the dam?" Ignoring positive externalities helps us cut to the heart of what is important, and also prevents us from overstating the value of the project to our donors.
Then we need to figure out whom to ask these questions and how many households we need to visit to create a good idea of the aggregated benefits of the dam. It's impractical to ask everyone in a 1,500-member community all these questions, so we ask those questions to a part of the community (like a poll). However, we don't yet know how big that part needs to be or how many people we need to ask to get a pretty good idea of the whole. So I've been brushing up on the ole' statistics to figure this sample size out.
Lots of work and only two weeks left!