Kindji Village, Aguégués Region, Ouémé District, Benin. 19 February 2013.
This is the second installment on my recent visit to CARE programs in Southern Benin. CARE, one of the world’s largest humanitarian and poverty alleviation organizations, began working in Benin in 1999.
Two weeks ago, I piled into a car and headed out of Cotonou to visit a few programs in the field along with ten other CARE board members. Our first stop was Aguégués, a group of 20-odd villages and 34,000 people on Lake Nokoue, a large, shallow saltwater basin where most houses are built over the lake on stilts and sanitation consists of a hole in the floor.
My first reaction: in low-lying Southern Benin, the gravest threat to human well-being is global warming. The area is prone to flooding, which seems to be getting worse. In 2010, the region suffered the worst disaster since the 1960s, making over 100,000 people homeless. CARE began operating in the region after the floods, first providing emergency assistance and later graduating to a range of programs aimed at reducing poverty.
We were greeted in Kindji, as we were at all the sites we visited, by hordes of dancing villagers in a song-filled procession, by a panel of local officials who made speeches expressing their gratitude, and by youth groups singing in CARE-branded t-shirts. This was a bit awkward and reminded me of popular critiques of big NGOs that accuse donors of “poverty tourism” versus genuine interest in understanding whether their funding is making a difference in people’s lives. I recall one article by Victoria Schlesinger about a donor visit to Jeff Sachs’ Millennium Village project that infuriated me — the trip unfolded like an elaborate theatre production, complete with crop counts doctored by villagers. The biggest value that Sachs’ project seemed to offer was a massive investment in a poor community, rather than an innovative model for ending extreme poverty.
Silly as it seemed, the official reception we received is an important aspect of CARE’s work and philosophy. In Benin, CARE staff told me the only way they could scale programs is with the help of community leaders who assign great value to hospitality. In much of West Africa, local chiefs, elders, and town mayors play a powerful role in the economic and social well-being of people in their communities. It seems hard to avoid the pomp and ceremony and also attain the buy-in of community leaders necessary to improve people’s lives.
I tried whenever I could to leave the group and speak directly with beneficiaries of CARE programs to get a better sense of our impact. My findings were mixed. Unlike newer models like Charity:water, GiveDirectly, or Samasource, which tend to focus on one narrow type of intervention, CARE follows the pattern of most large international NGOs — it’s essentially a portfolio of various programs funded by tightly restricted grants from different aid agencies. So in a country like Benin, CARE runs several parallel programs in each village in which it operates. In Kindji, there was a “Child to Child” sanitation program run by local youth, a Village Committee for Civil Protection to help the community prepare for natural disasters, a Village Committee for Environmental Protection, which encourages villagers to plant trees, and a Village Savings and Loan Association (VSLA). The idea, according to CARE Benin’s local staff, is to lift a community out of poverty by applying all the tools we have at once, rather than piecemeal efforts.
Village savings groups are one of CARE’s signature programs in Africa. As explained by the people we met, the model consists of teaching a group of women in a village how to set up a revolving savings and loan fund. Each woman contributes a tiny amount (50 cents to a dollar, in Benin) to the fund at regular intervals, and is eligible for a much larger loan (ten to twenty dollars) from the general pool about once a year. VSLAs cost next to nothing to start up, require no outside financing, and, due to their locally-driven nature, have the potential to “go viral” and spread without much intervention by outside NGOs. They seem like a no-brainer.
But the skeptic in me wanted to know whether VSLAs actually make poor women better off in the long term, and whether they had an effect on health, education, and other outcomes at a family level. I talked to a few women members of Aguegue’s VSLA to get some anecdotal evidence. One woman, Filomene, told me she was 50 years old and had four children. She’d lost her husband and her eldest son a few years ago after spending well over 20,000 CFA (US$40) and borrowing money to send them to a local hospital (a problem CARE hopes to improve by investing in health initiatives in the village). Last year, she got a loan of 10,000 CFA (about US$ 20) and used it to buy firewood. She earned about US$3 in profit over the course of one month. But before she’d sold all the wood, Filomene had already spent the profit on her living expenses, which add up to about US$3 a day for school fees, meals, and health expenses.
Though VSLA hadn’t done much to improve her financial situation, Filomene liked the idea and praised CARE for brining it to Kindji. Other women in the village had similar feedback on the program — it was certainly better than what they’d had before, and it gave them pride and a sense of community, but it didn’t seem to be lifting anyone out of poverty. It made me wonder: would CARE be better off spending what it costs to administer the VSLA program on direct cash transfers to women like Filomene, with no strings attached?
Last year, Nick Kristof wrote a glowing piece about a CARE village savings program he visited in Malawi fueled by the powerful story of a woman who used a tiny loan to start a business and fund her children’s education. It was emotional and encouraging — but it didn’t offer any data.
The good news is that CARE has taken steps to get such data, and to share it with the outside world. The organization commissioned an official evaluation of the program elsewhere in Africa by Innovations for Poverty Action (IPA), one of the most respected impact evaluation groups. The study found in a sample of 7,000 households that the effects of VSLAs on health, education, hunger, and poverty levels are negligible, though authors admitted that more time might be needed to see a bigger effect. IPA, like MIT economist Esther Duflo’s Poverty Action Lab, uses randomized controlled trials to help answer the question “in the absence of this program, what would have happened to people in this village?” as best we can. The approach is often compared with clinical trials, the argument being: if we wouldn’t subject people to a drug that hadn’t been tested rigorously, why should we subject them to a development program without ample evidence that it works?
The problem with evaluating impact in a broad model like CARE’s is that it’s terribly messy. One has several different outcomes to measure, and the task of determining causality in such a mixed bag is a nightmare. If villagers are less poor at the end of the measurement period, is it because of the savings association, civil protection, or something else? How do we know how each program contributes to the outcome, and thus which programs are most effective and should receive more funding? CARE operates in over eighty countries, each serving thousands of villages with the same impact measurement challenges. The result is an extremely complex system and few ways to measure true outcomes across the breadth of a large NGO’s programs.
Many new charities and social enterprises thus orient themselves around narrower interventions with clear outcomes. At Samasource, we set annual goals for a few unifying impact metrics: the number of people we move from below the poverty line to above it, and the “donor-dollar ratio” (a rough donor return on investment or ROI calculation). We make no distinction between different types of people, except for a requirement that at least 50% of our workforce be made up of women — all human lives have equal weight in our model.
In theory, Samasource’s impact philosophy makes it easier to understand the incremental impact of more donations and to make decisions about which populations to target. It also helps us stay lean and focus (something we’re big on; to kick off the year, we handed out copies of Eric Ries’s The Lean Startup to staff and unveiled company-wide experiments in a few areas). In our early days, for example, we tried to work in a remote refugee camp on the Somali border in Kenya. We found that the cost to impact a refugee, given security and infrastructure needs, was at least triple what it cost us to employ a similarly needy person in another part of Kenya. The following year, we made the tough decision to focus on where we could achieve the biggest “bang for the buck” in terms of poverty alleviation, and scaled back our efforts in the refugee camp.
But unlike CARE, Samasource is only operational in a handful of countries, and has only employed a few thousand people. It’s easy to keep it simple when you’re small. The big question is how to do this as an organization grows large. We have very few examples of singular impact-driven organizations in the international development world that report consistently on a key metric and the cost of achieving it. The best may be in global health — vaccine programs and mosquito nets were among the first innovations to be subjected to a randomized-controlled trial approach — and microfinance. With any luck, some of the “narrow” interventions I described earlier will scale up quickly in the next few years and offer patterns for others to follow.
Most large organizations aren’t incentivized to rigorously track impact; doing so is painful and often thankless. Mother Teresa famously bristled at attempts to quantify her charity’s outcomes, believing that hard-nosed ROI calculations had no place in helping the destitute. She kept no logbooks of the people she served, the amount of money her branches spent, or the impact her charities achieved. Consequently, we have no way to know whether her methods were more or less effective than others at alleviating suffering. Many argue that the methods used in her hospices ignored commonly-accepted standards of basic medical care and actually worsened some patients’ conditions. Donors, however, were taken with Mother Teresa’s humble reputation and the images they saw on the news–few took the time to understand whether her work was actually making people better off.
CARE is obviously quite different from the Missionaries of Charity. The organization has taken great steps to systematize impact measurement as best it can, publishing regional impact reports and engaging external evaluators where possible. And overall, despite my reservations about the outcomes of some of the programs we saw, the visit to Kindji was inspiring. In a short time, CARE has made moves to localize its operation, hiring entirely Beninese or West African staff, and operates without much of the overhead of large international organizations. This in itself is a feat. And I am not alone on the board, or in senior management — I voiced many of the ideas in this piece in the car on the drive home and heard support from others in the organization who’ve come to similar conclusions. Now our challenge is to figure out how to scale up what works.