Until he found Bin Laden, the CIA Agent in charge was a 100% failure.
If the third sector had a FB page, “evidence-based’’ would be the term with the most likes these days, possibly closely followed by Theory of Change. They’re related of course, as we must ensure our actions do in fact lead to our goals or we shouldn’t exist. We need to base this on good data, not just gutfeel. But gutfeel is important when it’s actually tacit knowledge or intuition honed by years of experience, and evidence can have its limitations and even worse, introduce distortions both in resource allocation, behaviour and outcomes. As Prof. Jerry Z. Muller says in his recent book, The Tyranny of Metrics, up until he found Bin Laden, the CIA agent in charge of looking was a 100% failure.
In The Tyranny of Metrics, Prof. Muller reminds us that there is no evidence metrics are useful for pay-for-performance purposes, whether individual or collective. In fact, there is plenty of evidence it can distort outcomes. A well-known example is US Secretary of Defence Robert McNamara’s tragic use of ‘’body count’’ to judge the likelihood of the US winning the war in Vietnam. Rather, as military strategist Edward Luttwik explains: ‘’the only true measure of progress must be political and nonquantifiable: the impact on the enemy’s will to continue to fight’’. Crime statistics are notoriously unreliable as mayors pressurise police to perform better but inadvertently incentivise them to misclassify crimes e.g. theft becomes ‘’lost property’’. Muller provides many more intriguing examples from education to medicine and business and finance (quoting Larry Fink of Blackrock’s critique of ‘’quarterly earnings hysteria’’).
Where a manageable set of evidence works well is for learning, particularly collectively as an organisation. How are we going to know if we’re successful? How will we learn from our failures? It is also useful for strategy development or reformulation, for deciding ‘’where to play’’ relative to others, or where we play particularly well, and provide a high return on investment. It is useful for fundraising but only insofar as we can tell a more convincing story based on evidence. However, a disproportional focus on trying to manage complexity with metrics can lead to too much time spent on reporting, meeting and coordinating, with little left for actual doing!
Recently, the Moshal Scholarship Program (a university scholarship programme where Kate is President) had an unusually high dropout year (though we must point out this is still way below the national average!). We spent a few weeks as a team analysing the data and thinking hard together about the ‘’why’’. Not only did it surface some very important findings, but it brought the team to a new level of insight, determination and skill to address these. For example, we discovered that computer sciences courses had an alarming failure rate at certain universities, and we had taken a steep new intake of students who had probably never seen a computer, and needed some kind of bridging course. We needed to support those still in the system doing high failure courses, with even more tuition. More systemically, we realised that high school grades have become even worse predictors of ability to succeed at university, and we are now introducing our own academic potential assessment as part of the selection process.
On the other hand, we have an expensive annual event that we can’t directly link to our performance indicators but through our own ‘’gutfeel’’ and qualitative research, we know makes a massive difference to motivation and belonging among first generation students, and is a key factor in keeping our alumni engaged and committed.
At the Moshal Scholarship Program, we are also very clear about the limits of metrics. Organisations that focus too much on the metrics for success, and forget to keep an eye on their vision and mission, can run the risk of going astray, dampening creativity, or avoiding risk when risk is important for learning and innovating. If we simply focused on KPIs like dropouts and completion on time, we could select great performers or exclude everyone who failed the first test. Hey presto - the problem goes away! But so would the mission of the Program: to enable determined and resilient students from challenging backgrounds to obtain sought-after university degrees, soft skills and values, that lead to successful professional careers and lives. And that must always lead the metrics.
Kate Kuper is a Director at Bateleur Partners – katekuper@bateleurpartners.net
