Evaluation Today – From Bullet Points to Business Models

What is evaluation? “Evaluation is a systematic determination of a subject’s merit, worth and significance, using criteria governed by a set of standards.” Says Wikipedia. So this is about determination, judgement, fact finding and then measurement against criteria. We all know these criteria: effectiveness, efficiency, sustainability etc. etc. And we all know the rationale for doing evaluations: accountability and learning, in mixed combinations. Let’s leave aside the accountability factor for now, even if unfortunately it can become the dominant force in evaluations in quite a few cases, often driven by relationships with donors. Let’s focus on the learning side. No evaluation without recommendations – giving clear and concrete guidance as to how to improve the work of the organisation. In that sense, working from findings to recommendations, an evaluation can be seen as a ‘closed’ package of work: looking at the facts, judging them and then charting out the way ahead. In practice this can often turn into a rather operational or tactical endeavour: a list of practical next steps, a ‘to do list’ for the organisation. One step further these recommendations are part (at least in theory) of a cyclic approach where strategies are designed, scrutinised and then adjusted on the basis of learning.

This approach can work well in a situation where we work in a relatively stable strategic environment – what we are doing then is trying to optimise the workings of the organization. We are talking reparation here, fixing flaws inside an existing paradigm – checking, adapting and improving the links in the theories of change that are used in programs or campaigns. We adjust. But perhaps today more than ever we should not just be working in a virtuous cycle, in a helix that moves steadily upwards, but we need to throw the whole helix away, build an entirely new one and re-interpret what we see as an organisation and what there is to do.

Today we often work in a dynamic strategic environment where adjusting individual strategies may not be enough. In today’s world not only the scale of the issues we are dealing with is perhaps bigger than ever, the levels of knowledge and awareness, and the levels of resistance are changing too. We seem to be part of that Angry Young Women scenario (see this site under ‘Facing your Futures’, the Global Engagement Scenarios) where a multifaceted global crisis leads to resistance to all sorts of systemic arrangements, from the way universities are run to the way the international financial system is moving ahead. This resistance and politisation leads to new opportunities but potentially also to completely new roles for NGOs. In other areas (e.g. those of religion, refugees and migrants) the debate is experiencing levels of polarisation that have not been there probably for decades, so that activists see themselves confronted with hardened and much more challenging contexts. In this environment probably ‘every’ evaluation should be one that informs deep strategic thinking – not just at theory of change, campaign or program level but also at organisation level.

This will change what the evaluation should ‘do’ and how it should look like. What will come into view more and more is questioning an organisation’s business model: it’s deepest strategic convictions, its key approaches and its vital role in a sector. Work with clients shows that in looking at the effectiveness of an organisation in an environment that experiences radical change, it is not enough to test the impact of individual theories of change. It is inevitable to start looking at some long standing boundaries and to start discussing some strong convictions: ‘We do objective, law based human rights work but we stay away from politics.’ ‘We do education, not lobby or campaigning.’ ‘Our core business is resistance and containment, not the facilitation of alternatives.’ Nowadays in a lot of cases it’s these key strategic choices, these ‘pre-cooked’ strategic ingredients that may not be the optimal approach anymore, or worse: that will lead to missing crucial opportunities to achieve the mission, or work for the values of an organisation. At the same time formulating changes and alternatives at this level of an organisation’s strategy is not a thing that can be simply taken of the shelf and put in a list of bullet points, let alone be addressed in the timeframe of a typical evaluation. In that sense a good evaluation these days may be one that leads not only to facts, recommendations and then ‘closure’ but also to much more discussion and the inherently uncomfortable uncertainty, the probing and the questions that are there at the level of a strategic conversation. Please refer to the diagram below that maps out the spectrum moving from accountability via learning to strategy. No doubt ‘opening up’ will turn an evaluation into a different type of enterprise: not delivering concrete goods but opening up unknown spaces. The evaluator takes on the challenging and uncomfortable role of doubt-monger, not that of the more comfortable producer of concrete improvement packages. This approach will no doubt get the evaluator involved in deep and interesting strategic conversations where there are no quick fixes – but in quite a few cases she might also face outright rejection of her findings and questions as they are challenging the status quo – not at program but at organisational level. Making sure there is room for these discussions when setting the scope and TOR of the evaluation is obviously the best approach here but there will inevitably be cases where evaluator and commissioning organisation will have to engage with issues as they emerge during the work

Addressing organisational, operational, tactical as well as strategic issues (ranging from the programmatic to the business model level) in evaluations is of course not new. What may be new is the balance between these different perspectives – see the box to assess your practice of evaluation. This changing balance will make today’s and tomorrow’s evaluations more challenging, more relevant and definitively more fun.