MindBlog Dansk

MindLab

citizen-centred innovation - anthropological methods - service design - public development - communication - idea and concept development - innovation strategy - cross-institutional collaboration

Christian Bason

“We believe in evidence-based policy. Yeah, right!”

By August 22nd 2012

This article has previously been published in the Danish weekly, Mandag Morgen

There is much talk about measurement, and how public services must deliver better results. But the fact is that only very few politically governed organisations base their operations on their ability to create a positive impact on the world.

Shortly after I started my first government job, I overheard a conversation about a brand new policy programme evaluation. The discussion between the two civil servants went something like this: “Well now we have to explain these evaluation results in the parliamentary committee. And then we need to say that we take them seriously.” To which the colleague replied: “Yes, because here we believe in ‘evidence-based policy’” – which prompted mutual laughter.

I was somewhat surprised by this conversation. I had dedicated a significant part of my professional life as an evaluation consultant and had actively participated in setting up the Danish Evaluation Society (Dansk Evalueringsselskab – DES). I was convinced that public organisations should, fundamentally, measure and operate based on the difference they make to citizens and society. In other words: Are we generating the learning and well-being in our schools that we want? Are we getting the health benefits we are paying for? Are we really helping our unemployed back into work? In short: Is Denmark becoming a better place in which to live and work as a result of public organisations’ efforts? At the end of the day, that’s what “results” are all about.

Now that I have spent half a decade in the public sector, it seems quite natural to me that civil servants – whether they work for the state, a region or a municipality – laugh at results measurement and at the double-digit millions of kroner we spend on it every year.

Firstly, many other important factors determine good decisions in public organisations, beyond concerns for “documented” impact. The most important factor is presumably political considerations, both tactical and strategic, including how to make the minister or mayor look good.  Another (related) factor could include the appropriate timing of publishing evaluation results. It could also be consideration for the organisation’s existing mission, operational priorities and focus, for its employees, for various stakeholders, etc.

Secondly, many impact measurements and evaluations are encumbered by a lack of reliability. They can be based on a weak empirical foundation; they can be difficult to interpret; data can arrive too late to be considered as a basis for decision-making; and there can – first and foremost – be a lack of clarity surrounding the causal connections: In a complex social environment, is it really our organisation’s efforts, or lack thereof, that are responsible for the unsatisfactory results?

Thirdly, many organisations are not at all clear about what their effects are, or should be. In my experience, a great number of them confuse activities (what is being done, such as delivering teaching) with outputs (e.g. that a number of students have completed a course) with effects (that the students have in fact learned something).

One could say that many organisations have not, at the end of the day, managed to put great effort into completely clarifying whether they have had a positive or a negative influence on their surroundings. And they do not seek to create systematic, organisational learning in connection with their efforts, such as through experiments or user involvement.

Finally, a factor that I have only recently started to realise is this: The Ministry of Finance, at least in Denmark, is not particularly preoccupied with results. They are more interested in the public sector’s relationship to inputs, in other words money and other production resources. At a time when public sector finances are under severe pressure and a new Budget Law is in the offing, this is not particularly surprising. But if the Ministry – which traditionally sets the tone for public sector management, reforms and good governance – does not exactly embrace result- and impact measurement, then who should?

Indeed, what justifies the public sectors existence is not a specific use of funds; rather, it is that the public sector makes a positive difference to citizens and to society – and (one might be tempted to add) does it for as few resources as possible.

We must therefore not reduce efforts to measure “results” to tactical political manoeuvring. Results and evidence are not that difficult to understand, nor that difficult to work with in a systematic manner. We have far too much to lose if we fail to take it seriously. And that is no laughing matter.

Comment RSS

Post a comment