Definition of evaluation
Using a description by the United Nations Development Programme, an evaluation should be a rigorous and structured assessment of a completed or ongoing activity, intervention, programme or policy that will determine the extent to which it is achieving its objectives and contributing to decision-making (Menon, Karl and Wignaraja, 2009).
In simple terms, an evaluation tells us what works and what does not.For example, did a policy or project reach its intended audience, was it implemented as planned, did it achieve its desired impacts, how and why did it have its effects?
When should an evaluation be conducted?
It is important to conduct an evaluation when:
• there has been substantial investment of resources (e.g. time or money)
• there is any chance of risk or harm
• the project/policy approach is new or innovative
• the intervention is the subject of public or political scrutiny
• there is a knowledge gap about how to address a problem or provide a particular service
Evaluation stages
There are usually 4 stages:
1. Defining the evaluation questions, for example:
• what do you want to discover?
• what outcomes are you assessing, and for whom?
• what time frame are you reviewing?
2. Data collection
3. Analysing the data collected to answer your evaluation questions
4. Clarifying the implications of the findings and producing recommendations
Types of evaluation
1. Outcome evaluation
Outcome evaluations are assessments of the results of an intervention and measure the changes brought about by the intervention (WK Kellogg Foundation, 2004). This is done by the collection and analyses of data on specific outcomes that are thought to be influenced by the intervention, thereby clarifying how effective the intervention is at changing those outcomes.
2. Process evaluation
Process evaluations assess whether a policy is being implemented as intended and what, in practice, is felt to be working more or less well, and why (HM Treasury, 2011). Process evaluations do not primarily focus on outcomes but rather on how an intervention or service works and are usually conducted alongside outcome evaluations (see above). They typically collect data on different aspects of the intervention, using mixed methods. A key component of a process evaluation is the creation of a logic model explaining how the intervention is expected to generate outcomes.
Process evaluations can help us understand why an intervention works in one area but not in another. They can also highlight for whom an intervention works best, enabling optimal targeting of interventions or services
3. Economic evaluation
Economic evaluations are assessments of the value gained from and the costs of resources used to implement a policy, programme or intervention (HM Treasury, 2011). An economic evaluation can clarify the costs and benefits of an intervention compared to an alternative course of action, thereby assisting decision-makers in allocating future resources, setting priorities and shaping policy.
Economic evaluations are based on the assessment and valuation of resources (for example, staff time) to estimate the costs involved in and stemming from an intervention helping to assess the outcomes or benefits, usually expressed in terms of quality-adjusted life years
Evaluation risks
Risks to consider:
• Care is needed when constant changes have been made to the intervention as the evaluation could be premature and inconclusive
• Right timing is essential in case the intervention is too early in development (unless the evaluation is designed as a formative evaluation with the aim of improving an intervention)
• A lack of clarity or consensus on objectives could make it difficult to establish the evaluations purpose
Conclusion
Evaluations are a necessary part of business as usual or a change intervention, therefore it is essential that the right method is chosen to address the requirement. Please contact Smart Consult today if an appraisal is required in your organisation.
Should you have a research project or request for proposal that you wish to discuss, then please submit the request here.
