Che materia stai cercando?

Anteprima

ESTRATTO DOCUMENTO

Contents

Introduction 9

I Presentation of evaluation tools 15

■ Tool, function, method 17

■ Properties of evaluation tools 19

■ Advantages and risks of technical complexity 22

■ Steps and functions of an evaluation 24

■ Evaluation situations 28

II Tools for the overall evaluation of a programme 33

Examples of the evaluation of multi­sectoral programmes 35

Tools for structuring an evaluation 39

H SWOT Analysis (Strengths­Weaknesses­Opportunities­Threats) 41

■ Concept mapping of impacts 47

■ Colour vote 55

Tools for observing changes in the field 59

■ Ind ivid ual interview 61

■ Focus Group 67

■ Case study 73

Tools for analysing data 81

■ Geographical Information Systems 83

■ Shift­Share analysis 89

■ Input­Output Model 95

■ Macro­economic model 103

Tools for making a judgement 115

■ Expert panels 117

■ Multicriteria analysis 123

III Tools for in­depth treatment of an evaluative question 129

Examples of evaluations with in­depth treatment of a question 131

Tools for structuring an evaluation 135

■ Logical framework 137

METAPLAN® 145

Principal evaluation techniques and tools

Tools for observing changes in the field 151

■ Questionnaire survey 153

■ Ethnographic observation 161

Tools for analysing data 167

■ Factor analysis 169

■ Delphi Survey 175

■ C omparison groups 179

■ Regression analysis 187

Tools for making a judgement 193

■ C ost-Effectiveness Analysis 195

■ Benchmarking 201

■ C ost-benefit analysis 207

IV Recommendations for selecting and assembling tools 217

Guide for identifying the most appropriate tools 219

Recommendations for assembling several tools 225

Conclusion 231

Index 233

Introduction

A professional who wants to know what the evaluation toolbox contains,

has to consult numerous texts and manuals devoted to particular

techniques in the economic and social sciences. Fortunately, some

reference documents present a range of tools for evaluation purposes.

Among the classics are Rossi & Freeman (1993), as well as the Program

1

Evaluation Kit collection . These books feature "technical information

sheets" illustrated by means of examples and all are organised according

to each author's own typology. The most frequent typological distinctions

separate tools in terms of their nature, i.e. quantitative or qualitative,

deductive or inductive, etc. A particularly interesting typology is presented

by C. Pollitt (Box 1).

Is there any need for a new book on evaluation tools and, if so, what might

its contribution be? The main need that this Volume has to fulfil is to make

existing knowledge readily available to readers working in the field of socio­

economic programmes. In fact, most documents on evaluation tools are

illustrated with examples of social programmes or development projects.

The present document provides a link between general methodological

works and the context of European Structural Funds.

It is secondly necessary to adapt available knowledge to specific difficulties

in the evaluation of socio-economic programmes. Most texts on evaluation

tools refer to simple programmes, aimed at a well-defined target group and

at producing precise change within that group. However, a socio-economic

development programme is typically multi-sectoral, with multiple

objectives. Its purpose is to produce economic and social transformations

on the scale of an entire territory. To achieve this goal, an integrated

strategy is adopted, oriented towards different publics (firms, workers, the

unemployed, communities, etc.). Its global impact is produced by means of

a combination of multiple specific impacts on various targets. In this

context, evaluation encounters complex technical problems for which the

existing literature provides few solutions. This book is the first

methodological guide to record and present a number of relevant solutions.

1 Rossi RH, Freeman H.E. (1993) Evaluation: a Systematic Approach (5th edition)

Newbury Park CA: Sage.

Program Evaluation Kit, Center for the Study of Evaluation, University of California.

Newbury Park CA: Sage Principal evaluation techniques and tools

Box 1 - Four approaches to evaluation

Experimental approaches treat phenomena of causality without analysing them

directly. The effects of programmes are demonstrated by comparing a treatment

group with a control group. Evaluations carried out in this framework are typical

of American practice in the 1960s and 70s. These evaluations are long,

cumbersome, and are not always conclusive. They are often criticised, sometimes

severely, by their commissioners who want operational conclusions that can be

used rapidly for improving programmes.

Economic approaches are based on an individualistic view of society, inspired by

welfare economics. From this point of view, the value of a public action is the sum

of the benefits it provides to individuals. All individuals are supposed to have a

system of references and it is assumed that these systems of reference can be

aggregated. These hypotheses are used to give an objective character to

evaluations carried out by economic techniques. Despite their elegance,

economic techniques have been strongly criticised, in both practical and

theoretical terms.

Naturalistic or pluralistic approaches are based on the idea that the political and

social world is a collective construction. This construction results from the

interaction of differing social groups that have their own interpretations of the

same phenomena and issues. Consequently public programmes are seen as

temporary compromises between groups of actors. By means of appropriate

techniques, based to a large extent on group work, evaluation acts as a mediator

between the different points of view. It is a tool for promoting conciliation between

the stakeholders, and its conclusions are all the more robust when they are a

product of consensus.

Pragmatic approaches adopt simplified views of the processes that they have to

describe. Theoretical references are less pure and more eclectic. Evaluation

objectives are of an essentially managerial rather than scientific nature. Evaluations

carried out in this spirit, using techniques inspired by management approaches,

are oriented more towards the implementation and improvement of efficiency.

Their cognitive dimension is less important than in other forms of evaluation.

Some of the technical suggestions in this Volume would not exist without

the numerous studies and experiments carried out by the European

Commission. To adapt available knowledge to the difficult field of socio­

economic programmes, substantial methodological investments were

required, including the precise definition of the basic concepts (e.g. tools,

methods), the definition of functions likely to be fulfilled by the tools, the

construction of new tools (e.g. specialised macro-economic models,

impact mapping) or the proposal of new ways of using classical tools.

Introduction

These investments have benefited the entire professional evaluation

community. This is the third and last reason for the existence of this manual.

The manual contains a detailed presentation of twenty­three evaluation

tools, in the form of technical information sheets. The choice of tools is a

compromise between the most frequently used tools and others that,

although rarer, have significant potential for the evaluation of socio­

economic programmes. Since all compromises are to some extent arbitrary,

certain frequently used tools are only mentioned in passing, without having

technical information sheets devoted to them (e.g. variance analysis).

rapid pre­selection of

Box 2 ­ For a appropriate tools O

S Θ O

High Medium Low Retrospective Prospective

technical technical technical use use

complexity complexity complexity 0

Φ Θ

@%

(è)

V^~"»// νΛ*·»*JJ) Innovative

Medium High

Low Stabilised use

cost cost cost use

A

O

Overall In­depth

evaluation evaluation

Each information sheet provides data on the functions of the tool, the

conditions of its use, and the main steps in its implementation. Possibilities

for application in the framework of Structural Fund evaluations are

suggested, and the strengths and weaknesses of the tool in practice are

highlighted. An example taken from a Structural Fund evaluation is

described, if the tool has been used in that context. Each sheet also

contains a commented bibliography of books and articles that refer to the

tool. The information sheets are accompanied by pictograms that enable

readers to rapidly select the tools applicable to a given situation. The

meaning of these pictograms is given in Box 2. Principal evaluation techniques and tools

This Volume is divided into the following four main sections:

• Presentation of evaluation tools

• Tools for the overall evaluation of a programme (12 tools)

• Tools for more in-depth treatment of an evaluative question (11 tools)

• Recommendations for selecting and assembling the tools

I

Presentation of evaluation tools

In this Volume the terms "tools" and "techniques" are taken to be

synonymous and applicable to standardised procedures which any

evaluator can learn to use and apply in carrying out her/his missions.

The evaluation "toolbox" contains over 23 techniques, presented in this

Volume. The evaluator's art therefore involves the choice of an appropriate

tool and the rejection of inappropriate ones, depending on the function they

are meant to fulfil in a particular situation.

Presentation of evaluation tools

Tool, function, method

Evaluation practitioners use various words to describe their instruments or

procedures: technique, tool, approach, procedure, method, design, etc. It

is essential to agree on certain definitions before reading this Volume. The

following example will be used to establish these definitions.

Box 3 - Example of a tool(s)-based evaluation

In 1989 the British National Audit Office evaluated eight regional development

programmes in Wales. The evaluation calculated the cost per job created, in

other words, it was based on cost-effectiveness analysis, a standard evaluation

tool. To evaluate the eight programmes, the NAO adopted an approach in several

stages:

1 - Selection of a single evaluation criterion: net jobs created.

2 - Construction of an indicator for measuring net jobs, in terms of the following

definition: net jobs = gross jobs - deadweight effects - displacement effects

+ supplier effects.

3 - Selection of a sample of about 300 assisted businesses.

4 - Questionnaire survey by telephone, estimation of deadweight and

displacement coefficients.

5 - Estimation of total jobs created by extrapolation from the coefficients

obtained from all jobs reported by all the assisted businesses.

6 - Estimation of supplier effects by extrapolation from studies of similar

programmes.

7 - Recapitulation of the public funds mobilised for the programmes, and cost-

effectiveness analysis in terms of cost per job created.

8 - Comparison with the cost per job created in the framework of similar

programmes - here, a regional development programme in Scotland.

For further information on this evaluation, see: S. Roberts and Ch. Pollitt, (1994)

Audit or Evaluation? a National Audit Office VFM Study, Public Administration

In this example, there are at least t w o tools commonly used in evaluation:

• the questionnaire survey (4th step)

• cost-effectiveness analysis (7th step)

It should also be noted that the evaluation work involved far more than

simply the use of t w o tools. At each of the eight stages, a decision was

made on how to carry out the work. Each decision depended on the

'specific context of this evaluation. The NAO thus constructed an ad hoc

evaluation method for fulfilling its mission. Principal evaluation techniques and tools

This example will be used to define three of the basic terms in this Volume:

"tool", "method" and "function".

A tool is used to carry out standard treatment and may be applied to any

evaluation requiring such treatment. For example:

• a questionnaire survey is used to gather the opinions of a group on one

or several points defined in advance;

• a cost-effectiveness analysis calculates the public funds spent to obtain

a given effect; the result is presented in a form, which allows programmes

and projects to be compared.

A technique is so similar to a tool that in this Volume they are often used as

synonyms. However, a tool can be considered to be an object (e.g. a

shovel) and a technique the way in which this object is used (e.g. the mani­

pulation of the shovel).

A method is an ad hoc procedure, specially constructed for a given

evaluation. It may include one or more tools, or none. In its evaluation of

regional development programmes in Wales, the NAO constructed a

method adapted to the questions raised (Box 3). By contrast, for the prior

appraisal of programmes proposed by the Member States, the Commission

requests appraisals that are carried out within a few weeks. In general, the

methods applied to these evaluations comprise no tools.

Thus defined, the term "method" is similar to "design". In the American

literature, "evaluation design" is applied to the way in which a given

evaluation is conducted. However, the term is often applied to the analysis

of causality, that is to say, to only one of the functions that an evaluation

method must fulfil. The terms "approach" and "procedure" are also similar.

They apply to a given method and to the spirit in which this method has

been constructed.

The method constructed for carrying out a particular evaluation can always

be broken down into several steps. Each of these steps consists of a task

and fulfils an elementary function. The same basic elementary functions

can be found in most evaluations. A function may be fulfilled with or without

a tool. For example:

• in the NAO example (Box 3) the data collection function is carried out by

means of a questionnaire survey;

• in certain evaluations of the Commission, the estimation of effects is

carried out by means of a macro-economic model.

Presentation of evaluation tools

In this Volume, methodology is defined as the "science" of the construction

of methods. The aim of this Volume is to promote the development of this

science. In reality, however, professional practice is still closer to expertise

than to a science. The word methodology is therefore somewhat ambitious

at this stage.

Properties of evaluation tools

A tool is used to process raw materials (inputs) to transform them into

products (outputs). Inputs and outputs are information, data, facts,

opinions, judgements, etc. Inputs and outputs can be different in nature.

For example, a questionnaire survey collects opinions and produces

quantitative data about these opinions. All types of information feed into an

expert panel to produce a judgement, etc. The tool can be defined because

the mode of processing is stable, that is to say, inputs have a standard

form, outputs are always of the same nature, and transformations occur

according to the same principles. Yet a tool always has a degree of

flexibility so that it can be adapted to use in specific situations. These

"adjustments" are limited so that the tool remains identifiable.

Box 4 - The road worker's shovel and cost-effectiveness analysis

The road worker's shovel is used to fulfil a standardised function, consisting of

moving a loose material from one place to another, for example, by filling a

wheelbarrow with earth. In this example, the "inputs" are the user's physical

strength and the earth to be moved, and the "output" is the earth that has been

moved in the wheelbarrow. A priori, there are not many types of shovel or different

ways of using them. One could say that this tool performs a stable transformation.

The main variants are the size of the shovel, the distance of the movement and

the nature of the material to be moved (i.e. how loose it is). The case of the shovel

may be transposed, term for term, to the field of evaluation, for example to cost-

effectiveness analysis.

The inputs in cost-effectiveness analysis are twofold. It is necessary first to

measure the impact obtained (in a single dimension, e.g. net jobs created) and

then the amount spent. The output is simple; it is the cost ratio per unit of impact.

The transformation is stable; it consists of a simple division. The degree of

freedom concerns, above all, the way of taking cost Into account: budgetary cost

for all the public partners, budgetary cost for the only public actor that commis­

sions the evaluation, opportunity cost of public funds, cost limited to the duration

of the programme or cost extended to all public support, etc.

Principal evaluation techniques and tools

A tool may be used freely by anyone who needs it. Most of the tools used

in evaluation are free of charge but some have to be bought. For example,

a tool used for conducting meetings, called METAPLAN®, presented in

this Volume, is a registered trademark. A tool has instructions for use,

described in one or more manuals. Training courses on tools and their use

are offered for beginners. Quality standards ensure that tools are used

professionally.

To summarise, a tool can be recognised from the following characteristics,

featured in B ox 5, it can be identified by a name, it is used to carry out a

process with standard inputs and outputs, it can be adapted but in a very

limited number of ways, it has a manual and quality standards. All these

properties make the tool transferable from one evaluation to another and

from one professional to another. In particular, its use is public and not

limited to its inventor.

Box 5 ­ Features of an evaluation tool

Name

Standards Standard

O ■=>¡

inputs outputs

O O O

V ' / Manual

Adjustments Quality

standard

This diagram corresponds perfectly to some of the tools presented in this

Volume, as the example of regression analysis developed below shows. In

this example, the basic characteristics of a tool can be recognised, i.e.:

• standard inputs (quantitative data on apparent effects, on participation in

the programme and on the other explanatory variables since these data

are available for a large number of individuals);

• some adjustments (number and nature of variables, nature of relations,

thresholds of validity);

Presentation of evaluation tools

• standard outputs (distribution of apparent or gross effects between the

net effects attributable to the programme and the effects of the other

explanatory variables).

Moreover, the tool is identified by means of a name (regression analysis),

irrespective of the variants corresponding to the different adjustments

(linear regression, logistic regression, etc.). Finally, the tool is described in

numerous manuals and in all European languages.

Box 6 - Regression analysis as an evaluation tool

Data for

numerous Regression

individuals : % of apparent

analysis 0

0

Apparent effect effects which are

attributable to

Participation in the programme

the programme

Other explanatory

variables

Several of the 23 information sheets presented in this Volume have only

some of the characteristics of a real "tool". For example, some have not

been fully standardised (e.g. SWOT analysis), others do not have a proper

manual (e.g. colour vote, METAPLAN) and others have no stabilised quality

standards (e.g. multicriteria analysis). Principal evaluation techniques and tools

Apart from those tools developed specifically for evaluation (e.g. cost

benefit analysis, multicriteria analysis), most tools are borrowed from the

main scientific disciplines. The questionnaire survey is taken from

sociology, regression analysis was created by statisticians, the input-output

model comes from economics, the expert panel draws largely upon legal

practices, and the geographic information system and ethnographic

observation emerged from their corresponding sciences.

When a technique is "borrowed" from a discipline for use as an evaluation

tool, it is often adapted and simplified. From a purely scientific point of view,

there are always regrets about the loss of the "canonical" form of the

original technique. However, evaluation is above all a pragmatic activity.

The technical information sheets presented in this Volume should not be

judged with the academic rigour of their original disciplines, but in terms of

their usefulness for evaluation practitioners.

Advantages and risks of technical complexity

The use of a tool has numerous advantages in terms of guaranteeing results,

evaluation content, comparability of results, and quality and cost control.

It also provides a guarantee as to the form of results, on condition that the

use of the tool complies with the established procedures. Thus, a

questionnaire survey conducted properly (e.g. confidentially, by telephone)

can serve to accurately estimate net effects.

The choice of certain tools makes it necessary to structure the entire

evaluation. For example, cost-effectiveness analysis means that the

evaluation will favour a principal effect (e.g. employment). The tool will, in

this case, offer the commissioner a guarantee of this main effect being

prioritised, if that is what is desired.

Use of a known tool facilitates the estimation of the time and budget

required for an evaluation because practitioners have references for

grounding their estimations. Moreover, the fact that an evaluation is based

on standardised tools facilitates control of the conditions of its execution.

Several evaluations based on the same tools produce results that are easier

to compare. In the example in Box 3, it was possible to compare the costs

per job created in several regions because all the evaluations had used

cost-effectiveness analysis.

Presentation of evaluation tools

In short, a more technical evaluation is often simpler to commission, easier

to control, more credible for readers of the report and more predictable for

the commissioner. All these advantages obviously make evaluation tools

highly attractive, but they are no without certain risks.

The risks of "tool(s)-based" evaluation can be summed up in the following

two sentences:

• "With a hammer in hand, I see nails everywhere" - risk of choosing the

wrong tool

• "Because I have a hammer, I can build the entire house" - risks of over­

estimating the use of a tool.

A tool must always be adapted to its context of use and to the functions it

is to fulfil. The risk is that someone with a hammer might try to use it to

hammer in a screw. In an example studied in the framework of the MEANS

programme, the evaluation commissioner and team decided to use

regression analysis. After a few weeks, they realised that the available data

were insufficient and that they were unable to apply the chosen tool. They

were forced to quickly reorganise the evaluation work. To know whether an

evaluation tool is appropriate in the context, it is necessary to examine the

object to be evaluated, the available data, the evaluation budget, and the

deadline for presentation of the results. Choosing a tool without taking all

these factors into account could have serious consequences: absence of

conclusions or irrelevant conclusions, deadlines that are not met, conclu­

sions that are not credible, etc.

The weaknesses of the tools used must also be recognised and accepted.

It is better to carry out a less technical evaluation by specifying the limits of

the work, than to base conclusions on the use of sophisticated tools which

were not skilfully used.

Since technical complexity is tempting, there is often a tendency to over­

value tools used in evaluation. Thus, the NAO evaluation (Box 3) was

presented as an exercise in "value for money", that is, cost-effectiveness

analysis. The entire evaluation was given the name of one of the tools

implemented and this tool became emblematic. This phenomenon has

become widespread and applies to numerous evaluations which,

depending on the case, may be called cost-benefit evaluation, econometric

evaluation, quality/price evaluation, etc.

Equating an evaluation to its emblematic tool is always a mistake. An

evaluation tool is used to fulfil a function and, most of the time, one function

Principal evaluation techniques and tools

only. Favouring one function amounts to neglecting all the other functions

required by evaluation, and thus undermining the overall quality of the work.

Steps and functions of an evaluation

In general, a tool fulfils the function for which it was designed, and does not

fulfil the others. An example of a specific tool is a macro-economic model,

which can fulfil no other function than that of estimating impacts on the

scale of a country or region.

Yet tools do exist which are capable of fulfilling several evaluation functions.

These are generic tools. It will be seen in this Volume that the case study

technique can be used for collecting data but also for other functions.

Similarly, an expert panel can be set up for making judgements and for

estimating impacts.

All evaluations have several steps with which one or more functions are

associated. In the example of the evaluation carried out by the NAO, it can

be seen that there were at least five functions.

Box 7 - Several steps in an evaluation

(Evaluation of economic development programmes in Wales)

Choosing the effects to be ι

evaluated (employment) \ 1

1 STRUCTURING

Defining an observation tool ι

(indicator of net jobs created)

Collecting data OBSERVING

(300 telephone interviews)

Estimating effects ANALYSING

JUDGING

(cost-effectiveness analysis)

By analysing a large number of evaluations in the same way, the most

common functions can be identified. These are presented below in eight

basic steps (Box 8), grouped together into four main stages: structuring the

Presentation of evaluation tools

evaluation, observing changes in the field, analysing data, and judging the

programme.

Box 8 ­ Standard steps of an evaluation

; Clarifying and grading the effects

; to _be_e_va[ua_ted,_defi_njng_ enterra .

Structuring I Choosing observation

l_ instruments ___..

%

; Defining the observation field ~»,

Observing r

; Collecting data

I Comparing data

Analysing J Estimating effects ,«·»'

| Judging in terms of the """■

' different criteria

Judging ι Rirmulating a synthetic judgement

The diamond­shaped diagram is borrowed from Scriven M. (1980)

The Logic of Evaluation, Inverness CA: Edgepress

The structuring steps normally occur before the evaluation work as such

(before the terms of reference are drawn up). Their place in the present

Volume may therefore be questioned. However, the evaluation team

frequently has to revert to these steps at the start of its work, for example

for explaining the judgement criteria in more detail. It will also be seen that

tools exist which may facilitate structuring work. It is therefore logical for

this Volume to cover evaluation as a whole, including the very first steps.

Clarifying and grading the effects to be evaluated, defining criteria

The first step consists of listing the questions to which the evaluation must

provide answers. For example: Are the taxpayers getting value for their

money? What are the most effective parts of the programme? Is the pro­

gramme justified? Does the programme have negative unexpected effects?

Each question can be subdivided into the effects to be evaluated, e.g. job

creation, investments induced, market share gained, increase in

employability of a category of the population or increase in the

attractiveness of a region. Each question also refers to a judgement

criterion, e.g. is the effect sought and/or obtained relevant, consistent with

the objectives, efficient, lasting and fair? Principal evaluation techniques and tools

The tools applicable in this step are used to list questions, clarify them,

grade them and select those for which the evaluation must provide answers

(see the information sheets relating to the following tools: logical

framework, impact mapping, SWOT analysis, METAPLAN®, colour vote).

Defining observation instruments

This second step consists of the translation of evaluation questions into

indicators or descriptors observable in the field. In the example of regional

development programmes in Wales (Box 3), the question was as follows: "Is

the employment effect of the programmes sufficient compared to the

money spent?" It was translated in the form of an efficiency indicator, as

follows: "cost to the national budget of a net job created".

This Volume proposes no tools for this step. The reader is referred to

Volume 2 devoted to indicators, and to Volume 4 for more information on

the construction and use of descriptors.

Defining the field of observation

This third step consists of defining a field of observation that is limited

enough not to generate excessive costs, large enough to produce credible

information, and contrasted enough to reveal the effects to be evaluated. It

is therefore a compromise involving choices such as: Should all categories

of addressees be observed? Should one or more populations which are

equivalent but which have not participated in the programme also be

observed? How can cases be selected to obtain the fullest picture possible

of the programme? Should cases of best practice be selected in order to

understand the key factors of success?

The field of observation, that is to say, the definition of "groups" to be

observed (businesses, trainees, inhabitants, farmers, etc.), should not be

confused with the field of evaluation (definition of the intervention to be

evaluated).

Collecting data

The fourth step consists of collecting primary data in the field. Numerous

survey tools exist for this purpose. They are borrowed from various

disciplines in the social sciences that collect data: sociology, political scien­

ce, ethnography, etc.

Presentation of evaluation tools

This step is essential if the evaluation cannot rely entirely on secondary data

such as the evaluation of previous research, statistical data and monitoring

data.

The tools applicable to this step allow the collection of facts and opinions

and to reconstitute them in a qualitative or quantitative form (see the

information sheets relating to the following tools: individual interview, focus

group, questionnaire survey, ethnographic observation).

Comparing observations

The fifth step consists of the presentation of primary and/or secondary data

in such a way as to suggest conclusions and comparisons that can be

made intuitively or inductively.

If the example of the evaluation of a training course for the long-term

unemployed is taken, the comparison of observations will consist of

breaking down the rate of return to work in terms of occupational sectors,

to see whether there are significant differences and to better understand the

factors of success.

The basic tools used for this function are graphs and comparative tables.

This Volume also proposes several more sophisticated tools (see the

information sheets relating to the following tools: geographical information

systems, factor analysis).

Estimating effects

The sixth step is the one in which the net effects of the programme are

estimated. This step requires an analysis of causality which almost always

follows a deductive approach, i.e. an approach based on the verification of

hypotheses of cause and effect. The technical literature on evaluation often

favours quantitative analysis tools used to fulfil this function (e.g.

comparative analysis, regression analysis).

In this respect, this Volume differs from the existing literature. It attributes

the same value to qualitative and quantitative analysis tools by

demonstrating the strengths and weaknesses of each one. Furthermore, it

does not over-value the analysis of causality, often considered as being the

single role of evaluation. The main assumption is that the functions of

observation and judgement are at least as important as the function of

estimating net effects. Principal evaluation techniques and tools

Tools applicable to this step allow the net effects of programmes to be

estimated, both in the past and the future, and at both the micro and

macro-economic level (see the information sheets relating to the following

tools: shift-share analysis, input-output models, macro-economic models,

Delphi survey, comparison groups, regression analysis).

Judging effects in terms of each criterion

The evaluation must be able to judge whether the effects that the programme

has produced or will produce are sufficient or insufficient. In the example of

regional development programmes in Wales (Box 3), the evaluation team had

to judge job creation in terms of the efficiency criterion. The tool used was

cost-efficiency analysis. It was thus possible to conclude that programmes

were more satisfactory when the cost per job created was low.

For the evaluation to be complete, it was also necessary to be able to say

whether the cost per job created was sufficiently low. The standard chosen

was the average cost per net jobs created by comparable programmes in

Scotland. By comparing Wales and Scotland it was possible to conclude

that the evaluated programmes were sufficiently efficient.

This Volume presents several tools which can be used in the judgement

step (see the information sheets relating to the following tools: cost-

effectiveness analysis, benchmarking).

Formulating a synthetic judgement

The eighth and last step in an evaluation consists of formulating conclusions

by making a final synthesis of the partial judgements on the different criteria.

This function is necessary when several criteria have been selected. To

make a synthetic judgement of a programme, it is necessary to combine,

weight or grade the different effects that the evaluation has assessed. In the

framework of socio-economic programmes, this function is particularly

difficult since there are always several partners who do not attribute the

same value to the criterion. There are several tools which totally or partially

fulfil this function (see the information sheets relating to the following tools:

expert panels, multicriteria analysis, cost-benefit analysis).

Evaluation situations

A tool is designed to fulfil a given function. There is therefore a strong

connection between tool and function. However, several tools are often

Presentation of evaluation tools

able to fulfil the same function (see Box 9). When several evaluation tools

are available, the most suitable tool for the situation has to be selected.

Box 9- Significance of the situation for the choice of a tool

The shovel is a tool of which the function is to move a loose material from one

place to another. To fill a 50-litre wheelbarrow, an ordinary shovel is used. To fill a

25-tonne truck, it would definitely be better to use a mechanical shovel.

In both examples, the function of the tool is the same, but the situation is very

different. The tool must be adapted to both its function and the situation in which

it is to be used.

For example, in a situation of massive public intervention on the scale of a

large territory, the function of estimating effects will be fulfilled by a macro-

economic model. Thus, the HERMIN model was constructed and

successfully used to estimate the macro-economic impacts of Structural

Funds in Ireland, Portugal and Spain. In a different situation, for example if

the amounts involved are small compared to the economy of the territory

under consideration, or if the territory is small, the macro-economic model

is not suited to the estimation of effects.

As the example of the HERMIN model suggests, one of the first differences

is that of the scale of effects to be evaluated, which may be micro (a small

group of potential addressees) or macro (a large region or a country). The

evaluation of an impact on the national economy requires the use of

specific tools, particularly macro-economic models, and disqualifies others

such as case studies.

Another important difference is that of partnership. In an evaluation

involving several decision-makers, the choice of effects to be evaluated and

of evaluation criteria is of major importance and has to respect the differing

points of view. Evaluations involving a multiplicity of decision-makers

require the use of open tools, such as group-work techniques or

multicriteria analysis. On the other hand, they are ill suited to closed

techniques such cost-effectiveness analysis.

The third difference is that of the use of evaluation: formative or

recapitulative. A case study, which makes it easier to understand and to

learn quickly from an experience, is a suitable tool for formative evaluation.

By contrast, this tool produces conclusions that are difficult to synthesise

for decision-makers and the public. As a result, it is less appropriate for

recapitulative evaluation. Principal evaluation techniques and tools

Box 10 - The three phases of evaluation

In the framework of European Structural Fund programmes for the period 1994-

99, evaluation is carried out in three stages.

Ex ante evaluation focuses on the planned programme. It concerns the coherence

and relevance of the project, and the realism of the expected effects.

Intermediate evaluations focus on the effects of the very first outputs of the

programme. They are used by decision-makers at all levels. They are

complementary to monitoring and are intended to reorient the programme mid­

way.

Ex post evaluations are intended primarily to report on the effects of programmes.

The fourth difference is that of the schedule (see Box 10). The evaluation

cannot use the same tools to evaluate those effects which have already

been obtained and those which are still to be produced.

The fifth difference is one of the most important for the choice of tools. It

concerns the complexity of the programme. The majority of socio­

economic programmes are multi-sectoral, that is to say, they intervene in

several different forms, are aimed at several different publics and produce

multiple kinds of impact. However, most evaluation tools have traditionally

been used in straightforward situations (a single sector of intervention, a

single public, and a predominant expected impact).

The latter difference is very important for the choice of tools. It is also

peculiar to socio-economic programmes, which is why it is a focal point in

this Volume. The presentation of tools is therefore set out to correspond to

two main situations:

• Global evaluation of an entire programme. In this situation a regional

multi-sectoral programme has to be evaluated globally, in its entire

complexity. The evaluation of impacts at a macro level is required.

• More in-depth consideration of an evaluative question. This situation is

similar to the evaluation of a simple programme, described in most

evaluation manuals.

In general, the greater the complexity, the more attention must be paid to

the functions of clarification and structuring, the higher the cost of empirical

observation, and the more difficult the analysis of causality. Normal practice

consists of breaking down the programme into categories of homogeneous

interventions. Each intervention is evaluated in terms of its relevance,

effectiveness or efficiency. The evaluation then makes a synthesis at the

Presentation of evaluation tools

programme level. The difficulty of this approach is that the different

components of the programme (measures, actions, etc.) must be evaluated

in terms of criteria common to the entire programme. If all the criteria were

specific the synthesis would be impossible. A variant of this situation is one

in which macro-economic effects on a regional or countrywide scale must

be estimated. This requires first an estimation of the micro effects of each

component of the programme according to a common analytical grid, then

the introduction of these effects into all the macro-economic mechanisms

in order, finally, to deduce the global effect which is not equal to the sum of

all the micro-economic effects.

Various issues may need to be considered in greater depth:

• Evaluations focused on an elementary component of the programme, for

example a priority axe, a measure or an action. In this case, the outputs

are homogeneous and the addressees may easily be identified and

questioned. The collection and analysis of primary data are easy, as is

judgement by comparison with similar interventions in other regions.

• Evaluations focused on a specific public such as SMEs or the long-term

unemployed. This case is very similar to the preceding one. The only

difference is that several elementary components of the programme may

affect the same public.

• Evaluations focused on a priority theme. In this case, the relevance or

impact of all the components of the programme on a particular theme,

such as employment, the environment or equal opportunities, is

evaluated. The difficulty in this type of situation is that the interventions

are multi-sectoral, as are the results and impact mechanisms. This

situation therefore involves almost as many difficulties as the overall

evaluation of a complex programme. It is considered in Volume 5 of this

Collection.

• Evaluations focused on a specific territory. In this case, the relevance or

effectiveness of all the measures of the programme on a limited part of

the area covered by the programme (e.g. problem neighbourhoods, the

most underprivileged areas) is evaluated. This situation presents exactly

the same difficulties as the overall evaluation of a complex programme.

Throughout this Volume, the term "in-depth evaluation" will refer to only the

first two sub-divisions that produce simple situations, in which traditional

evaluation tools are fully appropriate.

Tools for the overall evaluation

of a programme

Texts concerning evaluation tools usually refer to simple programmes aimed

at a well-defined public and at producing limited change within that public.

By contrast, a socio-economic development programme is typically multi-

sectoral and multi-objective. Its purpose is to create economic or social

transformations on a scale that covers an entire territory. To achieve this

goal, an integrated strategy is implemented, aimed at various publics

(businesses, employees, the jobless, communities, etc.). It has to produce a

global impact by means of the right combination of multiple specific im-

pacts on its different targets. In this context, evaluation encounters

technical problems for which the existing literature provides few solutions.

This section presents twelve tools adapted to this situation. Many of them

are innovative or are employed in an innovative way.

Examples of the evaluation of multi-sectoral programmes

The following three examples describe situations in which the programme

to be evaluated was clearly multi-sectoral. The evaluations used one or

more tools very successfully.

In the first example, the programme provided substantial funds to a region

that was lagging in its development. The programme financed basic

infrastructure as well as assistance to businesses. The objectives were,

however, numerous and not always precise. One of the questions answered

by the team responsible for intermediate evaluation was the following: "Is

the regional economic context developing in such a way that the goals will

be achieved?" The evaluation made it possible to ascertain whether

regional economic problems were improving or worsening.

Box 11 - Intermediate evaluation of a regional programme in Spain

In 1996, the Generalität de Valencia launched an intermediate evaluation of a

regional socio-economic development programme (Objective 1 Operational

Programme) co-financed by the European Structural Funds. The programme

comprised 25 measures intended to act, together or separately, on several

categories of target population. The numerous programme objectives were rarely

graded and quantified.

The evaluation started with a series of meetings during which the partners

established a list of the ten main expected impacts. This list was obtained by

means of the concept mapping of impacts technique. Each expected impact

was given a name (e.g. "diversification and efficiency of energy resources").

Each of the ten impacts was associated with one or more context indicators (e.g.

"number of regional firms that use natural gas"). The evaluation team then

gathered statistical data to establish the overall evolution of these indicators at a

regional level. The underlying evolution of each indicator was analysed by means

of the before-after comparison technique.

In this example, the concept mapping of impacts technique was used to

"clarify and grade the effects to be evaluated" (first evaluative function),

while the before-after comparison technique was used to compare data

in order to show any interesting developments (fifth function).

In the second example, the situation was that of a programme financing the

development of a French region. Here too, interventions are multi-sectoral

Principal evaluation techniques and tools

and multi-objective. The team responsible for the intermediate evaluation

wanted to show the spatial dimension of the outputs and results.

Box 12 - Intermediate evaluation of a regional programme in France

The decision to launch the Integrated Mediterranean Programmes (IMP) was taken

in the 1980s. The aim was to help those regions facing strong competition due to

the entry of Spain and Portugal into the European Community. The IMP of the

Languedoc-Roussillon region in France consisted of several sub-programmes,

oriented towards the different sectors: agriculture and fishing, industry, crafts,

services, forestry and the environment, and tourism. Over half the funds mobilised

were allocated to agriculture and fishing, since these sectors were hit hardest by

the integration of Spain and Portugal into the European Community.

An intermediate evaluation was carried out on the Languedoc-Roussillon IMP.

Since the IMPs were designed to be complementary to regional policies, the

evaluation had primarily to analyse the synergy between these two policy levels.

This evaluation focused above all on the location and spatial integration of the

different programme measures. The evaluation team gathered together all the

monitoring data as well as certain statistical indicators in the database, and noted

their location. A geographical Information system was used on this occasion for

the first time for evaluative purposes. The evaluation team presented its results in

the form of maps, some of which revealed numerous discrepancies between the

location of funds and the most threatened rural areas.

In this example, the geographical information system technique was

used to compare data, thus producing interesting conclusions (fifth

function).

The third example concerns the global evaluation of Structural Fund

assistance in the Republic of Ireland. The situation is that of a set of

programmes covering numerous sectors and providing massive funding

countrywide. The global objectives of the programme are of a macro-

economic nature. There are few of them and they are very clearly defined.

The team responsible for ex ante evaluation has to estimate likely macro-

economic impacts.

In this example, the evaluation team used a macro-economic model to

estimate effects (sixth function). It should be noted, however, that the tool

used includes a series of options concerning the variables to be modelled

or excluded. Thus, the tool implicitly served to clarify and grade the effects

to be evaluated (first function).

Examples of the evaluation

Box 13 - Ex ante evaluation of a set of programmes in the

Republic of Ireland

For the period 1989-93, all European development assistance for Ireland was co­

ordinated within a "Community Support Framework" (CSF). The European

Commission and the national government jointly commissioned an ex ante

evaluation of this highly complex set of aid. The evaluation had to estimate the

potential impacts of the CSF on the Irish economy. An important question was to

know whether a massive intervention (about 5% of the Irish GDP) would not

indirectly cause a worsening of the state of public finances because of co-

financing requirements.

The evaluation was carried out by the Economic and Social Research Institute

(ESRI) in Dublin. The evaluation team estimated the micro-economic impacts of

support by drawing on research work and the evaluations of similar support

mechanisms. These primary impacts were introduced into a macro-economic

model, making it possible to simulate the functioning of the national economy.

The ESRI had previously created this model for the European programme

HERMES. The model first had to be adapted to the specific evaluation situation,

particularly by breaking down its initial sectoral structure to more effectively take

into account the effects of European support. The model made it possible to

estimate impacts on the scale of the national economy, by taking into account

numerous macro-economic mechanisms. The risk of an excessive increase in the

public debt was confirmed.

These three examples illustrate the use of several tools in situations in

which evaluation covered an entire multi-sectoral programme. The

following section will show that the evaluation toolbox is sufficiently well

stocked for most of the functions of this type of evaluation to be tool(s)-

based.

Tools for structuring an evaluation

When a socio-economic programme is complex, the structuring of the

evaluation is a particularly important phase because it conditions the

comprehensive and comprehensible nature of the results. Moreover, if the

evaluation is being undertaken in a partnership context, this phase is

essential for clarifying the evaluation criteria suited to the different partners'

needs. It strongly determines the acceptability of the conclusions.

SWOT analysis is generally used ex ante for defining a programme

strategy. This tool is helpful in validating objectives and defining more

precisely the effects to be achieved and the most appropriate interventions.

It may also be used to evaluate a strategy that has already been established

or even implemented.

Concept mapping of impacts is used to define the effects that are to be

evaluated and, in cases where there are multiple objectives and where

these have not yet been firmly established, the associated indicators. This

tool may also be very useful if the set of objectives lack precision. It is

suited to evaluation in a partnership context because it is based on the

aggregation of individual points of view for the purpose of reaching

consensus between the partners.

The colour vote is particularly well suited to the clarification and grading of

evaluative questions in an evaluation involving several decision-makers. It

also provides a fast and robust means for choosing evaluation criteria,

validating conclusions and formulating recommendations. This tool is

useful when there are many stakeholders and when their differing points of

view have to be taken into account.

The impact matrix is a tool that provides a simplified overview of a complex

programme. It basically consists of grouping together the different

components of the programme into between ten and twenty sub-sets (the lines

of the matrix) and of clarifying the main expected impacts (the columns of the

matrix). Each cell in the matrix is given a score representing the probable im­

portance of a given impact for a given component. The tool helps to ensure

that all the essential effects of the programme have been considered before

selecting those that are to be evaluated more thoroughly. This tool is not

described in the following pages and for further information the reader is

referred to Volume 4 of the Collection devoted to innovative techniques.

Principal evaluation techniques and tools

Some classical structuring tools are not suited to the overall evaluation of a

complex programme. For example, the objectives tree and the logical

framework do not allow the multiplicity of actions and their expected

impacts to be adequately taken into account.

Tools for structuring an evaluation

SWOT Analysis Φ Φ

(Strengths-Weaknesses- O

Opportunities-Threats) Low

Overall Low

SWOT analysis was developed 50 evaluation technical cost

years ago to help firms define their complexity

strategy in a fluctuating and G Θ

restrictive competitive environment.

This decision-making tool owes its

name to the fact that it examines Use

Prospective stabilised

the strengths and weaknesses use

within the firm, as well as the

opportunities and threats of the

market. It is one of the classical tools of strategic analysis, like the BCG (Sos-

fon Consulting Group) matrix that is similar in many ways.

In the public sector, cities were the first to use SWOT analysis in the 1980s

as a framework for reflection on different development scenarios. The use

of this tool has recently spread to territorial diagnoses and an ex ante

evaluation of regional programmes.

Example: Prior appraisal of the Swedish Objective 6 plan

SWOT analysis was used in drawing up the Objective 6 regional development plan in

Sweden. The results produced through this method highlighted the main

environmental, structural and demographic characteristics of the context of the

chosen region. The identification of strategies and priorities, as well as the

formulation of the action plan, were subsequently made on the basis of major factors

identified through SWOT analysis. These can be summarised as follows:

• Strengths: Large nature reserves, considerable natural resources.

• Weaknesses: Isolated region with a low population density; vulnerability to economic

fluctuations and to fluctuations of public budgets; structural weaknesses of firms.

• Opportunities: Tourism, quality of forest resources, fresh-water piscicultural

production; mineral prospecting.

• Constraints: Population loss; lack of jobs.

The recommendations from this analysis based on regional socio-economic data

were judged by the managers as being relevant for their strategic planning. Their

usefulness for justifying the choices made in the framework of the Regional

Development Plan seemed to be recognised. Since this first attempt at the concrete

use of SWOT analysis in the framework of programmes financed by the European

Commission, this tool has been used in many other regions.

Source: Prior appraisal of the regional development plan for the Objective 6 region of

Sweden, Nordland Research Institute, Bodo, June 95 Principal evaluation techniques and tools

What is the purpose of the tool?

The purpose of SWOT analysis is to incorporate into the reflections on a

programme, both the intrinsic characteristics of the territory concerned and

the determining factors in the environment in which it takes place. The tool

is intended to reduce the areas of uncertainty related to the implementation

of a project or programme applicable to the relevant territory. It thus

enables the definition of a more relevant strategy in the context in which the

action is to take place. The aim of the tool is therefore:

• to highlight the dominant and determining factors, both within and

outside of the territory, likely to influence the success of the project;

• to produce relevant strategic guidelines by linking the project to its

environment.

SWOT analysis may be expanded by means of tools similar to those known

as "portfolio management", such as the BCG matrix, to examine the validity

of a strategy that has been proposed or is being applied, and to recommend

changes where relevant. The classification of the different possibilities takes

into account their feasibility (assets available "in house" = strengths and

weaknesses), as well as their potential (attractive features in relation to the

outside environment = opportunities and threats).

The usefulness of such tools in evaluative processes lies in their capacity

for drawing a systematic ¡mage of the relationship between the evaluated

programme and its direct environment.

• In which cases should it be used?

Developed about twenty years ago by specialists of private management,

strategic decision-making tools are, once again, being used increasingly in

public policy strategic reflection. Originally constructed in terms of

products, customers, markets and competitive advantages, their use has

now spread to public collectivities, in so far as the policies of these

collectivities are aimed at creating competitive advantages on the scale of

the territory for which they are responsible. The notions of strengths,

weaknesses, opportunities and threats may apply to a regional economy in

the framework of European competition or to a firm competing in the

market.

SWOT analysis serves to identify the most relevant strategic guidelines in

relation to a global objective of economic development. Use of the tool is

Tools for structuring an evaluation

therefore more particularly recommended in the planning of a programme

and during its ex ante evaluation, helping to improve the integration of the

programme into its context.

How should it be implemented?

The implementation of a strategic approach such as SWOT analysis

involves six distinct steps:

Step 1. "Scan" of the environment of the programme: This operation

enables the detection of the major trends and problems likely to affect the

future of the territory under consideration, by means of socio-demographic,

economic, political and physical indicators. Indicators of regional disparities

are particularly useful for revealing opportunities and threats. This phase

should not be exhaustive, for the aim is limited to obtaining an overall

picture to illustrate the key issues the community in question will have to

face.

Step 2. Inventory of possible actions: Possible actions are formulated in

general terms, and constructed in relation to the main problems identified.

Step 3. External analysis of opportunities and threats: This consists of a

list of the parameters of the environment which are not under the direct

control of the public authorities and which, it is assumed, will strongly

influence socio-economic development.

Step 4. Internal analysis of strengths and weaknesses: This step

completes the external analysis by making an inventory of the factors which

are at least partly under the control of the public authority, and which may

either promote or hinder development.

Step 5. Classification of possible actions: This is aimed at highlighting

those actions (strategic guidelines) most likely to reduce development

problems by focusing on the strengths and reducing or even eliminating the

weaknesses, with a view to maximising opportunities and minimising

threats.

Step 6. Evaluation of a strategy: This optional stage may be included if it

is appropriate forjudging the relevance of a strategy already implemented or

being planned. It is designed on the basis of a "portfolio of activities"

analysis. Like a firm with its products and markets, a socio-economic

programme contains a set of interventions some of which combine

strengths and opportunities while others try to compensate for weaknesses

Principal evaluation techniques and tools

or to warn of threats. The evaluation team places interventions on a plane

with two axes: (1) internal feasibility, strengths and weaknesses, and (2)

external environment, opportunities and threats. The discussion of the map

thus produced can be used to judge the relevance of the evaluated strategy.

Application to the Structural Funds

Strategic analysis is suited to the clarification of issues relating to the socio­

economic development of those regions covered by the Structural Funds.

It is recommended for the planning of regional development strategies that

have to be centred on strengths and weaknesses, while trying to overcome

weaknesses and threats. A tool such as SWOT analysis proposes a starting

point for defining the strategic guidelines to be applied in a specific

environment.

SWOT analysis may therefore serve as a management tool for assessing

the relevance of a strategy in the planning or implementation stage. In the

framework of ongoing evaluation, this tool may be used to check whether

a strategy remains relevant to the needs of the regional economy.

Strengths and limits of use

• SWOT analysis applied to public action is oriented towards the search for

an effective strategy.

• The matrix tool is derived from management theories that are already old

and are often criticised for being simplistic. This limitation should be kept

in mind in its transposition to public management, so as to avoid

restricting the analysis to a simplistic framework.

• SWOT analysis, as an evaluation tool, serves to make a simple

classification of activities in terms of their relevance. Its main weakness

derives from the often-subjective procedure used by the evaluation team

to classify the activities. The association of partners in this classification

is a way of enhancing the credibility and usefulness of the analysis.

• SWOT analysis requires a deliberate intention, on the part of the different

actors participating in its application, to reach a stable consensus. The

process of formulating strategic guidelines is only of value under this

condition. Otherwise, this model may tend to produce an erroneous

and/or inapplicable diagnosis.

Tools for structuring an evaluation

Further reading

Bryson Ü.M. and Roaring W.D. (1987) "Applying Private Sector Strategic

Planning in the Public Sector", Journal of the American Planning

Association, n°53, pp. 9-22.

Examination of the application of corporate strategic planning

methods to contingencies in the public sector.

Quinn, J.B. & al (1988) The Strategy Process, New Jersey: Prentice hall.

Critical study of strategic planning.

Tools for structuring an evaluation

Concept mapping 0 Φ

of impacts O

The concept mapping of impacts is Medium Medium

Overall

an adaptation of the technique technical

evaluation cost

known as "concept mapping", complexity

used over the past ten years for O O ©

programme evaluation. It was first

used to help to define strategies or

the reorganisation of firms, before Retrospective Prospective Innovative

use use use

being applied to public policy

making, in particular, as regards

social programmes.

Data processing software is used to obtain a graphic ¡mage of the actors'

representations of a socio­economic reality, in general, and of expected

impacts, in particular. The MEANS programme developed an application

specially designed for the clarification of the impacts of a regional

development programme and to find associated indicators within a

partnership framework (see Volume 4 in this Collection, devoted to

innovative techniques).

Example: C larification of expected impacts of the ERDF 1994­99 Operational

Programme of the Generalität de Valencia.

This example corresponds with that of Box 11.

The concept mapping of impacts used in the intermediate evaluation of the ERDF

1994­99 O.P. of the Generalität de Valencia had two objectives:

• clarification of specific and global impacts of the programme, in order to determine

criteria for judging the effectiveness of interventions for intermediate evaluation;

• proposal of intermediate and global impact indicators in order to complete the

battery of programme indicators.

The evaluation took place in two phases focusing first on intermediate impacts and

then on global impacts, with the assistance of the managers of programme measures

and of the overall programme managers, respectively.

Officials of the Generalität in charge of the implementation of measures within the

programme were divided into three groups corresponding to the fields of intervention

(direct support to businesses, indirect support to businesses, infrastructure). At the

end of a series of three meetings, each of the groups determined specific evaluation

criteria for actions in the field of intervention concerned. The same groups then

determined the intermediate impact indicators most likely to provide the information

Principal evaluation techniques and tools

required to make a judgement in terms of the specific evaluation criteria selected. In

a second phase and by means of the same process, the overall programme

managers (representatives of the European Commission, of the central Spanish

administration and of the regional authorities) constructed global evaluation criteria

for the programme. They drew upon the results of the groups' work. The process

ended with the selection of the corresponding global impact indicators.

Source: Programme MEANS 1996 - Instituto Valenciano de Investigationes

Económicas, "Projet pilote de cartographie conceptuelle des impacts"

What is the purpose of the tool?

The concept mapping of impacts is used to assist in the clarification of

impacts expected from a public intervention. The clarification/grading is

obtained by taking into account the representations of all the stakeholders.

The group's work is facilitated by the systematic nature of the approach

that avoids fruitless digressions, and by the visualisation of the similarity or

difference between impacts. It is thus possible to conduct collective

reflections in large heterogeneous groups (up to 200 persons) and to

structure and grade the main impacts expected from an intervention. The

tool makes it possible, in an ex ante context, to determine the main objec­

tives and priorities. In an ex post context, the impact mapping technique is

used to draw up the evaluation reference system (effectiveness criteria and

impact indicators).

In which cases can it be used?

This tool is valuable for structuring the content of a complex programme. It

was used, for example, in a neighbourhood in Northern Ireland to define a

social programme in partnership with the Protestant and Catholic

communities concerned. It is useful in evaluation when the evaluability of a

programme has to be enhanced, that is to say, when it is necessary to make

the objectives of a programme more explicit.

This technique is particularly relevant when the evaluability of a programme

has to be enhanced in the partnership framework involving several decision

makers/funding bodies. It is thus possible to produce a common evaluation

reference system, despite differences in objectives that may exist between

these partners.

Two characteristics frequently found in programmes can thus motivate the

use of impact mapping: the lack of clarity and precision of the programme

objectives and of its expected impacts, and the lack of impact indicators

Tools for structuring an evaluation

planned in the programme. Since its implementation is relatively

cumbersome, it is advisable to limit the use of impact mapping to complex

programmes with many expected impacts.

How should it be implemented?

Step 1. Constitution of a group of stakeholders in the programme. This

step is essential because it will determine the framework in which the

subsequent steps in the process are to take place. The group must be

balanced (in terms of the different actors in the programme), knowledgeable

(in-depth knowledge of the intervention by the members of the group) and

stable (full participation continues throughout the process as a whole).

Step 2. Inventory of expected programme impacts in the form of

descriptors. This step consists of producing an exhaustive list of all the

impacts expected from the programme, in the form of short qualitative

statements called "impact descriptors" (e.g. "actors in tourism became

more professional").

This list must include any unfavourable impacts of the programme which

were not expected but could have been foreseen.

This list draws upon two sources:

• identification by the evaluators of all the impact descriptors featured in

the programming documents;

• impact descriptors formulated by members of the group in relation to

their knowledge of interventions. These concern the impacts of the

measures or actions of programmes, which each member of the group

expects to observe in the short, medium or long term. Formulation of

these descriptors may take place during interviews or in the context of a

first group meeting.

A group meeting is essential for validating the descriptors and determining

the final list together.

Step 3. Weighting of descriptors and constitution of piles. In the context

of a second group meeting, each member is given a recapitulative list of the

descriptors, as well as a set of cards (with a descriptor marked on each

one). Each member of the group is asked, individually:

• to weight the different descriptors, that is to say, to give a score to the

descriptors in terms of their strategic importance for the programme. The

more important the role of the described impact in the success of the

programme, the higher the score. Principal evaluation techniques and tools

• to constitute piles of descriptors in terms of their conceptual proximity;

each member of the group will thus group together conceptually similar

descriptors, that is to say, those in the same "family" of impacts.

Step 4. Calculation of the impact map. The data produced by each

member of the group (weightings and piles) will be analysed statistically by

an appropriate software package.

At the end of a non-metric positioning, multidimensional analysis the

descriptors are represented in the form of a cloud of points in a two-

dimensional space. In this cloud of points, the descriptors that were often

placed in the same pile by the participants will be positioned close together.

Those descriptors which were never put in the same pile by the participants

will be furthest apart.

At the end of a second analysis consisting of a hierarchical classification in

clusters, the software will identify groups of descriptors. Those descriptors

that were most often grouped together by the different members of the

group will be put into the same cluster. Groups of descriptors that are

similar from a conceptual point of view are thus constituted.

Tools for structuring an evaluation

Finally, the average weighting of descriptors included in each of the groups is

calculated. The final result is a map showing weighted groups of descriptors

(in the diagram in Box 14 the weight is represented by a thicker line).

Step 5. Definition of effectiveness criteria. The impact map is examined

in a third meeting when the composition of the different groups of

descriptors is discussed. Some descriptors can be moved to make its

composition more coherent. Thus, the map calculated by the software is

not a final result but rather a basis for discussion.

Once the composition of the families has been validated, they are named.

The aim is for the members of the group to agree on a name for each family,

which is representative of all the descriptors in the family. This name

corresponds to an evaluation criterion. Thus, there will be as many

evaluation criteria as families of impact descriptors. Since the families are

weighted, the criteria obtained are also weighted. This allows these criteria

to be used directly for a multicriteria judgement.

Group reflection may be continued with the aim of specifying a time-scale

for the impact families (impacts which are expected in the short, medium or

long term), and the presumed causal links between these impact families

(e.g. a particular impact family constitutes an intermediate impact which will

help to achieve a more global objective).

This process may finally be taken further with the aim of proposing impact

indicators directly related to the different evaluation criteria. The descriptors

included in each of the families are helpful in defining these indicators. Thus,

some of these descriptors, and particularly those to which the most substantial

weightings were given, may directly inspire the definition of an indicator.

Discussion by the group on each of the proposed indicators will relate both

to the relevance of the indicator in terms of its relation to the evaluation

criterion, and the availability of sources of information.

Application to the Structural Funds

The concept mapping of impacts has proved to be particularly well adapted to

the context of structural programmes. The aims of this type of programme are

defined by the European Union and adapted to suit to the characteristics of

each Member State or beneficiary region. Since the Single Programming Docu­

ments are the product of negotiation, the programme objectives are frequently

not specific enough for constructing a reliable evaluation. Impact mapping is

Principal evaluation techniques and tools

consequently useful in the first stage of clarification. It helps to clarify evaluation

questions, grade effects and define evaluation criteria in a partnership context,

which facilitates the appropriation of the evaluation results.

The impact map is useful in the preparation of intermediate programme

evaluation. It makes it possible to systematically establish impact families

cutting across several measures, and thus to construct relevant criteria for

the evaluation of the programme as a whole. The procedure can be divided

into two phases, as in the case of Valencia, which are summarised in the

following diagram.

Box 15- Partne rship proce ss of impact mapping

Operators Operators Operators

Group A Group Β Group C

'

PHASE 1 ■ ' 1

Map of Map of Map of

specific specific specific

impacts A impacts Β impacts C

'

Group of

managers

PHASE 2 1

Map of global

impacts

Strengths and limits in practice

• Using a computer programme to mediate is a valuable support for

reaching agreement, despite the heterogeneity of initial points of view.

Tools for structuring an evaluation

• The group dynamics generated by the technique has proved to be useful

and positive for increasing mutual knowledge of different management

actions, and may thus favour adjustments to the implementation of pro­

grammes. By mobilising the stakeholders in the preparatory phase of

evaluation work, it facilitates the subsequent stages of the evaluation and

makes the results far more readily acceptable.

• Impact mapping is a technique that is used to enhance the evaluability of

programmes. It makes it possible to produce a common evaluation

reference system, taking into account the points of view of the different

partners. The successive group discussions, which are part of the

implementation of this tool, guarantee the legitimacy of the reference

system obtained.

• The results may be produced in a few days. The technique is thus

compatible with the evaluation cycle.

• The resources required (even if they are limited) and the mobilisation of

numerous stakeholders during successive meetings can be justified only

when the evaluation stakes warrant such an investment.

• The constitution of the group of partners can be difficult and care has to

be taken to ensure that there is a sound balance between the

stakeholders. Considerable efforts may be required in cases of conflict.

• Although the technique of drawing up impact maps limits risks related to

the leadership of a few members of the group, this risk remains during the

phases of production of descriptors and exploitation of the impact map.

• This tool can be used for all the functions of the structuring stage.

Further reading

Knox, C. (1995) 'Concept Mapping in Policy Evaluation: A Research Review

of Community Relations in Northern Ireland', in: Evaluation, Vol. 1,

Number 1.

An interesting example of use of the tool in a European context.

Trochim, W.M.K. (1989) An introduction to concept mapping for planning

and evaluation', in: Evaluation and Program Planning, 12 (1): 1-16.

Presentation of the tool and its advantages for evaluation.

In the MEANS Collection: Volume 4 devoted to innovative techniques.

Tools for structuring an evaluation

Colour vote Θ φ

O

In the field of evaluation, group

work is often essential, particularly Overall Low Low

technical cost

evaluation

when several partners are involved complexity

in commissioning and steering an O o

A

evaluation. The different voting

techniques can be used to In­depth Retrospective Prospective

structure collective reflection, when evaluation use use

the aim is to improve effectiveness, ©

stimulate creativeness or define a

common standpoint. While there is Innovative

a large number of these use

techniques, this information sheet

focuses more particularly on the "Abaque de Régnier", or colour vote.

Example: Evaluation of an economic development programme in Switzerland

As part of a study of the measurement of the effects of public actions in Switzerland,

the CEAT (research community for national development) evaluated a canton policy

for economic development. Two work groups were formed with the actors involved

in the implementation of the policy. The first group was to focus on the "objectives

and strategies" of the policy and the second group on its "implementation and co­

ordination".

The evaluation team used the Abaque de Régnier as a technique for conducting

group sessions. The groups based their work on assertions formulated by the

evaluation team. These were often provocative, the intention being to encourage the

participants to take a stand. By asking the participants to vote on these assertions,

the technique created a starting point from which to express their positions and to

become aware of the other opinions present in the group. The debate helped to

define a common position on the objectives of the policy of economic development.

Source: Alberton S., Rey M. et L. Vodoz (1993) La politique valaisanne de promotion

économique: une évaluation privilégiant le point de vue des acteurs. B erne:

Programme National de Recherche 27.

W h a t is the purpose of t h e tool?

Voting techniques facilitate the expression of opinions on a given subject by

all the participants. It is the way in which these opinions are obtained that

varies. The colour vote is based on the idea of making participants aware

Principal evaluation techniques and tools

of all opinions in the group, by means of a coloured visual presentation of

the results of individual votes. In this way the strong personality of some

does not conceal the opinions of other, more reserved, participants. With

the colour vote, issues on which there is consensus are soon distinguished

from conflictual issues. In this way time is saved by allowing the group's

reflections to be concentrated on most problematical issues.

Voting techniques also help to avoid certain drawbacks of group work such

as pressure to conform, convergence of opinions around core values,

predominance of certain actors, etc.

In which cases should it be used?

A work group may use these techniques for any type of evaluation: ex ante

to define challenges or needs to be met, ongoing to clarify objectives as in

the example above and, where relevant, ex post to validate evaluation

conclusions.

Very often the objectives and judgement criteria of a programme cannot be

legitimately clarified by a single person. In this case the voting techniques

facilitate the work of a group composed of people with very different skills

and levels of responsibility. In other words, meetings for managing the

performance of an evaluation are an ideal opportunity to apply voting

techniques. From this point of view, the colour vote is particularly useful for

grading programme objectives, defining evaluation criteria and selecting

the most important criteria.

How should it be implemented?

Step 1. Formulation of statements. The statements are assertions

expressed concisely, generally in a single sentence. They are formulated

beforehand by the evaluation team, on the basis of personal interviews and

of an initial group discussion, during breaks in the meeting.

In the above example the items were formulated as follows:

• "For industrial enterprises, financial support is derisory"

• "For industry, economic promotion should be aimed at 'high-tech' firms"

• "Economic promotion should benefit all areas of the canton equally"

• etc.

At this stage, it is useful to collectively check the understanding of each

statement.

Tools for structuring an evaluation

Step 2. Individual vote: the participants give a colour to each statement.

The signification of the colours is simple: Black

Light Dark White

Pink Yellow

Red green green Does

No

No clear- General Total

Total General not want

opinion

cut opinion agreement agreement

disagree­ disagree­ to

ment ment on the

on the participate

subject subject ¡n the vote

Step 3. Transcription of votes on a magnetic board or on a computer

screen. The result is thus immediately presented to the participants in the

form of a table filled with coloured disks. Tendencies are clearly visible and

this enables each person to know whether consensus exists or not in the

group.

Step 4. Group discussion. The group leader selects statements for

discussion. Most often, he or she will choose those on which there was

disagreement.

Step 5. Reformulation of statements with the group. This final step is

used to define a collective position on the subject.

Application to the Structural Funds

The Structural Funds are managed within a partnership framework, which

means that numerous working meetings are organised. Apart from those of

an administrative nature, all meetings with a "creative" aim can be based on

voting techniques to structure and motivate the work (meetings devoted to

the selection of projects, the evaluation, reorganisation or renewal of

programmes, etc.).

The colour vote is particularly well suited to situations in which a common

standpoint or consensus are required. It is also used to grade proposals

made in a group, which can make it easier to reach conclusions in a meeting.

Strengths and limits in practice

• Groups appreciate the colour vote when they want to reach consensus in

a relatively conflictual domain. The transparency and simplicity of the

technique ensure a high level of confidence in the process.

Principal evaluation techniques and tools

• The results of this vote are immediately visible and, thanks to the

technique, tendencies are easily perceived. The possibility of voting "whi­

te" or "black" helps to highlight a lack of information or points of

disagreement on the subject under discussion.

• The facilitator has a high level of responsibility; he or she therefore

requires extensive experience.

• The fact that the tool is not of a highly technical nature enhances the

participants' confidence (NB: the most recent version of the Abaque de

Régnier is computerised.) The game-like nature of the method also has

the advantage of relaxing the atmosphere during meetings. There is,

however, a risk of rejection if the participants find the technique too

scholastic.

• Potential predominance of one actor in the debates is limited by the non­

verbal nature of the tool.

Further reading

Anzieu D. et Martin J.-Y. (1994) La dynamique des groupes restreints. Paris:

PUF. 395 p.

Reference manual for the functioning of small groups. Annex

containing various methods of "discussion-meetings".

Mucchielli R. (1992) La conduite des réunions. Paris: Librairies techniques.

120 p.

Popularised presentation containing practical exercises.

Viveret P. (1992) 'Les outils d'intelligence collective et l'évaluation', in:

L'évaluation en développement. Paris: La Documentation Française,

pp. 179-183.

Summarising group techniques.

Tools for observing changes in the field

When a programme is complex and has to be evaluated in its entirety, the

field of observation to be covered is very broad. Exhaustive observation is

therefore extremely costly. Tools do, however, exist for collecting the data

needed for evaluation.

Observation in the field can be carried out by organising a series of

individual interviews with operators (indirect observation) or with the

addressees of the programme (direct observation). This technique is

particularly appropriate when the information to be collected is factual.

Focus groups , with operators or addressees as participants, are used to

collect both opinions and data. A well-conducted focus group allows

reliable, high quality data to be collected.

Systematic field observation can be carried out by means of carefully

selected case studies. In the evaluation of a complex programme, the case

study seems to be the only appropriate tool for observing effects on direct

or indirect addressees, while respecting the complexity of the different

programme components.

Some classical observation techniques are difficult to apply in the overall

evaluation of complex programmes. For example, the questionnaire

survey of direct addressees of the programme is too expensive if, as is

often the case, there are up to ten different categories of addressees.

Tools for observing changes in the field

Individual interview £3 <D

O

The interview technique is used to

gather qualitative information and Overall Medium Medium

the opinions of those persons evaluation technical cost

complexity

affected by a particular programme O G

A

or project, its context, implemen­

tation, results and impact. Several

forms of interview can be In-depth Retrospective Prospective

evaluation use use

distinguished, each of which fulfils Θ

a different purpose: the informal

conversation interview; the semi-

structured, guide-based interview; Use

stabilised

and the structured interview (the

most rigid approach).

The following presentation is based on a semi-structured interview, the

technique that is used most frequently in the evaluation of Structural Funds.

Example: Evaluation of the North Jutland Technology (NordTek) programme

The goal of the NordTek programme was the development of North Jutland through

the introduction of information technologies into the regional industrial fabric. The

programme had two aims: job creation and the improvement of a negative image as

a peripheral region. To this end, the development strategy implemented by this

programme was based on four objectives that were to be achieved through three

types of action.

The objectives were: reinforcing local networks of actors; improving the qualification

of workers in new technologies; orienting regional production towards local demand;

and, lastly, developing attractiveness for foreign investments. The actions

implemented for that purpose were measures favouring the knowledge and utilisation

of new technologies, measures for training people in the use of new technologies,

and measures providing investment support for SMEs.

The programme was evaluated by means of semi-structured interviews among 70

beneficiary firms. The goal of these surveys was to understand how firms used the

services received and what they got out of them. Those firms in which interviews

were held were selected from 250 beneficiary firms in five of the most important

projects in the NordTek programme (the different projects comprising the programme

concerned specific subjects such as consultancy or training and networking, for

example). The firms selected had to have received a relatively large amount of

assistance during at least six months, and to be representative of the different forms

of assistance that had been granted. Principal evaluation techniques and tools

The approach chosen was a method using an interview guide, so as to gather all the

information concerning the firms. The interview guide also made the approach more

rigorous, thus facilitating the comparison of the interview results obtained by several

different interviewers. After the interview, the survey continued with a more informal

conversation during a visit to the firm. It lasted a total of about two hours. The

interviewee then received the summary of the interview in the form of a 4-8-page

document, for validation.

The themes addressed during the interview were, for example: the reasons for joining

the project, the expected benefits from the assistance and the benefits actually

received, and the possibilities of obtaining assistance from a different source.

The results of the interviews made it possible to identify the reasons behind the

firms' decision to participate in the NordTek projects. On the whole, the interviews

revealed that the assistance provided by NordTek was unique in certain cases and

could not be found elsewhere, whereas for certain aspects it competed with local

companies and even other NordTek projects. The NordTek networking and

consultancy services were developed over a period of several months, and even

several years; personal knowledge and geographical proximity were then seen as

important factors for the firms. It nevertheless remained difficult to quantify the

benefits that the NordTek projects were able to provide to firms, due to the

numerous external factors capable of influencing the process in which the services

were provided.

Source: Olsen, Leif and Olaf Rieper. NordTek-evaluieringen, AKF Forlaget, 1991

W h a t is t h e p u r p o s e of t h e tool?

This type of survey is a way of learning about and examining the views of

the actors (addressees and other stakeholders) on a project or programme,

e.g. how far the project or programme meets needs, or its results as

c o m p a r e d to expectations. Interviews are also used in social science as a

tool for investigating users' reasoning.

The individual interview is an exploratory technique that serves to obtain

relevant information on the reasoning, conceptions and representations of

the persons questioned on a project or programme. Apart from subjective

representations, it also serves to gather information on individual practices.

It is particularly valuable in exploring the ways in which an intervention has

been implemented and for identifying g o o d practice. It is therefore often

used as part of a formative evaluation that is designed to test a theory of

action and/or to provide guidance about fine-tuning a policy and program­

me. It may also be helpful in providing a summative evaluation of a

Tools for observing changes in the field

programme which is intended to achieve changes in actors' behaviour or

perceptions (such as technology transfer or training programmes) as

opposed to the more tangible, 'harder' outcomes (such as creating new

jobs or constructing new facilities).

In which cases can it be used?

The interview is used in an exploratory context, in other words, when one

does not have a priori hypotheses or adequate knowledge on a project or

a target public to make a questionnaire survey possible.

It is also a relevant technique when the public concerned by the evaluated

programme or project is too small to be the subject of a statistically

representative survey.

Semi-structured, guide-based interviews are used when relatively

standardised information is required about the programme being studied. A

certain amount of prior knowledge about the subject is needed when using

this technique.

The individual interview technique may be used at any stage of the

evaluation process: to establish a theory on the evaluated programme or

identify the problems and needs of a region and the improvements needed

during the implementation of the programme. It may be particularly

powerful when used to gain the views and understandings of key actors

such as senior policy makers or programme managers.

• How should it be implemented?

In order to conduct an interview well, the following steps have to be

adhered to:

Step 1. Selection of interviewees: The samples needed for carrying out

interviews are smaller than for questionnaire surveys. The information

obtained is validated by the context and not by the probability of occurrence

peculiar to questionnaires. The number of interviews depends on the subject

of the study, on the variety of reactions to the subject, and on the resources

available (in general 20 to 60 interviews). The selected sample is most often

based on the selection of components said to be characteristic of the

population (diversified sample). The sample may be selected either by direct

access (e.g. via administrative lists) or through the intervention of a third party,

which allows for more targeted selection but presents risks of distortion.

Principal evaluation techniques and tools

Step 2. Planning the interview: Planning the interview includes the

drawing up of an interview guide. This consists of specifying the topics that

the interviewer wants to address. It is not essential to follow the interview

guide in any precise order. This guide is more a sort of checklist enabling

the interviewer to check that he or she is dealing with the essential

questions. The interviewer may modulate her/his intervention in relation to

the interviewee, and formulate new questions. The first interviews may give

rise to adjustments or amendments in the definition of the questions if the

interviewees have a problem with them. In many cases it is also helpful to

gather basic information from interviewees in advance to save time during

the interview itself. It may also be useful to provide the interviewee with a

guide to the issues that are to be covered so that he or she can gather

together any necessary factual information prior to the meeting. Certainly it

is usually worth establishing a 'contract' with the interviewee explaining the

purpose of the interview, how long it will last, the level of confidentiality, the

use made of findings etc.

Step 3. Selection and training of interviewers: If they are to be conducted

properly, interviews require a high degree of professionalism in the

interviewer. He or she must have skills in communicating, listening and note

taking. To facilitate the smooth running of the interview and ensure that the

interviewee feels at ease, it may be useful to ensure that there is social

proximity between interviewer and interviewee. The least structured inter­

views also require the interviewer to have substantial knowledge of the field.

Step 4. Course of the interview: An interviewer must have a "respectful"

attitude vis-à-vis the interviewee and the information gathered, but also be

able to convey a good understanding of the subject matter and context. The

initial contact is very important, for it is the basis of communication. The

interviewer must be careful not to influence the interviewee by approving or

orienting her/his answers. The interview may be recorded, to ensure that the

interviewee's assertions are not distorted and that the most important

remarks are not omitted from the report. However, in some cases interviewees

may feel less able to give full and frank answers on tape, and the costs of

transcribing interviews should be taken in to account in designing the

evaluation budget and timetable. The interview transcript or summary should

be checked with the interviewees and it may be useful at this stage to follow

up any unresolved questions. It may also be useful to thank interviewees by

letter for their contributions (their co-operation may be needed in future

studies and it is therefore worthwhile ensuring their goodwill).

Tools for observing changes in the field

Step 5. Analysis of results: This final phase consists of analysing the

conversations, interpreting and comparing the information given by the

interviewees, and finding common and divergent viewpoints so as to draw

up a review of the evaluation. A summary focused on the coherence of each

interview, or a transversal thematic analysis more suited to the search for

models capable of explaining individual practices, may be drawn up.

In order to produce results and then discuss them, the interviewer

establishes an analysis grid based on the reading of the interviews and of

descriptive hypotheses. This grid is an explanatory tool applied to each

interview report.

Application to the Structural Funds

Interview techniques are used extensively in the evaluation of structural

interventions and, in particular, for the evaluation of programmes.

Interviews with programme managers and addressees remain one of the

most commonly used tools in the intermediate evaluation of structural

programmes. The principle consists of selecting several addressees or

managers, depending on the characteristics of the action implemented and

the public concerned.

Used as a tool for analysis, the interview is a technique which makes it

possible to make a brief overview of the programmes. Very loosely

structured interviews - informal conversations with the managers and other

persons concerned by the programme - may be used to identify those parts

of the programme which need to be considered in more depth. Interviews

with policy makers may also form an important part of the early stage of

focusing an evaluation and determining the key outputs which policy

makers require.

The interviews may also prove relevant and provide useful information in the

framework of programmes of a social nature where the addressees often

lack motivation for filling in the questionnaires, such as measures aimed at

giving jobs to the long-term unemployed who have been on a training

course.

Strengths and limits in practice

• The advantage of this type of survey is that it provides in-depth

information on the values, facts and behaviour of the interviewees; it

Principal evaluation techniques and tools

makes it possible to link up a group of elements, thus producing a

relatively exhaustive study on a given subject.

• An individual interview takes into account situational and individual

factors. It is therefore difficult to draw general conclusions. Individual

interviews may allow for an exhaustive identification of effects and

possible causes, but cannot be used to measure impacts or grade

causes.

• A well conducted interview may provide insight into the mechanisms of

implementation and the causal links peculiar to a programme, and help

to identify success stories or obvious shortcomings. In that case, the

technique produces information that can easily be communicated in the

form of boxed examples in a text.

• If it is to be effective, this practice requires a lot of time and the

contribution of professionals. Specific skills are needed to plan, conduct

and interpret an interview; inadequate skills of this nature will produce

information of no value. Interviews are not always conducted with the

professionalism required to produce effective results.

• The benefit of these surveys depends on the knowledge of the

interviewees and on their co-operation in answering the questions.

Information may also be distorted due to the choice of interviewees.

Further reading

Blanchet Α., Gotman Α. (1992) L'enquête et ses méthodes: l'entretien.

Nathan Université

This book is entirely devoted to interview surveys, from an educative

point of view.

Casley D., Kumar K. (1991) Collecte, analyse et emploi des données de

suivi et d'évaluation. Banque Mondiale

Clear and accessible presentation of interview techniques - Ch. 2

and 3, p p . 1 2 - 4 7

Patton M.Q. (1987) How to use qualitative methods in Evaluation, Sage

Publications

This book devoted to all the qualitative techniques gives a glimpse

of their application in the field of evaluation.

Tools for observing changes in the field e m

Focus Group O

A focus group involves several

people with the same characte­ High Medium

Overall

ristics, and provides qualitative evaluation technical cost

complexity

information during a targeted O o

A

discussion.

The focus group technique was In-depth Retrospective Prospective

devised by marketing specialists to evaluation use use

analyse the potential impact of ©

advertising strategies and

advertisements. The method solicits Innovative

the participants' creativity in use

carrying out simulations. It is

particularly valuable for analysing themes or fields which give rise to divergent

opinions or which involve complex issues that need to be explored in depth.

Example: Thematic evaluation of the Objective 3 SPD, 1994-99, UK

In the framework of an evaluation of the Objective 3 SPD in the UK, a focus group

technique was used to complement other techniques. The general objectives of this

evaluation were to check: a) whether the actions implemented within the SPD had

effectively reached the targeted addressees and b) whether the structure of the SPD

favoured the emergence of effective projects in terms of the integration of

addressees into the job market.

The method devised for this evaluation had a quantitative section based on the

exploitation of monitoring data, and a qualitative section based, in particular, on four

focus groups formed in four regions representative of the diversity of situations in the

UK: Scotland, the "West Midlands", London and the Southwest.

These focus groups consisted of representatives of project promoters (notably

training centres, regional councils, higher education institutions, non-governmental

organisations, and Training and Enterprise Councils). The subjects and questions

addressed were structured as follows:

• examination of the sources of information available for planning projects

corresponding to the needs of targeted publics as well as for choosing the means

to reach these publics;

• highlighting the strengths and weaknesses of the modes of selection, approval and

management - especially financial - of projects, and their effectiveness in terms of

targeting;

• discussion on the means to measure and assess the results of these actions in

relation to the specificity of the publics targeted and employers' requests which

have been debated; Principal evaluation techniques and tools

• finally, debate on the effectiveness of the SPD as an instrument for promoting

projects suited to the addressees' needs.

These discussions served to reveal new ideas and considerations, often covering

themes closely related to the questions asked, such as the existence and advantages

of "integrated projects", the modes of functioning in networks of project promoters,

the "additionality" of projects, etc. The results drawn from these focus groups,

although not representative in the statistical sense of the term, served to highlight

"ideas to work on" in various fields such as: access to problem publics, conditions

facilitating the emergence of relevant projects in terms of targeting, and so on.

Source: Targeting and the ESF Objective 3 1994-1996 Single Programming

Document (SPD), final report to the Department for Education and Employment

(DfEE) ECOTEC, June 1996

What is the purpose of the tool?

Structured discussions in focus groups are starting to be used extensively

today, especially in the social sciences and particularly for evaluation. By

playing on the interaction and confrontation of points of view, this techni­

que serves to reveal the participants' perceptions on a given theme,

whether it concerns the relevance of the evaluated programme, its

implementation, results or impacts.

The focus group technique is a particular application of techniques for

chairing meetings. The tool is standardised, with a view to creating optimum

interaction for the collection of information. Other group techniques are used

in the social sciences, such as the Delphi technique or the nominal group

(recording of all individual points of view which are then submitted to the

group and discussed), that do not feature truly interactive discussion.

The focus group makes it possible to bring together, simultaneously or

sequentially, several stakeholders in a programme (managers, addressees

and/or operators) and to collect a large amount of qualitative information in

a relatively short space of time, by means of the "shared intelligence"

technique. The comparison of the experiences and representations of the

participants in the group make it easier to understand the phenomenon

under observation.

By involving the actors or addressees of a programme in the evaluation

work, the conclusions of the study will be more credible and more easily

accepted. The focus group technique may also be used for the validation

of data collection, or for completing quantitative data.

Tools for observing changes in the field

In which cases can it be used?

Focus groups are adapted to those cases where the topics to be addressed

provoke divergent opinions but where discussion remains possible. The

subjects dealt with will be in the "public interest", so as to be debatable in

a group.

How should it be implemented?

For the same reasons as those applying to individual interviews, several

steps are required to prepare and conduct focus groups properly:

Step 1. Selection of participants: The choice of participants in the group

depends on the characteristics of the population and the subject addressed.

It is preferable to form several types of groups (between 3 and 5) in the

selected categories. Limiting the work to a single group may undermine the

validity of the study. The group dynamic could, for example, be distorted due

to the heterogeneous composition of the sample. The participants will be

selected so as to ensure that there is a degree of homogeneity in the group.

The number of participants generally ranges from 6 to 10 per group, so that

each person has a turn to speak, and so that sub-groups are not formed.

Usually the participants do not know one another.

The participants are sometimes remunerated. However, rather than offering

an incentive remuneration, it is preferable to offer to refund their travel

expenses, where relevant, and to serve refreshments at the end of the

session.

Step 2. Designing the interview guide: Given the time limits and the

number of persons, it is important to carefully define and limit the topics

addressed. The guides for each group will be different. The group interview

guide must be short because more time is necessary when several persons

participate in the discussion and answer the same questions. Most of the

questions asked are open; they must also be carefully defined and arranged

in a series with the most general ones first.

Step 3. Choice and training of facilitators: The facilitator must prove

her/his creativeness and motivation, and have a good relationship with the

participants. In their training facilitators must learn which attitudes to have

vis-à-vis the group (e.g. respect for people's ideas) and what knowledge to

absorb concerning the objectives of the study and data analysis

techniques. Facilitators must also have communication skills, note-taking

Principal evaluation techniques and tools

skills, etc. It is useful to make provision for a second person per group,

particularly if the session is not tape recorded, so that one person can take

notes while the other leads and facilitates the discussion. Co-facilitators

may observe the discussion and then make recommendations to the

facilitator on the way the meeting was conducted. When this technique is

being used to test advertising or products, the co-facilitators usually

observe the group from another room by means of a two-way mirror or

video link. In the case of scientific study, the use of this approach without

the knowledge of the participants poses a number of ethical problems and

is not therefore recommended.

Step 4. Course of the discussion: The discussion may be launched fairly

openly, by introducing the subject of the session and asking a simple

question of general Interest. This will enable each participant to give an

initial opinion or a remark on the subject. The interview guide serves to

remind the facilitator of all the essential points that need to be addressed.

If all the advantages of the interview are to be maintained, there must be a

written report. It is strongly recommended to record or even to film the

interview, although the latter alternative is often seen as an "intrusion" by

the participant. Some rooms are designed in such a way that the

participants do not notice they are being recorded or filmed.

Step 5. Analysis and report on results: This final phase consists of

interpreting and comparing the information given by the participants, and

looking for shared and divergent opinions within each group. The informa­

tion collected is codified so as to organise the results in relation to the

objectives of the survey. The interpretation of data must take into account

and distinguish two major aspects of the discussion: what the participants

consider as interesting, and what they judge as being important. The

analysis will depend on the number of focus groups questioned, and on the

nature of the interviews (for example, did the focus group discussion have

a structured approach or not?). The results from the different groups are

compared so as to identify any convergence there may be. The report may

quote the most noteworthy statements made by the participants, together

with a summary of the discussions.

Application to the Structural Funds

The focus group technique may be used to test an innovative measure (ex

ante evaluation), clarify the objectives of a project, establish a theory of

Tools for observing changes in the field

action for the programme being evaluated, and identify the problems and

needs of a region and the improvements required during the

implementation of the programme. This technique may also be used at the

end of the programme in the framework of an ex post evaluation, to collect

information for identifying and/or interpreting the effects and impacts of the

programme concerned, and to fix new priorities and orientations.

Strengths and limits in practice

• This type of survey provides in-depth information on the values and

opinions of the interviewees;

• The fact of bringing several persons together provides a certain balance

in the answers given and makes it easier for the evaluation team to define

the general opinion on a particular programme. Owing to the participation

of several persons, the focus groups provide a certain "quality control"

over data collection, by judging the "pros and cons" of each person's

arguments and thus avoiding extreme opinions.

• In a short space of time (from one and a half to two hours), it is possible

to collect a large amount of qualitative information.

• Specific skills are required for managing the group dynamic and obtaining

a balanced discussion while avoiding the dominant influence of opinion

"facilitators", for example and the technique of time and resource

intensive. There is also a risk of "polarisation", that is to say, that at the

end of each group discussion the participants may be influenced by the

majority opinion expressed at the beginning of the discussion.

• The discussions may sometimes be biased, due to the fact that the

participants (addressees) of public policies are subject to an effect of

dependency and will produce a positive judgement a priori.

• The number of people does, however, limit the number of questions due

to the time required to answer them. Moreover, conflicts may emerge

between the different persons, which may undermine the quality and

effectiveness of the interview.

• It is possible that participation in a focus group changes peoples'

perceptions - either because of the 'Hawthorne effect' (the fact that the

behaviour of persons who know themselves to be under observation

changes) or because their interaction with other participants gives them

new insights and perspectives. Thus for example programme managers

Principal evaluation techniques and tools

may actually improve their performance as a result of participating in a

focus group. In this way the focus group methodology may therefore

have an impact on the programme being evaluated or on successor

programmes.

Further reading

Anzieu D, Martin J.Y., (1994) "La dynamique des groupes restreints", PUF

Le psychologue

Complete presentation, by psychosociologists, of the concept of a

limited group, the main group phenomena and some fields of

application.

Aubel J., (1992) "Guide pour des études utilisant les discussions de

groupe", BIT

Practical guide to conducting group interviews.

Greenbaum, TL., (1998) 'The Handbook for Focus Group Research'

(Second edition). London : Sage Publications ;

Krueger, R.A., (1988), 'Focus Group : A Practical Guide for Applied

Research'. London : Sage Publications.

Moore M. C , (1994) "Group techniques for idea building", Sage

A more detailed approach on group techniques, presenting specific

techniques (Nominal Group Technique, Ideawriting, Interpretive

Structural Modeling, etc.)

Morgan, D.L., (1997) 'Focus Groups as Qualitative Research' in Qualitative

Research Methods Series, 16. London : Sage Publications.

Tools for observing changes in the field φ

Case study 0

O

This tool is based on an in-depth

study of the data collected on Medium

Overall Medium

specific cases. It may concern evaluation cost

technical

individuals, programmes, organi­ complexity

sations, projects, groups of people O Θ

A

or decision-making processes. The

approach uses a range of

qualitative and quantitative data In-depth Retrospective Use

evaluation use stabilised

collection techniques. The case

study is recommended for

evaluations aimed at a detailed analysis of c o m p l e x processes of

implementation of programmes and the representation of their impacts. The

case study generally applies to situations that require an exploratory field

study due to the lack of available information.

Example : Evaluation of the NordTek programme, Denmark

This programme, intended for small and medium-sized firms, consisted of about 100

projects offering services to firms in the areas of training, consultancy, access to

modern computer equipment, etc. It was therefore a programme that could be

qualified as "soft", as opposed to "hard" programmes aimed at improving infrastruc­

tures. The results of such programmes are difficult to evaluate precisely because they

are not tangible.

Six large projects were selected as cases to be studied. In terms of the selection

criteria, the projects had to be representative of the diversity existing in intervention

strategies and locality. The selected projects also had to have benefited from

significant funding in the framework of the NordTek Programme and be oriented

exclusively towards firms (rather than research institutions or administrations). The

six projects evaluated represented such a large portion of the total Programme that

the analysis of the impacts of these interventions could be used to cover all the

important phenomena. It was recognised without ambiguity that the selected

projects represented the "best cases" of the programme.

The data were collected at two operational levels: at the level of the project as such,

and at that of the firms. As far as the project was concerned, the evaluators studied

each of the cases in the field four or five times over a period of two years. Each time

the managers were questioned and each project had to keep a record, in a "register"

designed especially for the purpose, of the services proposed and the firms

concerned. The analysis of the documents thus obtained was handed to the project

leaders and discussed during visits. As for the firms, the data were collected during

Principal evaluation techniques and tools

interviews with a sample of 20 SMEs that had used the services offered. Finally, 40

SMEs who had not used the proposed services were questioned in the final stage of

the evaluation.

Source: Olsen, Leif and Olaf Rieper. Nordtek-evaluieringen, AKF Forlaget, 1991.

What is the purpose of the tool?

The case study may have the following aims:

• illustration: the case study is a tool that may be used to add realism to an

evaluation if it is presented in a narrative form. The case must, however,

be chosen carefully because it must be representative of the programme

as a whole or illustrate a specific point - for example a particularly

effective action or an approach which was found to have serious

deficiencies and which should therefore be avoided in future.

• exploration: putting forward hypotheses for future investigations,

identifying the various points of view of the stakeholders.

• critical analysis: verify and validate a statement concerning a programme,

project or strategy.

• analysis of implementation: examine the diffusion of services and its

mechanisms, often in different places.

• analysis of the impacts of programmes: understanding the nature of the

processes producing impacts.

The results of a case study are always presented in a narrative form, as a

story, thus giving the reader an "inside view" of the case studied and an

impression of authenticity. The case study therefore has an analytical and

communicative aim. Readers are more likely to relate to cases where the

programme(s) and personnel involved are identified. However, particularly

when a case study is being used to illustrate and learn from failures it may

be necessary to anonymise some or all of the material in order to secure

access to data and personnel.

Case studies can often be designed in a cumulative way to help answer

evaluation questions. The same case programme may also be studied over

time to provide an analysis that is updated iteratively. Cases may be

descriptive, normative or designed to show causality. They may be

particularly useful in pedagogic/training situations for example being used

to enable officials to evaluate alternative evaluation methodologies.

Tools for observing changes in the field

In which cases should they be used?

Case studies have been used extensively in evaluation during the past

decade. Today this approach is known to provide valid information for both

the evaluation of programmes and the diffusion of new knowledge. Case

studies which use sophisticated selection procedures (e.g. "multiple case

studies with replication design") tend to replace large-scale quantitative

surveys carried out in diverse cultural contexts.

The case study is a method of holistic analysis applied to complex

situations. This means that its use is appropriate for the in-depth

understanding of behaviours and social phenomena, by using the persons

and organisations analysed as a frame of reference. Case studies are

valuable for identifying the effects of programmes inductively, by

developing assumptions concerning the phenomena linking cause and

effect. These assumptions must then be supported by information drawn

from the different case studies and tested through the search for alternative

explanations.

This may prove useful for observing expected results and impacts, but also

for revealing unexpected ones. The approach is less suited to the

identification of causal links, although it may be used to demonstrate that

they are likely to exist.

The case study is intended to be the most complete illustration possible of

a given situation, so as to give a precise image of current phenomena and

to understand their causes. This is obtained by means of the description

and then the analysis of examples situated in their context. It follows that

this type of analysis must be based on multiple data sources, such as

interviews, observations over time, statistics, physical information, etc. The

data must also be cross-checked in order to ensure its coherence. The

notion of "context" encompasses all the factors that could affect the case

studied. Thus, for example, the impacts of a specific project on the

addressees are influenced by a large number of external factors.

How should it be implemented?

The quantity of work required by a case study may vary widely. One must

bear in mind that the case study must be sufficiently rich to give the reader

an impression of what actually occurred. However, the case study is part of

the least standardised methods and may encompass a range of different

Principal evaluation techniques and tools

approaches in different situations. Carrying out a case study involves the

following steps:

Step 1. Selection of cases to study: There are at least three criteria for

selecting cases: convenience/access, the purpose to which they are to be put

and the extent to which they can be considered to provide wider insights

beyond the particular case in question. The selection of cases is a critical step

for generalising and answering evaluation questions. It is difficult to justify a

selection based only on convenience (easy access to data) and probabilistic

surveys are sometimes difficult to carry out. Thus, a choice based on purpose

is appropriate in most cases, as shown in the following table:

Basis of selection

Which questions can it answer? contrasting cases

What happens at the extremes?

What explains these differences? the best cases

What explains the effectiveness of a project? the worst cases

Why does a project not function? by sub-sets

How can the different types of project be

compared? representative cases

Among the examples chosen to represent

significant variations, what happens and why?

On a typical site, what happens and why? typical cases

particular cases

In these specific circumstances, what happens

and why?

(Adapted from GAO, 1990, p.23)

Step 2. Data collection and process: Theoretically, data collection covers

all available information about a case including that derived from project

documents, project meeting reports, and collected at the various

operational levels: interviews with project leaders and staff; observation of

the site of the project; surveys among the addressees of the services

provided by the project. These data must be collected, recorded

(compilation of a "register") and pieced together so that they can be used

in the final report.

Step 3. Case report: Drawing up the report on the case involves the

organisation of all the raw data on the case into a body of exploitable

information. This is then edited, redundant information is eliminated, and

the different parts are combined. The report is organised in such a way as

Tools for observing changes in the field

to be easy to consult, either chronologically orthematically. The report must

include all the information required for subsequent analysis, that is to say,

for constructing an account of the case study.

Step 4. Account: The case monograph should give the reader immediate

access to relevant information and to the particular situation of the case -

the situation of a project - and provide an understanding of the project as a

whole. Each case study, in an evaluation report, must be isolated (the size

may vary between one and five pages). Nevertheless, in the last steps of the

analysis the cases may be used as contrasts or comparisons, depending

on the evaluation objectives.

Application to Structural Funds

The multiple case study method is particularly well suited to analyses of the

various member States and regions, but also to thematic evaluations. The

flexibility of each case study makes it possible to draw up an adequate

description of the peculiarities of a given place or a project. The formulation

of a common set of questions, relative to the evaluation, facilitates the

analysis of the results obtained from multiple case studies. In fact the

results prove to be more sound when they are produced in relation to a

variety of places (through re-using case studies). Similarly, the specificity of

success stories or failures will then seem more obvious. It should be

recalled that the transversal analysis of cases consists of cross-referenced

qualitative examinations and a description based on the frame of reference

established by the evaluation questions. A cumulative process may be

sought when the evaluation is focused, for example, on operational

programmes in Objective 1 regions and when the conclusions have to be

synthesised on the scale of several member States.

The presentation of the results of several case studies could be a barrier to

more generalised use. This difficulty may be solved by means of a graphic

summary providing a brief report of the case history and a graphic

presentation of the results, in relation to each of the questions. In this form,

the answer to each of the evaluation questions, for each case, is set out on

a single page: a graphic presentation at the top, a short but rich summary

of the case history, with the main results, and a concise conclusion. The

summary of the transversal case could be sketched in the same way,

followed by conclusions and recommendations. Principal evaluation techniques and tools

Another type of case study that may be applied to the Structural Funds is

the "integrated approach". This approach takes into account, for example,

the study of results and impacts in the context of a specific programme. In

the NordTek example mentioned above, for each of the projects, a number

of addressees (SMEs) were selected for a mini case study. In that way the

report by the SMEs on their use of the results obtained could be used in the

context of the project. Thus, variations in the results among the different

SMEs can be explained by differences in the situation of the SMEs

concerned.

Strengths and limits in practice

• The case study is relevant for giving a view of processes and

complexities that are impossible to see in any other way. It may even

make outside persons, such as European managers who are hardly

involved in this field, aware of the reality of daily actions. It provides them

with a clearer view of the way in which the programme is put into practice

once the decision has been taken with the national authorities.

• Depending on the selection strategy opted for, if several case studies are

carried out, it may prove impossible to extrapolate beyond the results of

those case studies. It is however impossible to generalise, in the

statistical sense, the results to a global population; to go from a few

projects to a programme, or from a few addressees to all the addressees

of a certain category.

• This approach is less appropriate for measuring the amplitude of impacts

or for inferring the causality.

• Due to the cost of setting up a good case study (requiring sources of

multiple data and competent evaluators), it is necessary to limit the

number of observations. The case study may, however, be re-used and

applied to other contexts, thus providing economies of scale.

• The credibility of the results of the case study is likely to be undermined

if the method is not implemented correctly: incompleteness, arbitrary

selection of information, comments cut short, distortion of results, etc. To

enhance the reliability of a case study, several precautions are

recommended, e.g.: rereading of the case studies by the persons

concerned in order to verify the precision and the veracity of the data and

their interpretation; or having two different evaluators write down their

Tools for observing changes in the field

comments on the same case; involving outside professionals (such as

journalists) in the writing of the comments.

Further reading

Dufour S., Fortin D., Hamel J. (1994), 'L'enquête de terrain en sciences

sociales: l'approche monographique et les méthodes qualitatives',

Montréal: Saint-Martin

GAO (1990), 'Case Study Evaluations.' Washington DC: General Accounting

Office, 133 pp

Sound general approach to the method.

Yin R.K. (1994), 'Case Study Research - Design and Methods', 2nd ed.

Newbury Park, Sage Publications

Complete guide to the case study technique.

Tools for analysing data

An important function of evaluation is the comparison of primary and

secondary data. In the situation concerned, these data are generally

numerous and heterogeneous, making the use of sophisticated tools

difficult. Most evaluations use classical tools such as tables or graphs (see,

for example: Henry G.T. ed., 1997, "Creating Effective Graphs: Solutions for

a Variety of Evaluation Data", New Directions for Evaluation, no. 77).

An innovative and particularly useful technique in this situation is the

geographical information system which can be used to compare data by

considering their location. The maps produced by means of this tool are

used to make visual comparisons that are often highly informative.

Owing to its flexibility, the case study technique, mentioned above, is one

of the rare techniques allowing for an exhaustive, although qualitative and

non-graded, analysis of all the effects of a complex programme.

Most quantitative analysis tools are inappropriate because of the

heterogeneity of data and the multiplicity of publics and processes of

production of impacts. For example, it seems very difficult, in this situation,

to use tools such as factor analysis, regression analysis or comparison

groups.

On the other hand, macro-economic tools are appropriate in cases of vast

programmes covering an entire territory, provided that the situation meets

the following criteria. The funds spent in the framework of the programme

must be significant compared to the economy of the territory concerned.

This territory must be large enough (a country or a large region), or closed

enough (an island) for the global functioning of its economy to be considered

in isolation. When these conditions are met, the evaluation team can use

several tools borrowed from regional economics or macroeconomics.

Shift-share analysis consists of projecting national economic trends onto

the economy of a region. This tool is used to estimate a policy-off situation.

By comparing this with the policy-on situation, the global impact of the

programme can be evaluated.

The input-output model and the macro-economic model are used to

simulate the economic development of the region. These tools are generally

applied ex ante to two types of assumption (with / without the programme)

and provide an overall estimation of probable macro-economic impacts.

Tools for analysing data Φ ©

Geographical Information

Systems O

Geographical information systems High High

Overall

(GIS) were originally developed in evaluation technical cost

the 1970s to stock and process complexity

information related to the physical © G

©

environment (natural and man-

made). More recently, the interest Innovative Retrospective Prospective

of GIS users has spread to socio­ use use use

economic information. GIS have

thus become a way of assembling

any data that can be referred directly or indirectly to a geographical

location. They are now used in the field of spatial planning to collect, gather,

accumulate, analyse, exploit, display and update all spatially referenced

data and information.

Advances in computer technology, particularly computer-assisted

cartography, have led to the development of high-performance information

systems. The management of public interventions is thus able to cope with

a considerable number of interdependent factors, so as to better deal with

the complexities of the real world. Thanks to GIS, these complexities can

be described and analysed by combining information from the thematic,

spatial and temporal dimensions.

Example: Evaluation of the Languedoc-Roussillon IMP in France

This example corresponds with that of Box 12.

The GIS technique was used in an evaluation of the Languedoc-Roussillon Integrated

Mediterranean Programme (IMP), primarily to analyse its spatial effects. This

particular tool was chosen because the evaluation was required to account for the

territorial synergy of a programme composed of multiple actions consisting of

subsidies granted in selected areas.

A wide range of fields was affected by this programme. These concerned problems

of regional development such as the maintenance, diversification and adaptation of

agriculture, the creation of new jobs, the development of tourism, and the protection

of the environment, within spatial (in defined geographical zones), thematic (by

economic activity) and temporal (reference to the initial state) perspectives.

To assess the impact of the IMP measures and the effects of synergy, it was necessary

to observe the changes brought about by the programme and then to explain them.

Principal evaluation techniques and tools

For this purpose an "initial state" was constituted to describe the reference situation.

This step required a substantial effort gathering all the data and indicators of the

socio-economic structure in the territory under consideration - an effort that was

essential in order to be able to make a comparison with the situation observed after

the allocation of the subsidies. Once the data base had been compiled, the

evaluators, using mainly geographical representations, attempted to verify if the

actions undertaken respected the logic underpinning geographical location, spatial

integration and spatio-temporal integration.

The use of cartographic representations associating material and relational data

(social, political and economic) is relevant when the programme itself is based on the

identification of sensitive areas within a given territory. Thus, for example, actions to

prevent forest fires (reforestation, protection) were correlated cartographically with

the areas prone to fires (frequency) in order to assess the logic behind their spatial

integration.

While representation on a map is relevant in this case, it should be recalled that very

often the understanding of a map is not immediate and the results require

interpretation. Initially, only the actions of the programme in question and the current

year were registered. Actions of the same type undertaken by other organisations

between the time of the "reference situation" and the "observed situation" were not

recorded. This led to a biased interpretation of the real situation.

Source: Commission européenne DG XVI (1993) L'évaluation du PIM Languedoc-

Roussillon, MEANS pilote.

W h a t is t h e p u r p o s e of t h e tool?

In general, GIS are used to gather all sorts of information of a spatial nature

and to present it in a form that makes it easy to read. GIS is able to

incorporate socio-economic data, data from censuses, surveys and

inquiries, and monitoring data. The spatial representation may be direct

(cartographic co-ordinates, precise postal address) or indirect (postal code,

area of census).

In terms of evaluation, the main a d d e d value of a GIS is that it allows the

cross-referencing of data that other techniques do not allow. In particular,

the tool can c o m p a r e information on which the geographical references

differ, e.g. address of firms assisted and urban areas in difficulty, or areas

eligible for assistance and environmentally sensitive areas. A GIS also

makes it possible to estimate the value of an indicator for a given area,

when its vasue is registered on different statistical scales. (This point is

discussed in detail in Volume 2 of this Collection, devoted to indicators.)

Tools for analysing data

In which cases can it be used?

Constructing a GIS is a heavy investment, especially in terms of data input.

It is therefore difficult to make such an investment in the framework of an

evaluation with a limited budget and duration. Such an investment is,

however, justified in the context of monitoring.

If a GIS already exists, an evaluation would be well advised to use it. The

collection of evaluation data has to include systematic geographical

references, so that cross-referencing with other information already

contained in the system can be performed.

How should it be implemented?

GIS use a basic structure of special data. All the information is linked to a

system of geographical co-ordinates characterised by nodes, lines and

areas (and therefore also of points, arcs, polygons, etc.).

Box 16 - Principle of a Geographic Information System

Topography

Properties

Protected areas,

forests, etc.

Density of population,

Constructed areas, etc.

Fire protection projects

Resulting map

(selecting and

combining variables)

The different steps in a GIS are as follows:

Step 1. Collection of basic data: The first phase is the collection of

information. The basic data are taken from a wide variety of sources:

Principal evaluation techniques and tools

géomorphologie maps, aerial photographs, existing socio-demographic

data, etc. In some cases, when the raw data cannot be saved directly on

computer (e.g. a sample bank), it is recorded by means of indicator cards.

This step includes the digitisation and inputting of geographical data and

associated alphanumerical data.

Step 2. Storage and management of geographical data base: This

function includes the storage of data in an alphanumerical form and the

programming of recall mechanisms to meet the needs of the subsequent

steps, such as users' requests. The idea is to organise the data (creation of

an index) to facilitate its access based on geographical, thematic,

topological, etc. criteria.

Step 3. Exploitation and analysis of data: A wide range of techniques is

needed for the transformation and use of data in the framework of a specific

problem. The user proceeds either by "questioning", that is to say, he or she

asks the GIS a question directly, or by "extraction" when he or she wants

to compile a partial file for a specific purpose. Moreover, the GIS allows the

combination of data and the identification of correlation between certain

variables with a view to highlighting the characteristics of a geographical

area.

Step 4. Rendering results: A GIS comprises different graphical systems

allowing for the organisation of data in models, such as maps, graphs,

pictures, projections of sequences, etc. These facilitate communication and

allow for user-friendly exploitation of the data.

Application to the Structural Funds

This tool has considerable potential for integrating and synthesising

information. Its ability to integrate the territorial dimension is relevant for the

Structural Funds. It is used to centralise results from statistics offices,

monitoring, and evaluation surveys carried out in the eligible territory.

A GIS also allows easy visualisation of information relating to the

programme. This characteristic can be extremely useful during the

presentation of the results to the various committees and work groups

concerned.

This potential for synthesis facilitates the illustration of the coherence of the

measures implemented in the framework of the programmes, something

that is an asset for the publication of synthesis documents.

Tools for analysing data

Strengths and limits in practice

• Above all, GIS have a didactic faculty: results can be rendered in a user-

friendly way, which helps to enhance the awareness of actors. However,

cartographic language is not univocal; signs and keys are codes that

vary, depending on the designers, and need to be explained.

• Apart from the fact that the creation of a complete geographical database

often proves to be costly in both time and money, the use of a GIS may

be complex. Existing data are often collected on different spatial entities

(administrative partitioning into communes, cantons, etc.) which requires

time consuming transformations or sometimes even a new collection of

data. Existing data may also be stored on information systems that, due

to their incompatibility, make it impossible to use them.

• Before being integrated in a GIS, the data produced by monitoring must

be processed by means of statistical techniques that may be

cumbersome.

• In practice, GIS facilitate the processing of a substantial number of data.

It also has the advantage of allowing the transmission of data from one

user to another. The centralising nature of the tool facilitates the updating

of data and allows for the manipulation of geographical information on

varying scales (e.g. from a small area of land to an entire region).

• GIS are also useful when they are combined with other analysis

techniques such as factor analysis.

• Experience tends to show that the tool may generate conflict between

territorial officials due to their biased reading of the maps produced.

Further reading

Collet C. (1992), Système d'information géographique en mode image, coll.

"Gérer l'environnement", Lausanne: Presses polytechniques et

universitaires romandes. 186 p.

Simple but very effective for an introduction to GISs.

Fisher M. and Nijkamp P. (1993), Geographic Information Systems, Spatial

Modelling and Policy Evaluation, Berlin: Springer-Verlag. 280 p.

Volume comprising background articles with a specific focus on

applications to evaluation. Principal evaluation techniques and tools

Martin D. (1996), Geographic Information Systems, socioeconomic

applications, London: Routledge.

Introduction to GISs with a specific focus on socio-economic

applications.

Tools for analysing data

Shift-Share analysis Θ Φ

O

Shift-share analysis has been used

since the 1960s for evaluating Low Low

Overall

regional policies. It is a t o p - d o w n evaluation technical cost

evaluation technique in so far as it complexity

projects observable trends at the Ο Θ

level of a country, onto the

economy of a region. The starting Retrospective Stabilised

point of a shift-share analysis is not use use

the study of the consequences of a

programme, but the observed

trends at the national and regional levels. Shift-share analysis focuses only

on macro-economic dimensions, to construct a basis for comparison of

what would have happened without the programme.

Example: Employment impact of regional development policies in Scotland

100 This graph

represents a

difference in

thousands of jobs

between the policy-

on situation and the

policy-off situation

A positive value corresponds to a regional

creation of jobs greater than forecasts

based on national trends. The difference

(residue) is attributed to the effects of

regional policy.

-200

The case of Scotland is a classical example of shift-share analysis. It is particularly

meaningful because it presents two clearly distinct phases in regional policy: a

passive phase during the 1950s, during which there was no regional policy, followed

Principal evaluation techniques and tools

by a more active phase during the 1960s when the British government provided

abundant support for the development of Scotland. The application of shift-share

analysis to the evaluation of British regional policy in Scotland was used to draw the

graph above.

It can be seen that, in the 1970s, jobs in Scotland tended to increase faster than

forecasts based on national trends suggested. This leads to the conclusion that

regional policies had a positive impact on employment in this region.

Source: Diamond D., Spence N. (1983), Regional Policy Evaluation. A Methodological

Review and the Scottish Example, Gower House: Ashcroft

What is the purpose of the tool?

Shift-share analysis is used to give an indication of the effects of a policy in

a given region. The effect of the policy is determined by comparing the real

situation in the region, i.e. the policy-on situation, with a reconstruction of

what the regional situation would have been like had it followed national

trends, i.e. a policy-off situation.

In which cases can it be used?

Shift-share analysis is designed to evaluate the effects of regional policy at

a macro-economic level. Used for retrospective analysis, it is based on

statistical indicators and requires no survey on the target groups of the

programme.

How should it be implemented?

Shift-share analysis consists of two steps: the construction of a policy-off

situation, and the comparison. Both steps are illustrated by means of an

analysis in terms of employment (the most frequent use of shift-share

analysis).

Step 1. Construction of the policy-off situation: this is based on simple

modelling. The national employment trends are applied to the structure of

regional employment. The formulation of this procedure is as follows:

IM, = IM + NS

0

IM characterises the sectoral distribution of employment in the region (i.e.

the industry mix) and NS is the variation of the employment that would be

produced if each regional sector developed at the pace observed for this

sector nation-wide (i.e. the national share component).

Tools for analysing data

Step 2. Comparison for determining the impact of the action: This step

consists of the comparison between the policy-off situation, obtained

during the preceding step (IM and the policy-on situation. The difference

t)

between these two situations (the residue) is used to highlight the impacts

of the political action. RS = E, - IM,

Here, the programme impact is equivalent to the residue (i.e. the regional

shift component).

Box 17 - Example of shift-share analysis of regional employment

The number of jobs in a region in which the public authorities provide e m p l o y m e n t

incentives is 300,000 ( I M J in which 100,000 are industrial jobs, 150,000 are in the

service sector and 50,000 are in agriculture and the f o o d sector, during the period

t = 0.

Nationally, employment trends ¡n the industrial, tertiary and agricultural sectors

are - 1 % , 3 % and - 2 % respectively between t = 0 and t = 1 . Assuming that

regional employment developed at the same pace as national employment, the

increase in regional employment w o u l d increase by 2,500 (NS). The level of

employment in the policy-off situation w o u l d then be 302,500 (IM,).

By contrast, during the period t = 0 to t = 1 , employment in the assisted region

progressed by 5 % , i.e. 15,000 jobs. The number of jobs w a s then 315,000 (E,).

According to this analysis, the number of jobs created as a result of the

intervention is 12,500 (RS).

Application of national trends

to the region 315 000

jobs

IM-, =

IM = Residue

0 RS =

302 500

300 000 12 500

jobs Impact

jobs Service

sector

+ 3%

Distribution per

sector of

employment in

the region Industrial

sector

- 1%

Agricultural

sector

-2%

Before the Policy-off situation Policy-on situation

programme After the programme

Principal evaluation techniques and tools

Application to the Structural Funds

Because shift-share analysis was designed to evaluate the effects of

regional policies it is, theoretically, directly applicable to the Structural

Funds. It can therefore be used in ex post evaluations of regional

development programmes, whenever the emphasis is on the study of a

particular impact such as employment or value added. Shift-share analysis

is appropriate for use in programmes implemented in the framework of the

ERDF, due to their regionalised nature.

Because this technique has few requirements, it has the advantage of being

easy to use. The statistical data it requires are available for most European

regions, particularly in the form of employment data and value added for

NUTS 1 and 2, provided by EUROSTAT.

Strengths and limits in practice

• Shift-share analysis is used to obtain quantified conclusions on the

effects of policies.

• The implementation of a shift-share analysis is not standardised (dates of

start and conclusion, sectoral breakdown, national reference); as a result

it is difficult to compare the results of analyses of different programmes.

• It is an inexpensive tool because it can use widely available statistical

data. All that is needed is an impact indicator (employment or value

added), with a sectoral breakdown at the regional and national levels.

• The logic of the implementation of a shift-share analysis is often

simplified to the extreme. Usually, the variations observed in regional

employment can be explained by factors other than the simple projection

of national trends. Among such factors are the growth of the region's

exports, productivity, quantification of manpower, population-growth

pressure, immigration, and the intensity of research and development. It

is not credible to attribute the effect of all these factors to regional policy

only, unless this policy had a very strong hold over all the regional

economic agents.

• In shift-share analysis the impact consists of the residue, which is the

difference between the policy-on and the policy-off situations. However,

the residue may be attributed both to the programme under evaluation

and to all other national and European political actions, as well as to the

former trajectory of regional policy. In theory, shift-share analysis can be

Tools for analysing data

applied more satisfactorily if a very far-reaching regional policy is

evaluated at the end of a period characterised by the absence of regional

policy.

Further reading

Bartels C.P.A. (1982), Measuring Effects of Regional Policy. An introduction,

Regional Science and Urban Economics 12, 41 p.

A review of all techniques used to analyse the impact of

employment effects. Shift-share analysis is addressed from a critical

point of view. Highly accessible.

Steevens B.H., Moore C.L. (1980), A Critical Review of the Literature on

Shift-Share as a Forecasting Technique, Journal of Regional

Science, Vol. 20, N°4, pp. 420-435

A highly formalised article presenting the main shift-share models

and their possible uses in forecasting.

Tervo H., Okko P. (1993), A Note on Shift-Share Analysis as a method of

estimating the employment effects of regional economic policy,

Journal of Regional Science, Vol. 23, N°1, pp. 115-121

A critical article on the use of the shift-share technique for evaluating

regional employment effects. Based on formalised demonstrations.

Tools for analysing data 0

Input-Output Model φ)

O

Input-output matrices were

originally developed to analyse High High

Overall

interaction between the different technical cost

evaluation

sectors within relatively closed complexity

O

economies (low level of exports Θ

and imports). They describe the

way in which the productive Prospective Use

system satisfies final demand use stabilised

(consisting of consumption,

investments and exports). An

input-output matrix represents the links between an economy's resources

and their consumption. The matrix may vary from the simple (three sectors:

industry, services and agriculture) to the complex (over 500 branches). It is

one of the only techniques applicable to the evaluation of the sectoral

impacts of structural interventions, because it makes it possible to divide

up the productive structure of an economy in great detail. An input-output

matrix can be compared to a macro-economic model that is highly

simplified as regards the economic mechanisms represented, but extremely

detailed from the sectoral point of view.

Example: The impact of Structural Funds on Objective 1 countries and regions

A team of German academics adapted input-output analysis to the study of the

impact of Structural Funds on Objective 1 countries and regions. It used matrices

provided by EUROSTAT for the reference year 1985 and forecasts for the years 1991,

1994 and 1999. In this example, the team tried to estimate the impact on overall

economic growth and on sectoral distribution. The following results are the most

representative ones from this study:

Without the Structural Funds, average economic growth in Objective 1 areas would

have been less than 0.7% in absolute terms.

In 1999, the Structural Funds are expected to induce a 1.9% growth in employment

in the Objective 1 areas.

The analysis was differentiated for the different countries and effects on growth and

employment were shown to be greater in Greece and Portugal.

For the period between 1994 and 1999, changes in the sectoral structure of the

production (in terms of GDP) of these regions can be represented as follows:

Principal evaluation techniques and tools

19.0%

Public sector 119.9% ZJ42.8

Services _|41.0%

Building and l9.6%

civil engineering _J9.1 %

Manufactured 19.0%

products 19.1%

I5.594

Energy |5.7% D 1999

D 1994

u .2%

Agriculture 5.2%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45%

Source: J. Beutel, Konstanz University

What is the purpose of the tool?

Input-output matrices are used primarily in economic forecasting, where

they serve to verify policy scenarios, based on the technological structure

of the economy of the country and on the state of final demand. In an

evaluation they can be used with or without policy interventions, in the

same way that a macro-economic model is used (see corresponding sheet).

The tool is used to estimate impacts differentiated according to the different

branches of an economy. Among the numerous applications of input-output

matrices to the evaluation of development programmes, the above-

mentioned example will be considered because it has the advantage of

presenting harmonised findings on a European scale.

The model was produced by Konstanz University and EUROSTAT. It was

used for estimating the net effects of the main programmes financed by the

Structural Funds.

In which cases can it be used?

In general, this tool is used on a national scale, because this is where the

statistical data needed for the construction of matrices are collected.

Tools for analysing data

Regional matrices do exist and also lend themselves to input-output

analysis. However, their statistical value is not as great because of the

difficulty of monitoring the movement of goods that are not subject to

customs duty.

How should it be implemented?

The following lines describe the implementation of the tool from its

construction to effective simulations in the context of the evaluation. In

reality, an input-output matrix is rarely constructed for an evaluation. The

implementation of the tool starts, in practice, with step 4.

Step 1. Construction of a transaction matrix: the connections between

the different branches of an economy are described. The lines of this matrix

consist of the outputs, which concern resources supplied by a given sector

to each of the sectors of activity and to the final demand. The columns

consist of the inputs from the different sectors, i.e. the consumption per

sector required for production.

Outputs Industry Inter

Inputs Agri­ Services Final Total

mediate demand utilisa­

culture consump­ tion

tion

250 350

Agriculture 50 150 50 100

Industry 150 150 100 400 320 720

100 100 200 600

Services 400 1000

400 1050

350 1020 2070

Intermediate 300

products

Value 50 320 650 1020

added 720

Production 350 1000 2070

The values taken in this example are monetary. Thus, for example, the

agricultural branch provides a value of 150 to the industrial branch for its

production, whereas a value of 100 is intended for the final demand. The

industrial branch consumes a value of 150 in intermediate products from

the agricultural branch for its production, leading to the creation of value

added of 320. By adding the intermediate consumption of agricultural

products by the different sectors, to the final demand for the agricultural

Principal evaluation techniques and tools

branch, one obtains the total utilisation of agricultural production. The final

demand may itself be broken down into final consumption, investments and

exports. The sum of the intermediate inputs consumed by the agricultural

branch for its production, and the agricultural value added, equals the

agricultural production. The totals of the lines and the columns are therefore

equal.

Step 2. Construction of a matrix of technical coefficients: This is based

on the matrix of transactions. The technical coefficients are nothing other

than values taken from the matrix of transactions divided by total

production. The relative part of the value created by each branch for all the

other branches can thus be seen. The technical coefficients serve to

provide indications on the technical structure of the modelled economy;

they show which sectors have a high level of value added, productivity and

exports. These coefficients are artificially fixed for the following steps.

Step 3. Construction of an inverted matrix: This is the result of a matrix

transformation through which multiplier coefficients can be calculated.

These coefficients summarise all indirect effects.

Step 4. Hypotheses of primary impacts: A quantitative estimation is made

of the main impact of the programme, in the form of an increase in final

demand (increase in public spending, increase in consumption, increase in

infrastructure investment, etc.). The assumption of increased demand

includes a breakdown per sector, so that it can be introduced into the matrix.

Step 5. Estimation of total impact: The calculation of total impact is based

on the assumptions of primary impact and of the inverted matrix. Based on

the primary impact on a sector, the tool is used to estimate all indirect

effects on all sectors, particularly in terms of consumption, imports,

production and employment.

Application to the Structural Funds

Among the different applications of the input-output models, the focus here

will be on the model mentioned above, known as the "Beutel model". This

example is particularly interesting because the model was constructed

especially for evaluation and has the advantage of providing harmonised

estimations on a European scale.

The Beutel model makes use, above all, of a series of harmonised input-

output tables for the year 1985, drawn up by EUROSTAT for the economies

Tools for analysing data

of the European Union. These input-output tables are completed by regional

tables, particularly in the case of Spain and Italy. The countries covered are

Portugal, Greece, Ireland, Spain, Italy and the UK (Northern Ireland).

Germany is, moreover, included in the planning period 1994-1999. Since the

1985 database is obviously outdated for any practical purposes, it has to be

updated. For this purpose Beutel developed a forecasting technique adopted

by EUROSTAT (see more detailed information at the end of this section).

The main impact of Community funding is estimated by dividing total

expenditure into three categories: property investments, investments in

capital goods and salaries. A comparison is then drawn between the

development of the economy with and without Community funding. As far

as the evaluation of interventions in the framework of Objective 1 is

concerned, the Beutel approach provides estimations focused on the

following key macro-economic variables:

• Employment: comparison between cases with and without intervention is

established not only in terms of total employment, but with a distinction

being also made between employees on the one hand and independent

workers on the other.

• The GDP and its growth rate: It is possible to draw a comparison between

growth rates with and without intervention. Another interesting

comparison is made between the annual growth rate with intervention

and cases where interventions are interrupted during a specific year.

• Gross capital formation: This variable captures changes induced in

investments by the interventions and its annual growth rate.

• Value added by sector: The sectoral distribution of changes induced in

25 economic sectors by the interventions is calculated each year. A stan­

dard presentation exists of the evolution of the 6 sectors undergoing

structural change: agriculture, forestry and fishing, fuel and energy,

building and civil engineering, private services, and public services.

• Exports and imports: The impact of interventions on exports and imports

is calculated each year and sub-divided into exports and imports to and

from countries belonging to the European Union, and in exports and

imports to and from non-EU countries.

Strengths and limits in use

• The most interesting contribution of input-output matrices concerns

impacts on sectoral distribution and trade. Leakages due to imports by

Principal evaluation techniques and tools

the beneficiary country from Member States of the European Union and

third countries may prove to be important for EU policy-making. Similarly,

knowledge of the impact of sectoral demand may prove to be important,

particularly if it is considered that some sectors must be stimulated to

accelerate the growth rate.

• The value of this approach lies above all in the broadness of its scope and

in its coherent treatment of the main economies of the European Union

benefiting from Objective 1 support.

• The advantage of the tool is limited to the estimation of effects on

demand. Therefore, it does not take into account one of the most im­

portant effects aimed at by Objective 1 interventions, i.e. lasting effects

on productive potential. Most effects on supply, which are likely to lead

to a sustainable increase in the growth rate of assisted regions and to

enable them to catch up with more developed areas, are totally

overlooked (creation of new productive capacity, improvement of the

training and education of the workforce, construction of infrastructure,

productivity gains throughout the economy, spread of technological

progress and intensity of high-tech activities in the productive sector). It

is all these effects on supply which transform productive capacity in a

lasting and irreversible manner.

• A tendency to over-estimate the impact of Structural Funds on demand

has been observed, because the impact of interventions is not

moderated by feedback effects, particularly as regards price changes

originating in product markets, financial markets and the labour market.

Further reading

W. K. Brauers (1995), Prévisions économiques à l'aide de la méthode

Entrées - Sorties, Paris: Economica, 111p

An introduction to input-output analysis, for the general public.

Highlights the main techniques, recounts the debates that took

place around this subject and provides examples of application.

Concentrates on the most important aspects.

R. E. Miller and P. D. Blair (1985), Input-Output Analysis, Foundations and

Extensions, New-Jersey:Prentice-Hall, 464p

Intended for the initiated, this book describes the basis of input-

outpu: analysis and provides a wide view of the different types of

Tools for analysing data

analyses that may be carried out by means of this technique.

Readers can easily select those articles of particular interest to

them.

H. D. Kurz, E. Dietzenbacher and C. Lager (1998), Input-Output Analysis,

Cheltenham: Edward Elgar, 3 volumes, 1504p

In this work the evolution of input-output analysis is studied through

a collection of articles published since the origins of this type of

analysis.

More in-depth: the construction of Beutel model matrices

In the Beutel model, the EUROSTAT input-output matrices are updated by

means of economic forecasts by the European Commission Directorate

General for Economic and Financial Affairs. These forecasts are derived

from the QUEST model (see sheet on macro-economic models). Use is also

made of the forecasts of 25 productive activities, prepared by ERECO

(European Economic Research and Advisory Consortium, composed of Ifo-

Institut/Munich, BIPE/Paris, Prometeia/Bologna, NEI/Rotterdam,

WIFO/Vienna, and Cambridge Econometrics/Cambridge).

The data used for updating the model consist of the input-output table for

the reference year, including the import matrix, growth rate forecasts of

value added per sector (agriculture, industry, services) and the forecasts of

components of final demand (consumption, investments, exports and

imports). The updated input-output table is estimated by a procedure of

iteration which provides estimations on the intermediate consumption of

imported goods and services, the structural composition of final demand by

product, domestic production by product and imports by product.

This technique tries to avoid random changes of input coefficients by

treating all activities equally. Apart from its limited need for data and the fact

that it uses official sources, this technique has the further advantage of

producing sound updated versions at a low cost.

Tools for analysing data φ)

Β

Macro-economic model O

The purpose of the macro-

economic model is to represent the High

Overall High

functioning of an entire economy technical cost

evaluation

and of the different markets complexity

comprising it (e.g. product market, O Θ

job market, financial market, etc.).

A model schematises the Prospective Use

functioning of an economy at use stabilised

equilibrium. It also simulates the

evolution of this economy

following an exogenous shock (e.g. variation in oil prices, variation in

exchange rates, tax imposition), the adjustment of modelled economic

variables and the attainment of a new market equilibrium.

Macro-economic models are used by administrations for the preparation of

their budgets and for the study of political choices when the economic

stakes are high. Some public and private forecasting institutes also use

models for short- and medium-range economic forecasts.

The use of this type of model for evaluation is far more recent and still

involves methodological innovation. This does not, however, preclude the

existence of success stories such as the following case.

Example: Evaluation of the probable impact of European economic support to

the Republic of Ireland

The evaluation of the probable impact of European economic support to the Republic

of Ireland (CSF 89-93) was carried out by the Economic and Social Research Institute

in Dublin (ESRI), using a macro-economic model constructed by this institute in the

framework of the HERMES programme. The evaluation was to estimate the probable

impact of European support on the Irish economy as a whole and, in particular, on

variables such as the per capita GDP or the public debt.

The HERMES model was constructed to simulate the functioning of the Irish economy

in the context of an energy crisis. It was modified slightly during the evaluation in order

to take into account the impact mechanisms of European programmes more

satisfactorily. The model used for the evaluation contains over 600 equations

representing assumptions on the behaviour of the economic agents concerned directly

or indirectly by European assistance. The evaluation conclusions were as follows:

In the year 2000, the impact of the Community intervention is estimated at 2.7% of

Irish GNP. Principal evaluation techniques and tools

However, this impact alone will not be sufficient to allow the Irish economy to catch

up with the European average.

It is interesting to note that this evaluation was the subject of political debates during

the Irish referendum on the Maastricht Treaty, and that it served as an argument

during the renegotiations of European assistance for the period 1994-99. It can

therefore be seen that despite the inherent fragility of any estimation, macro-

economic evaluation served as a basis for constructive debate.

(Source: Bradley J. and Fitzgerald J., Medium-Term Review 1989-1994, Economic

and Social Research Institute, Dublin, 1989)

What is the purpose of the tool?

A macro-economic model provides a simplified representation of the

mechanisms of an economic system. Used in evaluation, the tool makes it

possible to simulate one scenario with public intervention and one without

it, and therefore to estimate the consequences of the intervention by

observing the differences between the two scenarios. The models are

entirely quantified and therefore provide an estimation of the extent of the

impacts on all the macro-economic variables that have been modelled.

Obviously a model cannot predict everything, nor even guarantee the full

reliability of these simulations. However, the tool remains relevant if an

estimation of macro-economic effects is required, for no technique is more

reliable. It provides reference values on the impact of specific measures

that are difficult to obtain by other methods due to the complexity of the

mechanisms in action (e.g. impact on employment and private investment).

These reference values are used to stimulate discussions and to provide

solid information about the real extent of the issues. Finally, the value of the

tool is often educational. Use of the model and the examination of the

results obtained make stakeholders in the evaluation pay more attention to

understanding macro-economic mechanisms.

In which cases can it be used?

Effects can be estimated ex ante. In this case, two forecasts (with and

without) are compared. This is the most frequent use in evaluation.

Although it has very seldom been done, a model can also be used in an ex

post situation. In this case, the comparison concerns past economic

developments as recalculated by the model, and a scenario without public

intervention, reconstructed ex post by the model (simulation of the policy-

Tools for analysing data

off situation). In the framework of intermediate evaluation, the tool may be

used, where relevant, to guide the re-orientation of the programme (e.g.

distribution of funds between investment support, infrastructure

development and support for R&D).

How should it be implemented?

The implementation of an evaluation using a macro-economic model must

be considered only if the evaluators have adequate knowledge, skills and

time. This is because numerous highly technical steps are involved.

Step 1. Adapting the model to evaluation: The construction of a model is

a heavy investment that may take up to a few years. A macro-economic

model is therefore never constructed for the specific needs of an evaluation.

It may happen that several models are available and that the first step

consists of choosing one, but this is exceptional. Generally, the evaluation

team's choice is dictated by the existence and availability of a single model.

The problem is then to decide whether the model can be applied in its

present form or whether it must be adapted to better simulate the impacts

of the evaluated intervention. In the first case, the adjustments can generally

only be minor ones because the equilibrium conditions of the model cannot

be challenged without the risk of having to reconstruct it entirely. In general,

adaptation of the model is limited to a few adjustments that make it possible

to take into account the content of the programme better, and to integrate

the primary impacts in the form of exogenous shocks.

Step 2. Adjusting the model: Before being used, and particularly if it has

been modified, the model must be adjusted to the most recent

developments observed and to the most recent statistics. A model is ready

for use when it is capable of explaining correctly what happened in the five

to fifteen preceding years. In practice, the model must regularly be re-

estimated to take into account revisions in basic information and thus to

allow for it to be "adjusted" to the most recent data.

Step 3. Estimation of primary impacts: The diagram (Box 18) shows the

macro-economic mechanisms taken into account in the model.

Employment depends on the production of firms, on productivity and on the

level of salaries. These three variables are endogenous, i.e. internal to the

model. To evaluate the effect of a public intervention, for example its effect

on employment, this public intervention must be considered to introduce

exogenous changes into the model (the term exogenous shock is used).

Principal evaluation techniques and tools

The evaluation team must then select a few variables or coefficients that are

part of the model and which will be directly affected by the public

intervention. These values constitute "bridges" between the evaluated

intervention and the model. In the following example they are public

spending and productivity. The primary impacts of the Structural Fund

interventions may also take the form of investments in productive structures

or in human capital. The evaluation team must then estimate the primary

impacts of the intervention (micro-economic impact) and introduce them

into the model.

Box 18- Primary impact and macro-economic mechanisms

Primary impact _^ Primary impact

CSF Demand side

Supply side

Salaries «-» Employment Public debt

A

Taxes Public

spending

Income I

Import.

Costs and Production ·* * Demand

Export.

productivity International

prices

Step 4. Simulations: In an evaluation context, the model will always be

used at least twice. The first time it will be used to adjust the intervention to

the situation (case of an ex post evaluation) or to simulate this situation

(case of ex ante evaluation). The second time it will be used to simulate the

situation without intervention. The impacts of the intervention are estimated

by establishing the difference between the two simulations.

Step 5. Presentation of results: The last phase is that of the formulation

and presentation of results. Given the complexity of the tool and the

numerous variables that can be quantified, it is necessary to take specific

steps to present results in a simple and easily accessible way. This phase

must therefore be considered as one of the most important in the

implementation of the tool.

Tools for analysing data

Application to the Structural Funds

Macro-economic evaluation of a Structural Funds programme depends on

the existence of a model for the eligible territory. Available macro-economic

models are generally limited to national models for reasons of availability of

statistical data and the cost of modelling. The models constructed on a

regional or local scale are rare, particularly because economic flows are

diluted as soon as they are not in the territory concerned. However,

modelling remains meaningful for small economies, provided that they are

relatively closed. That is why insular economies are often equipped with

macro-economic models.

Thus, the use of models is generally possible for big programmes for which

the eligible zone corresponds to national borders (Portugal, Ireland, Greece)

or to vast regional blocs (South of Spain or of Italy). Almost all the models are

ill suited to programmes for which the eligible zone is a small, open region.

Furthermore, the use of a model has meaning mainly if the intervention of

the programme attains sufficient mass in relation to the modelled economy.

It is considered that in order to attain this critical mass, the economic

importance of the programme must be at least around 1 % of the GDP of

the economy of the territory.

In the framework of European structural interventions, macro-economic

evaluation is limited to a few experiments of which the two most well known

used the QUEST and HERMIN models presented below.

In the case of the QUEST II model, Structural Fund spending is considered

to be an increase in both public spending and the national stock of capital.

From this point of view, the expenses incurred are considered as an

exogenous increase of overall public consumption (demand side effect), as

well as an exogenous increase in public investment, leading to an overall

increase in production (supply side effect).

This type of approach makes a very broad estimate of supply side effects

since all kinds of investments (infrastructure, R&D and training, both private

and public) are assumed to have similar impacts on productive potential.

Further refinements of the model might consider differentiated impacts

stemming from various publicly funded interventions. This would have the

advantage of fully taking into account the supply and demand effect of

Structural Fund interventions in a clear and fairly direct way. The supply

effects would be simulated separately and directly via the impact on

Principal evaluation techniques and tools

potential production. The combination of supply and demand effects would

indicate the extent to which the structure of the economy is likely to benefit

from the increase in productive potential

The HERMIN model was used in evaluations in Spain, Ireland and Portugal.

As in the experiment with the QUEST model, the demand effects of

spending in the framework of Structural Fund interventions were introduced

in the form of exogenous increases in public spending. Yet evaluations

carried out with HERMIN introduced an interesting innovation by explicitly

simulating the supply effects of public expenses, trying to make separate

estimates of the effects of three categories of interventions: basic

infrastructures, human capital and investments in the productive sector.

The primary impacts of Structural Funds in terms of supply are introduced

into the model in several ways. They first take the form of an increase in

productivity that affects one or more parameters of the production function.

In the Spanish and Irish versions, productivity is "embodied" in labour (a

single parameter). In the Portuguese model, it is total factor productivity

which is increased (all the parameters). The increase in productivity is a

function of investment generated by European support in the form of

private capital, human capital and public infrastructure capital. It is

assumed that there is a constant relationship (called elasticity) between the

increase in investments and the increase in productivity. A real difficulty in

evaluation relates to the choice of the value of this relationship. Empirical

observations which might allow its estimation are barely reliable and, to

avoid totally random values, similar studies based on American or

international data are often used. The credibility of the basis of this form of

estimation of supply effects is somewhat limited.

A second hypothesis on primary impacts is that the action of Structural

Funds helps to draw more foreign investments and therefore to increase the

competitiveness of exports. To estimate this chain of effects, their

equivalent in terms of increase in international demand is estimated. Here

again, it is assumed that there is a constant relationship between the

increase in investments generated by the European funds and the increase

in exports. The choice of a value for this relationship is even more

problematical here. First, arbitrariness is inevitable, given the total absence

of applicable empirical studies not only for the countries concerned but also

at an international level. Worse still, the final result seems to depend heavily

on the value chosen for this relationship. This is particularly true for the

long-term impact on GDP.

Tools for analysing data

To assess the potential of HERMIN for the evaluation of macro-economic

effects, it must be kept in mind that this work is still underway and that

many current difficulties are likely to be solved as the project progresses.

Various teams' estimations of the impact of the CSF should help to refine

the assumptions of impact, the parameters of the models, the quality of the

data, and the comparability of results.

Strengths and limits in practice

• The aim of the Structural Funds is economic and social cohesion. This

objective is essentially defined at the macro-economic level. The macro-

economic model is practically the only tool that can be used to formally

ascertain whether European policy has achieved its aim.

• In the evaluation of socio-economic programmes, one of the major limits of

models is that they do not give a credible explanation for "supply-side

effects". For example, if a description of the effect of new infrastructure on

the productivity of firms is required, it will be difficult to find an equation which

reflects this effect and which is convincingly adjusted to past statistics.

• One of the main limits of the tool relates to the construction of models.

This represents a substantial investment and always takes more than a

year. Moreover, the parameters of a model have to be continuously

updated, under the supervision of specialists with considerable know-

how. Experience shows that statistics and forecasting institutes are

obliged to "readjust" their models several times a year in accordance with

recent statistics.

• Only exceptionally is a model constructed specifically for an evaluation.

Use of the tool therefore depends on the existence of a model. The existing

model must often be adapted to the questions posed by the evaluation.

• Each macro-economic model is based on its own specific set of

assumptions. When two models are constructed for the same territory,

they never produce exactly the same simulations.

• The weakness of many macro-economic evaluations relates to the way in fi

which the primary micro-economic effects are evaluated before being

introduced into the model. In this sense, the macro-economic evaluation

encounters the same difficulty as the other evaluations of multi-sectoral

programmes: the effects of a multitude of heterogeneous interventions

have to be summarised in the form of a few standardised and quantified

micro-economic impacts. This task, which is very difficult to accomplish

Principal evaluation techniques and tools

for a normal evaluation, is only the first step in a macro-economic

evaluation. It is the step that was defined above as the estimation of

primary impacts. Irrespective of the accuracy of its internal assumptions,

the model only transforms estimations of primary impacts into macro-

economic estimations. Thus, it accumulates the difficulties of the two

levels of estimation.

Further reading

In the interests of clarity, the preceding paragraphs deliberately omitted the

technical aspects of the construction of models. For readers who would like

more detailed information, a list of publications on macroeconomics and

modelling is provided below.

Since the use of models is still at its experimental stage for the evaluation

of socio-economic programmes, it is difficult to find manuals with a detailed

explanation of techniques for estimating macro-economic impacts. This

information sheet has therefore been extended to cover the QUEST and

HERMIN models in greater depth.

Andersen, K.V. (1995) Information systems in the political world:

Implementation, Use and Implications of Economic Modeling

Amsterdam: IOS Press. , 283 p.

This book investigates the use of econometric models for policy

analysis and evalution in Europe and the USA, at central and

regional level. It focuses on the social and political dimensions of the

processes by which models are built and used.

Daloz J.P. and Goux C. (1985), Macro-économie appliquée. Du simple au

complexe, Paris: Cujas, 156p.

Very clear, basic approach which introduces econometrics in a very

simple way.

Gray A.W. (1995), EU Structural Funds and other public sector investments.

A guide to evaluation methods, Gill & Macmillan, 106 pp

Considers a set of models adapted to the evaluation of Structural

Funds, and explains problems in this field. Highly accessible.

Kebabdjian G. (1994), Les modèles théoriques de la macro-économie,

Paris: Dunod, 194 pp

A macroeconomics manual which addresses all current trends

without any bias. A demanding, synthetic and useful tool.

Tools for analysing data

More in-depth: the QUEST and HERMIN models

QUEST: a multinational model

The QUEST model (Quarterly European Simulation Tool) was developed by

DG II of the European Commission. Its main purpose is to make medium-

term and long-term forecasts. The model is used to quantify scenarios in

various fields: monetary policy, budgetary policy, and the economic

incidence of changes on the international scene, such as fluctuations in

exchange rates, interest rates, oil prices and other world prices. The way in

which the multinational dimension is modelled is an essential characteristic

of the model.

The QUEST model uses national sub-models to cover all the Member

States of the European Union (with Luxembourg associated with Belgium),

the United States and Japan. For the rest of the world, only international

trade is modelled. It is divided up into seven main OECD countries and six

other regions. The equations of bilateral exports for the commodity market

are calculated, on the basis of import volumes and export prices, for the 26

countries/regions. On this basis, the models of international commodity

trade can be represented in a series of equations on commercial relations.

This bloc of commercial relations can be used to monitor an economic

shock arising from international trade. It is separated from national sub­

models. It is thus possible to study an impact on a country, either taken

alone or embedded in international trade as a whole.

The national sub-models all have the same basic structure. They have a

relatively low number of behavioural equations, for which the parameters

have been adjusted for each country. The basic QUEST structure combines

three types of relationship: the supply bloc, the demand components, and

the wage/price ratio.

(a) Supply

The main purpose of the supply bloc is to determine productive capacity.

The basic assumption is that productive capacity is roughly limited by the

stock of national capital. It should be noted that QUEST is a single sector

model that establishes no differences between the products of the various

sectors. Productive capacity serves, together with actual production, to

define the degree of capacity utilisation. The latter variable plays an

important part in the model and partly determines demand in terms of

investments and imports, and therefore global demand and price levels.

Principal evaluation techniques and tools

Productive capacity depends on private and public capital, employment

and energy, plus an exogenous increasing productivity trend applying to all

factors. Full effects on potential output take about six years to materialise.

(b) Demand

Private consumption is determined by the present discounted income of

current and expected income and wealth of households. Also current

disposable income plays a role since a proportion of households is subject

to liquidity constraints.

Investment is positively related to the present value of profits and is a

negative function of the cost of capital.

Imports depend on domestic final demand and their relative price vis-à-vis

domestic prices.

(c) Salaries and prices

It is assumed that firms fix their prices on the basis of a mark-up over

marginal cost. Exports depend both on domestic production costs and

prices of international competitors. There is also short run price rigidity,

modelled using an adjustment costs constraint. Nominal wages principally

depend on four factors: (1) the disincentives that arise from the provision of

social protection and the value of leisure time, (2) labour productivity, (3)

labour market pressures and (4) the general price level. Wages do not

immediately adjust to changes in economic conditions since wage

agreements are signed for a period of several years.

Interest rates and exchange rates are also treated as endogenous variables

in the QUEST model (contrary to many other models), that allows crowding

out effects to be estimated (the fact that the funding of public expenditures

generates an increase in real interest rate, and a consequent decrease of

private investment).

HERMIN: a four-sector model

The HERMIN models (HER from the macro-sectoral model HERMES from

which it was inspired and MIN to indicate the limited dimensions of this

model) cover Ireland, where the prototype was originally developed,

Portugal, Spain and Greece. The HERMIN prototype was taken from

HERMES, a system of complex models (eleven sectors, over 600

equations), spawned by trans-European collaboration conducted by DG XII

Tools for analysing data

and covering, with various degrees of success, all the Member States of the

European Union. The HERMES models were developed to analyse energy

policies and therefore focused on the role of energy as a factor of

production. They consequently included a careful simulation of production-

related processes.

The HERMIN models are small- and medium-scale (4 sectors with 100 to

150 equations each). They were developed by national laboratories in each

of the four peripheral countries in the European Union. One of the main

objectives of these models is the comparative study of the impact of

Structural Funds on the periphery of the Union. The basic structure of

HERMIN combines three types of relations: the components of supply, the

components of demand and the wage/price ratio.

(a) Supply

The HERMIN model comprises four sectors: the exchangeable goods

sector, the non-exchangeable goods sector, the agricultural sector and the

public sector. These sectors have the following distinct characteristics:

The exchangeable goods sector is subject to international competition and

its production is determined primarily by international demand (expressed

in terms of world production) and competitiveness (measured by the ratio

of the unit cost of labour to world unit costs). Prices are determined

externally in international markets.

The non-exchangeable goods sector covers services, building and civil

engineering, and community facilities. It is subject to domestic demand and

prices are determined by the profit margin on labour costs.

The production of the exchangeable goods and non-exchangeable goods

sectors is modelled in the form of a function where the value added

depends on capital and labour. As a result, the levels of employment and

investments depend on the level of production. The demand of the two

factors of production also takes into account their respective cost (under

the assumption of the elasticity of constant substitution). Technological

progress is considered to be an exogenous, time-related trend that differs

from one sector to the next.

The agricultural sector covers agriculture, forestry and fishing. It is

considered to be essentially exogenous, with prices and production quotas

defined in the framework of the European Union Common Agricultural

Policy. Principal evaluation techniques and tools

In the public sector, which includes the administration, education and

health, the level of activity depends on government decisions. Spending is

limited by the rules of budget equilibrium and debt repayment.

(b) Demand

Consumption is determined by real available personal income and by real

financial wealth. Private non-residential investments are derived from the

production function as a demand for fixed capital. Residential investments

are a function of available income per capita. Finally, public investment is

an exogenous variable. The financial sector is not modelled and the

exchange rate, as well as the domestic interest rate, are exogenous

variables.

Agricultural exports are estimated exogenously. Exports from the

exchangeable goods sector are a function of the production of this sector

and of the GNP growth rate (a reflection of the economic cycle) in the case

of Spain. Tourism and service exports are a function of world production.

Imports are determined in a residual fashion to differentiate between the

total production of the four sectors and final demand (private consumption,

public consumption, investments and exports).

(c) Wages and prices

Wages are fixed by an equation that simulates the process of wage

negotiations between employers and unions. Negotiations on nominal

wage rates are influenced by the situation of the labour market, as reflected

in the current unemployment rate or its rate of variation. They are also

subject to the influence of "fiscal leakages", i.e. the difference between

gross salary before tax, expressed in terms of production costs, and net

salary after all deductions, expressed in terms of consumer prices. Finally,

the level of production costs and that of labour productivity also affect the

process of wage negotiations. In the case of Ireland, the job market is

assumed to be homogeneous and wages in the exchangeable goods

sector dominate all the other sectors. In Spain, salaries in the non-

exchangeable goods sector are evaluated separately, but the public and

agricultural sectors are also dominated by the exchangeable goods sector.

The supply of manpower is determined differently in the various countries.

Migration plays an important part in the case of Ireland, particular into and

from the UK. In the case of Spain, a distinction is made between the

employment rate of men and of women.

Tools for making a judgement

The overall evaluation of a complex programme is characterised by the fact

that there are numerous expected effects and that these do not all have the

same degree of importance for the different partners. Some tools for

making judgements are particularly well suited to this situation.

For example, the expert panel constructs a synthetic judgement of the

programme being evaluated. This tool, when implemented with optimum

efficiency, enhances the credibility and acceptability of the evaluation

conclusions because differences between points of view are respected and

consensus is reached.

Multicriteria analysis has significant advantages in terms of the

acceptance of a multiplicity of points of view. The tool includes a

mechanism for the identification and explanation of points on which there

is consensus.

The most well known tools for making a judgement are cost-benefit

analysis and cost-effectiveness analysis. For various reasons, these

tools do not have the flexibility required for taking into account the

complexity of a judgement on an entire socio-economic programme.

Benchmarking is a more flexible tool for making judgements, which could

theoretically be applied to the overall evaluation of a complex programme.

However, to date this tool has been used only for simple evaluations.

Consequently, it is presented further on in this document.

Some tools presented above, such as those for structuring an evaluation,

may also be used to formulate a judgement. For example, SWOT analysis

may be used to judge the relevance of a strategy, just as the colour vote

may help to validate key points in the final evaluation synthesis in a context

of partnership.

Tools for making a judgement

Expert panels Θ ©

O

Expert panels bring together

several independent specialists Overall Medium Medium

evaluation technical cost

recognised in the fields covered by complexity

O o

the evaluated programme. The A

experts collectively produce a

value judgement on the programme In-depth Retrospective Prospective

and its effects. This low cost tool evaluation use use

rapidly provides an overall Θ

judgement incorporating the main

information available on the Use

programme, as well as numerous stabilised

previous and external experiences.

The panel may be considered as an evaluation tool in so far as there is a

standard and reproducible procedure for forming it, bringing it together and

leading it to produce its conclusions. Inspiration for the tool was based on

university juries - which explains why it appeared in the early 1970s - in the

field of Research and Development programme evaluation.

An "expert panel" is a specially constituted work group that meets for

evaluation. A similar tool, the Delphi survey, is described elsewhere in this

Volume. It also relies on experts but differs in several other respects.

Example: Evaluation of the quality and relevance of development programmes

in Sweden

For several years the Swedish National Council for Technical Development has be

financing programmes for the development of basic skills and knowledge. Since

1981 the quality of these programmes has been regularly evaluated.

The first step consists of selecting five international experts who are recognised and

representative of the diverse points of view concerning the programme. These

experts make an undertaking to participate in all the panel meetings. One of them is

elected as the chairperson.

In the second stage the panel examines the programme documents concerning the

resources mobilised (staff, equipment, finances), the research themes, results,

publications and co-operation. The panel interviews several addressees.

The following stage consists of visits in the field by each expert, either individually or

by groups of two.

In the final stage the panel draws up a joint report. If the points of view diverge, the

panel discusses them until a consensus is reached. Experience has shown that the


PAGINE

248

PESO

19.13 MB

AUTORE

Atreyu

PUBBLICATO

+1 anno fa


DESCRIZIONE DISPENSA

Dispensa al corso di "Metodi di valutazione di programmi e progetti" del Prof. Claudio Mazziotta. Trattasi del volume redatto dalla Commissione Europea dal titolo "Evaluating socio-economic programmes", vol. 3, all'interno del quale sono presentate le principale tecniche e strumenti di valutazione dei programmi socio-economici. In particolare, rivolge l'attenzione a 23 strumenti utilizzati per la valutazione dei fondi strutturali europei.


DETTAGLI
Corso di laurea: Corso di laurea magistrale in politiche pubbliche
SSD:
A.A.: 2011-2012

I contenuti di questa pagina costituiscono rielaborazioni personali del Publisher Atreyu di informazioni apprese con la frequenza delle lezioni di Metodi di valutazione di programmi e progetti e studio autonomo di eventuali libri di riferimento in preparazione dell'esame finale o della tesi. Non devono intendersi come materiale ufficiale dell'università Roma Tre - Uniroma3 o del prof Mazziotta Claudio.

Acquista con carta o conto PayPal

Scarica il file tutte le volte che vuoi

Paga con un conto PayPal per usufruire della garanzia Soddisfatto o rimborsato

Recensioni
Ti è piaciuto questo appunto? Valutalo!

Altri appunti di Metodi di valutazione di programmi e progetti

Valutazione dei costi e dei benefici nell’analisi dell’impatto della regolazione
Dispensa
Indicatori o analisi di performance?
Dispensa
Statistica - Indice di contingenza quadratica media
Appunto
Statistica - Esercizi
Esercitazione