Section 1
Conceptual Foundations of Evaluation Models

- Programme information
- Evaluation models
- Conclusion

Programme information

Programme evaluation is concerned with making value-based judgments about a programme(Scriven, 1991). These judgments may take a number of forms including decisions on programmechanges, expansion, termination and implementation in other settings, and judgments about cost-effectiveness and efficiency. Their focus might be on the programme overall, or particular aspects of it such as its implementation or its impact.

Making soundly based judgments clearly requires information that is relevant and important to the judgments and decisions to be made. This typically would include information on the social and physical context of the programme and how it achieves its effects, as well as information on its outcomes (Pawson and Tilley, 1994).

When planning and conducting an evaluation study, it is important to take account of the likely uses of the information provided by the study. This will help to ensure that the scope and quality of the programme information are appropriate to the nature and significance of the judgments and decisions to be made.

However, having information about a community crime prevention programme is not sufficient for making sound judgments and decisions about it. The application of relevant standards and criteria considered important for making these judgments also is required.

Relevance of information to stakeholders

The standards and criteria to be used in making judgments and decisions about a programme, including relevant benchmarks, can be derived from a variety of sources including:

• programme objectives
• government or agency policy
• widely held community values
• ethical standards
• the performance of other programs with similar objectives
• the current or previous situation in the local community or in other comparable communities.

The relative importance of various standards and criteria for evaluating a programme will, however, almost always differ among individuals and groups and their relevance will depend partly on the main purposes of the evaluation study (Owen and Rogers, 1999).

To ensure that the information produced by an evaluation study will be useful to the people making judgments and decisions about a community crime prevention programme (usually the major stakeholders), their information needs and priorities must be determined and taken into account in the planning, design and conduct of the study. Clearly, this requires that they be consulted about the purposes of the study, the critical variables to be measured and the information to be obtained, as well as the standards and criteria of importance to them.

Evaluation models

There are several models which might be used to evaluate community crime prevention programs. These models guide decisions on the specific evaluation questions and procedures to be used in an evaluation study (Posavac and Carey, 1997). Each model represents a different combination of:

• evaluation form or purpose (Owen,1993; Owen and Rogers, 1999)
• evaluation approach (for example goal-based, decision-oriented)
• evaluation methods.

An appropriate evaluation model should also allow currently accepted standards of evaluation practice to be met, such as those outlined in the Programme Evaluation Standards (Joint Committee on Standards for Educational Evaluation, 1994). These standards are concerned with the utility of the information, the feasibility of collecting it, its technical adequacy and ethical practice in the conduct of an evaluation study.

Choosing an evaluation model

A key touchstone of any evaluation study is the extent to which the information it provides is useful and is used by the main stakeholders in the programme and other audiences of the information, i.e. the potential users of the information. Thus, the models chosen for evaluating community crime prevention programmess, and the way they are applied, need to enhance the appropriate and effective use of the information.

Information from evaluation studies is most likely to be used when the choice of evaluation model takes into account the nature of the programme and its specific characteristics. Community crime prevention programmes differ on a number of important dimensions which have significant implications for evaluation studies, in particular:

• the social, cultural and political context surrounding the programme, including the nature of the community and local support for the programme
• the degree of community participation and control
• the particular types of crime which are of major concern
• whether the programme emphasises a social or opportunity reduction/situational approach to crime prevention (O?Malley and Sutton, 1997)
• the assumed causal mechanisms that provide the rationale for the design of the programme (Pawson and Tilley, 1994).

Other factors which need to be taken into account are how these dimensions are thought to influence programme outcomes, particularly across programme sites, target groups and the various needs and problems being addressed.

Programme characteristics
Figure 1 indicates that the initial considerations in the choice of an evaluation model are the characteristics of the community crime prevention programme itself. These include the:

• prevention approach, that is the presumed underlying causal mechanisms and programme rationale that provided the basis for the design of the programme
• programme type, as reflected in its main purpose and associated objectives (for example public education or the provision of a service)
• programme specification, encompassing the specific programme setting, target group and need being addressed by the programme.

Figure 1 also indicates that these characteristics can only be properly understood by taking account of the social, cultural and political context surrounding the programme.

Figure 1: Considerations in the choice of an evaluation model

Social context of evaluation studies

Community crime prevention programs are designed to address only some of the broad range of community needs and concerns, and so must compete for resources with other social programmes and initiatives through the normal political processes of local communities (O'Malley, 1997). Evaluation studies of crime prevention programs also consume resources, of course, and so they too are competing for the limited resources available and must be justified by the value of the information which they provide.

The reasons for the lack of evaluation of community crime prevention programs have not been studied extensively, but social, political and financial considerations are likely to have a strong influence. There are also several other relevant factors including the limited knowledge and experience of evaluation theory and practice of many programme managers and organisers. In addition, evaluation evidence is often seen as 'bad news' since programme objectives tend to be over-optimistic and hence are rarely fully met: a situation that evaluation might expose.

Those responsible for commissioning or conducting evaluation studies need to recognise and take account of both these factors and the local social, cultural and political context if the evaluations are to produce evidence that is not only useful, but used. This point was made by Guttentag and Struening (1975) when they stated that:

Weiss (1993) has reinforced this point, suggesting that being aware of the political context of a programme 'is a precondition for useable evaluation research' (1993: 94), an assessment based on her extensive experience in a senior evaluation role in a major US government agency.

Conclusion

It will be apparent that the choice of an evaluation model, including the methods to be used, should be influenced by a number of considerations and should flow from an assessment of them. This view is in contrast to the impression given by a number of authors that experimental or strong quasi-experimental designs and quantitative measures are the most appropriate methods, or have greater merit than other methods for evaluating crime prevention programs (Sherman et al, 1997; Hagan, 1997; Davis and Taylor, 1997).

Programme evaluation, however, essentially is concerned with the development of a defensible argument about a programme, supported by sound evidence about:

• what the programme does and in what context
• its effects
• why the effects eventuate
• the extent to which the effects can be applied generally and repeated in other settings and populations.

Obtaining information about these aspects which is relevant to the evaluative argument and of importance to stakeholders is thus of primary concern in evaluation studies (Cronbach, 1982; Pawson and Tilley, 1994; Tilley, 1996).

The choice of evaluation models, then, and the methods to be used in particular, should depend on the value of the information which different methods are likely to produce, as determined by its contribution to the evaluative argument. On occasion, this might mean that only one type of method is used, producing only quantitative or only qualitative data, but most usually mixed methods yielding both qualitative and quantitative data will provide the most valuable and broadly useful information.