Determining What to Evaluate

Economic and social impact evaluation efforts require a lot of work by staff, even when conducted with the assistance of professional researchers.  It is critical to begin the process by clarifying what you wish to evaluate and by acknowledging what you will not address in your evaluation.

In this section we recommend a set of steps to take before engaging in a large evaluation project to ensure the greatest chance for success. Your ultimate goal is to have a set of results that are informative and useful to your organization.

Jump To:

Forming Evaluation Steering Committee

Assessing Data Availability

Setting Evaluation Objectives

Soliciting Bids for Paid Evaluation Services


Forming Evaluation Steering Committee

Whether your impact evaluation project involves just your own organization or a collaboration, the nature of the project should be defined with input from a range of stakeholders.  Typically the Executive Director of the cultural organization or cultural association initiating the study has certain purposes in mind for the effort, but projects are likely to stall or produce results with limited utility if they are too closely tied to a single individual.

We recommend that both staff and board members be represented on the evaluation steering committee, and if possible include at least one friendly university faculty member with experience in economic or social science research. The latter need not be a potential professional consultant for the project, but rather an individual who can provide a useful grounding in research practice.  A group of roughly 6-8 people is a manageable size to provide diverse perspectives among the leadership team for the evaluation project. 

Some organizations or partnerships have already chosen a research consultant to work with before the project scope has been defined, which can help provide focus and direction to the priority-setting process based on the experience and interests of that particular consultant.  However, you should also be aware that your choice of consultants has a major influence on what you will end up evaluating and what approaches you will use. 

Ideally, your evaluation steering committee would possess or retain the necessary competence to assess the available data and get to know the range of methodologies available to work with that data before selecting any outside consultants.  If consultants are deemed necessary, you would then solicit bids from consultants based on a generally defined scope of work and you would ultimately shape a workable marriage between project objectives, available consulting or in-house expertise, available data, and the financial resources allocated for the project.

You should expect that the first one or two meetings of the steering committee will be focused on developing shared objectives for the proposed evaluation project by gathering ideas from the different members on what sort of analysis is most desired and potentially useful, assessing data availability, assessing potential partnerships (with college faculty, professional consultants, funding partners, etc.), and determining a budget for the project. 


Assessing Data Availability

Data assessment is an excellent task to be divided among several steering committee members to be completed between meetings.  In the case of an organization that doesn’t currently collect visitor data, for example, one person (perhaps a box office manager) could be assigned to write up one or more scenarios for how visitor data collection could be effectively incorporated into regular operations on a temporary or permanent basis. 

A clear division of labor will encourage more members of the steering committee to think seriously about the practical logistics of the various evaluation methodologies the committee may be considering. The ultimate shape of your impact evaluation will depend to a large extent on the availability of data necessary to conduct robust analyses using various methodologies. 


Setting Evaluation Objectives

Once you have explored potential partnerships and have generally determined the availability of relevant data, your evaluation steering committee should make a list of evaluation objectives. With these objectives in mind, the committee should identify additional data required to complete the evaluation. Financial resources and staff time necessary to collect additional data and complete the evaluation should be identified. 

The resulting document will constitute a potential scope of work. At this point you are ready to consider soliciting bids from consulting researchers or evaluators for any aspects of the desired project scope that would require expert assistance outside of individuals you have already identified. 

Many basic economic and social impact evaluation projects can be carried out without any outside assistance, but there are occasions when paid services are valuable and worthwhile. Our website is designed to give you the insight necessary to weigh the cost/benefit trade-off of utilizing paid evaluation consulting.

After you have identified your full range of options for in-house, pro bono and paid evaluation services for each of the items on your initial scope of work, you can make decisions about how to allocate your project budget so as to get the most useful evaluation possible using scarce funds.  There are no hard-and-fast rules for such decision-making. You must do a reasonable amount of preparation, and then allocate available resources to best meet your evaluation needs.


Soliciting Bids for Paid Evaluation Services

When contacting any prospective consultant, you should be prepared to send them a copy of your potential scope of work indicating the tasks you expect to handle in-house as well as the evaluation services for which you are seeking assistance.  This will show them that you are a serious organization that understands the evaluation process.

You are likely to be asked about the budget for your evaluation project.  It is a good idea to indicate the upper limit that you can envision spending for the evaluation project so that you don’t waste their time and your own with bids that are not remotely within your price range. You should, however, also indicate that the quoted figure includes resources for in-house staff support related to the evaluation project and the possible use of paid consultants for other aspects of your evaluation project in addition to those elements for which you are currently seeking bids.  That will discourage bidders from simply devising a list of services that will justify the full amount of money budgeted. Rather, they will have to think carefully about how to offer you the maximum value for a cost that will fit within your budget and address the needs identified in your scope of work.


Site © 2007 Zack Sheppard and the Center for Creative Community Development