Evaluating Your Social Role in the Community
Determining your organization’s role in community improvements in education, civic engagement, and social integration is a more challenging task than estimating economic impact. Analyzing the role of your organization in the health and strength of the community can similarly seem daunting. Yet, the social impacts of your organization may well be more directly relevant to your mission and of greater interest to your organization’s leaders and key constituents and economic impact alone.
The techniques used to evaluate social and educational impacts are generally less standardized within the arts world than those used for evaluating economic impact. In this section we introduce some methodologies in use for evaluating social and educational impacts associated with cultural organizations.
Although long-term trends in social and educational indicators are rarely attributable to a particular cultural organization or program, they can provide important context and tell a piece of the story that may be critical to your evaluation and case-making activities. Before you embark on evaluations of impacts closely tied to your organization’s programs, it is helpful to be familiar with the data on broader trends within your community.
For educational data, an excellent national source is www.schoolmatters.com. On this site, you can search for particular schools or districts to analyze enrollment levels and performance on standardized tests, as well as trends based on up to five years of data for certain indicators.
Perhaps the most comprehensive site for obtaining social and educational indicators for communities all across the country is www.dataplace.org, a free nonprofit resource developed by KnowledgePlex. Dataplace is populated with a wide range of useful census data and much more, especially in the areas of housing and community development. A noteworthy feature is the “My Dataplace” button in the upper right corner of the screen, which allows you to select indicators on multiple locations and compare them side by side, and also compare data over time. Much of the data only exist in 1990 Census and 2000 Census form, although certain indicators such as county and zip code employment data and mortgage lending statistics are often available for five or six consecutive years.Dataplace also allows you to upload datasets of your own and use relatively simple GIS tools to geocode and map any geographically-based data. This tool is still in beta-testing mode, but it seems to function well for many simple mapping exercises.
Many arts administrators would like to be able to better illustrate the composition of their audiences because they define their social impact, in part, as delivering important cultural and educational experiences to members of diverse and/or disadvantaged communities.
If you have street address or zip code data on a meaningful portion of your general visitation or participants in a specific program, you can use several methods to analyze and show the geographic dispersion of your audience, and the economic and ethnic characteristics of the communities from which your audience is drawn.
Some basic zip code analysis can be done relatively easily by your own staff simply using an EXCEL spreadsheet of visitor addresses or zip codes with tools that are freely available online; this will give you a taste for the kind of information you may be able to extract if you invest funds and/or time in more complex analyses. You can work with zip code data in the form of an EXCEL file mailing list with one address per row, or simply a running list of visitor zip codes in and EXCEL spreadsheet. The advantage of zip code data in this form is that it is easy to collect and store, and can be used for analysis both within Excel and in other programs such as GIS programs.
Geographic information systems (GIS) are software tools that allows you to collect, edit and display geographically-referenced information. GIS has myriad uses (for detailed background, visit http://en.wikipedia.org/wiki/Geographic_information_system ), but for the present discussion, GIS tools can be considered a method for analyzing the geo-demographic characteristics of your visitors.
The simplest GIS techniques, using no more than a list of visitors with their zip codes, can produce interesting maps of the relative concentration of visitors by zip code, such as that seen below representing partial-year data of the number of parties visiting the Queens Museum of Art in New York City.
The mapping of visitor concentrations by zip code can sometimes reveal pockets of visitor engagement that might otherwise go unnoticed, such as the one red zip code in Brooklyn that is located some distance away from the other red zip codes directly adjacent to the museum. Even a simple map such as this one may have more power of persuasion than a chart or graph based on the same data, or at least can complement other forms of data presentation.
The map above was created using ArcGIS, a well-known GIS software product from ESRI. But a similar map can be created without GIS mapping expertise by using Dataplace . Detailed instructions for uploading a spreadsheet to Dataplace are available here at the site. For example, you might want to upload a spreadsheet that consists simply of a list of zip codes and the number of individuals or parties originating from that zip code. In the case of the Queens Museum of Art, it looked like this (but with 987 rows of zip codes in the worksheet):
When uploading zip code data, the system won’t be able to complete the process until you correctly classify the data. After a few simple steps you will get to a screen that looks like this:
Before you click on the button “Generate indicators,” you should click on the word “Address” in the column titled “Purpose.” This will bring you to a screen that looks like this:
You need to de-select the box titled “Part of Street Address” and select the box “Part of Region” instead. Then click the “Update” button, which should remove the “ERROR” message from the status box at the right of the screen. Then you are ready to click on the button “Generate indicators,” and after the data have processed (which may take several minutes for a significant amount of data), you will be taken to screen that looks like this:
The “Map” and “Rankings” options are both interesting ways to view your data. Unless you want to share your data with other users of Dataplace, you should not click on “Publish this dataset.”
The resulting map of zip code data from the Queens Museum of Art, as viewed in Dataplace, looks like this:
Dataplace allows you to zoom in or out, and also click on a particular area of the map to view the exact number of visitors associated with that zip code. If you have full street address information within a dataset of visitors or members or donors, you can also upload that list to Dataplace to map and view the block-by-block geographic distribution of those visitors instead of the concentration of visitors within a particular zip code.
As of September 2007, the beta version of this tool was experiencing some “bugs” and other limitations. Most significantly, the tool was not yet able to recognize and geocode zip codes starting with the number zero (sorry New England and New Jersey). That explains why the above map of visitors to the Queens Museum of Art shows no visitors from zip codes in New Jersey or Connecticut. There are also fewer options in how to present the data than you would find with a traditional GIS program, so the Dataplace maps you produce may not be able to portray your data in the most compelling format possible. Likewise, you cannot produce maps that simultaneously display data from an uploaded dataset of visitors and also Census data associated with the zip codes or Census tracts from which those visitors originate, which is a powerful feature of more sophisticated GIS tools. But the developers of Dataplace hope to address those limitations in future versions of the program, and even in its current form, the Dataplace uploading and mapping tools form an exciting resource in that it is free and is much simpler to use than most GIS programs.
Using advanced GIS techniques to analyze visitor data
While a free tool like Dataplace can be used for simple GIS analysis of visitors to your organization, more sophisticated GIS techniques can be utilized to create more customized maps and to overlay participant data with census data to observe both the concentration of participants geographically and various demographic characteristics of neighborhoods from which participants are drawn.
Most GIS applications, which tend to have strong visual appeal, can be especially powerful advocacy and evaluation tools if the data tell a compelling story. At a high level of sophistication, GIS techniques can provide exciting insights about the way a cultural organization interacts with its communities. Mapping Cultural Participation in Chicago http://culturalpolicy.uchicago.edu/mcpic/, a project of the Cultural Policy Center at the University of Chicago, looks at visitor and donor data from more than 60 cultural organizations in Chicago combined with Census data on socioeconomic status, race and ethnicity, creating a rich topography of cultural participation in the city, neighborhood by neighborhood and organization by organization, as in the map below. We recommend reading the Cultural Policy Center’s report to contextualize the map we have reproduced here and comparing it with other maps and analyses produced through the research study.
Full-blown GIS capabilities, even basic techniques, are not typically within the skill-set of cultural professionals. However, most colleges and universities teach GIS skills, and there may be undergraduate and graduate students who might be interested in undertaking a real-world analysis using your visitor addresses/zip codes as a class project or to refine their skills. You should consider contacting faculty in the geography, geoscience and/or economics department at a nearby college or university to inquire about GIS training offered.
In addition to mapping visitor addresses, you can also collect and analyze survey responses from visitors and program participants.
For many cultural advocacy and planning purposes, the primary “analysis” of survey data is simply tabulating the responses and presenting them in chart, graph and/or map form, along with narrative explanation. If you gathered the survey responses in an Excel spreadsheet, you can use Excel’s chart wizard to create a display format for various results. A simple Step by Step Excel Chart Tutorial is available.
A useful analysis is one that is easily understood at a glance and highlights any results that may be contrary to accepted wisdom. A good write-up of survey results will be reviewed by many constituents (board members, staff, and institutional funders, for example) who may have uneven knowledge of how the survey was conducted, so it is recommended to include a copy of the survey instrument along with some basic facts about when and how the survey was administered. For most purposes, the write-up need not take the form of a detailed paper, but should instead explain key information and be formatted so as to make it simple to find quickly find information.
Often the primary goals of a particular program or organization are not economic, nor can their success be evaluated effectively on the basis of participation levels alone. In other words, you may wish to measure not only that you engaged a diverse or targeted audience, but also that your program had significant impact on the knowledge or attitudes of that audience.
Surveys, if carefully designed, can provide quantitative evaluation of changes in learning or attitude within a given audience. Surveys can also be amazing time-wasters, so good forethought is essential to make sure that you seek a reasonable amount of information and focus on data that you will actually find useful.
Generally speaking, we recommend that your survey fit on a single page (or less). Take, for example, a survey project by Real Art Ways (RAW), a multidisciplinary organization in Hartford, Connecticut, with an annual budget of roughly $1 million. RAW received a grant to bring area middle school and high school students to RAW’s cinema to view the documentary film, “Mighty Times, the Legacy of Rosa Parks,” with facilitated discussion on personal empowerment and civic leadership. Real Art Ways wanted to develop a questionnaire for students to fill out at the end of each visit to help evaluate the success of the program in meeting its goals and encourage further reflection by the students on the messages in the film. RAW hoped the results would help them strengthen the program in the future and encourage continued support from the project’s funder, which it did. Here is a copy of the survey instrument
Mighty Times: the Legacy of Rosa Parks, is a powerful film, and Real Art Ways (RAW) is an engaging place for youth to visit, so the responses were predictably positive in general, but RAW staff were surprised to receive such thoughtful and useful feedback to the open-ended questions. The “data” that result from open-ended questions can be difficult to organize and quantify, but capturing personal voices can be extremely valuable and should be considered as a possible component of your next survey. Sample quotes can be compiled into a document and grouped under thematic headings to give readers a sense of the collective response to a particular program. RAW staff used a compilation of student quotes to help make the case for the program.
Children’s Discovery Museum (CDM) of San Jose, California, is another organization that has used surveying, along with focus groups and interviews, to assess impacts on learning associated with a particular arts program. Their report [PDF] contains detailed information on their methodology and findings and serves as an example of how to present a comprehensive summary of the results of an intensive evaluation effort, including an Executive Summary that captures the main findings CDM wanted to communicate to its funders and other key constituents.
Researchers can examine the extent to which a particular arts organization has developed linkages to other nonprofits in the community through the affiliations of board members, staff and volunteers, and can measure how those linkages change over time (in response to particular initiatives, for example).
Visually, a social network is a series of nodes or dots, connected by a set of links or lines. A social network can be graphed whenever you can articulate what the nodes represent and what the links connecting them represent. In work with arts organizations, the nodes are typically organizations. The organizations are usually linked by the action of individuals – individuals who are employed by the organization, or are on the board, volunteer at the organization, or work on one of its initiatives.
The first step of social network analysis is data collection. The example that follows focuses on building a social network database of links between organizations made by staff, board members and volunteers of organizations in the community. The database, an Excel spreadsheet, contains the names of staff, board members and volunteers of various organizations, and the analysis is performed with software that identifies links between organizations created by individuals who are affiliated with multiple organizations – perhaps serving on the staff of one organization, on the board of other organizations, and as a volunteer at still other organizations.
The necessary data can be obtained in one of two ways:
Along slightly different lines, the Social Impacts of the Arts Project at the University of Pennsylvania has produced a study titled “Artists and their Social Networks”. It is based on detailed record-keeping by a large cohort of Philadelphia artists that allowed the researchers to observe, among other things, how often particular types of artists turn to other artists and art service providers for support in advancing their careers.
|Site © 2007 Zack Sheppard and the Center for Creative Community Development|