There are many ways to evaluate program successes, mark progress, and identify challenges. This section provides program coordinators with ways to measure progress toward program outcomes. It provides a variety of options for assessing your program, including techniques that don’t require a lot of time, effort, or money. Even just identifying and prioritizing your program outcomes will help you critically review your program, improve it, and make smart decisions in the future.
If you are interested in pursuing a more in-depth program evaluation, know that it can be expensive and time consuming and require outside expertise. We recommend involving your university’s education department to discuss an evaluation of your program as a possible graduate student project or to get assistance in developing different components.
- Identifying Program Outcomes
- Decide Which Outcomes to Measure
- Finding Ways to Measure Your Outcomes
Identifying Program Outcomes
Simply identifying and measuring program outcomes is one way to evaluate. An outcome is a change or impact for participants, during or after a program. An outcome can be positive or negative, intended, or unintended.
The first step in measuring outcomes is to identify outcomes for your program. In choosing your program outcomes, you will acknowledge the stated and unstated objectives of the program and think about what matters to various stakeholders.
To begin, brainstorm a list of program outcomes. Your list should include outcomes for various participants and consider the perspectives of different stakeholders. For example, a brainstormed list of program outcomes for a program that supplies translation services at a health clinic might include the following:
- Patients can state their doctor's instructions
- Patients follow their doctor's instructions
- Patients take their medications
- Patients have a decrease in drug complications
- Doctors increase their understanding of other cultures
- Nurses pick up phrases in other languages
The following are not outcomes: the program provides translation services for 50 patients, ten people volunteer to be translators. They are inputs. Remember, an outcome is a change or impact.
Need help generating your list? Street Law, Inc. asked Law School Program Coordinators to share potential program outcomes.
High School Students Will:
- Know more about substantive areas of the law
- Improve their writing skills
- Improve their ability to advocate
- Have higher self-efficacy
- Be more engaged in learning
- Have increased respect for the law
- Understand reasoning behind laws
- Be less likely to engage in delinquent behavior
- Become better problem solvers
- Have increased civic values
- Understand the importance of the law
- Increase collaborative skills
- Consider attending college
- Consider pursuing legal careers
- Consider becoming a lawyer
- See themselves as potential lawyers
- Understand the roles of lawyers in civic leadership
Law Students Will:
- Improve lawyering skills beyond cognitive ones
- Increase their empathy for their students
- Have a stronger self concept
- Improve community connections
- Learn substantive law
- Appreciate the importance of pro bono
- Increase their likelihood to participate in law-related education in the future
- Appreciate the needs of low-income people
- Improve their organizational skills
- Improve their presentation skills
- Increase their participation in the law school community
- Improve their oral communication with lay people
The Law School Will:
- Receive good PR
- Have an increased awareness of the need for pro bono
- Improve community networking
- Increase involvement of alumni
- Increase financial support of law school
- Better meet accreditation standards
The High School Will:
- Have raised expectations of the students' capabilities
- Teachers will increase knowledge of legal information
- Experience increased student motivation
- Experience increased student interest
- Receive good PR
- Come closer to meeting course/teacher objectives
You may want to involve key program stakeholders in the brainstorming process. Ask on-site teachers/site coordinators what they hope students get from the program. Ask participating law students what they see as the outcomes.
Decide Which Outcomes to Measure
After a complete brainstorming session where you've written down all of your potential program outcomes, you should decide which outcomes are worth measuring.
Weed out any duplicative, overlapping, or clearly unimportant outcomes.
Apply the following three criteria to the remaining outcomes (The following worksheet can help facilitate this process: Worksheet: Criteria for Choosing Outcomes)
- Test #1: Is it reasonable to believe that our program can influence this outcome in a meaningful way?
- Example: It may not be reasonable to believe that your Street Law program will keep students from ever committing a crime. It may be reasonable to believe, however, that students will have more positive interactions with law enforcement
- Test #2: Would measurement of the outcome help identify program successes and address problems or shortcomings?
- Example: While it might be interesting to know whether students' reading skills improve after the program, if that information will not help you improve the program, then you shouldn't spend time measuring it.
- Test #3: Does the outcome represent meaningful benefits to or consequences for the participants?
- Example: One of your outcomes might be that law students will have a better understanding of the workings of your local public schools. If that increased understanding isn't a meaningful program benefit, then don't worry about trying to measure it.
Of the remaining outcomes, put a star next to those that you feel it would be particularly beneficial for the program to measure. Ideally, you should narrow your list to about five outcomes.
Finding Ways to Measure Your Outcomes
Below are four different levels of outcome measurement, listed from least time-consuming and expensive to most. You should choose which level to undertake based on your available time, funding, and other considerations.
Level 1: Critical Review
Now that you have decided what your program's target outcomes are, look back at your program activities with a critical eye. Are your components really designed to maximize progress towards these outcomes? This level of measurement--a critical review of whether your program is designed to meet your outcomes--is the most affordable and should be done frequently. As you prepare for each program cycle, think about those outcomes and ask yourself if your activities are in line with them. If you add a new component to the program, begin with these outcomes. Ask yourself how you can design the component to address the outcomes and proceed accordingly. Make sure that participating law students are aware of these program outcomes. As the direct service providers, they should surely understand what you hope the program will achieve.
Level 2: Tracking Basic Data
Set up systems to track simple program data. Keep a spreadsheet or database listing the law students who participate in the program and ways to contact them in the future in case you decide to survey them at some point down the road. Track how many high school classes you serve and how many students are enrolled in each. Track how many "contact hours" each high school class receives, along with the number of visits and topics covered. You can collect this information by asking each law student to keep a log of their visits noting length and topics. Any results from outcome measurement will be more meaningful with program data on hand. If you keep track of these things over time, it will help you to both make sense of outcome results and report about your program to stakeholders or funders.
Level 3: Specifying Indicators
Specify indicators for your chosen outcomes. What would you consider to be a successful result for each outcome? If one of your outcomes is "High school students will know more about legal careers" how do you identify success? Would you consider it a success if 50% of the students improved their scores on a basic test about legal careers? What about if the class as a whole showed an improvement of 3 points on the same test? Is a better indicator whether each student can list 5 things lawyers do or write a paragraph that clearly explains how people get to law school? Some outcomes may require more than one indicator. Indicators must be measurable, observable, and unambiguous. Terms like "substantial," "acceptable," and "sizeable" must be defined. What would you consider "substantial" progress to be? It is understandable that you may be hesitant to choose numerical targets without any basis for your choice. In that case, you may simply define success as an increase, improvement, or gain of any sort in the indicator. Your future targets might be to improve over previous years' results.
Collect Data for Each Indicator: There are several ways to collect information about the indicators you choose. You might survey or interview the high school students, interview the classroom teachers, look at test results or scores, conduct a focus group, observe classroom sessions, or collect observations and data from your law students. Developing your own data collection tools can be time consuming. Here are some measurement tools that can be modified to meet your program's specific needs:
As you design or modify tools, do so with a careful eye towards your indicators. Make sure that the questions you ask will be clear to the participants and that answers will provide you with information you need. Avoid adding too many items simply because it would be "interesting" to see the results. Develop a plan for processing and tracking the results as you develop the instrument.
Data Collection Considerations: Many school districts have strict rules about conducting research in the classroom. Since you are not undertaking a peer‐reviewed, published research study, but rather evaluating a program that students are participating in, you should be able to obtain approval to survey students, observe classes, or interview teachers. Remember that the high school students are likely all minors and you will probably not be allowed to collect any identifying or personal information about the students without parental permission. Be sure to investigate the policies in your school district before surveying students, and be prepared to provide your data collection instruments for review.
Level 4: Larger Plans for the Future
At some point, consider looking for funding for a full impact study. Outcome measurement is not synonymous with impact research, as it does not employ control groups or complex statistical analyses, and therefore cannot isolate a program as the contributing factor to results. Impact research is the most expensive and time consuming form of program evaluation, but provides the most specific results.