Below are four different levels of outcome measurement, listed from least time-consuming and expensive to most. You should choose which level to undertake based on your available time, funding, and other considerations.
Level 1: Critical Review
Now that you have decided what your program’s target outcomes are, look back at your program activities with a critical eye. Are your components really designed to maximize progress towards these outcomes? This level of measurement—a critical review of whether your program is designed to meet your outcomes—is the most affordable and should be done frequently. As you prepare for each program cycle, think about those outcomes and ask yourself if your activities are in line with them. If you add a new component to the program, begin with these outcomes. Ask yourself how you can design the component to address the outcomes and proceed accordingly. Make sure that participating law students are aware of these program outcomes. As the direct service providers, they should surely understand what you hope the program will achieve.
Level 2: Tracking Basic Data
Set up systems to track simple program data. Keep a spreadsheet or database listing the law students who participate in the program and ways to contact them in the future in case you decide to survey them at some point down the road. Track how many high school classes you serve and how many students are enrolled in each. Track how many “contact hours” each high school class receives, along with the number of visits and topics covered. You can collect this information by asking each law student to keep a log of their visits noting length and topics. Any results from outcome measurement will be more meaningful with program data on hand. If you keep track of these things over time, it will help you to both make sense of outcome results and report about your program to stakeholders or funders.
Level 3: Specifying Indicators
Specify indicators for your chosen outcomes. What would you consider to be a successful result for each outcome? If one of your outcomes is "High school students will know more about legal careers” how do you identify success? Would you consider it a success if 50% of the students improved their scores on a basic test about legal careers? What about if the class as a whole showed an improvement of 3 points on the same test? Is a better indicator whether each student can list 5 things lawyers do or write a paragraph that clearly explains how people get to law school? Some outcomes may require more than one indicator. Indicators must be measurable, observable, and unambiguous. Terms like “substantial,” “acceptable,” and “sizeable” must be defined. What would you consider “substantial” progress to be? It is understandable that you may be hesitant to choose numerical targets without any basis for your choice. In that case, you may simply define success as an increase, improvement, or gain of any sort in the indicator. Your future targets might be to improve over previous years’ results.
Collect data for each indicator
There are several ways to collect information about the indicators you choose. You might survey or interview the high school students, interview the classroom teachers, look at test results or scores, conduct a focus group, observe classroom sessions, or collect observations and data from your law students. Developing your own data collection tools can be time consuming. Here are some measurement tools that can be modified to meet your program’s specific needs:
As you design or modify tools, do so with a careful eye towards your indicators. Make sure that the questions you ask will be clear to the participants and that answers will provide you with information you need. Avoid adding too many items simply because it would be “interesting” to see the results. Develop a plan for processing and tracking the results as you develop the instrument.
Data collection considerations
Many school districts have strict rules about conducting research in the classroom. Since you are not undertaking a peer‐reviewed, published research study, but rather evaluating a program that students are participating in, you should be able to obtain approval to survey students, observe classes, or interview teachers. Remember that the high school students are likely all minors and you will probably not be allowed to collect any identifying or personal information about the students without parental permission. Be sure to investigate the policies in your school district before surveying students, and be prepared to provide your data collection instruments for review.
Level 4: Larger Plans for the Future
At some point, consider looking for funding for a full impact study. Outcome measurement is not synonymous with impact research, as it does not employ control groups or complex statistical analyses, and therefore cannot isolate a program as the contributing factor to results. Impact research is the most expensive and time consuming form of program evaluation, but provides the most specific results.