As part of ILLUME’s mid-year report focused on the grassroots education component of ILSFA, we outlined metrics that can be used to assess the effectiveness of grassroots education outreach activities and measure progress to overarching goals, using a program theory logic model (PTLM). We suggest that the program administrator adjusts the metrics so that they align with outputs for each of the activities that grassroots educators complete, and metrics that measure progress toward stated outcomes. The table below shares the evaluation team’s recommended set of metrics, the piece of the PTLM that those metrics relate to (including inputs, activities, and outcomes). We also recommend the source for gathering the data to track metrics and the cadence for updating and reviewing the metrics.
Key
GE = grassroots educator
AV = approved vendor
PA = program administrator
EJC = environmental justice community
| Type | Input | Activity | Outcome | Metric(s) | Data Soure | Cadence |
|---|---|---|---|---|---|---|
| Input | Training about ILSFA eligibility, processes, and benefits | -Number of GE trainings offered -Number of GEs who attend trainings | Program documentation | Annually | ||
| Input | Program materials to enable outreach | -Number of outreach materials offered (including the number of types of materials, topics covered, languages available) -The program offered the GEs an opportunity to provide input on materials and incorporated suggestions | Program documentation, GE evaluation interviews | Annually | ||
| Input | Events to connect grassroots educators and AVs | -Number of GE/AV networking events hosted, and number of GE and AV attendees | Program documentation | Annually | ||
| Input | Provide financial support to GEs | -Amount of funding distributed to GEs | Program documentation | Annually | ||
| Activity | Regularly attend program trainings | -GE self-reported understanding of program offerings | Qualitative assessment based on office hours, GE end-of-year reporting, GE evaluation interviews | Annually (every 2-3 years in evaluation interviews) | ||
| Activity | Perform educational outreach to homeowners, renters, property managers, and non-profits with a focus on EJCs | -Number of community outreach events and number of attendees (in general and in EJCs) | GE reporting and Salesforce tracking | Quarterly | ||
| Activity | Develop partnerships (i.e. with other organizations serving income-eligible households, municipalities, and other grassroots educators) | -Number of partners each GE is working with -Number of events or amount of outreach delivered alongside partners | GE reporting | Annually | ||
| Activity | Share information about complementary programs (i.e. energy efficiency programs, building repairs | -Amount of outreach completed alongside complementary programs -Number of GEs that deliver information about complementary programs | GE reporting | Bi-Annually (Twice per Year) | ||
| Activity | Form relationships with AVs to support community members in finding an AV | -Number of events with an AV present -Number of outreach attendees the GE connected with an AV (estimate okay) | GE reporting and Salesforce tracking | Bi-Annually (Twice per Year) | ||
| Activity | Support participants throughout the program participation process | -Number of outreach event attendees the GE followed up with -Number of attendees that went on to participate in ILSFA (estimate okay) | GE reporting and Salesforce tracking | Bi-Annually (Twice per Year) | ||
| Activity | Collect data from outreach engagements and enter into database | -GEs are entering data into Salesforce | Salesforce tracking | Quarterly | ||
| Activity | Collect community feedback on ILSFA and provide it to program administrators | -GEs are sharing feedback on the program | GE end-of-year reports, PA feedback tracker | Annually | ||
| Activity | Educate community members about job training opportunities | -GE self-reported understanding of program offerings | GE reporting and Salesforce tracking | Bi-Annually (Twice per Year) | ||
| Short-Term Outcome | Grassroots educators have increased knowledge of the program | -GE self-reported understanding of program offerings | GE evaluation interviews | Annually | ||
| Short-Term Outcome | Communities receive accurate and more easily accessible information about ILSFA | -Sources of awareness for participants -Future potential: General ILSFA awareness -Amount of outreach across geographic areas | Participant enrollment forms (recommend adding a filed asking about source of awareness), Salesforce tracking, gen pop survey (potential future opportunity) | Annually | ||
| Short-Term Outcome | Grassroots education budget is effectively used | -Amount of funding that GEs spend -Proportion of GE spend on outreach events | Program documentation | Bi-Annually (Twice per Year) | ||
| Short-Term Outcome | Grassroots educators reach a greater array of communities who are eligible for ILSFA | -Number of outreach events in EJCs -Geographic distribution of GEs, including number of GEs located in EJCs | GE reporting and Salesforce tracking, program documentation | Annually | ||
| Short-Term Outcome | Community members have the support they need to participate in ILSFA | -GE experience of the number of community members they work with that do not participate due to participation barriers or challenges | GE reporting and Salesforce tracking, evaluation interviews | Annually (more in-depth through evaluation every 2-3 years) | ||
| Short-Term Outcome | Program administrator and IPA adapt ILSFA design and implementation based on feedback | -Elevate is analyzing grassroots educator feedback on a regular cadence to make recommendations for program improvement | Program administrator feedback tracker | Annually | ||
| Short-Term Outcome | Communities have increased knowledge about job training programs and opportunities | -GE perspective on knowledge of job training in their communities | GE end-of-year reports, GE evaluation interviews | Annually (more in-depth through evaluation every 2-3 years) | ||
| Medium-Term Outcome | Communities have increased awareness and knowledge of ILSFA | -GE perspective on the proportion of people they speak with that have heard about ILSFA -Attendees have heard about ILSFA multiple times | GE reporting and Salesforce tracking, future potential: data collection at community events | Annually (more in-depth through evaluation every 2-3 years) | ||
| Medium-Term Outcome | Communities hear about the program from multiple trusted sources | -GE perspective on the proportion of people they speak with that have heard about ILSFA | GE end-of-year reporting, evaluation | Annually (more in-depth through evaluation every 2-3 years) | ||
| Medium-Term Outcome | Potential participants face fewer barriers to participation in ILSFA | -GE experience of the number of community members they work with that do not participate due to participation barriers or challenges | GE reporting, evaluation | Annually (more in-depth through evaluation every 2-3 years) | ||
| Medium-Term Outcome | Participation in job training programs increases | -Number of job trainees in communities served by GEs | Job training data | Annually, look for trends year over year | ||
| Long-Term Outcome | Participation in ILSFA increases | -Number of participants/ projects (including the number in communities that GEs serve) | Program tracking data | Annually, look for trends year over year | ||
| Long-Term Outcome | Communities have increased trust in ILSFA | -GE perspective on trust in ILSFA among eligible communities | GE end-of-year reporting, evaluation | Annually (more in-depth through evaluation every 2-3 years) | ||
| Long-Term Outcome | Participation in EJC communities increases | -Number of participants/ projects in EJCs | Program tracking data | Annually | ||
| Long-Term Outcome | Participants report high satisfaction over time | -Satisfaction reported by participants | Participant survey or interviews | Every 2-3 Years |