The objective of our research was to determine if HERs are still a useful behavioral energy efficiency program and calculate the average energy savings for HER programs across recent evaluations. We compared energy savings across different factors such as fuel type, HER vendor, and the mode of report distribution (printed, emailed, or both) to explore if HERs may be effective under different circumstances.
Whether it’s variation in the level of detail that is provided on the HER program design or the metrics and measures of precision that are chosen to report on, nearly every evaluation was like a new code to decipher.
As we collected data on various HER program evaluations to perform this analysis, we were surprised by how much variation exists in how findings are presented in evaluations. While we found measurable trends like increasing savings over time, HER evaluation inconsistencies made it more challenging to compare different program designs.
A call for greater standardization in HER program evaluations
Below are some of the variations that we noticed during this research study:
- Evaluations used different words for savings values, such as Relative Savings, Measured Savings, Verified Savings, Net Savings Prior to Uplift, or Savings with Double Counting. These terms could refer to a variety of savings values, such as vendor savings, evaluator-verified savings, unadjusted savings, or adjusted savings. In addition, some reported “net” versus “gross” savings for the HER program, while others reported that all savings are “net” savings because of the RCT design of HER programs. We had to carefully examine each report to determine what it meant by its reported values.
- Evaluations reported measures of precision (e.g., standard deviation, standard error, and confidence intervals) inconsistently. Some only provided this for total savings, some only for per household savings, and others only for percent savings. This meant that we were not only limited by what type of savings value an evaluation reported but also by which savings value an evaluation chose to provide a precision statistic. A cohort is a group of customers who receive home energy reports starting at the same time.
- Some evaluations provided savings values at the program level, while others provided savings values at the cohort level.
- Evaluations reported the number of participants in the treatment group inconsistently, reporting either the number of participants in the treatment group when that cohort began or the number at the beginning of the evaluation program year.
- While some evaluations provided information on the HER program design, such as report format, type of messaging, customer targeting, or distribution mode (i.e., email, print, or both), other evaluations were unclear about this type of information or did not address it at all.
- Some evaluations provide savings values for each treatment group cohort every year while a few evaluations provided this information every quarter or every month.
