Drawing on statistical methods from biomedical research, ILLUME applied meta-analysis techniques to measure savings from home energy report (HER) programs to understand if HERs continue to be an effective source of energy savings for residential utility customers.
The Challenge
Home energy reports (HERs) are a key part of utility behavioral energy efficiency programs, but many program administrators and industry experts are asking: are HERs still effective at getting the attention of customers and influencing them to save energy? Further, are there different levels of efficacy of HERs across factors such as fuel type, vendor, cohort maturity, and distribution mode?1A cohort is a group of customers who receive home energy reports starting at the same time.,2Cohort maturity refers to how long the group of customers have been receiving the home energy reports for (e.g., 1 year). Oracle Energy and Water, was interested in taking a systematic look at recent HER evaluations to find out.
The Approach
A meta-analysis systematically and statistically combines results from multiple studies to generate a weighted average for a particular metric. This provides an estimate of the metric across multiple studies and accounts for differences in sample size, study approach, treatment effect size, and sensitivity to inclusion in the study. This is a useful approach to assess multiple home energy report evaluations because it provides a systematic and statistically precise methodology to determine the energy savings that home energy reports generate across evaluations and how various factors impact energy savings. To complete this meta-analysis, ILLUME:
Identified relevant studies: We collected publicly available HER evaluations from sources such as HER vendors, utility web sites, state evaluation websites, and general internet searches. Due to differences in market share, longevity of programs, and state-level regulations on publishing evaluation reports, most of the studies we found were evaluations of Opower HER programs.
Screened studies: Using consistent criteria, we screened documents for inclusion in the study. We excluded those that did not describe the control group, did not report the number of treatment group participants, programs evaluated before 2018, or did not report unadjusted savings at the HER cohort level. These exclusion criteria were necessary to ensure that we focused on the efficacy of recent HERs and to ensure that we had the minimum data necessary to perform the meta-analysis on the included studies.
Screened cohorts: We screened the cohorts within the included evaluations and excluded cohorts that had received home energy reports for less than ten months or greater than five years. This allowed for greater consistency of experience with HERs among the cohorts in our study.
Collected data: From each evaluation we gathered data on information such as characteristics of the utility program, cohort maturity, program treatment experience, number of participants in the treatment and control group, and the unadjusted savings.
Analyzed data: We calculated weights for each cohort’s savings estimate based on the precision of their estimate. Each cohort’s weight is inversely proportional to the precision of their savings estimate. We used these weights to calculate a weighted mean, for electric and gas saving metrics.
The Results
HERs are still effective at influencing customer behavior to save energy.
For electric customers, HERs save 88 kWh on average annually per household and 1.16% percent of average household electricity usage. For gas customers, HERs save on average 4.4 Therms annually per household and 0.87% percent of average household gas usage. We found variation in the average savings across vendors, but the differences are not statistically significant. The figure below shows the variation in electric savings by vendor with the horizontal line showing the average 88 kWh savings across all vendors and cohorts.
As cohort maturity increases so does electric savings.
As customers receive home energy reports for more years, their average annual electric savings increases. We found a correlation (Pearson’s r) of 0.48 between years of receiving reports and electric savings We found a positive, but weaker relationship (Pearson’s r=0.19) between years of receiving reports and gas savings.3https://en.wikipedia.org/wiki/Pearson_correlation_coefficient
Percent savings goal achieved varies across HER vendors
We found differences across vendors in the percentage of savings goal that the program achieved, with some vendors saving as little as 32% of the program goal and others saving as much as 120% of the program goal for electric savings.4For this analysis, we relaxed most exclusion criteria, keeping all studies except for evaluations prior to 2018. We were not able to calculate confidence intervals because precision estimates were not given for these values. This means that these findings suggest differences, but are not conclusive. Similarly, there is variation in savings goal achieved across vendors for gas savings. However, due to differences in how goals are set, limitations caused by small number of cohorts for some vendors, and the lack of precision estimates reported for this metric, these findings should be interpreted carefully. We should move cautiously into digital-only approaches.
Most of the cohorts we were able to include in our analysis received both paper and email reports with a smaller number receiving only paper or only email, We compared savings by report type and found lower savings for email-only programs (54 kWh) compared to paper only (118 kWh) and paper and email (107 kWh). The differences between the two groups that received paper are not statically significant. We found a similar pattern for gas programs. We caution that the comparison is based on a small number of cohorts in the email-only group and urge the industry to do more research on impacts by report type.
Inconsistencies in HER evaluations hinders analysis
There is wide variation in the way evaluations report savings for home energy reports. There is considerable variation in naming conventions for different types of savings, which types of savings are reported, and how they are reported (such as level of granularity and whether a precision estimate is included). These inconsistencies limited the data we could use to a subset of available reports and cohorts.
The Takeaway
Overall, we found that home energy reports continue to save energy in recent years. They are still a helpful and effective tool to influence customer behavior and the longer a customer receives them, the more energy they are likely to save.
A primary finding from this study is the inconsistency in reporting conventions across home energy report evaluations. More consistency across evaluation reports would enable deeper analyses across more factors and could substantially improve our understanding of home energy reports and their role in residential energy efficiency.
ILLUME published a white paper that was presented at the ACEEE Summer Study 2022 Conference.
You can read the white paper here.
