Predicting Community Health Center Power and Cold-Chain Readiness in California

Establishing ground truth is the first step in preparing for and responding to disaster. Health systems throughout the world find themselves unable to verify or accurately estimate ground truth in real-time. California is no different. Direct Relief and the California Primary Care Association principally support community health centers in California and have a shared interest in determining the degree to which these health centers are prepared for the increasing threats posed by wildfires and power outages.

While California’s health centers have adapted and become more resilient to “wildfire season,” over the past few years, California’s crises have increased to include record-breaking heat waves, rolling blackouts, historic wildfires and poor air quality. The California Primary Care Association reached out to health centers in all fire-prone areas and has learned that many had to close operations due to evacuation orders, poor air quality and power outages.

In October 2019, California’s largest electric utility shut down its power grid in 34 of California’s 58 counties, leaving an estimated 2 million people without power and some customers cut off for up to a week. To CPCA’s knowledge, there were approximately 180 health center organizations with over 500 sites in these counties where power was shut off or was scheduled to be shut off. Losing access to power has become a recurring challenge for health centers as California’s utility companies plan to continue intentional power shutdowns during periods of extreme fire danger.

California’s power outages have forced the cancellation of thousands of patient visits at community health centers due to closure. Community health centers that lost power, but were still able to open, found themselves crippled by the loss of power, forced to slash services, close units like dental clinics, and attempt to operate without the computer systems that are the backbone of modern healthcare. Even if a health center stays in operation without power, without electronic health records, the doctors can’t access lab results, records of current prescriptions, schedules for screening tests like mammograms, records of blood pressure and cholesterol level, or reports from specialists.

Based on the information gathered to date, health centers with generators were able to keep their doors open to serve patients as usual. However, it is not yet clear how many health centers experienced challenges with storing medications that require refrigeration, pharmacy refills, or accommodating patients in need of medical services during the Public Safety Power Shutoff (PSPS). Loss of access to health care, including medications, is one of the most least-understood risks from natural or human-caused disasters.

California’s widespread power shutdowns have revealed a hidden weakness in our health care safety net, and PSPS events are likely to become more commonplace as California’s utility companies seek to prevent destructive wildfires during periods of extreme risk.

DEPLOYMENT

In January 2021, Direct Relief and the California Primary Care Association (CPCA) initiated an engagement with Macro-Eyes (now Pendulum) to apply machine learning to understand individual health facility and health safety network capacity and readiness for managing Covid-19 vaccinations for the communities they serve – specifically focused on federally qualified health centers (FQHC) and look-alike health centers in California.

The first phase of this project was a rapid two-month deployment of the Macro-Eyes (now Pendulum) health readiness product, Striata, which uses artificial intelligence (AI) to machine learn the current state of infrastructure at each FQHC while generating data on the catchment population. Striata learns about health infrastructure from publicly available data and data derived through a set of proprietary learning tools that power the platform. The ability to provide insight quickly without accessing proprietary health data enables rapid response and maintains health data safety and sovereignty.

The three main indicators for site readiness included in the initial deployment were as follows:

• Back-up power capacity. Identify presence of back-up power generation capabilities: solar, battery installation, generator.
• Refrigeration capacity. Identify capacity and reliability of refrigeration.
• Catchment population. Learn socio-economic indicators of site-specific catchment population.

These data driven insights allow decision makers to rapidly allocate resources and make precision investments in essential infrastructure. Every dollar and every scarce resource available for health – especially during natural disasters and pandemics – can be used effectively.

In the context of health systems broadly and COVID readiness specifically, Striata becomes a resource mapping and investment planning tool where decision makers can see in real time which facilities have a higher proportion of population at risk of not being able to access a vaccine, and which sites currently have low resilience to power outages where investment may improve their ranking.

RESULTS

In January 2021, the status of refrigeration capacity and back-up power availability across the California health safety net was known in 8% of facilities. Six weeks into deployment, Striata was able to provide visibility into 100% of federally qualified health centers (FQHC) and look alike sites in California – each rapidly updating in real time.

Striata yielded insight into 100% of community health centers included in the master facility list as compared to 3-8% at the beginning of the project. This was achieved over a two-month period. Predictive accuracies for refrigeration and back-up power were 71-85%% and 68-83.6% respectively, while catchment population insights yielded results using both defined vulnerability criteria and machine learned predictive vulnerability. The technology continues to learn from new data and improve predictive accuracy the longer it is run.

Striata found that out of 2059 sites, 1258 (61%) were without any form of back-up power (meaning no generator and/or no battery), while 212 (10%) were without refrigeration (170 of those are administrative and 42 are health service delivery sites.) 721 sites (35%) had both refrigeration and some form of back-up power.

Overlaid with counties at greatest risk for wildfires, these results clearly highlight a need to invest in infrastructure in those counties most at risk of fires and associated power outages in order to ensure uninterrupted services at CHCs serving the most vulnerable populations.

METHODS AND UNCERTAINTY

Macro-Eyes (now Pendulum) machine learning models were validated retrospectively using standard train-valid-test data splits. This rigorous machine learning model development framework ensured that the models were evaluated on data they never encountered during the training phase. Additional prospective validation has been done at a limited set of sites and is actively expanding. This is being done by means of user validation as well as access to additional labelled data newly entered in the public domain.

With each 100 pieces of labelled data – which refers to a data point known to be accurate and can also be referred to as ground truth data – the machine learning system is able to leverage at least twice that amount of unlabeled data. A 1:2 to 1:3 leverage is expected. At the level of current accuracy – 80% – if the machine learning system predicts 10 refrigerators at a site, the actual number of refrigerators will be between 8 and 12. Another way to interpret this is that 80% of the time, the predictions will only be off by 1 for every 10 refrigerators predicted.

IMPACT

Visibility is resilience. This level of insight has near term and long term strategic implications. The health safety net is ever changing, as are those who seek care. Shifts in population and environment require health systems to adjust the care delivery infrastructure and strategy. System leaders' ability to respond rapidly to disasters, pandemics, or other influences relies on an ability to know ground truth. Visibility from Striata can support targeted investment in the health safety net and identify sites ready to deliver essential services dependent on reliable refrigeration (such as a mass vaccination campaign), as well as sites able to continue to deliver and care services during power outages or times of restricted access. This is especially significant in regions where natural disasters are an annual occurrence.

In California alone, 2020 saw a total of 9,917 wildfires, according to the California Department of Forestry and Fire Protection, or CALFIRE. During the same timeframe, the Gulf Coast saw a total of 12 named storms or hurricanes.

In 2021 the state of Texas experienced a power grid failure resulting in 4.5 million homes and businesses left without power over several days. Together these events had the potential to impact as many as 5087 CHCs and 15,798,766 uninsured populations at the height of the Covid pandemic.

With these statistics in mind, Striata quickly becomes a resource mapping and investment planning tool where you can see in real time which facilities have a higher proportion of population at risk of not being able to access health services, and which sites currently have a low resiliency to power outages where investment may improve their ranking. The results of this process are linked to a user interface where decision makers at the state and local level can extract data to inform decision making. The user interface allows decision makers to quickly see where there may be a need to invest in infrastructure improvement, or where to allocate resources at a given point in time. Users can zoom in to a region and scroll over sites so as to understand whether or not that site has certain infrastructure in place. A user might focus on a region where there may be a lower percentage of eligible population vaccinated and then zoom in to see what the site level infrastructure is for that region. Users can then click to share feedback or to correct information, which allows validation and learning over time.

Vista Community Clinic participated in the validation of infrastructure readiness via an email exchange between DeeAnne McCallin (CPCA) and Dr. Sue Ann Park (Vista Community Clinic). It was conveyed that Dr. Park understood the importance and value of the data: “This is good information to know, will help to keep COVID vaccine safe if there is a power outage.”

OPPORTUNITIES FOR SCALE

After the rapid deployment, or build phase of the work, there is an opportunity to scale to include additional infrastructure elements, answer broader questions around resiliency, add features which emerged during stakeholder engagement, and to optimize ease of use for all stakeholders accessing the interface. While the use cases extend well beyond Covid, it is worth noting that the pandemic is far from over and the idea of equitable access becomes more important the further we get into the vaccination campaign, especially when this is combined with a natural disaster.

Striata can pinpoint populations most at risk of wildfires and power outages alongside the existing user interface. This provides insight into whether the health center is fire ready, what are the capacities that need to be built, and whether they have that capacity to improve resilience. It can cost a health center hundreds of thousands of dollars to be down for weeks at a time making the return on investment of resiliency significant.

Striata can empower regional associations to have a more influential role in the allocation of resources and decision-making power, positioning visibility as an advocacy tool. The ability to see into the current state of that infrastructure can generate enormous efficiencies for the CPCA and similar associations to provide a single source of ground truth for dialogue.

Finally, as daily vaccination rates continue to drop, the cost of vaccinating 70% of the population is no longer just about getting vaccines to sites, it's also about incentivizing people and ensuring that the infrastructure is in place to meet the demand and avoid wastage where possible.

‍

For more information on the work including the challenges encountered by the project, read the original post by Direct Relief here.

‍

Editor’s note: This article was produced in collaboration with Macro-Eyes (now Pendulum), Direct Relief, and the California Primary Care Association.

‍