Blog

Links

Distributed systems: vaccines, websites, and vaccine websites

posted: March 21, 2021

tl;dr: A widely distributed system can have the best performance and reliability, for both websites and vaccine distribution...

A little more than a week ago I received the Johnson & Johnson/Janssen COVID-19 vaccine at a local Walgreens. I freely admit that there was some amount of luck involved, especially in getting the one-dose J&J/Janssen vaccine instead of either of the two-dose vaccines, from Moderna and Pfizer. I actually signed up, on Walgreens website, for two vaccine appointments spaced four weeks apart, but when I got to the pharmacy counter for my first appointment and chatted with the pharmacy clerk, she told me that the store had received their first batch of J&J/Janssen vaccines a day ago. No appointments had been booked for these unexpected vaccines, so she offered it to me, as the batch was going to expire in a few days. I also learned about the process the store has established to use up vaccines before they expire, as they usually have more vaccine doses than appointments booked via the website.

Another aspect of luck was in the vaccine eligibility requirements. The age group I’m in became eligible several weeks ago but only for certain front line workers, which excluded me, since I can work from home. Then the authorities dropped the front line worker requirement, making the vaccine available to everyone in my age group. I read that this was done to speed the rollout: having people provide the paperwork to prove they are in a certain type of occupation, instead of just showing an ID card with date-of-birth on it, slows down the vaccine administration process.

Yet some people in Pinal County, Arizona, are even luckier: the county just announced that everyone 18 years and older is eligible to receive the vaccine at county-run sites. There have been statistically significant differences in the vaccination rates across different counties and locales within Arizona. Arizona claims to be one of the few states that is doing a better job of getting the vaccine out to socially vulnerable populations than richer parts of the state.

There are a wide variety of institutions administering the vaccine, each with their own signup procedures and websites. There are a few large state run sites, including (at the moment) at State Farm Stadium in Glendale, Phoenix Municipal Stadium near Tempe, the University of Arizona in Tucson, and Chandler-Gilbert Community College. There are county-run sites. Hospital systems are administering the vaccine. Several large employers, especially those with front line workers, are arranging to vaccinate their employees. And there is also the private sector, including drug stores like CVS and Walgreens.

This may appear to be a totally haphazard and chaotic system, yet I applaud it. To me, it is prioritizing the right goal, which is to get as many people vaccinated as quickly as possible, in rough priority order. It is a widely distributed system, as opposed to a more centralized system run with stronger control from the state or the federal government. In both vaccine distribution and websites, my experience is that a widely distributed system can scale up faster and perform better than a centralized system.

A widely distributed system can take advantage of existing low-cost infrastructure, instead of having to build expensive, brand new infrastructure from scratch. I got my COVID-19 vaccine in the same small vaccination booth at the Walgreens pharmacy where, at other times, flu shots are administered. Pharmacies and hospitals already had personnel trained in giving shots. Lots of places administering the vaccine already had websites capable of making appointments online: they just needed to be loaded with the data for how many vaccines could be administered in given time periods.

A widely distributed system is also more impervious to failure. If one vaccine site gets shut down for whatever reason, there are plenty of others. If one website has a major outage or performance problems, there are plenty of others; the entire system doesn’t grind to a halt. The primary downside of this system is that users can’t just go to one place to sign up for a vaccine appointment; they are better off trying multiple websites. A centralized system would be best if the top priority was not speed, but was instead ensuring that the vaccine was distributed in strict priority order, from the person deemed most in need of the vaccine to the least.

This system stands in stark contrast to another public health initiative, the launch of the Obamacare healthcare.gov website and program back in 2013. That architecture was much more centralized, with millions of people around the country attempting to enroll on the same federal government-run website. It was widely acknowledged that the initial website had major performance issues that required a “tech surge” to address. The Obamacare system itself, and the health insurance industry in general in this country, is also monstrously complex, which exasperated the website’s problems. System complexity even showed up in a bug in TurboTax which I found. Thankfully, the U.S. appears to have learned some lessons from that disaster, and the COVID-19 vaccine rollout was not governed by forcing everyone to enroll on a single federal website into an overly-complicated program.