Vaccine effectiveness, hospitalized vaccinated and Simpson’s paradox

Data presentations such as those in Figure 1 (representing 50 admissions of unvaccinated and 40 of vaccinated per week in the ICUs of a region) accompanied by headlines such as “the 44.4% of the covid patients admitted to the ICU were vaccinated ”(or its complementary“ 55.6% of the patients admitted to the ICU were not vaccinated ”). This type of statement raises doubts when, sometimes inadvertently, we try to infer causally about the effectiveness of vaccines .

Figure 1. Weekly ICU admissions by patients with covid-19. Each bed in the figure represents 10 admissions / week. / Salvador Peiró

These hospitalization figures, without being false, distort reality and affect the rationality of our judgment on the effectiveness of vaccines against covid. Below they have two major problems: the information they offer is badly “framed” and the Simpson paradox .

The frame effect ( framework effec t, framing effect ) is a cognitive bias [19459014 ] That modifies our preferences, making them less rational, according to the way the information is presented (“framed”) to us. The concept was introduced by Nobel Laureates in Economics Kahneman and Tversky and can be illustrated with one of their best-known experiments .

Very simply, the participants were given the choice between a hypothetical treatment for 600 patients with a serious illness that would save 200 of them, versus another hypothetical treatment with which 400 would die. Although both alternatives are identical (in both 200 people survive and 400 die), the majority of the participants preferred the life-saving treatment, the positively “framed” one.

The frame effect is a cognitive bias, which modifies our preferences, making them less rational, according to the way in which the information has been presented to us

In the hypothetical example in Figure 1, the framing is negative simply because it does not consider that the population of origin of these admissions (the number of vaccinated and unvaccinated patients in the region) is very different.

Continuing with the example, our ICUs have 90 weekly admissions for covid, 40 in vaccinated (40 * 100/90 = 44.4% of the admissions are vaccinated) and 50 in unvaccinated (50 * 100 / 90 = 55.6% of the admissions are not vaccinated). Now let’s add that the population of the region served by these ICUs is 5.5 million people over 12 years of age.

The total rate of admitted to the ICU would be (90 * 100,000 / 5,500,000 =) 1.6 admitted per 100,000 people over 12 years of age and week. But this joint number of vaccinated and unvaccinated, which is sometimes reported in official statistics, does not offer much information on the risk of admission to the ICU of vaccinated and unvaccinated.

Let’s add more data. Let’s say that 91% of the population> 12 years (5.5 * 0.91 = 5 million) is vaccinated, while the remaining 9% (500,000) are not. With data states we can now offer better framed information ( Figure 2 ): The 500,000 unvaccinated have generated 50 admissions / week in ICU, with a rate of (50 * 100,000 / 500,000) 10 admissions per 100,000 not vaccinated per week . The 5 million vaccinated have generated 40 admissions, with a rate of (40 * 100,000 / 5,000,000) of 0.8 admissions to the ICU per 100,000 vaccinated .

Figure 2. Admissions / week in ICU by patients with covid-19 in vaccinated and unvaccinated population. Each bed represents 10 admissions; each human figure represents 25,000 inhabitants. / Salvador Peiró

These population and group-separated rates already inform vaccinated and unvaccinated patients of their difference in risk of entering the ICU each week: 12.5 times more (1,150% more in relative terms) and offer better “framed” information for decision-making that confusingly conveys the idea that these risks are shared half and half. These data, population rates and relative risks (and not the proportion of people admitted), are what matter to inform rational preferences about vaccination.

Let us remember that, to the extreme, if 100% of the region’s inhabitants were vaccinated, all ICU admissions would come from this population. The big difference would be that the 500,000 previously unvaccinated would have produced only 4 admissions / week (0.8 / 100,000; 12.5 times less than the 50 in the example) and only a total of 44 people would have been admitted to the ICU that week in place of 90 ( Figure 3 ).

Figure 3. Admissions / week to the ICU by patients with covid-19 if the entire population was vaccinated. Each bed represents 10 admissions; each human figure represents 25,000 inhabitants. / Salvador Peiró

Age, severity, vaccination rate and Simpson’s paradox

The “frame effect” is not the only problem with the current presentation of rates when It is intended to make causal interpretations from the incidence of serious cases (hospitalizations, ICU admissions, deaths) and vaccination. Furthermore, there is significant confusion ( confounding ) created by the higher incidence of severe covid in elderly people (compared to younger ones) and the fact that these people are have been vaccinated in a higher proportion than the youngest.

In 1951, Edward H. Simpson described the statistical paradox that will help us understand what happens in some cases where, apparently, hospitals or ICUs they have more vaccinated than unvaccinated patients.

Those vaccinated have a higher incidence of severe cases not because they are vaccinated, but because of their older age; but within each group, those vaccinated have a lower incidence of severe cases

The Simpson paradox is a phenomenon that occurs when there is a trend in the general population (in the example of Figure 4 : the more vaccination, the higher the rate of ICU admissions) that disappears or reverses when the statistical analysis is stratified by groups.

In the example in the figure, stratifying by age reverses the trend and the higher the vaccination, the lower the rate of covid admissions within each age group. In other words: those vaccinated have a higher incidence of serious cases not because they are vaccinated, but because of their older age; but within each age group, those vaccinated have a lower incidence of severe cases than those not vaccinated.

Figure 4. Simpson’s paradox: the more vaccinated, the more income in the total population, but less when stratified by age. Each bed represents 10 admissions; each human figure represents 25,000 inhabitants. / Salvador Peiró

What Simpson’s paradox teaches when reporting vaccination preferences is that, if not stratified by age, severe cases in vaccinated will be overrepresented because there is a higher proportion of vaccinated older people and, in turn, older people are at increased risk of developing severe covid .

The information on covid-19 today

Although all the indicators that we use around covid have their indication , throughout the prevacunal period we have been using mainly 2 types: the cumulative incidence of new cases in 14 days (because it reported trends and was associated with a significant number of hospitalizations ) and the indicators of hospital or ICU capacity (because they reported on our ability to care for patients and prepare for saturation situations).

After the mass vaccination of more than 80% of the population, both indicators have modified (but not lost) their interest in favor of others. On the one hand, the accumulated incidence data for 14 days has been dissociated from what really matters: the serious covid data. Thanks to vaccines, and especially to the vaccination of people most at risk (over 60 years and some groups), transmission no longer produces as much hospitalization , ICU and [ 19459002] deaths as in the first waves. On the other hand, and due to this reduction in the incidence of serious cases, we do not expect the saturation of hospitals or ICUs.

The best way to represent the relevant information about covid is by separating the rates of new hospitalizations in vaccinated and unvaccinated, and stratifying them by age

We face a period in which the most relevant measures become the cumulative incidence of cases in older people (because this is where serious cases are generated, even to a lesser extent than before vaccination) and the incidence of new hospitalizations in vaccinated individuals (which despite being less than 10% of those over 12 years of age generate half of the admissions) and in unvaccinated individuals. It is not that the other indicators are not useful, but in order to convey a properly “framed” image of the situation and to decide on potential non-pharmacological measures, these would be the most important.

In this sense, the best way to represent the relevant information about covid today is probably the type used by the Centers for Disease Control and Prevention ( CDC ) of the United States, separating the rates of new hospitalizations in vaccinated and unvaccinated, and stratifying them by age ( Figure 5 ).

Possibly, aggregate transmission rates and the percentage of beds occupied were very important indicators to guide public health actions in the stages prior to mass vaccination. Also, possibly, times have changed and it is time to change the indicators.

Figure 5. Weekly hospital admissions rates for vaccinated and unvaccinated in the United States. Each bed represents 10 admissions; each human figure represents 25,000 inhabitants. / Salvador Peiró