I’m writing this in response to a series of blog posts from my former employer (whom I still respect tremendously along with all of those pieces’ authors for the record) as well as to get the monkey off my back from breaking my prior promise to reserve my next post to provide a deep-dive analysis of what can be done to improve policing in the United States (it’s still coming one day I promise, but I got very busy and didn’t get through all of the studies I wanted and have forgotten much outside of some notes). Also, I just really need to write here more!
Anyway, I’ve aired many grievances privately and publicly about those articles, but I’d rather walk through readers why at this time I think you can draw a pretty strong conclusion from the preliminary data that does exist that Michigan’s Pause to Save Lives altered behaviors in the state and not only saved lives, but did so within a pretty reasonable distance of the estimates provided by the University of Michigan’s School of Public Health. One area where my back of the envelope calculations differ strongly from UM is for some reason (almost certainly to bullshit inflate their statistics as much as possible) they chose to publish a case fatality rate of 2.6% in their analysis to arrive at a total of 2,800 deaths prevented in the period claimed.
A CFR of 2.6% is Michigan’s total for the entirety of the pandemic. It is an obscene one to choose for this analysis given the tremendous differences in quality of treatment and the age composition of the infected (which I will discuss more later) that Michigan and the US have had since spring. The CFR UM ended up eventually publishing as their lower bound, 1.8% (and 1960 deaths), is much more reasonable as it was roughly Michigan’s CFR for the period of interest; however, even then I think this is closer to a median or upper-bound estimate. Right now the CFR in the United States as a whole is 1.7%; however, there could be important demographic differences in the state that would make Michigan’s higher than average. I emailed some of UM’s research team seeking clarification along these lines to see if I was missing anything and they did not return any inquiries :/
Regardless, my very back of the envelope calculations come with the caveat that I have no personal expertise in any of this, but I expect my forthcoming model* to indicate there probably was somewhere between 1400-2100 lives saved in Michigan from the counterfactual of there being none of the policy interventions contained in the Pause. In the long-run, however, given the differences in suppression in Michigan relative to other states, I actually suspect the true total will end up being higher than those estimates, especially as vaccines are rolling out. Here is most of the explanation of how I get there…
Prior to comparing true infections between states, you need to try to standardize them by controlling for differences in testing such as deviations in the number of per capita tests being administered, the difference in positivity rate, and holiday reporting abnormalities. My preferred way to do this for months now has been Youyang Gu’s COVID model. I originally began following Gu after outlets like The Economist noted his impressive ability to forecast deaths relative to the field of models from universities and the like that received much more public funding. From the CDC to 538, his projections were widely included amongst those of ostensibly more prestigious institutions that were less accurate.
Gu ended up pulling the plug on the original forecasting intent of his website after the public persons whose actual jobs were to create these models improved them to a sufficient point where he no longer felt the need to update his. Instead, he pivoted his project towards nowcasting. Instead of projecting deaths, he is providing projections on how many people are infected (including asymptomatic individuals) currently, cumulatively, and on a given date.
These estimates should be taken with a grain of salt relative to other sources like serology studies; however, at this time those largely don’t exist for our purposes. In addition, to my naked eye it seems like his model may still undercount infections slightly in the spring and may slightly over-penalize places that conduct a relatively high amount of testing, but what the hell do I know. I’m just a half-rate wanna be economist, let alone an epidemiologist or a guy who made a model that consistently beat their collective asses. I personally suspect Gu’s project can control for testing/case differences in a way better than the vast majority of people ever could, self-included.
Therefore, using his data (which I’ve been saying you should use to evaluate this way before I knew what it was going to look like), what do you see in the difference in infections between states? Ideally, this comparison would be done for a larger region like the entire American Midwest (chosen for geographical and cultural proximity), rather than just a few neighboring states. However, for the sake of brevity, let’s take a look at just Michigan and its neighbors:
What observations can be made here? Michigan’s new/current infections likely max out exactly where you’d expect them to assuming there was an intervention on November 18. Not only this, but it does so at a lesser magnitude on a per person basis than our neighbors and it remains near those peak levels for a shorter time. Furthermore, remember that when you look at positive tests in the news or an aggregating site, those are generally people that were infected ~2 weeks prior. This is because you have to wait for people to develop symptoms + the time it takes for labs to turnaround results and submit them to the state. Gu’s model and his pics account for all that and more (or at least they try to and probably to a better degree than nearly anybody else).
You might think it seems crazy that MI has significantly fewer infections than its neighbors in aggregate given our deaths, but actually, even beyond accounting for improvements in treatment, the biggest difference between now and spring (when MI had disproportionately large numbers of deaths) is the enormous change in the age composition of the infected. You might think well yeah, olds were prioritized for testing given scarcity in the spring, but if that was only the case you’d expect lower positivity rates, while they actually had higher ones!
And I don’t have it as handy and can’t recall where I saw a good estimate of this (maybe someone can dig it up and post it in the comments), but differences in the early social mobility data are visible by age too. Olds (understandably) simply weren’t social distancing at the beginning of the pandemic and huge numbers were getting sent to the hospital and dying as a consequence because COVID is so much more fatal for them.
So, as I said, I would take the inexact infection estimates from Gu in the first few charts with a grain of salt, but I think they are very much directionally true and there’s little question a lot more Michiganders were infected in fall/winter than spring, and we’ve had fewer overall than our neighbors this fall/winter. I think it’s extremely likely our neighbors all surpassed us in cumulative infections as well.
Meanwhile, if you look at data measuring social distancing, you should be able to tell what state is Michigan without me even labeling them simply by checking the axes. Michiganders started from a lower point of social distancing relative to other states prior to the orders, but our encounter density (basically, how often are people interacting with one another/km^2) was driven far lower relative to other states and would not peak anywhere near as high over the holidays. I kind of arbitrarily chose this as my metric to post here in order to make a point, but it’s visible in any of the other social distancing metrics as well.
And it’s not just there, distancing differences are also especially pronounced in retail & recreation, which I suspect is because of the state’s closure of indoor dining (Also, I’m not sure how to get earlier data from Google’s mobility reports, so message me if you know).
To go off on a bit of a tangent here, the indoor dining thing strikes me as particularly important because there is in my estimation a large amount of evidence suggesting that restaurants are one of, if not the, largest vectors of transmission outside households. Indoor dining is an especially risky activity given that it is maskless and it does not take much time at all in such a setting to infect a bunch of others. You may have heard 6 feet distance and 15 minutes of contact for spread, but that’s mostly bunk from very early guidance, and our top available data confirmed with genomic sequencing from the previously linked South Korea study examining transmission within a restaurant and the NFL indicate transmission occurs over much larger enclosed distances and in as little as 5 minutes if you are not wearing a mask. Michigan has had a very difficult time contact tracing to prove outbreaks linked back to restaurants among non-staff, but that has always struck me as part of the problem. For the most part, it’s strangers infecting strangers and the staff isn’t keeping track, while we have mobility data suggesting that restaurants may even be the #1 spreaders outside the household by a significant margin. Michigan’s restaurants are also getting pulverized relative to other states – the data backs people just aren’t going there.
All of those strike me as very plausible reasons exploring the causality of how Michigan has fewer infections, but if that is really the case, do these alleged differences show up in other data as well. I think the answer is a resounding yes!
You see trends in hospitalization data that pretty closely mirror what you would expect to see in the infection forecasts, with them peaking and beginning a faster rate of decline in Michigan almost exactly 2 weeks after the Pause took effect.
Hospitalizations are a decent timeliness bridge in the data between cases and deaths; however, I also assume they are a little closer to cases because most hospitalizations are severe, but not life-threatening. ICU admissions are closer in timeliness to death data since they largely represent individuals on the verge of dying. I would expect these to peak something like 3-4 weeks after the policy took effect as a consequence.
Meanwhile, in the death report data, there are a couple of weeks that are largely complete and I think are reported enough to where you’d expect to see an effect from a policy difference. Keep in mind that this data is all ⚠️provisional⚠️ (4/17/2021, it’s no longer all that provisional, but I’m too lazy to go through and edit this further), hence why I chopped off the final 7 weeks of the data series. I am also unsure of reporting idiosyncrasies between states. In my experience, Michigan has been the fastest reporting to the CDC and has had the fewest revisions over time, but that could all be some sort of coincidence and bodies may be piling up to the point they are waiting to dump 1500 in a day (for the record, this is exactly what Indiana did a few days ago).
(EDIT: 2/12/2021 Hours after publishing, Ohio health officials announced they had 4,000 more bodies to report as well 🤦. However, from what I can gather the vast majority of these were already being reported to the CDC and just weren’t apparent in their official state dashboard’s total. I don’t think it alters my analysis all that much).
Additionally, most studies that measure policy impacts on death use a lag range of something like 3-4 weeks to measure the difference in backdated deaths (see one such study linked below that contains references to others and an in-depth explanation of why in its appendix). It’s essentially because it generally takes 5-6 days for an infected person to develop symptoms, and from there it takes something like 2 weeks for the average and median death to occur, with the distribution being slightly right-skewed. So, with that caveat in mind, how does Michigan compare to other states?
Sure enough, deaths in Michigan peak in that 3-4 week window after the policy took effect and did so at a significantly lesser magnitude than other states. Based on everything I’ve written in this post, you can expect that cumulative distance to widen over time as the data is more finalized.
Nevertheless, maybe these differences are entirely the result of death reporting discrepancies between the states? If that was the case, you would expect there to be a meaningful contrast between them and the COVID all-cause mortality data. Keeping in mind these are even more ⚠️provisional⚠️ than the COVID mortality data and all of these curves even with the final 7 weeks dropped will still be revised significantly upwards, there really doesn’t seem to be any tangible differences of note.
Maybe you’re still skeptical and believe the primary reason there are so many fewer infections/deaths in Michigan this fall/winter is that we have achieved herd immunity in large part during the spring wave of infections. That surely plays some role, but if that was the dominant factor you would not expect there to be any differences in county-level infections and deaths among areas that were relatively unaffected in the spring outbreak. Thankfully, border counties along the MI/IN/OH’s border and outside the Detroit metro fit that exact criterion, and given their geographical and cultural proximity, there is no particular reason we would expect there to be significant differences in cases/deaths across the county/state borders – unless there was some sort of policy differences among the people living in those imaginarily drawn lines. So what do we see?
With all of this being said, the idea that it is all some sort of coincidence is quite frankly preposterous to me. Especially when I have been making predictions this is exactly what you’d expect to see if the policies of the Pause did have an effect! It’s also all consistent with Michigan’s infection curve being relatively late compared to its neighbors this fall and coming to a comparatively abrupt halt from increased social distancing. And what caused that social distancing? Probably the government’s heavy-handedness!
Although I am not sure the likes of Indiana, Ohio, and Wisconsin will ever surpass Michigan in true cumulative COVID deaths on a per million people basis from the start of the pandemic, if they had similar treatment and a composition in demographics of those infected as we had in spring they would surpass us – it wouldn’t even be close.
Or maybe in a few weeks Michigan is going to release a massive Indiana-style data dump that turns this entire post and its analysis into 💩.
P.S. – The University of Michigan should really make the model they use publicly available. By fiddling around with Gu’s data I can also get pretty close to a difference of 100,000 “true” infections (Gu lists expected total infections including asymptomatic cases), but in many ways I am personally concerned I have basically no idea what I’m doing and am unintentionally twisting the data to herd with their estimates. Nevertheless, if experts claim something saved thousands of lives, they should really make the foundation of that claim publicly available before rushing to the presses!
*In the coming months I plan on publishing here the equivalent of what will be a mediocre thesis paper to unrust my econometric skills on the efficacy of Michigan’s interventions this fall that basically follows the approach of this paper, except the non-policy interventions I am interested in are things like closing/restricting indoor dining, schools, etc instead of masking. My model primarily focuses on backward induction from a lagged regression on backdated CDC death certificate/excess death data and comparing Michigan and midwestern states like it to the US Midwest Census region (plus maybe Pennsylvania since they are Great Lakes?, but I don’t want to create too much more work for myself unless somebody pays me…). Much of the requisite data is still outstanding and being finalized, but I think enough is in place for the first 1-2 weeks of our time of interest so that the conclusions within here can start being drawn with some degree of confidence. Once I am done, whenever that may be, I plan on making the model and all data I use publicly available at this site.
Edit 4/17/2021: This proved too difficult relative to my skill level and nobody actually paying me, so sorry for anyone hoping for more! Take this Reddit post as the closest thing to a compromise.