Making (Some) Sense of the FBI's Crime Numbers

They have always been extremely messy, but here are some basics.

Over at RealClearInvestigations, John Lott has drawn attention to recent revisions in the FBI’s violent-crime numbers. I keep an eye on these numbers as well—and even emailed the FBI a couple weeks ago about a related revision to the murder rate I noticed—so I wanted to provide a quick overview of the issue.

First, here are the controversial changes that occurred between the 2022 and 2023 Crime in the United States reports, via a quickie Excel chart I made for Twitter this week. (You can get the tables yourself here.) The 2023 edition is the most recent; it concerns crime during that year, besides offering the newest revisions of the historical trend, but came out last month.

The discussion here is often framed as “the FBI claimed crime fell in 2022, but now it says crime rose that year.” Some even misinterpret the situation as “crime is rising.” But (a) the change in direction for 2022 is driven by a downward revision to 2021 (the upwardly-revised 2022 number is 377.1 per 100,000, which is ever so slightly below the unrevised 2021 number of 377.6), and (b) no matter which estimate for 2022 you use, violent crime was down in 2023. Well, at least pending further revisions.

I doubt there’s anything malicious here, but I do think it’s illustrative of the chaotic and not-entirely-transparent approach the FBI takes to these statistics. Let’s take a quick tour of how they work, or at least what I happen to know about how they work.

First, these are based on reports submitted by law-enforcement agencies, but not all agencies participate (though participating agencies did cover 95% of the U.S. population in 2023). So to get national estimates, they need to statistically estimate what the non-reporting agencies likely would have said. Of course, being based on police reports also means these figures by design do not measure crime that isn’t reported to the cops.

Second, the FBI often uses different methods in different reports or tables—some might be based on agencies that provided all twelve months of data for the year, while others might include partial reporters, for instance. When I first started diving into these reports in a serious way years ago, one of the first things I noticed is that tables in the same report can sometimes report the same statistic a little differently!

And third, these data are collected through two different systems, the more basic SRS (Summary Reporting System) and the more detailed NIBRS (National Incident-Based Reporting System). In 2021, they tried to force all agencies to use the superior but more time-consuming NIBRS, which resulted in catastrophically low participation, missing about a third of the U.S. population. The 2021 numbers are thus more heavily estimated than the others, and the FBI even went back and got 2021 SRS numbers from some agencies to retroactively try to improve its 2021 numbers for later reports.

Meanwhile, the agency is also transitioning to a more real-time-focused system that automatically updates old estimates and provides numbers faster.

These dynamics help to explain why revisions happen—new data come in, methodologies change, etc.—and why 2021 is particularly affected. But I can’t say I entirely understand everything that happened between the 2022 and 2023 editions of the Crime in the United States report. And I don’t feel bad about not understanding, because Jeff Asher, who is way more knowledgeable about these datasets than I am, says he isn’t sure either, and the FBI’s responses to media inquiries about this and other revisions have been pretty vague. So I really hope this kerfuffle forces more transparency even if it doesn’t reveal a big sinister conspiracy.

Lastly, a few words about other ways to measure crime trends.

First there’s the National Crime Victimization Survey. It’s a survey of the general public asking about their experiences with crime, including crime they didn’t report to the police. This survey can tell us some things the FBI data can’t, but there are also severe limits: (a) surveys have confidence intervals, so apparent increases and decreases can easily just be noise; (b) crime, especially serious crime, is fairly rare, so a small percentage of people making things up (or even misremembering when things happened) can throw off the estimates; (c) some of the most victimized populations might not be too likely to wind up in a government survey; (d) you can’t measure murder by surveying its victims.

But at any rate, the 2023 NCVS finds no statistically significant change in total violent crime (best guess is a drop from 23.5 to 22.5 per 1,000) or violent crime that is reported to police (best guess an increase from 9.7 to 10.1 per 1,000).

The survey also, oddly, suggests violent crime dropped in 2020 and 2021 despite the murder rate being way up—more about which in a second—and then shot back up in 2022 and 2023 while murder was falling. Helpful huh? At least it picks up the famous 1990s crime decline though:

The last big way of measuring crime is to focus on the murder or homicide rate, which is my personal favorite. Murder is the most serious crime, and it’s very well-measured for two different reasons: Cops generally tend to find out about a dead body, and the Centers for Disease Control also keeps track of homicides through death certificates, so you can cross-check the estimates of two different agencies. Murders also tend to correlate with other kinds of crime, though far less than perfectly. (My colleague Charles Fain Lehman, for example, has pointed out that while murder has come down lately, many places are struggling to control more basic forms of disorder.)

The story with murder is that it skyrocketed in 2020 with the pandemic and especially the George Floyd unrest—from 5.2 to 6.8 per 100,000, per the FBI. Since then it’s gradually cooled; as of 2023, it stood at 5.7, not quite back to its 2019 levels.

Another quickie Excel chart, including both the CDC and FBI estimates. The CDC tends to be higher; its definition of “homicide” includes things the FBI wouldn’t consider murder or non-negligent manslaughter, such as justifiable homicide.

Even in these data, note that the FBI’s number for 2021 seems a bit wonky. Bizarrely, their original, unrevised estimate for 2021 was an increase similar to the CDC’s. It was a goofy year!