Starting from first semester of 1L year in law school, it’s been drilled into our heads that we must rely on a “citator” like LexisNexis’ Shepard’s, Westlaw’s Keycite, Bloomberg Law’s BCite, or Casetext’s SmartCite. We’ve been told that, if we don’t, we risk citing to “bad law” — a case that’s been overturned, reversed, or undermined by subsequent authority.

But new research shows that that over-reliance on a simplistic flag system is dangerous. A recent article by Paul Hellyer, a law librarian at William & Mary Law School, demonstrates that Shepard’s, KeyCite, and BCite are wrong a lot.

The article, Evaluating Shepard’s, KeyCite, and Bcite for Case Validation Accuracy, is “the largest statistical comparison study of citator performance for case validation.” The shocking takeaway of this study: in only 15% of the time did all three systems analyzed agree that a particular case has had negative treatment applied; “in 85% of these citing relationships, the three citators do not agree on whether there was negative treatment.” They can’t all be right — at least one system wrongly identified negative treatment 85% of the time. Yikes.

As Kristina Niedringhaus, the Associate Dean of Library and Information Services and an Associate Professor at Georgia State University College of Law, said in her review of the study: the results of the study are “distressing.”

Why do citators make so many mistakes?

To understand the results of the study, it’s important to understand why systems like Shepard’s, KeyCite, or BCite might miss negative treatment. Running a legal research company has given me insight into this issue.

The primary reason legacy research systems make these mistakes, I believe, is that they use largely editor-driven — i.e., human-driven — processes. Humans make mistakes.

It’s part of why we at Casetext rely on artificial intelligence to do some of the heavy lifting for SmartCite: machines are getting increasingly good at automating this process, and are less error-prone than humans.

We also rely on intelligence extracted from the text of legal opinions themselves. For example, Casetext SmartCite has yellow flags that will note where a case has been cited with a contradictory citation signal (but see, but cf., contra, etc.) We do this because attorneys want to see how judges cite the case they’re about to rely on. And because we’re leveraging the collective wisdom already contained in the common law, it’s much more likely to be useful information.

All that said, humans are a part of our process too. For example, attorneys hand-check the results of the machine learning algorithms. Although our internal analyses of our performance has us doing a lot better than those studied by Hellyer, I am certain we are not without error.

So even the most advanced, technology-driven system can make mistakes. What are you to do if you want to avoid citing bad law?

How to avoid citing bad law by accident

Given that none of these systems are perfect and relying on the presence or absence of a “red flag” may not be enough, how can you avoid citing to bad law?

There’s one simple tip that I’ve used since law school that almost always works:

• Look at the entire list of citing cases
• Sort by the cases that were most recently published
• See how the case you’re researching has been discussed by the 10-20 most recent cases that cite to it

Most legal research systems make this trivially easy. On Casetext, you simply click the “Citing Cases” tab and then “Sort by Newest to Oldest.”

Why do this? A few reasons. First, if a case has been overruled, that’s almost always made obvious by how recent cases talk about it. Or it won’t be talked about at all recently, which is itself a potential signal about the case’s continued validity (although that does not conclusively say anything one way or another).

For example, look at how Austin v. Michigan Chamber of Commerce has been discussed subsequent to its overruling by Citizens United. More recent courts make clear that the case has been overruled as they discuss it:

Second, even if a case has a “red flag” on it, the case may still be good law for some propositions. For example, McGore v. Wrigglesworth, an opinion from the Sixth Circuit Court of Appeals, has been overruled in part. But it is still being cited up to this day, multiple times a week, for propositions for which it is still good law. This indicates that it is still potentially citable.

Using this simple method can help you make sure that you avoid mistakes. Instead of relying on a system’s flag system, you will instead see for yourself in just a few moments how a case has been discussed recently. That signal may be as important, if not moreso, than anything else.

Next steps

All this said, citator systems like Shepard’s, KeyCite, or Casetext’s SmartCite let you see at a glance a lot of information, will save you a lot of time, and still have a lot of value. If this is the sort of stuff that interests you, I encourage you to check out what we’ve done with SmartCite — which goes far beyond the legacy publishing companies and gives a level of intelligence and information you’d expect from a modern legal research platform.