Ask Catalyst: What is the Difference Between TAR 1.0 and TAR 2.0?

[Editor’s note: This is another post in our “Ask Catalyst” series, in which we answer your questions about e-discovery search and review. To learn more and submit your own question, go here.]  

This week’s question:

Twitter_Ask_Catalyst_John_Tredennick

Your blog and website often refer to “TAR 1.0” and “TAR 2.0.” While I understand the general concept of technology assisted review, I am not clear what you mean by the 1.0 and 2.0 labels. Can you explain the difference?

Today’s question is answered by John Tredennick, founder and CEO.

We developed TAR 2.0 as a shorthand way to describe a new generation of technology assisted review engines that worked differently than TAR 1.0 engines and offered to significantly reduce the cost and time to find relevant documents during a review.  While TAR 2.0 is regularly associated with CAL, which stands for continuous active learning, it goes beyond CAL to address a number of shortcomings found in TAR 1.0 products.

Let me start with a description of a typical TAR 1.0 process.

TAR 1.0: One-Time Training

While different products follow different processes, the hallmark of TAR 1.0 is one-time training. In essence, a subject matter expert (SME) codes a control set for relevance and then trains against that control set. When the training is done, the system ranks the remaining documents and orders them by the likelihood of relevance.

Here are the typical steps for a TAR 1.0 process:

  1. An SME, often a senior lawyer, reviews and tags a random sample (500+ documents) to use as a control set for training.
  2. The SME then begins a training process using a mix of randomly selected documents, judgmental seeds (documents you find yourself) or documents selected by the computer algorithm. In each instance, the SME reviews documents and tags them relevant or non-relevant.
  3. The TAR engine uses these judgments to train a classification/ranking algorithm to identify other relevant documents. It compares its results against the SME-tagged control set to gauge its accuracy in identifying relevant documents.
  4. Depending on the testing results, the SME continues training to improve performance of the algorithm.
  5. The training and testing process continues until the classifier is “stable.” That means its search algorithm is no longer getting better at identifying relevant documents in the control set. There is no point in further training relative to the control set.

OneTimeTraining

The next step is for the TAR engine to run its ranking algorithm against the entire document population. For testing, the SME might review another random sample of ranked documents to determine how well the algorithm did in pushing relevant documents to the top of the ranking.

Once these procedures are completed, the review team can be directed to look at documents with relevance scores higher than the cutoff point. Documents below the cutoff point can be discarded.

Even though training is initially iterative, it is a finite process. Once the algorithm has learned all it can about the 500+ documents in the control set, that’s it. You simply turn it loose to rank the larger population (which can take hours to complete) and then divide the documents into categories to review or not review.

Problems with One-Time Training

When we originally developed Insight Predict, our chief scientist, Dr. Jeremy Pickens, felt the TAR 1.0 process had a number of limitations which he sought to overcome.

  1. One bite at the apple: The first limitation — and most relevant to a discussion of continuous active learning — is that you get only “one bite at the apple.” Once the team gets going on the review set, there is no opportunity to feed back their judgments on review documents and improve the ranking algorithm.
  2. SMEs required: A second problem is that TAR 1.0 generally requires a senior lawyer or SME for training. Expert training requires the lawyer to review thousands of documents to build a control set, train and then test the results. Not only is this expensive, but it delays the review until you can convince your busy senior attorney to sit still and get through the training.
  3. Rolling uploads: Another limitation of the TAR 1.0 approach is that it does not handle rolling uploads well, even though they are common in e-discovery. New documents render the control set invalid because they were not part of the random selection process. That typically means going through new training rounds, which is bothersome to say the least.
  4. Low richness: Low richness collections are a problem for TAR 1.0 because it can be hard to find good training examples based on random sampling. If richness is below 1 percent, you may have to review several thousand documents just to find enough relevant ones to train the system. Indeed, this issue is sufficiently difficult that some TAR 1.0 vendors suggest their products shouldn’t be used for low richness collections.

TAR 2.0: Continuous Learning

Working with our technologists and developers, Dr. Pickens created a ranking engine that could rank about 1 million documents in less than five minutes. As a result, it seemed obvious to build a predictive ranking system based on the notion of continuous learning.

Continuous learning means that the algorithm is not limited to one round of training. Rather, as the review progresses, the algorithm continues to learn, taking advantage of the additional judgments made by the reviewers. The reference to “active” means that the system sends documents to the review team based in part on the continuously updated rankings.

CAL’s superiority was first documented in a 2014 peer-reviewed study on the effectiveness of the various TAR protocols by Maura R. Grossman and Gordon V. Cormack, who gave CAL its name. They concluded that CAL was far more effective than the one-time training methods used in TAR 1.0 systems. There have been a number of studies since, all concluding that continuous learning is superior to earlier methods.

The process in a CAL review is quite simple.

TAR20withCAL

  1. Start with as many relevant seeds as you have or can easily find. These may be documents found through initial searches, through witness interviews or perhaps from earlier reviews. Use these for initial training of the algorithm.
  2. Begin the review process. As the review progresses, tagged documents are continuously fed to the algorithm to continue the training. The algorithm continues to rank the documents in relevance order based on the increasing number of training documents.
  3. Continue the review process until the number of relevant documents decreases substantially or runs out.
  4. Sample the un-reviewed documents to estimate the percentage of relevant documents found and those remaining. If the recall percentage is sufficient for your purposes, you can stop the review.

There is more to be said about the process, but these are the basics.

Other TAR 2.0 Features

While CAL is one characteristic that distinguishes between TAR 2.0 and TAR 1.0 systems, there are other key differences:

  1. Does not use control sets: A TAR 2.0 system does not use control (or reference) sets to train the algorithm. Rather, the system ranks all of the documents all of the time. We measure training progress by the fluctuation in the ranking of individual documents across the entire set. Among other benefits, this allows the system to handle rolling uploads. TAR 1.0 requires training against a reference set. This limits its ability to handle rolling uploads, because it assumes that all documents have been received before the ranking. TAR 2.0 analyzes and ranks the entire collection continually, allowing new documents to be added at any time.
  2. No need for SME training. With TAR 1.0 protocols, the review team cannot begin work until the SME does the initial training of the system. In some cases, this can hold up a review for days or even weeks. Our research and that of many others shows that CAL does not require training by SMEs, which means review starts right away. Meanwhile, the SME can focus on higher-level tasks, such as finding relevant documents and performing QC on review team judgments.
  3. Judgmental seeds. Many TAR 1.0 products use random seeds to train the system and avoid bias. This can preclude or limit the use of key documents found by the trial team. TAR 2.0 uses judgmental seeds so that training can immediately use every relevant document available. Bias is avoided through supplemental training with active learning.
  4. Contextual diversity: One of the key concerns for TAR users is whether the algorithm has missed important relevant documents. The fear many raise is that “you don’t know what you don’t know.” Because TAR 2.0 uses judgmental seeds rather than randomly selected documents, some wonder if it is missing other categories of relevant documents in the bargain. To address this issue, we developed what we call a contextual diversity algorithm. It is a part of our TAR 2.0 definition. In essence, the algorithm groups documents the review team hasn’t seen and presents representative samples from these groups in the review mix. If the reviewer tags the samples as relevant, similar ones are moved up the ranking. When samples are tagged not relevant, similar documents move to the bottom.
  5. QC algorithm: Our TAR 2.0 system includes a QC algorithm designed to help minimize review mistakes and inconsistent coding. The algorithm ranks documents by the likelihood that the review judgment is a mistake. It presents documents tagged as relevant which have a low ranking and documents with a high ranking but marked irrelevant.
  6. Works with low-richness collections. TAR 1.0 systems choke on low-richness collections and is impractical to use with small collections. The problem is that it is hard to find good training examples based on random sampling. With very low richness collections — say below 1 percent — you may have to review several thousand documents to find enough to train. A TAR 2.0 system lets you start with any relevant documents you can find, so neither low richness nor the small size of the collection is a problem.
  7. Run multiple rankings simultaneously: An additional advantage of some TAR 2.0 systems is the ability to run several ranking projects simultaneously. You might be ranking for “responsive” as one example while also running a QC ranking for privilege or a ranking based on different issues.

The most compelling benefit of using a TAR 2.0 system built around CAL is savings of cost and time. In an earlier blog post, I ran some numbers to estimate the potential cost savings using CAL based on the Grossman-Cormack research. I concluded that CAL would save $250,000 over one form of TAR 1.0 called Simple Passive Learning and $115,000 over another form called Simple Active Learning. Those are significant savings and they can be even greater in larger or more-challenging collections.

A companion to cost savings is time savings. In that same blog post, I estimated that in an average review CAL could save anywhere from 121 hours to more than 3,300 hours, depending on the protocol and the number of training seeds used. (See the original post for charts detailing these calculations.)

This is a very brief overview of what we mean by TAR 1.0 and TAR 2.0. If you would like to learn more, we have written extensively about TAR on this blog and we have a book you can download that goes into even greater depth on the subject.

mm

About John Tredennick

A nationally known trial lawyer and longtime litigation partner at Holland & Hart, John founded Catalyst in 2000. Over the past four decades he has written or edited eight books and countless articles on legal technology topics, including two American Bar Association best sellers on using computers in litigation technology, a book (supplemented annually) on deposition techniques and several other widely-read books on legal analytics and technology. He served as Chair of the ABA’s Law Practice Section and edited its flagship magazine for six years. John’s legal and technology acumen has earned him numerous awards including being named by the American Lawyer as one of the top six “E-Discovery Trailblazers,” named to the FastCase 50 as a legal visionary and named him one of the “Top 100 Global Technology Leaders” by London Citytech magazine. He has also been named the Ernst & Young Entrepreneur of the Year for Technology in the Rocky Mountain Region, and Top Technology Entrepreneur by the Colorado Software and Internet Association. John regularly speaks on legal technology to audiences across the globe. In his spare time, you will find him competing on the national equestrian show jumping circuit or playing drums and singing in a classic rock jam band.