Does Technology-Assisted Review Work for ‘Hot’ Documents?

[This post originally appeared at DSicovery, the blog of DSi (formerly Document Solutions, Inc.), a provider of high-quality litigation support services, electronic discovery and digital forensics to law firms and corporations worldwide. DSi was Catalyst’s first Summit Partner.]

One of the many unanswered questions about technology-assisted review (TAR) is whether or not – and how well – it works for finding “hot” documents, as opposed to merely responsive ones. We recently had a chance to work with our partner DSi and its law firm client on this very issue in conjunction with a large production they had received. I want to report on what we have learned.

Overview

DSi’s client received about 960,000 documents from defense counsel in several document productions. The first production had about 750,000 documents, with the remaining documents added in subsequent uploads.

The documents were primarily in image format with associated text. There were native files for spreadsheets as well.

Because depositions were looming, the client asked whether Insight Predict, our Predictive Ranking software, could help the law firm team identify hot documents that would be important for use in the depositions. Hot documents would become key deposition exhibits. Documents that weren’t hot but still important were tagged as “responsive” documents.

General Ranking Procedures

We started by loading the production documents we had into Insight Predict for analysis and ranking. By then the team had already reviewed about 3,000 documents. We used these documents as initial seeds (positive and negative) for the ranking algorithm and ranked the entire population.

We then took the top 2,000 ranked documents and foldered them for the team to review. They marked the documents as hot, responsive or otherwise tagged them as not important. After that review, we used the documents as additional seeds for a further ranking.

We continued the process in an iterative fashion, allowing the team to move forward with their deposition prep but re-ranking in 2,000 document increments as they moved through the documents. The team ranked about 20,000 documents in this fashion.

Success with Hot Documents

Did the process work? Did it provide an increased number of hot or responsive documents to the team? The goal of TAR is to increase the number of targeted documents presented at the front of the ranked order. Doing so allows the team to get to the important documents more quickly and, at their option, ignore the lower-ranked documents. The savings in review costs comes from more quickly finding important documents.

To answer this question, we needed baseline figures for comparison purposes. During the course of our work, we learned that counsel had run a series of key word searches to identify 180,000 potentially hot or responsive documents. The review team went through them in a linear fashion and tagged each of them as hot, responsive or some variant of not responsive.

We took the results of that review and compared it to the tagging done on the 20,000 documents we provided through Insight Predict. Here is how the numbers came out:

Screen-shot-2013-08-16-at-9.54.53-AM

In both cases, it seems clear that the produced documents contained a low percentage of either responsive or hot documents. A comparison of the two approaches suggests that Predict found a higher percentage of hot documents (3.95% vs. 1.34%) but a lower percentage of merely responsive documents (1.86% vs. 2.26%).[1]

In this case, the numbers suggest that Predict found almost three times as many hot documents per 100 as did the keyword searches. That is an improvement of 300%, which is a good result. We understand how users might look for dramatically higher returns, e.g. 70 out of a 100 being hot, but that is not within the realm of reasonable expectations. A 300% return is still a substantial improvement considering the alternatives.

Witness-Based Ranking

As the project progressed, the team asked us to perform rankings for specific witnesses scheduled for deposition. For this process, we took seed documents (already reviewed by the team) relating to each deponent and used them in separate, witness-specific rounds of rankings. We then identified the top documents that related to that specific deponent from each ranking and foldered them for review.

The custodian rankings seemed to bring an even higher percentage of hot and responsive documents than the overall averages. Here are the numbers elicited from that review[2]:

2

As you can see the percentage of hot documents varied from a low of 1% to a high of 30%. The percentage of responsive documents ranged from 0% to a high of 33%.

On balance, the numbers seemed far better than what we would have expected from a random sample of those same documents. That conclusion could only be confirmed statistically through a random sample for each custodian, which we did not ask the team to do. Nonetheless we feel these results are on target for the purposes of this analysis.

Here is a graphical view of the hot documents retrieved for each custodian by percentage.

3

For comparative purposes, the average richness across the population was in the range of 1.5 to 2%. Thus, we managed to improve the result set by a substantial margin using our Predict engine.

Conclusion

Based on our analysis, we feel that Insight Predict was successful in helping the team find and review hot and responsive documents. It is important to remember that the richness (number of hot or responsive) was low and the process ultimately was only going to yield about 6,000 documents that were of interest to the team. However, the analysis suggests that Predict could have helped the team find them much more quickly than using other means.

In reaching this conclusion, we note that we could have fine-tuned our results with more comprehensive samples and further analysis. We did not do that on this project because of the tight deadlines and our desire not to burden the review team with extensive sampling. Nonetheless, we think the results were indicative of what further sampling would disclose and support our ultimate, albeit practical, conclusions.

Using TAR techniques to master documents produced to you (as opposed to those you are producing) is a relatively new topic that few have written about. Ralph Losey recently wrote an excellent discussion on the topic in his must-read e-Discovery Team blog. I recommend that you give this a read as well when you have a chance: Why a Receiving Party Would Want to Use Predictive Coding?


[1] We attribute those results to the fact that we used only hot documents as Predict seeds. As a result, the engine was targeted towards those types of documents and seemed to do its job well.

[2] The names are fictitious to protect client confidentiality. Any resemblance to real people is purely coincidental.

mm

About John Tredennick

A nationally known trial lawyer and longtime litigation partner at Holland & Hart, John founded Catalyst in 2000. Over the past four decades he has written or edited eight books and countless articles on legal technology topics, including two American Bar Association best sellers on using computers in litigation technology, a book (supplemented annually) on deposition techniques and several other widely-read books on legal analytics and technology. He served as Chair of the ABA’s Law Practice Section and edited its flagship magazine for six years. John’s legal and technology acumen has earned him numerous awards including being named by the American Lawyer as one of the top six “E-Discovery Trailblazers,” named to the FastCase 50 as a legal visionary and named him one of the “Top 100 Global Technology Leaders” by London Citytech magazine. He has also been named the Ernst & Young Entrepreneur of the Year for Technology in the Rocky Mountain Region, and Top Technology Entrepreneur by the Colorado Software and Internet Association. John regularly speaks on legal technology to audiences across the globe. In his spare time, you will find him competing on the national equestrian show jumping circuit or playing drums and singing in a classic rock jam band.