What to Expect from Technology-Assisted Review in 2019

This article first appeared in Law360 on January 3, 2019.

2018 will be remembered as a transition year for technology-assisted review. The battle over whether we can use TAR has all but disappeared, and our attention has turned to how we will use TAR — an inquiry with two clear dimensions. In litigation, the question is whether, and to what extent, we will put TAR under a microscope and force the discussion of parameters surrounding the implementation of the technology-assisted review process. Outside of litigation, the focus has shifted to identifying alternative techniques and other applications for TAR technology within the legal space.

Calendar year 2019 will likely see a continuation of this trend, with refinement and expansion across the board. As document collections continue to grow in litigation, and technology-assisted review becomes increasingly prevalent, we will see the boundaries of the TAR process begin to be explicitly tested and delineated. And, given the effectiveness of technology-assisted review in efficiently locating documents, the use of TAR techniques will undoubtedly see growing adoption outside the arena of production in litigation.

Moving in the Direction of Even Greater Acceptance

One of the key cases addressing TAR protocols at the tail end of 2017, and widely analyzed in 2018, is largely overlooked for its implications on the level of judicial acceptance of TAR. In Winfield v. City of New York, Magistrate Judge Katherine Parker primarily addressed a TAR transparency dispute between the parties. The opinion, however, also reflects an explicit recognition of the value of TAR in effectively managing modern electronic discovery:

Plaintiffs lodged numerous complaints about the pace of discovery and document review, which initially involved only manual linear review of documents. Accordingly, given the volume of documents collected, this Court directed the City to … begin using Technology-Assisted Review (“TAR”) software (also commonly referred to as “predictive coding”) to hasten the identification, review, and production of documents responsive to Plaintiffs’ document requests.1

This preference for the cost-effective efficiency of technology-assisted review even made its way into court-mandated rules of practice in 2018. In July, the New York Commercial Division approved the following rule supporting the use of TAR:

The parties are encouraged to use the most efficient means to review documents, including electronically stored information (“ESI”), that is consistent with the parties’ disclosure obligations under Article 31 of the CPLR and proportional to the needs of the case. Such means may include technology-assisted review, including predictive coding, in appropriate cases.2

Courts are likely to continue to critically consider moving appropriate cases in the direction of technology-assisted review in 2019, particularly with the renewed emphasis on proportionality in the Federal Rules of Civil Procedure. Technology-assisted review has been shown to be more efficient and more economical than traditional linear review and keyword search. As a result, litigants will be hard pressed to continue to employ outmoded review techniques that unreasonably prolong discovery. Proportionality considerations similarly counsel in favor of technology-assisted review, as opposed to traditional, less efficient review methods. While courts cannot mandate the use of TAR in the absence of countervailing considerations,3 those considerations will only grow in importance as document collections swell and courts look for tools to efficiently manage discovery.

Exploring the Boundaries of Cooperation and Transparency

Transparency in technology-assisted review has been a somewhat contentious issue ever since the first written opinion endorsing TAR,4 as courts continue to seek the appropriate balance with a producing party’s ability to fulfill discovery obligations without preemptive restraint.5 2018 was no different, as we saw two decisions in which transparency took center stage — the above-referenced Winfield decision and the Entrata Inc. v. Yardi Systems decision out of the District of Utah.6

In Winfield, the plaintiffs asked Judge Parker to order sweeping disclosures of the city’s TAR workflow and review processes, and production of a sample of several categories of documents determined by the TAR process to be nonresponsive. While noting that “there is nothing so exceptional about ESI production that should cause courts to insert themselves as super-managers of the parties internal review processes, including training of TAR software … in the absence of good cause …,” Judge Parker ordered the city to provide an in camera submission addressing the TAR workflow and review processes in the first instance. Upon review, the court concluded that the submission was protected from disclosure as attorney work product. On the other hand, however, given that there was some evidence of human error in the TAR review process, Judge Parker did order the production of a random sample of nonresponsive documents to “increase transparency.”

In Entrata, Judge Clark Waddoups addressed the interplay between TAR and transparency, in the absence of any agreement on protocol. Despite being aware that plaintiff Entrata would be using technology-assisted review, defendant Yardi Systems essentially passed up several opportunities to influence the TAR process until production was complete. When Entrata refused Yardi’s subsequent demand for TAR disclosures, Yardi moved to compel disclosure of “the complete methodology and results” of Entrata’s TAR process. Yardi argued that Entrata was obligated to provide wholly transparent disclosures as a prerequisite to the use of TAR, and further that this “information was necessary to allow Yardi to assess the adequacy of Entrata’s document collection and review efforts and the completeness of Entrata’s productions …” Judge Waddoups reviewed both the Federal Rules of Civil Procedure and current case law, and rejected Yardi’s position that complete transparency is a precondition to the use of TAR. It is also worth noting that Yardi was unable to point to any deficiencies in the production that may have justified some measure of disclosure by Entrata.

There is little doubt that we will continue to see decisions refining the relationship between transparency and technology-assisted review in 2019. There are no clear and consistent guidelines that can be used to conform conduct. Lack of transparency certainly leads to mistrust between opposing parties, but there is also a legitimate concern that greater disclosure simply provides more fodder for criticism, disputes and expensive motions practice, particularly in asymmetrical litigation. Litigants will be well served to seek cooperative agreements addressing TAR protocols and associated transparency considerations that effectively manage the parties’ expectations and concerns. Doing so will provide a reasonable means of identifying and, perhaps, resolving disputes without judicial intervention. When those efforts fail, however, the courts will ultimately be called upon to strike the appropriate balance.

Focusing on Protocols and Metrics

In 2018, we also saw a growing emphasis on the negotiation, development and use of TAR protocols to manage the parties’ obligations and expectations without the need for extensive judicial intervention. And the substance has shifted from what has traditionally been little more than a general sanction of the use of TAR, to a much more sophisticated, specific and detailed prescription of the procedures and metrics that will govern the TAR process from start to finish.

As one example, the value of specifically addressing methods and metrics in an agreed TAR protocol is exemplified by a dispute that arose in the Domestic Airline Travel Antitrust Litigation.7 In that case, the parties had agreed to a specific method of validation, and targeted a production reaching 75 percent recall or more, but only at a reasonable level of precision. The TAR review resulted in a 3.5 million document production that was erroneously believed (based on certain intrinsic validation figures) to achieve 85 percent recall at 58 percent precision. Using the defined method of validation, however, it was determined that the production actually achieved 97.4 percent recall, but at the expense of precision, which was only 16.7 percent (meaning that there were roughly 2.9 million nonresponsive documents). Given that the expectation of a reasonable level of precision was not met, the court granted a requested extension of discovery — underscoring the value of clearly addressing metrics in a TAR protocol. Despite being widely criticized as a terrible TAR result, and even a “document dump,” the overall production was not unreasonable given the recall-precision trade-off inherent in every TAR review. Had the parties failed to address precision in the TAR protocol, there may not have been any grounds for relief.

This trend will certainly continue in 2019, as parties seek to reach reasonable agreements that clearly define expectations, rather than leaving issues open to dispute and judicial resolution. And, as we gain more experience in managing the specifics of a comprehensive TAR review, the scope of the procedures and metrics that will be addressed in a TAR protocol will only continue to expand.

Moving Beyond Responsive Document Production

Finally, with electronic document collections continuing to grow in every legal corridor, 2018 saw two clear developments that are undoubtedly going to continue into 2019.

First, there has been an effort to more closely align TAR techniques with the goals and risks of reviews that do not demand the level of scrutiny usually associated with document production in litigation — where, for example, privilege review of the entire production may not be quite as critical. Typical scenarios include government productions such as second requests, and productions pursuant to subpoena. In those situations, the goal is to produce the most reasonable, acceptable set of documents (in terms of both recall and precision), while reviewing the least number of documents. This calls for what is essentially a TAR 1.0 training approach, where a limited review can develop a ranking that will achieve a reasonable precision for a given level of recall. While the production set may well exceed the number of documents that would otherwise be reviewed with a TAR 2.0 continuous active learning protocol, the number of documents actually reviewed to generate the production set can be much less.

Second, given the efficacy of continuous active learning, technology-assisted review is being tested and utilized in applications that go beyond traditional production review. TAR is being used as an early case assessment tool to quickly locate the critical documents in what might otherwise be prohibitively large document collections, to allow counsel to develop and advise on litigation strategy at the earliest possible time. And TAR is being used to find the evidence necessary to promptly manage internal, compliance and regulatory investigations, particularly in combination with algorithms that actively seek unknown, contextually diverse documents. And, because continuous active learning is particularly suited for low richness collections, it is also being evaluated as a solution for continuous compliance monitoring.

Ultimately, the use of TAR is going to continue to grow and expand over the next year, both within and outside the traditional litigation applications. And with that will come an increased emphasis on identifying and resolving operational parameters, to facilitate cooperative, efficient and effective use.


1. Winfield v. City of New York , 2017 WL 5664852 (S.D.N.Y. 2017) (emphasis added).

2. New York Commercial Division Rule 11-e(f) (emphasis added).

3. Hyles v. New York City , 2016 WL 4077114 (S.D.N.Y. 2016).

4. Da Silva Moore, 2012 WL 607412 (S.D.N.Y. Feb. 24, 2012).

5. The Sedona Principles: Third Edition, Best Practices Recommendations & Principles for Addressing Electronic Document Production, Principle 6.

6. Entrata Inc. v. Yardi Systems Inc. , No. 2:15-cv-00102 (D. Utah Oct. 29, 2018 )

7 In Re Domestic Airline Travel Antitrust Litigation, No. 15-1404 (D.D.C. Sept. 13, 2018).


About Thomas Gricks

Managing Director, Professional Services, Catalyst. A prominent e-discovery lawyer and one of the nation's leading authorities on the use of TAR in litigation, Tom advises corporations and law firms on best practices for applying Catalyst's TAR technology, Insight Predict, to reduce the time and cost of discovery. He has more than 25 years’ experience as a trial lawyer and in-house counsel, most recently with the law firm Schnader Harrison Segal & Lewis, where he was a partner and chair of the e-Discovery Practice Group.