In A Milestone for Predictive Coding, Judge Peck Says, ‘Go Ahead, Dive In!’

Lawyers and predictive coding are like kids around the swimming hole — no one wants to be the first to dive in for fear the water is cold or it harbors scary creatures. But once someone takes the lead, dives in and declares the water fine, everyone else is quick to follow.

That is why U.S. Magistrate Judge Andrew J. Peck’s opinion published Friday marks a major milestone for the use of predictive coding in e-discovery. It is the first judicial opinion in which a court has expressly approved the use of computer-assisted review.

Just last October, a prescient Judge Peck published an article in which he described (metaphorically speaking) the lawyers standing around the predictive-coding swimming hole:

To my knowledge, no reported case (federal or state) has ruled on the use of computer-assisted coding. While anecdotally it appears that some lawyers are using predictive coding technology, it also appears that many lawyers (and their clients) are waiting for a judicial decision approving of computer-assisted review.

Well, on Friday he gave them what they were waiting for. “This judicial opinion now recognizes that computer-assisted review is an acceptable way to search for relevant ESI in appropriate cases,” he wrote in Da Silva Moore v. Publicis Groupe.

Predictive Coding Delivers Precision and Value

We wrote about this case just two weeks ago, recounting Judge Peck’s on-the-record colloquy about predictive coding with counsel and their e-discovery consultants. Near the end of that hearing, after discussing the issue at length, Judge Peck stated, “This may be for the benefit of the greater bar, but I may wind up issuing an opinion on some of what we did today.” If he did issue an opinion regarding predictive coding, I said then, it would no doubt be a milestone in the industry’s adoption of the technique.

In the opinion, Judge Peck readily concedes that his decision to allow predictive coding in this case was relatively easy, given that the parties agreed to its use. Their disagreement hinged on how to implement predictive coding, not whether it should be used.

Even so, he forcefully makes the case for predictive coding and computer-assisted review. In a section of the opinion titled, “Further Analysis and Lessons for the Future,” he describes why predictive coding delivers greater precision and value than either manual review or keyword searching.

The good old-fashioned method of linear, manual review is simply too expensive to be practical in cases with large numbers of documents, Judge Peck notes. In Da Silva Moore, for example, there are over 3 million emails. He then goes on to challenge the “myth” that manual review is the “gold standard.”

Moreover, while some lawyers still consider manual review to be the “gold standard,” that is a myth, as statistics clearly show that computerized searches are at least as accurate, if not more so, than manual review. Herb Roitblatt, Anne Kershaw, and Patrick Oot of the Electronic Discovery Institute conducted an empirical assessment to “answer the question of whether there was a benefit to engaging in a traditional human review or whether computer systems could be relied on to produce comparable results,” and concluded that “[o]n every measure, the performance of the two computer systems was at least as accurate (measured against the original review) as that of human re-review.”

Likewise, Wachtell, Lipton, Rosen & Katz litigation counsel Maura Grossman and University of Waterloo professor Gordon Cormack, studied data from the Text Retrieval Conference Legal Track (TREC) and concluded that: “[T]he myth that exhaustive manual review is the most effective – and therefore the most defensible – approach to document review is strongly refuted. Technology-assisted review can (and does) yield more accurate results than exhaustive manual review, with much lower effort.” … The technology-assisted reviews in the Grossman-Cormack article also demonstrated significant cost savings over manual review: “The technology-assisted reviews require, on average, human review of only 1.9% of the documents, a fifty-fold savings over exhaustive manual review.” (Citations omitted.)

The Limits of Keyword Searching

Next, Judge Peck reviews the limitations of keyword searching. Keyword searching has its place, he acknowledges, to cull data to a more manageable volume and to identify documents for predictive-coding seed sets. But far too often, he writes, “the way lawyers choose keywords is the equivalent of the child’s game of ‘Go Fish.'” (Judge Peck cites Ralph Losey’s blog for the analogy. We’ve used it here in a different context.)

The requesting party guesses which keywords might produce evidence to support its case without having much, if any, knowledge of the responding party’s “cards” (i.e.,the terminology used by the responding party’s custodians). Indeed, the responding party’s counsel often does not know what is in its own client’s “cards.”

Another problem with keyword searching is that it is often over-inclusive, finding large numbers of irrelevant documents, Judge Peck writes. Even worse, keyword searches are not very effective, he points out. He cites the landmark e-discovery study by David C. Blair and M.E. Maron, An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System, 28 COMMUNC’NS. OF THE ACM 289, 295 (1985). In that study, the attorneys were confident that their searches had found more than 75% of the responsive documents.  But they were wrong.  In fact, the searches had only found 20% of the relevant documents. More recent TREC Legal Track studies have corroborated these results.

Given the weaknesses in both manual review and keyword searching, Judge Peck concludes:

Computer-assisted review appears to be better than the available alternatives, and thus should be used in appropriate cases. While this Court recognizes that computer-assisted review is not perfect, the Federal Rules of Civil Procedure do not require perfection. … Courts and litigants must be cognizant of the aim of Rule 1, to “secure the just, speedy, and inexpensive determination” of lawsuits. Fed. R. Civ. P. 1. That goal is further reinforced by the proportionality doctrine set forth in Rule 26(b)(2)(C).

Returning to the case at bar, Judge Peck lists the the factors that led him to conclude that predictive coding was appropriate:

  1. The parties’ agreement.
  2. The vast amount of ESI to be reviewed (over 3 million documents).
  3. The superiority of computer-assisted review to the available alternatives (i.e., linear manual review or keyword searches).
  4. The need for cost effectiveness and proportionality under Rule 26(b)(2)(C).
  5. The transparent process proposed by defendants.

With reference to the transparency of the process, Judge Peck stresses that this was a key factor in his decision, describing himself as a strong supporter of the Sedona Conference Cooperation Proclamation. While not all e-discovery counsel will be as transparent as counsel were in this case, “such transparency allows the opposing counsel (and the Court) to be more comfortable with computer-assisted review, reducing fears about the so-called ‘black box’ of the technology. This Court highly recommends that counsel in future cases be willing to at least discuss, if not agree to, such transparency in the computer-assisted review process.”

Other ‘Lessons for the Future’

Magistrate Judge Andrew Peck

Judge Peck wraps up his “Lessons for the Future” with four additional points:

  • It is unlikely that courts will be able to approve a party’s proposal as to when review and production can stop until the computer-assisted review software has been trained and the results are quality control verified. Only at that point can the parties and the court see where there is a clear drop off from highly relevant to marginally relevant to not likely to be relevant documents.
  • Staging of discovery by starting with the sources and custodians that are most likely to be relevant–without prejudice to the requesting party seeking more after conclusion of that first stage review–is a way to control discovery costs.
  • In many cases, the requesting party’s client will have knowledge of the producing party’s records, either because of an employment relationship or because of other dealings between the parties. Lawyers should be sure to use their client’s knowledge of the opposing party’s custodians and document sources. Think of cooperation to mean “strategic proactive disclosure of information,” Judge Peck advises. “If you are knowledgeable about and tell the other side who your key custodians are and how you propose to search for the requested documents, opposing counsel and the court are more apt to agree to your approach.
  • It was helpful that the parties had their e-discovery vendors present at the court hearings where the ESI protocol was discussed, Judge Peck says. “Even where as here counsel is very familiar with ESI issues, it is very helpful to have the parties’ e-discovery vendors (or in-house IT personnel or in-house e-discovery counsel) present at court conferences where ESI issues are being discussed. It also is important for the vendors and/or knowledgeable counsel to be able to explain complicated e-discovery concepts in ways that make it easily understandable to judges who may not be tech-savvy.”

What This Means for E-Discovery

No one need wonder what this opinion means for the future of e-discovery, because Judge Peck answers that question himself.

The opinion does not mean that computer-assisted review must be used in all cases, he says. Nor should the opinion be considered an endorsement of any particular vendor or of any particular review tool.

What the Bar should take away from this Opinion is that computer-assisted review is an available tool and should be seriously considered for use in large-data-volume cases where it may save the producing party (or both parties) significant amounts of legal fees in document review. Counsel no longer have to worry about being the “first” or “guinea pig” for judicial acceptance of computer-assisted review. As with keywords or any other technological solution to e-discovery, counsel must design an appropriate process, including use of available technology, with appropriate quality control testing, to review and produce relevant ESI while adhering to Rule 1 and Rule 26(b )(2)(C) proportionality. Computer-assisted review now can be considered judicially-approved for use in appropriate cases.

In other words, Judge Peck has taken the dive into the scary pool of predictive coding and now he is beckoning to all of us, “Come on in, the water’s fine!”


About Bob Ambrogi

Bob is known internationally for his expertise in the Internet and legal technology. He held the top editorial positions at the two leading national U.S. legal newspapers, the National Law Journal and Lawyers USA. A long-time advisor to Catalyst, Bob now divides his time between law practice and media consulting. He writes two blogs, LawSites and MediaLaw, co-authors's Legal Blog Watch, and co-hosts the weekly legal-affairs podcast Lawyer2Lawyer. A 1980 graduate of Boston College Law School, Bob is a life member of the Massachusetts Bar Foundation and an active member of the Massachusetts Bar Association, which honored him in 1994 with its President's Award.