Treading Past Angels: Finding the Right Search Expert for Your Case

Last February, Assured Guaranty Municipal Corp. sued UBS Real Estate Securities Inc. for breach of contract, accusing the company of failing to meet obligations related to the pooling of residential mortgage-backed securities. As the case moved along, disputes arose over discovery and both sides filed motions to compel.

One of the discovery issues in dispute was the adequacy of the search terms that Assured proposed to apply to electronic documents. Ruling on this issue in a Nov. 21, 2012, memorandum, U.S. Magistrate Judge James C. Francis IV began by quoting the oft-cited words of another U.S. magistrate judge, John M. Facciola, in U.S. v. O’Keefe:

Whether search terms or ‘keywords’ will yield the information sought is a complicated question involving the interplay, at least, of the sciences of computer technology, statistics and linguistics. … Given this complexity, for lawyers and judges to dare opine that a certain search term or terms would be more likely to produce information than the terms that were used is truly to go where angels fear to tread.

Because neither party provided the expert affidavits that would have enabled him to decide on the most efficient search protocol, Judge Francis laid out three options for counsel:

  1. They can cooperate (along with their technical consultants) and attempt to agree on an appropriate set of search criteria.
  2. They can refile a motion to compel, supported by expert testimony.
  3. Or, they can request the appointment of a neutral consultant who will design a search strategy.

Note that in each of the three options, Judge Francis requires that an expert be involved in any determination, even when both counsel come to an agreement out of court.

So, if counsel are required to consult an expert and if search creation and testing is truly an area where “angels fear to tread,” how do you make sure to select the right search expert for your case? Below are some tips you may find helpful in finding the right consultant.

E-Discovery Expert  ≠ Search Expert

Electronic discovery is an extremely broad field of knowledge covering the entire gambit of the EDRM [link:]. While there may be a few e-discovery experts out there qualified to serve as search experts, Judge Franklin makes clear that the expert should have a solid understanding of the interplay between “the sciences of computer technology, statistics and linguistics.”  In the world of academia, this field of study is referred to as “Information Retrieval” (IR). [link:] Those specializing in IR seek to use searches to identify the relevant information from a larger sea of data. When selecting your expert, be sure she has expert knowledge in at least one of the areas of expertise listed by Franklin and foundational knowledge of the others. She should be able to understand how each science plays into the ultimate goal of creating defensible searches.

One Size Does Not Fit All

Like electronic discovery, IR is an extremely broad area of study. In 1992, the Text Retrieval Conference (TREC) [link: ] was created to support the IR community by providing the infrastructure to test and evaluate different text retrieval methodologies.  Since its inception, there have been 21 different research areas, or tracks. These tracks range from web search to spam filtering to gene sequencing. While the Legal Track has been a mainstay for TREC since 2006, it is far from the only IR area of study.

Much of IR focuses on returning the best results from an extremely large dataset. A good example of this would be web search. In web search, the user enters a query and the IR system compiles a list of matching results in a ranked order by likelihood or responsiveness. Users typically find what they are looking for within the first few pages of results and have no need to proceed further.

In contrast, an ideal e-discovery search methodology would need to make an accurate relevance assessment on each and every document in the collection. Instead of finding just the best documents, we need to identify those with even the slightest traces of relevant content. This task poses a unique and what I believe to be a significantly harder task than the majority of IR research.

When selecting a search expert, be sure to find one who is familiar with the intricacies of legal search. While the ideal candidate does not need to have a JD, the expert should be familiar with what is required under the Federal Rules of Evidence and be able to conform her processes to that standard.

Those Who Can, Do; Those Who Can’t, Teach; Those Wanting to Be My Expert, Do and Teach!

It’s obvious that you want to find an expert who has done similar work before and has the references to confirm it. Supporting documentation can be in the form of scholarly articles, prepared client reports, and documents submitted to the court. You want to read through these documents to make sure the information is presented clearly, that a structured workflow was followed, and the results are supported by iterative testing. You also want to make sure that throughout the expert’s career, she has been consistent in her work. Prior publications represent a minefield for prior inconsistent opinions and will likely be used by opposing counsel to discredit your expert

You also want to find an expert who can clearly explain the craft. Most experts are used to operating in a highly specialized academic community with shared foundational knowledge and common vocabulary. A good expert needs to be a teacher at heart. She should enjoy educating others by explaining complex principles in simplified terms.

Have the expert walk you through one of her reports. Make sure she can detail in basic vocabulary each step that was taken, why choices were made, and how the outcome was determined. For any part of the process that requires foundational knowledge, such as statistical sampling, she should be able to break down the process enough that the user can follow along and then return to the bigger picture. Be sure to ask questions, keeping a close eye to make sure she does not appear frustrated or inconvenienced.

Our Catalyst Consulting team of linguists, statisticians, attorneys, and technical experts has been helping clients construct repeatable and defensible search methodologies since 2008. For assistance with your case, or to find out more on selecting the perfect expert, be sure to contact them at

One thought on “Treading Past Angels: Finding the Right Search Expert for Your Case

  1. Pingback: The Many Faces of Mike McBride » Blog Archive » This Week’s Links (weekly)

Leave a Reply

Your email address will not be published. Required fields are marked *