Using LLMs to Identify Problematic Search Queries

If you’ve worked in search, you’ve probably had people tell you that the search system you’re overseeing is “bad”. Search is hard, and poor results happen sometimes even in well tuned systems. If you’re lucky, you might get some examples with the complaint. More commonly, it’s based on the overall feeling of people who use the system.

Tuning search is an important process, but it can easily get bogged down evaluating small differences in relevance between documents. Experts often disagree about exactly which document is the best response for a given query. What gets far less debate is what makes search results “bad”. No documents returned, irrelevant documents, essential documents missing, bad ordering, all of these are much easier to detect than small differences in relevancy.

In a search system, finding and fixing these bad queries can make a much bigger difference in the overall user experience of search than making small tweaks to improve mostly good results. Finding those problematic queries, however, is its own challenge though not an insurmountable one. Thankfully, Large Language Models (LLMs) can help.

Traditional Search Relevance Tuning

The typical search tuning process involves several steps:

First, problematic queries have to be identified as a starting point. In some cases particular bad searches kick off a round of tuning and a short list is already prepared. Other times a significant number of queries need to be reviewed to get a starting list. Usually several well performing queries need to be checked in order to identify one poor performing one.

Second, the list of poor performing queries has to be investigated to identify why search failed and how it can be improved. Common improvements involve adding synonyms to allow similar words to match (fridge/refrigerator), changing how text is analyzed, adjusting field weights, or adding boosting based on non-text factors like date or reviews.

Once modifications are complete, a sample of search results needs to be reviewed to ensure search quality didn’t degrade in other areas. This often involves a manual effort of testers trying various searches and comparing the results between the new and old systems as a spot check. A more complete version of relevance evaluation involves maintaining a defined set of queries with recorded human judgments about the results and algorithms to provide overall metric scores. Maintaining such a system takes considerable effort, so in practice it’s not appropriate for all use cases.

Using Large Language Models (LLMs) for Search Relevance Evaluation

Manually reviewing large sets of search terms is expensive in terms of time and attention. Using an LLM to give an approximate relevance score for each query can help focus efforts on the most likely candidates and reduce overall effort. In the search for problematic queries to fix, this can eliminate most of the manual testing of valid queries and get reviewers to a set of bad queries quickly.

Additionally, when evaluating results after a change, LLM scores can help identify areas of change for manual review and evaluation. This takes advantage of the strengths of both approaches. The LLM can quickly score queries where exact precision is less critical, while human reviewers provide a more thorough and nuanced evaluation of the areas showing poor performance or meaningful change.

How the LLM Relevance Evaluation Pipeline works

The first step in applying this in practice is to build a list of candidate searches. Pulling examples from logs or existing metrics systems is an easy way to get a good representative sample to start with. A non-representative sample (say management’s favorite list of terms) is still useful for finding examples of poor results, but won’t be as reliable for checking the overall impact of changes.

Once you have a list of queries, the next step is to fetch search results for each term. How many results to fetch and which fields to use should be determined based on the needs of your specific domain. That information is then sent to the LLM to generate a score for how relevant the document is for this search.

Finally, those individual document scores need to be combined into a single overall score for the search. There are many ways to do this, but the most consistent options are either to average the scores, or to calculate a search result specific metric like Discounted Cumulative Gain which accounts for order.

Once you have all that in place, simply look at the low scoring searches and investigate why their results are poor. With a bunch of examples, you’ll start to see overarching themes you can use to tune your search results, as well as fixing individual problem areas affecting your users.

With changes decided and set up in a functioning environment, you can then re-run the scoring on the same starting list of terms. Comparing the pre and post change scores will show what searches were most affected by your changes. This lets you review only the major changes to see what improved, what got worse, and whether the changes are a benefit overall.

Testing LLM Search Relevance Evaluation with eCommerce Data

To validate this approach, I used a set of ecommerce sample data. Queries were manually generated and scored with an eye towards searches with poor results to better check LLM effectiveness and limit manual evaluation. Two example searches with poor results were “bose earbuds” which brought back replacement tips instead of full earbud headphones, and “pixel” which brought back pixel themed gift cards over the phones.

While there are many possible Large Language Models to choose from, my testing used the open source cross-encoder/ms-marco-MiniLM-L6-v2 from Hugging Face to represent smaller open source models, and gpt-5.6-luna from OpenAI to represent a larger, newer, commercial model. A basic prompt was used to get Luna to evaluate the results and provide a single score. As a cross encoder, the Hugging Face model inherently provided a relevance score as output.

Scores from multiple documents were combined using max, average, and Discounted Cumulative Gain methods. These were compared to the manually curated scores using both correlation coefficients, and more directly by setting an appropriate threshold and calculating counts of true positive, false positive, and false negative results. A total of 72 queries were manually scored with 34 being considered “poor” results of various strengths.

To explain, the final row shows the OpenAI model using DCG scoring flagged 29 queries of which 26 were poor performers and 3 were actually fine. It missed an additional 8 poor performing queries. So, in this case, a reviewer would have reviewed 29 queries found 26 problematic ones rather than reviewing 72 queries and finding 34.

What it means

The more powerful OpenAI model performed significantly better overall in this test. The smaller Hugging Face model still provided a significant benefit over a full manual review of terms, though its performance wasn’t quite as strong as the OpenAI model.

In terms of combination algorithm, DCG showed a clear advantage in matching manual scoring. That’s unsurprising since it can account for mis-ordering as well as low overall scores.

While the manual scoring and small sample make this example far from statistically rigorous, it still shows that the technique provides a benefit in finding poor performing queries over a manual review. Using a more powerful model will provide better performance, though smaller and lighter-weight models are still beneficial. As always, with your data and queries you should expect different performance. Every search is unique after all.

Why this is Relevance Triage, Not an Overall Search Metric

Some search evaluation tools use Large Language Models to calculate overall metrics for how search is performing. These may be useful to get a general sense of how search is performing, but caution should be taken when treating these values as an objective measure of overall search quality.

Accounting for non-relevance factors like recency, user reviews, and brand trust may be difficult to accomplish accurately with Large Language Models even when the data is provided. Additionally, custom tuning of LLMs is extremely uncommon, and general purpose LLMs may not be fully aligned with your particular use case. Ultimately, if you assume the LLM is correct then you’ll tune search to the LLM’s expectations which are probably a bit different than the expectations of your users.

The process described here is designed to use the LLM as a first pass to identify areas for humans to review. This reduces the effect of bias or hallucinations while still saving time and effort compared to a manual tuning process. Having a human in the loop also allows the reviewer themselves to learn where the LLM’s scoring performs well and where it fails, allowing them to more effectively use it as a tool.

Using LLMs to Focus Relevance Work

LLM-assisted relevance evaluation does not eliminate the need for experienced human judgment. It makes that judgment easier to apply where it is most valuable. By filtering a large query set into a smaller collection of likely problems, a language model can help relevance practitioners spend less time confirming that acceptable searches are acceptable and more time understanding and correcting the searches that are not.