While many users still prefer short, keyword focused queries, the rise of AI has made it more common for users to send long grammatical queries and expect focused answers. Lexical search has long struggled to deliver good search relevance for these types of queries. Either omitting good matches because of unimportant words or ordering poorly because of how terms are weighted.
Take, for example, a query like “what treatments are available for pneumonia”. As a human, it’s easy to parse this and identify the key terms. A search for “pneumonia treatment” would probably give the correct results in a typical TF/IDF system. However, the search engine has no way of knowing which words are important and which aren’t without help, and “treatments available for smallpox” is a poor match when “pneumonia treatment” documents exist. If other factors like date or popularity are included in scoring it can be even harder to find unpopular relevant documents over popular irrelevant ones.
To address this, we can identify which parts of the user’s search are important to the user’s intent and carefully boost those to ensure their importance in the result set. Building a system to accomplish this takes several steps.
Identifying Important Terms
What Makes a Term Important to Query Intent?
For humans, identifying important terms is an automatic part of understanding language. To have a repeatable system, it helps to build explicit criteria. While no criteria will be perfect, the set below provides some functional guidelines.
Represents a well understood concept
A word like “flu” has a well understood meaning. “Testing”, is fairly clear but has a broader reach. A word like “thing” is generally too broad to be useful.
Is Not Highly Context-Dependent
A term like “swimming pool” generally stands on its own, while the meaning of “glass” can change a lot depending on context: “water glass”, “glass door”, “glassy stare”. Independent terms will be more consistent targets for boosting.
Cannot be removed Without Changing Query Intent
In “what treatments exist for smallpox”, the terms “treatment” and “smallpox” are essential, but the other can be removed without significantly changing the meaning.
Usage in the documents matches user expectations
If users search for a “weed whacker” but relevant documents use “string trimmer” then “weed whacker” is a poor choice without synonyms or similar interventions.
Appears Consistently in Relevant Documents
If a user is looking for a “quiet dishwasher” but only half the dishwashers actually include text about noise levels (or do mention noise level but use terms besides “quiet”) then boosting “quiet” will have a more random than beneficial effect. Such a term is a poor candidate.
Building an Important-Terms List
Knowing what makes a term important is only half the battle. We also have to create a process to identify them within a user’s search. That process should also be easy and fast to avoid long query times or errors during the search process.
The most effective way I found to efficiently identify important terms is to generate and review a list of candidate terms offline and simply compare the user’s search to that list. That way deciding term importance doesn’t need to occur within the latency requirements of the search system.
For building the list, that can be as simple as taking popular queries from the query logs and having subject matter experts manually review them. While this can take some time, it’s largely a one-time investment since queries drift slowly over time in most domains. A manual review also ensures that important terms in the list actually match document text and what users are searching. Manual changes (adding or removing terms) also become easy to accomplish.
Once you have a list of important terms, the next question is how to actually change search results with it.
Boosting Important Terms
How Important-Term Boosting Should Affect Relevance
To get a positive change from the search results we want to increase the influence of the important terms relative to the unimportant ones in the calculation of search scores. We also want to promote documents that match as many of the important terms as possible, since dropping important terms changes the intent of the query. However, we don’t want to completely ignore other terms since identifying important terms will never be perfect and less important words may still add useful context.
As an example, the query “what games are available for Xbox” would get worse if more xbox consoles or more games for non-xbox consoles creep into the results. Focusing on “xbox” and “games” together as important terms and reducing the influence of less important parts like “are available for” will give us results that better match the user’s intent.
Building the Important-Term Boosting Formula
Time for the fun part: Math! Boosting on individual term matches is pretty easy, but we want to increase the effect based on matching more important terms and decrease it when matching fewer. That means we’re looking for a function that increases when any individual term score increases, but increases a lot more if all terms increase. One common function that looks like this is the geometric mean.
Arithmetic mean: Amean(a,b,c,...) = (a+b+c+...)/n
The geometric mean is similar to the arithmetic mean that most people are familiar with, with n being the number of components. When all components have the same value, then Gmean(a,a,a,...) = Amean(a,a,a,...) = a, and both means are equal. However, the geometric mean is much more sensitive to one component being smaller: Amean(a,a,a,a/16) ~= 0.77a, while Gmean(a,a,a,a/16) = 0.5a, which we want. The one problem is that if any component is zero, then the entire geometric mean is also zero. That we don’t want, so we need to modify it slightly. A simple fix for this is to set a minimum value of 1 for each component. Then the boost won’t get zeroed out, but will still increase significantly for matching multiple components.
The last thing we need for a final formula is some way to adjust the overall score. Organic search scores tend to vary wildly based on configuration, so we need a factor we can adjust to give the boost the right amount of influence without totally taking over. Adding that in gives us a final formula:
Adding Intent-Aware Boosting to Search
To bring everything together, we need a way to introduce this boosting formula into our search engine so that it has the effect we want on searches.
Solr Edismax
The Apache Solr search engine provides the edismax request type which has many built in customization options for handling search. The bf query parameter is a shorthand for adding a function query to the score and makes it easy to add our calculated boost to an existing query.
To set the boost up for an edismax query, just add the following parameters:
&bf=product(<tuningFactor>,pow(product(max(1,$term1),max(1,$term2),...),div(1,2)))
&term1=query({!edismax bf='' boost='' v='"<term1>"~2'})
&term2=query({!edismax bf='' boost='' v='"<term2>"~2'})
To break that down:
- The “bf” parameter means the output will be added to the score.
- The “tuningFactor” is a numeric value used to change how much influence the boost has. Higher value means more boost influence.
- The section starting with “pow” calculates the modified geometric mean as outlined in the previous section.
Each &termX is a reference to the corresponding &termX query parameter. This is both useful for organization to show terms outside of the whole equation, and necessary to avoid parsing issues with nested localparameter syntax (the “{!...}” parts)
For each term, the “query(...)” section directs Solr to calculate scoring values for the given term using the existing edismax parameters. Notably, the “bf” parameter is removed to prevent recursion, and the “boost” parameter (used for multiplicative modifications to the score) is removed to prevent applying it twice.
Solr Function Query
When using the basic Lucene-based parser in Solr, boosting documents can still be accomplished but requires a bit more complicated setup. The basic form is to use a function query with the same calculation, but the query section should be modified to match normal system queries.
{!func}product(<tuningFactor>,pow(product(max(1,query(<query-for-term1>)),max(1,query(<query-for-term2>)),...),div(1,2)))This type of setup is less common than an edismax-based search for most installations, but does allow greater control over exactly how a search is processed.
Elasticsearch and OpenSearch
Unfortunately, neither Elasticsearch nor OpenSearch have built in support for using functions to combine multiple sub-queries. It is possible to use the dismax operator to add the results of additional searches to the main search by setting the “tie_breaker” parameter to 1. This does boost the important terms, but loses the factor that promotes documents matching more terms over those matching fewer. A similar additional only version was tested and showed some benefit but much less than the full system.
Comparison with other methods
Natural Language Processing (NLP)
At first glance, Natural Language Processing might seem like a natural choice to identify important words. As its own discipline, there are a wide array of tools to accomplish a variety of different tasks. There are two major problems applying these tools, though. First, many queries aren’t structured as natural language making these tools less reliable. Second, the structure of language is often unreliable in identifying important words for search. It’s relatively easy for modern tools to identify parts of speech and extract named entities. However, ignoring “red” and “iphone” in “red iphone case” because they operate as adjectives would make the search far worse. Beyond that, importance is highly domain specific. It takes knowledge of the actual data to say whether “red” in a search is important to the user's intent and actually present on the relevant products. It’s probably better to bring back rose-gold iphone cases than red android cases or red iphones.
Vector and Hybrid Search
Handling long queries and inexact language were two of the driving goals behind the development of vector search. While it can be extremely effective at handling some of the cases that the proposed lexical boosting method seeks to address, it also has its own challenges and trade-offs, notably slower query times, higher resource utilization, and difficulty with exact term matching. This method is intended to expand the capabilities of lexical search relevancy in a way that can be easily added to existing systems. That means that it can improve lexical relevancy in a hybrid search context.
AI-Assisted Candidate Generation
The suggested method is to get candidate queries from logs and manually review them to identify important terms. AI tools can be used in either part of this process, though there are tradeoffs. Anyone who has looked through user query logs knows that they tend to be much odder than expected. Generating candidates via AI will give a more tightly focused set of terms with many of the true rough edges missing. Additionally, if it’s not provided significant amounts of data, it may generate many candidates that don’t match the actual search domain.
As for using AI to judge whether a term is important, it can handle general language factors well, but may struggle with data specific concerns. As an example, it can probably identify that “refrigerator” is more important than “back”, but might miss that half of the refrigerators in the data are labeled “fridge” and will miss the resulting boost. A manual review by subject matter experts will be slower, but is more likely to uncover these kinds of edge cases, either to change which terms are considered important or to resolve the issues through search configuration changes.
Conclusion
Carefully identifying and boosting important terms from a user's query can help preserve query intent and improve search relevance for searches that are often poorly handled by traditional lexical search systems. While not a cure-all, these benefits can be achieved with minimal compromise in either system complexity or query performance. By pushing the query towards the user’s intent and away from naive text matching, search engines can significantly improve the user experience and help them find what they were looking for, rather than the text they used to ask.