With the rise of TypeSafe AI’s JEV and “system one” models that provide fast, cheap, and well-typed output changing the landscape of AI in search, it seemed important to do some practical testing of how these models would perform for various search use cases. Characterizing queries, identifying essential terms in a user’s query, and assigning queries and products to categories are all search tasks that could benefit from a JEV-like model either in accuracy or cost to implement. Obviously this isn’t an exhaustive list of use cases (could such a thing even exist?), but provides a window into what’s currently ready, what has potential, and what still needs improvement before it’s ready for production.
Characterize Queries
The Search Problem
One common problem in search is that different queries have different priorities. Searches for model numbers or highly specific products do best with highly weighted text matches. Broad category searches (like “refrigerators”) do better with strong boosting for other peripheral factors like user reviews or purchases. Separating which search is which and providing appropriate boosts or filters can improve search results over using one compromise setup for all queries.
To test how well JEV handles this kind of query characterization without relying on specific relevancy tuning, I had it identify whether a query was for a product or a product’s accessory. This is a common issue in lexical product search since an iPhone and an iPhone case both match a search for “iPhone”.
Test Approach and Results
Asking straightforwardly “Is this search looking for an accessory or product?” had mixed results. It correctly caught most straightforward queries like “phones” vs “phone cases”, “cables”, and “appliances”, but it struggled a bit with more complicated and ambiguous cases like lenses for cameras and monitors for computers (the former potentially conflicting with a search for the camera and the latter largely standing on its own and not being labeled as an accessory). Providing additional examples (“camera lenses are accessories”, “computer monitors are products”) in the prompt did allow JEV to correct these examples. That responsiveness is important since exactly what is and isn’t an accessory changes between domains.
Altering the prompt to “Is this search looking for a product that has accessories?” also fixed the monitors being labeled as an accessory and caught additional cases like “laptop” which could have confounding search results (“laptop bag”, “laptop riser”). This was closer to the intended use case to separate out products and accessories that match the same search. It wasn’t perfect though, with “fridge” being labeled as not having accessories even though fridge filters commonly show up highly in search results because they sell more often than fridges.
What it Means
First pass accuracy was decent but incomplete. Careful prompting significantly improved how well JEV characterized queries to the point where finding incorrect cases wasn’t easy. If you want to use JEV to characterize your queries, make sure you check how accurate it is on your data and carefully consider the prompts you use. Also make sure that the search benefits are worth the additional expense, complexity, and latency of including JEV in your query-side application stack.
Identifying Important Terms in Search Queries
The Search Problem
When users use longer, more natural language searches, they include terms that aren’t important to a lexical search. “What is the most quiet dishwasher” has 6 individual terms, but only “quiet” and “dishwasher” actually carry significant meaning. If the search engine requires and scores all of them, it’ll likely miss many important documents. If it only requires some of them, many irrelevant matches can creep in. Even with just the terms “quiet dishwasher”, it’s much better to show dishwashers that don’t mention being quiet than quiet appliances that aren’t dishwashers. Identifying the importance of query terms allows for both weighting essential terms more strongly, and relaxing the requirement that unimportant terms match.
Test Approach and Results
JEV performance was tested by taking a set of queries, tokenizing them on spaces, and building JEV noul queries for each term to assess its importance. Whitespace tokenizing was used to match how most query parsers interpret searches.
Overall results were mixed. While it did a good job weighting common unnecessary words like “which” and “are” low, it struggled to differentiate between essential products like “dishwasher” and fuzzy descriptors like “quiet”.
“Which dishwashers are quiet”
| Term | which | dishwashers | are | quiet |
|---|---|---|---|---|
| Score | 0.42 | 0.8 | 0.23 | 0.77 |
“A smart watch with a red band”
| Term | a | smart | watch | with | red | band |
|---|---|---|---|---|---|---|
| Score | 0.19 | 0.81 | 0.89 | 0.54 | 0.77 | 0.73 |
“Nintendo Switch game”
| Term | nintendo | switch | game |
|---|---|---|---|
| Score | 0.77 | 0.79 | 0.55 |
“Gas oven range”
| Term | gas | oven | range |
|---|---|---|---|
| Score | 0.83 | 0.79 | 0.55 |
“HDMI cable”
| Term | hdmi | cable |
|---|---|---|
| Score | 0.88 | 0.79 |
What it Means
While it’s possible more advanced prompting could give better results, this use case doesn’t seem ready for production. While there’s certainly some benefit to detecting and removing common words, the danger of mis-emphasizing terms seems likely to outweigh the benefit even before considering the costs of implementation and maintenance.
Product and Query Categorization
The Search Problem
Grouping products into categories has long been a way of helping users understand a vast catalogue of options. Ensuring that categorizations are accurate and well utilized can be a big help to search. There are two main ways to use JEV with categories.
First, JEV can be used to assign products to categories, either adding new products to a taxonomy or validating products that already have categories assigned. This can reduce human effort and help improve category consistency.
Second, JEV can be used to assign queries to categories to boost the relevant products. This requires evaluating each user query and modifying the search engine call to boost results that match the chosen category.
Test Approach and Results
To validate how well JEV can assign categories to products, I took a set of tech product data and had JEV assign a top level category from the list (approximately 20 total) to each product based on the product name. These assignments were then compared to the existing category assignments in the data.
| Uncertainty Threshold | # Agreeing | # Disagreeing | # Unsure | % Agreeing | % Disagreeing | % Unsure |
|---|---|---|---|---|---|---|
| None | 36176 | 6372 | 0 | 85.0% | 15.0% | 0% |
| 0.7 | 34511 | 3659 | 4378 | 81.1% | 8.6% | 10.3% |
| 0.8 | 33356 | 2515 | 6677 | 78.4% | 5.9% | 15.7% |
| 0.9 | 31142 | 1521 | 9885 | 73.2% | 3.6% | 23.2% |
| 0.95 | 27746 | 1031 | 13771 | 65.2% | 2.4% | 32.4% |
Manually reviewing the results showed that many of the “incorrect” categorizations were in areas of ambiguity like “Audio” vs “TV and Home Theater” vs “Musical Instruments”. This is important in two different ways. When assigning products to a category, small ambiguous differences are unlikely to have a large effect on overall search quality, especially since humans often disagree about the best category for some products.
When assigning queries to a category, this variability is more complicated. Boosting the best matching category can help push less relevant products like accessories or accidental matches further down the results, de-cluttering what the user sees. However, if JEV assigns a reasonable category but doesn’t match the category of the desired products (think “audio” vs “TV and Home Theater”), adjusting the search to favor that category may actually make results worse.
What it Means
JEV showed strong performance in identifying product categories. Matching 85% of products with no uncertainty filtering and matching 65% with only 2.4% errors with high uncertainty filtering is impressive. When assigning products to categories this could save a lot of time and effort by handling most the load and only requiring review on a fraction of the products. It could also be used as quality control to validate that existing category decisions are accurate.
Using JEV to identify query category is a bit more nuanced. Since it’s part of the query pipeline, there’s no opportunity for human intervention. That means that missed categories harm search results rather than just taking extra time. JEV also has to match the existing categorization even if it’s poor in order to boost the right results. The 2.4% error rate with uncertainty filtering is quite low, giving a good chance of a net positive change on search depending on search configuration and queries. Testing would be required to show that the searches it improves outweigh the searches it detracts from, especially since difficult and ambiguous cases it’s intended to fix tend to be difficult for both search and JEV categorization.
Additional Notes
Reliability
While interactive testing against the JEV API never ran into problems, the batch job to assign categories to products failed multiple times with error messages of high load from TypeSafe. While this is likely growing pains from their sudden spotlight, it’s something to keep an eye on when planning an integration into existing systems.
Alternative models
To test how a smaller open-source model performed in comparison, the same category test used to evaluate JEV was performed on Laya to test how viable it was as an alternative. The results were pretty striking with Laya getting only 41.5% correct with no uncertainty threshold, and an uncertainty threshold of 0.9 giving 22.4% correct, 8.13% incorrect, and 69.4% uncertain. Other models may perform better, and fine tuning Laya may provide a significant benefit, but out-of-the-box performance showed a pretty clear divide.
Overall Impression
The capabilities of TypeSafe’s JEV model are definitely intriguing. Overall practical testing for this set of search use cases had mixed results. Offline assignment of products to categories showed a strong win, while the benefit of query side characterization or categorization seemed to be more dependent on architecture and domain. In general, offline uses with human review or low cost of incorrect answers will likely see a benefit from using these models. In situations with a higher cost to errors, testing will be needed to ensure that deploying JEV or models like it provides a net benefit to search.