The Challenge
An archives organization relied on search to provide access to approximately 96 million documents. Users experienced slow searches, intermittent failures, and inconsistent results. Some content could not be found even when the searched words appeared in the original document.
The environment combined a distributed search platform with custom query processing and application-level load balancing. Although most queries completed quickly, certain searches ran for minutes. Troubleshooting required determining how user-interface behavior, application code, connection management, and search configuration contributed to the problems.
Innovent’s Approach
Innovent began by investigating inconsistent results, a visible problem that undermined users’ confidence in the system. The team reproduced the behavior and found that it occurred primarily on later pages of search results. Comparing those requests with search logs revealed a critical clue: the reported number of matching documents dropped by almost half.
The collection was divided across two groups of search nodes, and affected requests were reaching only one half of the data. Innovent traced the behavior to defects in the custom load balancer’s tracking of available nodes. Different instances disagreed about which nodes were available, causing users to receive different results for the same search. By the end of the first week on-site, code changes addressing the problem were under review.
The investigation also identified how performance problems contributed to apparent outages. The search interface retained a default search expression when users began typing, unintentionally turning ordinary searches into expensive queries. In controlled testing, affected searches took a minute or longer with the expression present and approximately half a second when it was removed.
Long-running requests exhausted undersized application connection pools. The custom load balancer interpreted connection errors as evidence that search nodes were unavailable, excluding healthy nodes and concentrating traffic on the remaining nodes. Because application requests explicitly selected which nodes to query, those exclusions could also produce incomplete results. Innovent corrected the interface behavior, connection handling, and routing so requests could use the search cluster’s own coordination capabilities.
A separate investigation traced missing content to custom text-processing filters. One corrupted words adjacent to encoded whitespace; another removed text enclosed in braces. Innovent corrected the processing and reindexed affected content so those passages could be retrieved.
Throughout the engagement, Innovent combined log analysis, application-code review, and controlled testing to isolate interacting defects. Monitoring dashboards helped the team correlate query behavior with request failures and traffic across search nodes. Alongside the immediate corrections, Innovent addressed expensive custom query behavior and advised on replacing legacy query-processing functionality to reduce architectural complexity.
The Result
Production changes corrected the identified causes of incomplete and inconsistent results, removed interface behavior that generated unnecessary query load, and resolved connection-handling defects that caused healthy nodes to be excluded from service. Corrections to text processing and reindexing also restored searchability to affected content.
The organization gained better visibility into search health through monitoring dashboards and a clearer path toward a more maintainable architecture.
The engagement demonstrated Innovent’s ability to trace user-visible search problems across application and search-engine boundaries, identify how individual defects amplified one another, and implement targeted corrections across the system.