The Challenge
A human capital management services provider relied on search applications to help customer service representatives find information and serve clients. Slow query response times and outdated data made it harder for representatives to quickly access the information they needed.
The client initially suspected that several queries needed optimization. Investigation revealed a broader problem: the existing data model and assumptions about how information would be retrieved contributed to an inefficient use of streaming queries and high latency.
After the initial review, the organization needed a robust enterprise-wide indexing solution spanning three separately maintained data centers. The pipeline had to support hundreds of millions of documents, full and partial updates, and data streams from multiple locations without placing excessive demands on source databases that also supported other critical operations.
Innovent’s Approach
Innovent began with an advisory and remediation engagement to investigate the slow searches. The team traced the latency to the interaction between the data model and the way streaming queries were used. Innovent recommended a nested data model better suited to the relationships in the data and the application’s search requirements.
The team prototyped the proposed model and tested it at scale, confirming that queries responded well. This gave the client a basis for moving beyond isolated query changes to a broader architectural solution.
In a second phase, Innovent provided architecture and development services to design and build a distributed search indexing architecture across the three data centers. The design brought full-load processing, incremental updates, data transformation, and cross-site coordination into a single pipeline.
The pipeline used a replicated staging environment to reduce repeated demands on the source databases. Data was collected and transformed into the structure required for search, then made available to indexing processes in each data center. This allowed the search indexes to be updated independently while reducing demands on the source systems.
For partial updates, applications consumed events from Kafka and applied changes to the staging data. Federated messaging coordinated transformation and indexing work across locations, allowing updates to reach each data center without requiring the applications to address one another directly.
Innovent also developed automated tests to support deployment and advised on query construction for the application interface. The architecture and its use of RabbitMQ and MongoDB are described in Integrating RabbitMQ and MongoDB to Maintain Search Indexes in Multiple Data Centers.
The Result
The initial prototype demonstrated that a data model aligned with the application’s search needs could address the underlying cause of slow queries. The subsequent development project delivered a distributed indexing pipeline for full and incremental updates across three data centers.
The client’s requirement was to make updated data searchable within four hours. In the completed solution, incremental updates reached the search indexes in seconds, with sufficient throughput to keep pace with thousands of updates per minute.
The staging architecture reduced demands on the source databases and supported concurrent indexing across data centers. Innovent’s work connected problem diagnosis and data modeling with the engineering needed to keep a large search environment current.