Contact
All Posts

Solr Alias Request Routing: Diagnosing a Query Coordination Hotspot (Part 1)

Abstract blue data streams converging into a central bottleneck, representing a Solr query coordination hotspot.

When running Apache SolrCloud at scale, it is standard practice to set up an array of nodes behind a load balancer to receive and coordinate queries. The underlying expectation is simple: any node in the cluster should be available to service incoming requests.

When multi-collection aliases aliases enter the equation, however, that assumption can break entirely, leading to a deeply frustrating performance bottleneck:

The load balancer hits all 40 nodes evenly, but CPU and network activity associated with query coordination spike on just three or four nodes, despite traffic being distributed across all 40.

We ran into this exact expectation gap in production while managing hundreds of gigabytes of data split across 35 tenant collections on a 40-node cluster. The load balancer was distributing HTTP connections evenly. Solr was then applying a second routing decision inside the cluster, and that second decision concentrated request handling around the nodes hosting the first collection in the alias. Here is the breakdown of why SolrCloud's HTTP request handling can behave differently from the usual coordinator mental model, how internal proxying creates hotspots, and the architectural workaround that solved it.

The Expectation Gap: Mental Model vs. Reality

Most experienced Apache Solr implementers would probably describe a SolrCloud query using roughly the following coordination model:

This is how Solr is generally understood to operate. However, official documentation rarely details the HTTP routing decisions that happen before the search engine takes over—specifically when aliases are involved. In the Solr versions we tested, a query targeting an alias did not necessarily proceed with request handling on the node that received the HTTP request. Instead, HttpSolrCall used the first collection in the expanded alias when locating a local core for request handling.

If the node receiving the request does not host an active replica of that first collection, Solr silently abandons local coordination and hands the raw request off to a node that does.

The Multi-Tenant SolrCloud Architecture Behind the Hotspot

To understand how severely this issue affected our deployment, some operational context is necessary:

  • Fixed Endpoint Constraint: An unmodifiable upstream content system required all search and indexing requests to use the same fixed Solr alias, with queries sent to /solr/my_alias/select and updates sent to /solr/my_alias/update.
  • Multi-Tenant Scale: Hundreds of gigabytes distributed across 35 single-shard collections, each configured with three replicas for fault tolerance.
  • Tenant Isolation: To ensure fault isolation, each tenant collection resided on a specific subset of nodes. If one tenant experienced a spike in traffic or index corruption, neighboring tenants remained mostly unaffected.
  • Infrastructure: A 40-node Solr cluster fronted by a Load Balancer (LB) distributing traffic roundrobin across every node, hosted in a cloud service.

On paper, every node should take its turn acting as the Query Coordinator, fanning out subqueries across the 35 tenant collections.

In reality, only three or four nodes were handling roughly 90% of the query coordination workload. The other nodes were still participating in distributed shard level execution. The disproportionate load was concentrated in the nodes handling the incoming requests after Solr's internal proxying.

This imbalance was clearly observable across all metrics—most noticeably in JVM active threads, which spiked to 10x the volume of neighboring nodes. Because the tenant collection at position 0 was relatively small, isolating it onto different nodes proved our hypothesis: any node hosting a replica of that first collection immediately ran hot. Removing the replica caused the hotspot to die down; adding it to another node caused that node to spike immediately.

The Root Cause: Under the Hood of Query Coordination

The culprit behind this behavior lives in HttpSolrCall.java, the class responsible for processing incoming HTTP requests in Solr. When a query hits /solr/<alias_name>/select, HttpSolrCall executes the following sequence before invoking the query engine:

  1. Alias Expansion: Solr looks up <alias_name> stored in ZooKeeper and resolves it into an ordered list of underlying collections: [coll_1, coll_2, ..., coll_35].
  2. First Core Match Check: Solr inspects the first collection in that expanded list (coll_1) and checks if the local receiving node hosts an active replica for coll_1.
  3. Internal Proxy Forwarding:
    • If YES: The node finds a matching local core and continues handling the request locally.
    • If NO: If the receiving node did not host an active replica of that first collection the request was internally proxied to a node that did.

Because coll_1 was a single-shard collection residing on only a few nodes for tenant isolation, every request hitting the remaining 30+ nodes was immediately bounced over the internal network to the handful of nodes hosting coll_1.

Our load balancer was distributing HTTP requests evenly, but Solr was immediately funneling them back to a tiny slice of the cluster.

The Fix: Distributing Query Coordination with a Gateway Anchor Collection

Because we could not modify the upstream client system or inject custom routing parameters (such as shards= or collection=), we needed every node in the cluster to recognize itself as a valid entry point for /solr/my_alias/*. We solved this with no application changes or additional coordinator nodes using this architectural pattern:

Step 1: Create a Lightweight “Anchor” Collection

We created a dummy collection named alias_anchor with 1 shard and a replication factor equal to our total node count (40), placing exactly one replica on every node:

curl "http://localhost:8983/solr/admin/collections?action=CREATE\
&name=alias_anchor\
&numShards=1\
&replicationFactor=40\
&maxShardsPerNode=1\
&createNodeSet=node1,node2,node3,...,node40"

The collection contains 0 documents and receives no application updates. Its data processing cost is minimal, but the replicas are still active SolrCores and therefore carry normal operational overhead, including cores/searchers, file descriptors, clusterstate entries, configuration, and recovery activity.

Step 2: Prepend the Anchor to the Alias

We updated the alias definition to place alias_anchor at index 0:

curl "http://localhost:8983/solr/admin/collections?action=CREATEALIAS\
&name=my_alias\
&collections=alias_anchor,coll_1,coll_2,...,coll_35"

The Result: Balanced Query Coordination Across the 40-Node SolrCloud Cluster

With alias_anchor occupying the first position in the alias, HttpSolrCall now evaluates the request locally on whichever node receives it from the load balancer:

  1. The upstream system sends queries to /solr/my_alias/select.
  2. The Load Balancer forwards the request to Node 12.
  3. Solr inspects the first collection in my_alias (alias_anchor).
  4. Node 12 checks its local cores, finds its active replica of alias_anchor, and skips internal proxy forwarding.
  5. Node 12 acts as the Query Coordinator locally, fanning out subqueries across the cluster to fetch actual tenant data.

Key Takeaways for Solr Alias Request Routing

  • Validate Request Routing: Do not assume incoming HTTP requests coordinate on the landing node when querying aliases.
  • Preserve Tenant Isolation: The Gateway Anchor pattern allows all 40 nodes to participate in request coordination, assuming the load balancer distributes requests across them.
  • Zero Client Overhead: The upstream legacy system continues using its single fixed endpoint URL without modification.