Solr Alias Request Routing: Maintaining a Gateway Anchor with Category Routed Aliases

In Solr Alias Request Routing: Diagnosing a Query Coordination Hotspot, we traced the hotspot to an unexpected property of Solr's alias request handling: when the receiving node did not host an active replica of the alias's first collection, HttpSolrCall forwarded the request internally.

Our Gateway Anchor Collection pattern solved that problem for a static multi-collection alias by putting an empty collection at position 0 and placing a replica on every node.

Our production workload, however, uses a Our production workload, however, uses a Category Routed Alias (CRA), and CRA doesn't leave the alias collection list alone., and CRA doesn't leave the alias collection list alone. New categories cause Solr to create new collections and modify the alias automatically. That creates a second problem: how do you keep the anchor at position 0 when Solr itself keeps changing the list?

Those dynamic behaviors create two new operational requirements: the anchor must participate cleanly in CRA update routing, and it must remain at position 0 as the CRA evolves over time.

Why a Static Gateway Anchor Collection Does Not Fit CRA Routing

In a standard multi-collection alias, an arbitrary, static collection can sit at position 0 without issue. Under a Category Routed Alias, however, an unmanaged static collection introduces two architectural complications:

1. Update Routing Incompatibility: CRA expects the collections represented by the alias to participate in its routing management. A manually created collection doesn't naturally carry that lifecycle metadata, so using one as the anchor requires additional effort.
2. Dynamic Array Mutation: When CRA encounters an unmapped category value, it automatically creates a new tenant collection and prepends it within /aliases.json in ZooKeeper.

This isn't merely an accidental ordering side effect. CRA deliberately prepends newly created collections to the alias. In the versions of Solr we tested (Solr 9.8.1 and Solr 9.10.1), MaintainRoutedAliasCmd adds the new collection at the head of the alias list because Solr's alias resolution defaults to the first collection. This immediately reinstates the original query-coordination hotspot issue.

Creating a CRA-Native Gateway Anchor Collection

A static collection is not a clean fit for a CRA-managed alias because CRA expects to manage the collections that belong to the alias. Rather than trying to inject an explicitly created collection, we chose to create the anchor through the CRA itself. We call this a CRA-native anchor: a collection created and registered by the Category Routed Alias itself, rather than a separately created collection manually attached to the alias.

Step 1: Seed a Document Using an Anchor Category Key

Index a single document using a reserved category value for the anchor, such as 00_anchor. Sending this document to the update endpoint forces CRA to instantiate a new, fully compliant collection:

curl -X POST "http://localhost:8983/solr/my_alias/update?commit=true" \
  -H "Content-Type: application/json" \
  -d '[
    {
      "id": "anchor_bootstrap_doc",
      "category_id": "00_anchor"
    }
  ]'

Solr intercepts the payload, creates the target collection (e.g., my_alias_00_anchor), updates /aliases.json in ZooKeeper, and routes the document into the new core. Critically, the newly created collection is added at position 0 in the alias's collection list.

Step 2: Delete the Seed Document

To keep the core footprint minimal and prevent synthetic data from polluting query results, immediately remove the bootstrap document (which does not delete the CRA collection itself):

curl -X POST "http://localhost:8983/solr/my_alias/update?commit=true" \
  -H "Content-Type: application/json" \
  -d '{
    "delete": { "query": "id:anchor_bootstrap_doc" }
  }'

The collection contains no application documents so its index footprint is negligible. It is, however, still a real Solr collection and its replicas consume the normal core, metadata, file-descriptor, and lifecycle resources described in Part 1. Deleting the document is optional, but still highly recommended.

Scaling Replica Coverage Across the Cluster

For HttpSolrCall to evaluate requests locally on every landing node, my_alias_00_anchor must have an active replica on all 40 nodes in the cluster.

Using the Collections API, expand replica placement so that every data node hosts an active core for the anchor:

curl "http://localhost:8983/solr/admin/collections?action=ADDREPLICA\
&collection=my_alias_00_anchor\
&shard=shard1\
&node=node40_host:8983_solr"

Maintaining Position 0 via ZooKeeper Automation

Even with a CRA-native anchor collection established, subsequent category creation events will eventually mutate /aliases.json and push my_alias_00_anchor out of position 0. Standard API calls like ALIASPROP and CREATEALIAS either ignore ordering modifications on managed aliases or strip the CRA routing metadata entirely.

As described in Part 1, our solution is to manage /aliases.json state directly in ZooKeeper. Because Solr watches alias state in ZooKeeper, changes to aliases.json are propagated to the Solr nodes without requiring node restarts. In our testing, the updated alias ordering was picked up without query interruption.

The following is a simplified example of the reconciliation logic. It inspects ZooKeeper, checks if my_alias_00_anchor has lost position 0, and restores it to the head of the collection array using Solr's bin/solr zk commands and jq:

#!/bin/bash
set -euo pipefail

ZK_HOST="localhost:2181"
ALIAS_NAME="my_alias"
ANCHOR_COLL="my_alias_00_anchor"
WORK_DIR="/tmp/solr_zk_sync"
SOLR_BIN="solr" # Set full path (e.g., /opt/solr/bin/solr) if not in PATH

mkdir -p "$WORK_DIR"

# 1. Download active aliases.json from ZooKeeper
"$SOLR_BIN" zk cp "zk:/aliases.json" "$WORK_DIR/aliases.json" -z "$ZK_HOST"

# 2. Extract current collection string for the alias
CURRENT_COLLECTIONS=$(jq -r ".collection.\"$ALIAS_NAME\" // empty" "$WORK_DIR/aliases.json")

if [[ -z "$CURRENT_COLLECTIONS" ]]; then
  echo "Error: Alias '$ALIAS_NAME' not found in ZooKeeper."
  exit 1
fi

FIRST_COLL=$(echo "$CURRENT_COLLECTIONS" | cut -d',' -f1)

# 3. Check if anchor lost position 0
if [[ "$FIRST_COLL" != "$ANCHOR_COLL" ]]; then
  echo "Notice: '$ANCHOR_COLL' lost position 0 (Current lead: '$FIRST_COLL'). Re-ordering..."

  # Strip any existing anchor occurrences, then prepend to index 0
  CLEAN_LIST=$(echo "$CURRENT_COLLECTIONS" | sed -E "s/(^|,)$ANCHOR_COLL(,|$)/\1/g" | sed 's/,,/,/g' | sed 's/^,//;s/,$//')
  UPDATED_LIST="${ANCHOR_COLL},${CLEAN_LIST}"

  # Inject updated string back into JSON structure
  jq --arg alias "$ALIAS_NAME" --arg list "$UPDATED_LIST" \
     '.collection[$alias] = $list' "$WORK_DIR/aliases.json" > "$WORK_DIR/aliases_updated.json"

  # 4. Push updated state back to ZooKeeper
"$SOLR_BIN" zk cp "$WORK_DIR/aliases_updated.json" "zk:/aliases.json" -z "$ZK_HOST"
  echo "Success: '$ANCHOR_COLL' restored to position 0."
else
  echo "OK: '$ANCHOR_COLL' is already at position 0."
fi

Running this script through a scheduled job or CI pipeline hook provides a reconciliation mechanism that restores the anchor to position 0 after CRA registers a new category collection.

Preserving Distributed Query Coordination Under CRA

Extending the Gateway Anchor Collection pattern to Category Routed Aliases resolves the tension between dynamic write management and query routing.

The approach bootstraps a CRA-native anchor collection through a seed document, expands its replica footprint across all nodes receiving queries, and reconciles its position to index 0 in ZooKeeper. We achieve compatibility on both traffic paths: incoming write requests route correctly to their underlying tenant collections, while read requests land on local cores and coordinate distributed queries evenly across the entire cluster.

The resulting architecture is admittedly a workaround rather than a feature native to Solr. CRA remains responsible for creating and routing tenant collections and our small reconciliation process keeps our gateway anchor at position 0. The important part is that the application still sees exactly the same endpoint it always has. No routing parameters, client changes, or additional coordinator tier are required.

The trade-off is operational complexity. We are deliberately relying on internal Solr behavior and maintaining ZooKeeper state outside the normal alias APIs. For our production environment that trade-off was preferable to adding another coordinator tier or changing an upstream system we could not modify.

Load-balanced ingress does not necessarily produce distributed query coordination when Solr aliases are involved. The Gateway Anchor Collection gives every receiving node a local path into the alias, while the CRA-native extension preserves that behavior as the alias evolves. Together, they let the load balancer do what we expected it to do in the first place: distribute query coordination across the cluster.