Elasticsearch Murmur3 Shard Routing
Elasticsearch's routing formula uses MurmurHash3, but the docs never say so. This post names the function, walks through the full shard calculation, and shows you how to reproduce it externally.
When I first started working with Elasticsearch, I knew that documents were distributed across shards and I knew there was some kind of hash involved. The documentation even shows you the formula:
routing_factor = num_routing_shards / num_primary_shards
shard_num = (hash(_routing) % num_routing_shards) / routing_factor
But that hash() call? Completely opaque. Click through to the routing
field documentation, and you won't find the name of the function. Ask around, and
you get told it's an implementation detail, deliberately abstracted so the team can
swap it out without updating docs.
That answer is technically defensible. It didn't help me or my customers understand uneven document distribution.
This post is the explanation I wish I'd found.
What is MurmurHash3, and why does Elasticsearch use it?
The function behind hash() is MurmurHash3. It's a
non-cryptographic hash function designed by Austin Appleby in 2008. Unlike
cryptographic hash functions (SHA-256, MD5), it isn't built for security; it doesn't
need to be irreversible or resistant to deliberate collision attacks. It's built for
one thing: spreading input values as evenly as possible across an output space,
fast.
That makes it a good fit for shard routing. Elasticsearch needs to take a document ID, produce a number, and use that number to decide which shard the document lives on. The requirements are exactly what Murmur3 delivers:
| Property | Why it matters for shard routing |
|---|---|
| Uniform distribution | Documents spread evenly across shards without manual intervention. |
| Determinism | The same document ID always routes to the same shard. |
| Speed | Runs on every index and get operation (latency cost must be negligible). |
A brief overview of the algorithm is on Wikipedia, if you want to go deeper. The short version: Murmur3 operates on blocks of bytes, mixing in a series of multiply-rotate-XOR steps to produce avalanche effects (small input changes cause large, unpredictable output changes). The result is a 32-bit or 128-bit integer. Elasticsearch uses the 32-bit variant.
How Elasticsearch calculates shard numbers: A worked example
For a six-shard index, Elasticsearch uses 768 routing shards and a routing factor of 128. Here's how to verify that with a real document.
Create an index with six primary shards:
PUT test
{
"settings": {"number_of_shards": 6}
}
Check the default settings, paying attention to
index.number_of_routing_shards:
GET test?include_defaults&flat_settings
With six primary shards, the response will show
index.number_of_routing_shards = 768. The docs describe this setting as:
The default is designed to allow you to split by factors of 2 up to a maximum of 1024 shards.
What it means in practice: Elasticsearch starts with your primary shard count and keeps doubling it until doubling again would exceed 1024. For six primary shards:
6 × 1 = 6
6 × 2 = 12
6 × 4 = 24
6 × 8 = 48
6 × 16 = 96
6 × 32 = 192
6 × 64 = 384
6 × 128 = 768
6 × 256 = 1536 → too high
This gives you enough routing space to double the index's primary shard count up to seven times without reindexing.
For our index:
num_primary_shards = 6
num_routing_shards = 768
routing_factor = 768 / 6 = 128
Now index a document:
PUT test/_doc/654321
{
"field": "value"
}
Check which shard it landed on:
GET _cat/shards/test?v
It ends up on shard 4. Let's verify why.
Computing the MurmurHash3 shard routing formula step by step
Using the values established above (num_routing_shards = 768,
routing_factor = 128, document ID 654321), the shard
calculation is:
shard_num = Math.floorMod(hash("654321"), 768) / 128
At this point, we finally need to answer the question that the docs leave open: What
is hash()? In Elasticsearch, it's MurmurHash3. For the string
"654321", it returns 1424940152.
Math.floorMod(1424940152, 768) = 632
632 / 128 = 4
Shard 4, exactly where Elasticsearch put it.
If you want to verify this with your own IDs, here's a small Java script that computes the same shard number externally. Index a document into Elasticsearch, check which shard it lands on, and then run the same ID through this script and confirm the result matches.
Save this as Murmur3Demo.java:
class Murmur3Demo {
// MurmurHash3 x86_32 — a fast non-cryptographic hash by Austin Appleby.
// These two constants were chosen empirically for their avalanche (bit-mixing) properties.
static final int C1 = 0xcc9e2d51;
static final int C2 = 0x1b873593;
static int murmurhash3(byte[] data, int seed) {
int len = data.length;
int h = seed;
// Mix in 4 bytes at a time (little-endian 32-bit words)
int blocks = len / 4;
for (int i = 0; i < blocks; i++) {
int k = (data[i*4 ] & 0xFF)
| (data[i*4+1] & 0xFF) << 8
| (data[i*4+2] & 0xFF) << 16
| (data[i*4+3] & 0xFF) << 24;
k *= C1;
k = Integer.rotateLeft(k, 15);
k *= C2;
h ^= k;
h = Integer.rotateLeft(h, 13);
h = h * 5 + 0xe6546b64;
}
// Mix in any leftover 1–3 bytes
int t = blocks * 4;
int k = 0;
if ((len & 3) == 3) k = (data[t+2] & 0xFF) << 16;
if ((len & 3) >= 2) k |= (data[t+1] & 0xFF) << 8;
if ((len & 3) >= 1) {
k |= (data[t] & 0xFF);
k *= C1;
k = Integer.rotateLeft(k, 15);
k *= C2;
h ^= k;
}
// fmix32: force all bits to fully avalanche before returning
h ^= len;
h ^= h >>> 16; h *= 0x85ebca6b;
h ^= h >>> 13; h *= 0xc2b2ae35;
h ^= h >>> 16;
return h;
}
public static void main(String[] args) {
String id = args[0];
int n = Integer.parseInt(args[1]);
// Elasticsearch encodes strings as UTF-16LE (2 bytes per char, low byte first)
byte[] bytes = new byte[id.length() * 2];
for (int i = 0; i < id.length(); i++) {
char c = id.charAt(i);
bytes[i*2 ] = (byte) c;
bytes[i*2+1] = (byte) (c >>> 8);
}
// To avoid modulo bias, Elasticsearch rounds n up to the next power-of-2 (r),
// takes floorMod(hash, r), then scales back down with / (r/n).
int r = n;
while (r * 2 <= 1024) r *= 2;
System.out.println(Math.floorMod(murmurhash3(bytes, 0), r) / (r / n));
}
}
To run it:
java Murmur3Demo.java 654321 6
This prints:
4
Which matches the shard assignment above.
If your index was created with an explicit
index.number_of_routing_shards, use that value directly instead of
deriving it from number_of_shards.
The Elasticsearch source code: How hash() encodes routing values
The Elasticsearch source encodes the routing value as UTF-16LE before passing it to MurmurHash3. Here's the relevant method:
public static int hash(String routing) {
assert assertHashWithoutInformationLoss(routing);
final int strLen = routing.length();
final byte[] bytesToHash = strLen * 2 <= MAX_SCRATCH_SIZE
? scratch.get()
: new byte[strLen * 2];
for (int i = 0; i < strLen; ++i) {
ByteUtils.LITTLE_ENDIAN_CHAR.set(bytesToHash, 2 * i, routing.charAt(i));
}
return hash(bytesToHash, 0, strLen * 2);
}
Two things worth noting here:
First, the routing value is a string, and each character is written as a
little-endian two-byte sequence before hashing. This is why you cannot reproduce
the hash by treating the ID as ASCII: "654321" is encoded as
12 bytes, not 6.
Second, the function returns a signed 32-bit integer, but Elasticsearch takes
Math.floorMod of the result to avoid negative shard numbers. If you're
trying to reproduce the routing calculation in another language, watch out for
signed/unsigned integer differences.
When MurmurHash3 routing causes hot shards and how to check
For most workloads, you never need to think about this. Index documents with auto-generated IDs, and Murmur3's uniform distribution will spread them evenly across shards. Where it causes issues is with custom IDs or routing values with poor effective distribution. Sequential integers, timestamp-derived values, or heavily patterned identifiers aren't guaranteed to cause skew, but they're worth checking if you're seeing hot shards. The hash function itself is not at fault: Murmur3 is designed to distribute inputs evenly. But if shard distribution looks wrong, your choice of IDs or routing values is one of the first things to test.
If you're seeing hot shards and your documents have custom IDs, do the same
calculation we just did for "654321" on a sample of your real
IDs. Hash each ID with Murmur3, apply
Math.floorMod(..., num_routing_shards), divide by the routing factor,
and see how many land on each shard.
Why Elasticsearch doesn't document the hash function name
Hiding implementation details behind a stable interface is good software design. The routing formula is documented and guaranteed. The specific hash function underneath it is not, because it doesn't need to be; for almost every use case, all that matters is that documents are distributed evenly and consistently, which Murmur3 delivers regardless of whether you know its name.
The exception is when things go wrong. If documents are clustered on the same shards and you don't know how routing works, you have nowhere to start debugging. Once you know the formula and the function, the path from Why are these shards hot? to a concrete answer is a few lines of arithmetic.
For most users, that's when knowing it's Murmur3 matters. And now you know.
Originally published on Elasticsearch Labs.