Hot keys, big keys, cache penetration, and measuring a cache

A hot key stays on one shard: serve it from each pod for seconds, and judge a cache by what still reaches the origin.

Before you read: predict

ExamHall's API runs in 10 pods behind a load balancer. During an exam, each of 1,200 candidates' browsers reads exam:42:config every 10 s, and each read lands on a random pod. The key sits on one of the three shards of the Redis Cluster, and that shard runs at 100% CPU. The team adds a 2-second in-process cache for this key in every pod.

In one minute, about how many GETs for the key reach its shard now?

The 30-second version

A key lives on one shard, so a hot key's load stays there however many shards you add. A short in-process cache in each pod cuts that shard to about one GET per pod per TTL. Big values stall Redis's single thread. Absent keys always miss unless "not found" is cached or a filter of real ids answers first.

A hot key stays on one shard: serve it from each pod for seconds, and judge a cache by what still reaches the origin.

How it works

Step 1 of 14

One key, one shard. Redis Cluster stores each key on exactly one shard, so every read of a hot key lands there.

In the kitchen

on a takeover day one stock's post is mobbed while the others stand idle; a new post just moves the crowd → a hot key on one shard (steps 1–2)

Analogy: A stock exchange floor

Each listed stock trades at exactly one post, fixed by its symbol.

On the floor	In Redis
on a takeover day one stock's post is mobbed while the others stand idle; a new post just moves the crowd	a hot key on one shard (steps 1–2)
each brokerage booth chalks the price on its own board, refreshed by a runner every two minutes	a short pod cache (steps 3–5)
the stock quoted at three posts	key replication (step 6)
one broker has the clerk read out a whole day of trades	a big value (step 7)
runners sent for unlisted symbols; a "not listed" slip pinned at the post for a while	absent keys, negative caching (steps 10–11)
a pocket list of listed symbols that may say "maybe" for an unlisted one	a Bloom filter (step 12)
quotes answered from the boards, runner trips still made	hit ratio, origin load (step 13)

In real terms: for each very hot key, decide how stale a pod may serve it; for each id a client can invent, decide what stops it before the database.

Breaks down when: a broker who sees the price move re-chalks at once; a pod's copy changes only when its TTL ends.

Contrasting case: reads spread over many keys, such as each user's profile. No shard is hot, and a pod rarely sees one key twice within a TTL, so a pod cache saves little.

Examples, run for real

4.1 ExamHall: the config key that melted one shard

EXH-5030: at ExamHall's 09:00 exam start, 2,000 candidates read exam:42:config and their own attempt key every 10 s through 20 pods. One shard ran hot, and growing to six shards did not help. MR !5330 adds a 2-second pod cache for the config key.

4.1 EXH-5030 2000 candidates read exam:42:config and their own attempt key every 10 s, 20 pods, one minute · exam:42:config slot 3899
4.1 EXH-5030 3 shards (slots 0-5460, 5461-10922, 10923-16383): GETs per shard 15942 / 4026 / 4032 · exam:42:config on shard 1: 12000 of them
4.1 EXH-5030 grown to 6 shards: GETs per shard 1962 / 13980 / 2058 / 1968 / 1956 / 2076 · exam:42:config on shard 2: 12000 of them
4.1 MR !5330 3 shards, 2-second in-process cache for exam:42:config in each pod: GETs per shard 4523 / 4026 / 4032 · exam:42:config: 581 (95% fewer)

The attempt keys spread evenly; the config key put 12,000 GETs on one shard, then on another (steps 1–2). The pods' copies cut it by 95% (steps 3–4).

4.2 Chat: finding the hot key and the big ones

A chat Redis on allkeys-lfu holds the lobby's pins (read 20,000 times), 2,000 rooms' small keys, a 200,000-message history hash and a 1 MB snapshot.

4.2 chat Redis 7.4.11, allkeys-lfu · 4003 keys · redis-cli --hotkeys: hottest key room:lobby:pins
4.2 redis-cli --bigkeys: biggest hash room:global:history (200,000 fields) · biggest string room:global:snapshot (1,048,576 bytes)
4.2 redis-cli --memkeys: biggest hash room:global:history (11.9 MB) · biggest string room:global:snapshot (1.3 MB)

--hotkeys reads LFU counters, so it needs an LFU policy. --bigkeys counted fields and bytes; --memkeys showed memory (step 8).

Picture it

many hands reaching toward one small shared drawer while the neighbouring drawers stay quiet and nearly empty (the asset map's description). — An illustration to help it stick. The mechanism is in the sections above.

Try it

Answer first, then check. Nothing is saved.

Question 1 of 6

A chat app's lobby is the room every user opens first. Its pinned messages live in room:lobby:pins on one of three Redis Cluster shards, placed on this lesson's model of the three-shard slot split, and 5,000 open lobby tabs read the key every 10 s through 25 API pods; that shard runs at 100% CPU. Moderators edit the pins a few times a day. Goal: the shard holding the pins serves under a tenth of today's GETs for them, and a moderator's edit reaches every reader within 3 s.

Which one change meets the goal?

Question 2 of 6

A chat service keeps each room's history in one hash, and room:global:history has grown to 200,000 messages. Its Redis 7.4 runs the default configuration except for two lines: lazyfree-lazy-user-del yes and lazyfree-lazy-server-del yes. The team tries four ways to drop the hash, each on a fresh copy: DEL, EXPIRE with 1 second and waiting for the expiry, UNLINK, and a SET of a tiny value over the key.

Which of them free the hash's memory on a background thread, so Redis's main thread never spends the time freeing 200,000 fields?

Question 3 of 6

A chat API serves GET /users/:id through a Redis cache in front of its database, and caches only the users it finds. Two clients ask for users that do not exist: a buggy mobile build polls the same 30 deleted users every 3 s, and a scraper asks for 10 random, never-repeated user ids a second. Redis has memory for millions of entries. The team adds negative caching: each not-found answer is cached in Redis for 60 s.

Over the next 5 minutes, how many database lookups does each client cause?

Question 4 of 6

A chat Redis is running short of memory, and two keys stand out: room:global:history, a hash of 200,000 short text messages of under 20 bytes each, and room:media:thumbs, a hash of 2,000 thumbnails of 16 KB each. The team runs redis-cli --bigkeys, then redis-cli --memkeys --memkeys-samples 0, which counts every field.

Which hash does each report name as the biggest?

Question 5 of 6

A video service has 100,000 videos. At 08:00 it builds a Bloom filter of their ids, sized for a 1% false-positive rate and kept in Redis, and checks it before its cache and database. Between 08:00 and 09:00 a crawler requests 20,000 random video ids that do not exist. At 09:00 the studio publishes one new video to the videos table, and viewers request it.

Which outcome follows?

Question 6 of 6

A ticketing site caches event pages in an LRU cache. It lists 50,000 events; a few take most of the reads, and the skew is the same all day. Replayed on yesterday's reads (an exact-LRU model), a 500-entry cache hit 47.7% and a 2,000-entry cache hit 62.1%.

About how many entries does the cache need to hit 85%?

Go deeper

Pro tips

Look for the hot key before you add shards. Under an LFU policy, redis-cli --hotkeys names the keys with the highest access counters. When: one shard's CPU or network is high while the others idle (steps 1–2).
Remove big keys with UNLINK, and keep them from growing. The 7.4 configuration file warns that deleting a value with millions of elements can block "for a long time (even seconds)". When: any hash, list, set or sorted set that grows without bound, such as a room's whole history (steps 7–9).

Source

Redis docs, Scale with Redis Cluster (source content/operate/oss_and_stack/management/scaling.md, redis/docs branch main): https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/ · fetched 2026-10-18. "There are 16384 hash slots in Redis Cluster, and to compute the hash slot for a given key, we simply take the CRC16 of the key modulo 16384"; "Every node in a Redis Cluster is responsible for a subset of the hash slots" (steps 1–2). Valid as of 2026-10-18.
Redis docs, Redis CLI (source content/develop/tools/cli.md, redis/docs branch main): https://redis.io/docs/latest/develop/tools/cli/ · fetched 2026-10-18. "It scans the dataset for big keys"; "allows you to scan the entire keyspace to find biggest keys as well as the average sizes per key type"; "Sample Redis keys looking for hot keys"; "only works when maxmemory-policy is *lfu". Valid as of 2026-10-18.
Redis docs, UNLINK (source content/commands/unlink.md, redis/docs branch main): https://redis.io/docs/latest/commands/unlink/ · fetched 2026-10-18. "the command performs the actual memory reclaiming in a different thread, so it is not blocking, while DEL is" (step 8). Valid as of 2026-10-18.
Redis 7.4 redis.conf, lazy freeing section (redis/redis branch 7.4): https://raw.githubusercontent.com/redis/redis/7.4/redis.conf · fetched 2026-10-18. "the server can block for a long time (even seconds) in order to complete the operation"; "The SET command itself removes any old content of the specified key in order to replace it with the specified string"; "In all the above cases the default is to delete objects in a blocking way, like if DEL was called"; "However you can configure each case specifically in order to instead release memory in a non-blocking way like if UNLINK was called"; "to modify the default behavior of the DEL command to act exactly like UNLINK" (steps 7–9, tip 2). Valid as of 2026-10-18.
Nishtala et al., Scaling Memcache at Facebook (NSDI 2013), §3.2.3 on replication within pools: https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf · read 2026-10-18. The paper replicates a category of keys across servers when the request rate is more than one server can manage, and notes that replicas need every invalidation (step 6). Cited, not quoted: the PDF's two-column text does not extract verbatim. Valid as of 2026-10-18.

Checked against the tools and docs as of 2026-10-18.

Predict first