Why a Mamba State at Token 8768 Cannot Be Stored Under 8704
SGLang scheduling, HiCache, Mamba state, and LMCache explained from first principles—and why checkpoint alignment must be fixed before the model forward pass.
SGLang scheduling, HiCache, Mamba state, and LMCache explained from first principles—and why checkpoint alignment must be fixed before the model forward pass.
This article follows SGLang PR #50706 at PR head
6e4771aon 2026-11-16. The change repairs a correctness contract spanning the scheduler, radix cache, and external storage. It is not merely an LMCache performance tweak.
A request has produced a model state at token 8768. The external cache stores data in 256-token chunks, so it generates the key 8704.
The difference is only 64 tokens. Why not store it anyway?
For Mamba, that would be like labeling “my memory after page 8768” as “my memory after page 8704.” A later request resuming at 8704 would receive knowledge from 64 future tokens. The cache would not be approximate; it would be semantically wrong.
The model first consumes the prompt in a prefill. It then generates one token at a time during decode. Recomputing the whole prompt for every new token would be wasteful, so the engine saves the intermediate information needed to continue.
A Transformer attention layer usually stores Key/Value data for every historical token. Imagine a numbered row of files: the first 8704 tokens correspond to the first 8704 files, so existing boundaries can be sliced.
Mamba resembles a reader who continuously rewrites a compact notebook. The state after token 8704 is h8704. After 64 more updates, it becomes h8768:
In general, h8768 cannot be sliced backward into h8704. If the earlier state was not captured, a storage backend cannot recreate it.
A hybrid model contains both Attention and Mamba layers. Reusing a prefix requires both the KV data and the recurrent state to describe exactly the same number of tokens.
This is the map for the rest of the article. Hover or focus each node to see what it owns.
Two terms are easy to conflate:
Not every HiCache setup uses LMCache. But when it does, both sides must agree on where a storable prefix ends.
The regression test is deliberately concrete:
8192 + 617 = 8809The old scheduler started from the matched prefix and rounded the extension down to 64:
The forward pass therefore retained h8768. LMCache, however, derived the last complete chunk key for 8809 as:
The system owned state content for 8768, while the external store accepted a chunk named 8704.
The 64-token gap is not ignorable padding. It is the semantic difference between h8704 and h8768.
LMCache’s defensive check skipped the store when lengths disagreed. That prevented silent corruption, but the prefix never entered external storage, so the next request could not retrieve it through HiCache.
Change the numbers below and compare the old checkpoint, the external key, and the repaired checkpoint. This is a simplified model; the production code also verifies that a checkpoint remains reachable from the current matched prefix.
The repair introduces a small capability contract:
BasePrefixCache.external_store_alignment() returns 1 by default. Caches without an external constraint remain unchanged.UnifiedRadixCache reports its connector’s chunk_size, such as 256.For this request:
It then selects the common boundary from the absolute request position:
The forward pass now produces h8704 directly. Radix Cache, HiCache, and LMCache all refer to the same token depth, so the store is safe.
Because a checkpoint is a computation result, not a byte array that storage can slice later.
Before forward, the Scheduler knows:
After forward has retained only h8768, the storage layers no longer possess h8704. A fix confined to lmcache_unified_radix_cache.py would have only invalid options:
This is silent data corruption: a request resuming at 8704 receives information from 64 future tokens.
This breaks the fixed-chunk addressing contract and does not ensure that all tensors and metadata share the boundary.
The intermediate evolution is normally gone. Replaying it would also turn a cache write into extra model inference.
That is not a resumable prefix for a hybrid model. A hit that cannot continue execution is not useful.
Reporting a constraint upward does not make the generic Scheduler depend on LMCache. It sees only one integer capability, external_store_alignment(), whose default is 1. Other backends do not need to know LMCache exists.
This is the same pattern as an allocator reporting alignment or a network device reporting MTU: the consumer declares a constraint; the producer satisfies it while producing the object.
Ask who knows each fact:
| Question | Owner |
|---|---|
| Which chunk boundaries can the external store accept? | LMCache connector |
| Where can this model capture a Mamba checkpoint? | SGLang model/scheduler path |
| Which state should the forward pass produce? | Scheduler, before computation begins |
LMCache must report its constraint honestly. SGLang must avoid producing a checkpoint that cannot be persisted through the selected cache path. Neither side can complete the contract alone.
More generally, this is an invariant between a recurrent-state checkpoint producer and an external chunked-store consumer. Any future backend with a fixed alignment greater than 1 needs the same capability to reach the production point.
The PR changes five focused areas:
chunk_size reported by the LMCache-backed cache;lcm-based alignment in the Scheduler, using an absolute boundary when external alignment exists;8192 + 617 → 8704.It does not hard-code an LMCache type check into the Scheduler, and it does not change default behavior for every cache. The new constraint activates only when a backend explicitly reports one.
The PR’s Buildkite #6665 end-to-end validation also exercises a hybrid model: two requests produce identical output, and the second request retrieves from LMCache. That matters because it verifies reusable state, not merely that a store API was called.
Read the implementation in this order:
schedule_batch.py: choose checkpoint alignmentbase_prefix_cache.py: generic capability defaultlmcache_unified_radix_cache.py: LMCache reports chunk_sizetest_mamba_checkpoint_depth.py: regression scenarioThat is why the schedule_batch.py change is not LMCache detail leaking into the scheduler. It completes a missing cross-layer contract at the only layer where correctness can still be achieved.