AI Infra

Why a Mamba State at Token 8768 Cannot Be Stored Under 8704

SGLang scheduling, HiCache, Mamba state, and LMCache explained from first principles—and why checkpoint alignment must be fixed before the model forward pass.

Maobaolong11/16/202630 min

This article follows SGLang PR #50706 at PR head 6e4771a on 2026-11-16. The change repairs a correctness contract spanning the scheduler, radix cache, and external storage. It is not merely an LMCache performance tweak.

A request has produced a model state at token 8768. The external cache stores data in 256-token chunks, so it generates the key 8704.

The difference is only 64 tokens. Why not store it anyway?

For Mamba, that would be like labeling “my memory after page 8768” as “my memory after page 8704.” A later request resuming at 8704 would receive knowledge from 64 future tokens. The cache would not be approximate; it would be semantically wrong.

The one sentence to remember The token depth of a Mamba state is part of its content. A state may be skipped, but it must never be renamed.

Meet the components in one minute

SGLangAn inference engine that runs model requests and decides what to compute and reuse.
SchedulerThe planner that chooses which requests run and how many tokens each forward pass processes.
HiCacheSGLang's hierarchical cache framework, extending prefix caching beyond GPU memory.
LMCacheOne external storage backend that HiCache can use. It organizes cached data in chunks.
MambaA recurrent state-space model that continuously updates a compact “memory up to here.”
Radix CacheSGLang's prefix tree for sharing already-computed beginnings across requests.

1. What does LLM inference cache?

The model first consumes the prompt in a prefill. It then generates one token at a time during decode. Recomputing the whole prompt for every new token would be wasteful, so the engine saves the intermediate information needed to continue.

Theskyisveryclear
prefill →
model cache“this history is already computed”
decode →
today
The cache is not a copy of the text. It is the model's intermediate state for continuing computation.

Attention keeps a row of KV entries

A Transformer attention layer usually stores Key/Value data for every historical token. Imagine a numbered row of files: the first 8704 tokens correspond to the first 8704 files, so existing boundaries can be sliced.

Mamba keeps a recurrent “memory up to here”

Mamba resembles a reader who continuously rewrites a compact notebook. The state after token 8704 is h8704. After 64 more updates, it becomes h8768:

ht+1 = f(ht, xt+1)

In general, h8768 cannot be sliced backward into h8704. If the earlier state was not captured, a storage backend cannot recreate it.

Attention KV
A row of files that can be cut at an existing boundary
Mamba state
h8768
One compressed summary whose depth must be exact

A hybrid model contains both Attention and Mamba layers. Reusing a prefix requires both the KV data and the recurrent state to describe exactly the same number of tokens.

2. How do Scheduler, Radix Cache, HiCache, and LMCache cooperate?

This is the map for the rest of the article. Hover or focus each node to see what it owns.

Two terms are easy to conflate:

  • HiCache is the framework and data path for SGLang’s cache hierarchy.
  • LMCache is one external backend attached to that path.

Not every HiCache setup uses LMCache. But when it does, both sides must agree on where a storable prefix ends.

3. The failure: why does 8768 collide with 8704?

The regression test is deliberately concrete:

  • cached prefix: 8192 tokens
  • extension in this batch: 617 tokens
  • request end: 8192 + 617 = 8809
  • Mamba/radix checkpoint grid: 64 tokens
  • LMCache chunk size: 256 tokens

The old scheduler started from the matched prefix and rounded the extension down to 64:

8192 + floor(617 ÷ 64) × 64 = 8768

The forward pass therefore retained h8768. LMCache, however, derived the last complete chunk key for 8809 as:

floor(8809 ÷ 256) × 256 = 8704

The system owned state content for 8768, while the external store accepted a chunk named 8704.

8192
cached prefix
8704
LMCache key
8768
old state
8809
request end

The 64-token gap is not ignorable padding. It is the semantic difference between h8704 and h8768.

LMCache’s defensive check skipped the store when lengths disagreed. That prevented silent corruption, but the prefix never entered external storage, so the next request could not retrieve it through HiCache.

4. Try it: the alignment lab

Change the numbers below and compare the old checkpoint, the external key, and the repaired checkpoint. This is a simplified model; the production code also verifies that a checkpoint remains reachable from the current matched prefix.

Request end8809
Old checkpoint8768
External key8704
Fixed checkpoint8704

5. The fix is not “make LMCache accept 8768”

The repair introduces a small capability contract:

  1. BasePrefixCache.external_store_alignment() returns 1 by default. Caches without an external constraint remain unchanged.
  2. The LMCache-backed UnifiedRadixCache reports its connector’s chunk_size, such as 256.
  3. Before forward, the Scheduler combines Mamba, radix-page, and external constraints:
Common grid G = lcm(Mamba grid, page grid, external grid)

For this request:

lcm(64, 64, 256) = 256

It then selects the common boundary from the absolute request position:

floor(8809 ÷ 256) × 256 = 8704

The forward pass now produces h8704 directly. Radix Cache, HiCache, and LMCache all refer to the same token depth, so the store is safe.

Step through it: when does the state become impossible to repair?1 / 5
Scheduler
chooses checkpoint
Forward
creates chosen state
Radix
attaches prefix state
LMCache key
aligns to chunk
Store
writes or skips

6. Why must the Scheduler change?

Because a checkpoint is a computation result, not a byte array that storage can slice later.

Before forward, the Scheduler knows:

  • where this request starts and ends in the current batch;
  • which Mamba checkpoints the model can capture;
  • the Radix Cache page constraint;
  • the external alignment exposed by the active Prefix Cache.

After forward has retained only h8768, the storage layers no longer possess h8704. A fix confined to lmcache_unified_radix_cache.py would have only invalid options:

Relabel state 8768 as 8704

This is silent data corruption: a request resuming at 8704 receives information from 64 future tokens.

Let LMCache use key 8768

This breaks the fixed-chunk addressing contract and does not ensure that all tensors and metadata share the boundary.

Recompute 8704 during store

The intermediate evolution is normally gone. Replaying it would also turn a cache write into extra model inference.

Store KV without Mamba state

That is not a resumable prefix for a hybrid model. A hit that cannot continue execution is not useful.

Reporting a constraint upward does not make the generic Scheduler depend on LMCache. It sees only one integer capability, external_store_alignment(), whose default is 1. Other backends do not need to know LMCache exists.

This is the same pattern as an allocator reporting alignment or a network device reporting MTU: the consumer declares a constraint; the producer satisfies it while producing the object.

7. Why is this not just an LMCache issue?

Ask who knows each fact:

Question Owner
Which chunk boundaries can the external store accept? LMCache connector
Where can this model capture a Mamba checkpoint? SGLang model/scheduler path
Which state should the forward pass produce? Scheduler, before computation begins

LMCache must report its constraint honestly. SGLang must avoid producing a checkpoint that cannot be persisted through the selected cache path. Neither side can complete the contract alone.

More generally, this is an invariant between a recurrent-state checkpoint producer and an external chunked-store consumer. Any future backend with a fixed alignment greater than 1 needs the same capability to reach the production point.

8. Is the scope reasonable?

The PR changes five focused areas:

  • a default capability on the Prefix Cache base class;
  • the real chunk_size reported by the LMCache-backed cache;
  • lcm-based alignment in the Scheduler, using an absolute boundary when external alignment exists;
  • a fake cache updated for the interface;
  • a regression test pinning 8192 + 617 → 8704.

It does not hard-code an LMCache type check into the Scheduler, and it does not change default behavior for every cache. The new constraint activates only when a backend explicitly reports one.

The PR’s Buildkite #6665 end-to-end validation also exercises a hybrid model: two requests produce identical output, and the second request retrieves from LMCache. That matters because it verifies reusable state, not merely that a store API was called.

9. Source map

Read the implementation in this order:

  1. schedule_batch.py: choose checkpoint alignment
  2. base_prefix_cache.py: generic capability default
  3. lmcache_unified_radix_cache.py: LMCache reports chunk_size
  4. test_mamba_checkpoint_depth.py: regression scenario

Four-sentence takeaway

  1. A Mamba state means “the model’s memory after exactly N tokens,” so N cannot be renamed.
  2. LMCache uses fixed-size chunk addressing and therefore accepts only certain token boundaries.
  3. The correct boundary must be selected before forward; once storage receives the wrong depth, it is too late to repair.
  4. LMCache should report its constraint, and the SGLang Scheduler must satisfy it at the state-production point.

That is why the schedule_batch.py change is not LMCache detail leaking into the scheduler. It completes a missing cross-layer contract at the only layer where correctness can still be achieved.

Discussion

Comments and Reactions

This article does not have an initialized discussion thread yet.