Blog Archive

Articles

Ongoing notes on AI infrastructure, distributed systems, databases, performance work, and engineering practice.

AI Infra11/16/202630 min

Why a Mamba State at Token 8768 Cannot Be Stored Under 8704

SGLang scheduling, HiCache, Mamba state, and LMCache explained from first principles—and why checkpoint alignment must be fixed before the model forward pass.

#SGLang#LMCache#HiCache
Read the article
AI Infra11/11/202626 min

CUDA VMM, POSIX FDs, FABRIC, and LMCache from First Principles

A visual, interactive introduction to legacy CUDA IPC, virtual memory management, POSIX file-descriptor exports, FABRIC handles, IMEX channels, and the memory-access negotiation between vLLM's extensible KV cache and LMCache MP.

#CUDA#VMM#LMCache
Read the article
AI Infra11/6/202630 min

LMCache P2P from First Principles: Control Plane, Data Plane, Connections, Reads, and Authentication

An animated walkthrough of LMCache multiprocess P2P: registration, peer discovery, the NIXL handshake, lookup and locking, address exchange, direct KV reads, unlock, and the exact boundaries that do or do not need authentication.

#LMCache#P2P#KV Cache
Read the article
AI Infra10/29/202624 min

When One vLLM Block Is No Longer One KV Page: LMCache PR #6019 Explained

A beginner-friendly explanation of KV cache blocks, kernel pages, why LMCache confused kernel paging with compression, how cache hits could silently restore the wrong KV, and how PR #6019 fixes every MP transfer path with num_blocks, k detection, and block-to-page expansion.

#lmcache#vllm#kv-cache
Read the article
AI Infra10/26/202638 min

KV Layout Should Not Be Guessed: vLLM #63699, RBLN and the LMCache Contract

17 diagrams and two interactive examples explain KV axis semantics, physical layout and encoding, trace allocation through connector registration, and show the RBLN, MLA, hybrid and LMCache integration boundaries.

#vllm#lmcache#kv-cache
Read the article
AI Infra10/26/202622 min

One Line, One Less Hop: How LMCache PR #6062 Sends SGLang KV Cache to the Copy Engine

A beginner-friendly guide to KV cache, paged memory, NHD/HND layouts, LMCache multiprocess kernel and direct-copy paths, and why one production-code line completes the fast path for SGLang unified MP.

#lmcache#sglang#kv-cache
Read the article
AI Infra10/21/202618 min

When a Legal Memory Gap Breaks LMCache CUDA IPC

Starting from LMCache Issue #5097, this article explains hybrid-model dim-0 padding, tensor strides, three CUDA IPC wrappers, and how PR #5693 and #5855 restore zero-copy transport with byte spans and as_strided.

#lmcache#vllm#cuda-ipc
Read the article
AI Infra9/21/202635 min

Unveiling the Mysteries of GDS: How LMCache MP Integrates KV Cache into NVMe

Starting with the DMA of SSD controllers, PCIe address mapping, and GPU registers, this article uses seven animated diagrams to walk through the initialization, STORE, and RETRIEVE of LMCache MP, and explains how the backend can be extended using Phoenix.

#lmcache#gds#kv-cache
Read the article
AI Infra9/17/202628 min

LMCache gRPC Support Details: From Enabling Methods to Protocol Evolution

An introduction to how LMCache multiprocess request transport evolved from ZMQ to gRPC: specific configurations for server and vLLM, implementation principles, codec design, benefits, and how to maintain compatibility when adding new service/message/rpc/field.

#lmcache#grpc#vllm
Read the article
AI Infra9/16/202631 min

From MTP to Mamba State: Why LMCache Treats a Draft Cache as a Prefix State

Starting from Prefill, Decode, and KV Cache, this article gradually explains Mamba, GDN, SWA, MLA, MTP, vLLM align-mode checkpoint, speculative block relocation, and the state misalignment issue fixed in LMCache PR #5004.

#lmcache#vllm#kv-cache
Read the article
AI Infra9/16/202635 min

Detailed Explanation of LMCache MP Registration Path: What Exactly Does register_kv_cache Do?

This article systematically explains why LMCache requires this layer of registration protocol, focusing on the KV cache passed after calling register_kv_caches in vLLM, client handling of the LMCache-driven multiprocess connector, server registration, EngineGroupInfo generation, and the complex model layouts of standard Attention, MLA, DeepSeek, Qwen, and Mamba.

#lmcache#vllm#kv-cache
Read the article
AI Infra9/16/202627 min

NHD, HND, BLHNC, BLNHC: What KV Layouts Are Chosen by vLLM and LMCache

From parsing vLLM's KVCacheLayout and the preferences of different attention backends to probing and converting boundaries of LMCache's EngineKVFormat, this system explains why there are multiple KV layouts.

#vllm#lmcache#kv-cache
Read the article
AI Infra9/11/202614 min

How vLLM Seamlessly Pulls Up LMCache: What Did PR #3476's MP Server AutoStart Do?

An analysis of LMCache PR #3476: How the LMCacheMPConnector automatically pulls up the local MP server when vLLM starts, how the port is resolved, how worker 0 takes on the responsibility of starting, and the lifecycle boundaries of this feature.

#lmcache#vllm#kv-cache
Read the article
AI Infra9/8/202616 min

Why Does Step Decide Whether GPU Blocks Can Be Released? LMCache Lazy Offload and vLLM Scheduler Persistence

Starting from the vLLM Step message pump and the boundaries of Scheduler/Worker/Cache Service, this explains why LMCache PR #4998 only persists in-flight lazy stores, and why the last batch of GPU blocks is delayed in release without it.

#lmcache#vllm#kv-cache
Read the article
AI Infra9/7/202624 min

In-Depth Analysis of KV Cache Controller: When Caching Transitions from Single Machine Optimization to Cluster Control Issues

Starting from the generation and reuse of KV Cache, this article dissects the layered storage, role boundaries, Router Hint, data transportation, state machine, fault recovery of KVCC/KVCR, and whether this architecture can become a universal infrastructure in the future.

#kv-cache#llm-inference#distributed-systems
Read the article
AI Infra9/3/202613 min

Concurrency State Machines from LMCache PR #4660: Why Atomic Operations Are Still Not Enough

Through a review of a LocalDiskBackend disk write, analyze duplicate submissions, capacity pre-checks, failure rollbacks, and resident/in-flight handover races, as well as how to validate fixes with deterministic testing.

#lmcache#concurrency#storage
Read the article
AI Infra9/3/202618 min

Understanding Pinned Memory from LMCache PR #4830: An Analysis of Silent KV Cache Corruption

Starting from pageable memory, pinned memory, device support matrices, and asynchronous copying, this analysis explores how LMCache can cause silent data corruption due to raw pointer lifetimes and incorrect synchronization boundaries.

#lmcache#cuda#pinned-memory
Read the article
AI Infra8/19/202614 min

Running sglang on MacBook with Docker: A Complete Journey from Source Code to curl Verification

The entire process of getting sglang running on Apple Silicon Mac—why Docker is the only option, three essential aarch64 patches, the HTTP/1.1 chunked trap of Docker Desktop vpnkit, and ultimately how to obtain real inference results using curl.

#sglang#docker#macos
Read the article
AI Infra8/12/20267 min

From "Can Run" to "Can Benchmark": How to Write More Persuasively in AI Infra Articles

What truly sets apart a technical blog is not the density of terminology, but whether you can clearly explain the design motivations, stress models, and key metrics together.

#serving#benchmark#writing
Read the article
Distributed Systems8/6/20266 min

9 Things I Check First When Troubleshooting Distributed Systems Online

When encountering complex failures, the most important thing is not to immediately guess the root cause, but to quickly reduce uncertainty.

#debugging#production#checklist
Read the article
Engineering Writing7/29/20265 min

Why Engineers Should Seriously Manage a "Publicly Visible" Technical Site

A technical site that can be updated long-term is not just about writing articles; it will also change the way you output, collaborate, and consolidate knowledge.

#blog#branding#knowledge
Read the article