LMCache P2P from First Principles: Control Plane, Data Plane, Connections, Reads, and Authentication
An animated walkthrough of LMCache multiprocess P2P: registration, peer discovery, the NIXL handshake, lookup and locking, address exchange, direct KV reads, unlock, and the exact boundaries that do or do not need authentication.
Maobaolong11/6/202630 min
At first glance, P2P sounds simple: connect two LMCache servers and copy KV Cache between them.
The code quickly raises harder questions. Who discovers whom? Does the Coordinator move data? Why are ZMQ or gRPC and NIXL both involved? Does lookup return KV bytes or addresses? Why must a hit be locked? Can one peer write another peer’s cache? If authentication is added, which messages need credentials—and should every DMA page carry a token?
This article starts from zero and follows only the P2P read path in LMCache multiprocess mode. The source baseline is the LMCache dev snapshot cf87a1ca from November 6, 2026, so every stage maps to concrete code. This is an architecture and defensive-design guide; it does not discuss unpublished security reports or attack proof-of-concepts.
LMCache P2P · visual field guide
First ask who decides what to read. Then ask who moves the bytes.
P2P is not one connection. It overlays three logical channels: the Coordinator supplies a directory, ZMQ or gRPC coordinates lookups and locks, and NIXL moves the actual KV payload. Once separated, the flow becomes much easier to reason about.
Directory: who is whereControl: which keys may be readData: where KV bytes move
1. Build the mental model
1.1 Think of two cache warehouses
Suppose Node B already has the required KV in its DRAM L1, while Node A does not. A must do four things:
find B’s control endpoint and transfer endpoint;
ask B whether specific ObjectKeys exist and lock the hits;
obtain each hit’s (offset, size) inside B’s registered shared-memory region;
ask NIXL to read those bytes into local L1 objects that A has already prepared, then tell B to unlock.
The Coordinator is a directory. ZMQ or gRPC is the dispatch phone. NIXL is the forklift. The forklift does not decide which crate to move, and the dispatch phone does not carry the crate.
Coordinator
Stores active instance IPs, RPC ports, and P2P advertise URLs. It assists discovery but does not enter the KV payload path.
P2PController
Serves lookup/query/unlock handlers while polling the directory and reconciling a local adapter per peer.
P2PL2Adapter
Makes Node B look like a read-only remote L2 to Node A and owns its RPC client, NIXL client, tasks, and address state.
TransferChannelAddress
An `offset + size` relative to a registered L1 region—not a raw process pointer.
NIXL handle
A prepared handle for imported remote memory descriptors. It is reused across reads rather than rebuilt for every key.
Read lock
Keeps B from evicting or reusing a hit after its address is returned but before A finishes reading.
1.2 Three logical channels do not mean three copies of KV
Directory / orchestration
HTTP registration, heartbeats, and `GET /instances`. Messages contain instance metadata and tolerate retries and eventual consistency.
Control plane
ZMQ or gRPC carries `p2p_lookup_and_lock`, `p2p_query_lookup_results`, and `p2p_unlock_objects`. Messages are small, but they decide what may be read.
Data plane
NIXL/UCX issues READ transfers using prepared memory descriptors. This is the large KV payload and therefore bypasses the Coordinator and normal RPC serialization.
Local execution
StorageManager looks up keys, holds read locks, maps MemoryObj to offsets, and polls completion. These are in-process calls, not network protocols.
A useful classification rule: messages containing keys, task IDs, status, offsets, and sizes are control plane. The path carrying actual K/V tensor bytes is the data plane.
2. Startup, registration, and discovery
2.1 Each node first turns local L1 into a remotely readable region
P2P is enabled when --p2p-advertise-url is configured. During startup, P2PController locates the one local DRAM L1, obtains L1MemoryDesc(ptr, size, align_bytes), and initializes the transfer-channel context.Initialization
The current NIXL implementation then:
creates a NIXL agent;
registers the whole local DRAM L1 as transferable memory;
prepares page-granular transfer descriptors using align_bytes;
starts a dedicated ZMQ REP handshake server on the P2P transfer URL.NIXL context
That ZMQ REP socket is not the LMCache request transport. Even when the main RPC transport is gRPC, the current NIXL metadata handshake still owns a separate ZMQ socket. It exchanges NIXL metadata; it does not carry business lookup RPCs.
2.2 Registration publishes addresses, not a live connection
The MP server registers instance_id, reachable IP, HTTP port, p2p_advertised_url, the request/RPC port, and chunk size with the Coordinator.Registrar
Registration says “this is where future peers can find me.” It does not connect A to B, and it never uploads KV. Heartbeats keep the entry alive; an explicit 404 causes re-registration.
2.3 Discovery creates an adapter and two clients
P2PController periodically fetches /instances. It only becomes REGISTERED after it can see its own matching entry. For each other valid peer, it creates P2PL2AdapterConfig with:
peer_mq_server_url, whose scheme follows the selected request transport;
peer_transfer_channel_server_url, used by the NIXL handshake/data path;
lookup and load timeouts.
The adapter constructor then creates a ZMQ or gRPC request client and obtains a transfer-channel client. The latter actively connects to the peer, performs the NIXL handshake, and caches a remote handle.Adapter registration
The NIXL handshake has two stages: exchange agent metadata, then exchange transfer descriptor lists. Both peers may connect at the same time, so the context keeps one canonical client per advertise URL and closes duplicates.
Loading the interactive walkthrough. Without JavaScript, the order is register → discover → NIXL handshake → lookup/lock → address result → NIXL READ → unlock.
3. One complete read: from ObjectKey to KV bytes
3.1 lookup_and_lock: ask whether it exists and reserve it
Node A’s storage manager treats Node B as a remote L2 and calls submit_lookup_and_lock_task(keys, group_layout_descs). The adapter sends:
B does not search its own L2 or fetch from disk on A’s behalf. The current handler is restricted to the local DRAM L1. Hits are placed under read reservation, and B returns an integer task ID.Lookup handler
The extra query phase follows StorageManager’s asynchronous prefetch contract. It leaves room for future implementations even when today’s local-L1 lookup usually completes quickly.
3.2 query_lookup_results: return coordinates, not data
None means not complete yet or already consumed. A miss is represented by offset=-1, size=0. For hits, B obtains the locked objects’ shared-memory positions and converts them into offsets relative to the whole registered L1.Address construction
3.3 submit_read: this is where KV payload finally moves
A has already allocated local MemoryObj destinations. It converts their (shm_offset, shm_byte_length) pairs to local transfer addresses and combines them with the remote addresses from B:
NIXL submit_read( local_addresses = destinations in A's L1, remote_addresses = hits in B's L1,)
NIXL translates addresses to page indices, prepares a READ, and submits it. A polls query_read_status; only DONE sets the corresponding success bits.Load path
The actual data plane. The hit pages in B are locked while NIXL writes the KV bytes into A's local destination pages. The Coordinator and business RPC are absent from this payload path.
Node A · destination L1
←
Node B · registered L1
3.4 unlock: return the reservation after reading
After completion or abandonment, A sends p2p_unlock_objects(keys). B calls finish_read_prefetched(keys) to release the read reservations, while A drops its cached remote addresses for those keys.
This is essential. An offset only identifies an object during a valid lifetime. After unlock, eviction or reuse may place different bytes at the same offset.
4. What the write path actually does
The current LMCache P2P adapter is not a general remote-memory read/write API.
P2PL2Adapter.submit_store_task sends nothing to the peer. It immediately records L2StoreResult(success=True, bytes_stored=0) so the generic store controller can finish its bookkeeping.Read-only store path
The two flows are therefore:
Normal local STORE / inference engine worker → local LMCache server → local L1P2P remote reuse Node A looks up Node B → NIXL READ → Node A local L1
P2P pulls an existing remote cache into the requester’s local L1. It does not push every new local KV object to all peers.
Direction in precise terms: A performs a remote read. Bytes flow out of B's memory and into A's memory. NIXL calls the operation READ because the requester reads a remote region.
5. Misses, timeouts, and cleanup
The directory is temporarily unreachable
A timeout moves orchestration to DISCONNECTED; other HTTP or schema errors lead to UNREGISTERED and reconciliation. The MP server itself stays up.
A peer disappears from discovery
The adapter is not deleted after one missing poll. The current controller tolerates three consecutive misses, reducing reconnect churn during directory blips.
The lookup RPC times out
The adapter treats it as a cache miss so upper layers can try another tier or recompute instead of blocking forever.
NIXL load times out or errors
The success bitmap stays false. A control-plane hit does not guarantee a successful data-plane transfer.
The requester abandons query or unlock
B eventually drops old lookup jobs. Read reservations mainly rely on L1 read TTL because a blind late unlock could release a newer reader's lock.
6. Where authentication belongs
Two terms matter:
Authentication establishes who the caller is, or at least that it possesses a trusted credential.
Authorization determines which objects and addresses that authenticated identity may access.
The immediate P2P boundary is to stop unauthenticated network peers from using control RPCs or importing a transfer region. Whether one authenticated peer should be able to address an entire region is a stronger least-privilege problem that an API key alone cannot solve.
① P2P startup configuration
Fail closed when P2P is enabled without credentials. Optional auth otherwise leaves the default P2P deployment anonymous.
② Coordinator registration and discovery
Authenticate writers and readers of the peer directory to prevent forged instances, endpoint poisoning, and metadata disclosure.
③ ZMQ / gRPC control RPC
Authenticate lookup, query, and unlock before business payload decoding and handler dispatch. Invalid keys must not touch state.
④ NIXL handshake
Authenticate before returning agent metadata or transfer descriptors and before registering the client. Descriptors are the capability entry point.
⑤ NIXL READ
Each DMA page need not carry another Bearer token. The read uses the session/handle established by an authenticated handshake; credential revocation must tear down sessions.
⑥ Local lookup and address conversion
In-process calls need locks, ownership, bounds, and alignment validation—not a user token threaded through every helper.
6.1 Why both control RPC and the NIXL handshake must authenticate
Protecting only control RPC is insufficient. A network peer that knows the transfer endpoint can still attempt the independent NIXL handshake. If complete region descriptors are returned before authentication, the control plane is guarded but the data-plane capability is not.
Protecting only the NIXL handshake is also insufficient. An anonymous caller could still create lookup jobs, hold read locks, enumerate hits and misses, consume results, or issue unlock calls.
6.2 Why local helpers do not all need token parameters
StorageManager.query_prefetch_status, unsafe_read, addresses_to_indices, and query_read_status operate after entering a trusted process/object boundary. Threading a token through them does not strengthen network identity. It spreads secrets into more APIs and logs.
Authenticate at the network edge, bind the resulting principal or session to the objects it owns, and keep using local type, lock, bounds, and lifetime invariants inside the process.
6.3 Authentication is not encryption
A Bearer token proves knowledge of a shared secret. Plain ZMQ, insecure gRPC, and NIXL do not become encrypted because a token is present. Crossing an untrusted network still requires TLS, mTLS, CURVE, IPsec, or a controlled encrypted tunnel.
A stronger handshake can use a nonce and HMAC to avoid repeatedly transmitting the raw key and to bind both agent identities, endpoints, and protocol versions into the transcript. KV payload encryption remains a separate property.
State the authorization limit honestly: the current NIXL context registers the whole L1 region. Authentication excludes strangers. If authenticated nodes are mutually untrusted, object-scoped or capability-scoped descriptors are required as well.
7. A practical implementation order
Start with a testable invariant:
With P2P enabled, only a peer holding a current credential may invoke P2P RPCs, obtain transfer descriptors, and construct a remote-memory handle. Unauthenticated input fails before business payload decoding, handler dispatch, and NIXL metadata operations.
Then implement in this order:
Fail closed at configuration time. Reject P2P without credentials, or require an explicit, loudly warned allow-unauthenticated-p2p escape hatch.
Reuse one key loader and verifier. Accept multiple inbound keys for rotation, choose one outbound client key, and redact secrets from representations, status, and environment output.
Protect request transports. Use a versioned auth frame for ZMQ and metadata interceptors for gRPC; authenticate before decoding business payloads.
Protect the NIXL handshake. Authenticate InitReq before returning agent metadata, then bind MemRegReq to the same identity/session.
Protect Coordinator HTTP. Add credentials to registration, heartbeats, and /instances; decide explicitly whether directory and peer credentials share a trust domain.
Define rotation and revocation. Existing prepared handles do not disappear when a key file changes. Restart or proactively tear down transfer clients.
Add observability. Distinguish authentication failure, lookup miss, RPC timeout, and transfer error instead of collapsing everything into a cache miss.
Scenario
Expected control-plane result
Expected data-plane result
Correct key
lookup/query/unlock succeed
handshake and NIXL READ succeed
Missing key
rejected before decode/dispatch
no agent metadata or descriptors returned
Wrong key
explicit UNAUTHENTICATED/PermissionError
no remote handle created
Malformed auth
bounded, fast failure with no secret leakage
no NIXL metadata parsing
Key rotation
old/new accepted inbound, one sent outbound
new connections work; old-session lifetime is explicit
P2P with no key
startup fails or requires explicit insecure opt-in
never opens silently
Authenticated hit
correct task and addresses
populated L1 bytes copy correctly
Authenticated miss
invalid address
no read is submitted
8. Source map and final checklist
Read from the business flow downward:
P2PController initializes the transfer context, polls /instances, maintains adapters, and implements the three P2P RPC handlers.
Must an in-flight NIXL read pass through the Coordinator? No.
Does query_lookup_results return KV? No, only offsets and sizes.
Does replacing ZMQ with gRPC move KV bytes through gRPC? No.
Does the P2P adapter write A’s KV into B? Not today; it is read-only.
Is adding an API key only to lookup RPC sufficient? No; the NIXL handshake must authenticate before disclosing descriptors.
In one sentence:
The Coordinator lets A find B; control RPC tells A which objects it may read and where they live; NIXL uses an imported remote-memory capability to copy those bytes into A; unlock ends the temporary reservation.
Discussion
Comments and Reactions
This article does not have an initialized discussion thread yet.