AI Infra

LMCache P2P from First Principles: Control Plane, Data Plane, Connections, Reads, and Authentication

An animated walkthrough of LMCache multiprocess P2P: registration, peer discovery, the NIXL handshake, lookup and locking, address exchange, direct KV reads, unlock, and the exact boundaries that do or do not need authentication.

Maobaolong11/6/202630 min

At first glance, P2P sounds simple: connect two LMCache servers and copy KV Cache between them.

The code quickly raises harder questions. Who discovers whom? Does the Coordinator move data? Why are ZMQ or gRPC and NIXL both involved? Does lookup return KV bytes or addresses? Why must a hit be locked? Can one peer write another peer’s cache? If authentication is added, which messages need credentials—and should every DMA page carry a token?

This article starts from zero and follows only the P2P read path in LMCache multiprocess mode. The source baseline is the LMCache dev snapshot cf87a1ca from November 6, 2026, so every stage maps to concrete code. This is an architecture and defensive-design guide; it does not discuss unpublished security reports or attack proof-of-concepts.

LMCache P2P · visual field guide

First ask who decides what to read. Then ask who moves the bytes.

P2P is not one connection. It overlays three logical channels: the Coordinator supplies a directory, ZMQ or gRPC coordinates lookups and locks, and NIXL moves the actual KV payload. Once separated, the flow becomes much easier to reason about.

Directory: who is where Control: which keys may be read Data: where KV bytes move

1. Build the mental model

1.1 Think of two cache warehouses

Suppose Node B already has the required KV in its DRAM L1, while Node A does not. A must do four things:

  1. find B’s control endpoint and transfer endpoint;
  2. ask B whether specific ObjectKeys exist and lock the hits;
  3. obtain each hit’s (offset, size) inside B’s registered shared-memory region;
  4. ask NIXL to read those bytes into local L1 objects that A has already prepared, then tell B to unlock.

The Coordinator is a directory. ZMQ or gRPC is the dispatch phone. NIXL is the forklift. The forklift does not decide which crate to move, and the dispatch phone does not carry the crate.

Coordinator

Stores active instance IPs, RPC ports, and P2P advertise URLs. It assists discovery but does not enter the KV payload path.

P2PController

Serves lookup/query/unlock handlers while polling the directory and reconciling a local adapter per peer.

P2PL2Adapter

Makes Node B look like a read-only remote L2 to Node A and owns its RPC client, NIXL client, tasks, and address state.

TransferChannelAddress

An `offset + size` relative to a registered L1 region—not a raw process pointer.

NIXL handle

A prepared handle for imported remote memory descriptors. It is reused across reads rather than rebuilt for every key.

Read lock

Keeps B from evicting or reusing a hit after its address is returned but before A finishes reading.

1.2 Three logical channels do not mean three copies of KV

Directory / orchestration

HTTP registration, heartbeats, and `GET /instances`. Messages contain instance metadata and tolerate retries and eventual consistency.

Control plane

ZMQ or gRPC carries `p2p_lookup_and_lock`, `p2p_query_lookup_results`, and `p2p_unlock_objects`. Messages are small, but they decide what may be read.

Data plane

NIXL/UCX issues READ transfers using prepared memory descriptors. This is the large KV payload and therefore bypasses the Coordinator and normal RPC serialization.

Local execution

StorageManager looks up keys, holds read locks, maps MemoryObj to offsets, and polls completion. These are in-process calls, not network protocols.

A useful classification rule: messages containing keys, task IDs, status, offsets, and sizes are control plane. The path carrying actual K/V tensor bytes is the data plane.

2. Startup, registration, and discovery

2.1 Each node first turns local L1 into a remotely readable region

P2P is enabled when --p2p-advertise-url is configured. During startup, P2PController locates the one local DRAM L1, obtains L1MemoryDesc(ptr, size, align_bytes), and initializes the transfer-channel context.Initialization

The current NIXL implementation then:

  1. creates a NIXL agent;
  2. registers the whole local DRAM L1 as transferable memory;
  3. prepares page-granular transfer descriptors using align_bytes;
  4. starts a dedicated ZMQ REP handshake server on the P2P transfer URL.NIXL context

That ZMQ REP socket is not the LMCache request transport. Even when the main RPC transport is gRPC, the current NIXL metadata handshake still owns a separate ZMQ socket. It exchanges NIXL metadata; it does not carry business lookup RPCs.

2.2 Registration publishes addresses, not a live connection

The MP server registers instance_id, reachable IP, HTTP port, p2p_advertised_url, the request/RPC port, and chunk size with the Coordinator.Registrar

Registration says “this is where future peers can find me.” It does not connect A to B, and it never uploads KV. Heartbeats keep the entry alive; an explicit 404 causes re-registration.

2.3 Discovery creates an adapter and two clients

P2PController periodically fetches /instances. It only becomes REGISTERED after it can see its own matching entry. For each other valid peer, it creates P2PL2AdapterConfig with:

  • peer_mq_server_url, whose scheme follows the selected request transport;
  • peer_transfer_channel_server_url, used by the NIXL handshake/data path;
  • lookup and load timeouts.

The adapter constructor then creates a ZMQ or gRPC request client and obtains a transfer-channel client. The latter actively connects to the peer, performs the NIXL handshake, and caches a remote handle.Adapter registration

The NIXL handshake has two stages: exchange agent metadata, then exchange transfer descriptor lists. Both peers may connect at the same time, so the context keeps one canonical client per advertise URL and closes duplicates.

Loading the interactive walkthrough. Without JavaScript, the order is register → discover → NIXL handshake → lookup/lock → address result → NIXL READ → unlock.

3. One complete read: from ObjectKey to KV bytes

3.1 lookup_and_lock: ask whether it exists and reserve it

Node A’s storage manager treats Node B as a remote L2 and calls submit_lookup_and_lock_task(keys, group_layout_descs). The adapter sends:

p2p_lookup_and_lock(
  keys = [ObjectKey(...), ...],
  group_layout_descs = {object_group_id: MemoryLayoutDesc(...)}
)

B does not search its own L2 or fetch from disk on A’s behalf. The current handler is restricted to the local DRAM L1. Hits are placed under read reservation, and B returns an integer task ID.Lookup handler

The extra query phase follows StorageManager’s asynchronous prefetch contract. It leaves room for future implementations even when today’s local-L1 lookup usually completes quickly.

3.2 query_lookup_results: return coordinates, not data

A polls with the remote task ID:

p2p_query_lookup_results(task_id) ->
  [TransferChannelAddress(offset, size), ...] | None

None means not complete yet or already consumed. A miss is represented by offset=-1, size=0. For hits, B obtains the locked objects’ shared-memory positions and converts them into offsets relative to the whole registered L1.Address construction

3.3 submit_read: this is where KV payload finally moves

A has already allocated local MemoryObj destinations. It converts their (shm_offset, shm_byte_length) pairs to local transfer addresses and combines them with the remote addresses from B:

NIXL submit_read(
  local_addresses  = destinations in A's L1,
  remote_addresses = hits in B's L1,
)

NIXL translates addresses to page indices, prepares a READ, and submits it. A polls query_read_status; only DONE sets the corresponding success bits.Load path

The actual data plane. The hit pages in B are locked while NIXL writes the KV bytes into A's local destination pages. The Coordinator and business RPC are absent from this payload path.
Node A · destination L1
Node B · registered L1

3.4 unlock: return the reservation after reading

After completion or abandonment, A sends p2p_unlock_objects(keys). B calls finish_read_prefetched(keys) to release the read reservations, while A drops its cached remote addresses for those keys.

This is essential. An offset only identifies an object during a valid lifetime. After unlock, eviction or reuse may place different bytes at the same offset.

4. What the write path actually does

The current LMCache P2P adapter is not a general remote-memory read/write API.

P2PL2Adapter.submit_store_task sends nothing to the peer. It immediately records L2StoreResult(success=True, bytes_stored=0) so the generic store controller can finish its bookkeeping.Read-only store path

The two flows are therefore:

Normal local STORE / inference
  engine worker → local LMCache server → local L1

P2P remote reuse
  Node A looks up Node B → NIXL READ → Node A local L1

P2P pulls an existing remote cache into the requester’s local L1. It does not push every new local KV object to all peers.

Direction in precise terms: A performs a remote read. Bytes flow out of B's memory and into A's memory. NIXL calls the operation READ because the requester reads a remote region.

5. Misses, timeouts, and cleanup

The directory is temporarily unreachable

A timeout moves orchestration to DISCONNECTED; other HTTP or schema errors lead to UNREGISTERED and reconciliation. The MP server itself stays up.

A peer disappears from discovery

The adapter is not deleted after one missing poll. The current controller tolerates three consecutive misses, reducing reconnect churn during directory blips.

The lookup RPC times out

The adapter treats it as a cache miss so upper layers can try another tier or recompute instead of blocking forever.

NIXL load times out or errors

The success bitmap stays false. A control-plane hit does not guarantee a successful data-plane transfer.

The requester abandons query or unlock

B eventually drops old lookup jobs. Read reservations mainly rely on L1 read TTL because a blind late unlock could release a newer reader's lock.

6. Where authentication belongs

Two terms matter:

  • Authentication establishes who the caller is, or at least that it possesses a trusted credential.
  • Authorization determines which objects and addresses that authenticated identity may access.

The immediate P2P boundary is to stop unauthenticated network peers from using control RPCs or importing a transfer region. Whether one authenticated peer should be able to address an entire region is a stronger least-privilege problem that an API key alone cannot solve.

① P2P startup configuration

Fail closed when P2P is enabled without credentials. Optional auth otherwise leaves the default P2P deployment anonymous.

② Coordinator registration and discovery

Authenticate writers and readers of the peer directory to prevent forged instances, endpoint poisoning, and metadata disclosure.

③ ZMQ / gRPC control RPC

Authenticate lookup, query, and unlock before business payload decoding and handler dispatch. Invalid keys must not touch state.

④ NIXL handshake

Authenticate before returning agent metadata or transfer descriptors and before registering the client. Descriptors are the capability entry point.

⑤ NIXL READ

Each DMA page need not carry another Bearer token. The read uses the session/handle established by an authenticated handshake; credential revocation must tear down sessions.

⑥ Local lookup and address conversion

In-process calls need locks, ownership, bounds, and alignment validation—not a user token threaded through every helper.

6.1 Why both control RPC and the NIXL handshake must authenticate

Protecting only control RPC is insufficient. A network peer that knows the transfer endpoint can still attempt the independent NIXL handshake. If complete region descriptors are returned before authentication, the control plane is guarded but the data-plane capability is not.

Protecting only the NIXL handshake is also insufficient. An anonymous caller could still create lookup jobs, hold read locks, enumerate hits and misses, consume results, or issue unlock calls.

The proper ordering is:

Network entry A: request transport
  authenticate → decode RPC payload → dispatch handler

Network entry B: transfer-channel handshake
  decode a bounded auth envelope → authenticate
  → add remote agent / deserialize descriptors / return local descriptors

6.2 Why local helpers do not all need token parameters

StorageManager.query_prefetch_status, unsafe_read, addresses_to_indices, and query_read_status operate after entering a trusted process/object boundary. Threading a token through them does not strengthen network identity. It spreads secrets into more APIs and logs.

Authenticate at the network edge, bind the resulting principal or session to the objects it owns, and keep using local type, lock, bounds, and lifetime invariants inside the process.

6.3 Authentication is not encryption

A Bearer token proves knowledge of a shared secret. Plain ZMQ, insecure gRPC, and NIXL do not become encrypted because a token is present. Crossing an untrusted network still requires TLS, mTLS, CURVE, IPsec, or a controlled encrypted tunnel.

A stronger handshake can use a nonce and HMAC to avoid repeatedly transmitting the raw key and to bind both agent identities, endpoints, and protocol versions into the transcript. KV payload encryption remains a separate property.

State the authorization limit honestly: the current NIXL context registers the whole L1 region. Authentication excludes strangers. If authenticated nodes are mutually untrusted, object-scoped or capability-scoped descriptors are required as well.

7. A practical implementation order

Start with a testable invariant:

With P2P enabled, only a peer holding a current credential may invoke P2P RPCs, obtain transfer descriptors, and construct a remote-memory handle. Unauthenticated input fails before business payload decoding, handler dispatch, and NIXL metadata operations.

Then implement in this order:

  1. Fail closed at configuration time. Reject P2P without credentials, or require an explicit, loudly warned allow-unauthenticated-p2p escape hatch.
  2. Reuse one key loader and verifier. Accept multiple inbound keys for rotation, choose one outbound client key, and redact secrets from representations, status, and environment output.
  3. Protect request transports. Use a versioned auth frame for ZMQ and metadata interceptors for gRPC; authenticate before decoding business payloads.
  4. Protect the NIXL handshake. Authenticate InitReq before returning agent metadata, then bind MemRegReq to the same identity/session.
  5. Protect Coordinator HTTP. Add credentials to registration, heartbeats, and /instances; decide explicitly whether directory and peer credentials share a trust domain.
  6. Define rotation and revocation. Existing prepared handles do not disappear when a key file changes. Restart or proactively tear down transfer clients.
  7. Add observability. Distinguish authentication failure, lookup miss, RPC timeout, and transfer error instead of collapsing everything into a cache miss.
Scenario Expected control-plane result Expected data-plane result
Correct key lookup/query/unlock succeed handshake and NIXL READ succeed
Missing key rejected before decode/dispatch no agent metadata or descriptors returned
Wrong key explicit UNAUTHENTICATED/PermissionError no remote handle created
Malformed auth bounded, fast failure with no secret leakage no NIXL metadata parsing
Key rotation old/new accepted inbound, one sent outbound new connections work; old-session lifetime is explicit
P2P with no key startup fails or requires explicit insecure opt-in never opens silently
Authenticated hit correct task and addresses populated L1 bytes copy correctly
Authenticated miss invalid address no read is submitted

8. Source map and final checklist

Read from the business flow downward:

  1. P2PController initializes the transfer context, polls /instances, maintains adapters, and implements the three P2P RPC handlers.
  2. P2PL2Adapter performs requester-side lookup, caches remote addresses, submits NIXL reads, and unlocks.
  3. nixl_impl.py owns agents, whole-L1 registration, the two-stage handshake, page-index translation, and transfer handles.
  4. registrar.py registers and heartbeats an MP server with the Coordinator.
  5. transfer_channel/api.py defines the small address and read-result model.

Five questions test whether the model is clear:

  • Must an in-flight NIXL read pass through the Coordinator? No.
  • Does query_lookup_results return KV? No, only offsets and sizes.
  • Does replacing ZMQ with gRPC move KV bytes through gRPC? No.
  • Does the P2P adapter write A’s KV into B? Not today; it is read-only.
  • Is adding an API key only to lookup RPC sufficient? No; the NIXL handshake must authenticate before disclosing descriptors.

In one sentence:

The Coordinator lets A find B; control RPC tells A which objects it may read and where they live; NIXL uses an imported remote-memory capability to copy those bytes into A; unlock ends the temporary reservation.

Discussion

Comments and Reactions

This article does not have an initialized discussion thread yet.