AI Infra

CUDA VMM, POSIX FDs, FABRIC, and LMCache from First Principles

A visual, interactive introduction to legacy CUDA IPC, virtual memory management, POSIX file-descriptor exports, FABRIC handles, IMEX channels, and the memory-access negotiation between vLLM's extensible KV cache and LMCache MP.

Maobaolong11/11/202626 min

The first encounter with these terms often makes them look like competing transfer protocols:

VMM / FD / FABRIC / IMEX / CUDA IPC

That mental model creates more questions than it answers. Is an fd a file containing GPU data? Does a FABRIC handle mean the bytes must travel over NVLink? Does VMM place GPU memory in a virtual machine? If the vLLM worker and LMCache server run on the same host, why not send the GPU pointer directly? What exactly does isolated_ipc isolate?

This article starts from one concrete problem:

A vLLM worker owns a GPU KV cache. The LMCache server is a different process. How can the server safely see and map the worker’s memory?

Once that boundary is clear, VMM, POSIX file descriptors, FABRIC, IMEX, and the automatic detection in LMCache PR #6499 become one causal chain rather than five unrelated terms.

The implementation status in this article is pinned to November 11, 2026. vLLM’s extensible KV cache #56492 and connector access contract #66605 are merged. The exportable POSIX-fd allocation work in #66674 is still open, and LMCache’s auto-detection #6499 is still a Draft. Follow the linked code for later changes.

CUDA memory sharing · visual field guide

The data is not inside the handle; the handle is a capability to map the allocation

Separate the physical memory, each process's address, the cross-process capability, and the channel that carries the capability. Nearly every term falls into place.

Virtual address: how this process sees it Physical memory: where the bytes live Shareable handle: who may remap it

1. Start without CUDA: why one process cannot send a pointer to another

Suppose a vLLM worker prints this address:

0x7f20_8000_0000

That number is not a global street address for a GPU location. It is meaningful only inside the current process and CUDA context. Another process that receives the same integer has no corresponding mapping. The same numeric address can refer to something else or to nothing at all.

CPU processes already work this way. Address 0x1234 in process A and 0x1234 in process B can map to different physical pages. Page tables translate each process’s virtual addresses into physical memory. GPUs have an analogous virtual-address and mapping model.

Cross-process GPU sharing therefore needs several distinct pieces:

1. Allocation

The physical GPU-memory object holding K/V bytes, not the pointer in a message.

2. Local mapping

The worker maps the allocation into its own GPU virtual address space so model kernels can use it.

3. Shareable handle

An exportable capability that lets the server refer to the same allocation.

4. Handle transport

The mechanism that carries the capability to the server; different handles require different channels.

5. Remote mapping

The server imports the handle and creates a mapping in its own address space, obtaining its own pointer.

6. Lifetime

The importer must stop using and unmap the memory before the exporter releases the allocation.

A pointer answers “where do I access it?” A handle answers “what authorizes me to map the same physical allocation?” The two processes do not need the same pointer value.

2. Legacy CUDA IPC: the traditional path for regular allocations

The classic path starts with a regular device allocation from cudaMalloc or cuMemAlloc. Process A calls cudaIpcGetMemHandle() or cuIpcGetMemHandle() to obtain an opaque IPC handle. Process B receives the handle and calls cudaIpcOpenMemHandle() or cuIpcOpenMemHandle(). The driver creates a mapping in B and returns a device pointer valid in B.

Process A                                      Process B

cuMemAlloc()                                  receive CUipcMemHandle
    │                                                  │
    ├─ local pointer                                   ├─ cuIpcOpenMemHandle()
    │                                                  │
    └─ cuIpcGetMemHandle() ── opaque handle ───────────┘
                                                       │
                                                       └─ B's local pointer

The handle does not contain the KV payload. Pointer values need not match, and the exporter still controls the allocation’s lifetime. Most importantly, this contract fits regular CUDA allocations; it does not describe the generic physical allocation handles created through CUDA VMM. NVIDIA documents this separately as legacy IPC. See the CUDA IPC programming guide.

When LMCache has use_vmm_api=false, a normal shared environment uses the PyTorch storage IPC wrapper, while isolated_ipc=true selects the driver-level RawCudaIPCWrapper. Both belong to the legacy CUDA IPC family. Neither one is a FABRIC handle.

3. CUDA VMM: reserve an address first, decide physical backing later

VMM means Virtual Memory Management. “Virtual” does not mean a virtual machine, and VMM does not automatically swap GPU memory to disk. It means the application explicitly controls the relationship between GPU virtual addresses and physical allocations.

A traditional cudaMalloc(size) bundles size selection, physical allocation, mapping, and pointer creation. CUDA VMM separates them:

  1. cuMemAddressReserve reserves a virtual-address range.
  2. cuMemCreate creates a physical allocation handle.
  3. cuMemMap maps a physical allocation into a virtual range.
  4. cuMemSetAccess grants device access.
  5. Cleanup proceeds through cuMemUnmap, cuMemRelease, and cuMemAddressFree.

Separate the address from physical backing

1 / 4

Virtual address rangestable pointer seen by CUDA graphs VA 0VA 1VA 2VA 3 Physical allocations chunk Achunk Bchunk C shareablehandle
The diagram omits allocation granularity, device access permissions, and the strict unmap/release sequence, but preserves the ownership model.

VMM is valuable because the address stays stable while backing can be bound later and controlled in physical chunks. NVIDIA also describes per-allocation peer access, fragmentation control, dynamic growth, and memory sharing. See the CUDA VMM programming guide.

Common mistake: a VMM allocation is not a regular `cudaMalloc` allocation with a new label. Its physical memory is represented by a `CUmemGenericAllocationHandle`, so legacy pointer-based IPC is not the right sharing contract.

4. POSIX FDs, FABRIC handles, and IMEX

A VMM physical allocation must declare its exportable handle types when it is created. LMCache currently cares about two.

4.1 POSIX file descriptors: the OS transfers the capability

With CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR, the worker can call:

cuMemExportToShareableHandle(..., POSIX_FILE_DESCRIPTOR, ...) -> fd

This fd is not a normal data file containing KV bytes. It is a process-local entry in the Linux descriptor table that references a driver memory object.

Serializing the integer value is insufficient. If the worker prints fd = 17, the number 17 in the server may name a log file or nothing at all. A real fd transfer uses ancillary data on a Unix-domain socket, specifically SCM_RIGHTS; the kernel installs a new descriptor in the receiving process that references the same object.

LMCache therefore has an out-of-band fd channel:

/dev/shm/lmcache_fd_<port>.sock
AF_UNIX + SOCK_SEQPACKET + SCM_RIGHTS

The request path may still use ZMQ or gRPC over TCP. That does not make an fd transferable over TCP. Worker and server must also see the same Unix-socket path, which generally means sharing the relevant /dev/shm in container deployments. The exact LMCache protocol is documented in vmm_cuda_ipc.md.

4.2 FABRIC handles: a driver-managed serializable capability

CUDA 12.4+ provides CU_MEM_HANDLE_TYPE_FABRIC. The exported CUmemFabricHandle is an opaque blob. LMCache carries its current 64-byte representation in the descriptor, so it does not require a shared Unix-socket path.

FABRIC is not free container-crossing magic. Exporter and importer must have device support, a running IMEX setup, access to the same IMEX channel, and an allocation created with the FABRIC handle type.

IMEX stands for inter-node memory exchange. The channel is a driver-enforced security and isolation boundary. A typical device node is:

/dev/nvidia-caps-imex-channels/channel0

NVIDIA documents an IMEX domain as either one OS or multiple OS instances securely connected through the NVIDIA IMEX daemon. On supported NVLink-fabric systems, FABRIC handles can support inter-node mapping. That does not by itself mean LMCache MP currently implements arbitrary cross-node KV-pool mapping; here the handle solves the worker/server allocation-sharing boundary.

4.3 Choose the conditions and inspect the path

Change deployment conditions and observe the sharing path

allocation
isolated IPC
same IMEX channel
vLLM workerowns KV allocationexports capability AF_UNIX + SCM_RIGHTSKV bytes stay in GPU memory POSIX FD LMCache serverimports handlemaps same allocation

The compressed rule is: regular allocations use legacy IPC; VMM allocations use a shareable handle requested at creation time. A non-isolated deployment can use a POSIX fd. An isolated deployment generally needs FABRIC plus a shared IMEX authorization domain.

5. Why vLLM needs VMM: CUDA graphs want stable addresses, but the KV cache wants late sizing

The KV cache often consumes most of the available GPU memory. vLLM historically had to size it before CUDA graph capture because captured graphs bake in addresses.

The final free-memory amount is only clear after graph capture, warmup, attention workspaces, sampling buffers, and speculative-decoding initialization. Early profiling either underestimates and risks OOM, or reserves excessive slack and wastes memory that could hold KV cache.

vLLM #56492 uses VMM to break that dependency:

before graph capture: reserve a large virtual-address range
during capture/warmup: keep those addresses stable
after final sizing: create and map only the required physical chunks

The model sees stable pointers while physical backing is chosen after final sizing. The V2 CUDA and ROCm model runners enable the extensible cache by default.

External connectors introduce another constraint. A virtually contiguous segment may contain multiple physical chunks. RDMA registration may require a single physical allocation, and LMCache MP needs to export that allocation to another process. vLLM cannot choose correctly unless each connector declares how it accesses the cache.

That is the purpose of KVCacheAccess from merged PR #66605:

Access Connector behavior Allocation requirement
IN_PROCESS kernels or copies inside the worker any allocation
REGISTERS UCX/RDMA registration one physical chunk per segment
EXPORTS_FD another process imports a POSIX fd single chunk, fd-exportable
EXPORTS_FABRIC another process imports a FABRIC handle single chunk, FABRIC-exportable, IMEX
LEGACY_IPC connector only supports legacy IPC extensible VMM cache is incompatible

The direction matters: vLLM does not parse LMCache-private flags. LMCache declares its memory-access contract, and vLLM combines that with platform support, IMEX availability, and other MultiConnector children.

As of November 11, 2026, the contract is merged, while actual EXPORTS_FD backing for extensible allocations remains in #66674. An enum value existing in main does not imply every allocator path can already produce it.

6. The LMCache MP path: export a handle, register a tensor view

The vLLM worker owns the GPU KV cache. LMCache server must map it from another process before it can store or retrieve blocks. The complete path has six steps:

1. Declare connector access

Before allocation, return `EXPORTS_FD`, `EXPORTS_FABRIC`, or `LEGACY_IPC` so vLLM knows the sharing requirement.

2. Resolve the allocation

vLLM combines platform support and all connectors to choose extensible mode and requested handle types.

3. Select the LMCache wrapper

The worker follows the resolved extensible/cuMem state and selects `VmmCudaIPCWrapper`; an explicit `use_vmm_api` remains an override.

4. Export the physical handle

The wrapper locates the allocation behind the pointer and exports a POSIX fd or 64-byte FABRIC handle together with shape, stride, and offset metadata.

5. Deliver descriptor and capability

A FABRIC handle travels with the descriptor. A POSIX fd must travel over the fd channel through `SCM_RIGHTS`, paired to the request by an export ID.

6. Import and map on the server

The server imports the same physical allocation, reserves and maps its own address, then rebuilds the tensor view. Registration does not copy the whole KV pool.

The request channel and fd channel have different jobs:

ZMQ / gRPC request channel
  └─ config, tensor metadata, export IDs, status

AF_UNIX fd channel (POSIX-fd path only)
  └─ the real descriptor capability via SCM_RIGHTS

GPU allocation
  └─ KV bytes remain on GPU; both processes map the same object

FABRIC avoids the fd channel because its handle is copyable opaque data, but replaces the shared-filesystem dependency with an IMEX authorization dependency. Neither path is universally “more advanced”; they represent different deployment and security boundaries.

7. Why automatic detection is necessary

Previously users had to set:

{
  "lmcache.mp.use_vmm_api": true
}

But vLLM’s final allocation is resolved from model-runner behavior, platform capabilities, extensible mode, cuMem allocator settings, connector access, IMEX, and possibly multiple connector children. Asking the user to predict that result creates a circular dependency:

LMCache: Are you using VMM? I need to select a wrapper.
vLLM: How do you need to share memory? I need to select an allocation.

LMCache #6499 splits this into two phases:

  1. Declare requirements before allocation. With no explicit override, normal IPC returns EXPORTS_FD, isolated IPC returns EXPORTS_FABRIC, and explicit false returns LEGACY_IPC.
  2. Follow the resolved result after allocation. During worker-adapter construction, LMCache derives the real mode from vLLM’s resolved enable_extensible_kv_cache or enable_cumem_allocator state.

Explicit true/false values remain authoritative for compatibility and rollback. Older vLLM revisions without KVCacheAccess receive None and retain import compatibility.

User settingPre-allocation declarationWorker behaviorMain requirement
unsetEXPORTS_FDfollow vLLM's resolved allocationfd-exportable extensible allocation and a shared fd-channel path
unset + isolatedEXPORTS_FABRICfollow vLLM's resolved allocationFABRIC support and the same IMEX channel on both sides
falseLEGACY_IPCforce legacy wrappervLLM must not select a VMM-only sharing path
trueFD or FABRICforce VMM wrapperthe actual allocation must export the matching handle
vLLM #56492

Reserve addresses before capture and map physical KV-cache chunks after final sizing.

vLLM #66605

Add `KVCacheAccess` so connectors declare registration, fd, FABRIC, or legacy requirements.

vLLM #66674

Create fd-exportable extensible allocations for `EXPORTS_FD` and compose FABRIC handle types.

LMCache #6499

Declare access automatically and make the worker follow vLLM's resolved allocation.

The complete production path exists only when the access contract, allocation support, LMCache export/import implementation, and deployment environment all agree.

8. Deployment and debugging checklist

Determine the allocation family first

  • Is vLLM using the extensible KV cache or the cuMem allocator?
  • Did LMCache select VmmCudaIPCWrapper from the resolved engine configuration?
  • Do not infer the result only from whether the user wrote use_vmm_api.

For POSIX FDs

  • Does the device support CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR?
  • Was the allocation created with that requested handle type?
  • Can worker and server see /dev/shm/lmcache_fd_<port>.sock?
  • Did the Linux fd channel start and connect?
  • Never serialize the raw fd integer as if it were the capability.

For FABRIC

  • Does the device report FABRIC-handle support?
  • Is nvidia-caps-imex-channels present?
  • Can exporter and importer access the same channel device?
  • Did allocation creation request the FABRIC handle type?
  • For CUDA_ERROR_NOT_PERMITTED, inspect IMEX access before tensor geometry.

For legacy IPC

  • Is the allocation truly regular CUDA memory rather than VMM backing?
  • Does the exporter remain alive and retain ownership?
  • Does the importer close its mapping before release?
  • Does an isolated deployment use the raw wrapper instead of shared storage IPC?

Finally, handle success only proves that the process obtained the memory object. Rebuilding the correct tensor still requires dtype, shape, stride, storage offset, and byte span. Hybrid-model padding is a separate layer; see How a Legal Memory Gap Breaks LMCache CUDA IPC.

Four sentences to keep

  1. VMM is an allocation model that separates virtual-address reservation from physical-memory creation and mapping.
  2. FD and FABRIC are shareable-handle types for VMM allocations, not KV payloads or ordinary network protocols.
  3. FD relies on the OS to transfer a capability; FABRIC relies on the driver and an IMEX authorization domain.
  4. Auto-detection is a two-phase negotiation: LMCache declares access needs, vLLM resolves allocation, and the worker follows the resolved result.

The vLLM worker and LMCache server each have their own pointer. The handle lets both pointers map the same physical allocation. Cross-process sharing is not about sending an address; it is about securely transferring the capability to create a valid mapping.

References

Discussion

Comments and Reactions

This article does not have an initialized discussion thread yet.