Skip to content

Cnuas Peer Memory

cnuas-peermem lets a CnuasNIC memory region refer directly to an allocation in the BAR1 device-memory window of a CnuasGPU. It uses the upstream Linux DMA-BUF RDMA interface; it does not implement or emulate the removed ib_register_peer_memory_client interface used by older versions of nvidia-peermem.

For the concise feature, interface, integration, and validation summary, see the Cnuas Peer Memory datasheet.

Experimental, in progress

The exporter, userspace library, standard verbs registration path, and CnuasNIC importer are implemented. Host ABI and ownership tests pass, but live allocation-lifetime and GPU-to-RDMA transfer validation remains gated on a guest exposing both devices. End-to-end operation is not yet claimed.

Data path

flowchart LR APP["RDMA application"] LIB["libcnuaspeermem"] GPUFD["/dev/cnuasgpuN"] DMABUF["DMA-BUF fd"] VERBS["ibv_reg_dmabuf_mr"] NIC["cnuas_ib.ko"] BAR["CnuasGPU BAR1 allocation"] APP --> LIB LIB -->|"EXPORT_DMABUF ioctl"| GPUFD GPUFD --> DMABUF LIB --> VERBS VERBS --> NIC NIC -->|"I/O-safe MR reads and writes"| BAR DMABUF --> BAR
  1. Allocate device memory through the existing CnuasGPU allocation API.
  2. Export an owned, page-aligned allocation range with CNUASGPU_IOC_EXPORT_DMABUF.
  3. Register the returned file descriptor with a CnuasNIC protection domain through standard ibv_reg_dmabuf_mr().
  4. Use the returned ibv_mr in SEND, RECV, RDMA READ, RDMA WRITE, or atomic work requests.
  5. Deregister the MR, close the DMA-BUF, and then free the GPU allocation.

The libcnuaspeermem library performs steps 2 and 3 and makes ownership explicit. It never closes a caller-owned GPU device or protection-domain handle.

Kernel implementation

CnuasGPU export

cnuasgpu.ko exports only an existing allocation or a page-aligned subrange owned by the calling file. Unsupported flags, nonzero reserved fields, misaligned ranges, overflow, and out-of-bounds ranges are rejected.

An allocation with an open export cannot be explicitly freed. Closing the original GPU file does not invalidate an exported DMA-BUF; an orphaned allocation is reclaimed after its final export closes. Device removal prevents new exports and defers BAR teardown until existing exports are released.

CNUASGPU_IOC_CAPS and the peer_rdma, export_align, and exports sysfs attributes report the interface and live export state.

CnuasNIC import

cnuas_ib.ko implements ib_device_ops.reg_user_mr_dmabuf. The uverbs core resolves the DMA-BUF fd and invokes the driver through the standard DMA-BUF MR registration command.

Ordinary user-memory MRs retain their ib_umem backing. DMA-BUF MRs retain the buffer and its iosys_map for the MR lifetime. Shared, bounds-checked MR helpers service both kinds of backing throughout SEND/RECV, RDMA READ/WRITE, and atomic paths. BAR memory is accessed with I/O-memory-safe copies rather than being treated as normal pinned RAM.

Scope and interpretation

This implementation removes the host-memory registration requirement from the Cnuas software-RNIC path. The emulated NIC still serializes network packets and copies their payload in software; it is not a claim of physical PCIe peer-to-peer DMA bandwidth or latency.

Capability Status
CnuasGPU allocation and subrange DMA-BUF export Implemented
Export lifetime after the GPU file closes Implemented
Standard DMA-BUF MR registration in CnuasNIC Implemented
Host-memory and DMA-BUF MR access through one data path Implemented
libcnuaspeermem shared and static libraries Implemented
Host ABI and ownership tests Passing
Guest GPU-to-RDMA transfer smoke Gated on CnuasGPU and CnuasNIC devices
In-process Soft-GPU peer-memory export Not supported; the node-less arena has no device fd
Host character-device DMA-BUF export Implemented for pinned allocations through /dev/cnuasgpu_hostN; live RDMA transfer validation remains gated
Legacy NVIDIA peer-memory client API compatibility Not supported
Physical FPGA or PCIe peer-DMA validation Not performed

Build and validation

make -C src/cnuasgpu/peermem
make -C src/cnuasgpu/peermem check

# Skips clearly unless a guest exposes the required GPU and RDMA devices.
make -C src/cnuasgpu/peermem check-device

The library requires a libibverbs development package that exposes ibv_reg_dmabuf_mr(). A build without that API can still compile the kernel-independent ABI tests, but cannot register a peer MR.

This path is separate from CnuasLink. CnuasLink carries framed GPU-to-GPU traffic through the accelerator fabric; cnuas-peermem registers GPU memory with the CnuasNIC RDMA stack.