Cnuas Peer Memory, Datasheet¶

Item	Value
Part	`libcnuaspeermem`
Type	GPU-memory export and RDMA registration library
Version	0.1.0
Status	Implemented and host-tested; live guest transfer validation is gated
Repositories	PacketFive/CnuasGPU, PacketFive/CnuasNIC

Functional scope

Cnuas peer memory exposes CnuasGPU memory to the software CnuasNIC through the standard Linux DMA-BUF RDMA interface. It does not claim physical PCIe peer-to-peer bandwidth, latency, or direct hardware DMA.

1. Overview¶

Cnuas peer memory lets a CnuasNIC memory region name an allocation in a CnuasGPU device-memory window without a host staging buffer in the application's data flow. cnuasgpu.ko exports an owned, page-aligned allocation or subrange as a DMA-BUF. libcnuaspeermem passes that descriptor to standard ibv_reg_dmabuf_mr(), and cnuas_ib.ko retains an I/O-safe mapping for the memory-region lifetime.

This path is separate from CnuasLink. Peer memory connects GPU memory to the RDMA network through CnuasNIC; CnuasLink carries framed GPU-to-GPU traffic over the accelerator fabric.

2. Data path¶

flowchart LR APP["RDMA application"] LIB["libcnuaspeermem"] GPU["cnuasgpu.ko"] BUF["DMA-BUF"] VERBS["ibv_reg_dmabuf_mr()"] NIC["cnuas_ib.ko"] APP --> LIB LIB -->|"export allocation"| GPU GPU --> BUF LIB --> VERBS BUF --> VERBS VERBS --> NIC

Allocate page-aligned CnuasGPU memory.
Export the allocation or an owned subrange as a DMA-BUF.
Register the descriptor with a CnuasNIC protection domain.
Use the resulting lkey or rkey in RDMA work requests.
Deregister the memory region and close the peer-memory region.

3. Functional specifications¶

Capability	State
Whole-allocation and page-aligned subrange export	Implemented
Read-only and close-on-exec export flags	Implemented
Export lifetime after the originating GPU descriptor closes	Implemented
Standard `ibv_reg_dmabuf_mr()` registration	Implemented
SEND and RECV access through DMA-BUF-backed MRs	Implemented
RDMA READ and WRITE access through DMA-BUF-backed MRs	Implemented
Atomic access through DMA-BUF-backed MRs	Implemented with software serialization
Shared and static userspace libraries	Implemented
Host ABI, validation, ownership, and error-path tests	Passing
Live guest export-lifetime test	Gated on `/dev/cnuasgpuN`
Live CnuasGPU-to-CnuasNIC transfer test	Gated on GPU and RDMA devices

The implementation uses the upstream DMA-BUF verbs path. It does not implement the removed out-of-tree ib_register_peer_memory_client() API used by legacy peer-memory clients.

4. Userspace interface¶

Header: cnuas_peermem.h

Group	Interface
Capability discovery	`cnuas_peermem_have_verbs`, `cnuas_peermem_have_dmabuf_mr`, `cnuas_peermem_query_caps`
Export	`cnuas_peermem_export`
Registration	`cnuas_peermem_register`, `cnuas_peermem_unregister`
RDMA keys	`cnuas_peermem_lkey`, `cnuas_peermem_rkey`, `cnuas_peermem_mr`
Region properties	`cnuas_peermem_fd`, `cnuas_peermem_offset`, `cnuas_peermem_size`, `cnuas_peermem_flags`
Lifetime	`cnuas_peermem_close`
Diagnostics	`cnuas_peermem_build_info`

Functions return zero or a negative errno and do not abort or log. The caller retains ownership of the GPU descriptor and protection domain. A peer-memory region owns only the DMA-BUF descriptor and verbs memory region that it creates.

5. Kernel interfaces¶

Component	Interface	Purpose
CnuasGPU	`CNUASGPU_IOC_CAPS`	Report export support, alignment, and limits
CnuasGPU	`CNUASGPU_IOC_EXPORT_DMABUF`	Export an owned allocation range
CnuasNIC	`ib_device_ops.reg_user_mr_dmabuf`	Import a DMA-BUF as an RDMA memory region
rdma-core	`ibv_reg_dmabuf_mr()`	Standard userspace registration operation

The GPU exporter keeps exported allocations and device state alive until the last DMA-BUF reference closes. The NIC importer uses iosys_map accessors so a device BAR is not treated as ordinary host memory. Memory regions are reference-counted across posted receives and in-flight READ or atomic work.

6. Build and validation¶

make -C src/cnuasgpu/peermem
make -C src/cnuasgpu/peermem check
make -C src/cnuasgpu/peermem check-device

Test	Environment	Coverage
`peermem_abi_smoke`	Any build host	ABI layout, ioctl values, flags, alignment, bounds, overflow, reserved fields
`peermem_api_smoke`	Any build host	Ownership, NULL handling, errno behavior, invalid descriptors
`peermem_lifetime_smoke`	Guest with CnuasGPU	Export lifetime, busy allocation, subranges, orphan reclaim
`cnuas-peermem-smoke`	Guest with CnuasGPU and CnuasNIC	Export, DMA-BUF MR registration, RC transfer, byte comparison

Device-dependent tests return exit status 77 when their required interfaces are absent. A skip confirms the gate behaved correctly; it does not validate the live transfer path.

7. Integration information¶

Requirement	Value
GPU device	`/dev/cnuasgpuN`; `/dev/cnuasgpu_hostN` can export pinned host pages through the same ABI
GPU driver	`cnuasgpu.ko` or `cnuasgpu_host.ko` with DMA-BUF enabled
RDMA driver	`cnuas_ib.ko` with `reg_user_mr_dmabuf`
Userspace RDMA	rdma-core exposing `ibv_reg_dmabuf_mr()`
Build without verbs	Export and capability APIs remain available; registration returns unsupported
Detailed design	Cnuas Peer Memory
Validation status	Validation Matrix

8. Current limits¶

Full live guest transfer validation remains gated.
The software RNIC copies payloads with the CPU; this is not physical peer-to-peer DMA.
Peer-memory atomics are serialized within CnuasNIC and cannot serialize against independent GPU-side mutation.
Legacy nvidia-peermem client API compatibility is not provided.

9. Revision history¶

Revision	Notes
A	First publication. Records the implemented DMA-BUF exporter, userspace library, CnuasNIC importer, host tests, and gated live-device tests.