CnuasGPU as a Standalone Soft-GPU¶

This page is published from docs/CnuasGPU_SoftGPU.md in the PacketFive/CnuasGPU submodule, which is the authoritative copy. Edit it there and refresh this one, rather than editing this file alone.

CnuasGPU runs as a software accelerator on an ordinary host, outside QEMU and without a guest. This document describes that form, what it provides, and how an inference runtime uses it.

It also sets out how CnuasGPU maps onto a tile-based accelerator architecture (TTU / TCP / CCP / TVP).

1. Standalone operation¶

CnuasGPU runs outside QEMU in two forms. The host device backend needs no kernel module, no /dev/cnuasgpuN, no guest and no configuration; it is the form this section describes and the one selected automatically when no character device node is present. The hostdev backend adds a character device, /dev/cnuasgpu_hostN, served from host pages by cnuasgpu_host.ko with no PCI function; it is described in CnuasGPU Host Character Device and compared with the node-less form in section 1.2.

CNUAS_DEVICE_BACKEND forces the choice, host, device or hostdev. Unset, the search order is /dev/cnuasgpuN, then /dev/cnuasgpu_hostN, then the in-process arena, so a host node is never chosen in preference to an emulated PCI device. Section 3.2.1 describes the integration that automatic selection serves.

The backend is implemented in lib/libcnuasdev/cnuasdev_host.c and is covered by tests that fail if it is removed:

Evidence	What it shows
`lib/test/cnuasdev_host_smoke`, 29 checks	The backend itself: enumeration, alloc alignment, H2D/D2H/D2D, bounds rejection, arena exhaustion, and that the pointer from `cnuasdev_mmap` survives `cnuasdev_munmap`.
`lib/test/cnuasrt_activation_smoke`, 19 checks	The full public runtime path, `cnuasInit` through `cnuasMalloc`, `cnuasMemcpy` and `cnuasReluF32`. With the backend deselected it reports skip, so it cannot pass for an unrelated reason.
ThreadSanitizer, 8 threads	Concurrent allocation is race free and hands out no duplicate offsets; removing the mutex reports races, so the test has detection power.

1.1 Why the host device is `/dev/cnuasgpu_hostN`¶

There are two host forms, and one does have a device node. The node-less host backend keeps its arena in the calling process and intentionally creates no node. The hostdev backend is implemented by cnuasgpu_host.ko and creates /dev/cnuasgpu_hostN without QEMU, a guest, or a PCI function.

The distinct name preserves the meaning of /dev/cnuasgpuN: those nodes are created by cnuasgpu_probe() only when the PCI bus matches an emulated CnuasGPU function. The PCI driver keeps /dev/cnuasgpuN, its own major, minor space and cnuasgpu class, while the host module owns /dev/cnuasgpu_hostN and the cnuasgpu_host class. Neither module depends on the other, and software can distinguish an emulated PCI device from a host-pages device without guessing from capabilities.

Both modules expose the same core ioctl and mmap contract. The implementation below it differs: the PCI driver accesses BAR registers and BAR memory, whereas the host module allocates and maps kernel-owned pages. This lets the user--kernel ABI, pinned mappings, DMA-BUF export and asynchronous queues run without a VM. The design, parameters and limits are in CnuasGPU Host Character Device.

References below to the node-less form mean the host backend specifically, not both host deployment modes.

The layers that run with no device are:

Layer	Runs with no device?
Compute backends (`cnuas_compute_ops`, including the activations)	Yes. `compute_backend_smoke` links the backend archive directly and needs neither a device nor a guest.
CnuasBLAS	Yes, through the test stub and through the host backend.
libcnuasrt public API (`cnuasMalloc`, `cnuasSgemm`, `cnuasReluF32`, ...)	Yes, on any host with no device node, without being asked. Where a node is present the character device is used, and `CNUAS_DEVICE_BACKEND=host` overrides that.

The host test suite runs with 0 skipped.

Two limits apply. The host backend emulates the interface of the device, not its timing or its memory system, so it serves correctness and bring-up rather than performance claims. Its allocator is a bump pointer that reclaims only the most recent block, matching the kernel driver rather than improving on it, so that code which works here also works against the emulated device.

The host backend is what allows a ggml backend to allocate and hand back buffers without a guest.

1.2 Node-less backend versus a host character-device backend¶

These are complementary deployment modes, not a fast and a slow edition of the same path. Both are implemented: CNUAS_DEVICE_BACKEND=host selects the node-less arena and CNUAS_DEVICE_BACKEND=hostdev selects the host node.

Area	Node-less host backend	Host character-device backend
Deployment	Ordinary shared libraries; no root, module, DKMS, udev rule or matching host kernel	Kernel module plus `/dev/cnuasgpu_hostN`; installation and kernel compatibility must be managed
Application API	CnuasRT and CnuasDev APIs work; applications that require a Linux GPU node do not see one	Same runtime APIs plus a discoverable device node and the QEMU-device ioctl/mmap ABI
Small-operation overhead	Lowest: allocation, copy and dispatch remain function calls in one process	Higher control-path cost from open/ioctl/poll and kernel transitions
Mapped-memory access	Direct process address; no kernel transition after allocation	Can also be direct after `mmap`, but setup, validation and teardown cross the kernel
Compute execution	CPU scalar/AVX2/AVX-512 backend runs in the caller	The node alone adds no compute engine; a userspace worker/backend still performs compute unless acceleration is added separately
Process isolation	Allocations are private process state and disappear with the process, but the backend cannot enforce a system-wide device policy	Kernel owns allocation/context lifetime and can validate mappings and permissions independently of each client
Multi-process sharing	No common allocation namespace or DMA-BUF export	Implemented DMA-BUF export of pinned allocations, a pollable completion queue, and per-node accounting
Peer-memory/RDMA	The process arena cannot use the CnuasGPU DMA-BUF export ABI	Uses the same export request as `cnuasgpu.ko`, but exports host pages rather than a BAR window
Resource governance	Per-process arena limit only; no central quotas, ownership, cgroup or device ACL enforcement	Device-node permissions, centralized accounting and future quota/cgroup policy are possible
Observability	Process-local logs and runtime counters	Stable sysfs/debugfs/device inventory and system-wide counters are possible
Failure boundary	Backend faults normally terminate only the calling process	Kernel bugs can affect the host; userspace worker faults can be isolated if the design keeps compute out of the module
ABI fidelity	Runtime-level fidelity; bypasses the ioctl, mmap and device lifetime path	Exercises the same user/kernel contract as the QEMU PCIe device without requiring a VM
Portability	Any supported userspace host architecture	Requires a supported Linux kernel; not the default path on Windows or macOS

Performance expectation¶

The existing node-less backend is not expected to be less performant merely because it lacks /dev/cnuasgpuN. For the current CPU compute backends it is normally the lower-overhead implementation:

calls do not cross the user/kernel boundary;
arena addresses are already in the caller;
no ioctl is required for allocation or copies; and
no daemon or worker-process context switch is required.

A host character device does not make scalar, AVX2 or AVX-512 arithmetic faster. If it maps the same pages and invokes the same userspace compute backend, large-kernel throughput should be similar while short operations can be slower because fixed syscall and synchronization costs are more visible.

The character-device design can outperform the node-less design only when it enables a capability that changes the data path: shared zero-copy mappings between processes, batched asynchronous queues, pinned memory, DMA-BUF exchange, peer devices, or a future FPGA/other hardware execution engine. The first four mechanisms now exist, but their benefit remains workload-dependent rather than proof that the device-node path is faster.

No performance ordering is claimed until both implementations are measured with the same workload, host, backend, arena size, thread count and copy policy. The comparison must report at least allocation latency, H2D/D2H/D2D bandwidth, launch latency, steady-state kernel throughput, concurrent-client scaling, CPU utilization, and memory footprint.

Selection guidance¶

Use the node-less backend for unprivileged installation, local inference, CI, numerical development, one-process applications, and the lowest current control-path overhead.

Use the host character-device backend when testing the real driver ABI without a VM, integrating software that discovers Linux device nodes, exporting DMA-BUFs, sharing resources across processes, enforcing system-wide ownership, or collecting system-wide device telemetry.

The node-less backend remains supported. Replacing it would add kernel requirements to workflows such as Ollama and ggml that currently need none.

2. Mapping onto a tile accelerator (TTU / TCP / CCP / TVP)¶

A conventional tile-based accelerator is usually described in four parts. Measured against that vocabulary, CnuasGPU stands as follows.

Unit	Role	State in CnuasGPU
TTU, Tile Tensor Unit	Matrix multiply with direct accumulation	Partial. `sgemm` computes `alpha·AB + beta·C`, so accumulation into a destination tile is present, and the AVX-512 backend already blocks internally with a 16×16 `C` tile and MC/KC/NC panels. What is absent is an exposed tile abstraction: callers cannot address a tile register file or issue an accumulate-into-tile primitive. `cnuasdev_device_info.tensor_size` exists and is 0 when there is no such unit, which is what the host backend reports.
TCP, Tile Control Processor	Pointer arithmetic and loop control feeding the tensor unit	Effectively present. This is what the CnuasIR scalar subset does, and what the CnuasCC code generator emits: induction variables, address computation, bounded loops.
CCP, Cluster Control Processor	Higher-level sequencing across several tiles/cores	Absent. There is no cluster-level scheduler; dispatch is one call, one backend, one thread team.
TVP, Tile Vector Processor	Elementwise/activation functions	Present (ABI 0.2). `relu`, `gelu`, `silu`, `sigmoid`, `tanh` and row-wise `softmax`, vectorised for AVX2 and AVX-512 by `lib/compute/common/cnuas_activations_simd.h`. See `CnuasGPU_Compute_Backends.md` §3.2.1.

In summary, CnuasGPU has a competent TCP, a usable but unexposed TTU, a TVP, and no CCP.

3. What an inference runtime actually needs¶

3.1 ggml / llama.cpp¶

Verified against llama.cpp at commit 0b1bad14ff20.

A backend is a shared object exporting ggml_backend_init() (via GGML_BACKEND_DL_IMPL) and optionally ggml_backend_score(). The loader dlopens it, checks an integer api_version, and registers its devices (ggml/src/ggml-backend-reg.cpp). ggml_backend_load_all_from_path() honours the GGML_BACKEND_PATH environment variable, which the code documents as the out-of-tree path.

Two consequences matter:

A partial backend is legitimate. There is no fixed minimum operation list; supports_op() is consulted per graph node. The ggml-blas backend supports only MUL_MAT and OUT_PROD under narrow layout and type conditions, plus view/no-op nodes, and returns false for everything else. ggml_backend_sched then splits the graph and routes the remainder to CPU, which is assumed to be the last backend in priority order.
Claiming an operation that graph_compute cannot execute is fatal, so supports_op() must be conservative rather than optimistic.

ggml-blas is therefore the right template: it reuses ggml_backend_cpu_buffer_type() and imports host pointers with ggml_backend_cpu_buffer_from_ptr(), which matches CnuasGPU's host backend exactly. ggml-cann and ggml-opencl assume a real device runtime and separate device buffers; ggml-rpc is a proxy/transport backend, not a kernel backend, and would only add serialisation overhead in the same-process case. (ggml-rpc becomes interesting only if the QEMU/VM boundary is the point - architecturally that resembles ggml-virtgpu, which remotes ggml through virtio_gpu.)

The binding constraint is ABI, not design: the plugin must be built against the same ggml revision as the host application, and dynamic loading requires GGML_BACKEND_DL with shared libraries.

3.2 Ollama¶

Ollama does not vendor a full llama.cpp fork; it fetches a pinned upstream revision with CMake FetchContent, applies a compatibility patch, and runs llama-server as a subprocess. Its default preset sets BUILD_SHARED_LIBS=ON and GGML_BACKEND_DL=ON, so on current Linux and Windows packages an out-of-tree backend can be loaded without rebuilding or patching Ollama. macOS packages set both options off and statically embed Metal, so this does not apply there.

This has been run rather than reasoned about. Ollama 0.32.9, which ships ggml 0.19.0, was installed from the upstream release archive and started as

$ GGML_BACKEND_PATH=/abs/path/libggml-cnuas.so ollama serve
$ ollama run qwen3:0.6b "..."

with no Cnuas variable set anywhere. The model answers correctly, and 41 of its matrix multiplications were measured executing on CnuasGPU during prompt processing, at shapes 1024x510x1024 and 1024x510x3072, for 71.7 GFLOP in total.

3.2.1 The environment is not propagated¶

The first attempt did not work, and the reason is worth recording because it is invisible from reading the code. It aborted in the device sgemm, and the debug log showed why. Ollama does not hand its own environment to the runner. It passes exactly three variables:

GGML_BACKEND_PATH=...  PATH=...  LD_LIBRARY_PATH=...

CNUAS_DEVICE_BACKEND=host, set in the parent, therefore never arrived. No PCI or host-device node was installed for that experiment, so automatic device discovery found neither /dev/cnuasgpuN nor /dev/cnuasgpu_hostN, reported zero device memory, and failed on the first multiply.

An earlier revision of this document asserted that Ollama propagates os.Environ() into the runner. That is not what it does, and the assertion is withdrawn.

The fix belonged in this project rather than in Ollama. A Soft-GPU whose only route to being a Soft-GPU is an environment variable that the embedding application refuses to forward is not usable, so cnuasdev_host_enabled now selects the host arena automatically when no character device node is present. CNUAS_DEVICE_BACKEND still forces the question either way, host or device, and the automatic case announces itself once on stderr so that a caller which believed it was driving real hardware can discover from the log that it was not. Automatic selection cannot mask a missing kernel module inside a guest, because there the node exists and is therefore chosen.

3.2.2 What runs on the accelerator and what does not¶

Only the prompt is processed on CnuasGPU under stock settings. The backend declines any multiply whose batch dimension is below CNUAS_GGML_MIN_BATCH, which defaults to 32, the same shape test ggml-blas applies. Autoregressive decode is a sequence of n=1 multiplies and so stays on the CPU. Since that variable is filtered out too, only the default is reachable from Ollama, and the split is therefore fixed at prompt on the accelerator, decode on the host. This is the same division of labour ggml-blas produces, and it is a deliberate consequence of the round trip through device memory rather than an oversight.

3.3 vLLM¶

vLLM has a genuine out-of-tree plugin mechanism (the vllm.platform_plugins entry point, used by vllm-ascend, vllm-spyre, vllm-gaudi, vllm-openvino), and a PyTorch PrivateUse1 device is not formally required. Platform.dispatch_key defaults to "CPU", and vllm-openvino declares device_type = "cpu".

But a platform must supply attention backends, worker and model-runner integration, and collectives, all built around torch.Tensor. With an fp32-only operation set and no quantised or attention kernels, a CnuasGPU platform would execute nearly everything on Torch CPU and intercept a few fp32 linear layers. That is substantial integration risk for little benefit.

Recommendation: ggml/llama.cpp first, vLLM not yet.

4. The performance question¶

A soft GPU running on the same host CPU cannot beat that CPU's native inference path. It adds indirection over the same silicon.

Against native llama.cpp CPU inference:

ggml has optimised quantised matrix-vector and matrix-matrix kernels with AVX2, AVX-512, VNNI, BF16 and AMX paths. CnuasGPU has fp32 SGEMM.
Typical GGUF models are quantised; serving them from CnuasGPU means dequantising or keeping fp32 copies, which multiplies memory traffic.
Autoregressive decode is dominated by low-batch matrix-vector work. Even llama.cpp's own BLAS backend refuses small matrices, because BLAS wins only on large ones.
Graph splitting adds synchronisation boundaries around each offloaded region.

Expect at best parity on large prefill GEMMs, and materially slower token decode. If an SGEMM path does match, the credit belongs to the underlying microkernel, not to emulation.

There is no precedent to appeal to. Searches of current llama.cpp and vLLM for gem5, verilator or simulator return no backend or platform integration whose purpose is to execute through a hardware simulator.

The value of a CnuasGPU ggml backend is not speed. It is:

bringing up and testing an accelerator software stack before silicon exists;
conformance-testing a backend implementation against a reference, in CI, with no hardware;
giving a reproducible, inspectable target for teaching and for research on accelerator software.