Support vendor VFIO vGPU devices

[yummybomb]

background

The current migration target is NVIDIA vGPU Manager R19 (R580) on Ubuntu 24.04 / kernel 6.8. For SR-IOV vGPUs, that host uses NVIDIA's vendor-specific VFIO framework: each VF is assigned a profile directly through its nvidia/ sysfs attributes and the VF itself is passed to the VM (no mdev UUIDs). Existing GPU hosts continue to use mdev, so both frameworks must coexist while hosts migrate one at a time: hypeman detects the host's framework at runtime and leaves the mdev path intact.

Note that /sys/class/mdev_bus/ itself may still be present on a vendor VFIO host, because nvidia_vgpu_vfio pulls in the mdev module. It just never gains parent entries, so detection keys off discovered mdev VFs rather than the directory existing.

summary

• detect NVIDIA's mdev and vendor-specific VFIO frameworks at runtime
• discover vendor VFIO virtual functions and parse creatable_vgpu_types
• assign and release profiles through current_vgpu_type with least-loaded-GPU placement
• protect newly created assignments with per-instance ownership until QEMU opens the VFIO device, then preserve active assignments and clear orphans by checking both iommufd cdevs (/dev/vfio/devices/*) and the legacy /dev/vfio/<group> node
• document both frameworks and verify that vendor VFIO vGPUs retain the existing standby rejection

Stacked on #322, which contains the framework-neutral lifecycle refactor.

creatable_vgpu_types format

The driver emits a table with a header row and the type ID first:

ID    : vGPU Name
1145  : NVIDIA L40S-1B
1147  : NVIDIA L40S-1Q

Parsing splits on the : separator and skips the header row. An earlier revision expected the name first and the type ID last, which failed on every line (parse vGPU type ID "Name"), and because getVGPUStatus discards profile errors the host advertised vGPU mode with all its slots and no profiles at all — no placement could succeed.

Once a profile consumes a GPU's entire framebuffer, the remaining VFs on that GPU print the header and nothing else. That is a legitimately empty list rather than a malformed file, so it yields zero profiles instead of an error; otherwise one 48Q allocation would remove the whole GPU from the status response.

host validation

Validated on an L40S host running vGPU Manager 580.159.01 on Ubuntu 24.04 / kernel 6.8.0-136, where the driver reports Host VGPU Mode: SR-IOV and /sys/class/mdev_bus/ has no parents:

• framework detection returns vendor VFIO; 64 VFs discovered across two L40S
• 30 profiles enumerated with correct framebuffer sizes
• availability drops from 64 to 32 slots after assigning NVIDIA L40S-48Q on one GPU, and returns to 64 after release. Assigning a smaller profile instead clamps that GPU's remaining VFs to the same framebuffer size class (2B/2Q/2A after a 2Q), matching how the mdev driver reports available_instances

Availability counts free VFs that can currently accept a profile, not how many of that profile fit — the driver enforces the framebuffer limit at assignment time. That is the pre-existing mdev behavior and this PR keeps it; tightening the count is separate work.

• assigning current_vgpu_type and booting QEMU with -device vfio-pci,sysfsdev=<VF> produces a live NVIDIA L40S-1Q in nvidia-smi vgpu, released by writing 0
• that QEMU process held the legacy /dev/vfio/<group> node, not an iommufd cdev, even with /dev/iommu present — so the legacy group fd check is the one that keeps assignments alive on this host
• ran this branch as the systemd service and booted both existing and Ubuntu 24.04 kernel-browser GPU images with NVIDIA L40S-2Q; guest GRID loaded, CDP and kernel-images-api started, hardware OpenGL reported the L40S through ANGLE, and NVENC completed an encode
• stop released the VF, start reacquired it, and standby returned the expected 409
• restarting hypeman with a live VM preserved its assignment; killing QEMU and restarting hypeman cleared the orphaned current_vgpu_type
• after the host provisioning role installed persistent SR-IOV ordering, a reboot restored 64 VFs and hypeman rediscovered all 30 profiles

tests

• go test -race ./lib/devices ./lib/instances -run 'VendorVFIO|CreatableVGPU|StoredVGPU|StandbyRejectsVendorVFIO' -count=1
• go test ./lib/devices ./lib/hypervisor/qemu ./lib/resources -count=1
• go test ./cmd/api ./cmd/api/api ./lib/instances ./integration -run '^$' -count=1
• go vet ./cmd/api ./lib/devices ./lib/hypervisor/qemu ./lib/instances ./lib/resources

review follow-ups

• discover the vGPU framework and its VFs in one sysfs scan (DiscoverVGPU); ListGPUProfilesWithVFs takes the framework instead of re-detecting, removing the repeated full VF scans per GPU status query
• GPU resource status now logs discovery and profile-listing failures instead of silently reporting a vGPU host with slots but no profiles (the failure mode the parser fix in this PR originally surfaced)
• reconcile shares one /proc scan across all VF in-use checks instead of walking /proc per allocated VF
• document why DestroyVGPU infers the framework for metadata written before GPUFramework existed
• merged the base branch so this PR carries its current head
• fail closed when VFIO usage probes are inconclusive, so reconciliation preserves active assignments
• propagate framework discovery errors through create and reconciliation instead of treating them as no vGPU support
• surface rollback failures after vendor VFIO assignment verification errors
• report capacity instead of “profile not found” when the creatable catalog is empty at full GPU capacity, with an all-GPUs-full test
• reconcile passes its shared /proc scan into each orphan release instead of rescanning per VF
• reconciliation skips allocated VFs whose device path belongs to an instance with a live hypervisor process, so a hypeman restart cannot clear an assignment from under a VM that has not opened the VFIO device yet (the interval the in-process owner map cannot cover across restarts); orphans with no live instance are still cleared. If the instance inventory cannot be listed, reconciliation is skipped entirely rather than proceeding unprotected

scope decisions

The implementation stays scoped to the validated R19 / Ubuntu 24.04 target:

• dropped the non-QEMU hypervisor rejection — the caller selects the hypervisor for vGPU instances, and hypeman keeps that selection with callers (mdev vGPU on Cloud Hypervisor was never gated either)
• dropped the name-first creatable_vgpu_types format; the parser accepts only the ID : Name table the target driver emits
• keep an in-process assignment ownership map for the interval between writing current_vgpu_type and QEMU opening the VFIO device; open-handle probing cannot cover that interval. After a restart, fail-closed VFIO handle probing protects recovered assignments
• dropped dead framework inference for vendor VFIO device paths in DestroyVGPU
• at full GPU capacity /resources reports an empty profile list (slot counts still show exhaustion); a last-known-catalog cache was considered and rejected — it would be stale across restarts and no caller makes placement decisions from that list — metadata with a device path always carries GPUFramework; only mdev-UUID-only legacy metadata needs inference
• release identifiers travel in a VGPUAssignment struct instead of positional string parameters
• listProfiles reads each VF's creatable_vgpu_types once per listing instead of twice (catalog + availability in one pass)

---

Note

High Risk
Changes GPU assignment, sysfs writes, and startup reconciliation on security-sensitive multi-tenant infrastructure; incorrect cleanup could disturb live VMs, though protection and fail-closed VFIO probes mitigate that.

Overview
Adds NVIDIA vendor VFIO as a second SR-IOV vGPU path next to mdev: hosts are classified at runtime via DiscoverVGPU() (mdev VFs first, else VFs with nvidia/creatable_vgpu_types). Vendor VFIO assigns profiles by writing current_vgpu_type, picks VFs with least-loaded-GPU placement, and releases by writing 0, with per-instance ownership until QEMU opens VFIO and reconciliation that scans open iommufd and legacy /dev/vfio/<group> handles.

API startup now runs ReconcileVGPUs with device paths protected for instances whose hypervisor PID still owns the instance socket (ListInstancesForReconcile + HypervisorProcessExists). If the instance list fails, vendor VFIO reconcile is skipped rather than running unprotected.

Instance lifecycle passes InstanceID into vGPU destroy, clears stale assignments when another live instance claims the same path, retains metadata when create-time vGPU teardown fails, and extends integration tests for sysfs assignment, stop/release, and start/reacquire on both frameworks. GPU resource reporting uses discovery instead of removed DetectHostGPUMode, with warnings on discovery/profile errors.

^{Reviewed by Cursor Bugbot for commit d92201d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[yummybomb]

Trimmed over-engineering after an architectural review against the parent task (host kernel 6.8 / Ubuntu 24.04 support). Most of the removed weight came from earlier review-round additions rather than the original design:

[cursor]

VFIO probe skips sibling check

High Severity

The vfioDeviceInUse function prematurely returns false if either the vfio-dev or iommu_group sysfs path is missing. This short-circuits the check, incorrectly marking a VF as unused even when a running QEMU VM holds an active device handle. This can lead to current_vgpu_type being cleared for an active VM during destroy or reconcile.

 

^{Reviewed by Cursor Bugbot for commit ff9462f. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 19b9602. Configure here.}