Skip to content

Add the vendor VFIO vGPU device backend - #364

Open
yummybomb wants to merge 8 commits into
hypeship/hypervisor-livenessfrom
hypeship/vendor-vfio-backend
Open

Add the vendor VFIO vGPU device backend#364
yummybomb wants to merge 8 commits into
hypeship/hypervisor-livenessfrom
hypeship/vendor-vfio-backend

Conversation

@yummybomb

@yummybomb yummybomb commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Layer 2 of the vendor VFIO vGPU stack (generalize-vgpu-devicehypervisor-livenessthisvendor-vfio-vgpu). Self-contained in lib/devices + lib/resources; nothing in the instance lifecycle calls it yet (that's the top layer).

Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are assigned by writing a type ID to a VF's nvidia/current_vgpu_type and passed to QEMU as a plain VFIO PCI device. This adds that backend behind the framework dispatch introduced in #322:

  • Discovery & placement — profile discovery from the capacity-dependent creatable_vgpu_types catalogs, least-loaded-GPU VF selection, create/verify/rollback.
  • Release guards — unlike mdev (fresh UUID per assignment), vendor VFIO reuses the same VF path across assignments, so a stale release could clear a later owner's vGPU. Release is guarded by an in-process owner map (covers the window before QEMU opens the device) and an open-VFIO-handle scan (refuses to clear a VF a running VM holds).
  • Reconciliation — clears orphaned assignments on startup, skips VFs in the caller-supplied protected set, and fails closed (skips vendor VFIO entirely) when that set is unavailable, while mdev reconciliation still runs.
  • Integration test — branched by discovered framework, extended to cover release on stop and reacquisition on start.

Testing

  • go build ./..., go vet clean
  • go test -race ./lib/devices/ ./lib/resources/ pass

Note

High Risk
Changes sysfs vGPU assignment, release semantics, and startup reconciliation on production GPU hosts; incorrect release or reconcile logic could disrupt running VMs or clear another instance’s VF assignment.

Overview
Introduces vendor VFIO as a second vGPU path for SR-IOV hosts (e.g. Linux 6.8 / R580) where profiles are assigned by writing type IDs to each VF’s nvidia/current_vgpu_type instead of creating mdevs. mdev remains preferred when usable VFs exist under mdev_bus; otherwise discovery falls back to PCI VFs that expose creatable_vgpu_types.

lib/devices centralizes behavior behind DiscoverVGPU, framework-aware ListGPUProfilesWithVFs, CreateVGPU / DestroyVGPU, and ReconcileVGPUs. Vendor VFIO adds profile parsing, least-loaded-GPU placement, create verify/rollback, an in-process owner map plus open VFIO handle checks before release, and startup reconciliation that honors a protected device-path set. mdev VF discovery now skips VFs without real mdev_supported_types entries. DetectHostGPUMode is removed in favor of vGPU discovery plus passthrough probing in resources.

lib/resources GetGPUStatus now takes context.Context, routes vGPU status through the active framework, and falls back to passthrough if vGPU discovery fails so whole-GPU status is not hidden.

Integration TestVGPU is framework-aware (GPUFramework, GPUDevicePath), asserts sysfs assignment per framework, exercises mdev release on stop / reacquire on start vs vendor VFIO retention on stop, and cleans up vendor assignments on teardown. GPU.md documents both frameworks and updated lifecycle/troubleshooting.

Reviewed by Cursor Bugbot for commit 5b47670. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread lib/devices/vendor_vfio_linux.go Outdated
Comment thread lib/resources/gpu.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 2bb8e86 to 7fc3b49 Compare August 6, 2026 19:26
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 7fc3b49 to f661e63 Compare August 6, 2026 19:40
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from f661e63 to d1207d0 Compare August 7, 2026 14:02
Comment thread lib/resources/gpu.go
Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are
assigned by writing a type ID to a VF's nvidia/current_vgpu_type and
passed to QEMU as a plain VFIO PCI device. Add a vendor VFIO backend
behind the existing framework dispatch: profile discovery from the
capacity-dependent creatable catalogs, least-loaded VF placement,
create/verify/rollback, and release.

Because the same VF path is reused across assignments (unlike mdev
UUIDs), release is guarded: an in-process owner map covers the window
before QEMU opens the device, and an open-VFIO-handle scan refuses to
clear a VF a running VM still holds. Reconciliation clears orphaned
assignments on startup, skipping VFs protected by the caller and
failing closed when the protected set is unavailable.

Branch the vGPU integration test by discovered framework and extend it
to cover release on stop and reacquisition on start.
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from d1207d0 to 5b47670 Compare August 7, 2026 15:04

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 5b47670. Configure here.

framebuffer, ok := s.framebufferByType[vf.ProfileType]
if !ok {
return "", fmt.Errorf("framebuffer size for allocated vGPU type %s is unknown", vf.ProfileType)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unknown type blocks placement

Medium Severity

selectLeastLoadedVF returns an error when an allocated VF’s ProfileType is missing from framebufferByType. That map is filled only from creatable catalogs, so an active assignment with an empty or stale creatable file can prevent any new vGPU creation on the host.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 5b47670. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant