Skip to content

Harden hypervisor process liveness checks - #363

Open
yummybomb wants to merge 7 commits into
hypeship/generalize-vgpu-devicefrom
hypeship/hypervisor-liveness
Open

Harden hypervisor process liveness checks#363
yummybomb wants to merge 7 commits into
hypeship/generalize-vgpu-devicefrom
hypeship/hypervisor-liveness

Conversation

@yummybomb

@yummybomb yummybomb commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Layer 1 of the vendor VFIO vGPU stack (generalize-vgpu-devicethisvendor-vfio-backendvendor-vfio-vgpu). Pure hypervisor-process hardening with no vGPU-specific code; reviewable in isolation.

The upper layers guard vGPU release decisions on "is this instance's hypervisor still alive", so the liveness answer has to be trustworthy first:

  • Unify liveness checks on ProcessExists — one exported, EPERM-aware, zombie-filtering definition instead of scattered bare kill(pid, 0) probes. EPERM means the process exists but cannot be signaled; treating it as dead would be wrong.
  • Wait for non-child hypervisor exit before finishing kill — after a hypeman restart the hypervisor is not our child, so Wait4 returns ECHILD immediately and the kill loop finished before the process had exited. Poll for actual exit in that case.
  • HypervisorProcessExists: verify socket ownership — a bare PID probe treats any process that reused a stored hypervisor PID as the owning VMM. On Linux, require the PID to own the instance's hypervisor socket before reporting it alive.

Testing

  • go build ./..., go vet clean
  • go test -race ./lib/instances/ targeted suites pass (TestCreateInstanceWithNetwork requires image pulls + iptables and fails in this environment on the unmodified base as well)

Note

Medium Risk
Changes process kill and liveness semantics on delete and PID refresh; incorrect behavior could spare a VMM or kill the wrong process, though the new tests target the stale-PID case explicitly.

Overview
Hardens how hypeman decides whether a hypervisor is alive and which PID to kill on instance delete, so stale or reused PIDs cannot target the wrong process.

Linux socket PID resolutionResolveProcessPID now returns (pid, confirmed, err) where confirmed is true only when the PID comes from FD ownership, not cmdline fallback. Listener lookup in /proc/net/unix filters on __SO_ACCEPTCON so accepted connections do not duplicate the path, and duplicate listener inodes error instead of picking arbitrarily. /proc scans surface partial read/link failures instead of silently treating them as “not found.”

Unified liveness — Internal processExists is exported as ProcessExists (EPERM-aware, zombie-filtered on Linux). New HypervisorProcessExists(pid, socketPath) requires socket ownership on Linux before treating a stored PID as the VMM.

Delete pathkillHypervisor only SIGKILLs a PID verified via HypervisorProcessExists, or a confirmed socket owner when metadata PID is wrong; unrelated processes that reused a stale PID are left alone. The post-SIGKILL wait loop no longer exits early on Wait4 ECHILD for non-child hypervisors; it polls until the process is actually gone.

Reviewed by Cursor Bugbot for commit 78fc483. Bugbot is set up for automated code reviews on this repo. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit a187984. Configure here.

Comment thread lib/hypervisor/socket_pid_linux.go
@yummybomb
yummybomb force-pushed the hypeship/hypervisor-liveness branch from a187984 to 76b9f42 Compare August 7, 2026 15:04
kill(pid, 0) returning EPERM means the process exists but cannot be
signaled, and a zombie PID passes a bare kill(0) probe. Export the
EPERM-aware, zombie-filtering processExists helper so every hypervisor
liveness check shares one definition.
After a hypeman restart the hypervisor is not our child, so Wait4
returns ECHILD immediately and the kill loop finished before the
process had exited. Poll for actual process exit in that case.
A bare liveness probe treats any process that reused a stored
hypervisor PID as the owning VMM. Require the PID to own the
instance's hypervisor socket on Linux before reporting it alive.
Accepted server-side sockets appear in /proc/net/unix with the same
bound path as the listener, so any connected API client made
socketRefForPath report multiple inodes and pid-reuse protection fell
back to unconfirmed while the control socket was in use. Only entries
with __SO_ACCEPTCON identify the owning process; duplicate listeners
from unlink-and-rebind still resolve as unconfirmed.
@yummybomb
yummybomb force-pushed the hypeship/hypervisor-liveness branch from 76b9f42 to 78fc483 Compare August 7, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant