docs(rfc): RFC 0012 - observability architecture - #2620
Open
rhuss wants to merge 22 commits into
Open
Conversation
Proposes a unified observability architecture for OpenShell covering tracing, logging, and monitoring. The centerpiece is a supervisor-local OTLP relay that collects agent-emitted traces from inside network-isolated sandboxes and forwards them through the gateway to an external collector. Relates-to: NVIDIA#1055, NVIDIA#2507, NVIDIA#2508, NVIDIA#909 Signed-off-by: Roland Huß <rhuss@redhat.com>
Replace ASCII art with Mermaid diagrams for GitHub rendering. Expand terse fragments into flowing paragraphs throughout the Current State, Deployment Configuration, and Open Questions sections. Signed-off-by: Roland Huß <rhuss@redhat.com>
Replace bold-keyword fragments in all Implementation Plan phases with flowing prose and natural transitions. Remove signposting, persuasive authority tropes, and promotional language throughout. Trim redundant Dapr description between Proposal and Prior Art sections. Add cross-references from Implementation Plan back to Proposal to reduce repetition. Signed-off-by: Roland Huß <rhuss@redhat.com>
Introduce the two observability layers (infrastructure-level and agent-level) before presenting the OTLP relay. The summary now establishes what exists on the infrastructure side, then positions the supervisor relay as the solution for the harder agent-level problem where sandbox network isolation prevents direct export. Signed-off-by: Roland Huß <rhuss@redhat.com>
Fix critical localhost vs veth-host-IP contradiction in relay section, diagram, and Phase 2. Standardize OTel/OTLP capitalization throughout. Expand all abbreviations on first use (OCSF, OTLP, OTel, SLI/SLO, RED, OPA, SSRF). Add Proposal and Implementation Plan roadmap paragraphs. Move Visibility domains earlier to establish two-domain framing before the relay design. Add new Multi-tenant observability section covering per-workspace OTLP endpoints, trace isolation, and config model. Add openshell.workspace.id to the enrichment attributes table. Add TOML and Helm YAML configuration examples. Fix Mermaid relay diagram to show gRPC+HTTP ports instead of misleading localhost. Add span link annotation to correlation diagram. Link RFC 0003 and reference NVIDIA#1055 in body. Replace temporal "recently merged" language. Reduce Proposal vs Implementation Plan redundancy. Tighten Do-nothing alternative and cardinality risk section. Signed-off-by: Roland Huß <rhuss@redhat.com>
Reframe the supervisor relay as a general telemetry channel carrying traces, logs, and metrics out of isolated sandboxes. Add log relay section (OCSF events + agent stdout/stderr via session protocol, solving NVIDIA#1922) and metrics relay section (push-based sandbox metrics, lower priority). Update Mermaid diagram to show all three pillars. Add Summary paragraph explaining why tracing gets architectural depth while logging and metrics extend through the same channel. Add Motivation bullet for centralized log collection gap. Signed-off-by: Roland Huß <rhuss@redhat.com>
The relay decouples agent development from observability operations: agent developers export to a fixed auto-injected endpoint, workspace admins configure where telemetry routes, global admins set defaults and policies. Reference this from the Alternatives section as a fifth reason to prefer the relay over direct collector access. Signed-off-by: Roland Huß <rhuss@redhat.com>
Fix OCSF scope claim: OCSF runs on both sandbox and gateway sides (TLS events, service routing, policy), not sandbox-only. Correct VM driver instrumentation count from "10+ using ErrorStatusGuard" to "15 instrumented operations (9 via instrument, 6 via .instrument()) and ErrorStatusGuard for error marking". Note that K8s and Podman drivers are dual-mode (in-process default + standalone gRPC binary). Clarify NVIDIA#2615 is a PR, not an issue. Signed-off-by: Roland Huß <rhuss@redhat.com>
The CLI runs on laptops, CI runners, and jump hosts where an OTel collector is typically unreachable. The gateway already traces every incoming request. Replace the full CLI OTel provider proposal with passive TRACEPARENT propagation: if the environment has TRACEPARENT set (e.g. CI pipelines), forward it to the gateway. No SdkTracerProvider, no collector configuration, no --otlp-endpoint flag needed. Signed-off-by: Roland Huß <rhuss@redhat.com>
Infrastructure and agent traces serve different user groups with different backend needs (Tempo vs MLflow). Initially both go to the same endpoint with collector-side routing. As a future extension, per-domain endpoints (infra_endpoint, agent_endpoint) would let the gateway route directly. This composes with per-workspace endpoints for multi-tenant deployments. Signed-off-by: Roland Huß <rhuss@redhat.com>
Expand the capacity calculation to cover traces, OCSF logs, agent stdout/stderr, and metrics. OCSF logs may be the dominant stream (10-300 events/sec per sandbox at 1-5 KB each). Add per-pillar volume table. Note that log sampling has different constraints than trace sampling (OCSF events serve audit/compliance, so rate limiting is preferred over probabilistic sampling). Add stdout/stderr byte rate limit as a mitigation. Signed-off-by: Roland Huß <rhuss@redhat.com>
The supervisor already pushes tracing events to the gateway via LogPushLayer and the PushSandboxLogs client-streaming RPC. The log relay section now builds on this existing mechanism rather than proposing to build from scratch. The two remaining gaps are OCSF JSONL events (which use a separate layer and stay local) and agent stdout/stderr (captured but not pushed). Signed-off-by: Roland Huß <rhuss@redhat.com>
Agent sandboxes are interactive sessions with terminal UIs, ANSI escape codes, and multiplexed SSH channels. That output stream is user-facing interaction, not structured log data. Internal agent logging is the agent developer's concern, not the platform's. The log relay scope is now: existing PushSandboxLogs tracing events (already working) plus OCSF event forwarding (the actual gap). Updated capacity table, diagram, and mitigations accordingly. Signed-off-by: Roland Huß <rhuss@redhat.com>
The supervisor has no metrics today (no metrics crate, no Prometheus exporter). Only the gateway exposes /metrics. Reframe the metrics relay as a future extension for when sandbox-level metrics are added, not as solving a current scraping limitation. Signed-off-by: Roland Huß <rhuss@redhat.com>
Replace bare "Phase 2" reference with descriptive text and an anchor link to the Implementation plan section, since it appears before the Implementation plan in the document. Signed-off-by: Roland Huß <rhuss@redhat.com>
Signed-off-by: Roland Huß <rhuss@redhat.com>
Span links are an OTel concept that may not be familiar to all readers. Add a brief explanation at the first use in the correlation section and a clarifying parenthetical at the first casual mention in the summary. Signed-off-by: Roland Huß <rhuss@redhat.com>
ServiceMonitor is a Prometheus Operator CRD, not a Helm concept. The Helm chart includes a template for it. Signed-off-by: Roland Huß <rhuss@redhat.com>
Describe the alternative as a design option considered and rejected, not as a prior version of this RFC that readers would not have seen. Signed-off-by: Roland Huß <rhuss@redhat.com>
Signed-off-by: Roland Huß <rhuss@redhat.com>
MLflow was used throughout as if it were the only agent trace backend. Reframe it as one example alongside Langfuse and use generic "agent trace backend" where the specific tool does not matter. This makes the RFC backend-agnostic. Signed-off-by: Roland Huß <rhuss@redhat.com>
Explain what span links are and what "surface links in the UI" means concretely (one-click navigation vs manual attribute filtering). Link to OTel docs and the RFC's own span links section. Signed-off-by: Roland Huß <rhuss@redhat.com>
rhuss
marked this pull request as ready for review
August 5, 2026 18:40
rhuss
requested review from
a team,
derekwaynecarr,
maxamillion and
mrunalp
as code owners
August 5, 2026 18:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Proposes a unified observability architecture for OpenShell covering tracing, logging, and monitoring. The architecture splits observability into two layers (infrastructure-level and agent-level) and introduces a supervisor-local telemetry relay that mediates all observability data between network-isolated sandboxes and the platform.
For traces, the supervisor listens on standard OTLP ports inside the sandbox, collects agent-emitted spans, enriches them with sandbox context, and forwards them through the gateway to an external collector. The same relay channel carries OCSF log batches for centralized collection and can push sandbox-level metrics to the gateway. The agent exports to a fixed, auto-injected endpoint and never needs to know about collector topology, keeping observability routing as an operational concern rather than a development one.
The RFC covers:
Related issues
Note on RFC number
RFC number 0012 is provisional. Per the RFC process, maintainers assign numbers from an issue. This PR uses the next available number; please reassign if needed.
Test plan