Skip to content

feat: discover OSS MLflow capabilities and model metadata - #334

Open
dgokeeffe wants to merge 8 commits into
databricks:mainfrom
dgokeeffe:review/oss-capability-discovery
Open

feat: discover OSS MLflow capabilities and model metadata#334
dgokeeffe wants to merge 8 commits into
databricks:mainfrom
dgokeeffe:review/oss-capability-discovery

Conversation

@dgokeeffe

Copy link
Copy Markdown

Issue and stack

Closes #332.

Depends on #217, #333, #223, #239, and #243. This fork-based PR must target main; review only the capability-discovery commit relative to the pushed integrated consolidation base:

dgokeeffe/ucode@review/integrated-consolidation-base...review/oss-capability-discovery

Equivalent immutable range: 168038b...347f81d. Do not merge before all prerequisites. The incremental range contains no proxy or cache-diagnostic changes.


Unit 05 — OSS/MLflow capability discovery and model metadata

Objective and user-visible behavior

Dynamically discover MLflow chat-completions-only models, persist reasoning/context/output metadata, project capabilities onto UC IDs, and render bounded Pi/OpenCode model entries. Unknown or partial metadata degrades conservatively; static GLM/Kimi behavior remains intact.

Candidate: review/oss-capability-discovery (347f81d), incremental base integrated dev (168038b). This is the capability follow-up for existing PR #217; it does not include the repair proxy.

Exact scope

Production: src/ucode/databricks.py, src/ucode/cli.py, src/ucode/agents/pi.py, src/ucode/agents/opencode.py.
Tests/fixtures: tests/conftest.py, tests/test_databricks.py, tests/test_cli.py, tests/test_agent_pi.py, tests/test_agent_opencode.py.
Non-goals: SSE repair/lifecycle, cache affinity/retention, Pi home isolation, GPT off-thinking.

Before / after reproduction

git show dev:src/ucode/databricks.py | rg 'discover_oss_model_specs' || echo '<absent>'
# <absent>

git show review/oss-capability-discovery:src/ucode/databricks.py \
  | rg 'def discover_oss_model_specs|_OSS_MAX_OUTPUT_TOKENS'
# structured discovery function and bounded output table are present

Focused validation:

uv run --frozen pytest tests/test_agent_pi.py tests/test_agent_opencode.py tests/test_cli.py tests/test_databricks.py -q
# 585 passed
uv run --frozen ruff check .
# All checks passed

Full suite:

uv run --frozen pytest -q
# 2 failed, 1763 passed, 36 skipped

Failures: installed Claude capture and the pre-existing Pi preferred-model state assertion. Both reproduce on integrated dev with exact focused commands; this unit does not touch Claude capture or state serialization/default selection.

Live Databricks evidence (sanitized)

A live configured workspace call to discover_oss_model_specs completed without printing workspace, token, or model IDs:

{"live_evidence":"completed","spec_count":14,"all_ids_nonempty":true,"all_reasoning_boolean":true,"context_metadata_count":8,"output_limit_count":13,"reason_redacted":null}

Deterministic tests cover malformed payloads, mismatched entity capabilities, native-API exclusion, duplicate IDs, context-description parsing, UC ID projection, metadata outage fallback, conservative limit completion, and state-to-agent propagation.

Impact map

  • Agents: Pi and OpenCode.
  • Provider: MLflow chat completions only; native Claude/Gemini/GPT Responses endpoints are excluded.
  • State/config: adds oss_model_specs; Pi/OpenCode consume it for reasoning and limits.
  • Compatibility: broad dynamic models require live validation; static GLM/Kimi survive metadata outages; unknown limits get conservative complete pairs.
  • Managed config/MCP/skills: no registration changes; existing state without specs retains static fallback.

Rollback and residual risk

Revert 347f81d; the original validated GLM/Kimi consolidation remains on dev. Residual risks: free-text context metadata can be absent; output ceilings are probed policy data and require maintenance as models evolve. The fallback direction caps conservatively rather than sending uncapped requests.

Hygiene

git diff --check dev..347f81d passes. Nine-file range; no proxy file, diagnostic scripts, generated files, credentials, uv.lock, .pi-subagents/, or goal.md; no merge markers or unresolved index entries.

…directing HOME

Pi honors the PI_CODING_AGENT_DIR env var to resolve its config
directory (~/.pi/agent), so redirecting /Users/david.okeeffe to APP_DIR/pi-home was
unnecessary. The HOME redirect broke macOS keychain default resolution
under ucode: the Security framework looks for the login keychain under
the redirected HOME, finds none, and security default-keychain returns
'A default keychain could not be found'. As a result gh auth, the git
credential helper, and any keychain-backed tool failed inside pi.

Setting PI_CODING_AGENT_DIR to the existing PI_CONFIG_DIR preserves
config isolation (models.json/settings.json/sessions still land under
APP_DIR/pi-home/.pi/agent) while leaving /Users/david.okeeffe as the user's real home,
so the login keychain stays discoverable.

Tests: the two pi e2e sites monkeypatched PI_UCODE_HOME/PI_CONFIG_PATH
to redirect pi at a tmp home. They now also patch PI_CONFIG_DIR (read by
build_runtime_env) and PI_SETTINGS_PATH/PI_SETTINGS_BACKUP_PATH (previously
masked because the HOME redirect made pi read settings from the un-patched
real APP_DIR path).
Expose the GLM and Kimi coding-model cohort through Pi and OpenCode with shared token limits and reasoning metadata. Keep unsupported chat models out of discovery, including Inkling until gateway issue databricks#215 is fixed, and retain the GPT-OSS Responses API routing guard.
Centralize Claude family/version parsing so Pi metadata, adaptive-thinking compatibility, and Claude Code's [1m] selector cannot drift. Cover Sonnet 4.5, Opus 4.6, future major versions, Fable fallback, and prefixed model IDs.
Configure the Databricks OpenAI Responses provider alongside the validated GLM/Kimi provider, and fall back to foundation-model serving endpoints when UC model services are unavailable.
…/consolidated-pi-opencode-upstream

# Conflicts:
#	tests/test_e2e.py
`_pi_gpt_model_entry` declared `reasoning: True` without an off-state, so for
the thinking-off case Pi's Responses builder fell back to
`reasoning: {effort: "none"}` (pi-ai openai-responses.js, the
`thinkingLevelMap?.off !== null` branch). `"none"` is only valid on gpt-5.1+,
so every request to gpt-5, gpt-5-mini, gpt-5-nano and gpt-5-5-pro was rejected:

  BAD_REQUEST: Unsupported value: 'none' is not supported with the 'gpt-5'
  model. Supported values are: 'minimal', 'low', 'medium', and 'high'.

Setting `thinkingLevelMap: {"off": None}` makes Pi omit `reasoning` entirely,
which the gateway accepts for all 14 codex ids. Verified against
/ai-gateway/codex/v1/responses: effort="none" 400s on gpt-5/-mini/-nano/-5-5-pro
and 200s on gpt-5-1..-5-6; omitting `reasoning` is 200 everywhere.
`{"off": "minimal"}` was rejected as an alternative because gpt-5-5-pro 400s on
it too. Same pattern already used for the Gemini 3.x entries.

The rest of Pi's Responses payload was bisected against the gateway and is
fine: store:false, prompt_cache_key, prompt_cache_retention:"24h",
prompt_cache_options, include:["reasoning.encrypted_content"], developer role,
flat tool schemas, and the session_id / x-client-request-id affinity headers.

Regression was hard to spot because the gateway returns
{"error_code","message"} rather than OpenAI's {"error":...}, so Pi's
error-body.js recovery no-ops and every 400 renders as
"OpenAI API error (400): 400 status code (no body)". Reported upstream as
earendil-works/pi#7748.

Refs databricks#286
…i-opencode-upstream

# Conflicts:
#	src/ucode/agents/pi.py
#	src/ucode/databricks.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Discover OSS/MLflow model capabilities and persist bounded metadata

1 participant