NaviComputerUse

Local macOS computer-use SDK. It observes and operates desktop applications without a WebRTC or server dependency.

Documentation

• 使用指南：权限、Swift API、JSON-RPC、典型操作流程与故障排查。
• 代码结构：Package targets、核心文件职责和依赖边界。
• 实现方案：技术选型、后台截图/输入方案、安全模型与限制。
• 运行逻辑：观察、点击、键盘、滚动、验证和生命周期时序。

Architecture

NaviComputerUse uses three native macOS mechanisms:

• Accessibility (AXUIElement) for bounded UI trees, semantic actions, text editing, and PID-scoped coordinate hit testing.
• ScreenCaptureKit with desktopIndependentWindow for complete per-window screenshots, including windows behind another application.
• Process/exact-window-targeted pixel and keyboard events as a last resort. Pixel click, drag, and scroll use runtime-resolved SkyLight event routing without moving the hardware cursor. Keyboard delivery prefers SLEventPostToPid and falls back to CGEvent.postToPid. macOS does not acknowledge whether a background application handled these events.

The navi-computer-use executable exposes the library over persistent JSON-RPC 2.0 on stdin/stdout. Every observation returns a random snapshotID, exact windowID, and opaque element tokens. An element action requires its token, originating snapshot, and originating sessionID. Snapshots are isolated by session and window, expire after five minutes, and are invalidated by a newer same-scope observation or any successful mutation of their process. The process identity and target AX element are revalidated before actions. The client is @MainActor isolated; synchronous mutations are ordered, while independent asynchronous captures may overlap. A state capture retries when a concurrent local mutation changes the revision between screenshot capture and AX observation.

Requirements

• macOS 14 or newer
• Accessibility permission
• Screen Recording permission for screenshots
• Event Synthesizing permission for raw keyboard fallbacks

Check without prompting:

swift run navi-computer-use permissions

Request the three permissions interactively:

swift run navi-computer-use request-permissions

For a distributed product, embed the library in a consistently signed helper app. TCC grants are bound to the responsible executable's identity; an unsigned development CLI is not a stable permission host.

Library usage

import NaviComputerUse

@MainActor
func automateNotes() async throws {
    let client = ComputerUseClient(
        accessPolicy: ComputerUseAppAccessPolicy(
            allowedBundleIdentifiers: ["com.apple.Notes"],
            allowUnidentifiedProcesses: false
        )
    )
    let state = try await client.getAppState(application: "com.apple.Notes")

    if let button = state.root.children.first(where: { $0.actions.contains("AXPress") }) {
        try client.click(
            application: "com.apple.Notes",
            elementToken: button.token,
            snapshotID: state.snapshotID
        )
    }
}

Call the client from the main actor. Token-based methods default to session "default"; pass the originating session explicitly when observing with another sessionID. A configurable ComputerUseRetentionPolicy controls observation, artifact, and idle-cleanup intervals.

The tree is bounded by default to 12 levels, 800 nodes, and 512 characters per exposed text value. AXSecureTextField values are never returned.

JSON-RPC usage

Start a persistent service:

swift run navi-computer-use serve

Example requests, one JSON object per line:

{"jsonrpc":"2.0","id":1,"method":"list_apps"}
{"jsonrpc":"2.0","id":2,"method":"get_app_state","params":{"app":"com.apple.Notes","include_screenshot":true}}
{"jsonrpc":"2.0","id":3,"method":"click","params":{"app":"com.apple.Notes","element_token":"<snapshot>.<index>","snapshot_id":"<snapshot>"}}
{"jsonrpc":"2.0","id":4,"method":"set_value","params":{"element_token":"<snapshot>.<index>","snapshot_id":"<snapshot>","value":"hello"}}

Available methods are capabilities, permissions, request_permissions, list_apps, end_session, cleanup_artifacts, pointer_state, move_pointer, get_app_state, verify_state, verify_window, capture_window, perform_action, set_value, select_text, click, drag, type_text, paste_text, press_key, and scroll. get_app_state returns a normalized structural diff from the previous same-session/window snapshot by default; pass disable_diff: true for full-state-only behavior.

Set NAVI_COMPUTER_USE_AUDIT_PATH when starting serve to append one bounded-metadata JSON audit event per request. Audit events include method, request ID, outcome, duration, and error message; request parameters and observed UI content are intentionally excluded.

The service host accepts comma-separated NAVI_COMPUTER_USE_ALLOWED_BUNDLE_IDS and NAVI_COMPUTER_USE_DENIED_BUNDLE_IDS. Set NAVI_COMPUTER_USE_ALLOW_UNIDENTIFIED=0 to reject raw processes without a bundle identifier. Pixel actions update a session-local virtual pointer exposed through pointer_state; physical_pointer_state exists for diagnostics and no-cursor-movement verification.

Action results include deliveryVerified. Semantic Accessibility actions set it to true; raw PID-targeted keyboard fallbacks set it to false because macOS confirms dispatch but not handling. Rejected JSON-RPC actions return error.data.delivery_status = "refused"; Swift callers receive a typed ComputerUseError. JSONL requests are limited to 1 MiB.

Verification

swift test
swift build
node Scripts/integration-smoke.mjs
node Scripts/chromium-smoke.mjs
node Scripts/restart-smoke.mjs

The integration smoke launches a disposable AppKit fixture behind the current foreground app and proves all of the following:

• Accessibility observation returns identified controls.
• A desktop-independent PNG screenshot is readable and non-empty.
• Element click uses AXPress.
• Coordinate click uses PID-scoped AX hit testing and AXPressAtPoint; it does not use positioned raw events when a semantic action exists.
• An AX-actionless custom view proves exact-window background pixel click, arbitrary drag, and scroll through runtime-resolved SkyLight routing without moving the real pointer.
• Text mutation uses AXValue and focused text insertion uses AXInsertText.
• Text matching and cursor placement use AXSelectedTextRange.
• Return is delivered as AXConfirm or an AXDefaultButton action where possible.
• Slider/scroll-bar drags and page scrolling prefer settable AX values.
• Superseded snapshots, ended sessions, and token/snapshot or window mismatches are rejected.
• Multi-window and cross-session observations remain correctly scoped.
• Cross-session token use is rejected unless the action declares the originating session.
• Deterministic verify_state predicates prove satisfied, unsatisfied, and unknown outcomes for values, focus, and element existence.
• Oversized JSONL input is rejected before unbounded buffering and the service remains usable.
• Secure text values are absent from both the AX model and serialized JSON.
• Background actions never raise their target application; the test still permits the user to switch foreground applications concurrently.
• A disposable Chrome profile and local canvas independently prove background pixel click, drag, and wheel delivery on Chromium without touching the user's profile.
• A helper-restart smoke proves old snapshots fail closed and fresh observations/actions recover.
• Unit tests prove concurrent captures do not block the main actor, cancellation removes temporary screenshots, and idle TTL cleanup runs without requiring another request.

Known limits

• Custom-drawn, Metal, game, and canvas controls without useful Accessibility actions may require a foreground fallback or an isolated VM.
• PID-targeted raw key events can be ignored by applications whose key window is not active.
• Pixel event results are dispatched, not verified; callers should use verify_state or a fresh screenshot. The SDK explicitly refuses the pixel route when required private symbols are absent.
• SkyLight event and focus-without-raise symbols are private SPI and may change across macOS releases. Semantic AX paths do not depend on them.
• AX mutations are main-actor ordered. Screenshot operations may overlap, but sessions isolate identity and lifecycle rather than promising unlimited throughput.