Skip to content

filer: switch workspace upload from import-file to /workspace/import - #6149

Draft
Sankalp-Mittal wants to merge 6 commits into
mainfrom
sankalp-mittal/workspace-import-migration
Draft

filer: switch workspace upload from import-file to /workspace/import#6149
Sankalp-Mittal wants to merge 6 commits into
mainfrom
sankalp-mittal/workspace-import-migration

Conversation

@Sankalp-Mittal

Copy link
Copy Markdown
Collaborator

Changes

Replace POST /api/2.0/workspace-files/import-file/{path} with the multipart
variant of POST /api/2.0/workspace/import (via the SDK's Workspace.Upload
with format=AUTO) as the transport for all bundle file uploads. This is a
single call site: WorkspaceFilesClient.Write in libs/filer.

The error handling in Write is rewritten to branch on SDK sentinels
(errors.Is against ErrNotFound, ErrResourceAlreadyExists,
ErrAlreadyExists, ErrInvalidParameterValue, ErrPermissionDenied) instead
of raw HTTP status codes and message-substring matching. The overwrite/type-
mismatch collision is detected via the AIP-193 ErrorInfo reason
WORKSPACE_OBJECT_TYPE_MISMATCH, with a message-substring fallback for
workspaces where that detail (WP-6031) has not yet rolled out.

Why

  1. /workspace-files/import-file is deprecated.
  2. /workspace/import has a higher rate limit (30 vs 20 requests/sec per
    workspace) served by a dedicated RLv2 group, and is ~1.5–2× faster for a
    typical bundle deploy.
  3. Structured error details let us stop parsing error message strings to detect
    path collisions.

Why multipart

/workspace/import's JSON body caps at 10 MiB for AUTO format
(databricks.webapp.autoExportFormatLimitBytes). The multipart variant accepts
the same payload sizes import-file did, so switching does not regress the
maximum uploadable file size. The 10 MiB server-side notebook cap
(databricks.notebook.maxNotebookSizeBytes) applies to both endpoints, so
maximum notebook size is unchanged.

Testserver

  • libs/testserver now decodes the multipart /workspace/import body and
    mirrors the real endpoint's format=AUTO notebook detection for
    .py/.sql/.scala/.r (header comment → NOTEBOOK, otherwise FILE), and
    returns the collision error shape the CLI branches on.
  • acceptance/internal/prepare_server.go records multipart bodies in a
    normalized, deterministic form (multipart_form with sorted fields; large or
    binary parts summarized) so recorded requests stay reviewable.
  • The fault-injection mechanism gains an optional bodyContains match
    (fault.py --body-contains SUBSTR). Because every upload now shares the
    /workspace/import method+path, a single file's upload can only be targeted
    by its multipart path field; the two migrate/destroy fault tests rely on
    this.

Tests

  • New acceptance test bundle/sync-upload-edge-cases covers the cases that
    differ between the two endpoints: a 12 MiB binary (over the JSON cap), an empty
    file, notebooks in three languages, a header-less .py (stored as FILE), a
    .lvdash.json dashboard, and non-ASCII / spaced filenames. It asserts each
    upload's object type/language and that every upload sets format=AUTO.
  • Unit tests in libs/filer/workspace_files_client_test.go cover the success
    path (format/overwrite) and every error mapping (already-exists shapes,
    type-mismatch via ErrorInfo reason, 403, 500).
  • libs/testserver/fault_test.go covers the new bodyContains match.
  • Existing acceptance fixtures regenerated for the new request shape; the jq
    assertions in affected scripts re-aimed from .path to
    .body.multipart_form.path.

Notes

  • The type-mismatch fallback (message-substring) is load-bearing until WP-6031
    is confirmed rolled out everywhere, and can be removed afterward.
  • This PR does not change the sync concurrency limit.

This pull request and its description were written by Isaac.

Sankalp-Mittal and others added 3 commits August 3, 2026 14:02
Replace POST /api/2.0/workspace-files/import-file/{path} with the multipart
variant of POST /api/2.0/workspace/import (via the SDK's Workspace.Upload +
format=AUTO). The previous endpoint is deprecated; the new one has a higher
rate limit (30 vs 20 rps/workspace) and is ~1.5-2x faster for typical bundle
deployments.

Error handling is rewritten to branch on SDK sentinels (errors.Is against
ErrNotFound / ErrResourceAlreadyExists / ErrAlreadyExists /
ErrInvalidParameterValue / ErrPermissionDenied) and the AIP-193 ErrorInfo
reason WORKSPACE_OBJECT_TYPE_MISMATCH, with a message-substring fallback for
workspaces where WP-6031 has not rolled out.

The testserver now decodes the multipart /workspace/import body and mirrors the
real endpoint's format=AUTO notebook detection. A new bodyContains option on the
fault mechanism lets a test target a single file's upload, which is no longer
possible by URL since every upload shares the /workspace/import path.

Co-authored-by: Isaac
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 97a20ed

Run: 30904967204

Env 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 4 4 323 1065 5:26
💚​ aws windows 4 4 325 1063 11:59
💚​ azure linux 4 4 322 1065 5:58
💚​ azure windows 4 4 324 1063 11:12
💚​ gcp linux 1 5 323 1065 5:33
💚​ gcp windows 1 5 325 1063 9:17
8 interesting tests: 4 RECOVERED, 4 SKIP
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo/root 💚​R 💚​R 💚​R 💚​R
💚​ TestFetchRepositoryInfoAPI_FromRepo/subdir 💚​R 💚​R 💚​R 💚​R
Top 6 slowest tests (at least 2 minutes):
duration env testname
8:54 aws windows TestAccept
8:17 azure windows TestAccept
6:24 gcp windows TestAccept
3:07 azure linux TestAccept
3:03 gcp linux TestAccept
2:59 aws linux TestAccept

@Sankalp-Mittal
Sankalp-Mittal marked this pull request as ready for review August 4, 2026 08:22
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Approval status: pending

/acceptance/bundle/ - needs approval

77 files changed
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @shreyas-goenka, @andrewnester, @anton-107, @lennartkats-db

/libs/filer/ - needs approval

Files: libs/filer/workspace_files_client.go, libs/filer/workspace_files_client_test.go
Suggested: @simonfaltum
Also eligible: @renaudhartert-db, @Divyansh-db, @hectorcast-db, @parthban-db, @tanmay-db, @tejaskochar-db, @mihaimitrea-db, @chrisst, @rauchy

General files (require maintainer)

9 files changed
Based on git history:

  • @denik -- recent work in libs/testserver/, .nextchanges/bundles/, libs/filer/

Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db) can approve all areas.
See OWNERS for ownership rules.

@Sankalp-Mittal
Sankalp-Mittal marked this pull request as draft August 4, 2026 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants