Skip to content

Load CppInterOp explicitly and resolve it relative to libcppjit - #21

Open
aaronj0 wants to merge 5 commits into
mainfrom
explicit-load-dladdr
Open

Load CppInterOp explicitly and resolve it relative to libcppjit#21
aaronj0 wants to merge 5 commits into
mainfrom
explicit-load-dladdr

Conversation

@aaronj0

@aaronj0 aaronj0 commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

The backend used to dlopen libclangCppInterOp inside a global constructor and locate it through absolute build-time paths, so a wheel built on one machine could not load on another. A constructor also runs under the loader lock, inside the dlopen of libcppjit itself: the nested dlopen re-enters the loader on partially consistent state, runs the inner library's initializers with no ordering guarantees, and holds the lock through the entire interpreter and JIT setup, deadlocking any other thread that touches the loader (as reported by @Vipul-Cariappa)

The ctor also had no error channel: on failure, the old code only prints and leave a half-initialized library. With this patch the explicit LoadCppInterOp() runs after loading completes and returns a status that turns into a proper exception on the Python side.

This PR implements the following:

  • CppInterOp loading and interpreter setup now run in an exported, idempotent LoadCppInterOp() that _cpython_cppjit.py calls before the extension module import
  • Library and headers are resolved relative to libcppjit's own load location, with the build-time install prefix as the fallback for build trees. This is necessary for the wheels building infrastructure (https://github.com/compiler-research/cppjit/tree/wheels-build-test-ci), which will be added as a separate PR.
  • Dead fork-inherited code and the fake cling environment (unused include paths, cling::runtime::gCling) are removed along the way, NFC

Validation:

  • full test suite passes locally across different cxx standard builds
  • wheel relocatability: a wheel built in an isolated environment installs and runs in a second fresh venv after the build
    environment is deleted
  • the install-prefix fallback and repeated LoadCppInterOp() calls verified against a bare build-directory libcppjit.so
  • clang-format-18 and ruff clean

@github-actions

github-actions Bot commented Aug 17, 2026

Copy link
Copy Markdown

Test Results

Configuration Result
macos-26-intel-llvm21-py3.14-cxx20 = 437 passed, 44 skipped, 85 xfailed, 32 xpassed, 100 warnings in 96.88s (0:01:36) =
macos-26-llvm21-py3.14-cxx20 ==== 438 passed, 37 skipped, 93 xfailed, 30 xpassed, 100 warnings in 45.68s ====
ubuntu-24.04-llvm21-py3.14-cxx20-vg ====== 553 passed, 25 skipped, 18 xfailed, 2 xpassed in 90.75s (0:01:30) =======
ubuntu-24.04-llvm22-py3.14-cxx20 ====== 553 passed, 25 skipped, 19 xfailed, 1 xpassed in 103.23s (0:01:43) ======
ubuntu-24.04-llvm22-py3.14-cxx20-cling ====== 543 passed, 25 skipped, 27 xfailed, 3 xpassed in 95.93s (0:01:35) =======
ubuntu-24.04-llvm22-py3.14-cxx23 ====== 574 passed, 3 skipped, 20 xfailed, 1 xpassed in 105.08s (0:01:45) =======

@aaronj0
aaronj0 force-pushed the explicit-load-dladdr branch from 6d3bf8a to dd99613 Compare August 17, 2026 23:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build-system enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant