Skip to content

[FIX] Give each upload its own temp file so concurrent tests stop clobbering each other's results - #1166

Merged
canihavesomecoffee merged 1 commit into
masterfrom
fix/upload-temp-file-race
Aug 9, 2026
Merged

[FIX] Give each upload its own temp file so concurrent tests stop clobbering each other's results#1166
canihavesomecoffee merged 1 commit into
masterfrom
fix/upload-temp-file-race

Conversation

@cfsmp3

@cfsmp3 cfsmp3 commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Problem

upload_type_request() saved every upload to TempFiles/<uploaded name>:

temp_path = os.path.join(temp_dir, filename)   # filename comes from the upload
uploaded_file.save(temp_path)
# ... hash the file ...
os.rename(temp_path, final_path)

That name is derived from the regression test output, so it is identical for every test running the same regression test. Two tests running at once share one path:

  1. A saves TempFiles/X.srt
  2. B overwrites TempFiles/X.srt
  3. A hashes it and renames it to TestResults/<hashA>.srt
  4. B's rename → FileNotFoundErrorHTTP 500

The 500 means no TestResultFile row is written. mod_test/controllers.py then sets got = 'error' for any regression test with no result files but expected output, which renders as "No output generated but there should be". The contributor sees their PR breaking tests it never touched.

There is a quieter variant: if B's save lands between A's save and A's hashing, A records the hash of B's output and its verdict silently flips. No error is raised in that case at all.

Evidence

From ccextractor PR #2309. Tests 9402 (master) and 9410 (the PR) ran concurrently on Linux, 15:32–16:02:

run HTTP 500s in VM log spurious "No output" genuine baseline
9402 master Linux 37 36 24
9410 PR Linux 21 21 24
9403 master Windows 0 0 24
9411 PR Windows 0 0 24

The two Windows runs of the same commits were staggered (15:35 vs 15:44), hit zero 500s, and produced identical verdicts. The two Linux runs disagreed on 57 tests, in two completely disjoint sets — the signature of a race, not a code difference.

Server-side traceback:

File "mod_ci/controllers.py", line 2680, in upload_type_request
    os.rename(temp_path, final_path)
FileNotFoundError: [Errno 2] No such file or directory:
  '/repository/TempFiles/3c77a844….srt' -> '/repository/TestResults/ca8d0dd4….srt'

error.log contains 2184 of these failed renames, so this has been corrupting results for a long time. It only became obvious once a backlog started launching two tests at the same moment.

Fix

Allocate a unique temp file per upload with tempfile.mkstemp(), and finish with os.replace() (atomic). A finally block removes the temp file if anything fails, so failures no longer leak into TempFiles.

Testing

  • test_upload_type_request_uses_unique_temp_path — two uploads of the same filename must get different temp paths, both results land, TempFiles ends empty. Fails on master with '…/TempFiles/shared_name.srt' == '…/TempFiles/shared_name.srt'.
  • test_upload_type_request_cleans_temp_file_on_failure — a failing save leaves nothing behind.
  • Updated test_upload_type_request for the new call shape (mkstemp + os.replace).

tests.test_ci.test_controllers: 202 tests, OK. isort, pydocstyle clean; mypy's only complaint is the pre-existing missing PyYAML stub, which CI installs via --install-types.

Note

This does not change how many tests genuinely fail — the baseline of 24 real failures ("Path is empty", CCExtractor exit code 10) is untouched. It stops the platform inventing extra ones and blaming them on whichever PR happened to be running.

…clobbering results

upload_type_request() saved every upload to TempFiles/<uploaded name>. That name
comes from the regression test output, so it is identical for every test running
the same regression test. Two tests running at once therefore share one path:

  A saves TempFiles/X.srt
  B overwrites TempFiles/X.srt
  A hashes and renames it to TestResults/<hashA>.srt
  B renames -> FileNotFoundError -> HTTP 500

The 500 means no TestResultFile row is ever written, and the platform then shows
that regression test as "No output generated but there should be" (got == 'error'
in mod_test/controllers.py when a test has no result files but expects output).
Contributors see it as their PR breaking tests that it never touched.

There is a quieter variant: if B's save lands between A's save and A's hashing,
A records the hash of B's output and its verdict silently flips.

Fix: allocate a unique temp file per upload with tempfile.mkstemp() and finish
with os.replace(), which is atomic. A finally block removes the temp file if
anything fails, so failures no longer leak into TempFiles.

Observed on ccextractor PR #2309. Tests 9402 (master) and 9410 (the PR) ran
concurrently on Linux; their VM logs contain 37 and 21 HTTP 500s, matching the
37 and 21 spurious "No output generated" results almost exactly, against a shared
baseline of 24 genuine failures. The Windows runs of the same two commits were
staggered, hit zero 500s, and produced identical verdicts. error.log holds 2184 of
these failed renames, so this has been corrupting results for a long time; it only
became obvious once a backlog started launching two tests at the same moment.
@sonarqubecloud

sonarqubecloud Bot commented Aug 9, 2026

Copy link
Copy Markdown

@canihavesomecoffee
canihavesomecoffee merged commit 7ad0ee5 into master Aug 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants