[FIX] Give each upload its own temp file so concurrent tests stop clobbering each other's results - #1166
Merged
Merged
Conversation
…clobbering results upload_type_request() saved every upload to TempFiles/<uploaded name>. That name comes from the regression test output, so it is identical for every test running the same regression test. Two tests running at once therefore share one path: A saves TempFiles/X.srt B overwrites TempFiles/X.srt A hashes and renames it to TestResults/<hashA>.srt B renames -> FileNotFoundError -> HTTP 500 The 500 means no TestResultFile row is ever written, and the platform then shows that regression test as "No output generated but there should be" (got == 'error' in mod_test/controllers.py when a test has no result files but expects output). Contributors see it as their PR breaking tests that it never touched. There is a quieter variant: if B's save lands between A's save and A's hashing, A records the hash of B's output and its verdict silently flips. Fix: allocate a unique temp file per upload with tempfile.mkstemp() and finish with os.replace(), which is atomic. A finally block removes the temp file if anything fails, so failures no longer leak into TempFiles. Observed on ccextractor PR #2309. Tests 9402 (master) and 9410 (the PR) ran concurrently on Linux; their VM logs contain 37 and 21 HTTP 500s, matching the 37 and 21 spurious "No output generated" results almost exactly, against a shared baseline of 24 genuine failures. The Windows runs of the same two commits were staggered, hit zero 500s, and produced identical verdicts. error.log holds 2184 of these failed renames, so this has been corrupting results for a long time; it only became obvious once a backlog started launching two tests at the same moment.
cfsmp3
requested review from
canihavesomecoffee and
thealphadollar
as code owners
August 9, 2026 18:42
|
canihavesomecoffee
approved these changes
Aug 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Problem
upload_type_request()saved every upload toTempFiles/<uploaded name>:That name is derived from the regression test output, so it is identical for every test running the same regression test. Two tests running at once share one path:
TempFiles/X.srtTempFiles/X.srtTestResults/<hashA>.srtFileNotFoundError→ HTTP 500The 500 means no
TestResultFilerow is written.mod_test/controllers.pythen setsgot = 'error'for any regression test with no result files but expected output, which renders as "No output generated but there should be". The contributor sees their PR breaking tests it never touched.There is a quieter variant: if B's save lands between A's save and A's hashing, A records the hash of B's output and its verdict silently flips. No error is raised in that case at all.
Evidence
From ccextractor PR #2309. Tests 9402 (master) and 9410 (the PR) ran concurrently on Linux, 15:32–16:02:
The two Windows runs of the same commits were staggered (15:35 vs 15:44), hit zero 500s, and produced identical verdicts. The two Linux runs disagreed on 57 tests, in two completely disjoint sets — the signature of a race, not a code difference.
Server-side traceback:
error.logcontains 2184 of these failed renames, so this has been corrupting results for a long time. It only became obvious once a backlog started launching two tests at the same moment.Fix
Allocate a unique temp file per upload with
tempfile.mkstemp(), and finish withos.replace()(atomic). Afinallyblock removes the temp file if anything fails, so failures no longer leak intoTempFiles.Testing
test_upload_type_request_uses_unique_temp_path— two uploads of the same filename must get different temp paths, both results land,TempFilesends empty. Fails on master with'…/TempFiles/shared_name.srt' == '…/TempFiles/shared_name.srt'.test_upload_type_request_cleans_temp_file_on_failure— a failing save leaves nothing behind.test_upload_type_requestfor the new call shape (mkstemp+os.replace).tests.test_ci.test_controllers: 202 tests, OK.isort,pydocstyleclean;mypy's only complaint is the pre-existing missing PyYAML stub, which CI installs via--install-types.Note
This does not change how many tests genuinely fail — the baseline of 24 real failures ("Path is empty", CCExtractor exit code 10) is untouched. It stops the platform inventing extra ones and blaming them on whichever PR happened to be running.