Split out of #36943 so the independent fixes there could land. The work is preserved on branch issue-36942-shard-rebalance-wip at f612b27ed8 — resume from there rather than rebuilding it.
It works
Measured on run 31203516115, with fail-fast disabled so every shard reported instead of the first failure cancelling the matrix:
IT shards 22.2 - 26.2m (was 38.9m)
Postman 18.6 - 28.8m (was 37.8m)
Bin-packing on measured per-class time balanced 557 integration classes to 18.4m each, spread 0.0m. Per-class times are stable run-to-run (115.2m vs 114.8m on a control set), so the balance should hold. Job wall times are noisy — an untouched shard swung +8m between runs — partly because runners vary by VM generation (D4ads_v5 / D4ds_v6 / D4ds_v7, all 4 vCPU / 15 GB).
Why it is not merged
3 of 7 integration shards and 5 of 9 Postman shards failed. Every failure was a test that only passed because of which suite-mates ran before it.
The Postman groups are dependency clusters, not labels. category-content grouped Category + ContentResourceV1 + Content_Resource because those collections share data. Rebalancing purely on time shattered every group. The GraphQL folder split failed the same way: "Page API - Testing 'page' field with inline fragments" needs setup performed by the Page API folder, which landed in the other shard.
Why it is still worth doing
Beyond the ~12m of wall clock, it is an effective detector of tests that depend on their neighbours. Every failure it surfaced was a real defect, not a false positive:
Outstanding, found but not yet fixed:
FieldUtilTest, ContentletAjaxTest — assume a suite-mate materialised the Languagevariable content typeMenuResourceTest.test_get_translation — language-variable lookupWebAssetHelperIntegrationTest ×3 — limited-user permissionsAjaxDirectorServletIntegrationTestPersonaAPITest — asserts an absolute global count (allPersonas should be 5, got:6), so any test that leaves a persona behind breaks it
Suggested approach when resumed
- Fix the tests above on their own, against the current shard layout, before re-sharding anything. Each is a real bug and lands independently.
- For Postman, treat existing groups as atomic. Rebalance by moving whole groups between shards, or by splitting a single oversized collection by file (as
default-split was done historically) — not by regrouping collections across clusters.
- For integration, consider contiguous splits of the largest suites, which preserve each class's neighbours, before free-form repacking.
- Keep
fail-fast: false while iterating — otherwise each ~40 minute run reveals only one shard's worth of coupling. Revert it before merging.
Note on measurement
.github/scripts/test-balance/find_swallowed.py landed in #36943. The packing scripts (parse_suites.py, pack_suites.py, pack_postman.py) are on the preserved branch.
The rebalance only pays off if both tails move: integration and Postman maxima were within a minute of each other (38.9m vs 37.8m), so fixing one alone changes wall clock by ~0.
Split out of #36943 so the independent fixes there could land. The work is preserved on branch
issue-36942-shard-rebalance-wipatf612b27ed8— resume from there rather than rebuilding it.It works
Measured on run 31203516115, with
fail-fastdisabled so every shard reported instead of the first failure cancelling the matrix:Bin-packing on measured per-class time balanced 557 integration classes to 18.4m each, spread 0.0m. Per-class times are stable run-to-run (115.2m vs 114.8m on a control set), so the balance should hold. Job wall times are noisy — an untouched shard swung +8m between runs — partly because runners vary by VM generation (
D4ads_v5/D4ds_v6/D4ds_v7, all 4 vCPU / 15 GB).Why it is not merged
3 of 7 integration shards and 5 of 9 Postman shards failed. Every failure was a test that only passed because of which suite-mates ran before it.
The Postman groups are dependency clusters, not labels.
category-contentgroupedCategory+ContentResourceV1+Content_Resourcebecause those collections share data. Rebalancing purely on time shattered every group. The GraphQL folder split failed the same way: "Page API - Testing 'page' field with inline fragments" needs setup performed by thePage APIfolder, which landed in the other shard.Why it is still worth doing
Beyond the ~12m of wall clock, it is an effective detector of tests that depend on their neighbours. Every failure it surfaced was a real defect, not a false positive:
AWSS3PublishingEndPointthrew aNullPointerExceptionfrom inside a catch block whenever no request was bound to the thread — fixed in fix(ci): drop dead build artifact, guard an NPE in a catch block, and make integration tests report their failures #36943system— Language Variable content type is not markedsystemand can be deleted, breaking i18n site-wide #36958RuleBundlerTestcreated fixtures in a static@DataProvider, evaluated at suite construction, ~65 classes before use — fixed in fix(ci): drop dead build artifact, guard an NPE in a catch block, and make integration tests report their failures #36943PublishingEndPointTestdepended on a leakedHttpServletRequestThreadLocal— fixed in fix(ci): drop dead build artifact, guard an NPE in a catch block, and make integration tests report their failures #36943Outstanding, found but not yet fixed:
FieldUtilTest,ContentletAjaxTest— assume a suite-mate materialised theLanguagevariablecontent typeMenuResourceTest.test_get_translation— language-variable lookupWebAssetHelperIntegrationTest×3 — limited-user permissionsAjaxDirectorServletIntegrationTestPersonaAPITest— asserts an absolute global count (allPersonas should be 5, got:6), so any test that leaves a persona behind breaks itSuggested approach when resumed
default-splitwas done historically) — not by regrouping collections across clusters.fail-fast: falsewhile iterating — otherwise each ~40 minute run reveals only one shard's worth of coupling. Revert it before merging.Note on measurement
.github/scripts/test-balance/find_swallowed.pylanded in #36943. The packing scripts (parse_suites.py,pack_suites.py,pack_postman.py) are on the preserved branch.The rebalance only pays off if both tails move: integration and Postman maxima were within a minute of each other (38.9m vs 37.8m), so fixing one alone changes wall clock by ~0.