Automatic cleanup of old repeater "ready" items

[adrianbj]

Automatic cleanup of old repeater "ready" items

Short description

ProcessWire creates a hidden + unpublished "ready" item for a repeater each time an edit form renders, and never reaps them. The cleanup routine already exists and is complete, but the only way to trigger it is a checkbox on each field's config screen. On a long lived site the items accumulate indefinitely.

The core source already asks this question, in FieldtypeRepeater/config.php next to the checkbox:

// @todo: should we just do this automatically?
$numOldReady = $fieldtype->countOldReadyPages($this->field);

Why it matters

On a site running since 2022 with 37 repeater fields:

• 19,197 ready items, of which 18,842 are older than the 3 day threshold countOldReadyPages() already uses
• That is 33% of all pages on the site
• One field, icon_grid, had 15,242 ready items against 20 real ones

They are not inert. Repeater deletes cascade through deleteRepeaterPage() with children('include=all'), so the orphans are walked and deleted too. One RepeaterMatrix item whose nested parent held 363 orphans took ~12 seconds to remove, which showed up as gateway timeouts when an editor deleted a few blocks and saved.

What already exists

FieldtypeRepeater::countOldReadyPages($field, $delete = false, $secondsOld = 259200) does the whole job. Nothing calls it with $delete = true except the config screen POST handler.

Requests

1. Run it automatically, or provide a documented hook that a site can schedule. Given deletion can be slow (see processwire-issues for FieldtypePage::hookPagesDelete), doing it inline during a normal request is probably the wrong place - a LazyCron or a maintenance entry point would fit better.

2. Bound the existing button. The _deleteOldReady handler calls countOldReadyPages($field, true), which finds and deletes the whole backlog in one request with no set_time_limit() and no limit - unlike the neighbouring "unnecessary pages" handler, which sets set_time_limit(600) and caps at 500 with a "run this again" notice. On a large backlog the button reliably times out, which is exactly when a site most needs it. Applying the same cap and notice would make it usable.

3. Consider reusing ready items more aggressively. On the site above, created and modified for recent ready items match exactly, meaning no existing ready item is ever reused - a fresh one is minted per render (~50/day across two fields). getNextReadyPage() only reuses an item present in the loaded value, so when a nested repeater's field row is absent the existing pool is invisible to it and a new item is created every time.

Workaround

A CLI script that calls countOldReadyPages($field, true) per field, bounded by count and wall clock so it can run from cron and resume across runs.

Environment

• ProcessWire 3.0.270, PHP 8.5, MySQL 8 (AWS RDS)
• 37 repeater / RepeaterMatrix fields, ~56,000 pages

[adrianbj]

Related: processwire/processwire-issues#2304 covers why the deletion itself is so expensive on installs with many page reference fields, which is what makes the unbounded cleanup button time out on a large backlog.

[adrianbj]

Follow-up on point 3 — I have now root-caused why ready items are never reused. It is a self-sustaining loop, and it explains why the accumulation is concentrated in a couple of fields rather than spread evenly.

Mechanism

FieldtypeRepeater::getNextReadyPage() looks for a reusable item only in $value, the already-loaded RepeaterPageArray:

if($value) {
    foreach($value as $item) {
        if($item->hasStatus(Page::statusUnpublished)
            && $item->hasStatus(Page::statusHidden)
            && $item->id
            && substr($item->name, -1) !== 'c'
            && !in_array($item->id, $notIDs)) {
            $readyPage = $item; // reuse

If $value is empty it falls through and creates a new item. And $value is empty whenever the field has no row in its own table, even when ready items exist under the repeater parent.

The loop, for any repeater that is rendered but never populated:

1. No real items, so ___sleepValue() skips them all (if(!$p->id || $p->isHidden() || $p->isUnpublished()) continue;) and no row is written.
2. Next edit form render, $page->get($field) returns an empty PageArray, so $value is empty.
3. getNextReadyPage() has nothing to reuse and mints + saves a new ready item.
4. That item is hidden + unpublished, so step 1 still applies and the row stays absent.
5. Repeat, once per render, indefinitely.

It cannot self-correct, because the thing that would make the existing pool visible — a row in the field table — only appears once somebody actually puts content in the field.

Reproduction

Take a page whose repeater has items, delete only its row from the field table, leave the child pages in place:

host page 35954, repeater parent 35955 holds 6 child page(s)
db row data: '35956,35957,36068,36069,35958,35959'

A: with the db row present
   get() returned RepeaterPageArray, count=6

B: db row deleted, children left in place
   get() returned RepeaterPageArray, count=0
   child pages still in the tree: 6

Six pages sitting under the parent, invisible to get(), therefore invisible to the reuse check.

Corroboration

On the site from the original report, the live-to-ready ratio tracks exactly how often a field is actually filled in:

field	live items	ready items	ratio
columns	2	3,321	1660:1
icon_grid	20	15,242	762:1
page_blocks	436	613	1.4:1
newsletter_blocks	402	1	0.002:1
stats_items	77	0	—

Fields that regularly hold content have essentially no backlog. The two that are almost never populated account for 98% of the 18,842 stale items.

Suggested fix

Have getNextReadyPage() fall back to querying the repeater parent for an existing unused ready item when $value yields no candidate — hidden + unpublished, right template, not a clone, not in $notIDs. One query on a path that already writes a page, and it would drain an existing pool instead of adding to it.

Fixing this would not remove the need for the cleanup in the original request, since ready items are still legitimately created, but it would stop the unbounded growth on exactly the fields where it runs away.