Replace searchsorted with a hash index in the reverse-pivot scatter

[Mmoncadaisla]

Why

The reverse pivot (to_dataset) resolves each result row's dim-coord values to array positions before scatter-writing into the dense output. Irregular (non-uniformly-spaced) axes — station networks, arbitrary point sets — used np.argsort once plus np.searchsorted per batch: O(log n) per row. Every fast engine (DuckDB's PhysicalPivot, ClickHouse's aggregator, Daft's pivot) does this same value-to-slot resolution with a hash table built once and probed vectorized; this PR adopts that discipline via pd.Index.get_indexer, whose persistent hash table is built once per dimension and probed per batch: O(1) amortized per row.

Uniformly spaced axes keep the existing affine fast path, so regular-grid workloads (ERA5 lat/lon/time) are completely unaffected — verified: ordered/unordered ERA5 reconstruction ratios stay ~1.0 on DataFusion, DuckDB, and Polars.

Measured

Isolated reconstruction through xql.to_dataset on shuffled rows over an irregular axis (GCP n2-standard-16, median of 5):

cells	searchsorted (main)	hash (this PR)	speedup
24 K	4.0 ms	1.9 ms	2.1x
240 K	40.4 ms	15.2 ms	2.7x
2.4 M	481 ms	162 ms	3.0x
6 M	1 945 ms	624 ms	3.1x
24 M	13 685 ms	4 168 ms	3.3x

Thread-scaling check (4 dask-style threads, 5 M probes over a 200 K-value axis): identical 3.7x scaling for both implementations — pandas' hash probe releases the GIL as well as searchsorted does — with the hash path ~7x faster per call.

What

The three position-resolution strategies now live in one place, _CoordLookup, built once per dimension per reconstruction:

• affine formula for uniformly spaced axes (unchanged);
• hash (pd.Index, built once, get_indexer per batch) for irregular axes with unique values;
• the previous argsort+searchsorted for axes a hash table cannot represent: duplicate values, or dtypes pd.Index rejects (float16 raises NotImplementedError on pandas 2.3.0).

Behavior change: a result value absent from an irregular unique axis now raises ValueError naming the dimension (previously a searchsorted misplacement surfaced as an opaque AssertionError deep in the scatter, or a silent wrong-cell write). Both callers construct requested such that this cannot fire on well-formed results (_dataset_from_batches derives coords via pd.unique of the same batches; SQLBackendArray._raw_getitem windows come from the engine filter), so it only converts an existing latent corruption into a loud error.

Tradeoff, quantified: the pandas hash table holds ~34 bytes/value transient vs ~16 bytes/value for the sorted copy it replaces (measured on a 2 M-value float64 axis) — per non-affine dimension, per reconstruction call, freed with the call frame. Affine axes build neither.

Validation

• tests/test_coord_lookup.py exercises each strategy through the public to_dataset contract on shuffled rows (irregular, descending-affine, NaN dim values, float16), plus two seam-level tests for behaviors unreachable through the eager public path (missing-value error, duplicate-axis fallback), with the reason documented.
• Adversarial review passes (dtype coercion incl. datetime64 unit mismatches, NaN/NaT, empty/len-1 axes, thread safety of per-call state, callers' reachability of the new ValueError) found no correctness refutation.
• Full suite on a clean release build: 322 passed, 2 skipped, 0 failed at head.
• mypy and ruff check/format match main's baseline exactly.

🤖 Generated with Claude Code