Rename batch_size config to queue_size and set better defaults - #125
Merged
Conversation
The value controls how many files are queued per processing chunk, not a batch or thread pool size. Renamed across config, client API, tests, and README. Old config files using batch_size are still accepted with a deprecation warning.
lfoppiano
force-pushed
the
feature/rename-queue-size
branch
from
August 18, 2026 09:36
caffcc8 to
849ec67
Compare
With a fixed default of 10, running with a higher --n left most of the thread pool idle (effective parallelism is min(n, queue_size)). An unset queue_size now resolves to n at processing time, so the pool is never starved by the default. An explicit config or constructor value still takes precedence.
lfoppiano
changed the title
An unset queue_size now defaults to 1000 for local directories (where the queue holds only file paths, as it historically did) and to 1.2 * n for archive and s3 streaming (where a whole chunk is materialized in memory), keeping 20% headroom over the thread pool. The queue_size entry was dropped from the shipped config.json so these defaults apply out of the box, and the README gains a "Choosing a queue size" section with tuning guidance.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The batch_size parameter name is misleading, the value controls how many files are queued per processing chunk, not a batch or thread pool size. Renamed across config, client API, tests, and README.
Old config files using batch_size are still accepted with a deprecation warning.