Skip to content

fix: Bound the shutdown wait so close() cannot block forever - #494

Open
aviadr1 wants to merge 1 commit into
launchdarkly:mainfrom
aviadr1:fix/bounded-close-timeout
Open

fix: Bound the shutdown wait so close() cannot block forever#494
aviadr1 wants to merge 1 commit into
launchdarkly:mainfrom
aviadr1:fix/bounded-close-timeout

Conversation

@aviadr1

@aviadr1 aviadr1 commented Aug 15, 2026

Copy link
Copy Markdown

Fixes #493.

The bug

LDClient.close() has no timeout anywhere on its shutdown path. If delivery of the final analytics event payload stalls, close() never returns.

The trigger is hanging DNS resolution, which HTTPConfig's connect_timeout and read_timeout do not cover: urllib3 calls socket.getaddrinfo() in util/connection.py before it applies sock.settimeout(), so name resolution sits outside both. When a resolver blackholes requests instead of returning NXDOMAIN, a flush worker blocks in getaddrinfo() indefinitely and the entire shutdown path queues up behind it.

Because close() is usually called from an atexit / interpreter-shutdown hook, a process in this state can never exit. We hit this on Kubernetes with a degraded cluster resolver and had worker pods alive for hours to days after finishing their work, each holding a concurrency slot that was never released.

Every wait in the chain was untimed:

DefaultEventProcessor.stop()
  -> _post_message_and_wait('stop')   # blocking put on a bounded queue, then reply.wait()
  -> EventDispatcher._do_shutdown()   # runs before the reply is set
  -> FixedThreadPool.wait()           # Event.wait() with no timeout
  -> the EventPayloadSendTask stuck in getaddrinfo()

Note there are two unbounded waits in _post_message_and_wait alone — the blocking Queue.put as well as reply.wait() — so bounding only the reply would still leave a way to hang.

The fix

Add a shutdown_timeout config option, defaulting to 5 seconds, and thread it through those waits:

  • FixedThreadPool.wait() takes an optional timeout and returns whether it actually drained.
  • _post_message_and_wait() takes an optional timeout, bounding both the inbox put and the reply wait, and returns whether the message was handled.
  • _do_shutdown() shares a single deadline across its flush-worker and diagnostic-worker waits, so they cannot add up to more than the budget.

When the budget is exhausted the SDK logs a warning and returns. Worker threads are daemons, so any left stuck do not keep the process alive.

On the default

I made the default bounded (5 seconds, matching flush_interval) rather than preserving the current unbounded wait, because an unbounded default is what makes this a hang rather than a delay, and the failure mode is silent and unrecoverable.

Dropping an undeliverable final payload is consistent with how analytics events are already treated — they are dropped when the inbox is full, when the outbox overflows, and when all flush workers are busy. Anyone who wants the old behavior can set shutdown_timeout=None.

Happy to change the default or the option name if you'd prefer something else.

Verification

Reproduction using the real SDK and real urllib3 stack, with only socket.getaddrinfo made to hang (full script in #493):

# before
ldclient version: 9.16.1
    [resolver] getaddrinfo('events.launchdarkly.com') -> hanging (simulated DNS blackhole)
calling stop() -- this is what LDClient.close() does first
RESULT: stop() has not returned after 20s and is waiting on a 600s resolver -- UNBOUNDED (bug reproduced)

# after
ldclient version: 9.16.1
    [resolver] getaddrinfo('events.launchdarkly.com') -> hanging (simulated DNS blackhole)
calling stop() -- this is what LDClient.close() does first
Timed out waiting for the event processor to shut down after 5 seconds; some analytics events may not have been delivered
RESULT: stop() returned after 5.0s -- BOUNDED (fixed)

Two tests added:

  • test_stop_returns_even_if_event_delivery_never_completes — the invariant. Stalls a flush worker on a payload that never completes and asserts stop() still returns. I verified this fails against the unfixed code (stop() never returned; it is waiting on a delivery that never completes) rather than only passing against the fix.
  • test_stop_still_delivers_buffered_events — a control, so that a stop() which simply gave up immediately could not pass the first test. It asserts buffered events are still delivered when delivery works normally.

LD_SKIP_DATABASE_TESTS=1 uv run pytest: 1417 passed, 279 skipped. mypy, isort --check, and pycodestyle all clean.

Not addressed here

DefaultAsyncEventProcessor.stop() has the same unbounded shape (async_event_processor.py:245), while flush_and_wait() directly above it already bounds its wait with asyncio.wait_for. I have not reproduced a hang on that path, so I left it out rather than change code I could not verify — but it looks worth a follow-up.


Note

Overview
Adds shutdown_timeout on Config (default 5 seconds) so LDClient.close() / event processor stop() no longer wait forever when final analytics delivery stalls—e.g. hung DNS outside HTTPConfig connect/read timeouts.

Shutdown now shares one deadline via _Deadline: FixedThreadPool.wait(timeout) returns whether workers drained; _post_message_and_wait bounds inbox put and reply wait; EventDispatcher._do_shutdown applies the same budget to flush and diagnostic pools. On timeout the SDK logs a warning and returns; undelivered events may be dropped. shutdown_timeout=None restores indefinite wait.

Tests cover stop() returning while delivery never completes and still flushing buffered events when delivery works.

Reviewed by Cursor Bugbot for commit f7d6c8d. Bugbot is set up for automated code reviews on this repo. Configure here.

LDClient.close() had no timeout anywhere on its shutdown path, so if delivery
of the final event payload stalled, close() never returned.

The trigger is hanging DNS resolution, which is not covered by HTTPConfig's
connect_timeout or read_timeout: urllib3 calls socket.getaddrinfo() before it
applies sock.settimeout(), so name resolution sits outside both. When a
resolver blackholes requests instead of returning NXDOMAIN, a flush worker
blocks in getaddrinfo() indefinitely and the whole shutdown path queues behind
it. Because close() is typically called from an atexit or interpreter-shutdown
hook, a process in this state can never exit.

Every wait in the chain was untimed:

  DefaultEventProcessor.stop()
    -> _post_message_and_wait('stop')   # blocking put, then reply.wait()
    -> EventDispatcher._do_shutdown()   # runs before the reply is set
    -> FixedThreadPool.wait()           # Event.wait() with no timeout
    -> the EventPayloadSendTask stuck in getaddrinfo()

Add a shutdown_timeout config option, defaulting to 5 seconds, and thread it
through those waits. FixedThreadPool.wait() now takes an optional timeout and
reports whether it drained; _post_message_and_wait() bounds both the inbox put
and the reply wait; _do_shutdown() shares one deadline across its waits. When
the budget is exhausted the SDK logs a warning and returns. Worker threads are
daemons, so any left stuck do not keep the process alive.

Dropping an undeliverable final payload is consistent with how analytics events
are already treated - they are dropped when the inbox is full, when the outbox
overflows, and when all flush workers are busy - and is far better than never
returning. Set shutdown_timeout=None to restore the previous unbounded wait.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@aviadr1
aviadr1 requested a review from a team as a code owner August 15, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LDClient.close() blocks forever when DNS resolution hangs

1 participant