fix: Bound the shutdown wait so close() cannot block forever - #494
Open
aviadr1 wants to merge 1 commit into
Open
fix: Bound the shutdown wait so close() cannot block forever#494aviadr1 wants to merge 1 commit into
aviadr1 wants to merge 1 commit into
Conversation
LDClient.close() had no timeout anywhere on its shutdown path, so if delivery
of the final event payload stalled, close() never returned.
The trigger is hanging DNS resolution, which is not covered by HTTPConfig's
connect_timeout or read_timeout: urllib3 calls socket.getaddrinfo() before it
applies sock.settimeout(), so name resolution sits outside both. When a
resolver blackholes requests instead of returning NXDOMAIN, a flush worker
blocks in getaddrinfo() indefinitely and the whole shutdown path queues behind
it. Because close() is typically called from an atexit or interpreter-shutdown
hook, a process in this state can never exit.
Every wait in the chain was untimed:
DefaultEventProcessor.stop()
-> _post_message_and_wait('stop') # blocking put, then reply.wait()
-> EventDispatcher._do_shutdown() # runs before the reply is set
-> FixedThreadPool.wait() # Event.wait() with no timeout
-> the EventPayloadSendTask stuck in getaddrinfo()
Add a shutdown_timeout config option, defaulting to 5 seconds, and thread it
through those waits. FixedThreadPool.wait() now takes an optional timeout and
reports whether it drained; _post_message_and_wait() bounds both the inbox put
and the reply wait; _do_shutdown() shares one deadline across its waits. When
the budget is exhausted the SDK logs a warning and returns. Worker threads are
daemons, so any left stuck do not keep the process alive.
Dropping an undeliverable final payload is consistent with how analytics events
are already treated - they are dropped when the inbox is full, when the outbox
overflows, and when all flush workers are busy - and is far better than never
returning. Set shutdown_timeout=None to restore the previous unbounded wait.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #493.
The bug
LDClient.close()has no timeout anywhere on its shutdown path. If delivery of the final analytics event payload stalls,close()never returns.The trigger is hanging DNS resolution, which
HTTPConfig'sconnect_timeoutandread_timeoutdo not cover: urllib3 callssocket.getaddrinfo()inutil/connection.pybefore it appliessock.settimeout(), so name resolution sits outside both. When a resolver blackholes requests instead of returning NXDOMAIN, a flush worker blocks ingetaddrinfo()indefinitely and the entire shutdown path queues up behind it.Because
close()is usually called from anatexit/ interpreter-shutdown hook, a process in this state can never exit. We hit this on Kubernetes with a degraded cluster resolver and had worker pods alive for hours to days after finishing their work, each holding a concurrency slot that was never released.Every wait in the chain was untimed:
Note there are two unbounded waits in
_post_message_and_waitalone — the blockingQueue.putas well asreply.wait()— so bounding only the reply would still leave a way to hang.The fix
Add a
shutdown_timeoutconfig option, defaulting to 5 seconds, and thread it through those waits:FixedThreadPool.wait()takes an optional timeout and returns whether it actually drained._post_message_and_wait()takes an optional timeout, bounding both the inbox put and the reply wait, and returns whether the message was handled._do_shutdown()shares a single deadline across its flush-worker and diagnostic-worker waits, so they cannot add up to more than the budget.When the budget is exhausted the SDK logs a warning and returns. Worker threads are daemons, so any left stuck do not keep the process alive.
On the default
I made the default bounded (5 seconds, matching
flush_interval) rather than preserving the current unbounded wait, because an unbounded default is what makes this a hang rather than a delay, and the failure mode is silent and unrecoverable.Dropping an undeliverable final payload is consistent with how analytics events are already treated — they are dropped when the inbox is full, when the outbox overflows, and when all flush workers are busy. Anyone who wants the old behavior can set
shutdown_timeout=None.Happy to change the default or the option name if you'd prefer something else.
Verification
Reproduction using the real SDK and real urllib3 stack, with only
socket.getaddrinfomade to hang (full script in #493):Two tests added:
test_stop_returns_even_if_event_delivery_never_completes— the invariant. Stalls a flush worker on a payload that never completes and assertsstop()still returns. I verified this fails against the unfixed code (stop() never returned; it is waiting on a delivery that never completes) rather than only passing against the fix.test_stop_still_delivers_buffered_events— a control, so that astop()which simply gave up immediately could not pass the first test. It asserts buffered events are still delivered when delivery works normally.LD_SKIP_DATABASE_TESTS=1 uv run pytest: 1417 passed, 279 skipped.mypy,isort --check, andpycodestyleall clean.Not addressed here
DefaultAsyncEventProcessor.stop()has the same unbounded shape (async_event_processor.py:245), whileflush_and_wait()directly above it already bounds its wait withasyncio.wait_for. I have not reproduced a hang on that path, so I left it out rather than change code I could not verify — but it looks worth a follow-up.Note
Overview
Adds
shutdown_timeoutonConfig(default 5 seconds) soLDClient.close()/ event processorstop()no longer wait forever when final analytics delivery stalls—e.g. hung DNS outsideHTTPConfigconnect/read timeouts.Shutdown now shares one deadline via
_Deadline:FixedThreadPool.wait(timeout)returns whether workers drained;_post_message_and_waitbounds inboxputand replywait;EventDispatcher._do_shutdownapplies the same budget to flush and diagnostic pools. On timeout the SDK logs a warning and returns; undelivered events may be dropped.shutdown_timeout=Nonerestores indefinite wait.Tests cover
stop()returning while delivery never completes and still flushing buffered events when delivery works.Reviewed by Cursor Bugbot for commit f7d6c8d. Bugbot is set up for automated code reviews on this repo. Configure here.