feat(claude-messages)!: emit invoke_agent, chat and execute_tool spans by apucacao · Pull Request #30 · launchdarkly/python-ai-sdk

apucacao · 2026-08-11T20:13:09Z

Replaces one flat span per call with the tree the TypeScript SDK emits, for claude-messages.

invoke_agent                     one per call, carries the LD identity and the run total
├── chat {model}                 one per model turn, with that turn's own tokens
└── execute_tool {name}          one per tool call, a sibling of chat

A multi-turn run with tools was previously one span with one set of token counts, so per-turn cost and latency were not recoverable from a trace, and a tool call left no trace beyond its LD metric event.

What else changed

Span construction moved to spans.py beside the handler, so the tool loop reads as a tool loop rather than as span bookkeeping with a provider call in the middle. The five handler PRs above follow this shape.
Cache tokens now reach the span. A turn that read 19,971 tokens from cache and wrote 3,580 more previously reported an input of 3; it now reports 23,554, and there is a test with those numbers in it.
The handler still returns the cache fields unfolded, in Anthropic's own names, because parse_usage folds exactly once. RawRunUsage carries that shape and is named so it cannot be confused with the client's cache-inclusive RunUsage.
Finish reasons are mapped: end_turn becomes stop, tool_use becomes tool_calls.
A failed run reports what its completed turns cost, but only when a turn actually reported usage. All-zero attributes would assert the run cost nothing, which a run whose first call died mid-flight cannot claim.
The streaming path gets a finally. A consumer that breaks out of the iteration skips except entirely, because GeneratorExit inherits from BaseException, so the root span was never ended and never exported: the whole run vanished from AI Config Monitoring along with the feature_flag event it carries.

The telemetry tests are rewritten rather than extended, because they pinned the old flat span. The single shared mock span is replaced with a recorder that keeps one object per span, since the old approach could not tell a parent from a child.

Not changed: the tool catalog is still unfiltered, unlike the TypeScript SDK. That difference predates this work and changes what the model is offered rather than what the span reports.

Breaking change

The span is renamed from claude.messages to invoke_agent. Queries selecting on the old name will not match. Prompt and completion content is no longer on spans unless the caller passes capture_content=True.

Where this sits

Needs the usage layer (#28) and the content layer (#29). Independent of the other five handler PRs; the stack orders them only because gh stack is linear.

Tests: 711 to 729.

Note

Overview
Breaking: The per-call span is renamed from claude.messages to invoke_agent, and prompts/completions are omitted from spans unless capture_content=True.

Replaces one flat span with the same tree as the TypeScript SDK: an invoke_agent root (LaunchDarkly identity, run-level tokens), one chat {model} child per model turn, and execute_tool {name} siblings per tool call (parented to the root, not under chat). Span helpers live in new spans.py; blocking and streaming paths share that shape.

Usage on spans now folds Anthropic cache fields into input totals via RawRunUsage while still returning unfolded cache fields for parse_usage. Finish reasons are mapped (end_turn → stop, tool_use → tool_calls). Failed or partial runs report token spend only when a turn actually reported usage.

Lifecycle: finally / end_unfinished_spans / end_span_once close spans on cancellation, stream abandonment, and serialization failures so traces still export to AI Config Monitoring.

^{Reviewed by Cursor Bugbot for commit 54b57f2. Bugbot is set up for automated code reviews on this repo. Configure here.}

apucacao · 2026-08-11T20:20:34Z

bugbot run

apucacao · 2026-08-11T20:44:49Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a4e837e. Configure here.}

apucacao · 2026-08-11T21:02:17Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 80c5a66. Configure here.}

apucacao · 2026-08-11T21:19:28Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 80c5a66. Configure here.}

apucacao · 2026-08-11T21:46:40Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 80c5a66. Configure here.}

apucacao · 2026-08-12T17:23:29Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 914e2bb. Configure here.}

apucacao · 2026-08-12T17:46:32Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0aeafc5. Configure here.}

apucacao · 2026-08-12T18:03:32Z

bugbot run

apucacao · 2026-08-12T18:08:38Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 4ad052e. Configure here.}

apucacao · 2026-08-12T18:20:57Z

bugbot run

apucacao · 2026-08-12T18:40:18Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 1ac5a89. Configure here.}

apucacao · 2026-08-12T18:51:55Z

bugbot run

apucacao · 2026-08-12T19:10:22Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 60586d4. Configure here.}

apucacao · 2026-08-12T19:22:19Z

bugbot run

apucacao · 2026-08-14T19:30:56Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 563063f. Configure here.}

One flat span per call, named claude.messages, becomes the tree the TypeScript SDK emits: an invoke_agent root, one `chat {model}` child per model turn, one `execute_tool {name}` child per tool call. A five-turn run with tools was previously one span with one set of token counts, so per-turn cost and latency were not recoverable from a trace at all, and a tool call left no trace beyond its LD metric event. BREAKING CHANGE: the span this handler emits is renamed from `claude.messages` and `claude.messages.stream` to `invoke_agent`. Queries that select on the old names will not match. Prompt and completion content is no longer on spans unless the caller passes capture_content=True. Span construction moved to spans.py so the tool loop reads as a tool loop rather than as span bookkeeping with a provider call in the middle. Tool spans take the root's context, not the chat span's, so they are siblings of chat rather than nested inside it. Both parents are passed explicitly: these handlers open a plain span rather than an active one, so there is no ambient span for a child to inherit, and a host app with its own tracer provider would otherwise get a flat trace. The root keeps what only it can carry: the launchdarkly.* identity, the feature_flag event, and the run's token total. It is the span a config-scoped query finds, and summing the children requires having already found them. A test asserts children carry none of it. Cache tokens now reach the span. Anthropic reports cache reads and writes beside input_tokens rather than inside it, so a turn that read 19,971 tokens from cache and wrote 3,580 more reported 3. The chat span now reports 23,554 for that turn, and there is a test with those numbers in it. The handler's return value keeps the cache fields unfolded, in Anthropic's own names, because parse_usage folds exactly once; a pre-folded figure returned alongside the fields would count the cache twice downstream. RawRunUsage carries that shape and is named so it cannot be confused with the client's SpanUsage-based RunUsage, which is cache-inclusive. Finish reasons are mapped rather than passed through: end_turn becomes stop, tool_use becomes tool_calls. A consumer grouping by this attribute across handlers previously saw two names for one outcome. A failed run now reports what its completed turns cost, on the root, but only when a turn actually reported usage. All-zero attributes would assert the run cost nothing, which a run whose first call died mid-flight cannot claim. The streaming path gets a `finally`. A consumer that breaks out of the iteration makes the generator skip `except` entirely, because GeneratorExit inherits from BaseException, so the root span was never ended and never exported: the whole run vanished from AI Config Monitoring along with the feature_flag event it carries. Every span now ends through end_span_once, and an abandoned one is marked and left UNSET rather than ERROR, because LaunchDarkly's own metrics record neither a success nor an error for abandonment and ERROR would put two dashboards in disagreement about one run. The success tail sets status without ending, so the `finally` owns every end. Ending twice is ignored by the OTel SDK but recorded as a diagnostic error, and would hide a genuine leak. Tests: the telemetry classes are rewritten rather than extended, because they pinned the old flat span. The single shared MagicMock span is replaced with a recorder that keeps one object per span, since the old approach could not tell a parent from a child. 82 tests here, up from 64. The fake usage object now declares only the fields Anthropic sets, so a handler cannot read a cache field the provider never reported. Not changed: the tool catalog is still unfiltered, unlike the TypeScript SDK, which offers the model only tools that have a registered handler. That difference predates this work and changes what the model is offered rather than what the span reports.

…apper The wrapper never passed capture_content to the factory, so it stayed in kwargs and reached config(), which takes no such argument. A caller asking for content on spans got a TypeError rather than content. Lifted out alongside variables, which was already handled the same way and for the same reason: one configures the handler, the other belongs to the invocation, and config() accepts neither. Two tests, one per branch, asserting the flag reaches the factory and does not reach config(). Found by Bugbot on #33 against openai-agents. Five of the six wrappers had it; each is fixed in its own layer.

… its span The success-side content write and the span finish sat outside the try, so a raise while recording the result skipped both the finish and the failure path. The tool span was never ended, so the exporter never saw it: the run showed a root marked ERROR and no sign the tool had been called. Reachable rather than theoretical. Serialising a tool result raises TypeError whenever capture_content is on and the result is not JSON-serialisable, which is any object a handler happens to return. The TypeScript handler has always done this inside the try. I put it outside when porting, and the two handlers that copied this file's shape inherited it, so they are fixed in their own layers. Found by Bugbot on #34, against the handler that copied it rather than this one.

… reads Span construction moved to spans.py, which holds the real _HAS_OTEL. The handler kept its own copy, plus the two imports it needed, alive only by a noqa. Nothing read any of it. That mattered because the tests patched the dead one. 7 tests set handler._HAS_OTEL to False and believed they were exercising the install without the otel extra; the flag was unread, so they exercised nothing and passed either way. They now patch spans._HAS_OTEL, which is the flag start_root_span actually consults: with it patched, span creation returns None, and with it set it does not. Found by Bugbot on #32. Five of the six handlers carried the dead gate, and four had tests aimed at it.

The streaming finally closed the model span and the root, but the in-flight execute_tool span was held only by a local. except Exception does not see a CancelledError or a GeneratorExit, so a tool cancelled mid-flight left its span open and unexported: the trace showed a closed parent above a child that never arrived, which reads as a tool still running long after the run ended. Tracked in open_tool_span and abandoned in the finally, the same way the model span already was. The tracker is cleared on the two paths that end the span and deliberately not in a finally, because a finally would also clear it for the BaseException case, which is the one case where the outer finally is the only thing left to close it. Found by Bugbot on the openai-messages layer. This handler is where the pattern was copied from, so the same gap reached four of the six.

…guard The output content write and the span finish sat outside the try that fails the chat span, and the blocking path has no finally that could recover it. A raise while serialising the completion left the span open and unexported, so the trace showed an errored root with no model call at all, and the turn was dropped from the run total even though Anthropic had already billed it. Reachable through capture_content, where serialising any non-JSON-serialisable value raises TypeError. The usage is now accumulated straight after the provider returns, before anything that can raise, and every span write happens inside the guard. Two tests: the span still ends and reports ERROR, and the tokens survive. Found while checking whether the openai-messages defect Bugbot reported on #32 reached the other handlers. It did.

…ntent failure The blocking path accumulates a turn's usage before any content work. The streaming path did it last, after the content write and the span finish, so a raise while serialising the completion dropped a turn Anthropic had already billed and left the root reporting less than the run cost. Both paths now accumulate straight after the provider returns. Found by Bugbot on #30.

The prompt write ran before the try that fails the span it writes to. Serialising conversation content raises on anything that is not JSON-serialisable, so a raise there left the chat span open in the tool loop, and on both root paths left the root open: never ended, never exported, so the run disappeared from AI Config Monitoring along with the feature_flag event it carries. The output writes were moved inside their guards earlier in this stack. The input writes were not, which is the same defect at the other end of the same span. Two tests, one per root path. Found by Bugbot on #34, which is this shape in langchain-messages.

…ot send RawRunUsage seeded both cache fields at zero, which undid raw_usage_of: that function omits the fields Anthropic did not send precisely so absent stays absent. The returned bag therefore always looked cache-aware, and parse_usage emitted an input_details breakdown of zeros for a model with no prompt caching at all. A zero cache read is a claim, and this accumulator has no grounds to make it. The two base counts are always present, because a completed turn always has them. A cache field joins the total only once some turn reports one. Two tests: a model with no caching returns the two base counts alone, and a reported cache figure is still carried unfolded so parse_usage folds it once. Found by Bugbot on #30.

…ing it The content write and the span finish sat after the try that fails the chat span, so a serialisation failure reached the outer finally with open_model_span still set. The chat span was ended as abandoned and left UNSET while the root was marked ERROR, which describes one turn as a consumer walking away and as a failure at the same time. The blocking path records the exception and ERROR for the same failure. Both writes now sit inside the guard, so the two paths agree. One test, and it fails on the reverted code with the chat span carrying no status at all. Found by Bugbot on this PR.

A timeout or a task.cancel() raises asyncio.CancelledError, which inherits from BaseException, so it walks past every except Exception this handler has. The blocking path ended its spans only from those clauses, so a cancelled run exported nothing at all. Not a wrong attribute: no span. The root carries the feature_flag event and every launchdarkly.* attribute, so the whole run vanished from AI Config Monitoring rather than showing as incomplete. Two finally blocks now own the ends the except clauses cannot reach: one in the turn loop for the chat and execute_tool spans, one in the caller for the root. This is the shape the streaming path has had since the earlier rounds, so both paths of this handler now agree. Open spans are tracked by clearing a local when a path ends one, rather than by asking the span. A mock span answers is_recording() truthily and both test suites here are built on mock spans, so asking would have made the finally fire a second end on every successful run. A cancelled root still reports the spend of the turns that completed, for the same reason the failure path does: those turns were billed. Spans are left at UNSET and marked launchdarkly.run.cancelled. Nothing failed, the caller went away. Two tests, driving a real task.cancel() against a provider call that never returns. Gutting either finally fails both. This is the reference for the other five handlers, which have the same gap.

…ndoned A CancelledError never enters except Exception, so the streaming teardown always ran its abandonment path and marked launchdarkly.stream.abandoned. A consumer that stops reading did abandon the stream, and that word is right for it. A timeout did not: nothing chose to stop reading, the run was cancelled underneath the consumer. The blocking path in this handler already reports launchdarkly.run.cancelled for that, so the two paths disagreed about the same event. The existing test for a tool cancelled mid-flight asserted stream.abandoned, which is what the defect looked like from inside. It now asserts run.cancelled and that the other key is absent. The consumer-break test above it still asserts stream.abandoned, which is the case that word is for, and it needed no change. No new attribute. Both keys already exist and are in the vocabulary lock. Removing the CancelledError branch fails the test. Found by Bugbot on the openai-agents layer, then found here by audit.

The write sat before the try that calls fail_span on that span, so a serialisation failure reached the outer handler instead. The root was marked ERROR while the chat span stayed open, and the finally then ended it as abandoned at UNSET. A reader saw a failed run whose model call merely stopped, which is two different stories about one turn. The blocking path has kept this write inside the guard from the start, and the streaming output write was moved inside for the same reason in an earlier round. This was the last of the four that was still outside. The test raises from the second call rather than the first, because the root's own input write happens before the chat span exists and must still succeed. Moving the write back outside fails the test. Found by Bugbot on this PR.

span_usage_of only forwarded to add_cached_tokens_to_input, and both call sites already call that directly. It was the last reason this module imported SpanUsage. A wrapper with no callers is worse than none: the next reader has to work out whether it is the intended entry point and the direct calls are the mistake, or the other way around. Found by Bugbot on this PR.

apucacao · 2026-08-14T19:40:35Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 54b57f2. Configure here.}

apucacao · 2026-08-14T19:57:23Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 54b57f2. Configure here.}

Handlers hold None for every span whenever the OpenTelemetry SDK is absent, and every other helper in this family already no-ops on it. This one did not, so a streaming cleanup path in a finally crashed with AttributeError on an install without the otel extra: the one place that should be hardest to break was the one place that was not guarded. Fixed in the shared helper rather than at six call sites, because all six handlers use it the same way and the next handler would hit the same edge. Found by Bugbot on #30.

The content write and the span finish sat after the try that fails the chat span, and the usage was accumulated last of all. A raise while serialising the response left the span for the finally to end as abandoned, which reads as a consumer who walked away rather than as the failure it was, and dropped a turn the provider had already billed. The blocking path in this same file already did both correctly, which is what made the streaming path's ordering look accidental rather than considered. It was. Two tests: the span is failed rather than abandoned, and the tokens survive. Found by auditing every handler for the ordering Bugbot reported on #30 and #34. This path had the same defect and had not been reported.

on_llm_end clears open_model_span before it writes, so close_open_spans can no longer reach that span: it is the only code left that can end it. A serialisation failure in the content write left the chat span open with nothing tracking it, never ended and never exported, while the run reported an error on the root with no sign a model call had happened. The langchain-agents callback guards the identical shape for the identical reason, and says so in a comment. This one did not. One test, and it fails on the reverted code. Found by auditing the handlers for the guard Bugbot reported missing on #30.

…rite fails on_llm_end accumulated the turn's usage after the content write. A raise while serialising completion content dropped a turn the provider had already billed, and that accumulator is what a failed run's root reports and what a successful run falls back to when the messages carry no usage of their own. The accumulation now happens before the write, matching what the other five handlers do. Found by auditing every handler for the ordering Bugbot reported on #30 and #34.

apucacao mentioned this pull request Aug 11, 2026

test: fail the build when the six handlers drift apart again #36

Merged

cursor Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/claude-messages/src/launchdarkly_ai_claude_messages/handler.py Outdated

apucacao force-pushed the ag/py-telemetry-claude-messages branch from 0ccb68d to a4e837e Compare August 11, 2026 20:43

cursor Bot reviewed Aug 11, 2026

View reviewed changes

cursor Bot reviewed Aug 12, 2026

View reviewed changes

Comment thread packages/claude-messages/src/launchdarkly_ai_claude_messages/handler.py

cursor Bot reviewed Aug 12, 2026

View reviewed changes

Comment thread packages/claude-messages/src/launchdarkly_ai_claude_messages/handler.py Outdated

cursor Bot reviewed Aug 12, 2026

View reviewed changes

Comment thread packages/claude-messages/src/launchdarkly_ai_claude_messages/spans.py

cursor Bot reviewed Aug 12, 2026

View reviewed changes

cursor Bot reviewed Aug 14, 2026

View reviewed changes

Comment thread packages/claude-messages/src/launchdarkly_ai_claude_messages/spans.py Outdated

cursor Bot reviewed Aug 14, 2026

View reviewed changes

apucacao added 14 commits August 14, 2026 15:38

apucacao force-pushed the ag/py-telemetry-claude-messages branch from 563063f to 54b57f2 Compare August 14, 2026 19:39

cursor Bot reviewed Aug 14, 2026

View reviewed changes

andrewklatzke approved these changes Aug 14, 2026

View reviewed changes

apucacao marked this pull request as ready for review August 14, 2026 21:18

apucacao merged commit 6a82ef5 into main Aug 14, 2026
12 checks passed

apucacao deleted the ag/py-telemetry-claude-messages branch August 14, 2026 21:20

Uh oh!

Conversation

apucacao commented Aug 11, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What else changed

Breaking change

Where this sits

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

Uh oh!

apucacao commented Aug 14, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 14, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 14, 2026

Uh oh!

apucacao commented Aug 11, 2026 •

edited by cursor Bot

Loading