feat(openai-messages)!: emit invoke_agent, chat and execute_tool spans by apucacao · Pull Request #32 · launchdarkly/python-ai-sdk

apucacao · 2026-08-11T20:13:18Z

Replaces one flat span per call with the tree the TypeScript SDK emits, for openai-messages.

invoke_agent                     one per call, carries the LD identity and the run total
├── chat {model}                 one per model turn, with that turn's own tokens
└── execute_tool {name}          one per tool call, a sibling of chat

Cached tokens were absent entirely

OpenAI reports them under input_tokens_details.cached_tokens, which nothing here read, so every span understated what a prompt-cached call actually reused.

They now appear in gen_ai.usage.cache_read.input_tokens and, unlike Anthropic, are not added on top of the input figure: OpenAI already counts them inside it, and adding them would double-count. Cache creation is always zero, because OpenAI has no such concept.

Two things specific to this handler

It is the only one of the six that reports the model which actually answered rather than the one requested, on both the root and the chat spans. OpenAI resolves an alias like gpt-4o to a dated snapshot, and this handler has the resolved value to hand.
Finish reasons are derived, not mapped. The Responses API has no finish_reason field, so the shared mapping table does not apply and is deliberately not imported. The value comes from a closed three-way check: a function call in the output means tool_calls, an incomplete status means length or content_filter depending on the reported cause, a completed status means stop, and anything else writes no attribute. The function-call check comes first, because status alone reports completed for a turn that stopped to call a tool.

Other changes

The streaming path gets a finally, so a consumer that breaks out of the iteration no longer leaves the root span unended and unexported, taking the whole run out of AI Config Monitoring along with the feature_flag event it carries.

Breaking change

The span is renamed from openai.response to invoke_agent. Queries selecting on the old name will not match. Prompt and completion content is no longer on spans unless the caller passes capture_content=True.

Where this sits

Needs the usage layer (#28) and the content layer (#29). Independent of the other five handler PRs; the stack orders them only because gh stack is linear.

Tests: 763 to 781.

Note

Overview
Replaces one flat openai.response span per call with the same tree as the TypeScript SDK: an invoke_agent root (LaunchDarkly identity and run-level token totals), one chat {model} child per model turn, and execute_tool {name} siblings parented to the root—not under the chat span.

Span logic moves into spans.py; blocking and streaming paths share RunUsage, explicit parent context, and teardown that ends open spans on errors, CancelledError, consumer abandonment, and content-serialisation failures without losing tokens already billed.

Telemetry fixes: cached tokens are read from input_tokens_details.cached_tokens into gen_ai.usage.cache_read.input_tokens without double-counting input; finish reasons are derived from Responses output/status (function calls before completed); tool arguments on spans are parsed objects, not raw JSON strings.

Breaking: root span name is invoke_agent; prompts/completions on spans only when capture_content=True (including openai_messages popping that kwarg so it does not hit config()).

^{Reviewed by Cursor Bugbot for commit 66bacd0. Bugbot is set up for automated code reviews on this repo. Configure here.}

apucacao · 2026-08-11T20:20:39Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 5b44877. Configure here.}

apucacao · 2026-08-11T20:44:53Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit b2f5fa1. Configure here.}

apucacao · 2026-08-11T21:02:19Z

bugbot run

apucacao · 2026-08-11T21:19:31Z

bugbot run

apucacao · 2026-08-11T21:46:43Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

1 issue from previous review remains unresolved.

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 910e8a7. Configure here.}

apucacao · 2026-08-12T17:23:31Z

bugbot run

apucacao · 2026-08-12T17:46:35Z

bugbot run

apucacao · 2026-08-12T18:03:35Z

bugbot run

apucacao · 2026-08-12T18:08:40Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 2c0c94e. Configure here.}

apucacao · 2026-08-14T19:30:58Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 35b2989. Configure here.}

One flat span named openai.response becomes the tree the TypeScript SDK emits: an invoke_agent root, one `chat {model}` child per model turn, one `execute_tool {name}` child per tool call. BREAKING CHANGE: the span this handler emits is renamed from `openai.response` and `openai.response.stream` to `invoke_agent`. Queries selecting on the old names will not match. Prompt and completion content is no longer on spans unless the caller passes capture_content=True. Cached tokens were absent entirely. OpenAI reports them under input_tokens_details.cached_tokens, which nothing here read, so every span understated what a prompt-cached call actually reused. They are now reported in gen_ai.usage.cache_read.input_tokens and, unlike Anthropic, not added on top of the input figure: OpenAI already counts them inside it, and adding them would double-count. Cache creation is always zero, because OpenAI has no such concept. This handler is the only one of the six that reports the model which actually answered rather than the one requested, on both the root and the chat spans. OpenAI resolves an alias like gpt-4o to a dated snapshot, and this handler has the resolved value to hand. Finish reasons are derived, not mapped. The Responses API has no finish_reason field, so the shared mapping table does not apply and is deliberately not imported. The value comes from a closed three-way check: a function call in the output means tool_calls, an incomplete status means length or content_filter depending on the reported cause, a completed status means stop, and anything else writes no attribute at all. The function-call check comes first because status alone reports completed for a turn that stopped to call a tool. The streaming path gets a finally, so a consumer that breaks out of the iteration no longer leaves the root span unended and unexported, taking the whole run out of AI Config Monitoring along with the feature_flag event it carries. Tests: 62 to 80.

…apper The wrapper never passed capture_content to the factory, so it stayed in kwargs and reached config(), which takes no such argument. A caller asking for content on spans got a TypeError rather than content. Lifted out alongside variables, which was already handled the same way and for the same reason: one configures the handler, the other belongs to the invocation, and config() accepts neither. Two tests, one per branch, asserting the flag reaches the factory and does not reach config(). Found by Bugbot on #33 against openai-agents. Five of the six wrappers had it; each is fixed in its own layer.

… its span The success-side content write and the span finish sat outside the try, so a raise while recording the result skipped both the finish and the failure path. The tool span was never ended, so the exporter never saw it: the run showed a root marked ERROR and no sign the tool had been called. Reachable rather than theoretical. Serialising a tool result raises TypeError whenever capture_content is on and the result is not JSON-serialisable, which is any object a handler happens to return. Inherited from the claude-messages handler this one was modelled on, which had it in the wrong place. The TypeScript handlers have always done this inside the try. Found by Bugbot on #34.

…t ends it The content writes on both sides of the provider call sat outside the try that fails the chat span, so a raise while serialising conversation content failed only the root. The chat span was never ended and never exported: a run showed an errored root with no sign a model call had happened. Reachable through capture_content, where serialising any non-JSON-serialisable value raises TypeError. The tool path in this same file already kept its serialisation inside the guard, which is what makes the model path's omission look accidental rather than considered. It was. Found by Bugbot on #32.

… reads Span construction moved to spans.py, which holds the real _HAS_OTEL. The handler kept its own copy, plus the two imports it needed, alive only by a noqa. Nothing read any of it. That mattered because the tests patched the dead one. 7 tests set handler._HAS_OTEL to False and believed they were exercising the install without the otel extra; the flag was unread, so they exercised nothing and passed either way. They now patch spans._HAS_OTEL, which is the flag start_root_span actually consults: with it patched, span creation returns None, and with it set it does not. Found by Bugbot on #32. Five of the six handlers carried the dead gate, and four had tests aimed at it.

The streaming finally closed the model span and the root, but the in-flight execute_tool span was held only by a local. except Exception does not see a CancelledError or a GeneratorExit, so a tool cancelled mid-flight left its span open and unexported: the trace showed a closed parent above a child that never arrived, which reads as a tool that is still running long after the run ended. Tracked in open_tool_span and abandoned in the finally, the same way the model span already was. The tracker is cleared on the two paths that end the span and deliberately not in a finally, because a finally would also clear it for the BaseException case, which is the one case where the outer finally is the only thing left to close it. Found by Bugbot on #32. openai-agents already did this through its hook object; three other handlers share the gap and are fixed in their own layers.

…ntent fails Moving the content write inside the span guard left the accounting behind it, so a raise while serialising a response dropped that turn from the run total. The provider had already billed the call. Failing to serialise its content is our problem, and it is not a reason to report the run as having spent less than it did: the root is the only span a config-scoped cost query can read the total from. The usage is taken and accumulated straight after the provider returns, before anything that can raise. Found by Bugbot on #32, reviewing the span-leak fix that introduced it.

The content write and the span finish sat after the try that fails the chat span, and the usage was accumulated last of all. A raise while serialising the response left the span for the finally to end as abandoned, which reads as a consumer who walked away rather than as the failure it was, and dropped a turn the provider had already billed. The blocking path in this same file already did both correctly, which is what made the streaming path's ordering look accidental rather than considered. It was. Two tests: the span is failed rather than abandoned, and the tokens survive. Found by auditing every handler for the ordering Bugbot reported on #30 and #34. This path had the same defect and had not been reported.

…string The Responses API sends function_call.arguments as an opaque JSON string. Every other handler puts a parsed object on a tool_call part, because Anthropic and LangChain hand over an object already. Passing the string through left the content carriers encoding it a second time, so a reader saw "arguments": "{\"q\":1}" on an OpenAI span and "arguments": {"q": 1} on an Anthropic span describing the same kind of call. The handler already parses this same string to call the tool, so only the span disagreed with the code beside it. The new helper parses a string and returns anything else untouched. A string that does not parse comes back verbatim rather than raising: a truncated stream is worth reporting as it arrived, and raising inside the telemetry path would end a run the provider has already billed. Applied at the four sites that build a tool_call part or write tool-call content. The shared content layer is untouched, per the call-site rule now written into TELEMETRY-CONTRACT.md. Two tests. Removing the helper fails one, removing its guard fails the other. Matches launchdarkly/js-ai-sdk#23. Found by Bugbot on this PR.

A timeout or a task.cancel() raises asyncio.CancelledError, which inherits from BaseException, so it walks past every except Exception this handler has. The blocking path ended its spans only from those clauses, so a cancelled run exported nothing at all. Not a wrong attribute: no span. The root carries the feature_flag event and every launchdarkly.* attribute, so the whole run vanished from AI Config Monitoring rather than showing as incomplete. Two finally blocks now own the ends the except clauses cannot reach: one around each provider turn, for its chat span, and one in the caller for the root and any tool span left open mid call. This is the shape the streaming path in this same handler has had since the earlier rounds, so both paths now agree. Open spans are tracked by clearing a local when a path ends one, rather than by asking the span. A mock span answers is_recording() truthily and the test suite here is built on mock spans, so asking would have made the finally fire a second end on every successful run. A cancelled root still reports the spend of the turns that completed, for the same reason the failure path does: those turns were billed. Spans are left at UNSET and marked launchdarkly.run.cancelled. Nothing failed, the caller went away. Two tests, driving a real task.cancel() against a provider call that never returns. Gutting either finally fails both. Found by Bugbot on the langchain-messages layer, then found here by audit.

…ndoned A CancelledError never enters except Exception, so the streaming teardown always ran its abandonment path and marked launchdarkly.stream.abandoned. A consumer that stops reading did abandon the stream, and that word is right for it. A timeout did not: nothing chose to stop reading, the run was cancelled underneath the consumer. The blocking path in this handler already reports launchdarkly.run.cancelled for that, so the two paths disagreed about the same event. The existing test for a tool cancelled mid-flight asserted stream.abandoned, which is what the defect looked like from inside. It now asserts run.cancelled. The consumer-break test still asserts stream.abandoned and needed no change. No new attribute. Both keys already exist and are in the vocabulary lock. Found by Bugbot on the openai-agents layer, then found here by audit.

apucacao · 2026-08-14T19:40:37Z

bugbot run

set_response_output_content recorded every Responses output item, including those whose parts list came back empty. output_item_parts returns nothing for a reasoning item with no summary text, which is what an encrypted reasoning item looks like from here, and those are common. An empty message still takes a slot. It landed at gen_ai.completion.0, which is the index the LaunchDarkly trace view renders, so a reader saw a blank answer with the real one sitting at completion.1 where nothing looks for it. The canonical carrier listed the empty message too. Only reachable with capture_content on, which is the setting whose entire purpose is making the transcript readable. One test, an encrypted-style reasoning item in front of a real answer. Removing the filter fails it. Found by Bugbot on this PR.

apucacao · 2026-08-14T19:57:26Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 66bacd0. Configure here.}

… reads Span construction moved to spans.py, which holds the real _HAS_OTEL. The handler kept its own copy, plus the two imports it needed, alive only by a noqa. Nothing read any of it. That mattered because the tests patched the dead one. 7 tests set handler._HAS_OTEL to False and believed they were exercising the install without the otel extra; the flag was unread, so they exercised nothing and passed either way. They now patch spans._HAS_OTEL, which is the flag start_root_span actually consults: with it patched, span creation returns None, and with it set it does not. Found by Bugbot on #32. Five of the six handlers carried the dead gate, and four had tests aimed at it.

…guard The output content write and the span finish sat outside the try that fails the chat span, and the blocking path has no finally that could recover it. A raise while serialising the completion left the span open and unexported, so the trace showed an errored root with no model call at all, and the turn was dropped from the run total even though Anthropic had already billed it. Reachable through capture_content, where serialising any non-JSON-serialisable value raises TypeError. The usage is now accumulated straight after the provider returns, before anything that can raise, and every span write happens inside the guard. Two tests: the span still ends and reports ERROR, and the tokens survive. Found while checking whether the openai-messages defect Bugbot reported on #32 reached the other handlers. It did.

…eads Span construction moved to spans.py, which holds the real _HAS_OTEL. The handler kept its own copy, plus the two imports it needed, alive only by a noqa. Nothing read any of it. That mattered because the tests patched the dead one. 3 tests set handler._HAS_OTEL to False and believed they were exercising the install without the otel extra; the flag was unread, so they exercised nothing and passed either way. They now patch spans._HAS_OTEL, which is the flag start_root_span actually consults: with it patched, span creation returns None, and with it set it does not. Found by Bugbot on #32. Five of the six handlers carried the dead gate, and four had tests aimed at it.

…ger reads Span construction moved to spans.py, which holds the real _HAS_OTEL. The handler kept its own copy, plus the two imports it needed, alive only by a noqa. Nothing read any of it. No tests aimed at this one, so only the dead code goes. handler._HAS_OTEL to False and believed they were exercising the install without the otel extra; the flag was unread, so they exercised nothing and passed either way. They now patch spans._HAS_OTEL, which is the flag start_root_span actually consults: with it patched, span creation returns None, and with it set it does not. Found by Bugbot on #32. Five of the six handlers carried the dead gate, and four had tests aimed at it.

…guard The output content write and the span finish sat outside the try that fails the chat span, and this path has no finally that could recover it. A raise while serialising the parsed object left the span open and unexported, and dropped the turn from the run total even though the provider had already billed it. Reachable through capture_content with any parsed object json.dumps refuses. The usage is now accumulated straight after the provider returns, before anything that can raise, and every span write happens inside the guard. Two tests: the span still ends and reports ERROR, and the tokens survive. Found while checking whether the openai-messages defect Bugbot reported on #32 reached the other handlers. It reached three of them.

…r reads Span construction moved to spans.py, which holds the real _HAS_OTEL. The handler kept its own copy, plus the two imports it needed, alive only by a noqa. Nothing read any of it. That mattered because the tests patched the dead one. 7 tests set handler._HAS_OTEL to False and believed they were exercising the install without the otel extra; the flag was unread, so they exercised nothing and passed either way. They now patch spans._HAS_OTEL, which is the flag start_root_span actually consults: with it patched, span creation returns None, and with it set it does not. Found by Bugbot on #32. Five of the six handlers carried the dead gate, and four had tests aimed at it.

apucacao mentioned this pull request Aug 11, 2026

feat(openai-agents)!: emit invoke_agent, chat and execute_tool spans #33

Merged

cursor Bot reviewed Aug 11, 2026

View reviewed changes

apucacao force-pushed the ag/py-telemetry-openai-messages branch from 5b44877 to b2f5fa1 Compare August 11, 2026 20:43

cursor Bot reviewed Aug 11, 2026

View reviewed changes

apucacao force-pushed the ag/py-telemetry-openai-messages branch from b2f5fa1 to 77d1074 Compare August 11, 2026 21:01

cursor Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/openai-messages/src/launchdarkly_ai_openai_messages/handler.py

apucacao force-pushed the ag/py-telemetry-openai-messages branch from 77d1074 to 910e8a7 Compare August 11, 2026 21:18

cursor Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/openai-messages/src/launchdarkly_ai_openai_messages/handler.py Outdated

cursor Bot reviewed Aug 11, 2026

View reviewed changes

apucacao force-pushed the ag/py-telemetry-openai-messages branch from 910e8a7 to 43fc252 Compare August 12, 2026 17:23

cursor Bot reviewed Aug 12, 2026

View reviewed changes

Comment thread packages/openai-messages/src/launchdarkly_ai_openai_messages/handler.py Outdated

apucacao force-pushed the ag/py-telemetry-openai-messages branch from 43fc252 to ba774df Compare August 12, 2026 17:45

cursor Bot reviewed Aug 12, 2026

View reviewed changes

Comment thread packages/openai-messages/src/launchdarkly_ai_openai_messages/handler.py Outdated

apucacao force-pushed the ag/py-telemetry-openai-messages branch from ba774df to 2c0c94e Compare August 12, 2026 18:07

cursor Bot reviewed Aug 12, 2026

View reviewed changes

Comment thread packages/openai-messages/src/launchdarkly_ai_openai_messages/handler.py

cursor Bot reviewed Aug 12, 2026

View reviewed changes

apucacao force-pushed the ag/py-telemetry-openai-messages branch from 2c0c94e to 440799c Compare August 12, 2026 18:20

cursor Bot reviewed Aug 14, 2026

View reviewed changes

apucacao added 11 commits August 14, 2026 15:38

apucacao force-pushed the ag/py-telemetry-openai-messages branch from 35b2989 to 76c4cef Compare August 14, 2026 19:39

cursor Bot reviewed Aug 14, 2026

View reviewed changes

Comment thread packages/openai-messages/src/launchdarkly_ai_openai_messages/spans.py

cursor Bot reviewed Aug 14, 2026

View reviewed changes

andrewklatzke approved these changes Aug 14, 2026

View reviewed changes

apucacao marked this pull request as ready for review August 14, 2026 21:18

apucacao merged commit 6a82ef5 into main Aug 14, 2026
7 checks passed

apucacao deleted the ag/py-telemetry-openai-messages branch August 14, 2026 21:20

Uh oh!

Conversation

apucacao commented Aug 11, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Cached tokens were absent entirely

Two things specific to this handler

Other changes

Breaking change

Where this sits

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 14, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 14, 2026

Uh oh!

Uh oh!

apucacao commented Aug 14, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

apucacao commented Aug 11, 2026 •

edited by cursor Bot

Loading