feat(langchain-agents)!: emit invoke_agent, chat and execute_tool spans by apucacao · Pull Request #35 · launchdarkly/python-ai-sdk

apucacao · 2026-08-11T20:13:29Z

Replaces one flat span per call with the tree the TypeScript SDK emits, for langchain-agents. This is the last of the six handlers.

invoke_agent                     one per call, carries the LD identity and the run total
├── chat {model}                 one per model turn, with that turn's own tokens
└── execute_tool {name}          one per tool call, a sibling of chat

The per-turn data was already being summed from each message's usage_metadata; it now drives a span per turn.

The same two provider-attribute fixes as #34

gen_ai.system is now the literal string langchain rather than the configured provider name: the key names the instrumentation, and here that is the framework.

gen_ai.provider.name is new and is a binary choice rather than a passthrough. It names who served the model, so it follows the client actually instantiated: anthropic when the config says so, openai for everything else, including Bedrock, Azure and an unset value. That mirrors the handler's own model resolution.

Other changes

Cached tokens are read from usage_metadata.input_token_details and reported per turn, not added to the input figure, which LangChain already reports inclusive of them.
Finish reasons go through the shared LangChain helper rather than being dropped, so a turn that stopped to call a tool is distinguishable from one that finished.
The streaming path gets a finally, so a consumer that stops reading no longer leaves the root span unended and unexported.

The graph span is untouched. ld.ai.graph and its two attributes already matched the TypeScript SDK and are out of scope.

Breaking change

The span is renamed from langchain.agent to invoke_agent. Queries selecting on the old name will not match. gen_ai.system changes value, as above. Prompt and completion content is no longer on spans unless the caller passes capture_content=True.

Where this sits

Needs the usage layer (#28) and the content layer (#29). Independent of the other five handler PRs; the stack orders them only because gh stack is linear. The oracle above (#36) needs all six.

Tests: 824 to 844.

Note

Overview
Replaces the single flat langchain.agent span with the same tree as the TypeScript SDK: an invoke_agent root (LaunchDarkly identity and run-level tokens), chat {model} per model turn, and execute_tool {name} per tool call as siblings under the root. Span lifecycle is driven by a new spans.py module and LangChain AsyncCallbackHandler hooks wired into ainvoke / astream.

Provider attributes now set gen_ai.system to langchain and gen_ai.provider.name to anthropic or openai (binary, matching which chat client is instantiated), not the configured provider string.

Usage and content: run totals reconcile message usage_metadata with callback llm_output.token_usage; per-turn usage, finish reasons, and cache token fields follow shared helpers. Prompt/completion on spans is opt-in via capture_content=True on the handler and langchain_agents() (no longer always on spans).

Teardown: blocking and streaming paths use finally / end_span_once so cancellation, consumer abandonment, and errors still end and export spans; abandonment vs cancellation use launchdarkly.stream.abandoned vs launchdarkly.run.cancelled without treating early stop as ERROR.

Breaking: span name langchain.agent → invoke_agent, gen_ai.system value change, and content off by default.

^{Reviewed by Cursor Bugbot for commit 1f4242b. Bugbot is set up for automated code reviews on this repo. Configure here.}

apucacao · 2026-08-11T20:20:46Z

bugbot run

apucacao · 2026-08-11T20:44:59Z

bugbot run

apucacao · 2026-08-11T21:02:22Z

bugbot run

apucacao · 2026-08-11T21:19:34Z

bugbot run

apucacao · 2026-08-11T21:46:46Z

bugbot run

apucacao · 2026-08-12T17:23:34Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c643fe3. Configure here.}

apucacao · 2026-08-12T17:46:40Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 5d7d073. Configure here.}

apucacao · 2026-08-12T18:03:38Z

bugbot run

apucacao · 2026-08-14T19:31:02Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0c848f0. Configure here.}

apucacao · 2026-08-14T19:40:41Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 48a752b. Configure here.}

One flat span named langchain.agent becomes the tree the TypeScript SDK emits: an invoke_agent root, one `chat {model}` child per model turn, one `execute_tool {name}` child per tool call. The per-turn data was already being summed from each message's usage_metadata; it now drives a span per turn. BREAKING CHANGE: the span this handler emits is renamed from `langchain.agent` and `langchain.agent.stream` to `invoke_agent`. Queries selecting on the old names will not match. Prompt and completion content is no longer on spans unless the caller passes capture_content=True. gen_ai.system changes value; see below. gen_ai.system is now the literal string langchain rather than the configured provider name, matching the TypeScript SDK: the key names the instrumentation, and here that is the framework. gen_ai.provider.name is new and is a binary choice rather than a passthrough. It names who served the model, so it follows the client actually instantiated: anthropic when the config says so, openai for everything else, including Bedrock, Azure and an unset value. That mirrors the handler's own model resolution. Cached tokens are now read from usage_metadata.input_token_details and reported per turn. They are not added to the input figure, which LangChain already reports inclusive of them. Finish reasons go through the shared LangChain helper rather than being dropped, so a turn that stopped to call a tool is distinguishable from one that finished. The streaming path gets a finally, so a consumer that stops reading no longer leaves the root span unended and unexported. The graph span is untouched. ld.ai.graph and its two attributes already matched the TypeScript SDK and are out of scope for this change. Tests: 54 to 74.

…oes not fail them The abandonment path reused close_open_spans, which records a synthetic exception and sets ERROR on every span still open, so an early consumer stop was indistinguishable from a provider failure in a trace. The comment three lines above it already claimed the opposite. Adds abandon_open_spans, mirroring the method openai-agents already had: every open span ends through end_span_once, staying UNSET and carrying launchdarkly.stream.abandoned. The failure path keeps its behaviour. Tested on the callback handler directly rather than through the streaming path. Reaching the state that matters, a chat or tool span still open at the break, needs a fake model that yields mid-turn, and with the fixtures here LangGraph has already run every callback by the time the first chunk reaches the consumer. My first attempt went through stream() and passed whether or not the fix was present, which is worse than no test. The test file says so, so the next person does not repeat it. Found by Bugbot on #35.

…rapper The wrapper never passed capture_content to the factory, so it stayed in kwargs and reached config(), which takes no such argument. A caller asking for content on spans got a TypeError rather than content. Lifted out alongside variables, which was already handled the same way and for the same reason: one configures the handler, the other belongs to the invocation, and config() accepts neither. Two tests, one per branch, asserting the flag reaches the factory and does not reach config(). Found by Bugbot on #33 against openai-agents. Five of the six wrappers had it; each is fixed in its own layer.

…nreachable Two mirror-image leaks in the callback handler, both reachable through content serialisation, which raises on any tool argument or result that is not JSON-serialisable. The end callbacks popped the span before doing that work. After the pop nothing else can reach it, so close_open_spans could not recover it and the span was never ended: the exporter never saw the turn or the tool call at all. Both now end it on the way out. on_tool_start had the reverse problem: it created the span, wrote the arguments, and only then inserted it into the tracking dict. A raise in between left a span no cleanup path knew about. It is now tracked first, so every later path can still close it. Two tests, each failing on the exact leak when the fix is reverted. The tracer patch has to stay active while the callbacks run rather than only while they are built, which is what my first attempt got wrong. Found by Bugbot on #35.

…t can raise _start_model created the span, wrote the conversation onto it, and only then inserted it into the tracking dict. Serialising conversation content raises on anything that is not JSON-serialisable, and a span created but never inserted is unreachable by close_open_spans, abandon_open_spans and the end callbacks alike: it never ends, so the exporter never sees it. on_tool_start already had this fix. The model-start path is the mirror of it and did not. Found by Bugbot on #35.

…r reads Span construction moved to spans.py, which holds the real _HAS_OTEL. The handler kept its own copy, plus the two imports it needed, alive only by a noqa. Nothing read any of it. That mattered because the tests patched the dead one. 7 tests set handler._HAS_OTEL to False and believed they were exercising the install without the otel extra; the flag was unread, so they exercised nothing and passed either way. They now patch spans._HAS_OTEL, which is the flag start_root_span actually consults: with it patched, span creation returns None, and with it set it does not. Found by Bugbot on #32. Five of the six handlers carried the dead gate, and four had tests aimed at it.

extract_llm_usage read the llm_output fallback with `or`, so a genuine 0 was skipped in favour of the next key. With both counts at zero the bag came back all None, lang_chain_span_usage read that as the provider having said nothing, and the run went unreported: a turn that completed and cost nothing became indistinguishable from one that never reported, which is the distinction the reported flag exists to preserve. Now keyed on presence rather than truthiness. Three tests: a zero prompt count survives, both-zero still counts as reported, and a genuinely absent count is still absent. Found by Bugbot on #35.

…wn spans The success path sums usage_metadata off each message. The callbacks read the whole LLMResult and fall back to llm_output.token_usage, which some providers use instead. For those providers the chat spans carried real tokens while the successful run's root, and the bag handed back to the caller, both stayed at zero: a config-scoped cost query undercounted completed work, and the two figures in one trace contradicted each other. The message-level sum stays authoritative wherever it has anything to say, so a provider that reports in both places cannot be counted twice. Only when it saw nothing at all do the callbacks stand in, because then they are the only record of what the run cost. The docstring claimed both sides computed the same numbers. They did not, and it now says which fields each one reads. Two tests: llm_output-only usage reaches the root, and usage_metadata still wins when both are present. Found by Bugbot on #35.

…rite fails on_llm_end accumulated the turn's usage after the content write. A raise while serialising completion content dropped a turn the provider had already billed, and that accumulator is what a failed run's root reports and what a successful run falls back to when the messages carry no usage of their own. The accumulation now happens before the write, matching what the other five handlers do. Found by auditing every handler for the ordering Bugbot reported on #30 and #34.

The prompt write ran before the try that fails the root, so a raise while serialising it left the root open: never ended, never exported, so the run disappeared from AI Config Monitoring along with the feature_flag event it carries. Both paths had it. Two tests, one per path. Found by Bugbot on #34, which is this shape in langchain-messages. All five handlers that had it are fixed in their own layers.

A timeout or a task.cancel() raises asyncio.CancelledError, which inherits from BaseException, so it walks past every except Exception this handler has. The blocking path ended its root span, and the chat and execute_tool spans the callback handler tracks, only from those clauses. A cancelled run exported nothing at all. The root carries the feature_flag event and every launchdarkly.* attribute, so the whole run vanished from AI Config Monitoring rather than showing as incomplete. A finally in _call_impl now owns the ends the except clause cannot reach. The root is tracked by clearing a local when a path ends it, the same shape as claude-messages, since asking the span via is_recording() would make the finally fire a second end on every successful run: the test double is a mock span and answers truthily. The chat and execute_tool spans are tracked differently here, in the callback handler's own dicts (LangGraph drives the provider call, not this handler, so a callback is the only hook into a turn's lifecycle), and are ended the same way close_open_spans already ends them on a failure: a new cancel_open_spans method on that handler, calling the shared end_unfinished_spans helper so a cancelled turn is marked launchdarkly.run.cancelled and left at UNSET, not the launchdarkly.stream.abandoned a stopped stream gets. A cancelled root still reports the spend of the turns that completed, for the same reason the failure path does: those turns were billed. Two tests drive a real task.cancel() against a chat model that never returns. Gutting the finally fails both. Found by Bugbot on the langchain-messages layer, then found here by audit.

…andoned A CancelledError never enters except Exception, so the streaming teardown always ran its abandonment path and marked launchdarkly.stream.abandoned. A consumer that stops reading abandoned the stream, and that word is right for it. A CancelledError is not a choice: something cancelled the run, usually a timeout, and the consumer was still reading. The blocking path already reports launchdarkly.run.cancelled for that, so the two paths disagreed about the same event. The SpanCallbacks wrapper took the new flag and dropped it, which would have left a cancelled run's tool spans saying abandoned under a root saying cancelled. It now forwards it. Two tests. One cancels a draining consumer while the model is mid-await, which is the shape a timeout has: suspending in the consumer's loop body instead unwinds as a GeneratorExit, which is abandonment and would test the wrong thing. The other calls the wrapper directly, because the first opens no tool span and so cannot see that line at all. The existing abandonment test breaks out of the loop and still asserts stream.abandoned, and needed no change. No new attribute. Both keys already exist and are in the vocabulary lock. Found by Bugbot on the openai-agents layer, then found here by audit.

…en the messages report nothing The streaming success path built its run total only from usage_metadata on the astream payloads. That walk expects per-step message updates, and sees nothing at all when the graph yields value snapshots, which is LangGraph's default. It also sees nothing from a provider that reports in llm_output.token_usage rather than on the message. A finished stream could then write zero on invoke_agent and in the done bag while its own chat spans held the tokens the provider had already billed. A root that says a run cost nothing, above children that say otherwise, is the one outcome neither figure can be right about, and the root is the span a config-scoped cost query finds. The blocking path has reconciled the two sources since an earlier round, with a comment explaining exactly this. The streaming path was the half that was missed. One test, driving a model that reports only in llm_output. Removing the fallback fails it. Found by Bugbot on this PR.

…an does The root's completion was taken from `output`, which is deliberately blank whenever the last message's content is a list of blocks, because `output` is also what this function returns to the caller. Chat models routinely reply in content blocks. The root then recorded an empty completion while its own chat child, which converts through lang_chain_span_messages, held the real text. Two spans, one reply, two different stories. The root now uses the same conversion as the child, so they cannot disagree. The string case is unchanged, and the fallback for a run with no messages at all is kept. The blank return value is a separate question. It predates this work and is not telemetry, so it stays as it is rather than changing what a caller receives. One test, a reply whose content is a typed block. Reverting to `output` fails it. Found by Bugbot on this PR.

apucacao · 2026-08-14T19:57:30Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 1f4242b. Configure here.}

apucacao mentioned this pull request Aug 11, 2026

test: fail the build when the six handlers drift apart again #36

Merged

cursor Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/langchain-agents/src/launchdarkly_ai_langchain_agents/handler.py Outdated

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 082eaaf to 9ccef53 Compare August 11, 2026 20:43

cursor Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/langchain-agents/src/launchdarkly_ai_langchain_agents/spans.py Outdated

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 9ccef53 to 326b8a6 Compare August 11, 2026 21:01

cursor Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/langchain-agents/src/launchdarkly_ai_langchain_agents/spans.py Outdated

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 326b8a6 to 65fc5f8 Compare August 11, 2026 21:18

cursor Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/langchain-agents/src/launchdarkly_ai_langchain_agents/spans.py Outdated

cursor Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/langchain-agents/src/launchdarkly_ai_langchain_agents/spans.py

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 65fc5f8 to c643fe3 Compare August 12, 2026 17:23

cursor Bot reviewed Aug 12, 2026

View reviewed changes

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from c643fe3 to 5d7d073 Compare August 12, 2026 17:45

cursor Bot reviewed Aug 12, 2026

View reviewed changes

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 5d7d073 to 0ec7a76 Compare August 12, 2026 18:02

cursor Bot reviewed Aug 12, 2026

View reviewed changes

Comment thread packages/langchain-agents/src/launchdarkly_ai_langchain_agents/spans.py

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 0ec7a76 to dfedf11 Compare August 12, 2026 18:07

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 0762796 to 0c848f0 Compare August 14, 2026 19:30

cursor Bot reviewed Aug 14, 2026

View reviewed changes

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 0c848f0 to 48a752b Compare August 14, 2026 19:39

cursor Bot reviewed Aug 14, 2026

View reviewed changes

apucacao added 14 commits August 14, 2026 15:55

apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 48a752b to 1f4242b Compare August 14, 2026 19:56

cursor Bot reviewed Aug 14, 2026

View reviewed changes

andrewklatzke approved these changes Aug 14, 2026

View reviewed changes

apucacao marked this pull request as ready for review August 14, 2026 21:18

apucacao merged commit 6a82ef5 into main Aug 14, 2026
12 checks passed

apucacao deleted the ag/py-telemetry-langchain-agents branch August 14, 2026 21:20

Uh oh!

Conversation

apucacao commented Aug 11, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The same two provider-attribute fixes as #34

Other changes

Breaking change

Where this sits

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

Uh oh!

apucacao commented Aug 11, 2026

Uh oh!

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 12, 2026

Uh oh!

Uh oh!

apucacao commented Aug 14, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 14, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

apucacao commented Aug 14, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

apucacao commented Aug 11, 2026 •

edited by cursor Bot

Loading