Telemetry: one tools-command record per invocation with CLI toolset and tool names, plus docs show-story reporting

[kasperpeulen]

Problem Statement

Telemetry for agent tool calls uses two different meanings of "toolset". The addon-mcp event reports the MCP grouping (dev, docs, test), while the CLI and SDK expose toolsets named stories, review, docs, and test. The tools-command event copies the MCP grouping, so a CLI call to review create is logged as toolset: dev, and the CLI toolset and tool names appear nowhere except inside a command string.

On top of that, one CLI invocation writes two tools-command records: one from the command wrapper (command, success, duration, attach fields) and one from the method handler (event, toolset, counters), with no link between them. Counting rows double counts every successful command.

Finally, docs show-story never reports telemetry on any surface. Its predecessor in @storybook/mcp never had a telemetry hook, and the port preserved that gap.

Solution

Every surface reports the same vocabulary: the CLI toolset and tool names, a generated event derived from them (tool:stories_changed), and the handler's payload. CLI and SDK records become one record per invocation and drop the command string since it is derivable from the two parts. addon-mcp records gain tool and switch toolset to the CLI name, but keep their pre-toolset event names (tool:getChangedStories) so MCP data stays continuous across versions. The show-story method gains a telemetry report on both surfaces.

A handler no longer sends telemetry and never names an event. It returns a payload as part of its outcome, next to data and markdown. The toolset layer that runs the handler completes that into one report: toolset and tool from where the method is registered, a generated event (tool:<toolset>_<method>), and the payload. The CLI and SDK forward that report as is and add their run fields. The MCP adapter forwards it too, adds its session fields, and replaces event with the pre-toolset name it keeps per tool in the addon's own registry; the toolsets know nothing of those names.

Before and after: CLI npx storybook tools stories changed

Field	Before, record 1	Before, record 2	After, single record
command	stories changed
event		tool:getChangedStories	tool:stories_changed
toolset		dev	stories
tool			changed
success	true		true
outcome	success		success
duration	312		312
client	cli	cli	cli
requestedMode	auto	auto	auto
resolvedMode	local	local	local
attachMode	local	local	local
host	in-process	in-process	in-process
storyCount		4	4
newStoryCount		1	1
modifiedStoryCount		3	3
affectedStoryCount		0	0

The same applies to every other method. The SDK produces the same record with client: sdk.

Before and after: CLI intercepts

Case	Before	After
unknown tool stories nope	command: stories nope	toolset: stories, tool: nope
garbage name ../x list	command: (invalid) list	toolset: (invalid), tool: list
attach gate, nothing parsed	command: (none)	no toolset, no tool

Before and after: MCP stories-changed

Field	Before	After
event	tool:getChangedStories	tool:getChangedStories
toolset	dev	stories
tool		changed
payload and session fields	unchanged	unchanged

docs and test keep their toolset value; only the dev grouping splits into stories and review.

New: show-story on both surfaces

Field	CLI	MCP
event	tool:docs_showStory	tool:getDocumentationForStory
toolset	docs	docs
tool	show-story	show-story
found	true	true
storyId	button--primary	button--primary
lookup	storyId	storyId
resultTokenCount	412	412

The CLI row also carries success, duration, and the attach fields.

User Stories

1. As a product lead building dashboards, I want every CLI tools-command record to carry toolset and tool in the CLI's own names, so that I can group agentic usage by stories, review, docs, and test without parsing strings.
2. As a product lead, I want one tools-command record per CLI invocation, so that counting records counts invocations.
3. As a product lead, I want the handler's payload (story counts, test results, token counts) on the same record as success and duration, so that I can correlate outcome with payload in one query.
4. As a product lead, I want CLI and SDK records to carry an event generated from toolset and tool, so that a new tool gets a consistent name without anyone inventing one.
5. As a product lead, I want addon-mcp records to keep their pre-toolset event names and payload and gain the same toolset and tool as CLI records, so that one query groups usage across surfaces and MCP data still joins to pre-10.6.1 data on event.
6. As a product lead, I want docs show-story reported on both surfaces with found, storyId, lookup, and resultTokenCount, so that the docs toolset is fully observable.
7. As a product lead, I want to see which input shape agents use for show-story, so that I can decide whether to keep both shapes.
8. As a product lead, I want intercepts and attach-gate failures to keep producing one record without payload, so that success rates stay comparable with the current release.
9. As a product lead, I want intercept records to carry the sanitized toolset and tool the agent typed, so that I can see which unknown names agents reach for.
10. As a product lead, I want SDK invocations to carry the same fields as CLI ones with client: sdk, so that the two clients are comparable.
11. As a product lead, I want a CLI call that fails inside the handler to still produce one record with success: false, so that failures are never lost.
12. As a maintainer, I want the toolset name to live only in the registry id, so that a method can never report a name that disagrees with where it is registered.
13. As a maintainer adding a new toolset method, I want toolset and tool filled in automatically on the CLI and SDK, so that I cannot forget them.
14. As a maintainer, I want the pre-toolset event names to live only in the addon-mcp registry, so that toolsets and the CLI never learn them and they can be dropped in one place later.
15. As a maintainer, I want a handler to return its report as data on its outcome, so that reporting cannot happen twice, too late, or on a path the surface does not see.
16. As a maintainer, I want the report a handler outcome carries to already be in the CLI vocabulary, so that every surface forwards it without translating.
17. As a maintainer, I want a handler outcome from a child host to carry its report inside the outcome that already crosses IPC, so that attachMode does not change what gets logged and no separate telemetry message exists.
18. As an SDK consumer, I want the handler's report on the outcome that call returns, so that I read it there and no separate telemetry option exists.
19. As a maintainer reading a handler test, I want the telemetry assertion to be a plain equality on the returned outcome, so that I can see what the handler reports without knowing how any surface forwards it.
20. As a maintainer, I want telemetry failures never to affect the tool result, so that agents are unaffected by a slow endpoint.
21. As a maintainer, I want the change shipped as a 10.6.1 patch, so that the data break happens once, at the point the product lead accepted.
22. As an agent using the CLI, I want no change in output or exit codes, so that my workflows keep working.
23. As a Storybook user, I want no change in what MCP tools are available or how they respond, so that telemetry changes stay invisible to me.

Implementation Decisions

• Handler contract: a handler outcome gains an optional telemetry field holding payload (counts and flags). That field is the only way a handler reports. The per-call telemetry callback on the handler context and the helper that wrapped it are removed. Handlers never write event, toolset, or tool. A handler that throws has no outcome and therefore no report; that case is dropped on purpose.
• Toolset layer completes the report: the code that invokes a handler, shared by all three surfaces, fills toolset and tool from the toolset id and the method name, spelled as the CLI spells them, and generates event as tool:<toolsetId>_<methodName> with the method name as written in code (tool:stories_findByComponent). The outcome that reaches a surface carries one complete report: toolset, tool, event, payload. A new method gets all three names without writing anything.
• CLI and SDK forward: the invocation reporter in the CLI command wiring and in the SDK call spreads the outcome's report into the one tools-command record and adds success, outcome, duration, and the attach fields. No translation, no collector, no per-call sink.
• MCP adapter keeps the old event names: the toolset-backed MCP tool adapter sends the report's toolset, tool, and payload as the addon-mcp event, plus the session fields it already adds, with event set to the pre-toolset name (tool:getChangedStories) that the addon's tool registry declares per tool. That is the only place those names exist. The MCP grouping (dev, docs, test) stays what it is for the enable gate and the X-MCP-Toolsets header, and is no longer written to telemetry. The UI-instructions tool and session:initialized are untouched.
• command removed: tools-command no longer carries command. toolset and tool are the two parts the agent invoked, spelled as the CLI spells them (stories, find-by-component), each sanitized to (invalid) when not a name-shaped token, and absent when no part was parsed. The SDK derives both from the method ref.
• Child host: the outcome already crosses IPC as the call result, so the report travels with it. The separate telemetry message type and the pending-sink bookkeeping in the child client are removed.
• SDK telemetry option removed: the per-call telemetry option on the SDK is deleted. Its only caller was the CLI, which now reads the report from the outcome, and any other caller gets the same report back from call.
• No compatibility layer: the Node SDK has no external users yet, so its options and return shapes change freely; no shim, wrapper, or fallback is kept for an older SDK shape. The pre-toolset event names on addon-mcp records are the one concession to existing data.
• show-story report: event tool:getDocumentationForStory, payload found, storyId (the resolved id when found, otherwise the requested one), lookup set to storyId or name, and resultTokenCount from the rendered markdown, mirroring the sibling show report.
• Intercepts and gates: unchanged apart from the command replacement, one record without payload.
• Telemetry types: the tools-command payload type gains toolset, tool, event, and an open payload bag, and loses command.

Testing Decisions

• A good test asserts either the outcome a handler returns or the payload a surface hands to its mocked telemetry function. Nothing in between is asserted.
• Handlers carry most of the coverage: the stories, docs, review, and vitest definition tests call the method through the toolset layer and assert the complete report on the outcome with a plain equality: toolset, tool, event, payload. Add showStory cases for found, missing story, missing component, and both lookup shapes.
• CLI: the tools-command telemetry tests in the command wiring test have the mocked run return an outcome with a report and assert one tools-command call that is the report plus success, duration, and the attach fields. Intercept, placeholder, and help cases assert toolset and tool instead of command.
• SDK: the createTools test asserts one record with client: sdk and that the outcome returned by call carries the report. The child-client test asserts the report arrives inside the outcome from the child.
• MCP: one adapter test asserts that the addon-mcp payload is the tool's declared pre-toolset event, the report's toolset, tool, and payload, and one that an outcome without a report sends nothing.
• Prior art: the tools-command telemetry describe block in the command wiring test, the surface-owned-fields test in the MCP adapter test, and the telemetry describe block in the vitest toolset definition test.

Out of Scope

• Removing the X-MCP-Toolsets header and the toolsets disable option, planned for Storybook 11 as its own PR.
• Renaming the legacy event names to the current tool names.
• Adding fields to the skills-get event or the MCP UI-instructions tool.
• Adding an invocation-level record with success and duration to the addon-mcp event.
• Documentation changes; the docs list no telemetry events.

Further Notes

• Discussed with Michael and Jeppe on 7 September 2026. Michael accepted one data break at 10.6.1; Jeppe reviews the PR.
• The pre-toolset event names such as tool:getChangedStories derive from the pre-10.6 MCP tool names. They no longer match any tool name an agent sees and survive only on addon-mcp records, as a join key to older MCP data. CLI records from 10.6.0 carried them too; from 10.6.1 the CLI uses the generated names, accepted as part of the same data break.
• Dashboards keyed on command for 10.6.0 data lose that key; accepted.
• Revised 9 September 2026, third time, after Jeppe's review and a Slack thread with Michael: toolsets stop naming events; the event is generated from toolset and method on every surface, and only addon-mcp keeps the pre-toolset names, in its own registry. Kasper's proposal.
• Revised 9 September 2026, after a second conversation with Michael: addon-mcp records also move to the CLI toolset and tool names, since the legacy event keeps joining them to older data. This drops the MCP-side toolset override the previous revision kept for compatibility.
• Ships as a patch:yes PR against next.
• Revised 9 September 2026: the first version of this spec kept the per-call telemetry callback and had the CLI collect and merge the handler's report. That was implemented in Telemetry: One tools-command record per invocation with CLI toolset and tool names #36210, and the collector, the SDK sink wrapping, and the IPC telemetry message it required made the change hard to read. Handlers now return the report on the outcome instead. The records that reach telemetry are identical in both versions.