Skip to content

Failed Send task loses UntrackedValue input when resumed from checkpoint #8582

Description

Checked other resources

  • This is a bug/behavior report, not a usage question.
  • I searched the issue tracker for combinations of Send, UntrackedValue, retry/resume, and checkpoint recovery and did not find a matching report.
  • The reproduction is self-contained and uses InMemorySaver only.
  • Reproduced against my local checkout of current main (d56666f7f).

Reproduction Steps / Example Code

import operator
from typing import Annotated

from langgraph.channels.untracked_value import UntrackedValue
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.graph import START, StateGraph
from langgraph.types import Send
from typing_extensions import TypedDict


class RuntimeResource:
    def __init__(self, name: str):
        self.name = name


class State(TypedDict):
    messages: Annotated[list[str], operator.add]
    resource: Annotated[RuntimeResource, UntrackedValue]


attempts = 0


def setup(state: State):
    return {
        "messages": ["setup"],
        "resource": RuntimeResource("runtime-secret"),
    }


def route_to_worker(state: State):
    return [Send("worker", state)]


def worker(state: State):
    global attempts
    attempts += 1

    # Present on the first execution, missing after checkpoint resume.
    resource = state.get("resource")
    print("attempt", attempts, "resource", resource)

    if attempts == 1:
        assert resource is not None
        raise ValueError("intentional-worker-failure")

    assert resource is not None, "resource was lost on resume"
    return {"messages": [f"used {resource.name}"]}


builder = StateGraph(State)
builder.add_node("setup", setup)
builder.add_node("worker", worker)
builder.add_edge(START, "setup")
builder.add_conditional_edges("setup", route_to_worker)

graph = builder.compile(checkpointer=InMemorySaver())
config = {"configurable": {"thread_id": "send-untracked-resume"}}

try:
    graph.invoke({}, config)
except ValueError:
    pass

snapshot = graph.get_state(config)
print("checkpointed values:", snapshot.values)
print("next:", snapshot.next)

# `resource` is correctly absent because it is UntrackedValue.
assert "resource" not in snapshot.values
assert snapshot.next == ("worker",)

# The failed dynamic Send task is retried, but its original Send input
# can no longer be reconstructed because the UntrackedValue was not persisted.
graph.invoke(None, config)

Actual behavior

First execution:

attempt 1 resource <RuntimeResource ...>

After the injected failure:

checkpointed values: {'messages': ['setup']}
next: ('worker',)

On resume:

attempt 2 resource None
AssertionError: resource was lost on resume

The failed worker task is still considered resumable (state.next == ('worker',)), but the input it receives after resume is structurally different from the original Send input because the UntrackedValue is gone.

Expected behavior / contract question

I understand that UntrackedValue is intentionally excluded from checkpoints, so persistence of the value itself is not expected.

The surprising part is that LangGraph still retries the failed dynamic Send task on resume even though part of that task's original input can no longer be reconstructed.

I would expect one of the following contracts:

  1. LangGraph preserves enough pending Send task input to retry the task faithfully, even when that input contains an UntrackedValue; or
  2. LangGraph detects that the failed task depends on untracked input and does not silently retry it with a different input shape; or
  3. This behavior is documented as an intentional limitation of UntrackedValue with failed dynamic tasks.

Could maintainers confirm which behavior is intended?

Why this matters

UntrackedValue is useful for runtime-only resources such as sessions, clients, locks, or other non-serializable objects. Send is commonly used for dynamic fan-out. Combining the two works normally during the initial run, but after a task failure the same task is retried with different input, which can lead to secondary errors or different behavior after recovery.

If maintainers consider this a bug, I would be happy to add a focused regression test and work on a minimal fix. Please assign the issue to @Hello-world-Prakash if that direction is approved.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions