🛡️ fix: Guard HITL checkpoint size against MongoDB 16MB limit (danny-avila#14157)

* 🛡️ fix: Guard HITL checkpoint size against MongoDB 16MB limit

A LangGraph HITL checkpoint embeds the whole serialized message history
in a single BSON document, so a large conversation (inlined base64 media,
big tool outputs, long history) can serialize past MongoDB's 16MB
document ceiling. `MongoDBSaver.put` would then throw a raw
`BSONObjectTooLarge` at pause time and the pause would be lost with no
legible error.

`LazyMongoSaver` now measures the serialized checkpoint on the persist
path (rare HITL pauses only — the clean-exit common path is untouched):
debug-logs the size, warns past a soft 8MB threshold, and throws a typed
`CheckpointTooLargeError` before the doomed write past a 15MB hard limit
(16MB minus headroom for the document's other fields). Thresholds are
overridable via the constructor for testing.

Adds integration coverage (real serde + mongodb-memory-server) for the
under-threshold, soft-warn, and hard-reject cases.

* 🧱 fix: Add explicit types for isolatedDeclarations build

The production build (tsdown + rolldown-plugin-dts) compiles with
--isolatedDeclarations, which requires explicit type annotations on
exported bindings whose initializers it can't infer syntactically.
`CHECKPOINT_HARD_LIMIT_BYTES` (arithmetic over two consts) tripped
TS9010; annotate it and `CHECKPOINT_WARN_BYTES` as `number`. Verified
with `tsc --isolatedDeclarations` over the package.

* fix: Codex review — include metadata in the size guard + flush parked bookkeeping

P1 (lost bookkeeping): `put` consumes the write anchor, then AWAITS
assertCheckpointFitsDocument (checkpoint serialization). A bookkeeping-only
putWrites dispatched in that window sees neither the anchor nor persistedIds,
so it parks — and `put` never flushed it, dropping the marker (e.g. a
completed Send-sibling's __no_writes__) so a resume re-executed the sibling.
Extract flushBufferedBookkeeping (shared with the anchoring putWrites) and call
it after super.put in the persist path.

P2 (metadata ignored by guard): MongoDBSaver.put stores the serialized
checkpoint AND metadata (plus metadata_search) in the SAME document, but the
guard measured only the checkpoint — a just-under-limit checkpoint with large
metadata fell through to a raw BSONObjectTooLarge. Measure checkpoint +
metadata; the fixed headroom now only covers metadata_search/ids/framing.

Two integration regressions added (both fail without the fix, pass with it):
metadata-pushes-over-the-ceiling, and flush-during-the-serialization-window.

* fix: count metadata_search (raw metadata copy) in the checkpoint size guard

Codex follow-up: MongoDBSaver.put stores metadata a SECOND time as
`metadata_search: metadata` — the whole raw metadata object as a queryable BSON
subdocument in the same agent_checkpoints document. Measuring only checkpoint +
serialized metadata undercounted by that raw copy, so a large metadata.writes
payload could pass the 15 MB preflight while metadata_search pushed the actual
BSON past 16 MB — the raw BSONObjectTooLarge the guard exists to prevent.

Add mongoose.mongo.BSON.calculateObjectSize(metadata) for the metadata_search
contribution (mongoose already imported; no new dep). Headroom now only covers
ids + BSON framing. New integration test sizes a case where checkpoint +
serialized metadata is under the limit but the raw metadata_search copy pushes
it over — green (24/24).

* ci: run packages/api agents integration specs (checkpointer) in CI

The checkpointer.integration.spec.ts (durable HITL checkpointer vs a real
in-process MongoDB) is a *.integration.spec.ts, which test:ci deliberately
excludes — and cache-integration-tests.yml only covers cache/cluster/mcp/
stream, not src/agents/**. So it ran nowhere and its regressions guarded
nothing. Add:
- test:agents-integration script (jest over src/agents/*.integration.spec.ts,
  runInBand — mongodb-memory-server is in-process, no external service);
- a dedicated agents-integration-tests.yml (mirrors the proven build setup,
  no Redis) triggered on packages/api/src/agents/** changes;
- babel-plugin-replace-ts-export-assignment as a packages/api devDep: the spec
  imports @langchain/langgraph-checkpoint (whitelisted for babel transform,
  uses `export =`), whose transform needs this plugin — it was only present
  under client/node_modules, unresolvable from packages/api, so the suite
  couldn't load. 24/24 pass locally.

Author: danny-avila

.github/workflows/agents-integration-tests.yml

@@ -0,0 +1,90 @@
+name: Agents Integration Tests
+
+# Runs the packages/api `src/agents/**` integration specs (e.g. the durable HITL
+# checkpointer against a real in-process MongoDB via mongodb-memory-server). These
+# are `*.integration.spec.ts`, which `test:ci` deliberately excludes — without this
+# job they run nowhere and their regressions guard nothing.
+on:
+  pull_request:
+    branches:
+      - main
+      - dev
+      - dev-staging
+      - release/*
+    paths:
+      - 'packages/api/src/agents/**'
+      - 'packages/api/package.json'
+      - '.github/workflows/agents-integration-tests.yml'
+
+permissions:
+  contents: read
+
+jobs:
+  agents_integration_tests:
+    name: Integration Tests that use in-process MongoDB
+    timeout-minutes: 20
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Use Node.js 24.16.0
+        uses: actions/setup-node@v4
+        with:
+          node-version: '24.16.0'
+
+      - name: Restore node_modules cache
+        id: cache-node-modules
+        uses: actions/cache@v4
+        with:
+          path: |
+            node_modules
+            api/node_modules
+            packages/api/node_modules
+            packages/data-provider/node_modules
+            packages/data-schemas/node_modules
+          key: node-modules-backend-${{ runner.os }}-24.16.0-${{ hashFiles('package-lock.json') }}
+
+      - name: Install dependencies
+        if: steps.cache-node-modules.outputs.cache-hit != 'true'
+        run: npm ci
+
+      - name: Restore data-provider build cache
+        id: cache-data-provider
+        uses: actions/cache@v4
+        with:
+          path: packages/data-provider/dist
+          key: build-data-provider-${{ runner.os }}-${{ hashFiles('packages/data-provider/src/**', 'packages/data-provider/tsconfig*.json', 'packages/data-provider/tsdown.config.mjs', 'packages/data-provider/package.json') }}
+
+      - name: Build data-provider
+        if: steps.cache-data-provider.outputs.cache-hit != 'true'
+        run: npm run build:data-provider
+
+      - name: Restore data-schemas build cache
+        id: cache-data-schemas
+        uses: actions/cache@v4
+        with:
+          path: packages/data-schemas/dist
+          key: build-data-schemas-${{ runner.os }}-${{ hashFiles('packages/data-schemas/src/**', 'packages/data-schemas/tsconfig*.json', 'packages/data-schemas/tsdown.config.mjs', 'packages/data-schemas/package.json', 'packages/data-provider/src/**', 'packages/data-provider/tsconfig*.json', 'packages/data-provider/tsdown.config.mjs', 'packages/data-provider/package.json') }}
+
+      - name: Build data-schemas
+        if: steps.cache-data-schemas.outputs.cache-hit != 'true'
+        run: npm run build:data-schemas
+
+      - name: Restore api build cache
+        id: cache-api
+        uses: actions/cache@v4
+        with:
+          path: packages/api/dist
+          key: build-api-${{ runner.os }}-${{ hashFiles('packages/api/src/**', 'packages/api/tsconfig*.json', 'packages/api/tsdown.config.mjs', 'packages/api/package.json', 'packages/data-provider/src/**', 'packages/data-provider/tsconfig*.json', 'packages/data-provider/tsdown.config.mjs', 'packages/data-provider/package.json', 'packages/data-schemas/src/**', 'packages/data-schemas/tsconfig*.json', 'packages/data-schemas/tsdown.config.mjs', 'packages/data-schemas/package.json') }}
+
+      - name: Build api
+        if: steps.cache-api.outputs.cache-hit != 'true'
+        run: npm run build:api
+
+      - name: Run agents integration tests (in-process MongoDB)
+        working-directory: packages/api
+        env:
+          NODE_ENV: test
+        run: npm run test:agents-integration

package-lock.json

@@ -33,6 +33,7 @@
     "test:cache-integration:stream": "jest --testPathPatterns=\"src/stream/.*\\.stream_integration\\.spec\\.ts$\" --coverage=false --runInBand --forceExit",
     "test:cache-integration": "npm run test:cache-integration:core && npm run test:cache-integration:cluster && npm run test:cache-integration:mcp && npm run test:cache-integration:stream",
     "test:s3-integration": "jest --testPathPatterns=\"src/storage/s3/.*\\.integration\\.spec\\.ts$\" --coverage=false --runInBand",
+    "test:agents-integration": "jest --testPathPatterns=\"src/agents/.*\\.integration\\.spec\\.ts$\" --coverage=false --runInBand --forceExit",
     "verify": "npm run test:ci",
     "b:clean": "bun run rimraf dist",
     "b:build": "bun run b:clean && bun run tsdown",
@@ -77,6 +78,7 @@
     "@types/winston": "^2.4.4",
     "@types/yauzl": "^2.10.3",
     "aws-sdk-client-mock": "^4.1.0",
+    "babel-plugin-replace-ts-export-assignment": "^0.0.2",
     "dedent": "^1.5.3",
     "get-stream": "^6.0.1",
     "jest": "^30.2.0",

packages/api/package.json

@@ -1,11 +1,14 @@
 import mongoose from 'mongoose';
+import { logger } from '@librechat/data-schemas';
 import { MongoMemoryServer } from 'mongodb-memory-server';
 import { MongoDBSaver } from '@langchain/langgraph-checkpoint-mongodb';
 import { emptyCheckpoint, ERROR, INTERRUPT } from '@langchain/langgraph-checkpoint';
 import {
   getAgentCheckpointer,
   deleteAgentCheckpoint,
   deleteAgentCheckpoints,
+  LazyMongoSaver,
+  CheckpointTooLargeError,
   __resetCheckpointerForTests,
 } from './checkpointer';
 
@@ -446,3 +449,197 @@ describe('LazyMongoSaver (lazy persistence — mongodb-memory-server)', () => {
     expect(effects).toEqual({ a: 1, c: 1 }); // siblings did NOT re-execute
   });
 });
+
+describe('LazyMongoSaver checkpoint size guard (mongodb-memory-server integration)', () => {
+  // Guards the single-document ceiling: a checkpoint embeds the whole message history, so a
+  // large conversation can serialize past MongoDB's 16 MB limit. The guard measures the
+  // serialized state on the persist path, WARNS past a soft threshold, and REJECTS past a hard
+  // limit BEFORE the write — a typed CheckpointTooLargeError instead of a raw BSONObjectTooLarge.
+  // Thresholds are shrunk here so payloads stay tiny while exercising the real serde + Mongo.
+  const clientForSaver = () =>
+    // mongoose vends the live MongoClient; the driver type resolves to a different `mongodb` copy
+    // than checkpoint-mongodb's, so the cast mirrors buildMongoSaver in checkpointer.ts.
+    mongoose.connection.getClient() as unknown as ConstructorParameters<
+      typeof MongoDBSaver
+    >[0]['client'];
+
+  const makeSaver = (overrides?: { warnBytes?: number; hardLimitBytes?: number }) =>
+    new LazyMongoSaver({
+      client: clientForSaver(),
+      checkpointCollectionName: 'agent_checkpoints',
+      checkpointWritesCollectionName: 'agent_checkpoint_writes',
+      ttl: 3600,
+      ...overrides,
+    });
+
+  /** Seed an interrupt anchor for a checkpoint whose serialized state is inflated to ~`payloadBytes`. */
+  async function seedSizedInterrupt(saver: LazyMongoSaver, threadId: string, payloadBytes: number) {
+    const checkpoint = emptyCheckpoint();
+    checkpoint.channel_values = { messages: 'x'.repeat(payloadBytes) };
+    await saver.putWrites(
+      { configurable: { thread_id: threadId, checkpoint_ns: '', checkpoint_id: checkpoint.id } },
+      [[INTERRUPT, 'approve?']],
+      'task-1',
+    );
+    const config = { configurable: { thread_id: threadId, checkpoint_ns: '' } };
+    const metadata = { source: 'input' as const, step: -1, writes: null, parents: {} };
+    return { checkpoint, config, metadata };
+  }
+
+  afterEach(() => {
+    jest.restoreAllMocks();
+  });
+
+  it('persists a checkpoint under the soft threshold', async () => {
+    const saver = makeSaver({ warnBytes: 5_000, hardLimitBytes: 50_000 });
+    await saver.setup();
+    const threadId = `convo-${new mongoose.Types.ObjectId().toString()}`;
+    const { checkpoint, config, metadata } = await seedSizedInterrupt(saver, threadId, 100);
+
+    await saver.put(config, checkpoint, metadata);
+
+    expect(await saver.getTuple(readConfig(threadId))).toBeDefined();
+  });
+
+  it('persists but WARNS when a checkpoint crosses the soft threshold', async () => {
+    const warnSpy = jest.spyOn(logger, 'warn').mockImplementation(() => logger);
+    const saver = makeSaver({ warnBytes: 500, hardLimitBytes: 50_000 });
+    await saver.setup();
+    const threadId = `convo-${new mongoose.Types.ObjectId().toString()}`;
+    const { checkpoint, config, metadata } = await seedSizedInterrupt(saver, threadId, 2_000);
+
+    await saver.put(config, checkpoint, metadata);
+
+    expect(await saver.getTuple(readConfig(threadId))).toBeDefined();
+    expect(warnSpy).toHaveBeenCalledWith(expect.stringContaining('soft threshold'));
+  });
+
+  it('REFUSES to persist and throws CheckpointTooLargeError over the hard limit', async () => {
+    jest.spyOn(logger, 'error').mockImplementation(() => logger);
+    const saver = makeSaver({ warnBytes: 500, hardLimitBytes: 2_000 });
+    await saver.setup();
+    const threadId = `convo-${new mongoose.Types.ObjectId().toString()}`;
+    const { checkpoint, config, metadata } = await seedSizedInterrupt(saver, threadId, 10_000);
+
+    await expect(saver.put(config, checkpoint, metadata)).rejects.toBeInstanceOf(
+      CheckpointTooLargeError,
+    );
+
+    // Nothing durable was written for that thread — getTuple reads the checkpoint document.
+    expect(await saver.getTuple(readConfig(threadId))).toBeUndefined();
+    expect(
+      await mongoose.connection.db!.collection('agent_checkpoints').countDocuments({
+        thread_id: threadId,
+      }),
+    ).toBe(0);
+  });
+
+  it('counts metadata toward the ceiling — refuses when the checkpoint is under but metadata pushes over', async () => {
+    // MongoDBSaver stores the serialized checkpoint AND metadata in one document, so a
+    // just-under-limit checkpoint with large metadata still overflows. The guard must catch it
+    // as a typed CheckpointTooLargeError, not let it fall through to a raw BSONObjectTooLarge.
+    jest.spyOn(logger, 'error').mockImplementation(() => logger);
+    const saver = makeSaver({ warnBytes: 500, hardLimitBytes: 2_000 });
+    await saver.setup();
+    const threadId = `convo-${new mongoose.Types.ObjectId().toString()}`;
+
+    const checkpoint = emptyCheckpoint();
+    checkpoint.channel_values = { messages: 'x'.repeat(400) }; // checkpoint alone well UNDER 2 KB
+    await saver.putWrites(
+      { configurable: { thread_id: threadId, checkpoint_ns: '', checkpoint_id: checkpoint.id } },
+      [[INTERRUPT, 'approve?']],
+      'task-1',
+    );
+    const config = { configurable: { thread_id: threadId, checkpoint_ns: '' } };
+    // Metadata alone pushes checkpoint + metadata over the 2 KB hard limit.
+    const metadata = {
+      source: 'input' as const,
+      step: -1,
+      writes: { payload: 'y'.repeat(4_000) },
+      parents: {},
+    };
+
+    await expect(saver.put(config, checkpoint, metadata)).rejects.toBeInstanceOf(
+      CheckpointTooLargeError,
+    );
+    expect(await saver.getTuple(readConfig(threadId))).toBeUndefined();
+  });
+
+  it('counts metadata_search (the raw metadata copy) — refuses when serialized-only would pass', async () => {
+    // MongoDBSaver stores `metadata_search: metadata` — the WHOLE raw metadata a
+    // SECOND time in the same document. Sized so checkpoint + serialized metadata is
+    // UNDER the limit but adding the raw metadata_search copy pushes it over: the guard
+    // must count that copy, else the doc slips the preflight and overflows on the wire.
+    jest.spyOn(logger, 'error').mockImplementation(() => logger);
+    const saver = makeSaver({ warnBytes: 500, hardLimitBytes: 6_000 });
+    await saver.setup();
+    const threadId = `convo-${new mongoose.Types.ObjectId().toString()}`;
+
+    const checkpoint = emptyCheckpoint();
+    checkpoint.channel_values = { messages: 'x'.repeat(200) };
+    await saver.putWrites(
+      { configurable: { thread_id: threadId, checkpoint_ns: '', checkpoint_id: checkpoint.id } },
+      [[INTERRUPT, 'approve?']],
+      'task-1',
+    );
+    const config = { configurable: { thread_id: threadId, checkpoint_ns: '' } };
+    // ~3 KB metadata: checkpoint + serialized metadata (~3.2 KB) is under 6 KB, but the
+    // raw metadata_search copy (~another 3 KB) pushes checkpoint+metadata+metadata_search
+    // over 6 KB. The pre-fix guard (checkpoint + serialized metadata only) would pass here.
+    const metadata = {
+      source: 'loop' as const,
+      step: 1,
+      writes: { payload: 'y'.repeat(3_000) },
+      parents: {},
+    };
+
+    await expect(saver.put(config, checkpoint, metadata)).rejects.toBeInstanceOf(
+      CheckpointTooLargeError,
+    );
+    expect(await saver.getTuple(readConfig(threadId))).toBeUndefined();
+  });
+
+  it('flushes bookkeeping parked during the size-serialization window (not dropped on resume)', async () => {
+    // `put` consumes the write anchor, then AWAITS `assertCheckpointFitsDocument` (serialization).
+    // A bookkeeping-only putWrites dispatched in that window sees no anchor and no persisted
+    // marker, so it parks — and must be flushed once the checkpoint persists, or a resume
+    // re-executes the completed sibling.
+    const NO_WRITES = '__no_writes__';
+    const saver = makeSaver({ warnBytes: 5_000, hardLimitBytes: 50_000 });
+    await saver.setup();
+    const threadId = `convo-${new mongoose.Types.ObjectId().toString()}`;
+    const checkpoint = emptyCheckpoint();
+    const writeCfg = {
+      configurable: { thread_id: threadId, checkpoint_ns: '', checkpoint_id: checkpoint.id },
+    };
+    await saver.putWrites(writeCfg, [[INTERRUPT, 'approve?']], 'task-gate');
+
+    // Pause the checkpoint serialization inside `put` so the bookkeeping batch lands mid-window.
+    const serde = (saver as unknown as { serde: { dumpsTyped: (v: unknown) => unknown } }).serde;
+    const realDumps = serde.dumpsTyped.bind(serde);
+    let releaseGate!: () => void;
+    const gate = new Promise<void>((resolve) => {
+      releaseGate = resolve;
+    });
+    let paused = false;
+    jest.spyOn(serde, 'dumpsTyped').mockImplementation(async (value: unknown) => {
+      if (!paused) {
+        paused = true;
+        await gate; // hold inside assertCheckpointFitsDocument (anchor consumed, not yet persisted)
+      }
+      return realDumps(value);
+    });
+
+    const config = { configurable: { thread_id: threadId, checkpoint_ns: '' } };
+    const metadata = { source: 'input' as const, step: -1, writes: null, parents: {} };
+    const putPromise = saver.put(config, checkpoint, metadata); // suspends at the gate
+    await saver.putWrites(writeCfg, [[NO_WRITES, null]], 'task-sibling'); // parks in the window
+    releaseGate();
+    await putPromise;
+
+    const tuple = await saver.getTuple(readConfig(threadId));
+    const channels = (tuple?.pendingWrites ?? []).map((w) => w[1]);
+    expect(channels).toContain(INTERRUPT);
+    expect(channels).toContain(NO_WRITES); // flushed, not dropped
+  });
+});