feat(sync_jobs): validate id_big NOT NULL (NAN-5491 Phase 3d)

[pfreixes]

Why

Prove id_big has no NULL rows so Phase 3e's swap stays metadata-only under ACCESS EXCLUSIVE — the long scan happens here, in a non-blocking lock mode.

How

• Stack: depends on Phase 3c (#6378) — the unique index is in place.
• Migrations: run manually in cloud prod from psql. Two separate statements (the second is NOT inside a transaction with the first), so the VALIDATE step runs under its native SHARE UPDATE EXCLUSIVE (non-blocking) instead of inheriting ACCESS EXCLUSIVE from the catalog write. The migration row is inserted into nango._nango_auth_migrations afterward so the auto-runner skips it. Self-hosted runs the migration automatically (instant + small-table scan, no concern).

Commands

(Schema-qualified for the operator runbook — the migration file itself uses unqualified table names and relies on search_path.)

1. Pre-flight check — backfill is still complete and unique index is in place:

SELECT count(*) FROM nango._nango_sync_jobs WHERE id_big IS NULL;
-- expect: 0

SELECT indisvalid, indisready
FROM pg_index
WHERE indexrelid = 'nango.sync_jobs_id_big_uidx'::regclass;
-- expect: t, t

2. Add the CHECK constraint (NOT VALID) — catalog entry only, wrapped with lock_timeout = '1s' so the brief ACCESS EXCLUSIVE can't queue up behind a busy moment:

BEGIN;
SET LOCAL lock_timeout = '1s';
ALTER TABLE nango._nango_sync_jobs
  ADD CONSTRAINT id_big_not_null CHECK (id_big IS NOT NULL) NOT VALID;
COMMIT;

3. Validate the constraint — runs OUTSIDE a transaction so it gets SHARE UPDATE EXCLUSIVE (concurrent reads/writes keep flowing). Aborts the whole step if any row has id_big IS NULL:

ALTER TABLE nango._nango_sync_jobs
  VALIDATE CONSTRAINT id_big_not_null;

4. Verify — convalidated must be t:

SELECT convalidated
FROM pg_constraint
WHERE conrelid = 'nango._nango_sync_jobs'::regclass
  AND conname = 'id_big_not_null';
-- expect: t

5. Mark migration applied — skip the auto-runner on next deploy:

INSERT INTO nango._nango_auth_migrations (name, batch, migration_time)
VALUES (
    '20260608120250_sync_jobs_validate_id_big_not_null.cjs',
    (SELECT COALESCE(MAX(batch), 0) + 1 FROM nango._nango_auth_migrations),
    NOW()
);

Emergency stop (if VALIDATE needs to be cancelled mid-flight)

The VALIDATE scan is interruptible since it runs under SHARE UPDATE EXCLUSIVE (no exclusive lock to release).

1. Find the PID (from a separate psql session):

SELECT pid, now() - query_start AS duration, state
FROM pg_stat_activity
WHERE query ILIKE '%VALIDATE CONSTRAINT id_big_not_null%'
  AND pid <> pg_backend_pid();

2. Cancel:

SELECT pg_cancel_backend(<pid>);

3. Clean up the unvalidated CHECK so a retry starts clean:

ALTER TABLE nango._nango_sync_jobs DROP CONSTRAINT id_big_not_null;

After the DROP, you're back to a clean pre-3d state and can retry from step 2.

What

• Add a validated CHECK (id_big IS NOT NULL) constraint named id_big_not_null to _nango_sync_jobs. From that point on, every INSERT/UPDATE is also constraint-checked, so the table can't drift back into containing NULLs while we wait to run 3e.
• 3e's swap then uses this validated CHECK as proof (PG 12+ optimization): ALTER COLUMN id SET NOT NULL becomes metadata-only, and ADD CONSTRAINT … PRIMARY KEY USING INDEX no longer has to scan under ACCESS EXCLUSIVE. The redundant CHECK is dropped inside the swap right before the PK supersedes it.
• Schema-qualifier dropped in the migration file (nango._nango_sync_jobs → _nango_sync_jobs) to match the project convention of relying on search_path for table resolution.

Linear: NAN-5491.

Test plan

• Cloud prod: pre-flight check (NULL count = 0, unique index valid)
• Cloud prod: run ADD CONSTRAINT … NOT VALID wrapped with lock_timeout = '1s'
• Cloud prod: run VALIDATE CONSTRAINT outside a txn
• convalidated = t on id_big_not_null
• INSERT INTO nango._nango_auth_migrations to mark applied
• Self-hosted auto-runs the migration on next deploy

[linear-code]

NAN-5491

[cubic-dev-ai]

1 issue found across 1 file

Confidence score: 3/5

• packages/database/lib/migrations/20260604120250_sync_jobs_validate_id_big_not_null.cjs combines ADD CONSTRAINT ... NOT VALID with VALIDATE CONSTRAINT in one transaction, which can keep a stronger lock until commit and block concurrent writes.
• This is a concrete runtime risk (severity 7/10, confidence 8/10) that can cause user-facing write disruption during deployment, so merge risk is moderate rather than low.
• Pay close attention to packages/database/lib/migrations/20260604120250_sync_jobs_validate_id_big_not_null.cjs - split/sequence validation to reduce lock duration and avoid write blocking.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/database/lib/migrations/20260604120250_sync_jobs_validate_id_big_not_null.cjs">

<violation number="1" location="packages/database/lib/migrations/20260604120250_sync_jobs_validate_id_big_not_null.cjs:4">
P1: This migration runs `ADD CONSTRAINT ... NOT VALID` and `VALIDATE CONSTRAINT` in the same transaction, which can hold the stronger lock until commit and block writes during validation.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P1: This migration runs ADD CONSTRAINT ... NOT VALID and VALIDATE CONSTRAINT in the same transaction, which can hold the stronger lock until commit and block writes during validation.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/database/lib/migrations/20260604120250_sync_jobs_validate_id_big_not_null.cjs, line 4:

<comment>This migration runs `ADD CONSTRAINT ... NOT VALID` and `VALIDATE CONSTRAINT` in the same transaction, which can hold the stronger lock until commit and block writes during validation.</comment>

<file context>
@@ -0,0 +1,9 @@
+/**
+ * @param {import('knex').Knex} knex
+ */
+exports.up = async function (knex) {
+    await knex.raw(`ALTER TABLE nango._nango_sync_jobs ADD CONSTRAINT id_big_not_null CHECK (id_big IS NOT NULL) NOT VALID`);
+    await knex.raw(`ALTER TABLE nango._nango_sync_jobs VALIDATE CONSTRAINT id_big_not_null`);
</file context>

[pfreixes]

check the description, we will run it separately in the cloud environment for that specific reason. But good catch!

[cubic-dev-ai]

Thanks, good catch!

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.