Skip to content

SK-2954: common/v2/v3 split + flowvault insert support#269

Open
saileshwar-skyflow wants to merge 3 commits into
mainfrom
saileshwar/SK-2954-flowdb-vaut-apis-support
Open

SK-2954: common/v2/v3 split + flowvault insert support#269
saileshwar-skyflow wants to merge 3 commits into
mainfrom
saileshwar/SK-2954-flowdb-vaut-apis-support

Conversation

@saileshwar-skyflow

@saileshwar-skyflow saileshwar-skyflow commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Why

Splits skyflow-python from a single-variant SDK into three build variants sharing a bundled common/ module (SK-2938 Option C, matching the pattern already used in skyflow-java's own v2/v3 split):

  • common/ — shared credential resolution, vault-URL resolution, bearer-token fetch/cache/expiry, enums, errors, and generic validators. Never published independently; bundled into each variant's wheel at build time.
  • v2/ — today's SDK, relocated from the repo root. Public API unchanged — same class names, signatures, import paths.
  • v3/ — new, built on the flowservice/flowdb API. Insert-only this round by design; every other operation is deliberately out of scope until a follow-up round. Publishes as skyflow-flowvault, starting at 1.0.0.

common/

  • New VaultController abstract base (mirrors Java's VaultController interface): declares insert/get/update/delete/query/detokenize as abstract methods. v2's PdbVaultController and v3's FlowVaultController each provide their own concrete/stub implementations — v2 already had all six; v3 only implements insert for real, with explicit "not implemented yet" stubs for the rest so it satisfies the abstract contract.
  • New BaseVaultClient abstract base: shared credential/bearer-token/vault-URL-resolution logic, with resolve_vault_url/initialize_api_client as per-variant hooks (v2 and v3 vaults live on different subdomains — vault.skyflowapis.* vs skyvault.skyflowapis.* — confirmed against a real vault).

v2/

  • Structural relocation only — zero behavior change. Vault is kept as a backward-compatible alias for the new internal PdbVaultController name, so from skyflow.vault.controller import Vault and everything downstream of it keeps working unchanged.
  • Fixed a real bug surfaced during this work: v2's own Env enum was failing cross-class comparisons (env not in Env) against common's Env because they were two structurally-identical-but-distinct enum classes. v2/skyflow/utils/enums/env.py is now a re-export of common's.

v3/ — insert support

  • New request/response types: InsertRequest/InsertRecord/Upsert (Java-parity "rich" shape — table/upsert can be set at the request level or overridden per-record). InsertResponse mirrors Java's v3 shape (summary/success/errors) as plain dicts — no custom response classes — with every result tagged with its index in the original record list (stable across batch boundaries, so callers can correlate a result back via request.records[result['index']]).
  • All four EnvUrls confirmed (DEV/SANDBOX/STAGE/PROD, each on skyvault.skyflowapis.*), with test coverage for each.
  • Validation ported directly from Java's v3/.../utils/validations/Validations.java, not reinvented:
    • Table must live in exactly one place — request-level (applies to every record) or per-record (every record sets its own) — never mixed, never both. Confirmed directly against a real vault's 400 response.
    • Upsert's placement must match table's placement (record-level upsert forbidden when table is request-level, and vice versa).
    • 10,000-record cap per request.
    • Per-record data dict keys/values can't be null or empty.
  • Error handling: parses the backend's actual structured per-record error body (a records list) into individual error entries instead of one flat message per failed batch, and propagates x-request-id onto each error for support/debugging. Verified against a real vault's actual 400 response (a NOT NULL column violation).
  • Batching: INSERT_BATCH_SIZE env var (default 50, max 1000), sequential (no concurrency this round), isolate-and-continue — a failing batch doesn't abort the rest.
  • New minimal Skyflow/Builder client facade and FlowVaultController/VaultClient wired to the flowservice REST client.
  • common is never part of the public interface — skyflow.error/skyflow.utils re-export what consumers need (SkyflowError, Env, LogLevel), matching v2's own established shim pattern.

CI/CD

Every existing workflow (main, ci, beta-release, internal-release, release, and the two shared reusable workflows) now works for both variants instead of just v2:

  • shared-tests.yml / shared-build-and-deploy.yml take a variant input and scope every build/test/bump/publish step to v2/ or v3/ via working-directory.
  • main.yml/ci.yml matrix over both variants (plus a new common/-only test job, since it now contains real shared logic worth gating on).
  • beta-release.yml/internal-release.yml/release.yml matrix over both variants with a job-level dispatch. v3 releases use a flowvault- tag/branch prefix (flowvault-1.0.0, flowvault-release/*) so a release trigger is never ambiguous between the two independently-versioned packages — v2's existing bare-semver tags (2.1.3, release/*) are completely untouched.
  • Fixed two latent bugs surfaced while doing this: ruff.toml/.codespellrc were still excluding a skyflow/generated path that stopped existing once the repo split (now excludes bare generated, matching at any depth); and bump_version.sh's version-bump sed had a collision with a code comment that happened to contain the literal text it was matching on.

Testing

  • common/tests — 36 tests, new.
  • v3/tests — 65 tests, new (validation parity, wire-shape mapping, batching, response splitting, error parsing, header injection, all four EnvUrls).
  • v2/tests — 426 tests, zero modifications to the existing suite (2 pre-existing failures are an unrelated gitignored credentials.json fixture, present before this change).
  • tests/contract/ — one shared insert() contract asserted against both variants' own installed environment.
  • Manually verified end-to-end against a real flowdb/v3 vault (SSL, auth, vault-URL resolution, batching, and the exact error-response shape all confirmed live, not just mocked).

Known follow-ups (not done here, intentionally out of scope)

  • v3's get/update/delete/query/detokenize are stubs (NotImplementedError) — planned for follow-up rounds, one operation at a time, same pattern as this round's insert.
  • CI: JFrog Artifactory publish target is reused as-is for both variants (a JFrog PyPI repo can host multiple distinctly-named packages); worth a careful look before the very first real flowvault- tag/branch push since some of this couldn't be dry-run outside GitHub's own infrastructure.

@github-advanced-security github-advanced-security AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Semgrep OSS found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

…lowdb (v3) insert support

Restructures the repo into three build variants sharing a bundled common/
module (SK-2938 Option C): v2 (today's SDK, behavior-preserving) and a new
v3 built on the flowservice/flowdb API, insert-only this round.

common/
- Shared credential resolution, vault-URL resolution, and bearer-token
  fetch/cache/expiry logic (VaultController, BaseVaultClient), enums,
  errors, service_account, and generic validators.
- VaultController declares insert/get/update/delete/query/detokenize as
  abstract methods (Java-interface-style); v2 and v3 each provide their
  own concrete/stub implementations.

v2
- Relocated from the repo root via git mv; public API unchanged (same
  class names, signatures, import paths). Vault is now a backward-compatible
  alias for the internal PdbVaultController class.
- Fixed a latent bug where v2's own Env enum failed cross-class comparisons
  against common's Env; both now share one definition.

v3 (skyflow-flowvault, starting at 1.0.0)
- New InsertRequest/InsertRecord/Upsert/InsertResponse types, FlowVaultController,
  and VaultClient targeting the flowservice REST API.
- Insert validation ported from Java's v3 Validations.java: table/upsert
  must live in exactly one place (request-level or per-record, matching
  in both), 10k record cap, empty key/value checks.
- InsertResponse mirrors Java's v3 shape (summary/success/errors) as plain
  dicts, each result tagged with its index in the original record list
  (stable across batch boundaries).
- Structured per-record error parsing from the backend's actual error
  body, plus x-request-id propagation onto error entries.
- Batching via INSERT_BATCH_SIZE (default 50, max 1000), sequential,
  isolate-and-continue on a failing batch.
- Vault URL resolution uses v3's own skyvault.skyflowapis.* domain for
  all four envs (DEV/SANDBOX/STAGE/PROD), confirmed to differ from v2's
  vault.skyflowapis.* domain.

CI/CD
- shared-tests.yml and shared-build-and-deploy.yml now take a `variant`
  input and scope every step to v2/ or v3/ via working-directory.
- main.yml, ci.yml, beta-release.yml, internal-release.yml, and
  release.yml matrix over both variants. v3 releases are distinguished
  from v2's via a flowvault- tag/branch prefix (flowvault-1.0.0,
  flowvault-release/*) so a release trigger is never ambiguous between
  the two independently-versioned packages; v2's existing bare-semver
  tags are untouched.
- Fixed ruff.toml/.codespellrc still excluding a pre-split "skyflow/generated"
  path that no longer existed after the relocation.
- Fixed a bump_version.sh sed collision with a comment that happened to
  contain the literal text "__version__ = ...".
- Added a common/ test job to main.yml/ci.yml.

Note: v3/samples/ and the root samples/ folder are deliberately excluded
from this branch/commit -- local working copies there contain
credentials used for live testing against a real vault and must not be
pushed.

Tests: common 36, v3 65, v2 426 (2 pre-existing unrelated fixture
failures), tests/contract passing for both variants.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@saileshwar-skyflow saileshwar-skyflow force-pushed the saileshwar/SK-2954-flowdb-vaut-apis-support branch from 57c72e0 to cadf5d4 Compare July 7, 2026 11:01
saileshwar-skyflow and others added 2 commits July 8, 2026 00:34
Distribution name was already skyflow-flowvault (setup.py) but the
importable package stayed skyflow, colliding with v2's skyflow
import name if both are ever installed in the same environment.
Rename the v3 directory to flowvault and its package to
skyflow_flowvault to match the distribution name and remove the
collision. generated/ content is left untouched (Fern-owned).
…d base insert response

Consolidates duplicated logic between v2 (PDB) and flowvault per architecture
review: shared validation (vault config, credentials, log level), LogLevel/Logger,
and insert field/table validation now live in common with per-variant message
injection; adds BaseInsertResponse alongside BaseInsertRequest so each variant's
InsertRequest/InsertResponse can extend a common base while keeping its own shape.
Also fixes flowvault's insert() response shape (drop redundant 'data'/'table',
flatten tokens, errors=None when empty) and a stale SDK_VERSION drift bug.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@saileshwar-skyflow saileshwar-skyflow changed the title SK-2954: common/v2/v3 split + v3 (flowvault) insert support SK-2954: common/v2/v3 split + flowvault insert support Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants