Skip to content

fix(experimentation): keep variant assignment stable across rollout updates#7912

Merged
gagantrivedi merged 1 commit into
mainfrom
fix/experiment-rollout-variant-reshuffle
Jun 30, 2026
Merged

fix(experimentation): keep variant assignment stable across rollout updates#7912
gagantrivedi merged 1 commit into
mainfrom
fix/experiment-rollout-variant-reshuffle

Conversation

@gagantrivedi

@gagantrivedi gagantrivedi commented Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Member
  • I have read the Contributing Guide.
  • I have added information to docs/ if required so people know about the feature.
  • I have filled in the "Changes" section below.
  • I have filled in the "How did you test this code" section below.

Changes

Under v2 versioning, updating an experiment's rollout went through update_flag, which clones the segment override into a new feature state each time. The multivariate split is salted on the feature state id, so every update re-randomised control/variant for already-enrolled identities, corrupting the experiment data.

Now, once the override exists, it's updated in place (id preserved → stable split) and the environment document is rebuilt directly. Creating the override, and all v1 versioning, still go through update_flag.

How did you test this code?

Added test_apply_experiment_rollout__reapplied_under_v2__keeps_variant_assignment (fails on main, passes here). Full experimentation + versioning unit suites and make typecheck pass.

@vercel

vercel Bot commented Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview, Comment Jun 30, 2026 7:17am
2 Skipped Deployments
Project Deployment Actions Updated (UTC)
flagsmith-frontend-preview Ignored Ignored Preview Jun 30, 2026 7:17am
flagsmith-frontend-staging Ignored Ignored Preview Jun 30, 2026 7:17am

Request Review

@github-actions github-actions Bot added the api Issue related to the REST API label Jun 30, 2026
@gagantrivedi gagantrivedi changed the base branch from main to feat/experiment-rollout-default-control-allocation June 30, 2026 05:23
@gagantrivedi gagantrivedi force-pushed the fix/experiment-rollout-variant-reshuffle branch from 8451662 to f05a0a8 Compare June 30, 2026 05:43
@codecov

codecov Bot commented Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.62%. Comparing base (c393d31) to head (cae154b).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files 
@@           Coverage Diff           @@
##             main    #7912   +/-   ##
=======================================
  Coverage   98.62%   98.62%           
=======================================
  Files        1488     1488           
  Lines       58543    58578   +35     
=======================================
+ Hits        57737    57772   +35     
  Misses        806      806

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@gagantrivedi gagantrivedi force-pushed the fix/experiment-rollout-variant-reshuffle branch from f05a0a8 to aa1b95c Compare June 30, 2026 06:39
@gagantrivedi gagantrivedi changed the title WIP: fix(experimentation): keep variant assignment stable across rollout updates fix(experimentation): keep variant assignment stable across rollout updates Jun 30, 2026
@gagantrivedi gagantrivedi marked this pull request as ready for review June 30, 2026 06:47
@gagantrivedi gagantrivedi requested a review from a team as a code owner June 30, 2026 06:47
@gagantrivedi gagantrivedi requested review from emyller and removed request for a team June 30, 2026 06:47
@github-actions

github-actions Bot commented Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Docker builds report

Image Build Status Security report
ghcr.io/flagsmith/flagsmith-e2e:pr-7912 Finished ✅ Skipped
ghcr.io/flagsmith/flagsmith-api-test:pr-7912 Finished ✅ Skipped
ghcr.io/flagsmith/flagsmith-frontend:pr-7912 Finished ✅ Results ✅
ghcr.io/flagsmith/flagsmith-api:pr-7912 Finished ✅ Results ✅
ghcr.io/flagsmith/flagsmith:pr-7912 Finished ✅ Results ✅
ghcr.io/flagsmith/flagsmith-private-cloud:pr-7912 Finished ✅ Results ✅

@github-actions github-actions Bot added the fix label Jun 30, 2026
@Zaimwa9 Zaimwa9 self-requested a review June 30, 2026 06:51
@Zaimwa9 Zaimwa9 removed the request for review from emyller June 30, 2026 06:51
@github-actions

github-actions Bot commented Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  4 passed

Details

stats  4 tests across 3 suites
duration  10.6 seconds
commit  aa1b95c
info  🔄 Run: #17945 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  4 passed

Details

stats  4 tests across 3 suites
duration  11.6 seconds
commit  aa1b95c
info  🔄 Run: #17945 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  5 passed

Details

stats  5 tests across 3 suites
duration  11.3 seconds
commit  aa1b95c
info  🔄 Run: #17945 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  6 passed

Details

stats  6 tests across 5 suites
duration  20.3 seconds
commit  aa1b95c
info  🔄 Run: #17945 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  3 passed

Details

stats  3 tests across 3 suites
duration  33.4 seconds
commit  c5b1a63
info  🔄 Run: #17947 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  5 passed

Details

stats  5 tests across 3 suites
duration  11.8 seconds
commit  c5b1a63
info  🔄 Run: #17947 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  4 passed

Details

stats  4 tests across 3 suites
duration  37.7 seconds
commit  c5b1a63
info  🔄 Run: #17947 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  57.8 seconds
commit  c5b1a63
info  🔄 Run: #17947 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  4 passed

Details

stats  4 tests across 3 suites
duration  39.3 seconds
commit  cae154b
info  🔄 Run: #17953 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  3 passed

Details

stats  3 tests across 2 suites
duration  11.7 seconds
commit  cae154b
info  🔄 Run: #17953 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  53.2 seconds
commit  cae154b
info  🔄 Run: #17953 (attempt 1)

@github-actions

github-actions Bot commented Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Visual Regression

17 screenshots compared. See report for details.
View full report

@gagantrivedi gagantrivedi force-pushed the fix/experiment-rollout-variant-reshuffle branch from aa1b95c to c5b1a63 Compare June 30, 2026 06:55
@gagantrivedi gagantrivedi requested a review from a team as a code owner June 30, 2026 06:55
@github-actions github-actions Bot added the docs Documentation updates label Jun 30, 2026
@github-actions github-actions Bot added fix and removed fix docs Documentation updates labels Jun 30, 2026
@gagantrivedi gagantrivedi marked this pull request as draft June 30, 2026 07:09
@gagantrivedi
fix(experimentation): keep variant assignment stable across rollout u…
cae154b 
…pdates

Under v2 versioning, updating an experiment's rollout routed the segment
override write through update_flag, which clones the override into a fresh
feature state on every call. The multivariate split is salted on the feature
state id, so each rollout update re-randomised the control/variant assignment
for already-enrolled identities, contaminating exposure and conversion data.

Once the rollout override exists, mutate it in place (preserving its id, and
therefore the variant assignment) and rebuild the environment document by hand
instead of publishing a new version. Creating the override, and v1 versioning,
still go through update_flag, which already reuses the feature state.
@gagantrivedi gagantrivedi force-pushed the fix/experiment-rollout-variant-reshuffle branch from c5b1a63 to cae154b Compare June 30, 2026 07:16
@gagantrivedi gagantrivedi changed the base branch from feat/experiment-rollout-default-control-allocation to main June 30, 2026 07:16
@gagantrivedi gagantrivedi marked this pull request as ready for review June 30, 2026 07:16
@github-actions github-actions Bot added docs Documentation updates fix and removed fix docs Documentation updates labels Jun 30, 2026
Zaimwa9
Zaimwa9 approved these changes Jun 30, 2026
View reviewed changes

@Zaimwa9 Zaimwa9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good as a temporary solution

@github-actions github-actions Bot added fix and removed fix labels Jun 30, 2026
@gagantrivedi gagantrivedi merged commit 772b9cb into main Jun 30, 2026
61 of 69 checks passed
@gagantrivedi gagantrivedi deleted the fix/experiment-rollout-variant-reshuffle branch June 30, 2026 07:33
@flagsmith-engineering flagsmith-engineering Bot mentioned this pull request Jun 29, 2026
Merged
@gagantrivedi gagantrivedi mentioned this pull request Jun 30, 2026
Draft
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Zaimwa9 Zaimwa9 Zaimwa9 approved these changes

Assignees

@Zaimwa9 Zaimwa9

Labels

api Issue related to the REST API fix

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@gagantrivedi @Zaimwa9 @emyller