Skip to content

fix(connect): decode registration into a fresh WgData to prevent stale hcStatus on reconnect#413

Open
lafoush wants to merge 1 commit into
fosrl:mainfrom
lafoush:fix/reconnect-stale-hcstatus
Open

fix(connect): decode registration into a fresh WgData to prevent stale hcStatus on reconnect#413
lafoush wants to merge 1 commit into
fosrl:mainfrom
lafoush:fix/reconnect-stale-hcstatus

Conversation

@lafoush

@lafoush lafoush commented Jul 14, 2026

Copy link
Copy Markdown

Problem

A resource with a health check can be reported permanently Unhealthy after a reconnect, even though it responds correctly, until newt is restarted. It's intermittent and only happens on reconnect (never on a clean boot).

Root cause

handleConnect decodes the newt/wg/connect registration payload into the persistent n.wgData:

json.Unmarshal(jsonData, &n.wgData)

Go's encoding/json merges a JSON array into an existing non-empty slice element-by-element, by position, and a JSON null is a no-op for a non-pointer field such as healthcheck.Config.Status (hcStatus, an int). n.wgData is reused across reconnects, so:

  1. First connect populates wgData.HealthCheckTargets, e.g. slot 0 = {id:47, hcStatus:401}, slot 1 = {id:5, hcStatus:null}.
  2. On reconnect the server returns the same targets but in a different order (the health-check config query has no ORDER BY), e.g. [{id:5, hcStatus:null}, {id:47, hcStatus:401}].
  3. Decoding into the reused slice merges new-slot-0 (id:5, hcStatus:null) into old-slot-0 (id:47, hcStatus:401). id is overwritten to 5, but the null hcStatus does not overwrite the retained 401.
  4. Result: target 5 now expects 401. Its check (GET /manifest.json200) fails forever with unexpected status code: 200 (expected: 401).

handleSync is unaffected because it decodes into a fresh local SyncData. A clean boot is unaffected because wgData starts empty. This matches the observed profile exactly: reconnect-only, intermittent (needs a row reorder), self-heals on restart.

Fix

Decode the registration payload into a fresh WgData and assign it on success, so no state is ever carried over from a previous connection:

var wgData WgData
if err := json.Unmarshal(jsonData, &wgData); err != nil {
    ...
    return
}
n.wgData = wgData

n.wgData is written nowhere else, so a wholesale replace is safe.

Test

Adds newt/wgdata_reconnect_test.go, which pins both halves:

  • decoding two reordered payloads into a reused WgData reproduces the stale 401 inheritance;
  • decoding each into a fresh WgData (what handleConnect now does) keeps target 5 free of target 47's expected code.

go build ./..., go vet ./newt/, and the full ./newt/ + ./healthcheck/ suites pass.

…e hcStatus

handleConnect decoded the newt/wg/connect registration payload into the
persistent n.wgData with json.Unmarshal(jsonData, &n.wgData). Decoding a
JSON array into an existing non-empty slice merges element-by-element by
position, and a JSON null is a no-op for non-pointer fields. So on
reconnect, when the server returns the health-check rows in a different
order (the config query has no ORDER BY), a target whose hcStatus is null
inherits the stale expected status code from whatever target previously
occupied that slice slot.

Concretely: a resource with no custom expected code (hcStatus null, so
newt uses the 2xx default) could inherit a sibling resource's 401 when the
two shared a backend and the rows reordered on reconnect, leaving the
resource permanently Unhealthy until newt restarted. handleSync was
unaffected because it decodes into a fresh local SyncData.

Decode into a fresh WgData and assign on success. Adds a regression test
covering the reused-slice hazard and the fresh-decode fix.
@lafoush

lafoush commented Jul 14, 2026

Copy link
Copy Markdown
Author

For reference, we are hot patching Newt in our product, the patch is getting rolled out to a portion of our fleet for testing. I'll update this issue if our monitoring raise any other issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant