Skip to content

[Rust] Reject server over-acks - #648

Draft
Pybsama wants to merge 1 commit into
databricks:mainfrom
Pybsama:codex/reject-server-over-acks
Draft

[Rust] Reject server over-acks#648
Pybsama wants to merge 1 commit into
databricks:mainfrom
Pybsama:codex/reject-server-over-acks

Conversation

@Pybsama

@Pybsama Pybsama commented Jul 31, 2026

Copy link
Copy Markdown

What changes are proposed in this pull request?

Fixes #638.

This change makes the Rust gRPC receiver reject malformed cumulative durability acknowledgements before they mutate client state. It:

  • tracks the highest physical offset handed to each active gRPC connection;
  • synchronizes outbound channel handoff and sent-watermark publication, so an observed request blocked on channel capacity is not considered sent;
  • rejects negative and beyond-sent ACK offsets before removing records or completing oneshots/callbacks;
  • treats duplicate and regressive cumulative ACKs as no-ops, preserving a monotonic watermark; and
  • retains an explicit invariant failure if an otherwise valid ACK cannot be applied to the observed queue.

A landing-zone observed count is not a safe upper bound because observe() happens before the bounded outbound channel handoff. A connection-local sent watermark closes that concurrency window and is reset together with the physical offset generator on recovery.

How is this tested?

  • cargo test -p databricks-zerobus-ingest-sdk (162 unit tests plus 24 passing doc tests; 9 doc tests ignored by existing annotations)
  • cargo test -p tests --test rust_tests (93 integration tests)
  • deterministic capacity-one sender test proving an observed request blocked in reserve() does not advance the sent watermark
  • mock gRPC regressions for a single record and a five-record batch, both receiving ACK 1 after only physical offset 0 was sent
  • receiver regressions for negative, duplicate, regressive, valid, and beyond-sent ACKs, including the 1 -> 0 -> 2 sequence
  • cargo clippy -p databricks-zerobus-ingest-sdk --all-targets -- -D warnings
  • make check
  • git diff --check

Track the highest physical offset handed to each gRPC connection and reject malformed cumulative acknowledgements before they can advance durability state. Preserve pending single records and batches on protocol violations.

Fixes databricks#638

Signed-off-by: sama Pyb <peiyibopybsama@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Rust] Receiver silently absorbs server over-ack and can wedge the stream

1 participant