perf(canton): use 3.5.1 query pagination for ACS and update reads - #309
Open
schronck wants to merge 1 commit into
Open
perf(canton): use 3.5.1 query pagination for ACS and update reads#309schronck wants to merge 1 commit into
schronck wants to merge 1 commit into
Conversation
New ledger_paging helper wraps GetActiveContractsPage / GetUpdatesPage and falls back to the streaming calls on participants below Canton 3.5.1. All 25 ACS reads in queries.rs go through it. Chain audit pages newest-first and stops once it has the limit, instead of draining the whole retained ledger before sorting and truncating. Cursor (before_offset / next_before_offset) runs through the endpoint and the UI. Fix get_party_metadata silently returning None for parties past the first 1000: page_size was hardcoded and next_page_token never read. Now filters server-side on filter_party and walks pages. /packages/vetted reported every uploaded DAR as vetted; it now reads real topology vetting state via the paginated ListVettedPackages. Shared PAGE_SIZE=25 in common::api, emitted into the generated TS by gen-types, drives both the wire page size and the UI page size.
There was a problem hiding this comment.
Pull request overview
This PR upgrades the decman backend and frontend to consistently use Canton’s paginated ledger APIs (where available) to reduce unbounded streaming reads, unify paging behavior, and align UI pagination with a single shared PAGE_SIZE constant generated into TypeScript.
Changes:
- Add
server/ledger_paging.rshelpers for paginated ACS / updates reads (with fallback to streaming for pre-3.5.1 participants) and migrate many ACS reads to use them. - Rework governance chain-audit to page newest-first using a cursor (
before_offset) and update the cache query to page by offset groups (transaction offsets) without splitting them across pages. - Introduce shared
PAGE_SIZE = 25incommon::api, emit it intotypes.generated.ts, and add reusable frontend pagination components/hooks applied across multiple views.
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| crates/decman/src/utils.rs | Adds a Ledger API PackageServiceClient creator for paged package reads. |
| crates/decman/src/server/types.rs | Re-exports PAGE_SIZE; updates chain-audit query DTO to include before_offset and default limit to PAGE_SIZE. |
| crates/decman/src/server/queries.rs | Routes many ACS reads through ledger_paging::fetch_active_contracts; fixes get_party_metadata paging by walking pages and using filter_party. |
| crates/decman/src/server/mod.rs | Registers the new ledger_paging module. |
| crates/decman/src/server/ledger_paging.rs | New paging abstraction for ACS, updates, and vetted packages (with pre-3.5.1 fallbacks for ACS/updates). |
| crates/decman/src/server/handlers/parties.rs | Switches /packages/vetted to ListVettedPackages via ledger paging helper and updates response mapping/error message. |
| crates/decman/src/server/handlers/governance.rs | Adds cursor-based chain-audit response shape (next_before_offset) and threads before_offset into cache/live paths. |
| crates/decman/src/server/chain_audit.rs | Implements descending paging via GetUpdatesPage, early-exit collection, and offset-group-safe truncation; adds before_offset range capping. |
| crates/decman/src/db/sqlite.rs | Updates chain-audit cache query to return whole offset groups and support before_offset. |
| crates/decman/src/db/schema.rs | Extends SchemaRead::get_chain_audit_cache signature with before_offset. |
| crates/decman/src/bin/gen_types.rs | Appends export const PAGE_SIZE = ... to generated TS types to prevent UI/wire drift. |
| crates/decman/frontend/src/usePagination.ts | New hook for client-side paging of fully-loaded lists. |
| crates/decman/frontend/src/constants.ts | Re-exports generated PAGE_SIZE as a UI constant. |
| crates/decman/frontend/src/components/PartyList.tsx | Adds client-side paging controls for the party list. |
| crates/decman/frontend/src/components/PartyDetail.tsx | Adds client-side paging controls for the contracts table. |
| crates/decman/frontend/src/components/Pagination.tsx | New PaginationControls and CursorPagination UI components. |
| crates/decman/frontend/src/components/PackagesPanel.tsx | Adds paging to both local and comparison package tables. |
| crates/decman/frontend/src/components/NotificationsView.tsx | Adds paging to the notifications feed; adjusts hook ordering via useMemo. |
| crates/decman/frontend/src/components/HoldingsSection.tsx | Adds paging to holdings table. |
| crates/decman/frontend/src/components/GovernanceAuditTrail.tsx | Switches chain-audit to cursor paging (server-driven), sets limit to PAGE_SIZE, and adds cursor pagination controls. |
| crates/decman/frontend/src/components/ExternalPartyList.tsx | Adds paging to external parties list. |
| crates/common/src/api.rs | Introduces shared PAGE_SIZE constant and extends ChainAuditResponse with next_before_offset. |
Suppressed comments (1)
crates/decman/src/server/chain_audit.rs:549
- The new “don’t split an offset group” truncation logic is subtle and has important correctness implications for cursor paging. Since this module already has unit tests, consider adding a focused test that feeds a synthetic list of entries with repeated offsets and asserts the truncation keeps whole offset groups (and that
before_offsetpaging doesn’t skip entries).
if entries.len() > limit {
let boundary = entries[limit - 1].offset;
let keep = entries
.iter()
.position(|e| e.offset < boundary)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+221
to
+226
| .list_vetted_packages(tonic::Request::new(ListVettedPackagesRequest { | ||
| package_metadata_filter: None, | ||
| topology_state_filter: None, | ||
| page_token: page_token.clone(), | ||
| page_size: PAGE_SIZE as u32, | ||
| })) |
Comment on lines
+1569
to
+1572
| Ok(fetch_active_contracts(config, token, event_format) | ||
| .await? | ||
| .into_inner(); | ||
|
|
||
| while let Some(response) = stream.message().await? { | ||
| if let Some(ContractEntry::ActiveContract(active)) = response.contract_entry | ||
| && let Some(created) = active.created_event | ||
| { | ||
| return Ok(extract_governance_state(&created)); | ||
| } | ||
| } | ||
|
|
||
| Ok(None) | ||
| .first() | ||
| .and_then(extract_governance_state)) |
Comment on lines
+402
to
+404
| // `collect_entries` already keeps governance entries only, sorted newest | ||
| // first and capped at `limit`. | ||
| let entries = match canton_filters { |
Comment on lines
+974
to
+978
| fn chain_audit_response(entries: Vec<ChainAuditEntry>, limit: usize) -> ChainAuditResponse { | ||
| let total_returned = entries.len(); | ||
| let next_before_offset = (total_returned >= limit && limit > 0) | ||
| .then(|| entries.last().map(|e| e.offset)) | ||
| .flatten(); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Canton added pagination to the bulk read endpoints in 3.5.1 (
GetActiveContractsPage,GetUpdatesPage) and 3.4.10 (ListVettedPackages). We're on 3.5.11 protos and were using none of it.What changed
server/ledger_paging.rs— one place that speaks the paged protocol, with a fallback to the old streaming calls when the participant returnsUNIMPLEMENTED(below 3.5.1). Three helpers:fetch_active_contracts,fetch_transactions_page,fetch_vetted_packages.All 25 ACS reads in
queries.rsnow go through it. The helper absorbs the client setup,GetLedgerEnd, request construction and the stream loop, soqueries.rsdrops ~590 net lines.Chain audit was streaming the whole retained ledger from the pruned offset, materialising every match, sorting, then truncating to
limit— O(entire ledger) to return 25 rows, growing without bound as the ledger ages. It now pages newest-first (descending_order) and stops as soon as it has enough.before_offset/next_before_offsetcarry a cursor through the endpoint into the UI.get_party_metadatabug —page_sizewas hardcoded to 1000 andnext_page_tokenwas never read, so on a participant hosting more than 1000 parties the lookup fell off page 1,find()returnedNone, and the caller gotOk(None): indistinguishable from "this party has no annotations". Now filters server-side viafilter_party(a prefix match, so a full id narrows to one row) and walks pages as a backstop.Frontend —
usePaginationhook +PaginationControls/CursorPagination. Applied to the audit trail (server cursor), party list, external parties, holdings, party-detail contracts, both packages tables, and the notifications feed.PAGE_SIZE = 25lives incommon::apiand is emitted intotypes.generated.tsbygen-types, so the wire page size and the UI page size are the same number and can't drift.Behaviour change to be aware of
/packages/vettedwas calling the adminListPackages, which lists every uploaded DAR — a strictly larger set than the vetted ones. It now reads real topology vetting state viaListVettedPackages. Packages that are uploaded but not vetted will disappear from that panel. That's the endpoint finally doing what its name says, but it is user-visible.Limits
_for_templatefan-out concatenates N template queries into one list and a single Canton page token can't address that. They read Canton a page at a time (bounded memory per round trip) but still return the full list, and the UI pages it at 25. True server-side paging there needs a composite cursor — not attempted here.FETCH_CHUNKis 1000, not 25.PAGE_SIZEis the wire/UI page size; using 25 as the Canton chunk when collecting a full result set would turn one stream into hundreds of round trips.tenant.rs(/v0/tenant/*/acs) is deliberately untouched — changing that response shape is a separate call.ContractsDialogis not paginated: it's an index-keyed editable form, not a data list.Compatibility
Paginated RPCs need Canton >= 3.5.1 on the participant. Devnet localnet is 3.5.8 and testnet is PV35; mainnet's version I could not verify from the repo, which is why the fallback exists rather than a hard switch.
Two edge cases worth a look in review
offset < cursor) skip the remainder. Both the live path and the cache SQL now return whole offset groups, overshootinglimitby at most one transaction's tail.Verification
cargo clippy --all-targets --workspaceclean,cargo fmt --checkclean,tsc --noEmitclean,vite buildsucceeds. ESLint adds no new errors (the repo has 63 pre-existing; the two in files I touched are identical onmain).No tests were run. The chain-audit cursor edge cases above are reasoned about and verified by inspection only.