chore: companion DIPs PR (do not merge yet)#1038
Draft
MoonBoi9001 wants to merge 18 commits into
Draft
Conversation
…RCA (#942) * feat(dips): implement SignedRCA validation and storage Implement RecurringCollectionAgreement (RCA) protocol for DIPS, aligned with the on-chain IndexingAgreement contract. Changes: - RcaStore trait and PostgreSQL implementation for RCA storage - EIP-712 signature verification via escrow-based authorization - validate_and_create_rca() with full validation pipeline: signature, IPFS manifest, network, pricing, deadline/expiry - Database migration for pending_rca_proposals table The indexer agent queries pending_rca_proposals directly and decides acceptance on-chain via RecurringCollector contract. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(dips): improve configuration ergonomics and validation - Add #[serde(default)] to DipsConfig for minimal config files - Validate recurring_collector != Address::ZERO at startup - Warn when tokens_per_second is empty (all proposals rejected) - Bump pricing rejection logs to info level for visibility Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs(dips): add module-level documentation Add comprehensive documentation explaining architecture, validation flow, trust model, and component responsibilities. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test(dips): expand unit test coverage to 43 tests Add comprehensive test suite with AAA pattern: - validate_and_create_rca: 11 tests covering all validation paths - PriceCalculator: 7 tests (previously 0) - SignerValidator implementations: 5 tests - Test doubles: FailingIpfsFetcher, FailingRcaStore, RejectingSignerValidator Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(dips): add IPFS fetch timeout and retry with backoff Add resilience to IPFS manifest fetching: - 30 second timeout per attempt - Up to 4 attempts with exponential backoff (10s, 20s, 40s) - Worst case: ~190 seconds before rejection Dipper gRPC timeout should be >= 220s. See edgeandnode/dipper#557. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
…etworks (#947) * feat(dips): improve config ergonomics with GRT pricing and explicit networks Changes human-readable GRT per 30 days pricing config and adds explicit network support list. Addresses #943 and #944. Config changes: - Add supported_networks list (proposals for unlisted networks rejected) - Add min_grt_per_30_days per-network base pricing (GRT/30 days) - Add min_grt_per_million_entities_per_30_days global entity pricing - Add 90+ networks with calculated pricing examples from IISA model Pricing derived from archive node costs (storage $25/TB, memory $1.50/GB, CPU $10/vCPU) divided by expected subgraph count per indexer. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(dips): add startup validation tests and fix backoff comment - Fix IPFS retry comment: actual delays are 10s, 20s, 40s (not 1s, 2s, 4s) - Add 5 tests for DIPS startup validation: - test_dips_absent_in_minimal_config - test_dips_config_defaults_recurring_collector_zero - test_dips_config_defaults_empty_supported_networks - test_dips_partial_config_uses_defaults - test_dips_maximal_config_parses Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(dips): use ceiling division to protect indexer minimums When converting GRT/30days to wei/second, truncating division caused indexers to accept slightly less than their configured minimum (up to 0.2% loss). Changed to ceiling division so minimums round UP, ensuring indexers never accept offers below their stated price floor. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When Dipper sends an RCA and times out (network partition, crash after INSERT but before response), it retries. Previously, the retry failed with a duplicate key error, causing Dipper to mark the agreement as failed even though it was stored successfully. Now uses ON CONFLICT DO NOTHING so retries succeed. Both first attempt and retry return success, enabling Dipper to safely retry without creating inconsistent state. Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
…n reasons (#954) * feat(dips): add /dips/info endpoint and rejection reasons to gRPC Add a public /dips/info HTTP endpoint on port 7600 that advertises the indexer's DIPS pricing configuration (min GRT per 30 days per network, min GRT per million entities, supported networks, and protocol version). This allows the Dipper to discover indexer pricing before sending RCA proposals. Update the gRPC protobuf to include a RejectReason enum on SubmitAgreementProposalResponse, distinguishing PRICE_TOO_LOW from OTHER rejection reasons. The server maps DipsError::TokensPerSecondTooLow and TokensPerEntityPerSecondTooLow to PRICE_TOO_LOW, with all other errors mapped to OTHER. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(dips): gate EscrowSignerValidator behind db feature The signers module was importing indexer_monitor unconditionally, but that crate is only available with the db feature. This caused compilation failures when using only the rpc feature (as dipper does). Changes: - Move EscrowSignerValidator and its imports into a conditionally compiled module (#[cfg(feature = "db")]) - Keep SignerValidator trait, NoopSignerValidator, and RejectingSignerValidator always available since they have no external dependencies - Gate escrow validator tests with #[cfg(all(test, feature = "db"))] - Restore dips_cancellation_eip712_domain function that was accidentally removed during the V2 migration (needed for backwards compatibility) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(dips): rename proto enum values to follow naming conventions The protobuf convention is to prefix enum values with the enum name. Changed PRICE_TOO_LOW -> REJECT_REASON_PRICE_TOO_LOW and OTHER -> REJECT_REASON_OTHER to match REJECT_REASON_UNSPECIFIED. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(dips): add unit tests for reject_reason_from_error Tests verify the mapping from DipsError variants to RejectReason: - TokensPerSecondTooLow -> PriceTooLow - TokensPerEntityPerSecondTooLow -> PriceTooLow - All other errors (UnsupportedNetwork, InvalidSignature, etc.) -> Other Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(dips): extract GRT formatting to helper function The format_grt() function converts wei (10^-18 GRT) to a human-readable GRT string with up to 18 decimal places, trimming trailing zeros. This removes duplicated formatting logic in the dips_info_state setup. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(dips): add tests for GRT formatting edge cases Tests cover: - Zero value - Whole numbers (1, 1000 GRT) - Small values less than 1 GRT (0.5 GRT) - Very small values (1 wei = 0.000000000000000001 GRT) - Mixed values with decimals - Trailing zeros are trimmed - Values with many decimal places - Large values with decimals Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: apply nightly rustfmt * fix: use maybe_dips_info for optional builder field * fix: make DipsInfoResponse and DipsInfoPricing public * chore: remove dips_version field from /dips/info response V1 never existed in production, so versioning is unnecessary. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: use GRT per billion entities instead of per million The entity pricing unit has been changed from "GRT per million entities" to "GRT per billion entities" for better human readability. At scale, "0.2 GRT per million entities" sounds negligible but actually translates to ~$4.50/TB/month - a meaningful cost that indexers might overlook. Using "200 GRT per billion entities" makes the cost more apparent. Changes: - Config: min_grt_per_million_entities_per_30_days -> min_grt_per_billion_entities_per_30_days - Default value: 0.2 -> 200 (same economics, just different unit) - /dips/info endpoint: field renamed in response - Internal conversion divisor: 1_000_000 -> 1_000_000_000 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(dips): add SIGNER_NOT_AUTHORISED rejection reason (#961) SignerNotAuthorised errors were mapped to RejectReason::Other, which causes dipper to block the indexer for 30 days. Signer authorization is a transient config issue that resolves once the operator registers the signer on the escrow contract, so a dedicated rejection reason allows dipper to apply a much shorter lookback window. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat(dips): align RCA struct with indexing-payments-management-audit contracts (#964) The RecurringCollector contract on the `indexing-payments-management-audit` branch removed `bytes16 agreementId` from the RCA struct and replaced it with `uint256 nonce`. Agreement IDs are now derived on-chain via `bytes16(keccak256(abi.encode(payer, dataService, serviceProvider, deadline, nonce)))`. The `deadline` and `endsAt` fields also changed from `uint256` to `uint64`. Updates the sol! struct definition, adds `derive_agreement_id`, simplifies `validate_and_create_rca` by removing fallible U256-to-u64 conversion, and updates all test RCA constructions. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * chore: remove misleading "v2" from DIPs migration and docs (#965) There is no DIPs v1 -- the off-chain voucher system was abandoned before deployment. DIPs refers exclusively to the on-chain RCA system. Rename the migration from dips_v2 to dips_pending_proposals and clean up doc comments that referenced "V2". Also clarifies the migration ownership comment in service.rs: the indexer-service does not run migrations by convention, the agent owns DDL, and the SQL files here are for local dev and tests only. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat(dips): add specific rejection reasons to gRPC proto (#966) The RejectReason proto enum only had 4 values (Unspecified, PriceTooLow, Other, SignerNotAuthorised), so 6 of the 8 validation failures in indexer-service mapped to the generic Other. Dipper uses the reason to set exclusion periods and Other gets 30 days, meaning transient issues like DeadlineExpired would incorrectly exclude an indexer for a month. Added DeadlineExpired, UnsupportedNetwork, SubgraphManifestUnavailable, UnexpectedServiceProvider, AgreementExpired, and UnsupportedMetadataVersion to the proto and updated reject_reason_from_error to map each DipsError variant to its specific reason. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Enables local image builds via `just build-image`, producing ghcr.io/graphprotocol/indexer-service-rs:local and ghcr.io/graphprotocol/indexer-tap-agent:local from the existing per-crate Dockerfiles. Lets downstream consumers (local-network) consume this repo as image tags instead of source clones.
…1029) * ci: add workflow_dispatch trigger to containers.yml Allows on-demand image builds from any branch via gh workflow run, producing :sha-<short> tags for downstream integration testing without widening the push-trigger branches list. * ci(containers): build multi-arch images (linux/amd64,linux/arm64) - Native runner per platform (ubuntu-24.04, ubuntu-24.04-arm), push-by-digest, manifest fused in a follow-up merge job. - Per-platform, per-target buildcache scopes to avoid collisions. - SHA-pin third-party actions with version comments. - Merge gate: !cancelled() && needs.build.result == 'success' + fork-PR check, so workflow_dispatch from a non-default branch doesn't leave orphan per-platform digest blobs in GHCR. - Target list owned by a small prepare job and consumed via fromJSON in build and merge. - Force type=sha,enable=true so meta.outputs.version is populated for the Inspect step on workflow_dispatch.
…tion (#983) The Solidity enum IndexingAgreementVersion has V1 as its first variant, which encodes as 0 in the ABI. The validation check was comparing against 1, causing all valid V1 proposals to be rejected with UnsupportedMetadataVersion. Test data also updated to use version 0. Companion to edgeandnode/dipper#583. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use version 0 for IndexingAgreementVersion.V1 in metadata validation The Solidity enum IndexingAgreementVersion has V1 as its first variant, which encodes as 0 in the ABI. The validation check was comparing against 1, causing all valid V1 proposals to be rejected with UnsupportedMetadataVersion. Test data also updated to use version 0. Companion to edgeandnode/dipper#583. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(dips): log proposal rejections at INFO level When an RCA proposal is rejected, log the rejection reason, error, and deployment ID (when decodable) at INFO level. Previously the rejection was returned via the gRPC response with no server-side log at INFO, making debugging difficult without access to the client. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style: fix nightly fmt Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(dips): log startup configuration at INFO level Log supported networks, recurring collector address, IPFS URL, and per-network minimum pricing when DIPs is enabled. Previously only a warning was emitted when supported_networks was empty. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: correct iterator and type errors in DIPs startup logging min_grt_per_30_days is destructured from a reference, so use .iter() instead of &ref to avoid &&BTreeMap. min_grt_per_billion_entities is GRT not Option<GRT>, so remove the if-let-Some wrapper. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(dips): add chain_id and structured fields to price rejection logs Price rejection logs now include chain_id (CAIP-2 identifier) and use structured tracing fields (offered, minimum) instead of format string interpolation. Makes it easier to filter and query rejection events in production log aggregation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add shared test vector for derive_agreement_id Pins the expected bytes16 output for a fixed set of RCA inputs. The same test vector exists in dipper (dipper-rpc/src/indexer.rs). If either repo's derivation drifts, the test fails with a message pointing to the counterpart. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Closed
The merge brought in main's lockfile pins for indexer-dips (rand 0.9.4, plus http, indexer-watcher, serde_json), but the merged Cargo.toml asks for rand 0.8 and no longer references those crates. Running cargo against the merged tree updated the lockfile to match. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The service tried at startup to check whether Horizon was active, and to talk to a separate escrow data source. Both are no longer needed — Horizon is always on, and escrow moved into the main data feed. Dropping the dead code fixes the build. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The escrow-accounts watcher is used in two places: once to wire into the router and once to drive signer validation later on. The first call consumed the value, so the second use no longer had anything to clone from. Clone at the first site. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(dips): switch to on-chain offer-based authorization The smart contract now verifies authorisation via on-chain offers at agreement acceptance, so the indexer-side signature check has no job left. The RCA struct gains a `conditions` field to match the contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(dips): cap IPFS manifest size at 5MB Real subgraph manifests are tens of kilobytes, but the IPFS hash in a proposal is chosen by the caller. Without a cap, a hostile caller can point at very large content and force the indexer to download it all. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(service): apply rate-limit and timeout to DIPs gRPC Adds two tower layers on the DIPs gRPC server: a 220-second timeout covering the IPFS retry budget plus headroom, and a 50-token-per- second global rate limit shared across callers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(dips): cut IPFS retries when many proposals are in flight When too many proposals are in flight, the IPFS retry budget lets a hostile caller hold handler slots for up to 190s. The new counter tracks in-flight requests; above 200 the IPFS client tries once. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(service): rate-limit DIPs proposals per source IP A new layer rate-limits per source IP: burst of 10, then 5 tokens per second sustained. Keyed on the peer IP via tonic's TcpConnectInfo extension. Single-source spam is cut off without touching other IPs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(dips): allowlist accepted payers via DipsConfig A new `allowed_payers` config field gates which payer addresses the indexer-service will even consider. Omit the field for legacy permissive behaviour; set an empty array to deny everyone; list addresses to allow. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(service): wrap DIPs gRPC middleware to make it cloneable The gRPC layer requires the wrapping around it to be cloneable, but the rate-limit and per-IP pieces aren't on their own. Putting a buffer around the whole chain makes it cloneable. The buffer's channel is sized so a healthy burst at the global rate never bumps it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(service): convert response body so DIPs middleware compiles Tower_governor wants axum's body type, while tonic's routing layer hands back its own; the rate-limit stack would not build. Convert in between, and turn any inner stack error into a gRPC status response rather than a dropped connection. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(config): drop deprecated [horizon] block from maximal example The [horizon] section sets a field the code now ignores; main already dropped the section from its copy of the example. Merging main back in here kept the block, so the test comparing maximal against the defaults-filled minimal failed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(dips): switch to on-chain offer-based authorization The smart contract now verifies authorisation via on-chain offers at agreement acceptance, so the indexer-side signature check has no job left. The RCA struct gains a `conditions` field to match the contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(dips): cap IPFS manifest size at 5MB Real subgraph manifests are tens of kilobytes, but the IPFS hash in a proposal is chosen by the caller. Without a cap, a hostile caller can point at very large content and force the indexer to download it all. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(service): apply rate-limit and timeout to DIPs gRPC Adds two tower layers on the DIPs gRPC server: a 220-second timeout covering the IPFS retry budget plus headroom, and a 50-token-per- second global rate limit shared across callers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(dips): cut IPFS retries when many proposals are in flight When too many proposals are in flight, the IPFS retry budget lets a hostile caller hold handler slots for up to 190s. The new counter tracks in-flight requests; above 200 the IPFS client tries once. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(service): rate-limit DIPs proposals per source IP A new layer rate-limits per source IP: burst of 10, then 5 tokens per second sustained. Keyed on the peer IP via tonic's TcpConnectInfo extension. Single-source spam is cut off without touching other IPs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(dips): allowlist accepted payers via DipsConfig A new `allowed_payers` config field gates which payer addresses the indexer-service will even consider. Omit the field for legacy permissive behaviour; set an empty array to deny everyone; list addresses to allow. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(service): wrap DIPs gRPC middleware to make it cloneable The gRPC layer requires the wrapping around it to be cloneable, but the rate-limit and per-IP pieces aren't on their own. Putting a buffer around the whole chain makes it cloneable. The buffer's channel is sized so a healthy burst at the global rate never bumps it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(service): convert response body so DIPs middleware compiles Tower_governor wants axum's body type, while tonic's routing layer hands back its own; the rate-limit stack would not build. Convert in between, and turn any inner stack error into a gRPC status response rather than a dropped connection. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(config): drop deprecated [horizon] block from maximal example The [horizon] section sets a field the code now ignores; main already dropped the section from its copy of the example. Merging main back in here kept the block, so the test comparing maximal against the defaults-filled minimal failed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(dips): drop CancelAgreement RPC and rename signed_voucher CancelAgreement was a no-op on the indexer side: the handler returned Unimplemented because cancellation now lives on-chain in the RecurringCollector contract. Dipper retried the dead-letter every 20s forever. Also rename signed_voucher to signed_rca to match the bytes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(dips): drop unused CollectPayment RPC and gateway proto CollectPayment was the off-chain voucher-era collection RPC: indexers submitted a signed work claim and dipper returned a TAP micropayment receipt. On-chain RCA collects directly via SubgraphService.collect(), so this RPC is dead. gateway.proto and its bindings can also go. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(dips): drop misleading payer allowlist The allowed_payers config gave operators false access control: the check ran on the payer field inside the proposal, which the caller writes, so anyone could spoof the address and walk past. The on-chain offer at acceptance is the real trust gate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(config): correct DIPs expansion and drop stale Horizon mode line The example listed "Decentralized Indexing Payment System" — the correct expansion of DIPs is "Direct Indexer Payments". The "Requires Horizon mode" line was stale too, since the runtime Horizon check has been dropped and the mode is now always on. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Companion to #967 on a new branch. It exists so the team can bring the DIPs work up to date with
mainwithout rewriting the history ofmain-dips, which is what #967 has been reviewed against.Motivation
The DIPs branch
main-dipshas had #967 open againstmainfor a while, and the commits on it have been through review.mainhas moved forward by roughly 50 commits since the last sync, and other PRs that build onmain-dips(currently #1009 and #1037) now have conflicts that need to be worked through. Rather than rewrite the history ofmain-dipsand lose the review record, we are using a new branchmain-dips-rebasedto do that work. Once the dependent PRs are brought up to date and a merge frommainis taken in here, this PR will replace #967.Summary
main-dips-rebasedforked frommain-dipsat its current head.mainhappens later, after feat: switch to on-chain offer-based authorization #1009 and feat: advertise DIPs prices via /dips/info endpoint #1037 land here.