CDDL based Node2Client LocalStateQuery API #25

Why

Users repeatedly have asked for a reliable API reference of the LocalStateQuery protocol available on the Node-to-Client (N2C) socket of the cardano-node.

The cardano-node implements this API through the ouroboros-network, ouroboros-consensus and cardano-ledger components, where the resulting protocol is comprised by messages and their binary CBOR encodings from these three components.

There are multiple clients/servers with varying completeness of this API:

• cardano-api
• ogmios
• pallas
• gouroboros and dingo
• ouroboros-miniprotocols-ts (even contains .cddl files).

Implementing them resulted in users asking for this in multiple issues scattered across repositories:

• [FEAT] - Provide CDDL specs for ledger state queries IntersectMBO/ouroboros-consensus#439 (links to reverse engineering)
• CDDL specs for ledger state queries IntersectMBO/cardano-ledger#4396
• Clarity on ledger state queries IntersectMBO/cardano-ledger#5610
• Write a CDDL spec for LocalStateQuery and LocalTxSubmission IntersectMBO/ouroboros-consensus#1370 (good comments on approach)
• Comprehensive information about Cardano wire protocol IntersectMBO/ouroboros-network#5049
• [FR] - Provide comprehensive CDDL grammar for local tx-submission protocol IntersectMBO/cardano-node#5139 (asks about tx submission, but good comments on why)
• Document serialisation used by Cardano IntersectMBO/ouroboros-consensus#7

This is a meta-issue to collect these issues and tackle them in course of the cardano-blueprint initiative. On top of these issues, there has been also repeated questions on communication channels about this API.

What

An API reference for the N2C protocols, especially the LocalStateQuery should be:

1. human readable to discover available functionality
2. machine readable to test and ensure consistency with implementations
3. easy to maintain

Concretely this means we need:

• A high-level API description like in ogmios
• An explanation of the protocol (acquiring, querying, ...) with messages defined as CDDL
• A definite list of all queries available
  • can be searched
  • can be linked to
• Each query holds an explanation (to answer questions like these)
• Each query and its possible responses are defined as a CDDL
  • Can be used to encoder/decoder implementations
  • Notable implementations like pallas, gouroboros and dingo have test suites to check conformance
• The API definition is automatically checked against the cardano-node reference implementation
  • Contributors to ouroboros-network, ouroboros-consensus and cardano-ledger get notified about inconsistencies
• The API definition is versioned, such that
  • it is clear which API is available in a given cardano-node version (or any of other client/server implementations)
  • modifications of the API can be done freely before releasing components

How

As also discussed in this and this issue, the concrete API is a combination of the ouroboros-network protocol, that delegates handling of a list of queries defined by ouroboros-consensus, which in turn often relies on cardano-ledger to provide the actual query results.

Current situation

• ouroboros-network defines a CDDL for localStateQueryMessage in local-state-query.cddl

  • This is using query = any and result = any placeholders

• ouroboros-consensus does not define a CDDL ledger queries, there is this PR though: Refine the CDDL spec of the LocalStateQuery mini-protocol IntersectMBO/ouroboros-network#5074, where (shelley era) query messages are defined in data instance BlockQuery and encoding/decoding is provided by encodeShelleyQuery and encodeShelleyResult.

  • Depending on the result type, the cardano-ledger is involved in providing the response.
  • Consensus "native" results are not explicitly covered by a CDDL

• cardano-ledger generates a composite CDDL for the conway era using cuddle into conway.cddl (with "huddle" source here)

  • It is not always clear which schema is used, e.g. the UTxO result of GetUTxOByTxIn query (needs to know how the Haskell type UTxO is mapped to CBOR)
  • This does not hold everything needed to describe query results, e.g. PParams seems not to be covered

Concrete solution idea

Create a binary, CDDL-based equivalence of the ledger state query documentation of ogmios:

• Written in markdown
• Rendered and hosted as static website
• Enumerates all queries and explains them
  • references .cddl files (see below)
  • (optional) includes examples / test data
• Create individual .cddl files for each query/result
  • References the relevant rules from cardano-ledger's conway.cddl
  • Using CDDL module extension
  • Add tests to ouroboros-consensus that test each Query
    • This requires the individual query .cddl files to be composed with conway.cddl
• Provide instructions to do similar tests for client implementations
  • (optional) Show how to generate arbitrary CBOR with cuddle to test client-side decoders
• Create a full, composite .cddl of the overall protocol
  • Can be used to explain all messages received on the wire
  • Enumerates all possible query and result combinations
  • Is based on ouroboros-network's local-state-query.cddl
    • (optional) Update ouroboros-network tests to use this composite cddl?
  • This may also require to be composed with conway.cddl
  • Using CDDL module extension

Steps

• Ouroboros Genesis IntersectMBO/ouroboros-consensus#29
• Validate example with cardano-node components
• Validate example with pallas or gouroboros clients
  • Cardano blueprint localstatequery tests txpipe/pallas#638
  • Cardano blueprint localstate query tests blinklabs-io/gouroboros#992
• Create full API description and CDDLs
  • Add NTN CDDLs IntersectMBO/ouroboros-consensus#1422

TBD

• Latest version and era is sufficient for a first pass?