diff --git a/.envrc b/.envrc new file mode 100644 index 00000000000..3550a30f2de --- /dev/null +++ b/.envrc @@ -0,0 +1 @@ +use flake diff --git a/.gitignore b/.gitignore index fa56a0bd882..f52fb03dbcd 100644 --- a/.gitignore +++ b/.gitignore @@ -21,6 +21,7 @@ result* stack.yaml.lock .ghcid .claude/settings.local.json +.direnv /.cache /db diff --git a/RELEASE.md b/RELEASE.md index 2e01cc4974e..1f9e1d0dd64 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -1,131 +1,241 @@ # Team -The release team is made up of the following: - -* *Release Squad Lead* (usually Cardano Head of Product) -* *Rotating Release Engineer* from one of the development teams -* *SRE Team* supporting deployment and CI/CD -* *Test Engineers* that focus on integration tests -* *Performance Engineers* running benchmarking tests -* *Dev Leads* for each of the components of the node +The release process is owned by **Intersect** (the Cardano MBO), which runs a +weekly Release Office Hours meeting attended by all team leads to give status +updates and plan upcoming releases. + +Code contributions and maintenance come from **IOG**, **Tweag**, and +**Ensurable Systems**, which also provide rotating release engineers. + +## Sign-offs + +| Role | Person | +|------|--------| +| Cardano Product Committee (CPC) | Samuel Leathers | +| Cardano Technical Steering Committee (TSC) | Kevin Hammond | +| Rotating Release Engineer | (varies per release) | +| QA Lead | Martin Kourim | +| SRE Lead | John Lotoski | +| Performance Lead | Michael Karg | + +A release may be published as a **GitHub pre-release** with sign-offs from +CPC, TSC, RE, and SRE. At that point QA kicks off the final integration test +suite. The remaining sign-offs (QA, Performance) are required before the +release is promoted to a full release. + +## Component Teams + +| Component | Responsible Team | +|-----------|-----------------| +| Ledger | Ledger team (IOG) | +| Consensus | Consensus team (IOG) | +| Network | Network team (IOG) | +| API & CLI | Node/API/CLI team (IOG/Tweag/Ensurable Systems) | +| Node | Rotating Release Engineer (no permanent owner) | + +# Release Cadence + +Releases target a **4–8 week** cycle. Although earlier planning aimed for +2–4 weeks, the active development of new eras (currently Dijkstra) has meant +nearly every release contains substantial changes, pushing the realistic +cadence to 4–8 weeks. # Release Process -This is the release process for node releases - -1. Release Squad Lead will work with Release Engineer and Dev Leads to determine where to cut the releases from ledger, - network and node. These will be published via CHaP and will follow the defined process of the team for versioning, etc... -2. These will be integrated up the stack to the node to produce a release branch -3. Release Engineer will work with Test/Performance sub-teams to initiate a testing round. - This may use a tagged release, a commit taken from master, - or a hotfix branch based on a previously released version. -4. A release candidate will be deployed to preview/preprod by SRE -5. Community will be notified of release candidate and given a few days to provide feedback -6. Release Engineer will create a draft GitHub Release containing an empty template for the notes -7. Release Engineer will ask Dev Leads to fill out their section of the high-level changelog in the release notes -8. Release Squad Lead will finalize the Release notes -9. Head of Engineering will meet with release team and identify if release should be published as stable or beta (pre-release) - in GitHub. - Stable releases can go all the way to mainnet, beta should only go to preprod and wait for that release to be relabeled as stable or a new stable release to be cut. -10. GitHub release is published +This is the release process for node releases. + +1. The weekly Intersect release meeting determines which commits or branches + to integrate, including any in-flight PRs that need to be included. +2. The Release Engineer works with component dev leads to integrate + dependencies bottom-up through the stack: + Ledger → Consensus → Network → API → CLI → Node. +3. Components pass their own property tests, golden tests, unit tests, and + integration tests before integration proceeds up the stack. Integration + tests using the `cardano-testnet` package (in this repository) provide the + primary integration signal at the node level. +4. When the full stack is stable, components are released to CHaP bottom-up + and the Release Engineer replaces source-repository-package (SRP) stanzas + with the CHaP releases, working up the stack until the node is ready. +5. SRE deploys the release candidate to preview/preprod networks. +6. The community is notified of the release candidate and given time to + provide feedback. +7. The Release Engineer creates a draft GitHub Release and asks dev leads to + fill in their changelog sections. +8. CPC, TSC, Release Engineer, and SRE Lead sign off → the release is + published as a **GitHub pre-release**. +9. QA runs the final integration test suite (`cardano-node-tests`). +10. Performance testing is run against the final tag + (see [Performance Testing](#performance-testing)). +11. SRE escalates deployments through all remaining environments up to + mainnet and monitors for issues. No new SRE sign-off is required — + their pre-release approval carries over — but any issues observed + during this window are reported back and can block promotion. +12. QA Lead and Performance Lead sign off → the GitHub release is + **promoted from pre-release to full release**. This is the release process for node release hot fixes: -1. **Create a hotfix branch** - - **Branch naming format**: `release/X.Y.Z` (e.g., for a hotfix on cabal version `10.5` already tagged and released as `10.5.0`, name it `release/10.5.1`). - - **Important**: CI will only run if the branch follows this exact naming convention. +1. **Create a hotfix branch** + - **Branch naming format**: `release/X.Y.Z` (e.g., for a hotfix on + `10.5.0` already released, name it `release/10.5.1`). + - **Important**: CI will only run if the branch follows this exact naming + convention. + +2. **Update the version** + - Bump the Node version in the `cabal` file to reflect the hotfix changes. + +3. **Follow the standard release process** + Resume from **Step 5** of the [normal release process](#release-process). + +# Testing + +## Continuous Testing (throughout integration) + +The bulk of testing happens during integration, at each layer of the stack: + +- **Property tests** — extensive property-based test suites in each component +- **Unit tests** — per-package unit tests +- **Golden tests** — encoding and serialisation stability tests +- **Integration tests** — the `cardano-testnet` package in this repository + provides the primary integration test suite + +## Final Integration Tests (post pre-release tag) + +Once CPC, TSC, and the Release Engineer have signed off and the pre-release +tag is cut, the QA team runs the `cardano-node-tests` suite — Python wrappers +that exercise end-to-end scenarios against the tagged binaries. + +## Performance Testing + +Performance tests are run against the final tagged version only. The typical +workload takes around 48 hours, though this is not a guaranteed SLA. Two +workloads are used: -2. **Update the version** - - Bump the Node version in the `cabal` file to reflect the hotfix changes. +- **Value workload** — raw payment transactions (throughput and latency + baseline) +- **Plutus workload** — intensive Plutus scripts designed to stress the system -3. **Follow the standard release process** - Resume from **Step 3** of the [normal release process](#release-process). +Results are reviewed and signed off by the Performance Lead before the release +is promoted to full release. # Rotating Release Engineer Role -All sprints are aligned across the node and its components. At the end of a sprint cycle the new rotating Release Engineer is decided on by the leadership team. -This person's primary duties are integration of new releases of dependencies up the stack to the node. They serve this role until the release is finalized -according to the above release process (ideally 1 sprint cycle). -The Release Engineer works with the Dev Leads to decide which changes should be included in their component, including any in-flight PRs that haven't been merged yet. +The Release Engineer is drawn from IOG, Tweag, or Ensurable Systems +development teams on a rotating basis, with a new engineer assigned from a +*different* team each release. They serve until the release is fully signed +off. + +Their primary responsibility is integrating component releases bottom-up +through the stack into a releasable node. They coordinate with dev leads, +triage build failures, cherry-pick required in-flight PRs to release branches, +and drive the release to completion. + +The Release Engineer works with the dev leads to decide which changes to +include in each component, including in-flight PRs not yet merged to `master`. ## Sub-Teams ### SRE (Site Reliability & Engineering) -The SRE team provides the tooling for monitoring, logging and measurement of live environments. The team initiates deployments of new versions to developer -testnets, public testnets and production systems. They are responsible for updating dashboards/alerts to align with new node features/refactoring. +The SRE team provides tooling for monitoring, logging, and measurement of live +environments. They initiate deployments to developer testnets, public +testnets, and production systems, and update dashboards and alerts to align +with new node features. + +The [Cardano World book site](https://book.play.dev.cardano.org/) hosts +configuration files for every release. Full releases appear in the main +release section; pre-releases have a dedicated pre-release section on the +site. ### DevX (Developer Experience) -The DevX team is responsible for CI/CD, the building process (using nix and compiling manually), OCI images (e.g. docker containers), systemd services, -and helper scripts associated in running the node for local development and remote deployment purposes. +DevX is responsible for CI/CD, the Nix build system, OCI images (Docker +containers), systemd services, and helper scripts for running the node locally +and remotely. ### Test Engineers -The test engineers are responsible for writing and running integration tests from `cardano-node-tests` repository. They execute integration tests as well as -tests that measure node synchronization times between releases. +Test engineers write and run integration tests from the `cardano-node-tests` +repository, and measure node synchronisation times between releases. ### Performance Engineers -Performance engineers run benchmarks of the node and report any improvements/regressions between node versions. +Performance engineers run benchmarks of the node and report improvements and +regressions between releases. # Versioning -The node uses a "pseudo semantic versioning" that takes into account breaking change (e.g. new eras) in the versioning logic. This new versioning -standard within the node is supported as of `8.0.0` release. The first part of the version always references the max protocol version allowed -on a stable network with no additional experimental override flags. This first part will remain `8` referencing Babbage era until the next era (Conway) -is finalized. The second digit is a new release incremented counter. Every release based off the master trunk will increment this number by one. The final -part should always be `0` *unless* an emergency bug fix is necessary that cannot wait until the next major release. This should be a rare occurrence going -forward and is used for forking a new version off a previous release and backporting fixes. +The node uses "pseudo semantic versioning": -Not all releases are declared stable. Releases that aren't stable will be released as *pre-releases* and will append a `-pre` tag indicating it is not ready -for running on production networks. The same version can be re-released without the pre tag without making any code changes by submitting a new tag without the -`pre` suffix. This means stable could jump from `8.0.0 -> 8.3.0` without ever releasing `8.1.0`, `8.1.1`, `8.2.0`, etc... +- **Major** (`X`) — references the maximum protocol version active on stable + networks. This increments when a new era is fully activated. +- **Minor** (`Y`) — incremented for each regular release cut from `master`. +- **Patch** (`Z`) — normally `0`; incremented only for emergency hotfix + releases branched from a prior tag. -Note that the version always has three parts, so a major release has a trailing `.0`. +## Pre-release vs Full Release + +All releases are tagged with the plain `X.Y.Z` format — there is no `-pre` +suffix in tags. Whether a release is a pre-release or a full release is +indicated solely by its **GitHub release type**. A pre-release may be promoted +to a full release in GitHub without any code changes or new tags. + +Example: `10.2.1` was published as a GitHub pre-release for approximately one +month before being promoted to a full release. The tag was always `10.2.1`; +only the GitHub release type changed. # Collaboration -The release team meets for a quick touch-point weekly where all team leads are invited. Currently these calls are closed to the public, but in the future we expect -to open them up to the larger community. The release team also meets ad-hoc as needed and collaborates asynchronously throughout the week. +Intersect runs a weekly **Release Office Hours** meeting that all team leads +attend. Async coordination uses the `#cardano-release-tech` Slack channel. -# Release notes +# Release Notes -The release notes are drafted and published via the GitHub releases UI. -Our current template contains the following sections. +Release notes are drafted and published via the GitHub releases UI. The +standard template contains: -- (no header) A very-high level summary of the release. - For a larger release, it may be best for the Cardano Head of Product to draft this summary instead of the Release Engineer, since they have more context. +- (no header) A high-level summary of the release. + For a larger release it may be best for CPC to draft this summary, since + they have more context. - Known Issues -- Technical Specification (usually unchanged) +- Technical Specification (usually unchanged between releases) - Links to `cardano-node` documentation - It seems to be a judgement call whether each of these should specify the upstream version. - Changelogs - + Summaries of the major dependencies' changelogs. - These are written as a few sentences that an interested user and/or dev would find helpful. - It may be best for the individual teams to draft these summaries instead of the Release Engineer, since they have more context. - + Links to the individual changelog of each upstream package that IOG maintains. - See the script explained below. + + Summaries of each major dependency's changes, written for an interested + user or developer. These are best drafted by the individual component + teams rather than the Release Engineer. + + Links to the individual changelog of each upstream package that IOG + maintains. See the script explained below. -Usually the release notes from the previous release are copied and used as a template. +The previous release's notes are typically used as a starting template. -## Detailed changelog table +## Detailed Changelog Table -There's a script (`scripts/generate-release-changelog-links.hs`) that generates a table of changelogs for each of the package versions included in a given `cardano-node` release. The script takes a cabal-generated `plan.json` and a GitHub API access token, and outputs a large table which contains links to the `CHANGELOG.md` file (if one exists) for each of the package versions contained in the build plan. +`generate-release-changelog-links` generates a table of changelogs for every +package version in a given `cardano-node` build plan. It reads `GITHUB_TOKEN` +from the environment (generate one at https://github.com/settings/tokens or +run `gh auth token`). -Example usage: +Nix devshell users have the binary available directly: ```shellsession -$ nix build .#project.x86_64-linux.plan-nix.json -... -$ scripts/generate-release-changelog-links.hs -- -o links.md result-json $GITHUB_API_TOKEN -... +$ nix build .#project.x86_64-linux.plan-nix +$ export GITHUB_TOKEN=$(gh auth token) +$ generate-release-changelog-links -o links.md ``` -For more information, including how to generate / retrieve a GitHub API token, use +`result/plan.json` is the default plan path. Pass an explicit path as the +first argument to override it. +Non-nix users can build and run via cabal: + +```shellsession +$ export GITHUB_TOKEN=$(gh auth token) +$ cabal run generate-release-changelog-links -- -o links.md ``` -scripts/generate-release-changelog-links.hs -- --help -``` -Note that this is a cabal script and may take a while to build all the dependencies. You will need to have the `zlib` native library available, either installed via your OS package manager or by using `nix-shell -p zlib`. +For full usage: + +```shellsession +$ generate-release-changelog-links --help +``` diff --git a/cabal.project b/cabal.project index 7d699e8d362..b33f9cd7315 100644 --- a/cabal.project +++ b/cabal.project @@ -35,6 +35,7 @@ package acts flags: -finitary packages: + scripts/generate-release-changelog-links cardano-node cardano-node-capi cardano-node-chairman diff --git a/nix/haskell.nix b/nix/haskell.nix index 372101c7323..3254ce105ee 100644 --- a/nix/haskell.nix +++ b/nix/haskell.nix @@ -66,6 +66,7 @@ let shellcheck scriv stylish-haskell + config.hsPkgs.generate-release-changelog-links.components.exes.generate-release-changelog-links ]; withHoogle = true; diff --git a/scripts/generate-release-changelog-links.hs b/scripts/generate-release-changelog-links.hs deleted file mode 100755 index bee0d0a491f..00000000000 --- a/scripts/generate-release-changelog-links.hs +++ /dev/null @@ -1,323 +0,0 @@ -#!/usr/bin/env -S cabal --verbose=1 --index-state=2025-04-16T18:30:40Z run -- -{- cabal: - build-depends: - base, - aeson, - bytestring, - cabal-plan, - case-insensitive, - containers, - foldl, - github ^>= 0.29, - http-client, - http-types, - network-uri, - optparse-applicative ^>= 0.18, - ansi-wl-pprint >= 1, - prettyprinter, - req, - text, - turtle ^>= 1.6.0, - uri-encode, - default-extensions: - BlockArguments, - DataKinds, - ImportQualifiedPost, - LambdaCase, - OverloadedStrings, - RecordWildCards, - ScopedTypeVariables - ghc-options: -Wall -Wextra -Wcompat --} - --- `nix build .#project.x86_64-linux.plan-nix.json` is a reliable way to --- generate the `plan.json` to be fed to this script. - -module Main (main) where - -import qualified Control.Foldl as Foldl -import Data.Aeson -import Data.ByteString.Char8 (ByteString) -import qualified Data.CaseInsensitive as CI -import Data.Foldable -import qualified Data.List as List -import Data.Map.Strict (Map) -import qualified Data.Map.Strict as Map -import Data.Maybe -import qualified Data.Text as Text -import qualified Data.Text.Encoding as Text -import qualified Data.Text.IO as Text -import Data.Version -import Network.HTTP.Client (HttpException (..), HttpExceptionContent (..), - responseHeaders, responseStatus) -import Network.HTTP.Req -import Network.HTTP.Types.Header (hLocation) -import Network.HTTP.Types.Status (found302) -import qualified Network.URI as URI -import qualified Network.URI.Encode as URIE -import Options.Applicative -import Prettyprinter -import qualified Prettyprinter.Util as PP - -import Cabal.Plan -import qualified GitHub -import Turtle - -main :: IO () -main = sh do - - (outputPath, planJsonFilePath, gitHubAccessToken) <- - options generateReleaseChangelogLinksDescription $ - (,,) <$> optPath "output" 'o' "Write the generated links to OUTPUT" - <*> argPath "plan_json_path" "Path of the plan.json file" - <*> fmap (GitHubAccessToken . Text.encodeUtf8) (argText "github_access_token" "GitHub personal access token") - - packagesMap <- getCHaPPackagesMap - - changelogPaths <- reduce Foldl.list do - - -- find all of the packages in the plan.json that are hosted on CHaP - printf ("Reading Cabal plan from "%w%"\n") planJsonFilePath - version@(PkgId n v) <- nub $ selectPackageVersion planJsonFilePath - - -- from cardano-haskell-packages, retrieve the package repo / commit / subdir - printf ("Looking up CHaP entry for "%repr version%"\n") - chapEntry <- lookupCHaPEntry version packagesMap - - -- from github, get the package's CHANGELOG.md location - printf ("Searching for CHANGELOG.md on GitHub for "%repr version%"\n") - changelogLocation <- findChangelogFromGitHub gitHubAccessToken chapEntry - - pure (n, v, changelogLocation) - - -- generate a massive markdown table - let res = generateMarkdown changelogPaths - liftIO . Text.writeFile outputPath $ format (s%"\n") res - -generateReleaseChangelogLinksDescription :: Description -generateReleaseChangelogLinksDescription = Description $ - mconcat - [ "generate-release-changelog-links.hs" - , line, line - , fillSep $ PP.words - "This script requires a GitHub personal access token, which can be \ - \generated either at https://github.com/settings/tokens or retrieved \ - \using the GitHub CLI tool with `gh auth token` (after logging in)" - ] - -selectPackageVersion :: FilePath -> Shell PkgId -selectPackageVersion planJsonFilePath = do - cabalPlan <- liftIO do - eitherDecodeFileStrict planJsonFilePath >>= \case - Left aesonError -> - die $ "Failed to parse plan.json: " <> fromString aesonError - Right res -> pure res - - Unit{..} <- select (pjUnits cabalPlan) - - -- we only care about packages which are hosted on CHaP - guard (isProbablyCHaP Unit{..}) - - pure uPId - -hackageURI :: URI -hackageURI = - URI "http://hackage.haskell.org/" - -isProbablyCHaP :: Unit -> Bool -isProbablyCHaP Unit{..} = - case uPkgSrc of - Just (RepoTarballPackage (RepoSecure repoUri)) -> repoUri /= hackageURI - _ -> False - -newtype CHaPPackages = CHaPPackages [PackageDescription] - deriving (Show, Eq, Ord) - -instance FromJSON CHaPPackages where - parseJSON v = CHaPPackages <$> parseJSON v - -data PackageDescription = PackageDescription - { packageName :: Text - , packageVersion :: Version - , packageURL :: Text - } - deriving (Show, Eq, Ord) - -instance FromJSON PackageDescription where - parseJSON = withObject "PackageDescription" $ \obj -> do - PackageDescription <$> obj .: "pkg-name" - <*> obj .: "pkg-version" - <*> obj .: "url" - -getCHaPPackages :: MonadIO m => m CHaPPackages -getCHaPPackages = do - fmap responseBody $ liftIO $ runReq defaultHttpConfig $ - req GET chapPackagesURL NoReqBody jsonResponse mempty - -type PackagesMap = Map (Text, Version) Text - -getCHaPPackagesMap :: MonadIO m => m PackagesMap -getCHaPPackagesMap = do - CHaPPackages ps <- getCHaPPackages - pure $ Map.fromList $ - map (\PackageDescription{..} -> ((packageName, packageVersion), packageURL)) ps - -chapPackagesURL :: Url 'Https -chapPackagesURL = - https "chap.intersectmbo.org" /: "foliage" /: "packages.json" - -lookupCHaPEntry :: PkgId -> PackagesMap -> Shell CHaPEntry -lookupCHaPEntry (PkgId (PkgName n) (Ver v)) packagesMap = do - chapURL <- maybe empty pure $ Map.lookup (n, Version v []) packagesMap - - case match packagesJSONUrlPattern chapURL of - [] -> do - printf ("Skipping "%repr n%" as its packages.json URL could not be parsed\n") - empty - chapEntry : _ -> - pure chapEntry - --- parses something like this: --- github:input-output-hk/cardano-ledger/760a73e89ef040d3ad91b4b0386b3bbace9431a9?dir=eras/byron/ledger/executable-spec -packagesJSONUrlPattern :: Pattern CHaPEntry -packagesJSONUrlPattern = do - void "github:" - owner <- plus (alphaNum <|> char '-') - void "/" - repo <- plus (alphaNum <|> char '-') - void "/" - revision <- plus hexDigit - subdir <- optional do - void "?dir=" - plus (alphaNum <|> char '.' <|> char '/' <|> char '-') - eof - pure $ CHaPEntry (GitHub.mkOwnerName owner) (GitHub.mkRepoName repo) revision subdir - -data CHaPEntry = - CHaPEntry { entryGitHubOwner :: GitHub.Name GitHub.Owner - , entryGitHubRepo :: GitHub.Name GitHub.Repo - , entryGitHubRevision :: Text - , entrySubdir :: Maybe Text - } - deriving (Show) - -findChangelogFromGitHub :: MonadIO m => GitHubAccessToken -> CHaPEntry -> m (Maybe (Text, Text)) -findChangelogFromGitHub accessToken c@CHaPEntry{..} = do - liftIO $ print c - let query = changelogLookupGitHub entryGitHubOwner entryGitHubRepo entrySubdir entryGitHubRevision - liftIO $ print query - contentDir <- liftIO (runGitHub accessToken query) >>= \case - Left (GitHub.HTTPError originalError@(HttpExceptionRequest _originalReq (StatusCodeException resp _))) -> do - if responseStatus resp == found302 - then do - let responseHeaders' = responseHeaders resp - case List.lookup hLocation responseHeaders' of - Nothing -> die "findChangelogFromGitHub: Got HTTP 302 redirect but no location header found" - Just redirectLocation -> do - - -- We must construct the redirect URL - -- We drop 2 characters at the end because the location appears to be malformed - let responseLocation = URIE.decodeText $ Text.dropEnd 2 $ Text.decodeUtf8 redirectLocation - finalResponseQueryURl = responseLocation - - newLocationQuery <- case query of - GitHub.Query _ queryString -> do - redirectPathSegments <- generateRedirectPathSegments finalResponseQueryURl - pure $ GitHub.query redirectPathSegments queryString - unexpected -> die $ "findChangelogFromGitHub: Expected a Query type but got: " <> repr unexpected - - r <- liftIO (runGitHub accessToken newLocationQuery) - case r of - Left e' -> die $ Text.unlines [ "Redirect failed: " <> repr e' - , "Original http error: " <> repr originalError - ] - Right (GitHub.ContentFile _) -> die - "Redirect result: Expected changelogLookupGitHub to return a directory, but got a single file" - Right (GitHub.ContentDirectory dir) -> pure dir - - else die $ - "GitHub lookup failed with HTTP exception: " <> Text.pack (show resp) - Left gitHubError -> die $ - "GitHub lookup failed with error " <> repr gitHubError - Right (GitHub.ContentFile _) -> die - "Expected changelogLookupGitHub to return a directory, but got a single file" - Right (GitHub.ContentDirectory dir) -> pure dir - - pure $ case Data.Foldable.find looksLikeChangelog contentDir of - Nothing -> Nothing - Just res -> do - let name = GitHub.contentName (GitHub.contentItemInfo res) - path = GitHub.contentPath (GitHub.contentItemInfo res) - Just (name, constructGitHubPath entryGitHubOwner entryGitHubRepo entryGitHubRevision path) - -generateRedirectPathSegments :: MonadIO m => Text -> m [Text] -generateRedirectPathSegments url = - case URI.parseURI (Text.unpack url) of - Just uri -> - let segments = map Text.pack $ URI.pathSegments uri - in if null segments - then die $ "generateRedirectPathSegments: No path segments found in URL: " <> url - else return segments - Nothing -> die $ "generateRedirectPathSegments: Invalid URL: " <> url - - -changelogLookupGitHub :: GitHub.Name GitHub.Owner - -> GitHub.Name GitHub.Repo - -> Maybe Text - -> Text - -> GitHub.Request k GitHub.Content -changelogLookupGitHub owner repo subdir revision = - GitHub.contentsForR owner repo (fromMaybe "" subdir) (Just revision) - -looksLikeChangelog :: GitHub.ContentItem -> Bool -looksLikeChangelog GitHub.ContentItem{..} = do - let caseInsensitiveName = CI.mk (GitHub.contentName contentItemInfo) - contentItemType == GitHub.ItemFile && caseInsensitiveName == "CHANGELOG.md" - -constructGitHubPath :: GitHub.Name GitHub.Owner - -> GitHub.Name GitHub.Repo - -> Text - -> Text - -> Text -constructGitHubPath = - format ("https://github.com/"%ghname%"/"%ghname%"/blob/"%s%"/"%s) - where - ghname = makeFormat GitHub.untagName - -newtype GitHubAccessToken = GitHubAccessToken ByteString - deriving (Show, Eq, Ord) - -runGitHub :: GitHub.GitHubRW req res => GitHubAccessToken -> req -> res -runGitHub (GitHubAccessToken tok) = - GitHub.github (GitHub.OAuth tok) - -generateMarkdown :: [(PkgName, Ver, Maybe (Text, Text))] -> Text -generateMarkdown changelogPaths = - let - rows = mkHeader : map mkRow changelogPaths - table = render rows - in Text.unlines $ "Package changelogs" : "" : table - where - mkHeader = ["Package", "Version", "Changelog"] - mkRow (PkgName n, v, linkMaybe) = [n , dispVer v, dispLink linkMaybe] - - -- example result: [CHANGELOG.md](https://github.com/IntersectMBO/cardano-base/blob/f11ddc7f/cardano-slotting/CHANGELOG.md "CHANGELOG.md") - dispLink (Just (file, link)) = format ("["%s%"]("%s%" \""%s%"\")") file link file - dispLink Nothing = "" - - render :: [[Text]] -> [Text] - render = map renderRow . List.transpose . map (separator . innerMargins . alignLeft) . List.transpose - where - renderRow = surroundWith '|' . Text.intercalate "|" - - alignLeft ts = - let maxLen = maximum (Text.length <$> ts) - in map (Text.justifyLeft maxLen ' ') ts - - surroundWith c = Text.cons c . flip Text.snoc c - - innerMargins = map (surroundWith ' ') - - -- insert separator line after the first entry (assumed to be the header in its final width) - separator (h:rs) = h : Text.replicate (Text.length h) "-" : rs - separator [] = [] diff --git a/scripts/generate-release-changelog-links/Main.hs b/scripts/generate-release-changelog-links/Main.hs new file mode 100644 index 00000000000..4007e23a1c7 --- /dev/null +++ b/scripts/generate-release-changelog-links/Main.hs @@ -0,0 +1,320 @@ +module Main (main) where + +import Control.Monad (guard) +import Data.Aeson +import Data.ByteString.Char8 (ByteString) +import qualified Data.CaseInsensitive as CI +import qualified Data.Foldable as Foldable +import qualified Data.List as List +import Data.Map.Strict (Map) +import qualified Data.Map.Strict as Map +import Data.Maybe +import Data.Text (Text) +import qualified Data.Text as Text +import qualified Data.Text.Encoding as Text +import qualified Data.Text.IO as Text +import Data.Version +import Network.HTTP.Client (HttpException (..), HttpExceptionContent (..), + responseHeaders, responseStatus) +import Network.HTTP.Req +import Network.HTTP.Types.Header (hLocation) +import Network.HTTP.Types.Status (found302) +import qualified Network.URI as URI +import qualified Network.URI.Encode as URIE +import Options.Applicative +import System.Environment (lookupEnv) +import System.Exit (exitFailure) +import System.IO (hPutStrLn, stderr) + +import Cabal.Plan +import qualified GitHub + +die :: Text -> IO a +die msg = Text.hPutStrLn stderr msg >> exitFailure + +main :: IO () +main = do + (outputPath, planJsonFilePath) <- execParser parserInfo + + gitHubAccessToken <- + lookupEnv "GITHUB_TOKEN" >>= \case + Nothing -> die "GITHUB_TOKEN environment variable is not set" + Just t -> pure $ GitHubAccessToken (Text.encodeUtf8 (Text.pack t)) + + packagesMap <- getCHaPPackagesMap + + hPutStrLn stderr $ "Reading Cabal plan from " <> show planJsonFilePath + versions <- selectPackageVersion planJsonFilePath + + changelogPaths <- + concat + <$> mapM (processVersion packagesMap gitHubAccessToken) (List.nub versions) + + let res = generateMarkdown changelogPaths + Text.writeFile outputPath (res <> "\n") + +processVersion + :: PackagesMap + -> GitHubAccessToken + -> PkgId + -> IO [(PkgName, Ver, Maybe (Text, Text))] +processVersion packagesMap gitHubAccessToken version@(PkgId n v) = do + hPutStrLn stderr $ "Looking up CHaP entry for " <> show version + mEntry <- lookupCHaPEntry version packagesMap + case mEntry of + Nothing -> return [] + Just chapEntry -> do + hPutStrLn stderr $ + "Searching for CHANGELOG.md on GitHub for " <> show version + changelogLocation <- findChangelogFromGitHub gitHubAccessToken chapEntry + return [(n, v, changelogLocation)] + +parserInfo :: ParserInfo (FilePath, FilePath) +parserInfo = info (argsParser <**> helper) $ + progDesc "Generate a Markdown changelog table for a cardano-node release" + <> footer + "Requires GITHUB_TOKEN in the environment. The token can be generated \ + \at https://github.com/settings/tokens, or retrieved via \ + \`gh auth token` after logging in with the GitHub CLI." + +argsParser :: Parser (FilePath, FilePath) +argsParser = + (,) + <$> strOption + ( long "output" + <> short 'o' + <> metavar "OUTPUT" + <> help "Write the generated links to OUTPUT" + ) + <*> argument str + ( metavar "plan_json_path" + <> help "Path of the plan.json file" + <> value "result/plan.json" + <> showDefault + ) + +selectPackageVersion :: FilePath -> IO [PkgId] +selectPackageVersion planJsonFilePath = do + cabalPlan <- eitherDecodeFileStrict planJsonFilePath >>= \case + Left aesonError -> + die $ "Failed to parse plan.json: " <> Text.pack aesonError + Right res -> pure res + return + [ uPId + | Unit{..} <- Map.elems (pjUnits cabalPlan) + , isProbablyCHaP Unit{..} + ] + +hackageURI :: URI +hackageURI = URI "http://hackage.haskell.org/" + +isProbablyCHaP :: Unit -> Bool +isProbablyCHaP Unit{..} = + case uPkgSrc of + Just (RepoTarballPackage (RepoSecure repoUri)) -> repoUri /= hackageURI + _ -> False + +newtype CHaPPackages = CHaPPackages [PackageDescription] + deriving (Show, Eq, Ord) + +instance FromJSON CHaPPackages where + parseJSON v = CHaPPackages <$> parseJSON v + +data PackageDescription = PackageDescription + { packageName :: Text + , packageVersion :: Version + , packageURL :: Text + } + deriving (Show, Eq, Ord) + +instance FromJSON PackageDescription where + parseJSON = withObject "PackageDescription" $ \obj -> + PackageDescription <$> obj .: "pkg-name" + <*> obj .: "pkg-version" + <*> obj .: "url" + +getCHaPPackages :: IO CHaPPackages +getCHaPPackages = + fmap responseBody $ runReq defaultHttpConfig $ + req GET chapPackagesURL NoReqBody jsonResponse mempty + +type PackagesMap = Map (Text, Version) Text + +getCHaPPackagesMap :: IO PackagesMap +getCHaPPackagesMap = do + CHaPPackages ps <- getCHaPPackages + pure $ Map.fromList $ + map (\PackageDescription{..} -> ((packageName, packageVersion), packageURL)) + ps + +chapPackagesURL :: Url 'Https +chapPackagesURL = + https "chap.intersectmbo.org" /: "foliage" /: "packages.json" + +lookupCHaPEntry :: PkgId -> PackagesMap -> IO (Maybe CHaPEntry) +lookupCHaPEntry (PkgId (PkgName n) (Ver v)) packagesMap = + case Map.lookup (n, Version v []) packagesMap of + Nothing -> return Nothing + Just chapURL -> + case parseCHaPEntry chapURL of + Nothing -> do + hPutStrLn stderr $ + "Skipping " <> show n + <> " as its packages.json URL could not be parsed" + return Nothing + Just chapEntry -> return (Just chapEntry) + +-- Parses CHaP package URLs of the form: +-- github:owner/repo/revision[?dir=subdir] +parseCHaPEntry :: Text -> Maybe CHaPEntry +parseCHaPEntry chapURL = do + rest <- Text.stripPrefix "github:" chapURL + let (owner, rest1) = Text.breakOn "/" rest + rest2 <- Text.stripPrefix "/" rest1 + let (repo, rest3) = Text.breakOn "/" rest2 + rest4 <- Text.stripPrefix "/" rest3 + let (rev, rest5) = Text.breakOn "?dir=" rest4 + subdir = if Text.null rest5 + then Nothing + else Just (Text.drop (Text.length "?dir=") rest5) + guard $ + not (Text.null owner) && not (Text.null repo) && not (Text.null rev) + pure $ CHaPEntry + (GitHub.mkOwnerName owner) + (GitHub.mkRepoName repo) + rev + subdir + +data CHaPEntry = CHaPEntry + { entryGitHubOwner :: GitHub.Name GitHub.Owner + , entryGitHubRepo :: GitHub.Name GitHub.Repo + , entryGitHubRevision :: Text + , entrySubdir :: Maybe Text + } + deriving (Show) + +findChangelogFromGitHub + :: GitHubAccessToken -> CHaPEntry -> IO (Maybe (Text, Text)) +findChangelogFromGitHub accessToken c@CHaPEntry{..} = do + print c + let query = changelogLookupGitHub + entryGitHubOwner entryGitHubRepo + entrySubdir entryGitHubRevision + contentDir <- runGitHub accessToken query >>= \case + Left (GitHub.HTTPError + originalError@(HttpExceptionRequest _ + (StatusCodeException resp _))) -> + if responseStatus resp == found302 + then + case List.lookup hLocation (responseHeaders resp) of + Nothing -> + die "findChangelogFromGitHub: HTTP 302 with no location header" + Just redirectLocation -> do + let loc = URIE.decodeText + $ Text.dropEnd 2 + $ Text.decodeUtf8 redirectLocation + newQuery <- case query of + GitHub.Query _ queryString -> do + segs <- generateRedirectPathSegments loc + pure $ GitHub.query segs queryString + _ -> + die "findChangelogFromGitHub: expected a Query request type" + runGitHub accessToken newQuery >>= \case + Left e' -> die $ Text.unlines + [ "Redirect failed: " <> Text.pack (show e') + , "Original error: " <> Text.pack (show originalError) + ] + Right (GitHub.ContentFile _) -> + die "Redirect result: expected a directory, got a file" + Right (GitHub.ContentDirectory dir) -> pure dir + else die $ + "GitHub lookup failed: " <> Text.pack (show resp) + Left gitHubError -> + die $ "GitHub lookup failed: " <> Text.pack (show gitHubError) + Right (GitHub.ContentFile _) -> + die "Expected a directory from changelogLookupGitHub, got a file" + Right (GitHub.ContentDirectory dir) -> pure dir + + pure $ case Foldable.find looksLikeChangelog contentDir of + Nothing -> Nothing + Just res -> + let name = GitHub.contentName (GitHub.contentItemInfo res) + path = GitHub.contentPath (GitHub.contentItemInfo res) + in Just (name, constructGitHubPath + entryGitHubOwner entryGitHubRepo + entryGitHubRevision path) + +generateRedirectPathSegments :: Text -> IO [Text] +generateRedirectPathSegments url = + case URI.parseURI (Text.unpack url) of + Just uri -> + let segs = map Text.pack $ URI.pathSegments uri + in if null segs + then die $ "No path segments in URL: " <> url + else return segs + Nothing -> die $ "Invalid URL: " <> url + +changelogLookupGitHub + :: GitHub.Name GitHub.Owner + -> GitHub.Name GitHub.Repo + -> Maybe Text + -> Text + -> GitHub.Request k GitHub.Content +changelogLookupGitHub owner repo subdir revision = + GitHub.contentsForR owner repo (fromMaybe "" subdir) (Just revision) + +looksLikeChangelog :: GitHub.ContentItem -> Bool +looksLikeChangelog GitHub.ContentItem{..} = + contentItemType == GitHub.ItemFile + && CI.mk (GitHub.contentName contentItemInfo) == "CHANGELOG.md" + +constructGitHubPath + :: GitHub.Name GitHub.Owner + -> GitHub.Name GitHub.Repo + -> Text + -> Text + -> Text +constructGitHubPath owner repo revision path = + "https://github.com/" + <> GitHub.untagName owner <> "/" + <> GitHub.untagName repo <> "/blob/" + <> revision <> "/" + <> path + +newtype GitHubAccessToken = GitHubAccessToken ByteString + deriving (Show, Eq, Ord) + +runGitHub :: GitHub.GitHubRW req res => GitHubAccessToken -> req -> res +runGitHub (GitHubAccessToken tok) = GitHub.github (GitHub.OAuth tok) + +generateMarkdown :: [(PkgName, Ver, Maybe (Text, Text))] -> Text +generateMarkdown changelogPaths = + Text.unlines $ "Package changelogs" : "" : render rows + where + rows = mkHeader : map mkRow changelogPaths + + mkHeader = ["Package", "Version", "Changelog"] + mkRow (PkgName n, v, linkMaybe) = [n, dispVer v, dispLink linkMaybe] + + dispLink (Just (file, link)) = + "[" <> file <> "](" <> link <> " \"" <> file <> "\")" + dispLink Nothing = "" + + render :: [[Text]] -> [Text] + render = + map renderRow + . List.transpose + . map (separator . innerMargins . alignLeft) + . List.transpose + + renderRow = surroundWith '|' . Text.intercalate "|" + + alignLeft ts = + let maxLen = maximum (Text.length <$> ts) + in map (Text.justifyLeft maxLen ' ') ts + + surroundWith c = Text.cons c . flip Text.snoc c + innerMargins = map (surroundWith ' ') + + separator (h:rs) = h : Text.replicate (Text.length h) "-" : rs + separator [] = [] diff --git a/scripts/generate-release-changelog-links/generate-release-changelog-links.cabal b/scripts/generate-release-changelog-links/generate-release-changelog-links.cabal new file mode 100644 index 00000000000..902ab821579 --- /dev/null +++ b/scripts/generate-release-changelog-links/generate-release-changelog-links.cabal @@ -0,0 +1,39 @@ +cabal-version: 3.8 +name: generate-release-changelog-links +version: 0.1.0.0 +synopsis: Generate a changelog table for a cardano-node release +description: + Reads a cabal plan.json (from plan-nix) and produces a Markdown table of + CHANGELOG.md links for every CHaP-hosted package in the build plan. +license: Apache-2.0 +author: IOHK +maintainer: operations@iohk.io + +executable generate-release-changelog-links + main-is: Main.hs + hs-source-dirs: . + default-language: Haskell2010 + ghc-options: -Wall -Wextra -Wcompat -O2 -flate-specialise -optl-s -threaded -rtsopts "-with-rtsopts=-A64m -I0" + default-extensions: + BlockArguments + DataKinds + ImportQualifiedPost + LambdaCase + OverloadedStrings + RecordWildCards + ScopedTypeVariables + build-depends: + , base + , aeson + , bytestring + , cabal-plan + , case-insensitive + , containers + , github ^>= 0.29 + , http-client + , http-types + , network-uri + , optparse-applicative + , req + , text + , uri-encode