flake: send identifying User-Agent on fetchurl (crates.io 403 workaround)#337
Merged
Conversation
crates.io enforces its long-standing crawler policy (https://crates.io/policies) by rejecting clients with unidentifying User-Agent headers. As of early 2026, default `curl/X.Y.Z` is on the blocklist, which makes `rustPlatform.importCargoLock` 403 on any crate tarball not already in the binary cache. The symptom: PRs that bump a crate to a version not previously cached fail the Nix engine build job with "cannot download crate-${name}-${version}.tar.gz from any mirror", while the actual code paths (clippy, fmt, test, webui) pass. Builds on `main` stay green because the binary cache shields every crate that's already been fetched once. Wrap `fetchurl` in `mkPkgs` so every curl invocation appends an identifying UA that matches the policy's recommended shape (`appname (contact)`). nixpkgs has a more surgical fix in flight (NixOS/nixpkgs#512735) that targets `fetchCargoVendor` specifically; this overlay is a temporary umbrella that unblocks dep work until that change reaches our channel. Verified with `nix flake check --no-build`: all outputs evaluate cleanly. The real test is the CI run on this PR.
CI on the previous commit failed during eval: somewhere inside the
rustPlatform → makeOverridable chain, `fetchurl` gets invoked with a
function-shaped value rather than an attrset (probably the
callPackage auto-call / makeOverridable wrapper checking shape).
Doing `args // {...}` on a function fails with "expected a set but
found a function".
Branch on isAttrs so the User-Agent injection only fires for actual
fetch requests (which always carry url + hash in an attrset).
Function-shaped calls pass through untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
crates.io has long had a crawler policy requiring an identifying
User-Agentheader. The policy was formalised in RFC 3463. What changed early 2026 is enforcement: the defaultcurl/X.Y.ZUA that Nix'sfetchurlsends started getting actively rejected withHTTP 403, alongside defaultpython-requests/*(rust-lang/crates.io#13482).The bite is silent on
mainbecause the Nix binary cache covers every crate that's been built before — only PRs that introduce a new crate version hit a live crates.io download and fail. PR #336 (http/jiffpatch bump) tripped it; every future dep bump will too until this is fixed.This overlay wraps
pkgs.fetchurlso every curl invocation appends an identifying UA of the shape the policy recommends (appname (contact)). It's an umbrella workaround until NixOS/nixpkgs#512735 — which addressesfetchCargoVendordirectly — propagates through our channel (estimated weeks).After this lands, retrying #336's CI run should turn the Nix engine job green.
Test plan
nix flake check --no-build— all outputs evaluate clean.http 1.4.1,jiff 0.2.27,jiff-static 0.2.27tarballs fetch successfully.curl: (22) ... error: 403.Risks / scope
fetchurlcall, not justrustPlatform.importCargoLock. Effect on other fetchers: they get a-A "nasty-engine-build (...)"appended, which servers almost universally ignore. The one corner is fetchers that deliberately set their own UA — curl uses the last-A, so ours would override theirs. No known callers in our derivation tree do that, but worth knowing.