rustPlatform.fetchCargoVendor: set a custom User-Agent

[andresilva]

Rust package builds using fetchCargoVendor have started failing intermittently with HTTP 403 responses when downloading crate tarballs from https://crates.io/api/v1/crates/<name>/<version>/download. The failures are not caused by any recent change in nixpkgs, they're triggered server-side by crates.io, which has begun blocklisting specific python-requests/<version> User-Agent strings. Testing directly against crates.io confirms this: python-requests/2.32.5 and python-requests/2.33.1 both return 403, while older versions, empty UAs, and any custom string return 302 as expected.

fetch-cargo-vendor-util.py builds its requests.Session without overriding the default User-Agent, so every tarball request goes out as python-requests/<version> and gets rejected. This also lines up with crates.io's long-standing published policy asking clients to identify themselves with a contact URL (https://crates.io/data-access), which the default UA doesn't satisfy.

The fix is a one-line change in create_http_session() to set User-Agent: nixpkgs fetchCargoVendor/2 (https://github.com/NixOS/nixpkgs) on the session. This identifies the client per crates.io's guidance and sidesteps the version-pattern blocklist. The PR also switches the download URL to https://static.crates.io/crates/<name>/<version>/download to bypass the 1 req/s rate limit on the API servers, as recommended by the crates.io maintainer in rust-lang/crates.io#13482.

Landing this naively would change fetch-cargo-vendor-util.py, which is an input to both the FOD vendorStaging stage (where HTTP downloads happen and the UA matters) and the input-addressed -vendor runCommand stage (where it doesn't). Even though vendorStaging is an FOD and its output path is stable, touching the util referenced by the -vendor stage rewrites every downstream buildRustPackage derivation in the tree, a mass rebuild that has to go through staging. To avoid that, fetch-cargo-vendor-util.py is left untouched and the fix is applied in a parallel copy fetch-cargo-vendor-util-v2.py that's wired only into the FOD. Since the FOD output is content-addressed by declared hash, the rest of the world sees no change. The duplication is deliberate and temporary, it can be collapsed back into a single file on the next staging cycle.

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

[andresilva]

This is a mass rebuild but right now I can't build any rust package, so not sure whether this should still go through staging. Maybe this is also targeted by geo IP and other people aren't getting 403?

[lunagl]

Can confirm this is an issue for me too (thank you so much for submitting this PR, I just spent a while desperately trying to figure out what's going on lol)

For some reason it doesn't block all requests, maybe it's still being rolled out or something somewhere is doing caching?

Oh just so people can find here when searching for the error message:

 Exception: Failed to fetch file from https://crates.io/api/v1/crates/<name>/<version>/download. Status code: 403

[B4dM4n]

I also experienced this today. Changing the user-agent fixed the issue for me.

While testing, I added the response.text to the raised exception, which revealed the following message (manually formatted):

{
  "errors": [
    {
      "detail": "We are unable to process your request at this time. This usually means that you are
        in violation of our API data access policy (https://crates.io/data-access). Please email
        help@crates.io and provide the request id b874fda9-5523-3db7-3c1a-f84d3db0a664"
    }
  ]
}

The https://crates.io/data-access#api page states:

1. A maximum of 1 request per second, and
2. A user-agent header that identifies your application. We strongly suggest providing a way for us to contact you (whether through a repository, or an e-mail address, or whatever is appropriate) so that we can reach out to work with you should there be issues.

Setting the user-agent would conform to 2., which sounds like the correct solution. It seems 1. is not actually enforced while downloading crates or the existing retry logic handles rate limits.

[NickCao]

Why would this create a mass rebuild? These are all FODs?

[andresilva]

@NickCao The tarball downloads themselves are FOD-cached, but fetch-cargo-vendor.nix has a second stage right after:

runCommand "${name}-vendor" {
  inherit vendorStaging;
  nativeBuildInputs = [ fetchCargoVendorUtil cargo replaceWorkspaceValues ];
} ''
  fetch-cargo-vendor-util create-vendor "$vendorStaging" "$out"
''

That -vendor derivation is the one that's actually consumed as cargoDeps by buildRustPackage and has no outputHash, so it's input-addressed. Its store path depends on the util's store path, which this PR changes. At least that's my understanding.

[NickCao]

but fetch-cargo-vendor.nix has a second stage right after:

That's unfortunate, I hope crates.io can grant us an exception for now.

[TomaSajt]

The staging part is a FOD, but there is a non-staging part, which is not a FOD.
And since both use the same file for their logic, it leads to a rebuild...
I have proposed some solutions to this rebuild issue, but have not pushed for them too strong.

(e.g. separating files or ditching the non-staging part completely and moving it into the setup hook)

[emilazy]

For now we can copy the Python file, fix the user agent in only one of them, and use that one in the FOD. That should avoid rebuilds and can be cleaned up on staging.

[emilazy]

Nixpkgs issues can easily go unnoticed if they don't ping anyone. I'm wondering if we should include a suggestion to ping here or perhaps an email address (Nixpkgs core would be fine for this although it would likely be pure indirection since I expect we'd just end up opening an issue ourselves).

[emilazy]

Also I think it would be a good idea to include some kind of version information here. That could be a manual integer specific to fetchCargoVendor for now as including the Nixpkgs version info would require the split Toma mentioned (so that release manager tasks don't go to staging).

[emilazy]

From the upstream issue:

I've temporarily disabled the block for the python-requests UAs again.

I would also recommend that you download the files directly from the CDN instead of going through our API servers: e.g. https://static.crates.io/crates/libc/0.2.185/download or https://static.crates.io/crates/libc/libc-0.2.185.crate. That addresses the rate limit concern mentioned in NixOS/nixpkgs#512735 (comment) since the 1 req/sec limit only applies to our API servers.

for completeness, the technically correct path for downloading crates is: load https://index.crates.io/config.json, look at the dl key in the JSON file, build the download URL via {dl}/{name}/{version}/download.

Since it sounds like the rate limit isn't an immediate problem for us, I suggest we do the file-copying hack now on unstable and 25.11 to get a UA set so the crates.io admins can reinstate the block, and if it would be feasible when cleaning up on staging, adjust the fetcher to follow the config.json protocol and possibly split up FOD-relevant parts of the fetcher cleanly. That should avoid hitting their API server at all and I wouldn't be surprised if it means faster downloads.

(importCargoLock will presumably need similar adjustment if it doesn't do this already? But is rarely used within Nixpkgs therefore less urgent.)

[andresilva]

@emilazy Will push in a bit, thanks for the suggestions 🙏

[andresilva]

diff pkgs/build-support/rust/fetch-cargo-vendor-util.py pkgs/build-support/rust/fetch-cargo-vendor-util-v2.py
42a43
>     session.headers["User-Agent"] = "nixpkgs fetchCargoVendor/2 (https://github.com/NixOS/nixpkgs)"
71c72,74
<     return f"https://crates.io/api/v1/crates/{pkg["name"]}/{pkg["version"]}/download"
---
>     # Use static.crates.io (CDN) instead of crates.io/api to avoid the 1 req/sec
>     # rate limit on the API servers.
>     return f"https://static.crates.io/crates/{pkg["name"]}/{pkg["version"]}/download"

[emilazy]

I think we shouldn’t hardcode this:

for completeness, the technically correct path for downloading crates is: load https://index.crates.io/config.json, look at the dl key in the JSON file, build the download URL via {dl}/{name}/{version}/download.

Let’s perhaps stick to the minimal change for now to reduce risk on unstable and 25.11 and unblock crates.io amins ASAP, and follow up with further improvements on staging? (Also, please split the copy into a separate commit to the other changes so that the diff is readable directly.)

[andresilva]

My thinking was that the current approach is already wrong and it already hardcodes an URL, so we might as well hardcode the URL that doesn't have rate limiting.

[andresilva]

I separated the changes into separate commits as requested.

[emilazy]

Fair enough.

[andresilva]

Maybe we can create an issue on nixpkgs for reports and just link to it here? WDYT @emilazy?

[andresilva]

We also don't want to make the UA a giant string, so maybe just an email might be the way to go?

[MiniHarinn]

The UA is technically valid but parses oddly against RFC 9110 section 10.1.5.

The current format is parsed as two separate products: nixpkgs with no version, and fetchCargoVendor/2, rather than one. Replacing the space with a dash would combine them into a single product token, something like this: nixpkgs-fetchCargoVendor/2 (...).

We should revisit this when we do the cleanup on staging with proper separation between the FOD stage and the input-addressed stage.

This is just a suggestion though, and not meant to block if this needs to land fast (since rust builds are currently broken).

[andresilva]

Updated with your suggestion.

[emilazy]

Replacing the repository URL with an issue URL for reports that the relevant parties can subscribe to and reporters can link their new issues in seems like a good idea to me.

[andresilva]

Feel free to push directly to this branch to fix any remaining issues (I may be offline for the next hours).