I discussed this yesterday on slack with @fredrikekre. But due to the slack-hole, I thought it would be good to note it down somewhere. If this is not the right repository, feel free to move it somewhere suitable.
The client code in Pkg.jl is quite liberal when it come to the possible file types of artifacts. In particular, it allows tarballs compressed as "zstd", "gzip", "bzip2", "xz", "lz4", "tar" (https://github.com/JuliaLang/Pkg.jl/blob/cc5f7c5f9ca34ed85a9cfad1bb0ed53264a6c0d5/src/PlatformEngines.jl#L452). The decompression/unpacking step is handled by Pkg.PlatformEngines.unpack.
The code on the package servers responsible for fetching, caching, and serving artifacts, however, only supports gzip, zstd and xz. I did not find any documentation that states this restriction. Even worse, I could not find any documentation at all that specifies for than "tarball".
We ran into this problem in oscar-system/GAP.jl#1407. tldr: The server hosting some of the artifacts for GAP.jl sometimes does not like many concurrent requests by github CI. For everything that is cached by the package servers, this is not a problem. But regularly, our CI fails to fetch a .tar.bz2 artifact.
In theory, it should be possible to adapt the artifact caching code in the package server to just use Pkg.PlatformEngines.unpack, thus removing this subtle difference between Pkg.jl and PkgServer.jl. I would provide a PR myself, but @fredrikekre told me that the relevant code is not public.
I discussed this yesterday on slack with @fredrikekre. But due to the slack-hole, I thought it would be good to note it down somewhere. If this is not the right repository, feel free to move it somewhere suitable.
The client code in Pkg.jl is quite liberal when it come to the possible file types of artifacts. In particular, it allows tarballs compressed as "zstd", "gzip", "bzip2", "xz", "lz4", "tar" (https://github.com/JuliaLang/Pkg.jl/blob/cc5f7c5f9ca34ed85a9cfad1bb0ed53264a6c0d5/src/PlatformEngines.jl#L452). The decompression/unpacking step is handled by
Pkg.PlatformEngines.unpack.The code on the package servers responsible for fetching, caching, and serving artifacts, however, only supports gzip, zstd and xz. I did not find any documentation that states this restriction. Even worse, I could not find any documentation at all that specifies for than "tarball".
We ran into this problem in oscar-system/GAP.jl#1407. tldr: The server hosting some of the artifacts for GAP.jl sometimes does not like many concurrent requests by github CI. For everything that is cached by the package servers, this is not a problem. But regularly, our CI fails to fetch a .tar.bz2 artifact.
In theory, it should be possible to adapt the artifact caching code in the package server to just use
Pkg.PlatformEngines.unpack, thus removing this subtle difference between Pkg.jl and PkgServer.jl. I would provide a PR myself, but @fredrikekre told me that the relevant code is not public.