I'm considering porting over an open source library (webdataset) for reading and writing large scientific and machine leanring datasets (petabytes) to Julia. We currently have implementations in Python, Golang, and C++. This library requires processing files as a stream rather than processing the entire archive at once; the code for the reader is actually quite simple.
Tar.jl seems to have the necessary low-level functionality, but its documented API only provides whole-archive operations. Would it be possible to expose the tar operations in a way similar to Python's tarfile API?
I'm considering porting over an open source library (webdataset) for reading and writing large scientific and machine leanring datasets (petabytes) to Julia. We currently have implementations in Python, Golang, and C++. This library requires processing files as a stream rather than processing the entire archive at once; the code for the reader is actually quite simple.
Tar.jl seems to have the necessary low-level functionality, but its documented API only provides whole-archive operations. Would it be possible to expose the tar operations in a way similar to Python's tarfile API?