Async support and fetch helper functions
- π Async now supported for streams in single request mode
RangeStreamand its subclasses now have amake_async_fetchermethod which will return anAsyncFetcherthat you can pass a list of URLs to (optionally: a client to use), and an async callback function (which will receive: theAsyncFetcher, theRangeStreamsub/class (with the received response), and the source URL which was requested)
Example showing the speedup this brings to PNGs, due to them being a very linear (unlike e.g. zip files) and many-chunked format which leads to a large number of requests which annihilate non-async performance:
import range_streams
from range_streams.codecs.png import PngStream
from some_urls import urls
import httpx
from tqdm import tqdm
from time import time
from random import shuffle
async def callback(fetcher, stream, url):
await stream.enumerate_chunks_async()
sample_size = 60
dbl_smpl_sz = 2 * sample_size
print(f"Sampling {sample_size} of {len(urls)} PNG URLs each time (200px thumbnails)\n")
for i in range(4):
urls_sample = [u for u in urls[:dbl_smpl_sz]]
shuffle(urls_sample) # Split a random sample
urls_a, urls_b = urls_sample[0:sample_size], urls_sample[sample_size:dbl_smpl_sz]
print("Synchronous")
t0 = time()
c = httpx.Client()
for u in tqdm(urls_a):
s = PngStream(url=u, client=c, enumerate_chunks=True, scan_ihdr=False)
del s
t1 = time()
print(f"Done in {t1-t0}s")
print("Asynchronous")
fetched = PngStream.make_async_fetcher(urls=urls_b, callback=callback)
fetched.make_calls()
t2 = time()
print(f"Done in {t2-t1}s")
print()