Requeue URLs giving HTTP 429

[Jamesking56]

How can we requeue URLs for a retry when receiving a HTTP 429?

[nakiviar]

Hi!! Nakk here

The cleanest approach is to configure a Guzzle retry when initializing the crawler.

use GuzzleHttp\HandlerStack;
use GuzzleHttp\Middleware;
use Spatie\Crawler\Crawler;

$handlerStack = HandlerStack::create();
$handlerStack->push(Middleware::retry(function ($retries, $request, $response) {
    return $response && $response->getStatusCode() === 429 && $retries < 3;
}));

Crawler::create()
    ->setGuzzleOptions(['handler' => $handlerStack])
    ->setCrawlObserver(new YourCrawlObserver())
    ->startCrawling($url);

This allows Guzzle to handle retries transparently before the crawler marks the URL as failed..