Retry Budgets for AI Agents: What ScraperAPI, Firecrawl and Bright Data Document About 429s, Retries and Charges

Last fact-checked: November 15, 2026

Advertising disclosure: we may receive a commission if you sign up through the Bright Data †, ScraperAPI † or Firecrawl † links marked † in this article, at no extra cost to you. We have not run production workloads on any of the three; everything below comes from their official documentation. See our Advertising Disclosure.

Note on experience: we have not load-tested these services. Pages were read through a web-fetch tool that summarises them, on November 15, 2026. Check the live docs before relying on any statement. "Not stated" means the page we read did not say.

Our earlier articles covered what the limits are (burst limits and 429s) and what failed requests cost (failed scrapes and your bill). This one is about the agent side: if your own code retries after an error, how many retries should it be allowed, and what do the vendors' docs tell you to assume?

1. Why an agent needs a retry budget

A retry budget is a cap you set in your own code: at most N retries per task, and at most M per run, after which the agent stops and reports. Without one, an agent that keeps retrying a rate-limited call can spend credits or time without finishing. The vendors do not set this cap for you, and two of them already retry on their side, which changes how you should count.

2. What each vendor documents

QuestionScraperAPI †Firecrawl †Bright Data †
What a 429 means"You are sending too many simultaneous requests."Rate limit exceeded, or the concurrency queue is full"Rate limit (rare)"; auto-throttling
Vendor-side retriesRetries failed requests for up to 70 seconds, then returns 500Not stated on the pages we readNot stated on the Web Unlocker features page
Retry guidance for youKeep concurrency within your plan; set your timeout to 70 secondsRetry-After documented only for the government/research indexes; the Rate Limits page points to an Errors page for a backoff snippetOpen a ticket or email support
Is a 429 charged?Not stated. Charges apply to 200 and 404 responsesNot statedNot stated; billing is described for successful requests

† Referral link.

3. ScraperAPI: the vendor already retries for 70 seconds

The status-code page says ScraperAPI retries failed requests for up to 70 seconds before it responds with a 500, and tells you to set your timeout to 70 seconds. A 500 is documented as "Request failed. You will not be charged for this request." The page says you are charged only for 200 and 404 responses. A 429 is described as too many simultaneous requests, with advice to keep concurrency within your plan's limit; whether a 429 itself is charged is not stated.

What that means for a budget: one call from your agent can occupy up to 70 seconds before you see a failure. If your client retries immediately on a 500, you are stacking your retries on top of the vendor's. Count wall-clock time as part of the budget, not only attempts. For a 429, the documented fix is to lower concurrency, not to retry harder.

4. Firecrawl: a lower timeout is the documented way to fail fast

Exceeding a rate limit returns a 429. Exceeding the concurrent-browser limit does not: jobs wait in a queue, and queue time counts against the request's timeout, so the docs say you can set a lower timeout to fail fast instead of waiting. Queued jobs expire after at most 48 hours, and when the team's queue is full, new jobs are rejected with a 429. The Retry-After header is documented for the government and research indexes; for ordinary endpoints, general retry timing is not stated on the page we read, which points to an Errors page for a retry-with-backoff snippet that we did not read. Whether a 429 or failed request consumes credits is not stated on the pages we read. A SCRAPE_TIMEOUT (408) error advises increasing the timeout parameter, up to 60000 ms.

What that means for a budget: a long timeout plus a retry loop can mean long waits in a queue. Set the timeout deliberately, and honor Retry-After where the docs give it.

5. Bright Data: contact support, billing depends on the feature

The Web Unlocker features page describes a 429 as a rate limit ("rare") from auto-throttling, for example "the request was auto-throttled due to low success rate", and says to contact support. It gives no numeric limit and no retry advice. On billing, the page says only successful Web Unlocker requests count toward billing, but for some custom features (the page names headers and cookies) it says you are billed for 100% of requests, successful and failed. Check which features your agent switches on before assuming failed calls are free.

6. A budget template you can adapt (our suggestion, not vendor guidance)

These numbers are placeholders we made up for illustration; none come from a vendor or a test.

Related: for ScraperAPI's max_cost cap and async jobs, see our article on max_cost and async jobs.

7. What we could not confirm

Sources (checked November 15, 2026)