What Happens When Your AI Agent Fires a Burst of Requests? Concurrency Limits, 429s and Timeouts
Last fact-checked: November 10, 2026
Advertising disclosure: we may receive a commission if you sign up through the Bright Data † link, the ScraperAPI † link or the Firecrawl † link marked † in this article, at no extra cost to you. We have not run production workloads on any of the three; everything below comes from their official documentation. See our Advertising Disclosure.
An AI agent does not call a scraping API politely, one URL at a time. It may fan out dozens of calls at once. What happens next depends on three things the vendors document unevenly: how many requests you may have in flight, what the error looks like when you exceed it, and how long to wait before giving up.
Note on experience: we have not load-tested any of these services. Pages were read through a web-fetch tool that summarises them, on November 10, 2026. Check the live docs before relying on any number here.
† marks a referral link (see the disclosure above). "Not stated" means the page we read did not say.
1. Side by side
| Question | ScraperAPI † | Firecrawl † | Bright Data † |
|---|---|---|---|
| What limits you | Concurrent threads, set by plan | Requests per minute per endpoint, plus concurrent browsers, both by plan | Not stated as numbers on the pages we read |
| Published numbers | Free 5; Hobby 20; Startup 50; Business 100; "Scaling" 200 concurrent threads (pricing page) | /scrape per minute: Free 10, Hobby 100, Standard 500, Growth 5,000, Scale 10,000. Concurrent browsers: Free 2, Hobby 5, Standard 25, Growth 50, Scale 100+ | None found |
| When you exceed it | HTTP 429: "You are sending too many simultaneous requests." | Rate limit: HTTP 429. Concurrency: extra jobs wait in a queue. Queue full: 429 | HTTP 429 means auto-throttling, described as rare |
| Retry guidance | Match concurrency to your plan's limit and try again | Retry-After documented for some endpoints only; general retry timing not stated on this page | Contact support on a 429 |
| Timeout advice | Set client timeout to 70 seconds | Queue time counts against the request's timeout | Not stated for Web Unlocker on the pages we read |
2. ScraperAPI
- The limit is on simultaneous requests, not on requests per minute. The pricing page lists concurrent threads per plan: 5 on Free, 20 on Hobby, 50 on Startup, 100 on Business and 200 on the plan it calls "Scaling". The 7-day trial lists no concurrency number.
- The status-code page says a 429 means "You are sending too many simultaneous requests" and tells you to keep your concurrency within your plan's limit. It gives no thread count itself, and no requests-per-second figure appears anywhere we looked.
- Retries happen on ScraperAPI's side for up to 70 seconds, so the docs say to set your own timeout to 70 seconds. Requests that fail after 70 seconds of retrying are not charged (see What Do You Pay For When a Scrape Fails?).
- We could not find a dedicated concurrency or rate-limit page in the docs index. We did not confirm whether a 429 itself costs credits.
3. Firecrawl
- Two separate controls apply. Requests per minute are capped per endpoint and per plan, and "rate limits are applied per team", so all API keys on a team share the same counters. Separately, concurrent browsers are capped per plan.
- Exceeding a rate limit returns HTTP 429. Exceeding the concurrency limit does not error: "any jobs beyond it wait in a queue until a browser frees up." Queue time counts against the request's
timeout, and queued jobs time out after a maximum of 48 hours. - Max queued jobs are 50,000 on Free, Hobby and Standard, 100,000 on Growth and 200,000+ on Scale and Enterprise. When the queue is full, new jobs are rejected with a 429 until existing jobs complete.
- Per-minute limits for the other endpoints (map, crawl, search, agent, status) differ by plan; the Free plan allows 2 crawl requests per minute and Hobby 20.
- The page documents a
Retry-Afterheader only for the government and research indexes. For ordinary endpoints it points to an Errors page for a retry-with-backoff snippet, which we did not read, so general retry timing is "not stated" here.
For what each operation costs in credits, see Firecrawl Credits Explained.
4. Bright Data
- The Web Unlocker introduction says it retries failed attempts with alternative configurations until the request succeeds. It states no rate limit, concurrency cap or request timeout.
- The Web Unlocker features page describes a 429 as "a rate limit (rare)" caused by auto-throttling, for example "the request was auto-throttled due to low success rate", and says to contact support. It gives no numeric threshold.
- We did not find limits for the Web MCP free tier on the pages we read. See our pre-flight checklist for what that tier does and does not state.
5. What to do with this
- Put your own cap below the plan's. An agent that spawns parallel tool calls can exceed 5 ScraperAPI threads or 2 Firecrawl browsers instantly. Use a semaphore in your client sized to the plan.
- Treat the three 429s differently. ScraperAPI's means too many in flight, so reduce parallelism. Firecrawl's can mean a minute-bucket limit or a full queue. Bright Data's means throttling on its side, where more parallelism will not help.
- Mind timeouts under queueing. On Firecrawl, time spent waiting counts toward your timeout, so a long queue can turn into a timeout error. On ScraperAPI, a client timeout under 70 seconds can cut off retries.
- Check billing on throttled calls before you scale. The pages we read do not say whether a 429 consumes credits. Send a small burst and compare your dashboard to your logs.
To turn plan prices into a monthly cost, use the cost-per-1,000 worksheet.
6. What we could not confirm
- ScraperAPI's requests-per-second behaviour, and whether 429s are billed.
- Any numeric rate or concurrency limit for Bright Data's Web Unlocker or Web MCP.
- Firecrawl's general retry-timing guidance for ordinary 429s.
- Real-world behaviour under load, success rates and speed of any of the three.