Which Web-Data API for an AI Agent? A Task-by-Task Chooser

Last updated: November 17, 2026

Advertising disclosure: we may receive a commission if you sign up through the Bright Data †, ScraperAPI † or Firecrawl † links marked †, at no extra cost to you. See our Advertising Disclosure.

Note on experience: this page is an index. It adds no new measurements and we have not load-tested any of these services. Every row summarises one of our own articles, which read the vendors' official pages on the dates shown there. Open the article for exact numbers and check the live docs before relying on them.

Three vendors covered here: ScraperAPI †, Firecrawl † and Bright Data †. Pick the row that matches what your agent is about to do.

Pick by task

If you…What the article coversRead
I want to know what a free tier gives meFree offers and concurrency figures for ScraperAPI, Firecrawl, Bright Data Web MCP and others are tabulated, with the date each was checked.Free tiers compared · MCP servers compared
I want to turn credits into a monthly costPer-credit and per-feature costs, plus a worksheet for cost per 1,000 requests. Figures are from vendor pages; see the articles for the exact numbers and dates.Cost-per-1,000 worksheet · Firecrawl credits · ScraperAPI vs ScrapingBee
I want to know what a failed scrape costsSide-by-side billing rules for 200, 404, 403 and 500 responses and timeouts, with the cases where a vendor's docs say nothing marked “not stated”.Failed scrapes and your bill
My agent fires many requests at onceHow each vendor limits concurrency or rate, what its 429 means, and timeout advice. Bright Data publishes no numeric limit on the pages we read.Burst limits and 429s
My agent retries after errorsWhat each vendor documents about retries and charges, plus a retry-budget template we suggest (our suggestion, not vendor guidance).Retry budgets
I need to cap spend per call (ScraperAPI)ScraperAPI's max_cost parameter and its asynchronous API with webhooks, and what the docs leave out.max_cost and async jobs
I need to limit a site crawl (Firecrawl)The crawl limit parameter and its default, the credit pre-check that can return a 402, async status, cancel and webhooks, including where two doc pages differ.Crawl limit and webhooks
I use Bright Data's MCP serverA pre-flight checklist of what the official pages state about the free allowance, tools and limits, and what they do not state.Bright Data MCP checklist
I use Bright Data Web UnlockerError codes sorted by whether a retry helps, and what the docs say and do not say about billing for failures.Web Unlocker error codes
I am choosing an automation platform to startA decision framework comparing Make and Apify, and a walkthrough of Apify's free plan.Make vs Apify · Apify free plan

† Referral link.

How we write these