ScraperAPI's max_cost and Async Jobs for AI Agents: What the Docs Say About Credit Caps, Retries and Webhooks

Last fact-checked: November 16, 2026

Advertising disclosure: we may receive a commission if you sign up through the ScraperAPI † links marked † in this article, at no extra cost to you. We have not run production workloads on ScraperAPI; everything below comes from its official documentation. See our Advertising Disclosure.

Note on experience: we have not load-tested ScraperAPI or used its async API in production. Pages were read through a web-fetch tool that summarises them, on November 16, 2026. Check the live docs before relying on any statement. "Not stated" means the page we read did not say.

If an AI agent calls a scraping API in a loop, two questions matter before the first run: how do I cap what one call can cost, and how do I find out what happened without holding a connection open? ScraperAPI † documents an answer to each: a max_cost parameter and an asynchronous API with webhook callbacks. This article lays out what the docs state, and what they leave out. For the related questions of 429s and failed-request billing, see our retry budgets, burst limits and failed scrapes articles.

1. The two ways to call it

Synchronous APIAsynchronous API
Endpointhttps://api.scraperapi.comJobs are submitted to async.scraperapi.com (single jobs to /jobs, batches to /batchjobs)
What you get backThe raw HTML of the target URLA job entry with an ID and a statusUrl; results come from that URL or a webhook
Timeout adviceA 70-second recommended timeout; complex domains can take as long as 60 secondsThe job keeps retrying until the pages return a successful response, for up to 24 hours per job
Size or batch limit50 MB maximum responseUp to 50,000 URLs per batch job

The sync page says it fits "pipelines or apps where data is consumed in real time". The async overview describes jobs that run until they succeed. Whether the two cost different amounts of credits per request is not stated on the pages we read; both say normal (flat) requests typically cost 1 API credit, and that extra parameters or certain domains may add cost.

2. max_cost: a per-request ceiling

The cost-control page says max_cost lets you define the maximum number of API credits used per scrape, so each request stays within your budget. It documents three places to set it:

If a request would exceed the limit, the docs say a 403 is returned with the message "This request exceeds your max_cost", and that you can see the cost per request in the response header or the API Playground.

One status code, two meanings. The status-code page documents 403 as "You have exhausted the API Credits available in this monthly cycle." So an agent that treats every 403 the same way will confuse "this one request was too expensive" with "the whole month's credits are gone". Read the message, not just the status. The examples we read set premium=true with max_cost=5; the docs we read give no table of costs per feature, so we cannot tell you which settings a given cap allows.

3. Async jobs: what the docs say about retries, polling and webhooks

4. What this means for an agent's budget (our reading, not vendor guidance)

5. What we could not confirm

† Referral link.

Sources (checked November 16, 2026)