ScraperAPI's max_cost and Async Jobs for AI Agents: What the Docs Say About Credit Caps, Retries and Webhooks
Last fact-checked: November 16, 2026
Note on experience: we have not load-tested ScraperAPI or used its async API in production. Pages were read through a web-fetch tool that summarises them, on November 16, 2026. Check the live docs before relying on any statement. "Not stated" means the page we read did not say.
If an AI agent calls a scraping API in a loop, two questions matter before the first run: how do I cap what one call can cost, and how do I find out what happened without holding a connection open? ScraperAPI † documents an answer to each: a max_cost parameter and an asynchronous API with webhook callbacks. This article lays out what the docs state, and what they leave out. For the related questions of 429s and failed-request billing, see our retry budgets, burst limits and failed scrapes articles.
1. The two ways to call it
| Synchronous API | Asynchronous API | |
|---|---|---|
| Endpoint | https://api.scraperapi.com | Jobs are submitted to async.scraperapi.com (single jobs to /jobs, batches to /batchjobs) |
| What you get back | The raw HTML of the target URL | A job entry with an ID and a statusUrl; results come from that URL or a webhook |
| Timeout advice | A 70-second recommended timeout; complex domains can take as long as 60 seconds | The job keeps retrying until the pages return a successful response, for up to 24 hours per job |
| Size or batch limit | 50 MB maximum response | Up to 50,000 URLs per batch job |
The sync page says it fits "pipelines or apps where data is consumed in real time". The async overview describes jobs that run until they succeed. Whether the two cost different amounts of credits per request is not stated on the pages we read; both say normal (flat) requests typically cost 1 API credit, and that extra parameters or certain domains may add cost.
2. max_cost: a per-request ceiling
The cost-control page says max_cost lets you define the maximum number of API credits used per scrape, so each request stays within your budget. It documents three places to set it:
- Sync API:
max_cost=5as a query parameter. - Async API:
max_costinside the request body (the examples nest it underapiParams; the Python example writesmaxCostat the top level). - Proxy mode: encoded in the username, for example
scraperapi.premium=true.max_cost=5.
If a request would exceed the limit, the docs say a 403 is returned with the message "This request exceeds your max_cost", and that you can see the cost per request in the response header or the API Playground.
One status code, two meanings. The status-code page documents 403 as "You have exhausted the API Credits available in this monthly cycle." So an agent that treats every 403 the same way will confuse "this one request was too expensive" with "the whole month's credits are gone". Read the message, not just the status. The examples we read set premium=true with max_cost=5; the docs we read give no table of costs per feature, so we cannot tell you which settings a given cap allows.
3. Async jobs: what the docs say about retries, polling and webhooks
- Retries: the overview says the Async API "keeps retrying the job until the requested pages return a successful response", and each job runs until it succeeds or for up to 24 hours. The retry interval is not stated.
- Status: the job-handling page shows two status values,
running(on submission) andfinished. You poll thestatusUrl(of the formhttps://async.scraperapi.com/jobs/<id>); a finished job returns aresponseobject with headers, body and status code. A running job can be cancelled with aDELETEto the job endpoint. - Retention: results are stored for up to 72 hours (24 hours guaranteed) or until you retrieve them, whichever comes first. After that you must submit the job again.
- Webhooks: only
webhookcallbacks are supported. By default the webhook is called for successful requests only; addexpectsUnsuccessReport: trueto also receive failed jobs (the page spells thisexpectUnsuccessReportin another example, so check which one the live docs use). A failed-job callback carries fields such asid,attempts,status,failReasonandurl. The system tries the webhook URL 3 times, then cancels the job, and the URL must be publicly reachable. - Your own tags: a
metafield is returned unchanged in callbacks and responses, and atimeoutSecfield sets a timeout in seconds.
4. What this means for an agent's budget (our reading, not vendor guidance)
- A job that retries for up to 24 hours is a long-running cost you do not see in a single response. Whether the retries themselves consume credits is not stated on the pages we read, so a cap is the only documented lever.
- If you rely on webhooks, remember the default: failures are not reported unless you ask. An agent waiting for a callback that never comes may be looking at a failed job.
- Because results can disappear after 72 hours, an agent that submits many jobs should fetch or receive them promptly.
- Whether the sync API's 70-second retrying and the async API's 24-hour retrying are billed differently is not stated; the status-code page says only 200 and 404 responses are charged, and that a request failing after 70 seconds of retrying is not charged.
5. What we could not confirm
- Whether a request rejected for exceeding
max_costis charged. - Whether
max_coston an async job caps one job's total cost across all of its retries. - The credit cost of each parameter (
premium,ultra_premium,render, and so on) from the pages we read. - Whether failed async jobs are charged.
- The async API's own concurrency or rate limits.
- Whether sync and async requests differ in credit cost.
† Referral link.