Crawl API
Turn public web pages into readable content for your products, research and AI assistants.
For teams that need page content without maintaining a crawler. Choose the format you need and pay only when a page is fetched.
Web content, ready for your workflow
- Get readable text, markdown or the original page
- Fetch pages that need a browser to load
- Give AI assistants content from the pages they find
Explore the details
Open the reference you need to connect the Crawl API to your workflow.
How page fetching works
How it works
Fetch any public web page as HTML, markdown or text. It starts with a plain request and uses a real browser only when the page needs one. Failed pages cost nothing.
Each request starts at the cheapest tier that can work and moves up only when it has to: when the page comes back blocked (a challenge page, or a 401, 403, 429 or 503 answer), or when a plain page looks like it needs a browser (very little text, or under 15 KB). The crawler remembers which tier worked for a site, so later requests there can start at it. Each browser request uses a fresh browser and profile, so nothing carries over between requests.
http
Plain HTTP
One HTTP request with a real Chrome browser's headers, from a residential IP. Enough for most pages.
1 credit per page
js
JavaScript rendering
A real browser that loads the page and runs its JavaScript.
3 credits per page
hard
Hard tier
A stealth browser for sites with strong bot protection.
10 credits per page
Some sites block every tier. Those requests fail with a 502 and cost nothing. Use max_tier to stop the crawler below a tier you do not want to pay for.
Parameters and authentication
Parameters
GET or POST https://api.crawlerdata.com/v1/crawl
Send them as a JSON body with POST, or in the query string with GET or POST. Put your API key in the Authorization: Bearer YOUR_API_KEY header, or in the query string as api_key=YOUR_API_KEY. The aliases render, premium, ultra_premium, wait_for_selector, output_format and device_type are accepted too.
| Name | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The page to fetch: an http or https URL on the public internet, up to 4,096 characters. Redirects are followed. URLs with a username or password, and private, local or internal addresses, are refused (400 TARGET_NOT_ALLOWED). |
| output | string | No | "html" (default), "markdown" or "text". Markdown leaves out scripts, styles, frames and SVG, and links that start with / become absolute URLs. Alias: output_format. return_page_markdown=true and return_page_text=true also work. |
| render_js | boolean | No | Start in a real browser that runs JavaScript (3 credits). Default false: the crawler starts with plain HTTP and moves to a browser by itself when the page is blocked or looks like it needs one. Alias: render. |
| premium_proxy | boolean | No | Start at the hard tier (10 credits), for a site you know blocks the cheaper tiers. Default false. Alias: premium. |
| stealth_proxy | boolean | No | Same as premium_proxy: start at the hard tier. Default false. Alias: ultra_premium. |
| country_code | string | No | 2-letter country code of the IP the page is fetched from. Default "us". If no IP is available in that country, the request fails and is not charged. |
| device | string | No | "desktop" (default) or "mobile" (an Android phone's browser and screen size). Alias: device_type. |
| wait | integer | No | Milliseconds to wait after the page loads before it is captured, 0 to 35000. Needs a browser, so the request starts at the JavaScript tier. |
| wait_for | string | No | CSS selector to wait for (up to 20 seconds) before the page is captured; the page is returned either way. Needs a browser, so the request starts at the JavaScript tier. Alias: wait_for_selector. |
| wait_browser | string | No | When the browser counts the page as loaded: "domcontentloaded" (default), "load", "networkidle0" or "networkidle2". |
| timeout | integer | No | Time limit for the whole request in milliseconds, 10000 to 180000. Default 90000. |
| block_resources | boolean | No | In a browser, skip images, fonts and media so the page loads faster. Default true. Screenshots always load them. |
| screenshot | boolean | No | Also take a PNG screenshot of the visible part of the page (+1 credit when taken). Needs a browser. Without json_response, the response is the PNG itself. Default false. |
| json_response | boolean | No | Return JSON with the page in body plus status_code, final_url, tier, credits, bytes, content_type and screenshot, instead of the page itself. Default false. |
| max_tier | string | No | The highest tier the crawler may use: "http", "js" or "hard" (default). It caps the price: with "http" a request never costs more than 1 credit. |
| max_cost | integer | No | The most credits the request may cost, 1 to 11 (screenshot included). The crawler stops at the highest tier that fits. Ignored when max_tier is set. |
Show the request and response
Example
A page as markdown, with its metadata as JSON:
curl -X POST https://api.crawlerdata.com/v1/crawl \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://www.rfc-editor.org/rfc/rfc9309.html","output":"markdown","json_response":true}'const res = await fetch('https://api.crawlerdata.com/v1/crawl', {
method: 'POST',
headers: { Authorization: 'Bearer YOUR_API_KEY', 'Content-Type': 'application/json' },
body: JSON.stringify({
"url": "https://www.rfc-editor.org/rfc/rfc9309.html",
"output": "markdown",
"json_response": true
}),
});
const page = await res.json(); // status_code, tier, credits, bodyimport requests
res = requests.post(
"https://api.crawlerdata.com/v1/crawl",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"url": "https://www.rfc-editor.org/rfc/rfc9309.html",
"output": "markdown",
"json_response": True,
},
)
data = res.json(){
"status_code": 200,
"final_url": "https://www.rfc-editor.org/rfc/rfc9309.html",
"tier": "http",
"credits": 1,
"bytes": 82967,
"content_type": "text/markdown; charset=utf-8",
"body": "RFC 9309: Robots Exclusion Protocol\n\nRFC 9309\n\nRobots Exclusion Protocol (REP)\n\nSeptember 2022\n\nKoster, et al.…"
}Long values are shortened in this example.
Or with GET, the way many scraping clients call: the response is the page itself.
curl "https://api.crawlerdata.com/v1/crawl?api_key=YOUR_API_KEY&url=https://example.com/&output=markdown"Response formats, headers and fields
Response
By default the response body is the page itself, with Content-Type text/html, text/markdown or text/plain (charset utf-8) for the output you asked for. It is sent with Content-Security-Policy: sandbox, so the page's scripts do not run if you open the response in a browser. With screenshot=true and no json_response, the body is the PNG (image/png). Pages, screenshots and JSON answers carry these headers (a 502 or 503 too, with X-Credits 0):
| Header | Value |
|---|---|
| X-Status-Code | The HTTP status the target site answered with. |
| X-Final-Url | The URL after redirects. |
| X-Tier | The tier that returned the page: http, js or hard. |
| X-Credits | Credits charged for this request (0 when it failed). |
| X-Job-Id | The request's ID. Quote it if you contact support about a request. |
With json_response=true you get one JSON object instead:
| Field | Value |
|---|---|
| status_code | The HTTP status the target site answered with. |
| final_url | The URL after redirects. |
| tier | The tier that returned the page: http, js or hard. |
| credits | Credits charged. |
| bytes | About how many bytes were downloaded to get the page, over every tier tried. |
| content_type | text/html, text/markdown or text/plain (charset utf-8), from output. |
| body | The page as HTML, markdown or text. |
| screenshot | Base64-encoded PNG, when screenshot=true and a screenshot was taken. |
A page whose site answers 404, or another client error that is not a block, is returned and charged; check status_code. Pages are not stored: the body goes back to you, and only the request's metadata (URL, options, status, tier, credits) is kept.
Pricing and credit holds
Pricing
You pay for the tier that returned the page, plus 1 credit when a screenshot was taken. A page that went to a browser only because the plain page looked like it needed one, and came back essentially the same, costs 1 credit. Blocked pages, timeouts and errors cost nothing. $1.00 per 1,000 credits, down to $0.30 with larger packs.
| Tier | Credits per page | Starter | Ultimate |
|---|---|---|---|
| Plain HTTP (http) | 1 | $0.0010 | $0.0003 |
| JavaScript rendering (js) | 3 | $0.0030 | $0.0009 |
| Hard tier (hard) | 10 | $0.0100 | $0.0030 |
| Screenshot | +1 | +$0.0010 | +$0.0003 |
Cost per page at $1.00 and $0.30 per 1,000 credits. While a request runs it holds the most it can cost: 10 credits (11 with a screenshot), or less with max_tier or max_cost. The rest comes back as soon as the page is served, and a balance below the hold gets a 402 before anything is fetched. See full pricing →
Errors and troubleshooting
Errors
No error is charged. Most errors have the usual shape, {"error": {"message": "...", "code": "..."}}. A page that could not be fetched (502) or a busy crawler (503) answers with a message in error and, when the site answered, its status_code and the last tier tried.
| Status | Code | When |
|---|---|---|
| 400 | VALIDATION_ERROR | A parameter is missing or out of range, for example no url, or wait over 35000. |
| 400 | TARGET_NOT_ALLOWED | The URL is not http or https, has a username or password, points at a private, local or internal address, or its host does not resolve. |
| 401 | UNAUTHORIZED, INVALID_API_KEY | The API key is missing, mistyped or revoked. |
| 402 | INSUFFICIENT_CREDITS | Your balance is below what the request reserves (see Pricing). |
| 403 | FORBIDDEN | The key came through an API marketplace. Use your own CrawlerData API key. |
| 429 | RATE_LIMITED | Over the key's rate limit: 20 requests per second or 100 per minute unless your key has its own limits. |
| 502 | message only | The page came back blocked on every tier the request allowed, the site answered with a server error, the page took longer than timeout, or the crawler could not fetch it. |
| 503 | message only | The crawler's browsers are busy. Try again in a moment. |
{
"error": "blocked (cloudflare) You were not charged for it.",
"status_code": 403,
"tier": "hard"
}Use it from an AI assistant
MCP tool: crawl_page
Fetches a public web page through the Crawl API and returns it as markdown (default), HTML or text. The tool result starts with the final URL, the status code and the credits used, then the page. It is on the same remote MCP server as the Google tools, https://api.crawlerdata.com/mcp, with the same API key or sign-in, and it is billed like the API: by the tier that returned the page, nothing when the page could not be fetched. Each call can climb to the hard tier, so it holds 10 credits while it runs. Set up the MCP server →
| Name | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The page to fetch (http or https). |
| format | string | No | "markdown" (default), "html" or "text". |
| render_js | boolean | No | Render JavaScript in a browser (3 credits). Off by default; the crawler switches to a browser by itself when the page needs one. |
| country_code | string | No | 2-letter country to fetch from. Default "us". |
| wait_for | string | No | CSS selector to wait for before the page is returned. |
Related APIs
Search Google for the pages you need, then fetch them with the Crawl API.