Skip to content
API

Crawl API

Turn public web pages into readable content for your products, research and AI assistants.

For teams that need page content without maintaining a crawler. Choose the format you need and pay only when a page is fetched.

Web content, ready for your workflow

  • Get readable text, markdown or the original page
  • Fetch pages that need a browser to load
  • Give AI assistants content from the pages they find

Explore the details

Open the reference you need to connect the Crawl API to your workflow.

How page fetching works

How it works

Fetch any public web page as HTML, markdown or text. It starts with a plain request and uses a real browser only when the page needs one. Failed pages cost nothing.

Each request starts at the cheapest tier that can work and moves up only when it has to: when the page comes back blocked (a challenge page, or a 401, 403, 429 or 503 answer), or when a plain page looks like it needs a browser (very little text, or under 15 KB). The crawler remembers which tier worked for a site, so later requests there can start at it. Each browser request uses a fresh browser and profile, so nothing carries over between requests.

http

Plain HTTP

One HTTP request with a real Chrome browser's headers, from a residential IP. Enough for most pages.

1 credit per page

js

JavaScript rendering

A real browser that loads the page and runs its JavaScript.

3 credits per page

hard

Hard tier

A stealth browser for sites with strong bot protection.

10 credits per page

Some sites block every tier. Those requests fail with a 502 and cost nothing. Use max_tier to stop the crawler below a tier you do not want to pay for.

Parameters and authentication

Parameters

GET or POST https://api.crawlerdata.com/v1/crawl

Send them as a JSON body with POST, or in the query string with GET or POST. Put your API key in the Authorization: Bearer YOUR_API_KEY header, or in the query string as api_key=YOUR_API_KEY. The aliases render, premium, ultra_premium, wait_for_selector, output_format and device_type are accepted too.

NameTypeRequiredDescription
urlstringYesThe page to fetch: an http or https URL on the public internet, up to 4,096 characters. Redirects are followed. URLs with a username or password, and private, local or internal addresses, are refused (400 TARGET_NOT_ALLOWED).
outputstringNo"html" (default), "markdown" or "text". Markdown leaves out scripts, styles, frames and SVG, and links that start with / become absolute URLs. Alias: output_format. return_page_markdown=true and return_page_text=true also work.
render_jsbooleanNoStart in a real browser that runs JavaScript (3 credits). Default false: the crawler starts with plain HTTP and moves to a browser by itself when the page is blocked or looks like it needs one. Alias: render.
premium_proxybooleanNoStart at the hard tier (10 credits), for a site you know blocks the cheaper tiers. Default false. Alias: premium.
stealth_proxybooleanNoSame as premium_proxy: start at the hard tier. Default false. Alias: ultra_premium.
country_codestringNo2-letter country code of the IP the page is fetched from. Default "us". If no IP is available in that country, the request fails and is not charged.
devicestringNo"desktop" (default) or "mobile" (an Android phone's browser and screen size). Alias: device_type.
waitintegerNoMilliseconds to wait after the page loads before it is captured, 0 to 35000. Needs a browser, so the request starts at the JavaScript tier.
wait_forstringNoCSS selector to wait for (up to 20 seconds) before the page is captured; the page is returned either way. Needs a browser, so the request starts at the JavaScript tier. Alias: wait_for_selector.
wait_browserstringNoWhen the browser counts the page as loaded: "domcontentloaded" (default), "load", "networkidle0" or "networkidle2".
timeoutintegerNoTime limit for the whole request in milliseconds, 10000 to 180000. Default 90000.
block_resourcesbooleanNoIn a browser, skip images, fonts and media so the page loads faster. Default true. Screenshots always load them.
screenshotbooleanNoAlso take a PNG screenshot of the visible part of the page (+1 credit when taken). Needs a browser. Without json_response, the response is the PNG itself. Default false.
json_responsebooleanNoReturn JSON with the page in body plus status_code, final_url, tier, credits, bytes, content_type and screenshot, instead of the page itself. Default false.
max_tierstringNoThe highest tier the crawler may use: "http", "js" or "hard" (default). It caps the price: with "http" a request never costs more than 1 credit.
max_costintegerNoThe most credits the request may cost, 1 to 11 (screenshot included). The crawler stops at the highest tier that fits. Ignored when max_tier is set.
Show the request and response

Example

A page as markdown, with its metadata as JSON:

curl -X POST https://api.crawlerdata.com/v1/crawl \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://www.rfc-editor.org/rfc/rfc9309.html","output":"markdown","json_response":true}'
{
  "status_code": 200,
  "final_url": "https://www.rfc-editor.org/rfc/rfc9309.html",
  "tier": "http",
  "credits": 1,
  "bytes": 82967,
  "content_type": "text/markdown; charset=utf-8",
  "body": "RFC 9309: Robots Exclusion Protocol\n\nRFC 9309\n\nRobots Exclusion Protocol (REP)\n\nSeptember 2022\n\nKoster, et al.…"
}

Long values are shortened in this example.

Or with GET, the way many scraping clients call: the response is the page itself.

curl "https://api.crawlerdata.com/v1/crawl?api_key=YOUR_API_KEY&url=https://example.com/&output=markdown"
Response formats, headers and fields

Response

By default the response body is the page itself, with Content-Type text/html, text/markdown or text/plain (charset utf-8) for the output you asked for. It is sent with Content-Security-Policy: sandbox, so the page's scripts do not run if you open the response in a browser. With screenshot=true and no json_response, the body is the PNG (image/png). Pages, screenshots and JSON answers carry these headers (a 502 or 503 too, with X-Credits 0):

HeaderValue
X-Status-CodeThe HTTP status the target site answered with.
X-Final-UrlThe URL after redirects.
X-TierThe tier that returned the page: http, js or hard.
X-CreditsCredits charged for this request (0 when it failed).
X-Job-IdThe request's ID. Quote it if you contact support about a request.

With json_response=true you get one JSON object instead:

FieldValue
status_codeThe HTTP status the target site answered with.
final_urlThe URL after redirects.
tierThe tier that returned the page: http, js or hard.
creditsCredits charged.
bytesAbout how many bytes were downloaded to get the page, over every tier tried.
content_typetext/html, text/markdown or text/plain (charset utf-8), from output.
bodyThe page as HTML, markdown or text.
screenshotBase64-encoded PNG, when screenshot=true and a screenshot was taken.

A page whose site answers 404, or another client error that is not a block, is returned and charged; check status_code. Pages are not stored: the body goes back to you, and only the request's metadata (URL, options, status, tier, credits) is kept.

Pricing and credit holds

Pricing

You pay for the tier that returned the page, plus 1 credit when a screenshot was taken. A page that went to a browser only because the plain page looked like it needed one, and came back essentially the same, costs 1 credit. Blocked pages, timeouts and errors cost nothing. $1.00 per 1,000 credits, down to $0.30 with larger packs.

TierCredits per pageStarterUltimate
Plain HTTP (http)1$0.0010$0.0003
JavaScript rendering (js)3$0.0030$0.0009
Hard tier (hard)10$0.0100$0.0030
Screenshot+1+$0.0010+$0.0003

Cost per page at $1.00 and $0.30 per 1,000 credits. While a request runs it holds the most it can cost: 10 credits (11 with a screenshot), or less with max_tier or max_cost. The rest comes back as soon as the page is served, and a balance below the hold gets a 402 before anything is fetched. See full pricing →

Errors and troubleshooting

Errors

No error is charged. Most errors have the usual shape, {"error": {"message": "...", "code": "..."}}. A page that could not be fetched (502) or a busy crawler (503) answers with a message in error and, when the site answered, its status_code and the last tier tried.

StatusCodeWhen
400VALIDATION_ERRORA parameter is missing or out of range, for example no url, or wait over 35000.
400TARGET_NOT_ALLOWEDThe URL is not http or https, has a username or password, points at a private, local or internal address, or its host does not resolve.
401UNAUTHORIZED, INVALID_API_KEYThe API key is missing, mistyped or revoked.
402INSUFFICIENT_CREDITSYour balance is below what the request reserves (see Pricing).
403FORBIDDENThe key came through an API marketplace. Use your own CrawlerData API key.
429RATE_LIMITEDOver the key's rate limit: 20 requests per second or 100 per minute unless your key has its own limits.
502message onlyThe page came back blocked on every tier the request allowed, the site answered with a server error, the page took longer than timeout, or the crawler could not fetch it.
503message onlyThe crawler's browsers are busy. Try again in a moment.
{
  "error": "blocked (cloudflare) You were not charged for it.",
  "status_code": 403,
  "tier": "hard"
}
Use it from an AI assistant

MCP tool: crawl_page

Fetches a public web page through the Crawl API and returns it as markdown (default), HTML or text. The tool result starts with the final URL, the status code and the credits used, then the page. It is on the same remote MCP server as the Google tools, https://api.crawlerdata.com/mcp, with the same API key or sign-in, and it is billed like the API: by the tier that returned the page, nothing when the page could not be fetched. Each call can climb to the hard tier, so it holds 10 credits while it runs. Set up the MCP server →

NameTypeRequiredDescription
urlstringYesThe page to fetch (http or https).
formatstringNo"markdown" (default), "html" or "text".
render_jsbooleanNoRender JavaScript in a browser (3 credits). Off by default; the crawler switches to a browser by itself when the page needs one.
country_codestringNo2-letter country to fetch from. Default "us".
wait_forstringNoCSS selector to wait for before the page is returned.

Related APIs

Search Google for the pages you need, then fetch them with the Crawl API.