Node.js client for Google Shopping and Google Search data

Step by step

1. 01Add the client

   Save this as crawlerdata.mjs. Requests time out after 200 seconds. Network errors, 429 and 5xx are retried with backoff, waiting until X-RateLimit-Reset when the API sends it. 400, 401 and 402 throw a CrawlerDataError with the HTTP status.

   // Minimal CrawlerData client for Node 18+: live requests, async jobs with polling, retries. No packages.
   const API = 'https://api.crawlerdata.com/v1';
   const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
   
   export class CrawlerDataError extends Error {
     constructor(message, status) { super(message); this.status = status; }
   }
   
   export class CrawlerData {
     constructor({ apiKey = process.env.CRAWLERDATA_API_KEY, timeoutMs = 200_000, maxRetries = 4 } = {}) {
       if (!apiKey) throw new Error('Set CRAWLERDATA_API_KEY');
       this.apiKey = apiKey;
       this.timeoutMs = timeoutMs;
       this.maxRetries = maxRetries;
     }
   
     async request(method, path, body) {
       for (let attempt = 0; ; attempt++) {
         let res;
         try {
           res = await fetch(API + path, {
             method,
             headers: { Authorization: `Bearer ${this.apiKey}`, 'Content-Type': 'application/json' },
             body: body ? JSON.stringify(body) : undefined,
             signal: AbortSignal.timeout(this.timeoutMs),
           });
         } catch (err) {
           // Network error or timeout: back off and retry.
           if (attempt >= this.maxRetries) throw err;
           await sleep(2 ** attempt * 1000 + Math.random() * 1000);
           continue;
         }
         // 429 = rate limited, 5xx = temporary. Wait and try again.
         if ((res.status === 429 || res.status >= 500) && attempt < this.maxRetries) {
           const reset = Number(res.headers.get('x-ratelimit-reset'));
           const waitS = reset ? Math.max(1, reset - Math.floor(Date.now() / 1000)) : 2 ** attempt;
           await sleep(Math.min(waitS, 60) * 1000 + Math.random() * 1000);
           continue;
         }
         // 400 bad params, 401 bad key, 402 out of credits: retrying will not help.
         if (!res.ok) throw new CrawlerDataError(`HTTP ${res.status}: ${(await res.text()).slice(0, 300)}`, res.status);
         return res.json();
       }
     }
   
     /** Submit a job. On the live queue the response holds the result. */
     run(endpoint, params, { queue = 'live', postbackUrl } = {}) {
       return this.request('POST', '/jobs', { endpoint, params, queue, ...(postbackUrl && { postbackUrl }) });
     }
   
     getJob(id) {
       return this.request('GET', `/jobs/${id}`);
     }
   
     /** Run a live request and return the result, or throw if the job failed. */
     async live(endpoint, params) {
       const { job, result } = await this.run(endpoint, params);
       if (job.status !== 'completed') throw new CrawlerDataError(`job ${job.id} failed: ${job.error}`);
       return result;
     }
   
     /** Queue a job on fast or regular and return its id. */
     async submit(endpoint, params, { queue = 'fast', postbackUrl } = {}) {
       const { job } = await this.run(endpoint, params, { queue, postbackUrl });
       return job.id;
     }
   
     /** Poll a job until it completes or fails. */
     async wait(id, { intervalMs = 5000, maxWaitMs = 600_000 } = {}) {
       const deadline = Date.now() + maxWaitMs;
       while (Date.now() < deadline) {
         const { job, result } = await this.getJob(id);
         if (job.status === 'completed') return result;
         if (job.status === 'failed') throw new CrawlerDataError(`job ${id} failed: ${job.error}`);
         await sleep(intervalMs);
       }
       throw new Error(`job ${id} still running after ${maxWaitMs / 1000}s`);
     }
   }

2. 02Use it

   live() returns the result directly. submit() queues a job and returns its ID, and wait() polls until it completes. Submitting with Promise.all sends the jobs together; keep batches under 20 per second.

   import { CrawlerData } from './crawlerdata.mjs';
   
   const cd = new CrawlerData(); // reads CRAWLERDATA_API_KEY
   
   // Live: the result comes back in the same response.
   const result = await cd.live('google-search', { query: 'best crm software', country: 'us' });
   for (const r of result.organic_results.slice(0, 3)) console.log(r.position, r.title, r.link);
   
   // Async: submit a batch on the fast queue, then poll each job.
   const ids = await Promise.all(['running sho', 'trail runn'].map((query) => cd.submit('google-autocomplete', { query })));
   for (const id of ids) {
     const { suggestions } = await cd.wait(id);
     console.log(suggestions.slice(0, 3).map((s) => s.value));
   }

3. 03Receive results by webhook

   Pass { postbackUrl } to submit() and the result is POSTed to that URL when the job finishes, with jobId, endpoint, status, data, credits, latencyMs and timestamp.

   With a webhook secret set in the dashboard, X-CrawlerData-Signature holds the hex HMAC-SHA256 of the raw body. The receiver below verifies it on the raw bytes, before JSON.parse. Answer quickly, and keep polling as a fallback for any job whose webhook you did not receive.

   // Receive CrawlerData webhooks (fast and regular queues). Node 18+, no packages.
   import { createServer } from 'node:http';
   import { createHmac, timingSafeEqual } from 'node:crypto';
   
   // The webhook secret you set in the dashboard (API Access > Webhook). Leave unset to skip the check.
   const SECRET = process.env.CRAWLERDATA_WEBHOOK_SECRET || '';
   
   function validSignature(rawBody, signature = '') {
     const expected = Buffer.from(createHmac('sha256', SECRET).update(rawBody).digest('hex'));
     const given = Buffer.from(signature);
     return given.length === expected.length && timingSafeEqual(given, expected);
   }
   
   createServer(async (req, res) => {
     if (req.method !== 'POST') { res.writeHead(405).end(); return; }
     const chunks = [];
     for await (const c of req) chunks.push(c);
     const raw = Buffer.concat(chunks);
     if (SECRET && !validSignature(raw, req.headers['x-crawlerdata-signature'])) { res.writeHead(401).end(); return; }
     const payload = JSON.parse(raw);
     // Answer with a 2xx within 10 seconds, then do the slow work. Poll GET /v1/jobs/{id} for any job you missed.
     res.writeHead(204).end();
     if (payload.status === 'completed') console.log(payload.jobId, payload.endpoint, JSON.stringify(payload.data).slice(0, 200));
     else console.log('job failed', payload.jobId, payload.error);
   }).listen(Number(process.env.PORT || 8000));

If something goes wrong

401 INVALID_API_KEY

The key is missing, mistyped or revoked. Send it as "Authorization: Bearer YOUR_API_KEY" (with the word Bearer and one space) and check the key is active under API Access in the dashboard.

402 INSUFFICIENT_CREDITS

Your balance is too low for the request. Buy a credit pack in the dashboard. Requests that fail for other reasons are not charged, so a 402 never costs credits.

429 RATE_LIMITED

Each API key allows 100 requests per minute and 20 per second. Wait until the time in the X-RateLimit-Reset header (a Unix timestamp) and retry, or spread requests out.

Timeouts on live requests

A live request returns when the result is ready. Search requests usually take a few seconds; a Google Shopping sellers request collects every seller for the product, so allow more time. Set the client timeout to about 200 seconds, or use the fast queue and poll the job.

job.status is "failed"

The HTTP request worked but the job did not; job.error says why. Failed jobs cost 0 credits, so you can retry them.

fetch is not defined

You are on Node 16 or older. Upgrade to Node 18 or newer, which has fetch built in.

Request and response schemas: OpenAPI 3.1 document. Still stuck? Email contact@crawlerdata.com.

APIs you can call

Solutions built on it

Other integrations