Run thousands of Google searches in Python: workers, retries and queues

At some point a script that runs one search at a time stops being enough. You have 2,000 keywords to check, or a nightly job that has to finish before morning, and the question is how to run them safely: how many in parallel, what to retry, and when to switch from live requests to a queue.

We measured it. This guide has the numbers from tests we ran on 16 October 2026 against the Google Search API, and tested Python for a batch runner with retries and for the fast queue. The throughput figures come from one account on one afternoon, so use them as a starting point and measure your own.

What we measured

Autocomplete scales with workers. We ran 40 autocomplete requests with 1, 3, 6 and 10 parallel workers, with no retries, so every outcome is raw:

Each step up in workers raised throughput almost in proportion, and there were no 429s or failures up to 10 workers.

Searches do not. We ran 30 real google-search requests, live queue, at three worker counts:

The batch finished in about the same time every time. Doubling the workers doubled the time each search took and left the total unchanged. We did not look into why. Whatever the cause, extra workers added waiting and no speed.

For planning, that gives a rough rule: about 0.36 searches per second, so roughly 46 minutes for 1,000 searches and about 7.7 hours for 10,000. We measured one account on one afternoon. Your number may differ, and it can change, so run 30 searches first and use what you get.

The fast queue is for connections, not speed

The live queue holds an HTTP connection open until the search finishes. The fast queue accepts the job straight away, with a 202, and you poll for the result. We submitted 10 searches to it:

• All 10 were accepted in 3.1 seconds.
• Polling every 3 seconds took 38 poll requests in total, and every one returned 200.
• All 10 completed, between 6.1 and 24.0 seconds after we started polling, which is about 0.4 searches per second.

So the fast queue finished at about the same rate as live requests, and it cost the same 1 credit each. Its advantage is practical: you are not holding thousands of connections open, you can submit a whole batch in seconds, and your script can stop and resume.

A batch runner with retries

Put your key in an environment variable:

export CRAWLERDATA_API_KEY="YOUR_API_KEY"

import os
import time
from concurrent.futures import ThreadPoolExecutor

import requests

API_URL = "https://api.crawlerdata.com/v1/jobs"
HEADERS = {"Authorization": f"Bearer {os.environ['CRAWLERDATA_API_KEY']}"}


def search(query, country="us", retries=4):
    """One live search. Retries on 429 and on transient failures whose message says to retry."""
    body = {"endpoint": "google-search", "params": {"query": query, "country": country}, "queue": "live"}
    for attempt in range(retries):
        resp = requests.post(API_URL, json=body, headers=HEADERS, timeout=180)
        if resp.status_code == 429:
            time.sleep(int(resp.headers.get("retry-after", 10)))
            continue
        resp.raise_for_status()
        job = resp.json()["job"]
        if job["status"] == "completed":
            return resp.json()["result"]
        if "retry" in (job["error"] or "").lower():  # "service is busy" and "temporarily blocked" both say retry
            time.sleep(5 * (attempt + 1))
            continue
        raise RuntimeError(job["error"])
    raise RuntimeError(f"gave up on {query!r} after {retries} tries")


def run_batch(queries, workers=3):
    """Run searches in parallel. Returns ({query: result}, {query: error message})."""
    results, errors = {}, {}

    def one(q):
        try:
            results[q] = search(q)
        except Exception as e:  # keep going: one bad search must not stop the batch
            errors[q] = str(e)

    with ThreadPoolExecutor(workers) as pool:
        list(pool.map(one, queries))
    return results, errors

Three details matter:

• It retries only what can succeed on a second try. A 429 waits as long as the retry-after header says. Transient failures whose message says to retry, such as The service is busy. Please retry shortly. and Google temporarily blocked the request. Please retry., back off 5, 10, then 15 seconds. We saw each of those messages during our testing. A validation error, such as an unsupported country, raises at once, because it would fail every time.
• One bad search does not stop the batch. Failures go into errors, so you can look at them afterwards and re-run only those.
• workers=3 is the default for a reason. For searches it matched the throughput of 10 workers with much lower latency per request.

Failed requests are not charged, so retrying costs nothing extra.

The fast queue, in code

def run_fast(queries, country="us", poll_every=3, timeout=300):
    """Submit all searches to the fast queue (HTTP 202), then poll until each finishes."""
    ids = {}
    for q in queries:
        body = {"endpoint": "google-search", "params": {"query": q, "country": country}, "queue": "fast"}
        resp = requests.post(API_URL, json=body, headers=HEADERS, timeout=60)
        resp.raise_for_status()
        ids[q] = resp.json()["job"]["id"]
    results, errors = {}, {}
    deadline = time.time() + timeout
    while len(results) + len(errors) < len(ids) and time.time() < deadline:
        for q, job_id in ids.items():
            if q in results or q in errors:
                continue
            resp = requests.get(f"{API_URL}/{job_id}", headers=HEADERS, timeout=30)
            job = resp.json().get("job") if resp.status_code == 200 else None
            if job and job["status"] == "completed":
                results[q] = resp.json()["result"]
            elif job and job["status"] == "failed":
                errors[q] = job["error"]
        time.sleep(poll_every)
    return results, errors

The poll reads the response defensively. In one of our runs a poll came back without the job key, and we could not reproduce it, so the function checks the status code and the key and simply tries again on the next pass, instead of crashing the batch.

We ran both on six and four searches:

live: 6 ok, 0 errors in 22s
fast: 4 ok, 0 errors in 17s

For a large list, save ids to disk after submitting. If your script stops, you can poll the same job IDs later instead of paying for the searches again.

Picking an approach

What it costs

Concurrency does not change the price. Every request is 1 credit on every queue, and failed requests are not charged. Starter is $1.00 per 1,000 requests and the largest pack is $0.30 per 1,000, see pricing. The tests behind this guide cost 160 credits for the autocomplete runs, 90 for the live searches and a few dozen for the fast queue runs.

Next steps

You now have measured numbers for how the live queue behaves with more workers, a batch runner that retries only what can succeed, and a fast-queue version for large or restartable jobs. For what to do with the results, see the rank tracking guide and AI Overview tracking, and the Google Search API page for every parameter.