Get Google search results as JSON with Python

If you want Google results in a Python script, the hard part is not the HTTP request. It is everything after it: parsing a page whose layout changes, dealing with blocks, and pulling out the parts you need, such as the organic results, People Also Ask and the AI Overview.

This guide uses the Google Search API to skip that step. You send a query, you get JSON back, and the rest is ordinary Python. We’ll cover a first request, every part of the response you are likely to use, desktop vs mobile, pagination, batch requests and saving the output to JSON and CSV.

All the response excerpts below are real, trimmed responses from requests we ran in October 2026. Your results will differ: Google results change over time and by country and device.

What you need

• A CrawlerData API key. New accounts get 50 free credits after verifying a work email, and every Google Search request costs 1 credit.
• Python 3.9 or newer and the requests package (pip install requests).
• The key in an environment variable, so it never ends up in your code or your repository:

export CRAWLERDATA_API_KEY="YOUR_API_KEY"

Your first request

Every endpoint uses the same URL, POST https://api.crawlerdata.com/v1/jobs, with a Bearer token in the Authorization header. The body says which endpoint you want, its parameters, and which queue to use. With curl:

curl -X POST https://api.crawlerdata.com/v1/jobs \
  -H "Authorization: Bearer $CRAWLERDATA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "endpoint": "google-search",
    "params": { "query": "how to read a csv file in python", "country": "us", "device": "desktop" },
    "queue": "live"
  }'

The same thing in Python, wrapped in a function we’ll reuse for the rest of the guide:

import os
import requests

API_URL = "https://api.crawlerdata.com/v1/jobs"
HEADERS = {"Authorization": f"Bearer {os.environ['CRAWLERDATA_API_KEY']}"}


def google_search(query, country="us", language="en", device="desktop", page=1):
    body = {
        "endpoint": "google-search",
        "params": {
            "query": query,
            "country": country,
            "language": language,
            "device": device,
            "page": page,
        },
        "queue": "live",
    }
    # A live request answers when the search is done, so allow a generous timeout.
    resp = requests.post(API_URL, json=body, headers=HEADERS, timeout=180)
    resp.raise_for_status()
    data = resp.json()
    if data["job"]["status"] != "completed":
        raise RuntimeError(f"Search failed: {data['job']['error']}")
    return data["result"]


result = google_search("how to read a csv file in python")
for r in result["organic_results"]:
    print(r["position"], r["title"], r["link"])

The live queue returns the result in the same HTTP response. The body has two parts: job, which describes the request (status, credits used, latency), and result, which holds the search results. If a live request fails, job.status is "failed", job.error says why, job.credits is 0 and result is null. You are not charged for it, which is why the function checks the status rather than assuming success.

Here is the job part of a real response, trimmed:

{
  "job": {
    "endpoint": "google-search",
    "queue": "live",
    "status": "completed",
    "params": {
      "query": "how to read a csv file in python",
      "country": "us",
      "language": "en",
      "device": "desktop",
      "page": 1
    },
    "credits": 1,
    "latencyMs": 6998,
    "error": null
  }
}

What is in the result

The result object has the same keys on every search. A feature Google did not show comes back as an empty list or null, so your code does not need to check whether a key exists.

Organic results

These are the first two organic results for our query:

"organic_results": [
  {
    "position": 1,
    "title": "Python CSV File Reading and Writing - ProTech Training",
    "link": "https://www.protechtraining.com/blog/post/python-for-beginners-reading-manipulating-csv-files-737",
    "displayed_link": "https://www.protechtraining.com › blog › post › python-...",
    "snippet": "Mar 18, 2014 — How to Read and Write CSV Files in Python · Opening a CSV file · ..."
  },
  {
    "position": 2,
    "title": "How do I read and write tabular data? - Pandas - PyData |",
    "link": "https://pandas.pydata.org/docs/getting_started/intro_tutorials/02_read_write.html",
    "displayed_link": "https://pandas.pydata.org › getting_started › intro_tutorials",
    "snippet": "pandas provides the read_csv() function to read data stored as a csv file into a pandas DataFrame . ..."
  }
]

link is the destination page, not a Google redirect URL, so you can store it, match it against your own URLs or fetch it directly. displayed_link is the breadcrumb text Google shows under the title and is for display only.

This page had 8 organic results, not 10. The count varies with the query and with what else is on the page, so don’t hard-code 10.

People Also Ask, related searches and AI Overview

The same response had four People Also Ask questions and eight related searches:

"people_also_ask": [
  { "question": "How to open a CSV file in Python code?" },
  { "question": "What are the different file methods in Python?" },
  { "question": "What is pandas used for in Python?" },
  { "question": "How do I create a CSV file?" }
],
"related_searches": [
  { "query": "How to read a csv file in python using" },
  { "query": "CSV to list Python" },
  { "query": "Convert CSV to dictionary Python pandas" }
]

It had no AI Overview, so ai_overview was null. A search for project management software did have one. Its text is a long string and sources lists the pages it cites:

"ai_overview": {
  "text": "Project management software helps teams plan, track, and execute tasks, schedules, resources, and budgets. ...",
  "sources": [
    { "title": "Wikipedia", "link": "https://en.wikipedia.org/wiki/Project_management_software" },
    { "title": "PCMag", "link": "https://www.pcmag.com/picks/the-best-project-management-software" },
    { "title": "Zapier", "link": "https://zapier.com/blog/free-project-management-software/" }
  ]
}

A small helper that flattens these features, and copes with a missing AI Overview:

def serp_features(result):
    aio = result.get("ai_overview")
    return {
        "people_also_ask": [q["question"] for q in result.get("people_also_ask", [])],
        "related_searches": [s["query"] for s in result.get("related_searches", [])],
        "ai_overview": aio["text"] if aio else None,
        "ai_overview_sources": [s["link"] for s in aio["sources"]] if aio else [],
    }

People Also Ask questions make good follow-up queries: run one through google_search() to see the pages Google ranks for it.

Desktop vs mobile

Pass device="mobile" to see the page as a phone user would. The default is desktop, and both cost the same 1 credit.

mobile = google_search("project management software", device="mobile")

The response has the same keys, but don’t expect the same values. In our comparison of one query, the mobile page had different organic URLs, different People Also Ask questions and fewer related searches than the desktop page. A few format details also differed in that sample:

• displayed_link was just the site URL, such as https://www.proofhub.com, and was null for some results.
• snippet was null for some results.
• search_information.total_results was null.

Treat those fields as optional in your code. If you track rankings, request each device separately and store it as part of the key. The desktop vs mobile difference is large enough that mixing them gives you meaningless trends.

Paginate through results

The page parameter goes from 1 to 10. Each page is one request and 1 credit. Two details matter when you combine pages:

1. position restarts at 1 on every page. Page 2 of our query started again at position 1.
2. The same URL can appear on more than one page. Two results from page 1 of our query showed up again on page 2.

So count your own running rank and skip URLs you have already seen:

def search_pages(query, pages=3, **kwargs):
    seen, rows = set(), []
    for page in range(1, pages + 1):
        result = google_search(query, page=page, **kwargs)
        organic = result["organic_results"]
        if not organic:
            break
        for r in organic:
            if r["link"] in seen:
                continue  # the same URL can show up on two pages
            seen.add(r["link"])
            rows.append({**r, "page": page, "rank": len(rows) + 1})
    return rows


rows = search_pages("how to read a csv file in python", pages=2)

For our query, page 1 had 8 results and page 2 had 9, two of them repeats, so rows ended up with 15 unique results for 2 credits.

Save to JSON and CSV

Keep the raw JSON if you might want other fields later, and write a CSV for spreadsheets. Record when you fetched the data, because a SERP is only true at the moment you requested it.

import csv
import json
from datetime import datetime, timezone


def save(rows, query, stem):
    fetched_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
    with open(f"{stem}.json", "w", encoding="utf-8") as f:
        json.dump({"query": query, "fetched_at": fetched_at, "results": rows}, f, ensure_ascii=False, indent=2)

    fields = ["rank", "page", "position", "title", "link", "displayed_link", "snippet"]
    with open(f"{stem}.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore")
        writer.writeheader()
        writer.writerows(rows)


save(rows, "how to read a csv file in python", "csv-python")

encoding="utf-8" matters: titles and snippets contain characters such as › and — that break on some default encodings. newline="" stops blank lines appearing between rows on Windows.

Run batches on the fast queue

The live queue keeps an HTTP connection open until the search finishes. For a list of hundreds of queries, submit jobs to the fast or regular queue instead. The API answers straight away with 202 Accepted and a job ID, and you collect the result later. Every queue costs the same.

import time


def submit(params, endpoint="google-search", queue="fast"):
    body = {"endpoint": endpoint, "params": params, "queue": queue}
    resp = requests.post(API_URL, json=body, headers=HEADERS, timeout=30)
    resp.raise_for_status()  # 202 Accepted
    return resp.json()["job"]["id"]


def wait_for(job_id, every=10, max_wait=900):
    deadline = time.time() + max_wait
    while time.time() < deadline:
        resp = requests.get(f"{API_URL}/{job_id}", headers=HEADERS, timeout=30)
        resp.raise_for_status()
        data = resp.json()
        status = data["job"]["status"]
        if status == "completed":
            return data["result"]
        if status == "failed":
            raise RuntimeError(data["job"]["error"])
        time.sleep(every)  # still "pending" or "processing"
    raise TimeoutError(job_id)


job_id = submit({"query": "project management software", "device": "mobile"})
result = wait_for(job_id)

Submit all your jobs first, then poll them, rather than waiting on each one in turn. If you’d rather not poll, add a postbackUrl to the request body and the result is sent to that URL when the job finishes.

Errors and cost

• Cost. 1 credit per request, on every queue and device. Credit packs work out to $1.00 per 1,000 requests on the Starter pack, down to $0.30 per 1,000 on the largest. See pricing.
• Failed requests are not charged. A request that completes with no results is a successful request and costs 1 credit.
• HTTP errors come back with an error object containing code, message and status. The ones to handle are 400 (VALIDATION_ERROR, for example an unsupported country), 401 (bad or missing key), 402 (INSUFFICIENT_CREDITS) and 429 (RATE_LIMITED, slow down and retry).
• Countries. country takes one of 31 two-letter codes, such as us, gb, de or jp.

Next steps

To look at a single search before building anything, our free AI Overview checker runs without an account and shows the AI Overview, its sources and the top results for the country and device you pick.

You now have a reusable google_search() function, a way to page through results without duplicates and an export to JSON and CSV. The Google Search API page lists every parameter with a full example response. If you’re tracking positions over time, see SEO rank tracking. If you’re comparing providers before you build, cheapest SERP API covers cost per 1,000 searches across the main options.