How to Parse Paginated REST APIs with Python and Trigger n8n Workflows

Step‑by‑step guide to fetch paginated REST API data using a Python automation script and automatically start an n8n workflow for each page.

How to Parse Paginated REST APIs with Python and Trigger n8n Workflows - Technology Tutorials
Short answer: Use a Python automation script that respects the pagination contract of the target API (cursor, offset, or link‑header), streams each page to n8n via its HTTP Request node or webhook, and handles edge‑cases such as empty pages, rate limits, and inconsistent total counts. The core steps are: fetch a page, yield results, detect the next page token, repeat until finished, and POST each batch to an n8n endpoint.

Introduction

Many SaaS services expose large result sets through paginated REST endpoints. Pulling the entire dataset with a naive loop often leads to missed records, throttling errors, or memory exhaustion. A reliable python automation script can abstract those details, stream data page‑by‑page, and hand each chunk off to n8n so the rest of the workflow runs in a serverless, low‑maintenance environment.

After following this guide you will be able to:

  • Identify the pagination style used by an API (cursor, offset, or RFC 5988 link headers).
  • Write a reusable Python generator that yields items safely, respects rate limits, and recovers from transient failures.
  • Configure an n8n webhook or HTTP Request node to consume the streamed payloads without storing the whole result set locally.
  • Deploy the script locally or as a lightweight container that runs on a schedule.

What You'll Learn

  • How to detect and handle the three most common pagination mechanisms.
  • Best practices for error handling, retries, and back‑off.
  • How to integrate Python code with n8n using webhooks or the built‑in HTTP Request node.
  • Trade‑offs between pulling all pages at once vs. streaming to n8n in real time.

Overview of Pagination Patterns

REST APIs usually adopt one of three approaches:

Pattern Typical Fields Pros Cons
Offset/Limit offset, limit Simple arithmetic, easy to test. Can skip records if data changes between calls.
Cursor (token) next_cursor or page_token Stable ordering, safe for mutable collections. Requires storing the token between calls.
RFC 5988 Link Header Link: <...>; rel="next" Standardised, works with any HTTP client. Parsing headers adds a tiny overhead.

Identify the pattern by inspecting the API docs (e.g., Google Drive uses page tokens, while GitHub uses Link headers).

Environment Setup

  1. Install Python 3.11 or newer (official installers: python.org).
  2. Create a virtual environment:
python -m venv .venv
source .venv/bin/activate  # Linux/macOS
.\venv\Scripts\activate   # Windows
  1. Install required packages:
pip install requests backoff tqdm

requests handles HTTP, backoff implements exponential retries, and tqdm gives a lightweight progress bar for debugging.

Building a Pagination‑aware Generator

General structure

A generator yields items one‑by‑one, keeping memory usage constant. The core loop looks like this:

import time
import requests
import backoff

def fetch_page(url, params, headers):
    response = requests.get(url, params=params, headers=headers, timeout=10)
    response.raise_for_status()
    return response.json(), response.headers

@backoff.on_exception(backoff.expo,
                      (requests.exceptions.RequestException,),
                      max_time=300,
                      jitter=backoff.full_jitter)
def safe_get(*args, **kwargs):
    return fetch_page(*args, **kwargs)

def paginate(base_url, auth_token, pagination_type, page_size=100):
    headers = {"Authorization": f"Bearer {auth_token}"}
    params = {"limit": page_size} if pagination_type == "offset" else {}
    next_token = None

    while True:
        if pagination_type == "offset":
            params["offset"] = params.get("offset", 0)
        elif pagination_type == "cursor":
            if next_token:
                params["page_token"] = next_token
        # Link‑header case does not need extra params

        data, resp_headers = safe_get(base_url, params=params, headers=headers)

        # Yield each record
        for item in data.get("items", []):
            yield item

        # Detect termination condition
        if pagination_type == "offset":
            if len(data.get("items", [])) < page_size:
                break
            params["offset"] += page_size
        elif pagination_type == "cursor":
            next_token = data.get("next_cursor")
            if not next_token:
                break
        elif pagination_type == "link":
            links = requests.utils.parse_header_links(resp_headers.get("Link", ""))
            next_link = next((l["url"] for l in links if l["rel"] == "next"), None)
            if not next_link:
                break
            base_url = next_link  # replace for next iteration
        else:
            raise ValueError("Unsupported pagination_type")

Edge‑case handling

  • Empty pages: Some APIs return an empty array but still provide a next token. The generator checks the token first before breaking.
  • Rate limits: If the response status is 429, backoff automatically respects the Retry-After header (if present) before retrying.
  • Inconsistent totals: Do not rely on a “total” field; stop only when the pagination mechanism signals completion.

Pro Tip: Wrap the generator in tqdm.tqdm(paginate(...), total=expected) during development to visualise progress, but remove it in production to avoid extra stdout noise.

Configuring n8n to Receive Streams

n8n can ingest data via a Webhook node or an HTTP Request node that calls an external endpoint. For streaming, the webhook approach is simplest because each POST can contain a small batch (e.g., 100 records).

Step‑by‑step webhook setup

  1. Create a new workflow in n8n.
  2. Add a Webhook node:
    • Method: POST
    • Path: api/paginated-data
    • Response: 200 OK (optional JSON acknowledgment)
  3. Connect the webhook to the rest of your pipeline (e.g., a Function node that transforms data, then a PostgreSQL node that inserts rows).
  4. Activate the workflow; n8n will display the full URL (e.g., https://your-n8n-instance.com/webhook/api/paginated-data).

Sending data from Python

Use requests.post inside the pagination loop. Batch size can be tuned to stay under typical payload limits (n8n defaults to 10 MB per request).

def push_to_n8n(batch, webhook_url, auth_header=None):
    headers = {"Content-Type": "application/json"}
    if auth_header:
        headers.update(auth_header)
    response = requests.post(webhook_url, json=batch, headers=headers, timeout=15)
    response.raise_for_status()

Integrate it with the generator:

WEBHOOK_URL = "https://your-n8n-instance.com/webhook/api/paginated-data"
BATCH_SIZE = 100

batch = []
for item in paginate(BASE_URL, TOKEN, "cursor", page_size=BATCH_SIZE):
    batch.append(item)
    if len(batch) == BATCH_SIZE:
        push_to_n8n(batch, WEBHOOK_URL)
        batch.clear()

# Send any remainder
if batch:
    push_to_n8n(batch, WEBHOOK_URL)

Putting It All Together: The Full Script

#!/usr/bin/env python3
import os
import requests
import backoff
import tqdm

# ---- Configuration -------------------------------------------------
BASE_URL = os.getenv("API_BASE_URL")          # e.g. https://api.example.com/v1/items
TOKEN = os.getenv("API_TOKEN")                # Bearer token
PAGINATION_TYPE = os.getenv("PAGINATION", "cursor")  # offset | cursor | link
PAGE_SIZE = int(os.getenv("PAGE_SIZE", "100"))
WEBHOOK_URL = os.getenv("N8N_WEBHOOK_URL")
BATCH_SIZE = int(os.getenv("N8N_BATCH_SIZE", "100"))
# --------------------------------------------------------------------

def fetch_page(url, params, headers):
    resp = requests.get(url, params=params, headers=headers, timeout=10)
    resp.raise_for_status()
    return resp.json(), resp.headers

@backoff.on_exception(backoff.expo,
                      (requests.exceptions.RequestException,),
                      max_time=300,
                      jitter=backoff.full_jitter)
def safe_get(*args, **kwargs):
    return fetch_page(*args, **kwargs)

def paginate(base_url, auth_token, pagination_type, page_size):
    headers = {"Authorization": f"Bearer {auth_token}"}
    params = {"limit": page_size} if pagination_type == "offset" else {}
    next_token = None
    while True:
        if pagination_type == "offset":
            params["offset"] = params.get("offset", 0)
        elif pagination_type == "cursor" and next_token:
            params["page_token"] = next_token

        data, resp_headers = safe_get(base_url, params=params, headers=headers)

        for item in data.get("items", []):
            yield item

        if pagination_type == "offset":
            if len(data.get("items", [])) < page_size:
                break
            params["offset"] += page_size
        elif pagination_type == "cursor":
            next_token = data.get("next_cursor")
            if not next_token:
                break
        elif pagination_type == "link":
            links = requests.utils.parse_header_links(resp_headers.get("Link", ""))
            next_link = next((l["url"] for l in links if l["rel"] == "next"), None)
            if not next_link:
                break
            base_url = next_link
        else:
            raise ValueError("Unsupported pagination_type")

def push_to_n8n(batch, webhook_url):
    resp = requests.post(
        webhook_url,
        json=batch,
        headers={"Content-Type": "application/json"},
        timeout=15,
    )
    resp.raise_for_status()

def main():
    batch = []
    iterator = paginate(BASE_URL, TOKEN, PAGINATION_TYPE, PAGE_SIZE)
    for item in tqdm.tqdm(iterator, desc="Fetching pages"):
        batch.append(item)
        if len(batch) >= BATCH_SIZE:
            push_to_n8n(batch, WEBHOOK_URL)
            batch.clear()
    if batch:
        push_to_n8n(batch, WEBHOOK_URL)

if __name__ == "__main__":
    main()

Store the script as run_paginated.py and execute it with the required environment variables. The script respects rate limits, retries on network errors, and never holds more than BATCH_SIZE records in memory.

Running the Script in Docker

Packaging the script as a container makes scheduling trivial (e.g., via cron, GitHub Actions, or a Kubernetes CronJob).

# Dockerfile
FROM python:3.11-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY run_paginated.py .

ENV PYTHONUNBUFFERED=1
CMD ["python", "run_paginated.py"]

Build and run:

docker build -t paginated‑to‑n8n .
docker run --rm \
  -e API_BASE_URL="https://api.example.com/v1/items" \
  -e API_TOKEN="YOUR_TOKEN" \
  -e N8N_WEBHOOK_URL="https://your-n8n-instance.com/webhook/api/paginated-data" \
  paginated‑to‑n8n

Common Edge‑Cases & How to Avoid Them

  • Changing total count mid‑run: Do not stop when a “total” field is reached; rely on the pagination token.
  • Duplicate records: If the API uses offset and data can be added during extraction, consider adding a deduplication step in n8n (e.g., a Function node that tracks IDs in a Set).
  • Partial failures: Wrap the push_to_n8n call in its own backoff block to retry only the failed batch without re‑fetching previous pages.
  • Large payloads: Keep BATCH_SIZE low enough to stay under n8n's 10 MB default limit. Adjust the limit in n8n settings if needed (n8n docs).
  • HTTPS verification errors: Use requests.get(..., verify=False) only for internal testing; production should always verify TLS.

Pro Tip: Log the next_token or Link header after each successful page. If the script crashes, you can resume from that token by passing it as an initial parameter.

Frequently Asked Questions

Can I use async libraries like httpx instead of requests?

Yes. The generator pattern works the same; replace requests.get with await httpx.AsyncClient().get and adjust the backoff decorator to support async functions.

What if the API returns mixed pagination methods?

Some APIs support both cursor and offset. Choose the method that guarantees stable ordering (usually cursor) and implement only that branch in the generator.

Do I need to authenticate the webhook in n8n?

n8n supports HTTP Basic, Header Auth, or OAuth2 on webhook nodes. Add the appropriate header in push_to_n8n and configure the webhook node to require it.

How can I monitor the script’s health?

Emit a simple heartbeat log or send a small JSON payload to a monitoring endpoint after each batch. In Docker, you can also use the built‑in health‑check directive.

Is there a limit on how many requests n8n can handle per minute?

n8n itself does not impose a hard request limit; the bottleneck is usually the underlying server or reverse proxy. Review your hosting provider’s rate limits and scale horizontally if needed.

Can I store the pagination state in a database for later resumption?

Yes. Persist the last successful next_token or offset in a key‑value store (Redis, SQLite) and read it on script start to continue where you left off.

Does n8n support bulk inserts into databases?

Yes. The PostgreSQL node can receive an array of objects and execute a single INSERT … VALUES statement when you enable “Batch Mode” in the node settings.

Next Steps

  • Deploy the Docker image to a CI/CD pipeline or a Kubernetes CronJob for automated, scheduled runs.
  • Extend the script to write failed batches to an S3 bucket for later re‑processing.
  • Explore n8n’s built‑in error handling (e.g., “Execute Workflow” node) to trigger alerts when the Python script reports a non‑recoverable error.
  • Consider adding a small cache (e.g., Redis) to de‑duplicate records across multiple runs.

Official documentation