Introduction
Many SaaS services expose large result sets through paginated REST endpoints. Pulling the entire dataset with a naive loop often leads to missed records, throttling errors, or memory exhaustion. A reliable python automation script can abstract those details, stream data page‑by‑page, and hand each chunk off to n8n so the rest of the workflow runs in a serverless, low‑maintenance environment.
After following this guide you will be able to:
- Identify the pagination style used by an API (cursor, offset, or RFC 5988 link headers).
- Write a reusable Python generator that yields items safely, respects rate limits, and recovers from transient failures.
- Configure an n8n webhook or HTTP Request node to consume the streamed payloads without storing the whole result set locally.
- Deploy the script locally or as a lightweight container that runs on a schedule.
What You'll Learn
- How to detect and handle the three most common pagination mechanisms.
- Best practices for error handling, retries, and back‑off.
- How to integrate Python code with n8n using webhooks or the built‑in HTTP Request node.
- Trade‑offs between pulling all pages at once vs. streaming to n8n in real time.
Overview of Pagination Patterns
REST APIs usually adopt one of three approaches:
| Pattern | Typical Fields | Pros | Cons |
|---|---|---|---|
| Offset/Limit | offset, limit |
Simple arithmetic, easy to test. | Can skip records if data changes between calls. |
| Cursor (token) | next_cursor or page_token |
Stable ordering, safe for mutable collections. | Requires storing the token between calls. |
| RFC 5988 Link Header | Link: <...>; rel="next" |
Standardised, works with any HTTP client. | Parsing headers adds a tiny overhead. |
Identify the pattern by inspecting the API docs (e.g., Google Drive uses page tokens, while GitHub uses Link headers).
Environment Setup
- Install Python 3.11 or newer (official installers: python.org).
- Create a virtual environment:
python -m venv .venv
source .venv/bin/activate # Linux/macOS
.\venv\Scripts\activate # Windows
- Install required packages:
pip install requests backoff tqdm
requests handles HTTP, backoff implements exponential retries, and tqdm gives a lightweight progress bar for debugging.
Building a Pagination‑aware Generator
General structure
A generator yields items one‑by‑one, keeping memory usage constant. The core loop looks like this:
import time
import requests
import backoff
def fetch_page(url, params, headers):
response = requests.get(url, params=params, headers=headers, timeout=10)
response.raise_for_status()
return response.json(), response.headers
@backoff.on_exception(backoff.expo,
(requests.exceptions.RequestException,),
max_time=300,
jitter=backoff.full_jitter)
def safe_get(*args, **kwargs):
return fetch_page(*args, **kwargs)
def paginate(base_url, auth_token, pagination_type, page_size=100):
headers = {"Authorization": f"Bearer {auth_token}"}
params = {"limit": page_size} if pagination_type == "offset" else {}
next_token = None
while True:
if pagination_type == "offset":
params["offset"] = params.get("offset", 0)
elif pagination_type == "cursor":
if next_token:
params["page_token"] = next_token
# Link‑header case does not need extra params
data, resp_headers = safe_get(base_url, params=params, headers=headers)
# Yield each record
for item in data.get("items", []):
yield item
# Detect termination condition
if pagination_type == "offset":
if len(data.get("items", [])) < page_size:
break
params["offset"] += page_size
elif pagination_type == "cursor":
next_token = data.get("next_cursor")
if not next_token:
break
elif pagination_type == "link":
links = requests.utils.parse_header_links(resp_headers.get("Link", ""))
next_link = next((l["url"] for l in links if l["rel"] == "next"), None)
if not next_link:
break
base_url = next_link # replace for next iteration
else:
raise ValueError("Unsupported pagination_type")
Edge‑case handling
- Empty pages: Some APIs return an empty array but still provide a next token. The generator checks the token first before breaking.
- Rate limits: If the response status is 429,
backoffautomatically respects theRetry-Afterheader (if present) before retrying. - Inconsistent totals: Do not rely on a “total” field; stop only when the pagination mechanism signals completion.
Pro Tip: Wrap the generator in
tqdm.tqdm(paginate(...), total=expected)during development to visualise progress, but remove it in production to avoid extra stdout noise.
Configuring n8n to Receive Streams
n8n can ingest data via a Webhook node or an HTTP Request node that calls an external endpoint. For streaming, the webhook approach is simplest because each POST can contain a small batch (e.g., 100 records).
Step‑by‑step webhook setup
- Create a new workflow in n8n.
- Add a Webhook node:
- Method:
POST - Path:
api/paginated-data - Response:
200 OK(optional JSON acknowledgment)
- Method:
- Connect the webhook to the rest of your pipeline (e.g., a Function node that transforms data, then a PostgreSQL node that inserts rows).
- Activate the workflow; n8n will display the full URL (e.g.,
https://your-n8n-instance.com/webhook/api/paginated-data).
Sending data from Python
Use requests.post inside the pagination loop. Batch size can be tuned to stay under typical payload limits (n8n defaults to 10 MB per request).
def push_to_n8n(batch, webhook_url, auth_header=None):
headers = {"Content-Type": "application/json"}
if auth_header:
headers.update(auth_header)
response = requests.post(webhook_url, json=batch, headers=headers, timeout=15)
response.raise_for_status()
Integrate it with the generator:
WEBHOOK_URL = "https://your-n8n-instance.com/webhook/api/paginated-data"
BATCH_SIZE = 100
batch = []
for item in paginate(BASE_URL, TOKEN, "cursor", page_size=BATCH_SIZE):
batch.append(item)
if len(batch) == BATCH_SIZE:
push_to_n8n(batch, WEBHOOK_URL)
batch.clear()
# Send any remainder
if batch:
push_to_n8n(batch, WEBHOOK_URL)
Putting It All Together: The Full Script
#!/usr/bin/env python3
import os
import requests
import backoff
import tqdm
# ---- Configuration -------------------------------------------------
BASE_URL = os.getenv("API_BASE_URL") # e.g. https://api.example.com/v1/items
TOKEN = os.getenv("API_TOKEN") # Bearer token
PAGINATION_TYPE = os.getenv("PAGINATION", "cursor") # offset | cursor | link
PAGE_SIZE = int(os.getenv("PAGE_SIZE", "100"))
WEBHOOK_URL = os.getenv("N8N_WEBHOOK_URL")
BATCH_SIZE = int(os.getenv("N8N_BATCH_SIZE", "100"))
# --------------------------------------------------------------------
def fetch_page(url, params, headers):
resp = requests.get(url, params=params, headers=headers, timeout=10)
resp.raise_for_status()
return resp.json(), resp.headers
@backoff.on_exception(backoff.expo,
(requests.exceptions.RequestException,),
max_time=300,
jitter=backoff.full_jitter)
def safe_get(*args, **kwargs):
return fetch_page(*args, **kwargs)
def paginate(base_url, auth_token, pagination_type, page_size):
headers = {"Authorization": f"Bearer {auth_token}"}
params = {"limit": page_size} if pagination_type == "offset" else {}
next_token = None
while True:
if pagination_type == "offset":
params["offset"] = params.get("offset", 0)
elif pagination_type == "cursor" and next_token:
params["page_token"] = next_token
data, resp_headers = safe_get(base_url, params=params, headers=headers)
for item in data.get("items", []):
yield item
if pagination_type == "offset":
if len(data.get("items", [])) < page_size:
break
params["offset"] += page_size
elif pagination_type == "cursor":
next_token = data.get("next_cursor")
if not next_token:
break
elif pagination_type == "link":
links = requests.utils.parse_header_links(resp_headers.get("Link", ""))
next_link = next((l["url"] for l in links if l["rel"] == "next"), None)
if not next_link:
break
base_url = next_link
else:
raise ValueError("Unsupported pagination_type")
def push_to_n8n(batch, webhook_url):
resp = requests.post(
webhook_url,
json=batch,
headers={"Content-Type": "application/json"},
timeout=15,
)
resp.raise_for_status()
def main():
batch = []
iterator = paginate(BASE_URL, TOKEN, PAGINATION_TYPE, PAGE_SIZE)
for item in tqdm.tqdm(iterator, desc="Fetching pages"):
batch.append(item)
if len(batch) >= BATCH_SIZE:
push_to_n8n(batch, WEBHOOK_URL)
batch.clear()
if batch:
push_to_n8n(batch, WEBHOOK_URL)
if __name__ == "__main__":
main()
Store the script as run_paginated.py and execute it with the required environment variables. The script respects rate limits, retries on network errors, and never holds more than BATCH_SIZE records in memory.
Running the Script in Docker
Packaging the script as a container makes scheduling trivial (e.g., via cron, GitHub Actions, or a Kubernetes CronJob).
# Dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY run_paginated.py .
ENV PYTHONUNBUFFERED=1
CMD ["python", "run_paginated.py"]
Build and run:
docker build -t paginated‑to‑n8n .
docker run --rm \
-e API_BASE_URL="https://api.example.com/v1/items" \
-e API_TOKEN="YOUR_TOKEN" \
-e N8N_WEBHOOK_URL="https://your-n8n-instance.com/webhook/api/paginated-data" \
paginated‑to‑n8n
Common Edge‑Cases & How to Avoid Them
- Changing total count mid‑run: Do not stop when a “total” field is reached; rely on the pagination token.
- Duplicate records: If the API uses offset and data can be added during extraction, consider adding a deduplication step in n8n (e.g., a Function node that tracks IDs in a Set).
- Partial failures: Wrap the
push_to_n8ncall in its own backoff block to retry only the failed batch without re‑fetching previous pages. - Large payloads: Keep
BATCH_SIZElow enough to stay under n8n's 10 MB default limit. Adjust the limit in n8n settings if needed (n8n docs). - HTTPS verification errors: Use
requests.get(..., verify=False)only for internal testing; production should always verify TLS.
Pro Tip: Log the
next_tokenorLinkheader after each successful page. If the script crashes, you can resume from that token by passing it as an initial parameter.
Frequently Asked Questions
Can I use async libraries like httpx instead of requests?
Yes. The generator pattern works the same; replace requests.get with await httpx.AsyncClient().get and adjust the backoff decorator to support async functions.
What if the API returns mixed pagination methods?
Some APIs support both cursor and offset. Choose the method that guarantees stable ordering (usually cursor) and implement only that branch in the generator.
Do I need to authenticate the webhook in n8n?
n8n supports HTTP Basic, Header Auth, or OAuth2 on webhook nodes. Add the appropriate header in push_to_n8n and configure the webhook node to require it.
How can I monitor the script’s health?
Emit a simple heartbeat log or send a small JSON payload to a monitoring endpoint after each batch. In Docker, you can also use the built‑in health‑check directive.
Is there a limit on how many requests n8n can handle per minute?
n8n itself does not impose a hard request limit; the bottleneck is usually the underlying server or reverse proxy. Review your hosting provider’s rate limits and scale horizontally if needed.
Can I store the pagination state in a database for later resumption?
Yes. Persist the last successful next_token or offset in a key‑value store (Redis, SQLite) and read it on script start to continue where you left off.
Does n8n support bulk inserts into databases?
Yes. The PostgreSQL node can receive an array of objects and execute a single INSERT … VALUES statement when you enable “Batch Mode” in the node settings.
Next Steps
- Deploy the Docker image to a CI/CD pipeline or a Kubernetes CronJob for automated, scheduled runs.
- Extend the script to write failed batches to an S3 bucket for later re‑processing.
- Explore n8n’s built‑in error handling (e.g., “Execute Workflow” node) to trigger alerts when the Python script reports a non‑recoverable error.
- Consider adding a small cache (e.g., Redis) to de‑duplicate records across multiple runs.



