requests (or httpx) to call the external endpoint, transform the JSON payload as needed, and then POST the result to the n8n webhook URL. The script can be run manually, scheduled with cron, or triggered from CI/CD pipelines, giving you a repeatable “api integration python” workflow.Many teams need to move data from third‑party services (e.g., a weather API, a CRM, or a public dataset) into their internal automation platform. n8n provides a flexible webhook node that can start a workflow with any JSON payload, but n8n does not itself fetch external APIs. By pairing a lightweight Python automation script with an n8n webhook, you get a clear separation of concerns: Python handles authentication, pagination, and data shaping; n8n handles downstream processing, branching, and integration with other services.
In this guide you will learn how to:
- Configure a Python environment for reliable API calls.
- Write reliable code that deals with timeouts, retries, and error handling.
- Transform the raw response into the exact shape your n8n workflow expects.
- Securely send the transformed data to an n8n webhook.
- Automate the script execution with
cronor a CI job.
What You’ll Learn
- Setting up a virtual environment and installing
requests/httpx. - Fetching paginated data from a REST endpoint.
- Common transformation patterns (flattening, renaming, filtering).
- Posting to an n8n webhook with proper headers and retries.
- Scheduling and monitoring the script.
Prerequisites
- Python 3.9+ installed on the host that will run the script.
- Access to the external REST API (API key, OAuth token, or other credentials).
- An n8n instance with a webhook node created. The webhook URL can be copied from the node’s settings.
- Basic familiarity with
curlorhttpfor testing endpoints.
Set up the Python project
1. Create a virtual environment
python3 -m venv venv
source venv/bin/activate
2. Install dependencies
Two popular HTTP clients are supported. requests is simple and widely used; httpx adds async support and HTTP/2.
pip install requests # synchronous version
# or
pip install httpx[http2] # async version with HTTP/2 support
3. Optional: Add a linter and type checking
pip install flake8 mypy
Fetch data from the external API
Basic GET request with requests
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
API_URL = "https://api.example.com/v1/items"
API_KEY = "<YOUR_API_KEY>"
# Configure a retry strategy (see Pro Tip below)
retry_strategy = Retry(
total=5,
backoff_factor=1,
status_forcelist=[429, 502, 503, 504],
allowed_methods=["GET"]
)
adapter = HTTPAdapter(max_retries=retry_strategy)
session = requests.Session()
session.mount("https://", adapter)
def fetch_page(page: int = 1) -> dict:
params = {"page": page, "per_page": 100}
headers = {"Authorization": f"Bearer {API_KEY}"}
response = session.get(API_URL, headers=headers, params=params, timeout=10)
response.raise_for_status()
return response.json()
Handling pagination
Assume the API returns a next_page field when more data is available.
def fetch_all() -> list:
results = []
page = 1
while True:
data = fetch_page(page)
results.extend(data.get("items", []))
if not data.get("next_page"):
break
page += 1
return results
Pro Tip: UseRetryfromurllib3(as shown) to automatically retry transient failures. Adjustbackoff_factorandstatus_forcelistto match the API’s rate‑limit behavior.
Transform the payload for n8n
n8n expects a JSON body that downstream nodes can reference (e.g., {{ $json["field"] }}). Common transformations include flattening nested structures, renaming keys, and filtering out unwanted records.
Example transformation
def transform(items: list) -> dict:
# Keep only id and name, rename "name" → "title"
simplified = [
{"id": i["id"], "title": i["name"]}
for i in items
if i.get("status") == "active"
]
return {"records": simplified}
Validation (optional)
Use pydantic for schema validation without adding heavy runtime overhead.
from pydantic import BaseModel, ValidationError
from typing import List
class Record(BaseModel):
id: int
title: str
class Payload(BaseModel):
records: List[Record]
def validate(payload: dict) -> dict:
try:
validated = Payload(**payload)
return validated.dict()
except ValidationError as e:
raise ValueError(f"Payload validation failed: {e}")
Send the data to an n8n webhook
POST request
WEBHOOK_URL = "https://n8n.example.com/webhook/your-webhook-id"
def post_to_n8n(payload: dict) -> None:
headers = {
"Content-Type": "application/json",
# If your n8n instance uses basic auth, add:
# "Authorization": "Basic <BASE64_USER_PASS>"
}
response = session.post(WEBHOOK_URL, json=payload, headers=headers, timeout=10)
response.raise_for_status()
print(f"Webhook delivered, status {response.status_code}")
Full script flow
def main():
raw_items = fetch_all()
transformed = transform(raw_items)
payload = validate(transformed) # optional step
post_to_n8n(payload)
if __name__ == "__main__":
main()
Schedule or trigger the script
Using cron (Linux/macOS)
# Edit the crontab for the user that runs the script
crontab -e
# Example: run at 02:00 UTC every day
0 2 * * * /path/to/venv/bin/python /path/to/script.py > /var/log/api_to_n8n.log 2>&1
Using GitHub Actions (CI/CD)
name: Pull API → n8n
on:
schedule:
- cron: '0 2 * * *' # daily at 02:00 UTC
workflow_dispatch: # manual trigger
jobs:
run-script:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install deps
run: |
python -m venv venv
source venv/bin/activate
pip install requests
- name: Execute script
env:
API_KEY: ${{ secrets.API_KEY }}
WEBHOOK_URL: ${{ secrets.N8N_WEBHOOK_URL }}
run: |
source venv/bin/activate
python script.py
Tool comparison
| Option | Learning curve | Pricing model | Typical limits | When to choose |
|---|---|---|---|---|
| Python + requests (sync) | Low – straightforward procedural code | Free (open source) | CPU‑bound; limited by host resources | Simple one‑off calls, low concurrency |
| Python + httpx (async) | Medium – requires async/await syntax | Free (open source) | Better for high‑throughput or many parallel calls | When you need to fetch many pages concurrently |
| n8n HTTP Request node (no Python) | Low – drag‑and‑drop | Free tier up to 2,000 executions/month; paid plans per execution (n8n pricing) | Execution timeout 30 s (free), 60 s (paid) | Very simple APIs, no complex transformation |
| cURL in a shell script | Low – basic shell knowledge | Free | Same as host limits; harder to maintain complex logic | Quick prototyping or when Python cannot be installed |
Frequently Asked Questions
Can I use environment variables instead of hard‑coding secrets?
Yes. Access them via os.getenv() in Python and reference them in cron or CI pipelines. This keeps keys out of source control.
What if the external API uses OAuth 2.0 client credentials?
Obtain a token first (POST to the token endpoint), cache it for its lifetime, and include Authorization: Bearer <token> in subsequent calls. The requests-oauthlib library simplifies this flow.
How do I debug a failing webhook call?
Enable logging in the script (logging.basicConfig(level=logging.DEBUG)) and inspect the n8n execution log (click the workflow execution in the n8n UI). The webhook node shows the received payload and any HTTP error codes.
Is there a rate‑limit on n8n webhooks?
n8n itself does not impose a hard rate limit, but the underlying hosting (e.g., Vercel, Railway) may. Review the hosting provider’s limits and add client‑side throttling if needed.
Can I chain multiple API calls before sending to n8n?
Absolutely. Build a pipeline inside main() that calls several endpoints, merges the results, and then posts a single combined payload.
What Python version should I target?
Python 3.9 or newer is recommended because it includes the zoneinfo module for timezone handling and receives security updates. All libraries used in this guide support 3.9+.
How do I monitor the script in production?
Use a simple log file rotation (e.g., logrotate) for cron jobs, or configure a health‑check endpoint in the script that returns 200 OK when the last run succeeded. In CI, the job status provides immediate feedback.
Next steps
- Copy the example script into a repository and replace placeholders with your API credentials and n8n webhook URL.
- Run the script locally to verify the payload arrives in n8n (check the workflow execution log).
- Choose a scheduling method (cron, GitHub Actions, or a container orchestrator) that fits your operational model.
- Add monitoring: ship logs to a central system (e.g., Loki, CloudWatch) and set up alerts for non‑200 responses.
- Iterate on the transformation step as your downstream n8n workflow evolves.



