Large language model (LLM) APIs are a core building block for llm api for developers who want to add code‑completion, documentation generation, or test‑case synthesis to their pipelines. When you run those calls inside GitHub Actions, the CI environment can fire many requests in a short burst, easily hitting the provider’s rate limits and causing the build to fail.
In this guide you will learn how to diagnose the limit that was hit, re‑architect your workflow to stay under the quota, and optionally use an AI agent to handle throttling and retry logic automatically. By the end you will have a reliable CI pipeline that respects API limits without sacrificing the benefits of AI‑assisted development.
What You’ll Learn
- How to locate the exact rate‑limit response from common LLM providers (OpenAI, Anthropic, Cohere, etc.).
- Ways to restructure GitHub Actions jobs to reduce request spikes.
- Implementation of exponential back‑off and jitter in Bash/Python.
- How to create a lightweight AI agent that monitors usage and injects throttling.
- Best practices for secret management and observability in CI.
Understanding Rate Limits
LLM providers typically enforce two kinds of limits:
- Request‑per‑minute (RPM) – caps the number of API calls in any rolling 60‑second window.
- Token‑per‑minute (TPM) – caps the total number of input + output tokens processed.
For example, OpenAI’s rate‑limit documentation shows a default of 3500 RPM for the gpt‑4o model and a TPM of 90 K. Anthropic’s limits page lists 30 RPM for the claude‑3‑sonnet model. These numbers are subject to change and can be increased with higher‑tier plans.
Diagnosing the “Rate Limit Exceeded” Error
Inspect the HTTP response
curl -s -D - -H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{"model":"gpt-4o","messages":[{"role":"user","content":"Hello"}]}' \
https://api.openai.com/v1/chat/completions | head -n 10
The response will contain a 429 status and a JSON body similar to:
{
"error": {
"message": "Rate limit reached for requests per minute. Limit: 3500 RPM.",
"type": "rate_limit_error"
}
}
GitHub Actions also surfaces the same message in the job log, making it easy to grep for “rate limit”.
Enable provider‑specific headers
Some APIs include usage headers (e.g., OpenAI-Processing-Msec, OpenAI-Request-Id). Adding -i to curl or inspecting the ResponseHeaders object in Python will reveal the remaining quota.
Adjusting Request Patterns
Batch prompts
Instead of sending one request per source file, concatenate related files into a single prompt (respecting the model’s context window). This reduces RPM dramatically.
Introduce a fixed inter‑request delay
# Bash example in a GitHub Action step
for FILE in $(git diff --name-only ${{ github.sha }}); do
curl -s -X POST ... && sleep 2 # 2‑second pause → ~30 RPM max
done
Parallelism limits
GitHub Actions defaults to 20 concurrent jobs for public repositories. Reduce the strategy.matrix size or set max-parallel to a lower value.
Implementing Retry & Back‑off
A reliable retry strategy prevents a single spike from aborting the entire workflow.
Exponential back‑off with jitter (Python)
import time, random, requests
def call_api(payload, max_retries=5):
backoff = 1
for attempt in range(max_retries):
resp = requests.post(
"https://api.openai.com/v1/chat/completions",
headers={"Authorization": f"Bearer {os.getenv('OPENAI_API_KEY')}"},
json=payload,
)
if resp.status_code != 429:
return resp.json()
# 429 → rate limit
sleep_time = backoff + random.uniform(0, 0.5)
time.sleep(sleep_time)
backoff *= 2 # exponential growth
raise RuntimeError("Exceeded retry limit")
GitHub Actions built‑in retry (v3)
steps:
- name: Call LLM
id: llm
uses: actions/http-client@v3
with:
url: https://api.openai.com/v1/chat/completions
method: POST
headers: |
Authorization: Bearer ${{ secrets.OPENAI_API_KEY }}
body: ${{ toJSON(inputs.payload) }}
continue-on-error: true
retry:
max-attempts: 5
delay: 3s
Pro Tip: Combine exponential back‑off with a fixed minimum delay to smooth out bursty traffic from matrix builds.
Using an AI Agent for Automated Throttling
AI agents can monitor usage counters and dynamically pause or resume jobs. Below is a minimal example using CrewAI (open‑source) that runs as a sidecar container in the same workflow.
Agent workflow
- Read the provider’s usage endpoint (e.g.,
/v1/usagefor OpenAI). - Maintain an in‑memory token budget for the current minute.
- If the next request would exceed the budget, sleep until the next minute window.
- Expose a tiny HTTP endpoint (
/throttle) that the main job calls before each LLM request.
Dockerfile for the agent
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY agent.py .
CMD ["python", "agent.py"]
agent.py (simplified)
import os, time, threading
from flask import Flask, jsonify
import requests
app = Flask(__name__)
budget = {"tokens": 0, "reset": time.time() + 60}
lock = threading.Lock()
def refresh_budget():
while True:
time.sleep(60)
with lock:
budget["tokens"] = 0
budget["reset"] = time.time() + 60
threading.Thread(target=refresh_budget, daemon=True).start()
@app.route("/throttle")
def throttle():
needed = int(os.getenv("EXPECTED_TOKENS", "500"))
with lock:
if budget["tokens"] + needed > int(os.getenv("TOKEN_LIMIT", "90000")):
wait = budget["reset"] - time.time()
return jsonify({"wait": max(wait, 0)}), 429
budget["tokens"] += needed
return jsonify({"wait": 0}), 200
if __name__ == "__main__":
app.run(host="0.0.0.0", port=8080)
Calling the agent from a job step
curl -s http://localhost:8080/throttle | jq -r .wait | while read w; do
if [ "$w" -gt 0 ]; then
echo "Throttling for $w seconds"
sleep $w
fi
# now safe to call the LLM API
done
CI Configuration Examples
Full workflow with back‑off and agent
name: AI‑Assist CI
on:
push:
branches: [main]
jobs:
ai-assist:
runs-on: ubuntu-latest
services:
throttle-agent:
image: ghcr.io/yourorg/llm-throttle:latest
ports:
- 8080:8080
env:
TOKEN_LIMIT: "90000"
EXPECTED_TOKENS: "500"
steps:
- uses: actions/checkout@v4
- name: Install dependencies
run: pip install -r requirements.txt
- name: Generate code suggestions
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: |
for FILE in $(git diff --name-only ${{ github.sha }}); do
# Ask the agent if we need to wait
WAIT=$(curl -s http://localhost:8080/throttle | jq -r .wait)
if [ "$WAIT" -gt 0 ]; then
echo "Rate limit hit – sleeping $WAIT seconds"
sleep $WAIT
fi
python generate_suggestion.py $FILE || exit 1
done
Matrix build with limited concurrency
strategy:
matrix:
python-version: [3.10, 3.11]
os: [ubuntu-latest, macos-latest]
max-parallel: 2 # prevents 4 concurrent LLM calls
Monitoring & Alerting
- GitHub Action annotations: Use
::warningto flag near‑limit conditions. - External metrics: Push
tokens_usedto a Prometheus pushgateway and set an alert for >80% of the quota. - Provider dashboards: OpenAI’s usage page updates in near real‑time and can be linked from the CI dashboard.
Common Mistakes & How to Avoid Them
- Hard‑coding limits: Provider limits change; read them from the API or keep them in a config file that is version‑controlled.
- Ignoring token count: A single request can consume many tokens; always estimate output length.
- Running parallel jobs without coordination: Use a shared throttling service (as shown) or a Redis lock to serialize calls.
- Storing API keys in plain text: Keep keys in GitHub Secrets and reference them via
${{ secrets.NAME }}.
Frequently Asked Questions
What is the difference between RPM and TPM?
RPM limits the number of HTTP calls, while TPM limits the total token volume processed across all calls. Exceeding either returns a 429 error.
Can I increase my rate limit?
Most providers offer higher limits on paid tiers or via a quota‑increase request. Check the provider’s pricing page (e.g., OpenAI’s pricing) for details.
Do retries count against the rate limit?
Yes. Each retry is a new request, so implement back‑off to keep the retry frequency low.
Is it safe to store usage counters in a file?
For short‑lived CI runs a file works, but in concurrent matrix builds a centralized store (Redis, the sidecar agent, or a cloud key‑value service) avoids race conditions.
How can I test the throttling logic locally?
Run the sidecar Docker image and invoke curl http://localhost:8080/throttle with different EXPECTED_TOKENS values to simulate high‑load scenarios.
Will the agent add noticeable latency?
The agent’s check is a single HTTP call (< 5 ms on a local Docker network). The added latency is negligible compared to the LLM response time.
Do other CI systems (GitLab, CircleCI) need a different approach?
The core concepts—rate‑limit detection, back‑off, and a shared throttling service—apply universally. Adapt the YAML syntax to the target platform.
Conclusion & Next Steps
Implement the following actions to eliminate “rate limit exceeded” failures in your GitHub Actions CI:
- Enable detailed logging and capture the provider’s limit headers.
- Batch prompts and insert a fixed inter‑request delay to stay under RPM.
- Add exponential back‑off with jitter to every API call.
- Deploy the lightweight throttling agent as a sidecar service for dynamic control.
- Set up monitoring (GitHub annotations, Prometheus, provider dashboards) to catch approaching limits early.
After these changes, your CI pipeline will reliably use the llm api for developers without unexpected interruptions, freeing you to focus on building AI‑enhanced automation and code quality checks.
Official documentation
- OpenAI Rate Limits
- Anthropic Rate Limits
- GitHub Actions Workflow Syntax
- Cohere Rate Limits
- CrewAI (open‑source AI agent framework)



