Definition:
The Rate-Aware Processing Pattern is a design pattern that respects the rate limits of an external system (an API, a database, a service) while maximizing throughput — by tracking how many requests have been sent, checking remaining capacity before each request, and waiting when the limit is reached. It’s the pattern that keeps you from “hitting the API wall” while still moving as fast as the system allows.
The core tension it solves: every external system has limits (X requests per minute, Y operations per second). Ignore them and your system breaks (requests fail, throttled, or rejected). Respect them blindly and you leave performance on the table. The rate-aware pattern tracks the limit precisely so you use all of it — without exceeding it.
THE PROBLEM THE PATTERN SOLVES:
═══════════════════════════════════════════════════════════════
THE NAIVE APPROACH (Fire everything at once):
──────────────────────────────────────────────────────────
You have 500 items to process. The API allows 60 req/min.
┌─────────────────────────────────────────────────────────┐
│ │
│ 500 items ──► [fire all 500 at once] │
│ │ │
│ ▼ │
│ ┌────────────────────────────────────────────┐ │
│ │ 60 requests succeed (under the limit) │ │
│ │ 440 requests fail (rate-limited / 429) │ │
│ │ │ │
│ │ ❌ Now you need: │ │
│ │ • Retry logic for the 440 failures │ │
│ │ • Error handling everywhere │ │
│ │ • Processing time just TRIPLED │ │
│ │ │ │
│ │ The system broke because it ignored │ │
│ │ the rate limit. │ │
│ └────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────┘
THE RATE-AWARE APPROACH (Respect the limit, maximize speed):
──────────────────────────────────────────────────────────
500 items ──► [processor tracks rate]
│
▼
┌─────────────────────────────────────────────────────────┐
│ │
│ Track 3 things: │
│ 1. Requests sent in the current window │
│ 2. When the window resets │
│ 3. Max allowed per window │
│ │
│ Before each request: │
│ • Have capacity? → send it │
│ • No capacity? → wait until the window resets │
│ │
│ → All 500 items process smoothly at the max rate │
│ → No 429 failures, no retry storm, no tripled time │
│ │
└─────────────────────────────────────────────────────────┘Example:
A visual walkthrough of the rate-aware processing pattern applied to a real workload.
A system needs to process 500 customer records through an AI API that allows **60 requests per minute**.
═══════════════════════════════════════════════════════════════
HOW RATE-AWARE PROCESSING WORKS (Step by Step)
═══════════════════════════════════════════════════════════════
Minute 1 (60 req/min allowed):
┌─────────────────────────────────────────────────────────┐
│ │
│ Request 1 → check: 0/60 used, have capacity → SEND │
│ Request 2 → check: 1/60 used, have capacity → SEND │
│ ... │
│ Request 60 → check: 59/60 used → SEND (limit reached) │
│ │
│ Request 61 → check: 60/60 used, NO capacity → WAIT │
│ → sleep until window resets │
│ │
└─────────────────────────────────────────────────────────┘
Window resets →
┌─────────────────────────────────────────────────────────┐
│ │
│ Request 61 → check: 0/60 used, have capacity → SEND │
│ ... process next 60 ... │
│ │
│ (repeat each minute until all 500 are done) │
│ │
│ Result: ~500 records in ~9 minutes, no failures. │
│ │
└─────────────────────────────────────────────────────────┘
═══════════════════════════════════════════════════════════════
KEY TECHNIQUE: USE THE API'S OWN HEADERS, NOT GUESSES
═══════════════════════════════════════════════════════════════
┌─────────────────────────────────────────────────────────┐
│ │
│ Many APIs (like Anthropic) return rate-limit info │
│ in the response headers: │
│ │
│ Response header: │
│ │ X-RateLimit-Remaining: 32 ← requests left │
│ │ X-RateLimit-Reset: 45 ← seconds until │
│ │ window resets │
│ │
│ → Use these headers to know exactly how much capacity │
│ you have left. More accurate than tracking yourself. │
│ │
│ KEY: Don't guess the limit. Let the API tell you. │
│ │
└─────────────────────────────────────────────────────────┘
═══════════════════════════════════════════════════════════════
ADAPTIVE RATE MANAGEMENT (The Advanced Version)
═══════════════════════════════════════════════════════════════
Static rate limiting works, but leaves performance on the
table. Adaptive rate management is smarter:
┌─────────────────────────────────────────────────────────┐
│ │
│ START: Begin at 50% of the stated limit │
│ │ │
│ ├─ If NO errors after 100 requests ──► bump to 75% │
│ │ │ │
│ │ └─ If still clean ──► bump to 90% │
│ │ │ (never go to 100% — other processes │
│ │ │ might share your quota) │
│ │ │
│ │ If you get a 429 (rate limit): │
│ └─► CUT your rate in half immediately │
│ │ │
│ └─ Then ramp back up slowly │
│ │
│ This is the SAME logic TCP uses for network │
│ congestion control — start conservative, ramp up, │
│ back off hard on failure, ramp up again. It works. │
│ │
└─────────────────────────────────────────────────────────┘
═══════════════════════════════════════════════════════════════