Load Shedding Architecture

Definition:

Load Shedding Architecture is a deliberate system design strategy where a service or system intentionally drops, rejects, or degrades a portion of incoming requests when it detects it is operating beyond its safe capacity — protecting core system stability and serving remaining requests well, rather than attempting to process everything and collapsing entirely under the overload.

Load Shedding is the engineering equivalent of a controlled sacrifice — deliberately letting some work fail gracefully so that the system as a whole continues to function, rather than letting everything fail catastrophically.

Background — Why Load Shedding Exists Every system has a capacity limit — and what happens beyond that limit defines survivability:

What happens when traffic exceeds capacity:

  WITHOUT Load Shedding:
    Traffic doubles beyond capacity
         │
         ▼
    All requests slow down
         │
         ▼
    Threads/connections exhausted
         │
         ▼
    Memory fills, GC thrashes
         │
         ▼
    Response times: 30 seconds
         │
         ▼
    Entire system collapses ❌
    ALL users get nothing
    Recovery time: minutes to hours

  WITH Load Shedding:
    Traffic doubles beyond capacity
         │
         ▼
    System detects overload
         │
         ▼
    Sheds 50% of lowest-priority requests
    (returns HTTP 429 / 503 immediately)
         │
         ▼
    Remaining 50% served normally ✅
    Response times: still fast
         │
         ▼
    System stays healthy and responsive
    50% of users fully served
    50% get fast "try again" response

  Partial service > total collapse — always

Example:

Streaming Platform During Live Event A streaming platform hosts a massive live concert — expecting 10x normal traffic. Here is how Load Shedding Architecture keeps the platform alive.

                     Internet
                        │
                 50M concurrent users
                 (normal: 5M)
                        │
                        ▼
           ┌────────────────────────┐
           │      CDN Layer         │
           │  (handles static       │
           │   assets, edge cache)  │
           └───────────┬────────────┘
                       │ dynamic requests only
                       ▼
           ┌────────────────────────┐
           │   API Gateway with     │
           │   Load Shedding Gate   │◀─── Real-time metrics
           │                        │     collector
           │   Priority classifier  │
           │   Capacity monitor     │
           │   Shed decision engine │
           └──────┬─────────────────┘
                  │
      ┌───────────┼─────────────┐
      │           │             │
      ▼           ▼             ▼
  TIER 1       TIER 2       TIER 3
  Protected    Important    Deferrable
  Stream API   Search API   Recommend
  Auth API     Profile API  API (shed)
  Payment API  Chat API     Social API