Human-in-the-Loop (HITL)

Definition:

Human-in-the-Loop (HITL) is a design pattern where a human being is embedded in the decision loop of an automated system — reviewing, validating, correcting, or approving outputs before they take effect. Instead of a fully autonomous pipeline (data in → decision out), HITL creates a checkpoint where domain expertise, judgment, ethics, or intuition can override or refine the machine’s output.

HITL exists on a spectrum of automation:

Where HITL is critical:

• High-stakes decisions — Medical diagnosis, loan approvals, legal document review, hiring filters
• Edge cases — ML models are confident on 95% of data; the remaining 5% needs human judgment
• Training data — Active learning uses HITL to prioritize which unlabeled examples a human should annotate
• Safety & compliance — Regulated industries (healthcare, finance, defense) often require a human signature on automated decisions
• Model improvement — Human feedback becomes training signal (RLHF — Reinforcement Learning from Human Feedback)

Example:

A medical imaging pipeline where an AI model flags suspicious X-rays, but a radiologist must confirm before the report is sent to the patient.

import json
import random
from dataclasses import dataclass, field
from typing import Optional

─── Domain Models ───
@dataclass
class XRayScan:
    patient_id: str
    image_id: str
    findings: list[str] = field(default_factory=list)
    ai_confidence: float = 0.0

@dataclass
class DiagnosisReport:
    scan: XRayScan
    ai_summary: str
    radiologist_notes: Optional[str] = None
    reviewed_by: Optional[str] = None
    final_diagnosis: Optional[str] = None

─── Stage 1: AI Inference ───
class AIModel:
    """Simulates a chest X-ray classifier."""

    def predict(self, scan: XRayScan) -> dict:
        # Simulate: model returns findings + confidence score
        findings_pool = [
            "No significant abnormalities",
            "Minor opacities detected — likely benign",
            "Nodule detected in upper left lobe — further investigation recommended",
            "Signs of pleural effusion — moderate severity",
            "Cardiomegaly (enlarged cardiac silhouette)",
            "Consolidation consistent with pneumonia",
        ]
        # Simulate uncertain prediction
        is_confident = random.random() > 0.3  # 70% confident

        if is_confident:
            findings = [random.choice(findings_pool[:2])]  # "easy" cases
            confidence = random.uniform(0.92, 0.99)
        else:
            findings = [random.choice(findings_pool[2:])]  # "hard" cases
            confidence = random.uniform(0.55, 0.82)

        scan.findings = findings
        scan.ai_confidence = round(confidence, 4)
        return {
            "findings": findings,
            "confidence": scan.ai_confidence,
        }

─── Stage 2: HITL Routing ───
CONFIDENCE_THRESHOLD = 0.85

def route_for_review(scan: XRayScan) -> str:
    """Decide if this scan needs a radiologist review."""
    if scan.ai_confidence >= CONFIDENCE_THRESHOLD:
        return "auto_approve"
    else:
        return "needs_review"

─── Stage 3: Human Review ───
class Radiologist:
    """A human expert who reviews low-confidence predictions."""

    def __init__(self, name: str):
        self.name = name
        self.reviews_completed = 0

    def review(self, scan: XRayScan, ai_summary: str) -> DiagnosisReport:
        self.reviews_completed += 1

        print(f"\n👩‍⚕️ Radiologist {self.name} is reviewing scan {scan.image_id}...")
        print(f"   AI findings: {scan.findings}")
        print(f"   AI confidence: {scan.ai_confidence:.2%}")

        # Simulate human judgment — may agree, refine, or override
        if random.random() > 0.15:  # 85% agreement
            final_diagnosis = scan.findings[0]
            notes = "Confirmed. No additional findings."
        else:
            final_diagnosis = "Benign scarring, no action needed"
            notes = (
                "AI flagged nodule, but upon review this is linear scarring "
                "from prior infection. No follow-up required."
            )

        return DiagnosisReport(
            scan=scan,
            ai_summary=ai_summary,
            radiologist_notes=notes,
            reviewed_by=self.name,
            final_diagnosis=final_diagnosis,
        )

─── Stage 4: Pipeline Orchestration ───
def run_hitl_pipeline(scans: list[XRayScan]) -> list[DiagnosisReport]:
    model = AIModel()
    radiologist = Radiologist("Dr. Sarah Chen")
    reports = []

    for scan in scans:
        # Step 1: AI predicts
        prediction = model.predict(scan)
        ai_summary = f"Findings: {', '.join(prediction['findings'])} | Confidence: {prediction['confidence']:.2%}"

        # Step 2: Route based on confidence
        action = route_for_review(scan)

        if action == "auto_approve":
            # Auto-approved — no human needed
            report = DiagnosisReport(
                scan=scan,
                ai_summary=ai_summary,
                final_diagnosis=scan.findings[0],
                reviewed_by="AI (auto-approved)",
                radiologist_notes="Confidence threshold met; auto-approved.",
            )
            print(f"✅ Scan {scan.image_id}: AUTO-APPROVED (confidence: {scan.ai_confidence:.2%})")

        else:
            # Human-in-the-loop
            report = radiologist.review(scan, ai_summary)
            print(f"🔍 Scan {scan.image_id}: REVIEWED by {report.reviewed_by}")
            print(f"   Final diagnosis: {report.final_diagnosis}")
            if report.radiologist_notes != "Confirmed. No additional findings.":
                print(f"   ⚠️ Override: {report.radiologist_notes}")

        reports.append(report)

    return reports

─── Run the Pipeline ───
print("🏥 Medical HITL Pipeline — X-Ray Diagnosis")
print("=" * 50)

Simulate 8 patient scans
patients = [
    XRayScan(patient_id=f"P-{i:04d}", image_id=f"X-{random.randint(1000, 9999)}")
    for i in range(8)
]

reports = run_hitl_pipeline(patients)

• Distance field: grid where each cell contains distance to nearest obstacle
• BFS (Breadth-First Search) algorithm to compute this field
• Used in pathfinding, collision detection, shadow mapping
• Key properties:
  • Euclidean distance (true straight-line distance)
  • Signed distance (negative inside obstacles)
  • Smooth gradients (useful for interpolation)

Example:

Visualization of BFS distance field computation:
  - Obstacle grid (walls, trees)
  - Wavefront propagation from obstacles
  - Final distance field (heatmap)

Key Concepts:

• BFS guarantees shortest path in un