Definition:
“A batch processing system takes a large amount of input data, runs a job to process it, and produces some output data.
Designing Data-Intensive Applications, M Kleppmann”
There are three important parts to this definition:
Takes a large amount of input data: This means that it does not make sense to run batch processing on top of a small amount of data that can be easily handled by request-response pattern. Runs a job: This means it could be run periodically or based on some external trigger. In the context of batch processing, every time the system processes a bunch of data, we call that the system is running a job. This job runs for a few minutes to hours, or even for days. Produces some output data: This means there is an outcome when the batch job finishes. Generally, the outcome is the generation of a set of output data. One important aspect of batch processing is the execution of the job by using many processes, typically using a cluster of nodes.
Example:
- Payroll Processing System
Monthly Salary Calculation:
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Employee DB │───▶│ Batch Processor │───▶│ Salary Reports │
│ (10M records) │ │ (Multiple nodes) │ │ & Payments │
└─────────────────┘ └──────────────────┘ └─────────────────┘- E-commerce Analytics:
# Daily sales report generation
Input: 24 hours of transaction data
Process: Calculate metrics across multiple servers
Output: Sales dashboard, inventory updates, recommendations