No table data
Monday morning: a dashboard that normally refreshes in seconds now lags by minutes. Overnight batch jobs keep retrying and never finish, so downstream tables stop updating. Someone says “it’s big data now” as if the label explains the outage. What actually changed is simpler: the workload crossed a constraint boundary. A single machine can no longer store, ingest, or serve the data with the required reliability. Once that happens, distribution stops being an optimization and becomes the only way to keep the system within its SLOs.
Big data is data whose constraints force you to spread storage and processing across multiple machines. The size itself matters only because it drives those constraints.
A practical way to diagnose the trigger is the 4Vs, each tied to a concrete system limit:
Rule of thumb
If meeting your SLO requires sharding, replication, or asynchronous pipelines, you are already in big data territory.
Generate custom courses on any topic — with hands-on practice, AI guidance, and visuals built in.
Already have an account?
Once you distribute, you start trading one property for another. Throughput is how much work you finish per unit time; latency is how long one request takes end to end. Batch systems push throughput high by buffering and parallelizing. Interactive systems protect latency by limiting queueing, precomputing, or reserving capacity.
Consistency means readers observe the latest committed write under a defined model; availability means the system keeps serving requests during faults. Under a network partition, you usually pick which one degrades.
A third lever is disaggregated compute and storage, where stateless compute clusters read and write shared storage. This improves elasticity and utilization, but increases network dependency and makes data movement a first-class design concern.
After you see the diagram, notice that every “fix” pulls on another edge. Faster queries often mean more copies and more coordination. Higher availability often means weaker coordination under partitions.
In distributed systems, partial failure is routine: one node, one disk, one rack, or one network link fails while the rest keeps running. That reality drives architecture choices more than ideal-case performance.
Retries help, but only when operations are idempotent, meaning repeated execution produces the same end state. Without idempotency, retries create duplicates, double-counts, and phantom updates. When velocity is high, the system must also apply backpressure, a mechanism that slows producers or sheds work when downstream cannot keep up, so queues do not grow without bound.
Reliability lens
Treat retries, deduplication, and backpressure as part of your data model, not as afterthoughts in ops.