Large-Scale Data Processing

Processing 25M+ records (roughly 4GB of data) in one pass isn't a matter of writing a script that works — it's making sure it doesn't run out of memory, stall on duplicates, or leave the dataset in a half-processed state if it fails partway through.

At this volume, naive approaches break: loading everything into memory at once isn't viable, duplicate detection across the full dataset needs an efficient strategy, and any failure mid-run has to be recoverable without reprocessing everything from scratch.

Data is read and processed in bounded batches rather than loaded wholesale, with validation and duplicate detection applied per batch before writes hit the database. Progress is tracked so failures don't require a full restart.

• Batch-based reads keep memory usage bounded regardless of total dataset size.
• Duplicate detection applied at the batch level to avoid comparing every record against the full dataset in memory.
• Database writes structured to minimize lock contention and round-trips under sustained load.
• Progress tracking so a failure mid-run doesn't mean starting over.

• Bounded memory usage regardless of total dataset size.
• Progress tracking allows recovery without reprocessing completed batches.
• Validation applied before writes to avoid corrupting the database with malformed records.

A batch pipeline that reliably processes the full 25M+ record dataset without memory exhaustion or unrecoverable failures mid-run.