Dead Letter Notes https://deadletternotes.com/ Notes on data pipelines, message queues, and the failure modes in between. en-us The small files problem, and four ways I've dealt with it https://deadletternotes.com/posts/small-files-problem/ https://deadletternotes.com/posts/small-files-problem/ Fri, 12 Jun 2026 09:00:00 GMT Why millions of tiny files wreck data lake performance, and practical ways to fix it. Schema drift: catching breaking changes before your consumers do https://deadletternotes.com/posts/schema-drift/ https://deadletternotes.com/posts/schema-drift/ Sun, 03 May 2026 09:00:00 GMT How upstream schema changes quietly break downstream pipelines, and how to catch them early. Backfilling a partitioned table without taking down production https://deadletternotes.com/posts/backfilling-partitioned-tables/ https://deadletternotes.com/posts/backfilling-partitioned-tables/ Sat, 28 Mar 2026 09:00:00 GMT A calm way to backfill large partitioned tables while production keeps reading and writing. Idempotent consumers, or how to stop fearing retries https://deadletternotes.com/posts/idempotent-consumers/ https://deadletternotes.com/posts/idempotent-consumers/ Sun, 15 Feb 2026 09:00:00 GMT Making consumers idempotent so at-least-once delivery stops keeping you up at night. Dead letter queues in practice https://deadletternotes.com/posts/dead-letter-queues-in-practice/ https://deadletternotes.com/posts/dead-letter-queues-in-practice/ Tue, 20 Jan 2026 09:00:00 GMT What actually goes wrong with dead letter queues once real traffic hits them.