Post Mortem: Emails to hostedemail.com Fail, Causing Delivery Queue Backup
By Matthew Tse
Summary
A delivery failure to hostedemail.com caused a cascading backup in our email delivery queue, temporarily delaying deliveries to all destinations.
What Happened
Starting at 08:00 UTC, hostedemail.com began rejecting all incoming connections with a 421 error (temporary failure). Our delivery workers retried these messages according to our standard retry policy, but the volume of retries consumed delivery worker capacity.
Impact
Email deliveries to all destinations were delayed by 15-30 minutes as the delivery queue backed up. No emails were lost.
Root Cause
hostedemail.com experienced their own infrastructure issues and began rate-limiting all incoming SMTP connections. Our retry logic, while correct, created a feedback loop where retry attempts consumed worker threads that would otherwise process new deliveries.
Resolution
Implemented destination-aware queuing so that failures to a single destination don't impact deliveries to other destinations. Added circuit-breaker logic that temporarily pauses retries to a destination experiencing persistent failures.
Lessons Learned
Single-destination failures should never cascade to affect the entire delivery pipeline. Our new circuit-breaker approach isolates problematic destinations while maintaining full throughput for healthy ones.