Post Mortem: Slow SQL Queries and Missing SQL Indexes
By Matthew Tse
Summary
On October 8, 2025, several API endpoints experienced degraded performance due to slow SQL queries. The root cause was identified as missing database indexes on frequently queried columns.
What Happened
Starting around 09:15 UTC, response times for alias lookup and domain listing endpoints increased from ~50ms to over 2 seconds. Our monitoring flagged the degradation within minutes.
Root Cause
As our user base grew, certain query patterns that previously performed adequately began to slow down. Specifically, queries filtering by account_id on the aliases table were performing full table scans because the column lacked an appropriate index.
Impact
API response times were degraded for approximately 45 minutes. Email forwarding was unaffected. Dashboard loading times increased significantly during this period.
Resolution
Added composite indexes on the identified columns. Query performance returned to normal immediately after index creation.
Lessons Learned
We've implemented automated slow query detection that alerts before performance degrades to user-impacting levels. Regular index analysis is now part of our monthly database maintenance.