Incident

Starting at roughly 15:30 UTC on March 20, 2015 we started seeing that link tracking stopped working as well as an increase in response time for the API in general.

We began investigating the issue and discovered that our API processes had slowed down because they were flooded with the requests serving our stats endpoints. The traffic pattern that arrived triggered an edge condition that caused counters to be aggregated at a higher rate than usual slowing down all queries.

Resolution

To immediately fix the issue we patched the edge condition as well as redirected all traffic that hits the stats endpoint to isolated API workers. This ensures that even if another edge case were to cause the stats endpoint to take longer than normal to respond, it would not affect the other API endpoints.

Normal operation resumed at 17:30 UTC on March 20, 2015.

No emails were lost during the outage.

Future Work

We have been working on our next generation stats engine that allows us to store counters in a pre-aggregated manner which will ensure that this kind of outage does not occur again. We expect to release the next generation stats engine in a few weeks and will be updating customers with more information soon.