Incident

On January 22, 2015 at roughly 3:00 PM UTC our monitoring system started reporting sending delays. Shortly thereafter our customers started reporting delayed delivery times to our support team. After looking at our internal system monitoring tools, we noticed that that message delivery time had been slowly going up and was now above our 9 second threshold.

Investigation

Our investigation revealed that we didn’t have enough resources to deliver the amount of requests we were receiving. We had enough workers to accept and deliver messages, but our workers that process messages were incorrectly allocated. We had 40 worker processes that were sitting idle due to a failure in our load balancing algorithm.

Resolution

We manually rebalanced our worker processes to quickly clear out our queue. This brought us back into a state of having more than enough resources dedicated to processing messages.

Scope

No customer messages were lost. Most customers’ message delays were measured in minutes, some customers saw delays that could be measured in hours.

Future Work

We had already begun work improving our load balancing and rebalancing algorithms. We will continue that work on this aspect of Mailgun as well as adding an additional focus on multi-tenant isolation. This will ensure that all customers get a fair share of Mailgun processing time and their messages can be delivered in a timely manner.