Incident

Starting at roughly 11:00 AM PDT on March 16, 2015 we started seeing up uptick in Temporary Delivery Failures and MX lookup failures.

We began investigating the issue and soon discovered that one of our delivery servers had become overloaded and almost stopped delivering mail. Internal failover mechanisms had been tripped and we were routing messages to different delivery servers to ensure continued delivery.

Resolution

After investigating the issue, we discovered that the server had exhausted all memory and gotten into a state that didn't allow us to login or access the server remotely. We worked with our Rackspace Support Team and DC Operations Team to restart the server after which we regained access and were able to re-start the server and our delivery software.

Normal delivery resumed at 1:30 PM PDT on March 16, 2015.

No emails were lost, some messages experienced delayed delivery.

Future Work

We are hunting down the memory issues that caused this outage and do not expect this issue to reoccur once it is resolved.

Beware the Ides of March.