Network outage

Incident Report for mailgun

Postmortem

Description

On June 24 2015 at roughly 17:50 UTC our IAD data center began dropping all inbound and outbound traffic, our ORD data center remained fully functional. Because of this, customers began experiencing intermittent connection issues when their request would reach IAD instead of ORD.

At roughly 18:30 UTC we noticed that we were unable to connect to our IAD data center as all connections were timing out. Our monitoring tools also reported that network traffic in and out of our data center was close to zero. We updated our status page to tell customers to connect to ORD directly to avoid the connectivity issues.

At roughly 19:00 UTC we updated our DNS records to remove the A record for our IAD data center. Once this DNS change was propagated, connections were only made to ORD and this temporarily resolved the connectivity issues for most customers. However, because we were running out of a single data center, some customers have have experienced degraded performance during this time.

At roughly 23:00 UTC once we had confirmed the issues with our IAD data center were resolved, we updated our DNS records to add back the A record for our IAD data center. Once this DNS change was propagated, load on our ORD data center decreased and response times returned to normal.

Cause

At the end of March 2015 our parent company Rackspace performed network maintenance in our IAD data center to upgrade our networking equipment and F5. Unbeknownst to us this maintenance was never completed. This left our IAD data center in a half complete state. This wasn't noticed earlier because the incorrect configuration was never applied to our F5 or other networking equipment.

However at roughly 17:50 UTC Mailgun triggered an event which caused automated tools at Rackspace to apply the incorrect configuration to our networking equipment. This lead to our primary and secondary F5 load balancers to both go into standby mode. This effectively dropped all traffic in and out of our IAD data center.

Prevention

As of roughly 23:00 UTC Rackspace had applied the correct configuration to our networking equipment and completed the network maintenance to ensure that incorrect configuration will not be applied again.

We have simulated the event that triggered failure and can verify it did not cause any further issues.

Posted Jun 24, 2015 - 17:48 PDT

Resolved

Our queues are down to normal levels, response times are normal. The cause of the network outage in our alternate data center has been resolved.

Posted Jun 24, 2015 - 16:57 PDT

Monitoring

We have restored our secondary data center; queues are reducing in size. We expect to resume to normal response times in a few minutes.

Posted Jun 24, 2015 - 16:18 PDT

Update

We are experiencing high queue volumes which are causing timeouts on our HTTP API endpoint.

Posted Jun 24, 2015 - 15:56 PDT

Update

DNS entries have been updated to direct traffic to our unaffected data center. Please be patient while the DNS propagates.

Posted Jun 24, 2015 - 11:51 PDT

Update

For API and SMTP endpoints, you may use api2.mailgun.net if your application requires a hostname until our DNS update propagates.

Posted Jun 24, 2015 - 11:35 PDT

Update

For SMTP endpoint, use 50.56.21.178.

Posted Jun 24, 2015 - 11:24 PDT

Update

Mailgun is experiencing a network outage in one of our data centers. If possible, please use https://50.56.21.178 for the Mailgun API endpoint until we can stop resolving to our unresponsive endpoint.

Posted Jun 24, 2015 - 11:23 PDT

Identified

Mailgun is experiencing a network outage in one of our data centers. If possible, please use http://50.56.21.178 for the Mailgun API endpoint until we can stop resolving to our unresponsive endpoint.

Posted Jun 24, 2015 - 11:21 PDT