We have resolved the final connectivity issues to the SMTP endpoint. Again, we'll prepare a full RCA within 72 hours of this point. Thanks for your patience throughout this incident!
Posted Sep 30, 2014 - 11:13 PDT
Update
We are still seeing a very small subset of users experiencing connectivity issues, if you are still seeing these issues, please contact us.
Posted Sep 30, 2014 - 10:07 PDT
Verifying
At this time, we've made additional adjustments and have seen SMTP connections improve. We will be moving this status to Verifying/Monitoring to collect any late reports. In addition, we are marking all services as fully operational. We've received several requests for an RCA and will provide one within 72 hours of closing the incident. Please subscribe to the Status Page alerts to receive a notification when the RCA is published.
Posted Sep 29, 2014 - 22:05 PDT
Update
We've failed over Load Balancers, made additional adjustments, and are seeing success with SMTP connections. If you're still experiencing issues, please open or update your ticket, so we can gather additional details from you. Thanks!
Posted Sep 29, 2014 - 21:06 PDT
Update
In 10 minutes, we will be failing over to an alternate load balancer. For those with persistent connections, they will be terminated, but can be re-established immediately. All other connections should continue on to the alternate load balancer without interruption.
Posted Sep 29, 2014 - 18:22 PDT
Update
We are continuing to work with customers reporting SMTP connectivity issues. If you're still having trouble connecting, please ensure you have a ticket open with us, so we can work with you on debugging.
Posted Sep 29, 2014 - 15:01 PDT
Update
Some domains utilizing an IP within the block 209.61.151.0/24, with messages queued between Sun, 28 Sep 2014 18:50:00 GMT - Mon, 29 Sep 2014 08:00:0 GMT, may experience duplicate email deliveries. We are trying to minimize this as much as possible. Thanks for your patience!
Posted Sep 29, 2014 - 12:40 PDT
Update
We believe we've made substantial progress on resolving the majority of SMTP connectivity issues. We're continuing to monitor.
Posted Sep 29, 2014 - 12:14 PDT
Update
SSO from Rackspace's Cloud Control Panel to Mailgun's Control Panel has been resolved. If you continue seeing issues with this functionality, please contact help@mailgun.com.
Posted Sep 29, 2014 - 11:42 PDT
Update
On Friday, Sep 26th at 4pm PDT, we were informed that Rackspace would be performing reboots of their performance cloud instances (https://status.rackspace.com/).
We planned to fail over to our alternate locations to minimize the impact. We have subsequently noticed connectivity issues caused by erroneous F5 LB behavior. This is resulting in blocked packets causing connection timeouts for some customers.
We are currently working with Rackspace and F5 to resolve this issue.
Posted Sep 29, 2014 - 11:24 PDT
Update
We have identified TLS connectivity issues. These are caused by errorneous F5 LB behavior of unidentified nature, resulting in randomly blocked packets causing connection timeouts. We are continuing our research.
Posted Sep 29, 2014 - 03:40 PDT
Update
The API and SMTP endpoints have been stable for an hour. We are still working through the previously mentioned dedicated IPs with messages queued. For clarity, if you have a dedicated IP in the 209.61.151.0/24 subnet, we are working on releasing those messages as quickly as we can. All other messages are delivering promptly.
Posted Sep 28, 2014 - 23:47 PDT
Update
A small percentage of customers with dedicated IPs may currently see messages Accepted, but not yet Delivered, within their logs. Those messages remain queued and will be released shortly. Thanks for your patience.
Posted Sep 28, 2014 - 22:34 PDT
Update
Between 3:00AM UTC - 3:12AM UTC, you may see a lack of events on the Events API and Logs Control Panel tab. Messages related to the events were unaffected in their normal delivery process. We do not anticipate being able to recover these events. However, we'll keep you updated.
Posted Sep 28, 2014 - 20:56 PDT
Update
We're speedily recovering nodes that have been rebooted. Several nodes are remaining. If a message is queued, it will be delivered. However, some delays may be present. We'll work to crunch any queues through the night. Thanks for your patience.
Posted Sep 28, 2014 - 20:20 PDT
Update
We are currently aware of failures accessing Mailgun's Control Panel from Rackspace's Cloud Control Panel. We do not have an ETA on resolution. We can provide an emergency workaround if you email help@mailgun.com.
Posted Sep 28, 2014 - 19:00 PDT
Update
Rackspace is now beginning rolling reboots of Cloud instances throughout the ORD datacenter, where a majority of Mailgun infrastructure resides. We planned to fail over to our alternate location to minimize impact, however, we have been unable to do so. As a result, you may see connectivity issues to Mailgun's API and SMTP endpoints. We have all hands on deck to recover services immediately. We are truly sorry for the inconvenience this will cause. Please reach out to us if we can help in any way.
Posted Sep 28, 2014 - 18:27 PDT
Update
API connectivity has been restored. Thanks for your patience.
Posted Sep 28, 2014 - 16:49 PDT
Update
We're seeing higher failure rates on the API. Hang in there, we're digging in.
Posted Sep 28, 2014 - 16:03 PDT
Update
We've made a few adjustments to resolve the API connectivity issues. We're continuing to monitor. Thanks for your patience.
Posted Sep 28, 2014 - 14:52 PDT
Update
We are currently seeing intermittent connectivity on the API endpoint. All hands on deck working to resolve.
Posted Sep 28, 2014 - 13:50 PDT
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Sep 28, 2014 - 04:00 PDT
Scheduled
Rackspace, our parent company and hosting provider, will be performing datacenter maintenance that may impact Mailgun's infrastructure between Sunday 9/28, 11:00AM UTC through Monday, 9/29 11:00AM UTC.
We do not expect the API or SMTP endpoints to be affected. Mail delivery should be unaffected as well.
Services such as events/domains/validation APIs may experience periodic network disruption during the maintenance window.
Please follow this maintenance notification. We will keep you informed through our status page (http://status.mailgun.com) and Twitter (@mail_gun).