Around 10:00 UTC we got alarms about service degradation. We had several dips in 'accepted' events over a two hour period. After inspecting our own logs and graphs, we spent about an hour on the phone Rackspace's Network Security team to get to the bottom of things.
We checked the history of our load balancers and found spikes in the number of connection attempts around the same time of the dips in our graphs. Rackspace DNS infrastructure was experiencing DDoS attack around the same time: https://status.rackspace.com/index/viewincidents?group=14&start=1419224400. This affected some parts of our internal infrastructure relying on DNS.
Posted Dec 22, 2014 - 17:14 PST
Resolved
We figured out that the issue was related to a side-effect caused by last night's DDoS attack on Rackspace DNS servers. All services are fully operational now.
Posted Dec 22, 2014 - 15:48 PST
Update
We are working with our Rackspace colleagues to identify the cause of dropped connections to our SMTP and API endpoints. At this moment, everything looks to be back to normal, but we are still analyzing logs.
Posted Dec 22, 2014 - 04:26 PST
Investigating
We have noticed short-lived drops in the number of accepted messages and are investigating the cause.