Today at 15:36 UTC two of our Cassandra nodes died.
In all our Cassandra clusters we use production-recommended settings of replication factor 3 and consistency level 2 which provides strong data consistency and allows us to tolerate one node being down without any noticeable impact. Due to the unfortunate circumstances both died nodes belonged to the same cluster.
After discovering and analyzing the issue we immediately started the process of decommissioning the faulty nodes from the cluster. The errors stopped completely at 17:51 UTC and things have been performing smoothly since then.
We utilize multiple Cassandra clusters for different types of messages. The affected cluster is used for storing large (>500KB) messages only so only requests with such messages could have been rejected. Requests with smaller messages are routed to a different set of clusters so they weren't affected.
Investigation of logs also shows that no routes, batch messages and mailing lists were rejected during that time interval.
Some messages and routes that already were in the queue at the time of the incident were delayed due this issue and retried later.
First of all, we're updating our API (both HTTP and SMTP) to return a more appropriate response codes in case of such issues. For example, in case of SMTP we return 554 while the more appropriate code would be 421 Service Not Available which would mean a temporary error making sender retry the message.
Also, we're improving monitoring so that we can react to such issues faster and introducing fallback clusters for big messages so that we can retry messages in case the primary one fails.
Please open a support ticket with any questions.
Thanks, Mailgunners