While performing a root cause analysis of last nights outage, we discovered that the outage affected more customers than originally assessed.

As we mentioned in the incident, last night during routine maintenance of our etcd cluster we had an outage caused by a misconfiguration that took longer than expected to resolve. During that time job processing was also affected as it relies on the etcd cluster for internal job routing. Job processing consists of batch sending (sending to 20 or more recipients), routing (incoming messages), and mailing lists.

During the outage, because the job processing service could not route jobs correctly, jobs were retried at 30 minute intervals which lead to some customers experiencing double delivery during the outage. Once the cluster was restored, jobs were once again routed correctly, and double delivery stopped.

Note: No batch jobs, routes (incoming messages), or mailing lists were lost. Once the etcd cluster was restored, double delivery also ceased.