At 07/14 ~01:40 AM UTC one of our beanstalkd servers which we’re currently using for asynchronous jobs started replying with OUTOFMEMORY error to any attempt to insert a new job. The logic our clients used to fallback to standby beanstalkd server was considering connection errors only and didn’t take erroneous responses from beanstalkd into account so the fallback didn’t happen.

At 07/14 ~05:00 AM UTC the malfunctioning beanstalkd server was restarted at which point the normal operation of asynchronous jobs resumed.

Unfortunately, due to this deficiency in the producers’ fallback mechanism, majority of incoming messages (routes), batches and mailing lists during that time period were lost. No other systems, including regular messages processing, were affected.

We’re very sorry about this incident and are working to resolve the outstanding issues as soon as possible:

• We have tweaked our monitoring systems to alert us about such incidents as soon as they are detected which will let us to react faster.
• We are working on improving the fallback algorithm in job producers.

Please do not hesitate to reach out to our support for any questions.

-Mailgunners