Routing, Batch Sending and Mailing Lists

Incident Report for mailgun

Postmortem

This morning we identified delays in processing jobs for batch sending, routing (inbound mail), and mailing lists.

We were able to identify the issue as a sub-optimal database query in our new job queue that caused cascading slowdowns. We've since fixed the issue by utilizing a more efficient index as well as upgrading the version of the database we use which includes numerous improvements to the query planner.

We've been monitoring the new job queue since this morning and performance has returned to normal. We'll continue monitoring the job queue to ensure that a slowdown in delivery does not occur again.

Note, no inbound or outbound mail was lost as the new job queue was designed to be more robut in the case of failures such as the one that occurred in the morning.

-Mailgunners

Posted Dec 03, 2015 - 17:16 PST

Resolved

This incident has been resolved.
Posted Dec 03, 2015 - 12:25 PST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Dec 03, 2015 - 11:54 PST

Investigating

We are currently investigating reports of delays with batch sending, routing and mailing lists.
Posted Dec 03, 2015 - 08:02 PST