Earlier today, a poorly formed search query was executed against one of our Elasticsearch clusters containing customer event logs. This query consumed all available search queues across the cluster and after numerous attempts, we were unable to terminate the queries while the cluster was online. As a result, we performed a rolling restart of the cluster, which resulted in a rebalance of the nodes. Once the rebalancing was completed, we re-enabled the events API and logs section of the control panel.
To help mitigate future issues, we've deployed timeouts on queries that will help prevent long-running operations from compromising the performance of our search infrastructure.
Posted Jul 14, 2017 - 16:03 PDT
Monitoring
The Events API and Logs tab have been bought back up and are now in a functional state. We are monitoring for continued issues.
Posted Jul 14, 2017 - 15:15 PDT
Identified
We have temporarily suspended the Events API and Logs tab while we continue to bring our Logs cluster back to a functional state.
Posted Jul 14, 2017 - 12:46 PDT
Investigating
We are currently investigating timeouts and errors with Logs and Event API.