On Monday February 16, around 23:32 UTC, a spike in memory usage was recorded for all data nodes in our Elasticsearch cluster for customer logs. A few minutes later, this caused the queue of to-be-indexed events to grow rapidly and induced much slower query response times in the Events API. This triggered several alarms.
Memory spikes like the above can happen when a memory-intensive query is performed and is usually immediately resolved by regular Garbage Collection processes in Elasticsearch. In this case the cluster did not recover as usual, and there was no evidence of memory-intensive queries.
As a temporary measure, we shut down the service that provides the Events API that is backed by this Elasticsearch cluster, to remove immediate strain. When that didn't yield the desired effect, we traced the problem down to one of the nodes not recovering properly, which caused the cluster to slow down queries and stop indexing new events.
After several tests, the node was forcibly restarted at 00:37. The cluster immediately recovered and started indexing the large backlog of events along with new events. Indexing was caught up at 00:52. At 01:20, the Elasticsearch cluster was fully recovered, at which point the Events API was re-enabled.
No events were lost during this outage.
We haven't been able to trace down the immediate cause of the large memory spike, so we are adding instrumentation to be able to dig further into future similar scenarios.