On February 19 at 00:00 UTC a configuration change in our Elasticsearch backend became activated. This prevented new events from being indexed properly, resulting in the loss of events between 00:00 and 00:17 UTC.
The goal of the configuration change was to make it easier to re-index events in cases where system failure would result in events not being indexed. Each event normally results in one document in Elasticsearch. Every event is assigned a unique identifier, and Elasticsearch would be configured to treat that identifier as the document id (instead of an identifier generated by Elasticsearch itself). Indexing an event a second time would then just override the original document, instead of adding a new one, making indexing an idempotent action.
The approach taken was to change the mapping of the _id field, so that it points to the id field present in each event. Elasticsearch could then take the value for that field and use it as the document id.
However, it turns out that field path references can be ambigious, so that a path of id might also refer to account.id. This results in all new events with the same account.id to be written to a document with that id, yielding a single document for each customer. Whether or not the right field is used, is a matter of chance at index creation time. Unfortunately, this was not encountered during testing.
Indexing was stopped around 00:17 UTC. Then the template for new indices with the old mapping was restored, and the index recreated. Finally, indexing was resumed around 00:20 UTC to work back the backlog since 00:17 and index future events properly.
Initially, it was thought that all events between 00:00 and 00:17 UTC were lost. Fortunately, we were able to recover about 60% of the events in this time frame. These events were retroactively indexed around 13:00 UTC.
With the ambiguity well known with Elasticsearch, as mentioned in the ticket referenced above, this particular way of providing a document id will likely be deprecated entirely. The alternative is to provide document ids at indexing time explicitly. This approach has since been tested and will be implemented in the near future.