Define timestamp offsets for incremental alerts (Optional)

Incremental alerting is based on a document’s analysis timestamp. Since there is a lag between analysis and indexing, there is the chance an alert may run after the analysis of a new batch of documents has started, but before all of these documents have been added to the index.

As a result, there’s a risk that some matching documents will be excluded from both the current alert and the alert that follows.

To prevent this, there is a timestamp offset. The default value is 900s (15 minutes), which you can change. The value needed for the offset depends on typical lag time between analysis and indexing, and the frequency of alerts.

For example, say 10 documents enter the analysis pipeline at 1:00.

This page discusses:

See Also
Changing the timestamp offset

Without an offset

  • An alert runs at 1:05. It looks for all matching documents in the index with an analysis timestamp between “now” (1:05) and the previous alert.

    However, at this time only 4 of the 10 documents from the 1 pm analysis have been saved to the index, so only these 4 documents are included in the alert notification.

  • The next time the alert runs at 2:05, it looks for all matching documents in the index with an analysis timestamp that falls between 2:05 and the previous alert from 1:05.

  • Even though the 6 remaining documents from the 1:00 analysis are now saved to the index, since their analysis timestamp (1:00) precedes the previous alert (1:05), they are excluded from the 2:05 alert notification.

With an offset

  • An alert runs at 1:05. It looks for all matching documents in the index with an analysis timestamp that falls between “now minus 15 min” (12:50) and the previous alert.

    By offsetting “now”, none of the 10 documents from the 1:00 analysis are included in the 12:50 alert notification.

  • The next time the alert runs at 2:05, it looks for all matching documents in the index with an analysis timestamp between “now minus 15 min” (1:50) and the previous alert (12:50).

  • Therefore, all 10 matching documents with a 1:00 analysis timestamp are included in the 2:05 alert notification.

Conversely, if alerts are so frequent that several run between analysis and indexing, an offset that is too large could result in the same documents with the same analysis timestamp being included in multiple alert notifications.