NerdGraph tutorial: Loss of signal and gap filling

You can customize New Relic alerting loss of signal detection and gap filling using our NerdGraph API. For example, you can configure how long to wait before considering the signal lost, or what value should be used for filling gaps in the time series.

Loss of signal occurs when New Relic stops receiving data for a while; technically, we detect loss of signal after a significant amount of time has elapsed since data was last received in a time series. Loss of signal can be used to trigger or resolve an incident, which you can use to set up alerts.

Gap filling can help you solve issues caused by lost data points. When gaps are detected between valid data points, we automatically fill those gaps with replacement values, such as the last known values or a static value. Gap filling can prevent alerts from triggering or resolving when they shouldn't.

In this guide we cover the following:

• Customize loss of signal detection
• Customize gap filling

Customize your loss of signal detection

Loss of signal detection opens or closes incidents if no data is received after a certain amount of time. For example, if you set the duration of the expiration period to 60 seconds and an integration doesn't seem to send data for more than a minute, a loss of signal threshold would be triggered.

You can configure the duration of the signal loss and whether to open an incident or close it by using these three fields in NerdGraph:

• expiration.expirationDuration: How long to wait, in seconds, after the last data point is received by our platform before considering the signal as lost. This is based on the time when data arrives at our platform and not on data timestamps. The default is to leave this null, and therefore this wouldn't enable Loss of Signal Detection.
• expiration.openViolationOnExpiration: If true, a new incident is opened when a signal is lost. Default is false. To use this field, a duration must be specified.
• expiration.closeViolationsOnExpiration: If true, open incidents related to the signal are closed on expiration. Default is false. To use this field, a duration must be specified.

You also have the option to skip opening an incident when the signal is expected to be lost. To do this, use this field in NerdGraph:

• expiration.ignoreOnExpectedTermination: If true, an incident will not be opened when the signal is expected to be lost. To indicate a signal is expected to be lost, the tag termination: expected must be present on the entity. Default is false. For infrastructure host entities, the tag hostStatus: shutdown will also indicate the signal is expected to stop and prevent an incident from opening.

View loss of signal settings for an existing condition

Existing NRQL conditions may have their loss of signal settings already configured. To view the existing condition settings, select the fields under nrqlCondition > expiration:

{
  actor {
    account(id: YOUR_ACCOUNT_ID) {
      alerts {
        nrqlCondition(id: NRQL_CONDITION_ID) {
          ... on AlertsNrqlStaticCondition {
            id
            name
            nrql {
              query
            }
            expiration {
              closeViolationsOnExpiration
              expirationDuration
              openViolationOnExpiration
              ignoreOnExpectedTermination
            }
          }
        }
      }
    }
  }
}

You should see a result like this:

{
  "data": {
    "actor": {
      "account": {
        "alerts": {
          "nrqlCondition": {
            "expiration": {
              "closeViolationsOnExpiration": false,
              "expirationDuration": 300,
              "openViolationOnExpiration": true,
              "ignoreOnExpectedTermination": false
            },
            "id": "YOUR_ACCOUNT_ID",
            "name": "Any less than - Extrapolation",
            "nrql": {
              "query": "SELECT average(value) FROM AlertsSmokeTestSignals WHERE wave_type IN ('min-max', 'single-gap') FACET wave_type"
            }
          }
        }
      }
    }
  },
  ...

Create a new condition with loss of signal settings

Let's say that you want to create a new create a NRQL static condition that triggers a loss of signal threshold after no data is received for two minutes. You would set expirationDuration to 120 seconds and set openViolationOnExpiration to true, like in the example below.

mutation {
  alertsNrqlConditionStaticCreate(
    accountId: YOUR_ACCOUNT_ID
    policyId: YOUR_POLICY_ID
    condition: {
      name: "Low Host Count - Catastrophic"
      enabled: true
      nrql: {
        query: "SELECT uniqueCount(host) FROM Transaction WHERE appName='my-app-name'"
      }
      signal {
        aggregationWindow: 60
        aggregationMethod: EVENT_FLOW
        aggregationDelay: 120
      }
      terms: [{
        threshold: 2
        thresholdOccurrences: AT_LEAST_ONCE
        thresholdDuration: 600
        operator: BELOW
        priority: CRITICAL
      }]
      valueFunction: SINGLE_VALUE
      violationTimeLimitSeconds: 86400
      expiration: {
        expirationDuration: 120
        openViolationOnExpiration: true
      }
    }
  ) {
    id
    name
  }
}

Update the loss of signal settings of a condition

What if you want to update loss of signal parameters for an alert condition? The following mutation allows you to update a NRQL static condition with new expiration values.

mutation {
  alertsNrqlConditionStaticUpdate(
    accountId: YOUR_ACCOUNT_ID
    id: YOUR_STATIC_CONDITION_ID
    condition: {
      expiration: {
        closeViolationsOnExpiration: true
        expirationDuration: 300
        openViolationOnExpiration: false
        ignoreOnExpectedTermination: true
      }
    }
  ) {
    id
    expiration {
      closeViolationsOnExpiration
      expirationDuration
      openViolationOnExpiration
      ignoreOneExpectedTermination
    }
  }
}

Customize gap filling

Gap filling replaces gap values in a time series with either the last value found or a static, arbitrary value of your choice. We fill gaps only after another data point has been received after the gaps in signal (after data reception has been restored).

You can configure both the type of filling and the value, if the type is set to static:

• signal.fillOption: Type of replacement value for lost data points. Values can be:
  • NONE: Gap filling is disabled.
  • LAST_VALUE: The last value seen in the time series.
  • STATIC: An arbitrary value, defined in fillValue.
• signal.fillValue: Value to use for replacing lost data points when fillOption is set to STATIC.

For example, here's how to create a static NRQL condition with gap filling configured:

mutation {
  alertsNrqlConditionStaticCreate(
    accountId: YOUR_ACCOUNT_ID
    policyId: YOUR_POLICY_ID
    condition: {
      enabled: true
      name: "Example Gap Filling Condition"
      nrql: { query: "SELECT count(*) FROM Transaction" }
      terms: {
        operator: ABOVE
        priority: CRITICAL
        threshold: 1000
        thresholdDuration: 300
        thresholdOccurrences: ALL
      }
      valueFunction: SINGLE_VALUE
      violationTimeLimitSeconds: 28800
      signal: {
        aggregationWindow: 60
        aggregationMethod: EVENT_FLOW
        aggregationDelay: 120
        fillOption: STATIC
        fillValue: 1
      }
    }
  ) {
    id
  }
}