Drop data using Prometheus remote write

You can drop data you don't want to keep by changing the remote_write section of the YAML config file.

Drop entire metric data points from remote write integration

If a target is sending a noisy metric that you don't want sent to New Relic, you can specify that New Relic should drop that data.

Example

Let's say you don't want to receive data for the metric node_memory_active_bytes from an instance running at localhost:9100. Using the write_relabel_config entry shown below, you can target the metric name using the __name__ label in combination with the instance name.

remote_write:
- url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=macbook-server-cluster
  bearer_token: <redacted>
  write_relabel_configs:
    - source_labels: ['__name__', 'instance']
      regex: 'node_memory_active_bytes;localhost:9100'
      action: 'drop'

This tells Prometheus that you want to do some action against metrics with these labels. To limit which metrics with these labels are affected, you must include some value for regex. By default this value is set to .* and it will include all metrics. In this case, it will drop all metric data points coming out of Prometheus via remote write.

Drop specific labels or attributes from data points

If a target is sending specific labels or attributes you're not interested in receiving, you can drop these from the metrics you receive.

Example

Let's say one of your targets is sending a bunch of extra attributes you're not interested in receiving. These might include things like high cardinality attributes such as unique machine identifiers, JVM IDs, or similar. In this case, you need to change both the remote_write and the scrape_configs section of the YAML file.

The result will look something like this:

remote_write:
- url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=macbook-server-cluster
  bearer_token: <redacted>
  write_relabel_configs:
    - regex: 'extraLabelToRemove.*'
      action: 'labeldrop'
...
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any time series scraped from this config.
  - job_name: 'node'
    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']
        labels:
          group: 'production'
          keepLabelName1: 'please-keep-me'
          extraLabelToRemove: 'please-remove-me'
          extraLabelToRemove1: 'please-remove-me'
          extraLabelToRemove2: 'please-remove-me'
          extraLabelToRemove4: 'please-remove-me'
          extraLabelToRemove3: 'please-remove-me'
          extraLabelToRemove5: 'please-remove-me'

Prometheus or NerdGraph?

There are advantages to both dropping data using the method described on this page and using NerdGraph. This section is intended to help you figure out which method is better for your specific needs and preferences.

Considerations for the Prometheus config file method

With this method, your dropped data never leaves the associated Prometheus instance. This is a valuable feature if bytes transferred is a cost consideration on the app hosting side.

However, this method may be less appealing than the NerdGraph option due to the following considerations:

• Maintained via config yaml files that need to be loaded onto each Prometheus instance (or via a shared storage mechanism)
• Requires access to Prometheus server, meaning that either:
• • The server needs to be restarted
• • Served must be be accessed at port with path /-/reload (assuming the server has lifecycle management enabled as described here in the Prometheus configuration docs.

Considerations the NerdGraph method

NerdGraph is a great option if you want to manage all your data dropping in a single place. It can also be updated easily via the API and requires no restart or interaction with Prometheus. However, this method applies rules to all incoming data points. This means that you should set up your rules with careful consideration using WHERE filtering.

For more information, see Drop data using NerdGraph.

Learn more

• Send Prometheus metric data to New Relic
• Prometheus High Availability (HA)