Reduce log complexity and cost by filtering

Modern systems create massive amounts of logs. Not all of those are useful. In fact, there's a high chance when you look at your logs you'd find most aren't useful. You might have a service that spews logs for every page load or a backup service which you never need to monitor logs for.

With New Relic you can create drop rules that look at your logs and ignore logs that you haven't selected for ingest. This has a few key benefits:

• Lower costs by storing only the logs relevant to your account.
• Lower costs by dropping specific attributes
• Reduce toil by only storing valuable logs.

How drop filter rules work

A drop filter rule matches data based on a query. When triggered, the drop filter rule removes the matching data from the ingestion pipeline before it is written to the New Relic database (NRDB).

This creates an distinction between the logs being forwarded from your domain and the data that New Relic collects. Since the data removed by the drop filter rule doesn't reach our backend, it can't be queried: the data is gone and cannot be restored.

Decide which logs to drop

Deciding which logs to keep and which logs to drop is a highly specific decision for each team and organization. Logs valuable to one organization may not be valuable to another. Regardless, here are a few suggestions on how to decide which logs are valuable and which to drop:

• What logs does your team rely on today?: If your team already manually reviews a subset of logs regularly, that indicates those logs are valuable and should not be dropped. Likewise, if there is a set of logs your team never looks at that might indicate they should be dropped.
• What apps and systems produce the most logs?: An app or system that creates a large amount of logs indicates you should spend time deciding what to do with those logs. Is it a valuable and widely used app which indicates you should keep most of the logs? Is it a redundent system which is spewing logs with minimal value?

Do take note that while an app or system may be rarely used, that doesn't mean its logs have no value. You would hate to drop logs from an application that is barely used only for that application to go down in a few months with no easy way to troubleshoot.

Filter your log ingest

The following steps will guide you through how to drop logs in the New Relic UI.

Let's say Acme Corp creates 2TB of logs each day. They decide this is too many logs to ingest for both cost and usability reasons. They take a look at their logs and realize over half of their daily logs are from a legacy Node.js application. While they need to keep the app around, they don't care for the logs created by this app. They decided to drop all logs created from the Node.js app.

Navigate to the UI

Go to one.newrelic.com > Logs

Create your drop rule

Filter or query to the specific set of logs that contain the data you want to drop.

There are a few ways to do this, but the easiest is to query for the logs you want to drop. In this case, you would do the following:

1. Select All partitions near the search bar.
2. Enter their query. In this case logtype=node.
3. Press enter and confirm the correct logs appear.
4. Once the query is active, click Create drop filter on the left nav.
5. Give the drop rule a meaningful name.
6. Save the drop filter rule.

Drop attributes

Acme Corp still wants to reduce their ingest. They decided that they don't need certain attributes in their stored logs, so they decide to drop attributes such as purchase_order.

1. In All logs click on a log that contains the attribute you want to drop to open log detail view.
2. Click on the attribute you want to drop to open the attribute menu. In this case purchase_order.
3. Click Create drop filter from attribute.
4. Give the drop rule a meaningful name.
5. Save the drop filter rule.

Repeat the above steps as many times as required until you're happy with your log ingest. If you need help querying for logs and attributes, check out our doc on log specific syntax or our doc on more complex log filtering.