Introduction to AWS Lambda monitoring

As your organization transitions to a Function-as-a-Service (FaaS) architecture with AWS Lambda, understanding and optimizing function performance becomes critical for ensuring a seamless and efficient user experience. Each Lambda invocation represents a potential bottleneck or opportunity for improvement.

New Relic's instrumentation for AWS Lambda provides deep visibility into the inner workings of your functions. By adding the New Relic agent to your function, each time your function is invoked, New Relic is too. You can gain insights into key performance metrics such as duration, cold starts, exceptions, and tracebacks.

This doc explains New Relic's Lambda monitoring architecture and walks you through linking your AWS and New Relic accounts. You must link your accounts before you can instrument your functions.

How the Lambda instrumentation works

These are the essential elements of AWS Lambda instrumentation with New Relic:

Your function: Your function is the code you want to understand. You want to know when it's encountering errors, why it's slow, or how often it gets invoked.
The New Relic agent or SDK: Depending on the language your function is written in, New Relic provides different agents or SDKs. Their job is to do the actual monitoring of your code. It measures the duration of your function invocations, notes errors that occur, records details about the source events, and your functions responses. To do this, it needs to wrap around your Lambda invocation handler function.
The New Relic Lambda extension: When you instrument the New Relic Lambda extension in your function, it will run inside the Lambda execution environment, alongside your code. It enhances the telemetry that the agent collects and sends it to New Relic's backend in batches. It can also send your function's logs to New Relic. You don't need the Lambda extension to monitor your function with New Relic.

Choose your shipping method

There are three ways to ship your New Relic AWS Lambda telemetry. Which shipping method you choose depends on your data needs.

A screenshot depicting the different shipping methods for Lambda

Cost: The most cost-effective way to instrument with AWS Lambda is by using any of our layers, which include the Lambda extension. By default, our extension never sends data to AWS CloudWatch.
Reliability: You can ship your data through CloudWatch only. To do this, you will disable the New Relic Lambda extension. This means that your Lambda function will be lighter and won't have any additional processes running when it's invoked. In this case, logs and payloads will be sent to New Relic through CloudWatch instead of the extension.
Best of both worlds: Using the extension with AWS CloudWatch as a fallback helps keep costs down while providing a fail-safe in case there is a problem with the extension.

We will show you to instrument your Lambda function using each method in our instrumentation docs:

Before you begin

Before enabling serverless monitoring using our Lambda layer, you'll need:

A New Relic account with either an admin role or have the Infrastructure manager add-on role.
A .
An AWS account with permissions for creating IAM resources, managed secrets, and Lambdas. You also need permissions for creating CloudFormation stacks and S3 buckets.

Then you'll need to complete the following:

Install AWS CLI v2 configure using aws configure. You will need an AWS Access Key ID and AWS Secret Access Key.
Install Python version 3.3 or higher.
Install the newrelic-lambda CLI. To install it, run this command:
bash
```
$pip3 install newrelic-lambda-cli
```
By default, we use the AWS Managed Policy ReadOnlyAccess. This allows the Infrastructure integration to see all the resources in your account, rather than just your Lambda functions and CloudWatch metrics. New Relic recommends this default, but we understand that some organizations prefer a strict security posture for third party integrations. If you'd like to limit New Relic's access, the IAM role can be granted with a minimum of these permissions:
```
Resource: "*"
  Action:
    - "cloudwatch:GetMetricStatistics"
    - "cloudwatch:ListMetrics"
    - "cloudwatch:GetMetricData"
    - "lambda:GetAccountSettings"
    - "lambda:ListFunctions"
    - "lambda:ListAliases"
    - "lambda:ListTags"
    - "lambda:ListEventSourceMappings"
```

Before running the CLI, you will need to grant New Relic a minimum of these permissions in AWS:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CLIAccessPolicy",
      "Action": [
        "cloudformation:CreateChangeSet",
        "cloudformation:CreateStack",
        "cloudformation:DescribeStacks",
        "cloudformation:ExecuteChangeSet",
        "iam:AttachRolePolicy",
        "iam:CreateRole",
        "iam:GetRole",
        "iam:PassRole",
        "lambda:AddPermission",
        "lambda:CreateFunction",
        "lambda:GetFunction",
        "logs:DeleteSubscriptionFilter",
        "logs:DescribeSubscriptionFilters",
        "logs:PutSubscriptionFilter",
        "s3:GetObject",
        "serverlessrepo:CreateCloudFormationChangeSet",
        "secretsmanager:CreateSecret"
      ],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Sid": "NRLogAccessPolicy",
      "Effect": "Allow",
      "Action": [
        "serverlessrepo:CreateCloudFormationTemplate",
        "serverlessrepo:GetCloudFormationTemplate"
      ],
      "Resource": "arn:aws:serverlessrepo:us-east-1:463657938898:applications/NewRelic-log-ingestion"
    }
  ]
}

If you'd like to learn more about our CLI, see our CLI repo.

Recommended AWS Lambda language runtimes

We recommend the following runtimes:

Node.js: nodejs16.x, nodejs18.x, nodejs20.x
Python: python3.8, python3.9, python3.10, python3.11, python3.12
Go: provided.al2, provided.al2023
Java: java8.al2, java11, java17, java21
Ruby: ruby3.2, ruby3.3
.NET: dotnet6, dotnet8

Consider costs

Enabling serverless monitoring for AWS Lambda may result in Amazon Web Services charges. Our newrelic-log-ingestion Lambda function, which reports your Lambda data to us, is considered a Third Party Service: AWS charges resulting from your use of it are your responsibility.

If you use our newrelic-lambda-extension bundled in our New Relic Lambda Layers, you can avoid sending logs and telemetry via CloudWatch and the newrelic-log-ingestion function. It's also possible to configure the newrelic-log-ingestion function as a CloudWatch fallback method for functions that require high availability of observability data.

Install Lambda monitoring

Lambda monitoring includes:

An AWS integration to ingest CloudWatch metrics and perform entity synthesis within New Relic.
A New Relic agent operating in "serverless" mode, included in our New Relic Lambda Layers or in your deployment.
A method of sending "instrumented" payloads and optionally logs to New Relic via our newrelic-log-ingestion function or Extension, also included in our New Relic Lambda Layers.

Link your AWS and New Relic accounts

Before you can receive telemetry for your "instrumented" Lambda function in New Relic, you will need to link your AWS account to New Relic. By linking accounts, you're granting New Relic permission to create an inventory of your AWS account and automatically gather CloudWatch metrics for your Lambda functions. After you've linked accounts, resources in your AWS account will appear as entities in the entity explorer. Lambda function entities will be found in the Serverless Functions view.

Set up an AWS Metric Streams integration (recommended)

For New Relic to monitor your Lambda functions, you'll need to set up an integration between New Relic and AWS. We recommend using Metric Streams to help reduce latency and avoid throttling of our polling requests.

Set up an API Polling integration with AWS

If you're not using a Metric Streams integration, you can choose to set up an API Polling integration instead. By default, an API Polling integration with the Lambda service enabled will poll every 5 minutes for CloudWatch metrics.

You can use the newrelic-lambda-cli to set up an API Polling integration, but not a Metric Streams integration.

bash

$newrelic-lambda integrations install --nr-account-id YOUR_NR_ACCOUNT_ID \
>--nr-api-key YOUR_NEW_RELIC_USER_KEY

One step in the integrations install command will create a NEW_RELIC_LICENSE_KEY secret in AWS Secrets Manager for improved security. This is configurable with a --disable-license-key-secret flag, along with other options and flags.

Instrument your Lambda function

There are several ways to go about adding instrumentation to your Lambda function. For "no code change" instrumentation, use a New Relic layer. You can add the layer added directly to your Lambda function or included in your dockerfile. Each layer, except the extension layer, includes an agent, handler wrapper, and extension for payload and log shipping.

The newrelic-lambda-cli is the quickest and easiest method to add a New Relic Lambda Layer. It's often used for testing rather than for CI/CD.

To add a New Relic layer via your existing CI/CD pipeline, we have a Serverless Framework plugin along with examples for SAM and Terraform.

For performance or to more tightly integrate New Relic with your function's code, you can alternatively use layerless instrumentation for all runtimes except Ruby.

Configure your Lambda function

After instrumenting your function, check in the AWS console to ensure the configuration is correct.

Sending of function logs is disabled by default for the extension. Here are some common environment variables:

NEW_RELIC_EXTENSION_SEND_FUNCTION_LOGS to send function logs. The default is false.
NEW_RELIC_LICENSE_KEY to override the secret in the AWS Secrets Manager. This is a good check in case the secret has been created with the wrong ingest license key.
NEW_RELIC_DATA_COLLECTION_TIMEOUT to reduce time spent sending logs and payloads to New Relic when a timeout occurs. The default is 10s.

Here are some additional environment variables to consider setting for the Node.js agent:

NEW_RELIC_USE_ESM if your function uses the ECMAScript standard instead of CommonJS. Make sure to instrument your handler manually to avoid performance impacts on cold starts.
NEW_RELIC_NATIVE_METRICS_ENABLED should be disabled to avoid performance impacts. Instrumentation of native metrics are not needed in Lambda. Defaults to true.

See the next step for more commonly used environment variables by runtime.

Environment variables

When you instrument New Relic's Lambda layer with the CLI, your environment variables with be configured automatically. You may want to change some of the default settings and configure your Lambda functions with the environment variables that match your monitoring needs. Choose your runtime to see the available environment variables and our recommendations for default settings.

Environment variable	Default value	Options	Description
`NEW_RELIC_ACCOUNT_ID`			Your New Relic account ID
`NEW_RELIC_TRUSTED_ACCOUNT_KEY`			Your New Relic account ID or parent ID, if it exists

Environment variable	Default value	Options	Description
`NEW_RELIC_ACCOUNT_ID`			Your New Relic account ID
`NEW_RELIC_TRUSTED_ACCOUNT_KEY`			Your New Relic account ID or parent ID, if it exists
`NEW_RELIC_LAMBDA_HANDLER`			Used by our handler wrapper to find your function's actual handler.
`NEW_RELIC_DISTRIBUTED_TRACING_ENABLED`	`false`	`true`, `false`	Enable or disable distributed tracing
`NEW_RELIC_PRIMARY_APPLICATION_ID`			Your New Relic account ID
`NEW_RELIC_DEBUG`	`false`	`true`, `false`	Agent log level

Environment variable	Default value	Options	Description
`NEW_RELIC_ACCOUNT_ID`			Your New Relic account ID
`NEW_RELIC_TRUSTED_ACCOUNT_KEY`			Your New Relic account ID or parent ID, if it exists
`NEW_RELIC_LAMBDA_HANDLER`			Used by our handler wrapper to find your function's actual handler. Not needed if using the manual wrapping method.
`NEW_RELIC_DISTRIBUTED_TRACING_ENABLED`	`false`	`true`, `false`	Enable or disable distributed tracing (Java excluded)
`NEW_RELIC_NO_CONFIG_FILE`	`false`	`true`, `false`	The Node.js agent uses environment variables in Lambda, so this should be set to `true`
`NEW_RELIC_APP_NAME`			Should be set but is not used in the New Relic UI. Instead, entity names come from the AWS integration.
`NEW_RELIC_LOG_ENABLED`	`false`	`true`,`false`	Enable or disable agent logging
`NEW_RELIC_LOG_LEVEL`	`info`	`fatal`, `error`, `warn`, `info`, `debug`, `trace`	Agent log level
`NEW_RELIC_LOG`	`stdout`	`stdout`	Agent log path should be stdout for the Node.js agent in serverless mode
`NEW_RELIC_USE_ESM`	`false`	`true`, `false`	ESM functions that use async/await and not callbacks
`NEW_RELIC_NATIVE_METRICS_ENABLED`	`true`	`true`, `false`	Should be set to false to reduce cold start duration. When set to `false` it does not collect VM metrics

You can find more environment variables in our Node.js configuration documentation.

Environment variable	Default value	Options	Description
`NEW_RELIC_ACCOUNT_ID`			Your New Relic account ID
`NEW_RELIC_TRUSTED_ACCOUNT_KEY`			Your New Relic account ID or parent ID, if it exists
`NEW_RELIC_DISTRIBUTED_TRACING_ENABLED`	`false`	`true`, `false`	Enable or disable distributed tracing (Java excluded)
`NEW_RELIC_LAMBDA_HANDLER`			Used by our handler wrapper to find your function's actual handler.
`NEW_RELIC_NO_CONFIG_FILE`	`false`	`true`, `false`	Agent uses environment variables in Lambda so this should be set to true
`NEW_RELIC_APP_NAME`			Should be set but is not used in the New Relic UI. Instead, entity names come from the AWS integration.
`NEW_RELIC_LOG`	`stderr`	`stderr`	Agent log path should be `stderr` for the Python agent in serverless mode
`NEW_RELIC_LOG_LEVEL`	`info`	`critical`, `error`, `warning`, `info`, `debug`	Agent log level
`NEW_RELIC_SERVERLESS_MODE_ENABLED`	`false`	`true`, `false`	This should be set to `true` so the agent will run in serverless mode
`NEW_RELIC_PACKAGE_REPORTING_ENABLED`	`true`	`true`, `false`	Python agent package reporting should be set to `false` to improve cold start times

You can find more environment variables in our Python configuration documentation.

Environment variable	Default value	Options	Description
`NEW_RELIC_ACCOUNT_ID`			Your New Relic account ID
`NEW_RELIC_TRUSTED_ACCOUNT_KEY`			Your New Relic account ID or parent ID, if it exists
`NEW_RELIC_DISTRIBUTED_TRACING_ENABLED`	`false`	`true`, `false`	Enable or disable distributed tracing
`NEW_RELIC_LAMBDA_HANDLER`			Used by our handler wrapper to find your function's actual handler.

Environment variable	Default value	Options	Description
`CORECLR_ENABLE_PROFILING`	`0`	`0`, `1`	Required: This must be set to `1` in order for the .NET agent to instrument your application.
`CORECLR_PROFILER`		`{36032161-FFC0-4B61-B559-F6C5D41BAE5A}`	Required: This must be set to `{36032161-FFC0-4B61-B559-F6C5D41BAE5A}` in order for the .NET agent to instrument your application.
`CORECLR_NEWRELIC_HOME`		`/opt/lib/newrelic-dotnet-agent`	Required: This must be set to `/opt/lib/newrelic-dotnet-agent` in order for the .NET agent to instrument your application via our layer.
`CORECLR_PROFILER_PATH`		`/opt/lib/newrelic-dotnet-agent/libNewRelicProfiler.so`	Required: This must be set to `/opt/lib/newrelic-dotnet-agent/libNewRelicProfiler.so` in order for the .NET agent to instrument your application via our layer.
`NEW_RELIC_LAMBDA_HANDLER`	the runtime handler		Required if the runtime handler is not in the `assembly::class::method` format, where `assembly` is the name of the DLL file that contains your code, `class` is the complete name (including namespace) of the class that contains your handler method, and `method` is the name of the function handler method. Note that this environment variable has the same name as one used by our handler wrappers for other agents, though the .NET agent does not need to utilize a handler wrapper.
`NEW_RELIC_APP_NAME`			Should be set but is not used in the New Relic UI. Instead, entity names come from the AWS integration.
`NEW_RELIC_LOG_ENABLED`	`1`	`1`, `0`	Enable or disable agent logging
`NEW_RELIC_LOG_CONSOLE`	`0`	`1`, `0`	Must be set to `1` to output logs to CloudWatch
`NEW_RELIC_LOG_LEVEL`	`info`	`info`, `debug`, `finest`	Agent log level

You can find more environment variables in our .NET configuration documentation and layerless instrumentation method.

Environment variable	Default value	Options	Description
`NEW_RELIC_LAMBDA_EXTENSION_ENABLED`	`true`	`true`, `false`	Enable or disable the extension
`NEW_RELIC_LICENSE_KEY`			Your New Relic ingest key. This overrides Secrets Manager.
`NEW_RELIC_LICENSE_KEY_SECRET`	`NEW_RELIC_LICENSE_KEY`		Custom secret name in AWS Secrets Manager
`NEW_RELIC_DATA_COLLECTION_TIMEOUT`	10s		Reduce timeout duration when for `Telemetry client error`
`NEW_RELIC_EXTENSION_LOGS_ENABLED`	`true`	`true`, `false`	Enable or disable `[NR_EXT]` log lines
`NEW_RELIC_EXTENSION_LOG_LEVEL`	`INFO`	`INFO`, `DEBUG`	log level for NR_EXT log lines
`NEW_RELIC_EXTENSION_SEND_FUNCTION_LOGS`	`false`	`true`, `false`	Send function logs
`NEW_RELIC_LOG_ENDPOINT`	`https://log-api.newrelic.com/log/v1`	`https://log-api.newrelic.com/log/v1`, `https://log-api.eu.newrelic.com/log/v1`	Logs endpoint
`NEW_RELIC_TELEMETRY_ENDPOINT`	`https://cloud-collector.newrelic.com/aws/lambda/v1`	`https://cloud-collector.newrelic.com/aws/lambda/v1`, `https://cloud-collector.eu01.nr-data.net/aws/lambda/v1`	Telemetry endpoint

See more environment variables for the New Relic extension in our documentation.

Try our example functions

After you've instrumented New Relic's Lambda layer, we strongly recommend trying out our example functions. These working examples should be used as a starting point for instrumenting your own serverless functions. Using these examples can help you become familiar with the the New Relic Lambda layer, testing the account link, and they can be used as a reference for your own instrumentation. Each example demonstrates adding permissions, runtime-specific techniques for wrapping your handler, managing function log retention in CloudWatch, and more.

While there are many ways to manage and deploy Lambda functions, AWS CloudFormation is the mechanism we use for our examples.

Our examples are published, alongside the New Relic Lambda Extension, in this GitHub repository. There's one for each Lambda runtime New Relic supports:

You can also learn how to instrument distributed tracing into a non-trivial serverless application in our distributed tracing example. It illustrates manual trace propagation for SQS and SNS, two of the more popular services that might invoke Lambda functions, with Node, Python, Ruby and Java functions.

Sugerencia

When manually testing, telemetry data can be delayed. We recommend waiting seven seconds before invoking the function again, allowing any buffered telemetry to be delivered.

Troubleshooting

If you run into issues instrumenting your Lambda function, here are some common troubleshooting tips:

The newrelic-lambda CLI should be run once per region, with the --aws-region parameter. Use the same linked account name, and the tool will detect that the account link has been created already. The secret needs to be created in each region.

Similarly, several AWS accounts can be linked to a New Relic account. Give each account a different linked account name. The --aws-profile argument to the CLI tool will select the named profile. The tool uses the same configuration as the AWS CLI.

You've instrumented your Lambda function but it is not showing as instrumented in the Amazon Web Services -> Lambda functions section of New Relic.

You've linked both an API polling and Metric Streams integration to your New Relic account manually in the UI.

In this scenario with two integrations for the same AWS account in one New Relic account, a race condition will occur when instrumented payloads are received from the instrumented Lambda function at the New Relic cloud-collector endpoint. The payload will be randomly assigned to one of the two integrations. If assigned to the integration not linked to your function entity, the payload will be dropped and the function will not show as instrumented. Only functions that have received at least one payload in the last 30 days to the AwsLambdaInvocation event will show as instrumented.

FROM AwsLambdaInvocation
SELECT count (*)
SINCE 30 days ago
WHERE entityGuid = 'ENTITY_GUID'
LIMIT 1

To prevent creating two integrations for the same AWS account, we recommend you use the newrelic-lambda CLI, as mentioned above, because it will detect an existing integration and use it.

If two integrations have already been created, choose one to keep and unlink the other by clicking Unlink this account at Infrastructure > AWS in New Relic.

Sugerencia

There are some limitations to Metric Streams integrations that should be considered before unlinking an API polling integration. There are also some limitations to Infrastructure Dimensional metrics queries which should be considered before fully committing to a Metric Streams integration.

Your lambda code requires the execution role which has permission to read AWS Secrets Manager. If you find a log like the following, add the appropriate permission to the policy of the execution role. In our examples, check out the template.yaml file to see an easy way to grant this permission.

bash

Failed to retrieve license key AccessDeniedException: User: <ARN> is not authorized to perform: secretsmanager:GetSecretValue on resource: <ARN>

What's next

Containerized Lambda functions

If you have containerized Lambda functions, click here to instrument.

Non-containerized Lambda functions

If you have non-containerized Lambda functions, click here to instrument.

Layerless Lambda functions

If you have layerless Lambda functions, click here to instrument.

How the Lambda instrumentation works

Choose your shipping method

Before you begin

Recommended AWS Lambda language runtimes

Consider costs

Install Lambda monitoring

Link your AWS and New Relic accounts

Set up an AWS Metric Streams integration (recommended)

Set up an API Polling integration with AWS

Instrument your Lambda function

Configure your Lambda function

Distributed Tracing

Extension

Node.js

Python

Environment variables

Go

Java OpenTracing

Node.js

Python

Ruby

.NET

Lambda extension

Try our example functions

Sugerencia

Troubleshooting

Cannot use AWS secrets manager

Multiple AWS regions and accounts

Lambda function not showing as instrumented

Sugerencia

Failure to retrieve license key `AccessDeniedException`

What's next

Containerized Lambda functions

Non-containerized Lambda functions

Layerless Lambda functions

Introduction to AWS Lambda monitoring

How the Lambda instrumentation works .css-21sua1{background:none;border:none;width:0;padding:0;}

Choose your shipping method

Before you begin

Recommended AWS Lambda language runtimes

Consider costs

Install Lambda monitoring

Link your AWS and New Relic accounts

Set up an AWS Metric Streams integration (recommended)

Set up an API Polling integration with AWS

Instrument your Lambda function

Configure your Lambda function

Extension

Node.js

Python

Environment variables

Go

Java OpenTracing

Node.js

Python

Ruby

.NET

Lambda extension

Try our example functions

Sugerencia

Troubleshooting

Cannot use AWS secrets manager

Multiple AWS regions and accounts

Lambda function not showing as instrumented

Failure to retrieve license key `AccessDeniedException`

What's next

Containerized Lambda functions

Non-containerized Lambda functions

Layerless Lambda functions

How the Lambda instrumentation works