Amazon SageMaker integration

By integrating Amazon SageMaker's integration with New Relic, you'll be able to instrument, analyze, troubleshoot, and optimize your machine-learning performance across your entire system. Rigorously observe your capabilities to react quickly to changes in the model's input or output and the relationship between the two.

Take the next steps to monitor your Amazon SageMaker metrics and objects (that are sent to AWS CloudWatch) and view them as entities and dashboards in New Relic.

For a look at our SageMaker integration in action, watch this short YouTube video (2:57 minutes).

Stream AWS CloudWatch Metrics to New Relic

Start benefiting from New Relic model performance monitoring entities in a single simple step (and just a few minutes!):

FROM Metric SELECT * WHERE aws.Namespace = '/aws/sagemaker/Endpoints' LIMIT MAX SINCE 1 WEEK AGO

Manual option

Follow our docs to set up CloudWatch Metric Streams.

Automated option

You may automate the setup with the Terraform code:

module "example_usage" {
  source = "modules/nr-cloudwatch-metric-stream"

  name_suffix    = "suffix" # optional
  aws_account_id = "your-aws-account-id"

  newrelic_collector_endpoint = "newrelic-endpoint-url"
  newrelic_trusted_account_id = "12345678"
  newrelic_license_key        = "YOUR_INGEST_LICENSE_KEY"
}

When calling the module, please write the correct newrelic_collector_endpoint:

• HTTP endpoint URL - US datacenter: https://aws-api.newrelic.com/cloudwatch-metrics/v1
• HTTP endpoint URL - EU datacenter: https://aws-api.eu01.nr-data.net/cloudwatch-metrics/v1

When you set the metric stream you can choose to stream the metric from all the namespaces, or you can specify namespaces.

Monitor your data and model in Amazon SageMaker, and send the metrics to CloudWatch

SageMaker automatically monitors your endpoints’ performance, and sends statistic metrics to CloudWatch. For more information, see Endpoint CloudWatch Metrics.

To obtain more benefits from the Amazon SageMaker integration, use the Amazon SageMaker Model Monitor tools. You'll have to define scheduled monitoring jobs to monitor the quality of your machine learning models in production and send metrics to CloudWatch.

The Amazon SageMaker Model Monitor provides the following types of monitoring:

• Monitor Data Quality: Monitor drift in data quality.
  • Example notebook: Amazon SageMaker Model Monitor
  • Namespace: aws/sagemaker/Endpoints/data-metrics
• Monitor Model Quality: Monitor drift in model quality metrics, such as accuracy.
  • Example notebook: Amazon SageMaker Model Quality Monitor
  • Namespace: aws/sagemaker/Endpoints/model-metrics
• Monitor Bias Drift for Models in Production: Monitor bias in you model's predictions.
  • Example notebook: Monitoring bias drift and feature attribution drift Amazon SageMaker Clarify
  • Namespace: aws/sagemaker/Endpoints/bias-metrics
• Monitor Feature Attribution Drift for Models in Production: Monitor drift in feature attribution.
  • Example notebook: Monitoring bias drift and feature attribution drift Amazon SageMaker Clarify
  • Namespace: aws/sagemaker/Endpoints/explainability-metrics

Advanced options

You can aso publish metric data points to Amazon CloudWatch and define the namespaces and one of the above using the put_metric_data function.

If you use your own algorithm for hyperparameter tuning, make sure that it sends at least one metric by writing evaluation data to stderr or stdout. Read more on how to define metrics in automatic model tuning. See also the example notebook Develop, Train, Optimize and Deploy Scikit-Learn Random Forest.

Explore your entities and dashboards

We generate aws-entities (under the MLOps entity domain) for the detailed namespaces. For these entities, you can get out-of-the-box dashboards and views. You can also create your own dashboard to view metrics that aren't displayed as part of the entities' views.

Go to one.newrelic.com > All capabilities > Model performance to view:

• The dashboard for the metrics of the endpoint from one of the Amazon SageMaker entities

• The dashboard for the model data entity