Monitor and troubleshoot ATP

• To view recent process metrics:

  FROM Metric SELECT *
  WHERE metricName LIKE 'process.%'
  LIMIT 100

  コピー

• To count unique process metric types:

  FROM Metric SELECT uniqueCount(metricName)
  WHERE metricName LIKE 'process.%'

  コピー

• To view recent process metrics for your cluster:

  FROM Metric SELECT *
  WHERE k8s.cluster.name='your-cluster-name'
  AND metricName LIKE 'process.%'
  LIMIT 100

  コピー

• To count unique process metric types for your cluster:

  FROM Metric SELECT uniqueCount(metricName)
  WHERE k8s.cluster.name='your-cluster-name'
  AND metricName LIKE 'process.%'

  コピー

Problem: All process metrics are being sent, not filtered.

Problem: Collector memory increases over time

Solution: Adjust retention settings:

processors:
  adaptivetelemetry:
    retention_minutes: 15  # Reduce from default 30

Problem: Expected processes are missing from telemetry

Solution: Must use full path to add processes to include list. Entries without path separators won't match any process. For example:

processors:
  adaptivetelemetry:
    include_process_list:
      - "/usr/bin/myapp"  # Always captured regardless of thresholds

Problem: Earlier, ATP data was shown but now it is not. This could be due to hitting cardinality limits. ATP filters out many processes to reduce telemetry volume, but if you have a very high number of unique processes, you could still hit limits. You can monitor your cardinality usage to track this.

Solution:

To troubleshoot, check whether you've hit cardinality limits by going to the cardinality management page in New Relic. If you've breached limits, you can increase your cardinality limit.

processors:
  adaptivetelemetry:
    metric_thresholds:
      process.cpu.utilization: 0.10  # Increase from 0.05 to 0.10 (10% CPU)
      process.memory.utilization: 0.10  # Filter out more processes

Problem: ATP logs show storage-related errors or warnings about being unable to create storage directories.

Solution:

ATP automatically selects and creates storage directories based on your installation method. If storage fails, verify the following:

1. For systemd installations: Ensure the service is running as the nrdot-collector user and the systemd service file includes the StateDirectory directive:

   bash

   コピー

   $
   sudo systemctl cat nrdot-collector.service | grep StateDirectory

   Expected output: StateDirectory=nrdot-collector

2. For manual binary installations: Verify your user has write permissions to ~/.local/state/:

   bash

   コピー

   $
   mkdir -p ~/.local/state/nrdot-collector
   $
   touch ~/.local/state/nrdot-collector/test && rm ~/.local/state/nrdot-collector/test

3. Check collector logs for storage-related messages:

   bash

   コピー

   $
   sudo journalctl -u nrdot-collector | grep -i storage

ATP automatically manages storage paths based on your installation method:

For systemd service installations (DEB/RPM packages):

• Storage is automatically configured at /var/lib/nrdot-collector/
• The systemd service uses StateDirectory to create and manage the directory
• No manual configuration required

For manual binary installations:

• Storage uses ~/.local/state/nrdot-collector/ for regular users
• No root privileges required for storage
• Directory is automatically created if writable

To enable disk.io data collection by the OpenTelemetry hostmetrics receiver, you need escalated privileges because the io data stored at /proc/[PID]/io requires root user access. This is not an ATP requirement, ATP only processes data that's already been collected by OpenTelemetry.

Add the following security context to your Helm chart:

securityContext:
  allowPrivilegeEscalation: true
  privileged: true
  readOnlyRootFilesystem: false
  runAsNonRoot: false
  runAsUser: 0