Troubleshoot Docker container monitoring

If you're having issues with your Docker monitoring setup, check these common problems and solutions.

팁

General NRDOT troubleshooting: For collector startup issues, configuration validation, network connectivity, and NrIntegrationError diagnostics, see the NRDOT Troubleshooting Guide. The issues below are specific to Docker container monitoring.

Issue: Docker containers are running but no Docker metrics appear in New Relic after 10+ minutes.

Diagnostics:

bash

$# Check Docker daemon accessibility
$docker info
$
$# Test Docker socket permissions
$ls -l /var/run/docker.sock
$
$# Verify containers are running
$docker ps
$
$# Check collector logs for Docker-specific errors
$journalctl -u nrdot-collector | grep -i "docker"

Solutions:

Docker socket access: Add nrdot-collector user to docker group:

bash

$sudo usermod -aG docker nrdot-collector
$sudo systemctl restart nrdot-collector

Docker API version: Ensure Docker API version is 1.25+:
bash
```
$docker version --format '{{.Client.APIVersion}}'
```

Issue: Collector logs show "permission denied" when accessing Docker socket.

Diagnostics:

bash

$# Check socket ownership and permissions
$ls -la /var/run/docker.sock
$
$# Check collector user membership
$groups nrdot-collector
$
$# Check collector logs for permission errors
$journalctl -u nrdot-collector | grep -i "permission denied"

Solutions:

bash

$# Option 1: Add collector user to docker group (recommended)
$sudo usermod -aG docker nrdot-collector
$sudo systemctl restart nrdot-collector
$
$# Option 2: Adjust socket permissions (less secure)
$sudo chmod 666 /var/run/docker.sock
$
$# Option 3: Run collector as root (not recommended for production)
$# Modify systemd service file User= setting

Issue: Some containers show metrics while others don't, or metrics are sporadic.

Diagnostics:

bash

$# Check for containers with networking issues
$docker inspect $(docker ps -q) | grep -A5 "NetworkSettings"
$
$# Review collector configuration
$cat /etc/nrdot-collector/docker-stats-config.yaml | grep -A10 "docker_stats:"
$
$# Check for resource constraints
$docker stats --no-stream
$
$# Check collector logs for specific errors
$journalctl -u nrdot-collector | grep -i "docker_stats"

Solutions:

Container visibility: Ensure all containers are accessible via the Docker socket path configured in the collector
Resource limits: Increase collector timeout values if containers are slow to respond. Edit your /etc/nrdot-collector/docker-stats-config.yaml:
```
receivers:
  docker_stats:
    timeout: 10s  # Increase from default 5s
    collection_interval: 30s  # Reduce frequency if needed
```
Network namespaces: Some containers in restricted network namespaces may not be accessible

Issue: Configured log collection but container logs don't appear in New Relic.

Diagnostics:

bash

$# Check log directory permissions
$ls -la /var/lib/docker/containers/
$
$# Verify receiver creator is discovering containers
$journalctl -u nrdot-collector | grep -i "receiver.*creator\|docker.*observer"
$
$# Check if containers have exposed ports (required for discovery)
$docker ps --format 'table {{.Names}}\t{{.Ports}}'
$
$# Verify logs pipeline is configured
$journalctl -u nrdot-collector | grep -i "logs"

Solutions:

File permissions: Grant log directory access:

bash

$sudo chmod -R 755 /var/lib/docker/containers/
$# Or add collector to docker group
$sudo usermod -aG docker nrdot-collector
$sudo systemctl restart nrdot-collector

Container port exposure: Expose ports for container discovery:

bash

$# When running containers, expose at least one port
$docker run -p 8080:8080 my-app:latest
$# Or use --expose flag
$docker run --expose 80 my-app:latest

Verify logs config: Ensure your log collection configuration is included:

bash

$# Check systemd override
$sudo systemctl cat nrdot-collector | grep -A2 "ExecStart="
$# Should point to: --config=/etc/nrdot-collector/docker-stats-config.yaml (or combined config with logs)

Query for logs in New Relic:

FROM Log SELECT * WHERE newrelic.source='api.logs.otlp' SINCE 1 hour ago

Issue: OpenTelemetry Collector fails to start or crashes immediately.

Diagnostics:

bash

$# Check collector status
$sudo systemctl status otelcol-contrib
$
$# View recent logs
$journalctl -u otelcol-contrib --since "1 hour ago" -f
$
$# Test configuration syntax
$otelcol-contrib --config-validate --config=/etc/otelcol-contrib/config.yaml

Solutions:

Configuration errors: Fix YAML syntax errors and ensure all required fields are present
Missing permissions: Ensure collector user has read access to configuration file
Environment variables: Verify NEWRELIC_LICENSE_KEY and NEWRELIC_OTLP_ENDPOINT are set correctly

Issue: Docker containers are running but no metrics appear in New Relic after 10+ minutes.

Diagnostics:

bash

$# Verify collector is running and processing data
$journalctl -u otelcol-contrib | grep -i "docker\|error\|export"
$
$# Check Docker daemon accessibility
$docker info
$
$# Test Docker socket permissions
$ls -l /var/run/docker.sock
$
$# Verify containers are running
$docker ps

Solutions:

Docker socket access: Add collector user to docker group:

bash

$sudo usermod -aG docker otelcol-contrib
$sudo systemctl restart otelcol-contrib

License key issues: Verify your license key is correct and has proper permissions:

bash

$echo $NEWRELIC_LICENSE_KEY
$# Should be 40-character NRAK-prefixed key for US, or EU equivalent

Network connectivity: Test OTLP endpoint accessibility:

bash

$# For US region
$curl -v -H "api-key: $NEWRELIC_LICENSE_KEY" https://otlp.nr-data.net/v1/metrics
$
$# For EU region
$curl -v -H "api-key: $NEWRELIC_LICENSE_KEY" https://otlp.eu01.nr-data.net/v1/metrics

Docker API version: Ensure Docker API version is 1.25+:
bash
```
$docker version --format '{{.Client.APIVersion}}'
```

Issue: Collector logs show "permission denied" when accessing Docker socket.

Diagnostics:

bash

$# Check socket ownership and permissions
$ls -la /var/run/docker.sock
$
$# Check collector user membership
$groups otelcol-contrib

Solutions:

bash

$# Option 1: Add collector user to docker group (recommended)
$sudo usermod -aG docker otelcol-contrib
$sudo systemctl restart otelcol-contrib
$
$# Option 2: Adjust socket permissions (less secure)
$sudo chmod 666 /var/run/docker.sock
$
$# Option 3: Run collector as root (not recommended for production)
$# Modify systemd service file User= setting

Issue: Some containers show metrics while others don't, or metrics are sporadic.

Diagnostics:

bash

$# Check for containers with networking issues
$docker inspect $(docker ps -q) | grep -A5 "NetworkSettings"
$
$# Review collector configuration
$grep -A20 "docker_stats:" /etc/otelcol-contrib/config.yaml
$
$# Check for resource constraints
$docker stats --no-stream

Solutions:

Container visibility: Ensure all containers are accessible via the Docker socket path configured in the collector

Resource limits: Increase collector timeout values if containers are slow to respond:

docker_stats:
  timeout: 10s  # Increase from default 5s
  collection_interval: 30s  # Reduce frequency if needed

Network namespaces: Some containers in restricted network namespaces may not be accessible

Issue: OpenTelemetry Collector consuming excessive system resources.

Diagnostics:

bash

$# Monitor collector resource usage
$top -p $(pgrep otelcol-contrib)
$
$# Check metric collection frequency
$grep -A10 "docker_stats:" /etc/otelcol-contrib/config.yaml
$
$# Count active containers
$docker ps --format 'table {{.Names}}\t{{.Status}}' | wc -l

Solutions:

Adjust collection interval: Reduce monitoring frequency for high-container-count environments:
```
docker_stats:
  collection_interval: 30s  # Increase from 15s
```

Disable expensive metrics: Turn off per-CPU metrics if not needed:

docker_stats:
  metrics:
    container.cpu.usage.percpu:
      enabled: false

Configure batch processing: Optimize batching for your environment:
```
batch:
  timeout: 60s
  send_batch_size: 1024
```

Issue: Configured log collection but container logs don't appear in New Relic.

Diagnostics:

bash

$# Check log directory permissions
$ls -la /var/lib/docker/containers/
$
$# Verify receiver creator is discovering containers
$journalctl -u otelcol-contrib | grep -i "receiver.*creator\|docker.*observer"
$
$# Check if containers have exposed ports (required for discovery)
$docker ps --format 'table {{.Names}}\t{{.Ports}}'

Solutions:

File permissions: Grant log directory access:

bash

$sudo chmod -R 755 /var/lib/docker/containers/
$# Or add collector to docker group
$sudo usermod -aG docker otelcol-contrib

Container port exposure: Expose ports for container discovery:

bash

$# When running containers, expose at least one port
$docker run -p 8080:8080 my-app:latest
$# Or use --expose flag
$docker run --expose 80 my-app:latest

Observer configuration: Verify Docker observer settings:

docker_observer:
  endpoint: unix:///var/run/docker.sock
  use_hostname_if_present: true
  use_container_labels: true

Issue: Metrics appear in wrong New Relic account or organization.

Diagnostics:

bash

$# Verify license key format and region
$echo $NEWRELIC_LICENSE_KEY | cut -c1-4
$# Should show "NRAK" for US accounts
$
$# Check configured endpoint
$echo $NEWRELIC_OTLP_ENDPOINT

Solutions:

US accounts: Use https://otlp.nr-data.net
EU accounts: Use https://otlp.eu01.nr-data.net
Verify license key: Ensure you're using the correct account's license key

For persistent issues, check the OpenTelemetry Collector troubleshooting guide and New Relic OpenTelemetry documentation.

Next steps

If you've resolved your issues:

Get back to setup: Return to the Docker monitoring setup guide to complete your configuration
Explore your data: Learn how to query and visualize your Docker metrics with the metrics reference guide
Set up alerts: Create proactive monitoring with alert conditions for critical container metrics

Troubleshoot Docker container monitoring

팁

No Docker metrics appearing in New Relic

Permission denied errors

Incomplete or missing container metrics

Log collection not working

Collector not starting or crashing

No metrics appearing in New Relic

Permission denied errors

Incomplete or missing container metrics

High memory or CPU usage by collector

Log collection not working

Data not appearing in correct New Relic account

Next steps

Troubleshoot Docker container monitoring

팁

Permission denied errors

Incomplete or missing container metrics

Log collection not working

Collector not starting or crashing

No metrics appearing in New Relic

Permission denied errors

Incomplete or missing container metrics

High memory or CPU usage by collector

Log collection not working

Data not appearing in correct New Relic account

Next steps .css-21sua1{background:none;border:none;width:0;padding:0;}

Next steps