Containerized private minion (CPM) maintenance and monitoring

After installing your containerized private minion (CPM), you can keep track of its maintenance and monitoring in several ways:

• Check if the CPM is healthy and working with the CPM status endpoint.
• See if a private location is under-provisioned and needs more minions.
• Review your Docker logs or Kubernetes logs.

Check CPM status using HTTP

Connecting to a running CPM using HTTP is the easiest way to check if it's healthy and working. The container exposes two ports: 8080 and 8180. You can check the CPM with the following endpoints:

• :8080/status/check: provides details about internal health checks that the minion performs. HTTP 200 means the status is healthy.
• :8080/status: provides details about a minion's status, which is the same data published in a SyntheticsPrivateMinion event.
• :8180/: provides JVM application admin endpoints. This is an advanced view of a minion's Java Development Kit (JDK) internal state.

Check if your private location requires more minions

If your private location has multiple monitor checks queued up and you experience delays, you may need more minions available to execute the monitor checks.

To learn how to verify this, see Does my private location need more minions?

Review logs

You can monitor your minion's health by looking at CPM container logs.

$
docker logs YOUR_CONTAINER_NAME
2018-10-10 11:33:29,856 - Minion ID: a21f6d7f-4f65-4dec-92fb-88cb975d2a19
2018-10-10 11:33:29,869 - Publishing resources for Private Minion API: /status/check, /build-info, /status
2018-10-10 11:33:40,527 - Minion is configured, checking if it is healthy...
2018-10-10 11:33:43,471 - Launching in PRIVATE Location: 123456-example_private_loc-480
2018-10-10 11:33:43,723 - Configured 2 heavy worker threads, and 50 light worker threads
2018-10-10 11:33:43,796 -
2018-10-10 11:33:43,796 - **************************************************************************
2018-10-10 11:33:43,796 - * Synthetics Minion is ready and servicing location 'example_private_location'
2018-10-10 11:33:43,796 - **************************************************************************
... logging continues ...

$
kubectl get pods -n YOUR_NAMESPACE

$
kubectl logs -n YOUR_NAMESPACE YOUR_CPM_NAME
2020-05-11 22:57:24,084 - Minion will use 2 heavy workers
2020-05-11 22:57:24,149 - Minion will use 50 lightweight workers
2020-05-11 22:57:27,973 - Minion Container System: KUBERNETES
2020-05-11 22:57:30,158 - Minion deployment mode: PRIVATE_MINION_POD_KUBERNETES
2020-05-11 22:57:30,178 - No volume mounted at '/var/lib/newrelic/synthetics' in ':rw' mode: Private Minion's ID will change with each boot
2020-05-11 22:57:30,284 - Minion ID: a21f6d7f-4f65-4dec-92fb-88cb975d2a19
2020-05-11 22:57:30,654 - Publishing resources for Private Minion API: /status/check, /build-info, /status
2020-05-11 22:57:31,595 - Minion is configured, checking if it is healthy...
2020-05-11 22:57:35,457 - Launching in PRIVATE Location: 123456-example_private_loc-480
2020-05-11 22:57:36,060 - Executor for async-worker-* threads configured with a max pool size of 16
2020-05-11 22:57:36,072 - Configured 2 heavy worker threads, and 50 lightweight worker threads
2020-05-11 22:57:36,087 -
2020-05-11 22:57:36,087 - **************************************************************************
2020-05-11 22:57:36,087 - * Synthetics Minion 3.0.1 is ready and servicing location 'example_private_location'
2020-05-11 22:57:36,087 - **************************************************************************
2020-05-11 22:57:36,087 -
... logging continues ...

Enable debug logs

If you experience issues with your CPM, you can enable debug logs to help troubleshoot issues.

The default level of logging is set to only inform the user of key information and actionable errors. If this is insufficient, you can enable a more verbose logging by using the MINION_LOG_LEVEL environment variable.

$
docker run ... -e MINION_LOG_LEVEL=DEBUG ...
$
docker logs -f YOUR_CONTAINER_NAME
... verbose logging continues ...

$
helm install YOUR_CPM_NAME YOUR_REPO_NAME/synthetics-minion -n YOUR_NAMESPACE --set synthetics.privateLocationKey=YOUR_PRIVATE_LOCATION_KEY --set synthetics.minionLogLevel=DEBUG

$
kubectl get pods -n YOUR_NAMESPACE

$
kubectl logs -f -n YOUR_NAMESPACE YOUR_CPM_POD_NAME
... verbose logging continues ...

Retrieve Kubernetes debugging information

If you experience issues with your CPM in a Kubernetes container orchestration system environment, you can retrieve information about the CPM pod and the node it is running on to help troubleshoot.

To retrieve information for the CPM pod:

$
kubectl describe pod -n YOUR_NAMESPACE YOUR_CPM_POD_NAME

To retrieve information for the node the CPM pod is running on, identify the node, and then:

$
kubectl describe node NODE_ASSOCIATED_WITH_YOUR_CPM_POD_NAME

Monitor CPMs with New Relic Infrastructure

New Relic's infrastructure monitoring supports advanced Docker monitoring and advanced Kubernetes monitoring. To add support for this, synthetic monitoring labels the containers spawned by CPM with a series of informative labels, all prefixed with synthetics-minion-. The CPM spawns containers called "runners" which process non-ping monitors like: simple browser, scripted browser, api test, and step function. You can use these labels to identify these runner containers. Example labels include:

• synthetics-minion-runner-role
• synthetics-minion-runner-version
• synthetics-minion-container-id
• synthetics-minion-id
• synthetics-minion-build-number
• synthetics-minion-job
• synthetics-minion-account
• synthetics-minion-monitor
• synthetics-minion-monitor-version
• synthetics-minion-monitor-type
• synthetics-minion-monitor-type-label

Runner containers last a short time. One runner container is created to process one non-ping monitor job. The runner is created, processes the job, and is quickly deleted. A runner container exists for only a few seconds and will be created only if there is a non-ping monitor job to process. Ping monitors will not trigger runner container creation, so the above labels will not be present.

If you are using the infrastructure agent to monitor these runner containers, configure at least one monitor to run each minute. The infrastructure agent will have more opportunity to notice and collect the above labels from the docker inspect of the container before it is deleted.

Note: the synthetics-minion-id label refers to the ID of the minion which spawned this particular runner container. The ID of the runner itself is not tracked.