The container_memory_usage_bytes metric isn't an accurate indicator for out of memory (OOM) prevention as it includes cached data (i.e., filesystem) that can evict in memory pressure scenarios. However, the container_memory_working_set_bytes metric excludes cached data and is what Kubernetes uses for OOM/scheduling decisions, making it a better metric for monitoring/alerting memory saturation.
This change will be reflected within New Relic One as follows:
- Predefined alert conditions. The “Container memory usage % is too high” alert condition now uses the new
memoryWorkingSetUtilizationattribute instead of (memoryUsedBytes/memoryLimitBytes)/100.- Please note that existing alert conditions won’t be automatically updated. If you are using our predefined alert policy, we strongly recommend you to update it accordingly.
- The following widgets are now using working set bytes in Kubernetes dashboard:
- Resources Used
- Container Memory Usage - % Used vs Limit
- Container MBytes of Memory Used
- Node Resource Consumption
- Golden metrics definition for containers will now use
memoryWorkingSetBytes - The Kubernetes cluster explorer now shows the container's memory consumption compared to the limit, using the
WorkingSetBytesmetric in the Pod preview - New alerts can now be created based on:
- Memory Working Set Bytes
- Memory Working Set Utilization
- Requested Memory Working Set Utilization
These attributes can be used to define thresholds in the integrations section of the infrastructure alerts when selecting the K8sContainerSample data source, and they can also be used for nrql alerts.