The New Relic Java agent includes a circuit breaker that protects applications from the effects of over-instrumentation. When the circuit breaker detects early symptoms of memory exhaustion, it automatically "trips" and limits instrumentation. The agent stops collecting transaction data until the circuit breaker automatically resets after deciding that resetting is safe.
The circuit breaker takes two parameters into account (heap usage and time spent in garbage collection) to determine when it should trip. The default values for these thresholds are percentages:
Memory threshold: 20%
Garbage collection CPU threshold: 10%
When the percentage of free heap memory is less than memory_threshold, and the CPU time spent doing garbage collection is greater than gc_cpu_threshold, the circuit breaker trips. When the circuit breaker trips, the agent stops collecting transaction data. Throughput reported in the UI will be underreported, and you will not see transaction traces for a period of time.
Reasons for memory exhaustion
The circuit breaker trips when it detects signs of memory exhaustion. This can happen for several reasons:
Your application shows early signs of memory exhaustion due to recently deployed custom instrumentation (using XML, API calls, trace annotations, or the Java agent's custom instrumentation editor) or due to built-in instrumentation.
Your application experienced a load spike and showed signs of memory exhaustion. In this case, the agent isn’t contributing to the spike, but the circuit breaker can help conserve resources and ensure that the agent doesn’t contribute to OutOfMemoryErrors.
Your application is tuned to run close to its memory limit.
Troubleshooting
If the circuit breaker trips, try these troubleshooting tips.
Use the Top methods by call count table on the circuit breaker Events page to find methods that might be over-instrumented. Identify and disable custom instrumentation.
In general, agent memory usage is proportional to the call count of a method. Custom instrumentation should be used on methods that are called no more than ten or so times per transaction. If the instrumentation is built into the agent, review New Relic's custom instrumentation documentation for Java. If you need additional help, get support at support.newrelic.com.
Carefully review your application's memory usage history and determine whether increasing the maximum Java heap size is necessary.
If your application is behaving as expected, you may want to disable the circuit breaker. To disable the circuit breaker, add enabled: false under the circuitbreaker section in your newrelic.yml configuration file:
common:&default_settings
circuitbreaker:
enabled:false
To detect early signs of memory exhaustion, the circuit breaker uses a formula with two variables: memory_threshold and gc_cpu_threshold. When the percentage of free heap memory is less than memory_threshold, and the CPU time spent doing garbage collection is greater than gc_cpu_threshold, the circuit breaker trips. Adjust these values as needed, based on your application's operating performance and behavior.
For configuration details, see memory_threshold and gc_cpu_threshold.