Currently we don't flush OSR nmethods when CodeCache is full. It could prevent freeing codecache to get space for new compilations.

Martin Traverso wrote:

We're seeing some strange behavior with the JIT compiler in jdk7. As far as we can tell, at some point in the lifetime of the process the vm decides to stop generating native code.

I wrote a simple program that reproduces the behavior. It generates classes dynamically, with a method that loops and performs some computation. It does this over and over and it calls the method each time to make the vm optimize and generate native code. Separately, it gathers and prints the following stats: cumulative compilation time, code cache size, perm gen size and invocations per second.

You can take a look at the code here: http://github.com/martint/jittest. Here's a chart that shows how those stats behave over time: https://raw.github.com/martint/jittest/master/stats-7u40.png.

As you can see in the chart, the code cache grows steadily in the beginning. So does the cumulative compilation time. At some point, the cache fills up and the vm stops compiling (my interpretation) and invocation throughput drops significantly. Nothing much happens for a while after that, until the perm gen fills up and goes through a GC cycle. Some time after (not sure if related to the perm gen GC), the code cache flushes and the vm starts compiling again. Eventually, the code cache fills up again, but at a lower level that the first time. The vm stops compiling once again and performance drops to an even lower level that before. I haven't seen it recover from this condition, which is consistent with what we see in our production system.

I've been able to reproduce it in both 7u10 and the 7u40 early access that's available in the openjdk site. Interestingly, java 8 does not exhibit this issue.

Is this a known problem? Is there any way around it (other than restarting the vm) or things to try out while we wait for java 8 to come out?

Thanks!

Martin