FULL PRODUCT VERSION :
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
[aivanov@lva1-app9961 ~]$ uname -a
Linux hostname 2.6.32-431.11.2.el6.x86_64 #1 SMP Mon Mar 3 13:32:45 EST 2014 x86_64 x86_64 x86_64 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
We are using JDK7u51. Recently our image resizers / watermarkers JVMs started hanging up.
Issue appears couple of times per day manifesting itself in JVM hogging one CPU but not doing any real work.

Here is sample trace:
Thread 6956: (state = IN_NATIVE)
 - sun.java2d.cmm.kcms.CMM.cmmColorConvert(long, sun.java2d.cmm.kcms.CMMImageLayout, sun.java2d.cmm.kcms.CMMImageLayout) @bci=0 (Compiled frame; information may be imprecise)
 - sun.java2d.cmm.kcms.ICC_Transform.colorConvert(java.awt.image.Raster, java.awt.image.WritableRaster) @bci=34, line=644 (Compiled frame)
 - java.awt.image.ColorConvertOp.filter(java.awt.image.Raster, java.awt.image.WritableRaster) @bci=381, line=571 (Compiled frame)
 - com.sun.imageio.plugins.jpeg.JPEGImageReader.acceptPixels(int, boolean) @bci=19, line=1263 (Compiled frame)
 - com.sun.imageio.plugins.jpeg.JPEGImageReader.readImage(long, byte[], int, int[], int[], int, int, int, int, int, int, javax.imageio.plugins.jpeg.JPEGQTable[], javax.imageio.plugins.jpeg.JPEGHuffmanTable[], javax.imageio.plugins.jpeg.JPEGHuffmanTable[], int, int, boolean) @bci=0 (Compiled frame)
 - com.sun.imageio.plugins.jpeg.JPEGImageReader.readInternal(int, javax.imageio.ImageReadParam, boolean) @bci=1107, line=1231 (Compiled frame)
 - com.sun.imageio.plugins.jpeg.JPEGImageReader.read(int, javax.imageio.ImageReadParam) @bci=15, line=1034 (Compiled frame)

We always see same {{libcmm}} call in {{IN_NATIVE}} state. I think it's blocking all other threads from being executed.

We have `perf` output for that TID. It seems like {{Java_sun_java2d_cmm_kcms_CMM_cmmColorConvert}} is spinning in {{KpLeaveCriticalSection}}.

If I disassemble that function - it seems like it's badly broken:
{code}
       â”‚ 00000000000484e0 <KpLeaveCriticalSection>:
       â”‚ push %rbp
       â”‚ mov %rsp,%rbp
       â”‚ push %rbx
       â”‚ mov %rdi,%rbx
       â”‚ sub $0x8,%rsp
       â”‚ callq KpGetCurrentThreadId
       â”‚ cmp 0x4(%rbx),%eax
       â”‚ je 18
100.00 │16:└─ jmp 16
       â”‚18: mov 0x8(%rbx),%eax
       â”‚ sub $0x1,%eax
       â”‚ test %eax,%eax
       â”‚ mov %eax,0x8(%rbx)
       â”‚ je 2c
       â”‚25: add $0x8,%rsp
       â”‚ pop %rbx
       â”‚ leaveq
       â”‚ retq
       â”‚2c: lea 0x10(%rbx),%rdi
       â”‚ movl $0x0,0x4(%rbx)
       â”‚ callq pthread_mutex_unlock@plt
       â”‚ test %eax,%eax
       â”‚ je 25
       â”‚40: jmp 40
{code}
note that it's spinning on {{'16:└─ jmp 16'}}

Java's source code [1] suggests that it's intentional:
{code}
void KpLeaveCriticalSection (KpCriticalFlag_t FAR *CriticalFlag)
{
KpInt32_t myThreadId;

myThreadId = KpGetCurrentThreadId ();

/* If this thread doesn't own the
critical section something is wrong */
if (myThreadId != CriticalFlag->ThreadId) {
for (;;);
}

/* Decrement the lock count and if the
new lock count is zero then really
unlock the critical section */
CriticalFlag->Count--;
if (0 == CriticalFlag->Count) {

CriticalFlag->ThreadId = 0;
#if defined (KP_POSIX_THREADS)
if (pthread_mutex_unlock (&CriticalFlag->CriticalFlag) != 0) {
#else
if (mutex_unlock (&CriticalFlag->CriticalFlag) != 0) {
#endif
for (;;); /* something is wrong */
}
}

}
{code}
Note 'for (;;);’ case.


REPRODUCIBILITY :
This bug can be reproduced occasionally.