awt_image_ImagingLib_transformBI ends up calling Runtime.gc although thread is s

SUN AWT Native code enters awt_image_ImagingLib_transformBI but ends up calling java/lang/Runtime.gc although thread is still in critical via GetPrimitiveArrayCritical - this breaks the JNI spec.

For JRockit, this causes a deadlock since there will be contention for the GC lock. GC thread is waiting for image thread to exit critical, but image thread calls java/lang/Runtime.gc() even though its still in critical.

Seems to involve the awt_NewHandler::handler(size_t) which is described as:


// Called when malloc or operator new runs out of memory. We try to
// compact the heap by initiating a Java GC. If the amount of free
// memory available after this operation increases, then we return
// (1) to indicate that malloc or operator new should retry the
// allocation. Returning (0) indicates that the allocation should fail.

I cannot reproduce this issue, I only have a .mdmp file from a customer. I can get this stack trace:

5d17f4d8 58b116a9 java!java/lang/Runtime.gc()V (Runtime.java) (+0x6f bytes)
5d17f508 005bf556 java!jrockit/vm/RNI.c2java(IIIII)V (RNI.java) (+0x49 bytes)
5d17f530 5b545378 jvm!stubsCallC2JavaStub+0x26 [d:\tendril4\work\install\src_pb_jrockitq_5462931\s\jvm\src\jvm\code\gen\lir\stubs_x86.c @ 66]
5d17f630 004c76a7 jvm!call_java+0xcd [d:\tendril4\work\install\src_pb_jrockitq_5462931\s\jvm\src\jvm\native\jni\jnimethod.c @ 234]
5d17f658 004ad173 jvm!jniCallIntMethodV+0x37 [d:\tendril4\work\install\src_pb_jrockitq_5462931\s\jvm\src\jvm\native\jni\jnimethod.c @ 498]
5d17f698 6c9808d5 jvm!xjniCallIntMethodV+0x43 [d:\tendril4\work\install\src_pb_jrockitq_5462931\s\jvm\src\jvm\native\jni\jni.c @ 199]
5d17f700 77bbd0a2 awt!Java_sun_awt_windows_WCanvasPeer_resetTargetGC+0xcb
5d17f718 6c9611ea msvcrt!malloc+0x82
5d17f738 6c94936d awt!Java_sun_awt_image_ImagingLib_init+0x182e5
5d17f764 6c9480a0 awt!Java_sun_awt_image_ImagingLib_init+0x468
5d17f7f8 6d2ba599 awt!Java_sun_awt_image_ImagingLib_transformBI+0x1e4
5d17f800 6d2ba376 java!sun/awt/image/ImagingLib.transformBI(Ljava/awt/image/BufferedImage;Ljava/awt/image/BufferedImage;[DI)I (ImagingLib.java) (+0xe9 bytes)
5d17f858 6d2b9f89 java!sun/awt/image/ImagingLib.filter(Ljava/awt/image/BufferedImageOp;Ljava/awt/image/BufferedImage;Ljava/awt/image/BufferedImage;)Ljava/awt/image/BufferedImage; (ImagingLib.java:202) (+0x206 bytes)
5d17f888 6d2b6ced java!java/awt/image/AffineTransformOp.filter(Ljava/awt/image/BufferedImage;Ljava/awt/image/BufferedImage;)Ljava/awt/image/BufferedImage; (AffineTransformOp.java:262) (+0x3b9 bytes)
5d17f8e8 6d2b5433 java!sun/java2d/pipe/DrawImage.transformImage(Lsun/java2d/SunGraphics2D;Ljava/awt/Image;IILjava/awt/image/BufferedImageOp;Ljava/awt/geom/AffineTransform;Ljava/awt/Color;)V (DrawImage.java:304) (+0xa8d bytes)
5d17f9a0 6d2b4fff java!sun/java2d/pipe/DrawImage.scaleImage(Lsun/java2d/SunGraphics2D;Ljava/awt/Image;IIIILjava/awt/Color;)Z (DrawImage.java:109) (+0x1e3 bytes)
5d17fa08 59a8d093 java!sun/java2d/pipe/DrawImage.scaleImage(Lsun/java2d/SunGraphics2D;Ljava/awt/Image;IIIILjava/awt/Color;Ljava/awt/image/ImageObserver;)Z (DrawImage.java:774) (+0x5f bytes)
5d17fa48 6d2b51f5 java!sun/java2d/SunGraphics2D.drawImage(Ljava/awt/Image;IIIILjava/awt/Color;Ljava/awt/image/ImageObserver;)Z (SunGraphics2D.java:2716) (+0x103 bytes)
5d17fa90 6d2b508e java!sun/awt/image/ImageRepresentation.drawToBufImage(Ljava/awt/Graphics;Lsun/awt/image/Image;IIIILjava/awt/Color;Ljava/awt/image/ImageObserver;)Z (ImageRepresentation.java:758) (+0x145 bytes)
5d17fad0 6d2b4f99 java!sun/java2d/pipe/DrawImage.scaleImage(Lsun/java2d/SunGraphics2D;Ljava/awt/Image;IIIILjava/awt/Color;Ljava/awt/image/ImageObserver;)Z (DrawImage.java:781) (+0xee bytes)
5d17fb10 59a8d093 java!sun/java2d/pipe/ValidatePipe.scaleImage(Lsun/java2d/SunGraphics2D;Ljava/awt/Image;IIIILjava/awt/Color;Ljava/awt/image/ImageObserver;)Z (ValidatePipe.java:162) (+0x79 bytes)
5d17fb48 59a8cf89 java!sun/java2d/SunGraphics2D.drawImage(Ljava/awt/Image;IIIILjava/awt/Color;Ljava/awt/image/ImageObserver;)Z (SunGraphics2D.java:2716) (+0x103 bytes)
5d17fb90 59a4324a java!sun/java2d/SunGraphics2D.drawImage(Ljava/awt/Image;IIIILjava/awt/image/ImageObserver;)Z (SunGraphics2D.java:2669) (+0x29 bytes)

Since i dont have the symbols for awt.dll, the:

awt!Java_sun_awt_image_ImagingLib_init references on the stack is wrong (only export symbols).

But it seems that somewhere from the entry awt!Java_sun_awt_image_ImagingLib_transformBI, several calls to *(env)->GetPrimitiveArrayCritical and *(env)->ReleasePrimitiveArrayCritical exist.

Somewhere in the code path I would assume there is an allocation being made to the heap which fails and as such triggers the code in awt_NewHandler::handler(size_t) EVEN though the code is in nested critical. This breaks the JNI specification.

I have also tried the 1.4.2_21 distribution of the JDK but the problem is still there.

If i can get the symbol files for either (or both) for awt.dll version:

1.4.2_21: Tue May 05 22:19:49 2009 (4A009F65)

1.4.2_17: Sat Feb 09 04:17:43 2008 (47AD1B57)

I could probably give you more information about which code path potentially have the unmatched *(env)->GetPrimitiveArrayCritical and *(env)->ReleasePrimitiveArrayCritical pair.

Summary:

Entry into Java_sun_awt_image_ImagingLib_transformBI ends up eventually calling java/lang/Runtime.gc() through awt_NewHandler::handler(size_t) EVEN though the thread is in critical. This breaks the JNI specification and causes different kinds of problems, deadlocking most prevalent.

Cannot reproduce, but I could extract more information if symbols for awt.dll (win_ia32) is provided by SUN.

Thank you
Markus Gronlund
Oracle JRockit Sustaining Engineering

Release Regression From : 1.4.2_21
The above release value was the last known release where this
bug was not reproducible. Since then there has been a regression.
I guess problem seems to be not reproducible, submitter wants to see more with symbols.

http://jre.sfbay/java/re/j2sdk/1.4.2_21/promoted/fcs/b03/j4b/bundles/windows-i586/

this usually contains the symbols files as requested.
j2sdkfb-1_4_2_21-fcs-bin-b03-windows-i586-05_may_2009.zip