When a Java Applet returns a big string to a calling java script, the applet will crash.
Customer reported, the crash will not happen at 1.5.0_11 or 1.6.0_04.
This issue happen on enabled NextGen Plugin only.

There is a testcase attached, modified from customers original used case.
The original used case is, that customer have a applet, do lookup a large pricebook
for many products with the help of a java applet, and return this long string back to
the calling java script.

Root Cause:
A effect is seen that once the string size to return is bigger then 5603322 byte, the applet will crash.
If it is equal or lower from this size, we see no applet crash.

Steps to reproduce from testcase:
- download the testcase and extract.
- configure Java console to start automatic, when a applet will be loaded.
- execute the index.html from the testcase, the applet will be loaded
you will see 3 buttons (Reload Home, Check Applet, View Pricebook)
- click on "Check Applet" button, it confirms Applet is loaded
- click on "View Pricebook" button, a popup html is loaded which should usual show the count in bytes returned.
But we see an endless gif scollbar only, from the popup window,
and Java Console dies unexpected. (JVM crashed)

- close these popup windows
- click button "Check Applet", it confirms applet is not longer running.
click on "View Pricebook" will show "failure" message, cause applet is not loaded.

Steps to prevent the error:
- open the bigStringApplet.java code with a text editor
- change these to lines from
// int size=100; //very small return string size works as expected
int size=8806595; //original customers used string size, jvm dies unexpected
to
int size=100; //very small return string size works as expected
// int size=8806595; //original customers used string size, jvm dies unexpected
- recompile the applet using
javac bigStringApplet.java
- the browser is still open, so click button "Reload Home", then "Check Applet", then "View Pricebook"
-> we will see the string size count 100 bytes is successfully returned.
Could you try with latest jdk-7 plugin.
I could not reproduce the "crash" in two scenarios:

A) Just above the crash limit (size=5603323 or 9603323):

I tested with size larger than the "crash" limit and still could not get the jvm crashed. Clicked on the "Check Applet" still get the good value. (Note that for the given testcase you should always start testing with "Check Applet" to have cookie initialized, otherwise "View Pricebook" always fails). Java Console output:

------------------------------------------------------------------------------
¿liveconnect: Invoking method: public boolean java.applet.Applet.isActive()
liveconnect: Invoking method: public java.lang.String bigStringApplet.getBigString(int)
liveconnect: Needs conversion: java.lang.Integer --> int
debug message: start bigString;
5603323
debug message: return bigString;
liveconnect: Invoking method: public boolean java.applet.Applet.isActive()
------------------------------------------------------------------------------


B) Inducing OutOfMemoryError:

I also tested with very large string buffer (size=32MB) with APPLET PARAM jvm_arguments=-Xmx=32m to force OutOfMemoryError.
After the error, the applet is still alive. I replace out the catch(Throwable t) with catch(Exception e) to see how the plugin code handle the error. The result is that an error dialog is displayed and the applet is still alive. Here is the log:

------------------------------------------------------------------------------
¿Trace level set to 5: all ... completed.
liveconnect: Invoking method: public boolean java.applet.Applet.isActive()
liveconnect: Invoking method: public java.lang.String bigStringApplet.getBigString(int)
liveconnect: Needs conversion: java.lang.Integer --> int
debug message: start bigString;
Ignored exception: java.lang.OutOfMemoryError: Java heap space
Ignored exception: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
liveconnect: Invoking method: public boolean java.applet.Applet.isActive()
------------------------------------------------------------------------------
Note that the isActive() ping before and after the OutOfMemoryError.