Customer application is getting java.lang.StackOverflowError after updating to java 6.0u20.

The SAM application uses SUN JDK 1.6 update 17 without any problems on all the supported platforms.
One of the SAM application JMS server has the following JVM options in update 17:

JVM_SAM_THREAD_DUMP_ENABLED="-Dsam.thread.dump=true"
JVM_OPTIONS_MEM="-XX:ThreadStackSize=128 -Xms512m -Xmx512m"
JVM_OPTIONS_PERM="-XX:PermSize=40m -XX:MaxPermSize=40m"
JVM_OPTIONS_GCLOG="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps"
JVM_OPTIONS_OTHER="-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseParNewGC -XX:+UseCMSCompactAtFullCollection -XX:CMSInitiatingPermOccupancyFraction=95"
JVM_OPTIONS_REMOTE="$JVM_OPTIONS_COMMON_REMOTE"
JVM_OPTIONS_DEBUG="$JVM_OPTIONS_COMMON_DEBUG -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5004 $JVM_OPTIONS_REMOTE"
JVM_HIGH_OPTIONS="$JVM_OPTIONS_MEM $JVM_OPTIONS_PERM $JVM_OPTIONS_OTHER $JVM_SAM_THREAD_DUMP_ENABLED $JVM_OPTIONS_DEBUG $JVM_OPTIONS_GCLOG"

Recently we upgrade JDK to 1.6 update 20. We tested SAM using update 20 on the following supported platforms:

(1) No problem on Windows (3.00GHz, 3GB RAM), using 32-bit OS.
(2) No problem on Solaris x86 using 64-bit JVM (typical hardware spec: 4CPU, 8GB RAM).
(3) Problem on two Solaris sparc machines using 32-bit JVM.
    Machine (a) 2 CPU, 4GB RAM (minimum harware spec that SAM supports).
    We increased the ThreadStackSize to 256, still failed. Then increassed to 512, succeeded.
    Machine (b) 2 CPU, 10GB RAM. We increased the ThreadStackSize to 256, succeeded.


During the start-up of JMS server on Solaris sparc, got the following errors:

<2010.06.21 10:55:19 954 -0400><W><orbit><main><org.jboss.mx.loading.LoadMgr3> Not resheduling failed loading task, loadTask=org.jboss.mx.loading.ClassLoadingTask@1b31fd9{classname: com.arjuna.ats.internal.arjuna.objectstore.FileSystemStore, requestingThread: Thread[main,5,jboss], requestingClassLoader: org.jboss.mx.loading.UnifiedClassLoader3@58213c{ url=null ,addedOrder=2}, loadedClass: nullnull, loadOrder: 2147483647, loadException: java.lang.StackOverflowError, threadTaskCount: 0, state: 1, #CCE: 1}
java.lang.StackOverflowError
       at sun.misc.VM.allowArraySyntax(VM.java:209)
       at java.lang.ClassLoader.checkName(ClassLoader.java:776)
       at java.lang.ClassLoader.findLoadedClass(ClassLoader.java:948)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:292)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
       at org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:154)
       at org.apache.log4j.Category.forcedLog(Category.java:388)
       at org.apache.log4j.Category.log(Category.java:853)
       at org.jboss.logging.Log4jLoggerPlugin.warn(Log4jLoggerPlugin.java:188)
       at org.jboss.logging.Logger.warn(Logger.java:216)
       at org.jboss.mx.loading.LoadMgr3.nextTask(LoadMgr3.java:431)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClassImpl(RepositoryClassLoader.java:540)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClass(RepositoryClassLoader.java:428)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
       at java.lang.ClassLoader.defineClass1(Native Method)
       at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
       at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
       at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
       at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
       at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
       at org.jboss.mx.loading.RepositoryClassLoader.findClassLocally(RepositoryClassLoader.java:699)
       at org.jboss.mx.loading.RepositoryClassLoader.findClass(RepositoryClassLoader.java:679)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClassLocally(RepositoryClassLoader.java:213)
       at org.jboss.mx.loading.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:131)
       at org.jboss.mx.loading.LoadMgr3.nextTask(LoadMgr3.java:399)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClassImpl(RepositoryClassLoader.java:540)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClass(RepositoryClassLoader.java:428)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
       at java.lang.ClassLoader.defineClass1(Native Method)
       at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
       at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
       at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
       at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
       at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
       at org.jboss.mx.loading.RepositoryClassLoader.findClassLocally(RepositoryClassLoader.java:699)
       at org.jboss.mx.loading.RepositoryClassLoader.findClass(RepositoryClassLoader.java:679)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClassLocally(RepositoryClassLoader.java:213)
       at org.jboss.mx.loading.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:131)
       at org.jboss.mx.loading.LoadMgr3.nextTask(LoadMgr3.java:399)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClassImpl(RepositoryClassLoader.java:540)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClass(RepositoryClassLoader.java:428)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
       at java.lang.ClassLoader.defineClass1(Native Method)
       at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
       at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
       at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
       at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
       at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
       at org.jboss.mx.loading.RepositoryClassLoader.findClassLocally(RepositoryClassLoader.java:699)
       at org.jboss.mx.loading.RepositoryClassLoader.findClass(RepositoryClassLoader.java:679)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClassLocally(RepositoryClassLoader.java:213)
       at org.jboss.mx.loading.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:131)
       at org.jboss.mx.loading.LoadMgr3.nextTask(LoadMgr3.java:399)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClassImpl(RepositoryClassLoader.java:540)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClass(RepositoryClassLoader.java:428)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
       at java.lang.ClassLoader.defineClass1(Native Method)
       at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
       at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
       at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
       at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
       at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
       at org.jboss.mx.loading.RepositoryClassLoader.findClassLocally(RepositoryClassLoader.java:699)
       at org.jboss.mx.loading.RepositoryClassLoader.findClass(RepositoryClassLoader.java:679)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClassLocally(RepositoryClassLoader.java:213)
       at org.jboss.mx.loading.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:131)
       at org.jboss.mx.loading.LoadMgr3.nextTask(LoadMgr3.java:399)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClassImpl(RepositoryClassLoader.java:540)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClass(RepositoryClassLoader.java:428)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
       at java.lang.ClassLoader.defineClass1(Native Method)
       at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
       at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
       at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
       at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
       at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
       at org.jboss.mx.loading.RepositoryClassLoader.findClassLocally(RepositoryClassLoader.java:699)
       at org.jboss.mx.loading.RepositoryClassLoader.findClass(RepositoryClassLoader.java:679)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClassLocally(RepositoryClassLoader.java:213)
       at org.jboss.mx.loading.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:131)
       at org.jboss.mx.loading.LoadMgr3.nextTask(LoadMgr3.java:399)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClassImpl(RepositoryClassLoader.java:540)
       at org.jboss.mx.loading.RepositoryClassLoader.loadClass(RepositoryClassLoader.java:428)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
       at com.arjuna.ats.internal.arjuna.objectstore.HashedActionStoreSetup.createVoid(HashedActionStoreSetup.java:49)
       at com.arjuna.ats.internal.arjuna.gandiva.inventory.StaticInventory.createVoid(StaticInventory.java:76)
       at com.arjuna.ats.arjuna.gandiva.inventory.Inventory.createVoid(Inventory.java:84)
       at com.arjuna.ats.arjuna.objectstore.ObjectStore.<init>(ObjectStore.java:128)
       at com.arjuna.ats.arjuna.coordinator.TxControl.getStore(TxControl.java:133)
       at com.arjuna.ats.arjuna.recovery.ActionStatusService.<init>(ActionStatusService.java:71)
       at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
       at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
       at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
       at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
       at java.lang.Class.newInstance0(Class.java:355)
       at java.lang.Class.newInstance(Class.java:308)
       at com.arjuna.ats.arjuna.recovery.TransactionStatusManager.start(TransactionStatusManager.java:149)
       at com.arjuna.ats.arjuna.recovery.TransactionStatusManager.<init>(TransactionStatusManager.java:78)
       at com.arjuna.ats.arjuna.coordinator.TxControl.<clinit>(TxControl.java:312)
       at com.arjuna.ats.jbossatx.jta.TransactionManagerService.startService(TransactionManagerService.java:145)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
       at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
       at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
       at $Proxy0.start(Unknown Source)
       at org.jboss.system.ServiceController.start(ServiceController.java:417)
       at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy4.start(Unknown Source)
       at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
       at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
       at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
       at $Proxy5.deploy(Unknown Source)
       at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482)
       at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362)
       at org.jboss.Main.boot(Main.java:200)
       at org.jboss.Main$1.run(Main.java:513)
       at java.lang.Thread.run(Thread.java:619)