See related CR filed for Access Manager : CR 6844567.
Customers are experiencing diminished throughput in Access Manager / OpenSSO products on T51xx / T52xx machines (high concurrency). CPU util on those machines are capped at 10~20%.

All thread dumps, generated with kill -3, show on average 80 threads waiting on a lock to be released in :

javax.security.auth.login.Configuration.getConfiguration

The stack trace below shows the thread that currently has the lock. Most of the other worker threads are waiting for a monitor entry, as shown below :

"service-j2ee" daemon prio=3 tid=0x01e7b060 nid=0x137 waiting for monitor entry [0x32d0d000..0x32d0f7d8]
        at javax.security.auth.login.Configuration.getConfiguration(Configuration.java:190)
        - waiting to lock <0xec228f50> (a java.lang.Class)
        at com.sun.identity.authentication.service.AMLoginContext.getModuleFromAuthConfiguration(AMLoginContext.java:1804)

I saw a bug / fix regarding the PolicyFile.implies call, but moving to update 19 doesn't seem to change anything.
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6552236

It looks to me that the synchronized getConfiguration() is a major bottleneck in highly multi-threaded environment.


"service-j2ee" daemon prio=3 tid=0x0294ac88 nid=0x12f runnable [0x32f8d000..0x32f8f7d8]
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.provider.PolicyFile.addPermissions(PolicyFile.java:1370)
        at sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1325)
        at sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1291)
        at sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1227)
        at sun.security.provider.PolicyFile.implies(PolicyFile.java:1182)
        at com.sun.enterprise.security.provider.PolicyWrapper.implies(PolicyWrapper.java:135)
        at java.security.ProtectionDomain.implies(ProtectionDomain.java:195)
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:249)
        at java.security.AccessController.checkPermission(AccessController.java:427)
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
        at javax.security.auth.login.Configuration.getConfiguration(Configuration.java:192)
        - locked <0xec228f50> (a java.lang.Class)
        at com.sun.identity.authentication.service.AuthUtils.isPureJAASModulePresent(AuthUtils.java:1399)
        at com.sun.identity.authentication.service.AMLoginContext.executeLogin(AMLoginContext.java:299)
        at com.sun.identity.authentication.server.AuthContextLocal.login(AuthContextLocal.java:327)
        at com.sun.identity.authentication.server.AuthContextLocal.login(AuthContextLocal.java:263)
        at com.sun.identity.authentication.server.AuthContextLocal.login(AuthContextLocal.java:183)
        at com.sun.identity.authentication.UI.LoginViewBean.getLoginDisplay(LoginViewBean.java:792)
        at com.sun.identity.authentication.UI.LoginViewBean.processLoginDisplay(LoginViewBean.java:1034)
        at com.sun.identity.authentication.UI.LoginViewBean.processLogin(LoginViewBean.java:734)
        at com.sun.identity.authentication.UI.LoginViewBean.forwardTo(LoginViewBean.java:444)
        at com.iplanet.jato.ApplicationServletBase.dispatchRequest(ApplicationServletBase.java:981)