FULL PRODUCT VERSION :
$ /data/icon/linux-x86-64/java/java/jre/bin/java -version
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
$ uname -a
Linux icslvb17 4.4.0-36-generic #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux


EXTRA RELEVANT SYSTEM CONFIGURATION :
Java Web Start with active SecurityManager.
'All grant permission' Java policy file.

A DESCRIPTION OF THE PROBLEM :
The deadlock is divided between the javax.security.auth.SubjectDomainCombiner and javax.security.auth.Subject.SecureSet (java.util.Collections.SynchronizedSet).

I. Part
---------
Calls one thread java.security.AccessController.getContext(), this call goes via java.security.AccessControlContext.optimize() into javax.security.auth.SubjectDomainCombiner.combine().
In javax.security.auth.SubjectDomainCombiner.combine() a mutex will aquired on the cached ProtectedDomain-Map.
Behind this mutex the cached Principal-Set is checked against the Principal-Set of the associated javax.security.auth.Subject.
Within this 'equals' call the second mutex will aquire. This mutex is locked by part II.

-----8<----------8<-----snipp-----8<----------8<-----
    public ProtectionDomain[] combine(ProtectionDomain[] currentDomains,
                                ProtectionDomain[] assignedDomains) {
[...]
        synchronized(cachedPDs) {
            if (!subject.isReadOnly() &&
                !subject.getPrincipals().equals(principalSet)) {
[...]
-----8<----------8<-----snipp-----8<----------8<-----

II. Part
---------
At the same time another thread tried to add a new principal into Principal-Set on the same javax.security.auth.Subject.
This Principal-Set is a __synchronized__ javax.security.auth.Subject.SecureSet.
The add-method aquires the internal mutex of this synchronized Set and call the add-method on the underlying SecureSet. The SecureSet checks the permissions and this call also ends in javax.security.auth.SubjectDomainCombiner.combine() and tried to aquire the mutex on the cached Principal-Set. This mutex is locked by part I.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
* compile test case
* execute test case:
$ java -Djava.security.manager -Djava.security.policy=target/test-classes/test.java.policy -cp target/test-classes/ TestDeadlockInSubjectDomainCombiner_combine

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
No deadlock.
ACTUAL -
The deadlock is alive and the coordinates are printed on std-out.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
"Thread[Thread-39]" - Thread t@81
   java.lang.Thread.State: BLOCKED
at java.util.Collections$SynchronizedSet.equals(Unknown Source)
- waiting to lock <a028d6> (a java.util.Collections$SynchronizedSet) owned by "J00000000001 (""): " t@72
at javax.security.auth.SubjectDomainCombiner.combine(Unknown Source)
- locked <1112eec> (a javax.security.auth.SubjectDomainCombiner$WeakKeyValueMap)
at java.security.AccessControlContext.optimize(Unknown Source)
at java.security.AccessController.getContext(Unknown Source)

"J00000000001 (""): " - Thread t@72
   java.lang.Thread.State: BLOCKED
at javax.security.auth.SubjectDomainCombiner.combine(Unknown Source)
- waiting to lock <1112eec> (a javax.security.auth.SubjectDomainCombiner$WeakKeyValueMap) owned by "Thread[Thread-39]" t@81
at java.security.AccessControlContext.optimize(Unknown Source)
at java.security.AccessController.checkPermission(Unknown Source)
at java.lang.SecurityManager.checkPermission(Unknown Source)
at com.sun.javaws.security.JavaWebStartSecurity.checkPermission(Unknown Source)
at javax.security.auth.Subject$SecureSet.add(Unknown Source)
at java.util.Collections$SynchronizedCollection.add(Unknown Source)
- locked <a028d6> (a java.util.Collections$SynchronizedSet)


REPRODUCIBILITY :
This bug can be reproduced often.

---------- BEGIN SOURCE ----------

import static java.util.concurrent.TimeUnit.MILLISECONDS;

import java.lang.management.ManagementFactory;
import java.lang.management.ThreadInfo;
import java.lang.management.ThreadMXBean;
import java.security.AccessControlContext;
import java.security.AccessController;
import java.security.Principal;
import java.security.PrivilegedActionException;
import java.security.PrivilegedExceptionAction;
import java.security.ProtectionDomain;
import java.util.Arrays;
import java.util.Collections;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;
import java.util.concurrent.CyclicBarrier;

import org.apache.commons.collections.set.SynchronizedSet;

import javax.security.auth.Subject;
import javax.security.auth.SubjectDomainCombiner;

/**
 * Try to test a possible deadlock in
 * {@link SubjectDomainCombiner#combine(java.security.ProtectionDomain[], java.security.ProtectionDomain[])} between a
 * internal monitor on the cached domains and a implicit monitor by use
 * {@link SynchronizedSet#equals(java.lang.Object)} ({@link Subject#getPrincipals()}).
 *
 */
public class TestDeadlockInSubjectDomainCombiner_combine {

  private static final long BARRIER_TIMEOUT_MS = 250;

  private static final Subject SUBJECT = new Subject();

  public static void main(String[] args) throws Throwable {
    final CyclicBarrier barrier = new CyclicBarrier(3);
    // '3': 2 created threads and the current thread (test execution)
    final Map<String, Throwable> uncaughtExceptionMap = Collections.synchronizedMap(
            new TreeMap<String, Throwable>());

    final BarrierSubjectDomainCombiner barrierSubjectDomainCombiner = new BarrierSubjectDomainCombiner(barrier,
                                                                                                       SUBJECT);

    final Thread.UncaughtExceptionHandler uncaughtExceptionHandler = new Thread.UncaughtExceptionHandler() {
      @Override
      public void uncaughtException(final Thread t, final Throwable e) {
        uncaughtExceptionMap.put(t.getName(), e);
      }
    };
    try {

      AccessController.doPrivileged(deadlockAction(barrier, uncaughtExceptionHandler, barrierSubjectDomainCombiner),
                                    new AccessControlContext(AccessController.getContext(), barrierSubjectDomainCombiner));
    } catch (PrivilegedActionException pae) {
      throw pae.getException();
    }
    if (!uncaughtExceptionMap.isEmpty()) {
      throw uncaughtExceptionMap.values().iterator().next();
    }

    final ThreadMXBean threadMXBean = ManagementFactory.getThreadMXBean();

    final long[] deadlocks = threadMXBean.findDeadlockedThreads();
    if (deadlocks != null) {
      final ThreadInfo[] threadInfos = threadMXBean.getThreadInfo(deadlocks, true, true);
      System.err.println(Arrays.deepToString(threadInfos));
      System.exit(1);
    }
    System.exit(0);
  }

  private static PrivilegedExceptionAction<Void> deadlockAction(
          final CyclicBarrier barrier,
          final Thread.UncaughtExceptionHandler uncaughtExceptionHandler,
          final BarrierSubjectDomainCombiner barrierSubjectDomainCombiner) {
    return new PrivilegedExceptionAction<Void>() {
      @Override
      public Void run() throws Exception {
        final Set<Principal> principalSet = SUBJECT.getPrincipals();
        final Principal userSubjectIdPrincipal = new Principal() {
          @Override
          public String getName() {
            return "userSubjectId";
          }
        };

        final Thread accThread = new Thread("AccessController.getContext()") {
          @Override
          public void run() {
            try {
              barrierSubjectDomainCombiner.armBarrier(); // arm barrier after thread creation
              AccessController.getContext();
            } catch (Exception ex) {
              throw new RuntimeException(ex.getMessage(), ex);
            }
          }
        };
        accThread.setUncaughtExceptionHandler(uncaughtExceptionHandler);

        final Thread subjectThread = new Thread("subject.getPrincipals().add()") {
          @Override
          public void run() {
            try {
              barrierSubjectDomainCombiner.armBarrier(); // arm barrier after thread creation
              principalSet.add(userSubjectIdPrincipal);
            } catch (Exception ex) {
              throw new RuntimeException(ex.getMessage(), ex);
            }
          }
        };
        subjectThread.setUncaughtExceptionHandler(uncaughtExceptionHandler);

        accThread.start();
        subjectThread.start();

        barrier.await(BARRIER_TIMEOUT_MS, MILLISECONDS);
        return null;
      }
    };

  }

  /**
   * This class drills the test code down just before the interesting method.
   * <p>
   * The passed {@code barrier} holds the method execution and waits until all threads are at this point. The risk to
   * fall down into the deadlock is very hight.
   * </p>
   * <p>
   * It's required to deactivate the {@code barrier} while the thread creation. Within the thread constructor this
   * method will also called but at this point this call isn't interesting for the specific failure.
   * <br>The barrier will armed later after the thread construction.</br>
   * </p>
   */
  private static final class BarrierSubjectDomainCombiner extends SubjectDomainCombiner {

    private final CyclicBarrier barrier;
    private boolean armedBarrier = false;

    public BarrierSubjectDomainCombiner(final CyclicBarrier barrier, final Subject subject) {
      super(subject);
      this.barrier = barrier;
    }

    void armBarrier() {
      this.armedBarrier = true;
    }

    @Override
    public ProtectionDomain[] combine(ProtectionDomain[] currentDomains, ProtectionDomain[] assignedDomains) {
      if (armedBarrier) {
        try {
          this.barrier.await(BARRIER_TIMEOUT_MS, MILLISECONDS);
        } catch (Exception ex) {
          throw new RuntimeException(ex.getMessage(), ex);
        }
      }
      return super.combine(currentDomains, assignedDomains); //To change body of generated methods, choose Tools | Templates.
    }

  }
}

---------- END SOURCE ----------