FULL PRODUCT VERSION :
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux ... 3.11.0-19-generic #33-Ubuntu SMP Tue Mar 11 18:48:34 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
This is a variant of the problem reported in bug 4744057 that requires multiple selectors.

Like 4744057, deadlock is possible because the selectNow() locks the cancelled key set, then the key lock of a cancelled key, but SelectableChannel.close() obtains the locks in the opposite order.

The test case registers SelectableChannels with two Selectors. It uses two threads to repeatedly select from each selector, and two other threads which close channels. Combined with some randomization to ensure that the selector cancelled key sets may be stored in different orders, it is easy to create a deadlock such as the following:

Found one Java-level deadlock:
=============================
"pool-1-thread-4":
  waiting to lock monitor 0x00007f6354007668 (object 0x00000000d8a66920, a java.util.HashSet),
  which is held by "pool-1-thread-2"
"pool-1-thread-2":
  waiting to lock monitor 0x00007f6354010838 (object 0x00000000da1339a0, a java.lang.Object),
  which is held by "pool-1-thread-3"
"pool-1-thread-3":
  waiting to lock monitor 0x00007f6350010ee8 (object 0x00000000d8a63e90, a java.util.HashSet),
  which is held by "pool-1-thread-1"
"pool-1-thread-1":
  waiting to lock monitor 0x00007f6350010a18 (object 0x00000000da134810, a java.lang.Object),
  which is held by "pool-1-thread-4"

Java stack information for the threads listed above:
===================================================
"pool-1-thread-4":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:88)
- waiting to lock <0x00000000d8a66920> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000000da134910> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:237)
- locked <0x00000000da134810> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000000da134800> (a java.lang.Object)
at Deadlock$3.run(Deadlock.java:62)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"pool-1-thread-2":
at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:127)
- waiting to lock <0x00000000da1339a0> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:185)
at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:177)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:150)
- locked <0x00000000d8a66920> (a java.util.HashSet)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000d8a669f0> (a sun.nio.ch.Util$2)
- locked <0x00000000d8a669e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d8a668c8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
at Deadlock$2.run(Deadlock.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"pool-1-thread-3":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:88)
- waiting to lock <0x00000000d8a63e90> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000000da133aa0> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:237)
- locked <0x00000000da1339a0> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000000da133990> (a java.lang.Object)
at Deadlock$3.run(Deadlock.java:62)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"pool-1-thread-1":
at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:127)
- waiting to lock <0x00000000da134810> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:185)
at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:177)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:150)
- locked <0x00000000d8a63e90> (a java.util.HashSet)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000d8a640b0> (a sun.nio.ch.Util$2)
- locked <0x00000000d8a64030> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d8a63c00> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
at Deadlock$1.run(Deadlock.java:22)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)



STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the attached test case on a multi-core machine.

It always fails for me on my laptop (Intel(R) Core(TM) i7-3667U CPU @ 2.00GHz) in under a second.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
JVM continues to run, observable by using jstack to check for deadlocks and high CPU.
ACTUAL -
The JVM running the test will quickly deadlock. This can be verified using jstack.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.io.IOException;
import java.nio.channels.Selector;
import java.nio.channels.SocketChannel;
import java.util.concurrent.Executor;
import java.util.concurrent.Executors;

public class Deadlock
{
    public static void main(String[] args) throws Exception
    {
        final Selector selector = Selector.open();
        final Selector selector2 = Selector.open();

        final Executor executor = Executors.newFixedThreadPool(4);

        executor.execute(new Runnable() {
            @Override
            public void run() {

                while (true) {
                    try {
                        selector.selectNow();
                        Thread.yield();
                    }
                    catch (Exception e) {
                    }
                }
            }
        });

        executor.execute(new Runnable() {
            @Override
            public void run() {
                while (true) {
                    try {
                        selector2.selectNow();
                        Thread.yield();
                    }
                    catch (Exception e) {
                    }
                }
            }
        });

        while (true) {
            final SocketChannel channel = SocketChannel.open();
            channel.configureBlocking(false);

            if (toss()) {
                channel.register(selector2, 0, null);
                channel.register(selector, 0, null);
            }
            else {
                channel.register(selector, 0, null);
                channel.register(selector2, 0, null);
            }

            executor.execute(new Runnable() {
                @Override
                public void run() {
                    try {
                        channel.close();
                    }
                    catch (IOException e) {
                    }
                }
            });
        }
    }

    private static boolean toss() {
        return Math.random() < 0.5;
    }
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Only workaround we have is to refactor to use a single selector.