Submitter has issue with timeouts in unit tests for WLS environment.

SERVER end hanging thread :

ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" #29 daemon prio=5
os_prio=0 tid=0x00007f7f800dd800 nid=0x7dde runnable [0x00007f7fe8497000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x00000000c206f350> (a sun.nio.ch.Util$3)
    - locked <0x00000000c206f340> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000000c206f228> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
    at weblogic.socket.NIOSocketMuxer.selectFrom(NIOSocketMuxer.java:653)
    at weblogic.socket.NIOSocketMuxer.processSockets(NIOSocketMuxer.java:560)
    at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:30)
    at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:43)
    at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119)

CLIENT end - (threads of interest) :

[javatest.batch] "ExecuteThread: '2' for queue: 'default'" #13 daemon prio=5
os_prio=0 tid=0x00007f7a2b111800 nid=0x7fe9 waiting for monitor entry [0x00007f7a05d4c000]
[javatest.batch] java.lang.Thread.State: BLOCKED (on object monitor)
[javatest.batch] at java.nio.channels.Channels.writeFully(Channels.java:96)
[javatest.batch] - waiting to lock <0x0000000770d61350> (a java.lang.Object)
[javatest.batch] at java.nio.channels.Channels.access$000(Channels.java:61)
[javatest.batch] at java.nio.channels.Channels$1.write(Channels.java:174)
[javatest.batch] - locked <0x000000077175acc0> (a java.nio.channels.Channels$1)
[javatest.batch] at weblogic.iiop.IIOPOutputStream.writeTo(IIOPOutputStream.java:564)
[javatest.batch] at weblogic.iiop.MuxableSocketIIOP.send(MuxableSocketIIOP.java:398)
[javatest.batch] at weblogic.protocol.AsyncMessageSenderImpl.sendOutMsg(AsyncMessageSenderImpl.java:90)
[javatest.batch] at weblogic.protocol.AsyncMessageSenderImpl.send(AsyncMessageSenderImpl.java:46)
[javatest.batch] at weblogic.iiop.MuxableSocketIIOP$IIOPConnection.send(MuxableSocketIIOP.java:472)
[javatest.batch] at weblogic.iiop.EndPointImpl.send(EndPointImpl.java:733)


[javatest.batch] "ExecuteThread: '3' for queue: 'default'" #14 daemon prio=5
os_prio=0 tid=0x00007f7a2b113000 nid=0x7fea runnable [0x00007f7a05c4b000]
[javatest.batch] java.lang.Thread.State: RUNNABLE
[javatest.batch] at sun.nio.ch.Net.poll(Native Method)
[javatest.batch] at sun.nio.ch.SocketChannelImpl.poll(SocketChannelImpl.java:954)
[javatest.batch] - locked <0x0000000770d61360> (a java.lang.Object)
[javatest.batch] at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:204)
[javatest.batch] - locked <0x0000000770d61350> (a java.lang.Object)
[javatest.batch] at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
[javatest.batch] - locked <0x0000000770dc1610> (a sun.nio.ch.SocketAdaptor$SocketInputStream)
[javatest.batch] atweblogic.socket.SocketMuxer.readFromSocket(SocketMuxer.java:1018)
[javatest.batch] at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:952)
[javatest.batch] at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:919)
[javatest.batch] at weblogic.socket.JavaSocketMuxer.processSockets(JavaSocketMuxer.java:385)
[javatest.batch] at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:30)
[javatest.batch] at weblogic.work.ExecuteRequestAdapter.execute(ExecuteRequestAdapter.java:21)

Further input from submitter :

After investigating more, the reason seems to be that NIO non-blocking
reading intermittently hang at specific Linux kernel. I checked the history
of Jenkins rmi-iiop tests, all tests run on Linux kernel
3.8.13-118.20.2.el6uek.x86_64 can pass, but the tests run on Linux
4.1.12-112.14.13.el6uek.x86_64 intermittently fail with the "response
timeout" error. So this might be Linux kernel related problem