-
Bug
-
Resolution: Incomplete
-
P4
-
None
-
17.0.16
-
x86_64
-
linux
ADDITIONAL SYSTEM INFORMATION :
OS: 5.4.54-2.0.4.std7c.el7.x86_64
JDK: open jdk 17.0.16
A DESCRIPTION OF THE PROBLEM :
we use netty server, we found netty thread block on SocketDispatcher.read0
"NettyServerNIOSelector_3_3" #161 prio=5 os_prio=0 cpu=106798282.45ms elapsed=407181.15s tid=0x00007f6d300032c0 nid=0x1f4fa runnable [0x00007f6f38611000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.SocketDispatcher.read0(java.base@17.0.16/Native Method)
at sun.nio.ch.SocketDispatcher.read(java.base@17.0.16/SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(java.base@17.0.16/IOUtil.java:330)
at sun.nio.ch.IOUtil.read(java.base@17.0.16/IOUtil.java:284)
at sun.nio.ch.IOUtil.read(java.base@17.0.16/IOUtil.java:259)
at sun.nio.ch.SocketChannelImpl.read(java.base@17.0.16/SocketChannelImpl.java:417)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(Unknown Source)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(Unknown Source)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(Unknown Source)
at io.netty.channel.nio.NioEventLoop.run(Unknown Source)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(java.base@17.0.16/Thread.java:840)
Locked ownable synchronizers:
- <0x00001002df4f2278> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
Then, we use strace to trace this thread, it stuck on read(0, ....
but in our server, fd0 is not a socket. we dump heap, found the true fd is 3465, not 0.
we suspect fdval(env, fdo) return 0.
JNIEXPORT jint JNICALL
Java_sun_nio_ch_SocketDispatcher_read0(JNIEnv *env, jclass clazz,
jobject fdo, jlong address, jint len)
{
jint fd = fdval(env, fdo);
void *buf = (void *)jlong_to_ptr(address);
jint n = read(fd, buf, len);
if ((n == -1) && (errno == ECONNRESET || errno == EPIPE)) {
JNU_ThrowByName(env, "sun/net/ConnectionResetException", "Connection reset");
return IOS_THROWN;
} else {
return convertReturnVal(env, n, JNI_TRUE);
}
}
For more details, check this github issue. https://github.com/netty/netty/issues/15746
OS: 5.4.54-2.0.4.std7c.el7.x86_64
JDK: open jdk 17.0.16
A DESCRIPTION OF THE PROBLEM :
we use netty server, we found netty thread block on SocketDispatcher.read0
"NettyServerNIOSelector_3_3" #161 prio=5 os_prio=0 cpu=106798282.45ms elapsed=407181.15s tid=0x00007f6d300032c0 nid=0x1f4fa runnable [0x00007f6f38611000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.SocketDispatcher.read0(java.base@17.0.16/Native Method)
at sun.nio.ch.SocketDispatcher.read(java.base@17.0.16/SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(java.base@17.0.16/IOUtil.java:330)
at sun.nio.ch.IOUtil.read(java.base@17.0.16/IOUtil.java:284)
at sun.nio.ch.IOUtil.read(java.base@17.0.16/IOUtil.java:259)
at sun.nio.ch.SocketChannelImpl.read(java.base@17.0.16/SocketChannelImpl.java:417)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(Unknown Source)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(Unknown Source)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(Unknown Source)
at io.netty.channel.nio.NioEventLoop.run(Unknown Source)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(java.base@17.0.16/Thread.java:840)
Locked ownable synchronizers:
- <0x00001002df4f2278> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
Then, we use strace to trace this thread, it stuck on read(0, ....
but in our server, fd0 is not a socket. we dump heap, found the true fd is 3465, not 0.
we suspect fdval(env, fdo) return 0.
JNIEXPORT jint JNICALL
Java_sun_nio_ch_SocketDispatcher_read0(JNIEnv *env, jclass clazz,
jobject fdo, jlong address, jint len)
{
jint fd = fdval(env, fdo);
void *buf = (void *)jlong_to_ptr(address);
jint n = read(fd, buf, len);
if ((n == -1) && (errno == ECONNRESET || errno == EPIPE)) {
JNU_ThrowByName(env, "sun/net/ConnectionResetException", "Connection reset");
return IOS_THROWN;
} else {
return convertReturnVal(env, n, JNI_TRUE);
}
}
For more details, check this github issue. https://github.com/netty/netty/issues/15746