FULL PRODUCT VERSION :
java version "1.8.0_162"
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)

java version "9.0.4"
Java(TM) SE Runtime Environment (build 9.0.4+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.4+11, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Linux 3.0.101-108.21-default #1 SMP Fri Dec 29 10:25:37 UTC 2017 (5f5299b) x86_64 x86_64 x86_64 GNU/Linux
Linux 4.4.87-18.29-default #1 SMP Wed Sep 13 07:07:43 UTC 2017 (3e35b20) x86_64 x86_64 x86_64 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
We are implementing a service that uses a TLSv1.2 connection for secure this connection.

When the connection fails our customer want to receive an TLS-Alert followed by an TCP-FIN Message.
As described in RFC 5246 section 7.2.2 (https://tools.ietf.org/html/rfc5246#section-7.2.2) a fatal alert have to be sent when an error becomes detected while the TLS-handshake.

At server side we are using our own X509TrustManager to checkClientTrusted and throwing a CertificateException if the check fails.
In such a case the error (javax.net.ssl.SSLHandshakeException: General SSLEngine problem) becomes detected by the server when TLS-Handshaking.
Stack Trace: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
  at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
  at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
  at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
  at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
  at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1906)
  at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:233)
  at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
  at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
  at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
  at java.security.AccessController.doPrivileged(Native Method)
  at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
  at org.opends.server.extensions.TLSByteChannel$ByteChannelImpl.doHandshake(TLSByteChannel.java:230)


If we configure the accepted SocketChannel with socket().setSoLinger(true, 10) and close the connection using shutdownInput and shutdownOutput we get a FIN from the server and an ACK from the client. This FIN, ACK happens only in this configuration.

In all other cases (no SO_Linger, using java.nio.channels.SocketChannel close-Method) we get a RST-message from the server what seems not to be RFC 5246 conform.

So we assume that the Java-Implementation of the "Transport Layer Security (TLS) Protocol Version 1.2" contains an error that prevents dispatching the TLS alert before sending a TCP FIN-message.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Start TLS-Connection from client.
configure the accepted ClientChannel.socket().setSoLinger(true, 10);
throw CertificateException from the used X509TrustManager when checkClientTrusted is called

call shutdownInput and shutdownOutput at the accepted ClientChannel

Trace the TCP traffic


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
TCP traffic:
server -> client: TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Certificate Unknown)
server -> client: TCP [FIN, Ack]
client -> server: TCP [ACK]
client -> server: TCP [FIN, Ack]
server -> client: TCP [ACK]
ACTUAL -
TCP traffic:
server -> client: TCP [FIN, Ack]
client -> server: TCP [ACK]
client -> server: TCP [FIN, Ack]
server -> client: TCP [ACK]

REPRODUCIBILITY :
This bug can be reproduced always.

SUPPORT :
YES