FULL PRODUCT VERSION :
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Linux 2.6.35-30-generic #54-Ubuntu SMP Tue Jun 7 18:41:54 UTC 2011 x86_64 GNU/Linux


EXTRA RELEVANT SYSTEM CONFIGURATION :
Accessing a HTTPS webservice hosted in IIS. IIS service is secured with SSL "Require Client Certificate" methodology.

A DESCRIPTION OF THE PROBLEM :
I have a program which uses Java NIO and SSL Engine API. I am using this program as a client to connect to a web service hosted in IIS (7.0). In IIS I have enable “Require Certificate” option under SSL configurations. In other words it requires client certificate to successfully establish the connection.
By default IIS server does not always negotiate client certificate. Therefore in my Java client program I had to use “-Dsun.security.ssl.allowUnsafeRenegotiation=true” parameter. But still I was not able to successfully communicate with IIS server using my Java client program.
I tested the same scenario with a synchronous SSL implementation and program worked successfully after specifying “-Dsun.security.ssl.allowUnsafeRenegotiation”.

It seems when used NIO, SSL implementation does not honor “allowUnsafeRenegotiation” parameter. With NIO following log is printed, (When SSL logging is enabled)

>Using SSLEngineImpl.
>Allow unsafe renegotiation: false
>Allow legacy hello messages: true
>Is initial handshake: true
>Is secure renegotiation: false

With normal transport following log is printed,

>executing requestGET /WebHost/SampleService.svc?wsdl HTTP/1.1
>main, setSoTimeout(0) called
>Allow unsafe renegotiation: true
>Allow legacy hello messages: true
>Is initial handshake: true
>Is secure renegotiation: false

In the latter case “Allow unsafe renegotiation” is set to true where as in NIO case it is set to “false”.


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Write a client program which uses SSLEngine and Java NIO
Setup HTTPS endpoint in IIS and enable "Require Certificate" option
Try to access HTTPS endpoint using Java client program by passing “-Dsun.security.ssl.allowUnsafeRenegotiation=true” parameter.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Java client successfully handshake with IIS endpoint and successfully communicating
ACTUAL -
Handshake is not successful

ERROR MESSAGES/STACK TRACES THAT OCCUR :
SSL Error Log - Note that unsafe renegotiation is set to false
===============================================

Using SSLEngineImpl.
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
%% No cached client session
*** ClientHello, TLSv1
RandomCookie: GMT: 1311169244 bytes = { 55, 97, 53, 183, 255, 145, 196, 9, 102, 31, 227, 30, 212, 165, 243, 190, 68, 178, 59, 217, 169, 230, 24, 170, 229, 129, 146, 112 }
Session ID: {}
Cipher Suites: [SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_RC4_40_MD5, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods: { 0 }
***
[write] MD5 and SHA1 hashes: len = 81
0000: 01 00 00 4D 03 01 4E 27 DB DC 37 61 35 B7 FF 91 ...M..N'..7a5...
0010: C4 09 66 1F E3 1E D4 A5 F3 BE 44 B2 3B D9 A9 E6 ..f.......D.;...
0020: 18 AA E5 81 92 70 00 00 26 00 04 00 05 00 2F 00 .....p..&...../.
0030: 35 00 33 00 39 00 32 00 38 00 0A 00 16 00 13 00 5.3.9.2.8.......
0040: 09 00 15 00 12 00 03 00 08 00 14 00 11 00 FF 01 ................
0050: 00 .
https-Sender I/O dispatcher-1, WRITE: TLSv1 Handshake, length = 81
[write] MD5 and SHA1 hashes: len = 110
0000: 01 03 01 00 45 00 00 00 20 00 00 04 01 00 80 00 ....E... .......
0010: 00 05 00 00 2F 00 00 35 00 00 33 00 00 39 00 00 ..../..5..3..9..
0020: 32 00 00 38 00 00 0A 07 00 C0 00 00 16 00 00 13 2..8............
0030: 00 00 09 06 00 40 00 00 15 00 00 12 00 00 03 02 .....@..........
0040: 00 80 00 00 08 00 00 14 00 00 11 00 00 FF 4E 27 ..............N'
0050: DB DC 37 61 35 B7 FF 91 C4 09 66 1F E3 1E D4 A5 ..7a5.....f.....
0060: F3 BE 44 B2 3B D9 A9 E6 18 AA E5 81 92 70 ..D.;........p
https-Sender I/O dispatcher-1, WRITE: SSLv2 client hello message, length = 110
[Raw write]: length = 112
0000: 80 6E 01 03 01 00 45 00 00 00 20 00 00 04 01 00 .n....E... .....
0010: 80 00 00 05 00 00 2F 00 00 35 00 00 33 00 00 39 ....../..5..3..9
0020: 00 00 32 00 00 38 00 00 0A 07 00 C0 00 00 16 00 ..2..8..........
0030: 00 13 00 00 09 06 00 40 00 00 15 00 00 12 00 00 .......@........
0040: 03 02 00 80 00 00 08 00 00 14 00 00 11 00 00 FF ................
0050: 4E 27 DB DC 37 61 35 B7 FF 91 C4 09 66 1F E3 1E N'..7a5.....f...
0060: D4 A5 F3 BE 44 B2 3B D9 A9 E6 18 AA E5 81 92 70 ....D.;........p
[Raw read]: length = 5
0000: 16 03 01 02 61 ....a
[Raw read]: length = 609
0000: 02 00 00 46 03 01 4E 27 DB DA 48 A7 DB FC D8 45 ...F..N'..H....E
0010: AA 65 30 47 BD 22 A9 24 E3 E1 38 BF DE EF 9C 54 .e0G.".$..8....T
0020: 78 3E 7F 7E 8D 92 20 B6 03 00 00 92 3F 20 3D B6 x>.... .....? =.
0030: CC 3E 60 AC 68 D9 10 AC 1B 17 08 E7 27 91 9D 46 .>`.h.......'..F
0040: BB 91 40 38 96 48 9F 00 2F 00 0B 00 02 0F 00 02 ..@8.H../.......
0050: 0C 00 02 09 30 82 02 05 30 82 01 6E A0 03 02 01 ....0...0..n....
0060: 02 02 10 BE 43 09 77 97 76 6C AA 4A F9 DC 4F 0A ....C.w.vl.J..O.
0070: 9B EB 62 30 0D 06 09 2A 86 48 86 F7 0D 01 01 04 ..b0...*.H......
0080: 05 00 30 12 31 10 30 0E 06 03 55 04 03 13 07 61 ..0.1.0...U....a
0090: 6A 2D 63 65 72 74 30 1E 17 0D 39 39 31 32 33 31 j-cert0...991231
00A0: 31 38 33 30 30 30 5A 17 0D 34 39 31 32 33 31 31 183000Z..4912311
00B0: 38 33 30 30 30 5A 30 12 31 10 30 0E 06 03 55 04 83000Z0.1.0...U.
00C0: 03 13 07 61 6A 2D 63 65 72 74 30 81 9F 30 0D 06 ...aj-cert0..0..
00D0: 09 2A 86 48 86 F7 0D 01 01 01 05 00 03 81 8D 00 .*.H............
00E0: 30 81 89 02 81 81 00 C2 A3 E8 A6 E4 51 70 85 B4 0...........Qp..
00F0: A0 E8 93 EC 39 38 FB 94 89 02 23 76 21 1B A3 A1 ....98....#v!...
0100: 57 AD 09 9E 2A 84 2E FB 11 A5 23 71 73 7F 9F 75 W...*.....#qs..u
0110: 2D 2E CB 87 C6 CE C3 55 AB ED 45 04 C3 AA D8 DF -......U..E.....
0120: CA 7F 6F C4 DA 2E D5 A4 E9 B1 19 EE C6 7C 87 E5 ..o.............
0130: AF 8B C3 83 5F AF 07 71 92 C0 2A 6E C9 33 65 89 ...._..q..*n.3e.
0140: 0F E7 C9 CD 36 8D 40 FE 0F 40 16 97 28 B5 55 DF ....6.@..@..(.U.
0150: 4D 51 02 11 89 2B 15 89 50 DF 7C D9 58 61 47 42 MQ...+..P...XaGB
0160: C6 40 27 3C 71 C2 A1 02 03 01 00 01 A3 5C 30 5A .@'<q........\0Z
0170: 30 13 06 03 55 1D 25 04 0C 30 0A 06 08 2B 06 01 0...U.%..0...+..
0180: 05 05 07 03 01 30 43 06 03 55 1D 01 04 3C 30 3A .....0C..U...<0:
0190: 80 10 AC 17 4A 34 DF C8 CD E7 E8 99 53 F4 D5 EF ....J4......S...
01A0: EE A0 A1 14 30 12 31 10 30 0E 06 03 55 04 03 13 ....0.1.0...U...
01B0: 07 61 6A 2D 63 65 72 74 82 10 BE 43 09 77 97 76 .aj-cert...C.w.v
01C0: 6C AA 4A F9 DC 4F 0A 9B EB 62 30 0D 06 09 2A 86 l.J..O...b0...*.
01D0: 48 86 F7 0D 01 01 04 05 00 03 81 81 00 67 23 E0 H............g#.
01E0: 8B 3D B0 2C 9D 8D B0 DA 55 70 D9 A7 AE C7 1A F2 .=.,....Up......
01F0: 05 43 0B 95 0C FA F9 CA 82 D9 FA B7 42 CF 88 CF .C..........B...
0200: 04 00 2A 08 ED 7D 7C AB 2F 2B E3 07 D3 E5 4E 6F ..*...../+....No
0210: 38 35 B8 8F 77 9F 06 2F F8 ED 08 4B 5E 7F 7F 0B 85..w../...K^...
0220: D2 B2 03 56 09 E7 82 1B 69 2A 9E 46 E0 40 91 AA ...V....i*.F.@..
0230: 3F 33 A6 F4 E0 69 0F 4B 12 56 54 2B 4E 92 54 0C ?3...i.K.VT+N.T.
0240: EF 6C B6 1A 6B B3 D3 EC 7B CB 40 37 3D C0 19 B8 .l..k.....@7=...
0250: 72 1B 90 FE 4F A3 31 64 25 4C 6E 36 28 0E 00 00 r...O.1d%Ln6(...
0260: 00 .
https-Sender I/O dispatcher-1, READ: TLSv1 Handshake, length = 609
*** ServerHello, TLSv1
RandomCookie: GMT: 1311169242 bytes = { 72, 167, 219, 252, 216, 69, 170, 101, 48, 71, 189, 34, 169, 36, 227, 225, 56, 191, 222, 239, 156, 84, 120, 62, 127, 126, 141, 146 }
Session ID: {182, 3, 0, 0, 146, 63, 32, 61, 182, 204, 62, 96, 172, 104, 217, 16, 172, 27, 23, 8, 231, 39, 145, 157, 70, 187, 145, 64, 56, 150, 72, 159}
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA
Compression Method: 0
***
Warning: No renegotiation indication extension in ServerHello
%% Created: [Session-1, TLS_RSA_WITH_AES_128_CBC_SHA]
** TLS_RSA_WITH_AES_128_CBC_SHA
[read] MD5 and SHA1 hashes: len = 74
0000: 02 00 00 46 03 01 4E 27 DB DA 48 A7 DB FC D8 45 ...F..N'..H....E
0010: AA 65 30 47 BD 22 A9 24 E3 E1 38 BF DE EF 9C 54 .e0G.".$..8....T
0020: 78 3E 7F 7E 8D 92 20 B6 03 00 00 92 3F 20 3D B6 x>.... .....? =.
0030: CC 3E 60 AC 68 D9 10 AC 1B 17 08 E7 27 91 9D 46 .>`.h.......'..F
0040: BB 91 40 38 96 48 9F 00 2F 00 ..@8.H../.
*** Certificate chain
chain [0] = [
[
  Version: V3
  Subject: CN=aj-cert
  Signature Algorithm: MD5withRSA, OID = 1.2.840.113549.1.1.4

  Key: Sun RSA public key, 1024 bits
  modulus: 136681044873253699075218075320211068862424281882421768761631619303131241089317130259510057330712150821551051103403717999192665146330081144812780875864357549631096435866124471499932510506486896855140092040222659105925611946394695700894664860668204515986901190206235587067148894307622736257095571604305270129313
  public exponent: 65537
  Validity: [From: Sat Jan 01 00:30:00 IST 2000,
               To: Sat Jan 01 00:00:00 IST 2050]
  Issuer: CN=aj-cert
  SerialNumber: [ -41bcf688 68899355 b50623b0 f564149e]

Certificate Extensions: 2
[1]: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
]

[2]: ObjectId: 2.5.29.1 Criticality=false
Extension unknown: DER encoded OCTET string =
0000: 04 3C 30 3A 80 10 AC 17 4A 34 DF C8 CD E7 E8 99 .<0:....J4......
0010: 53 F4 D5 EF EE A0 A1 14 30 12 31 10 30 0E 06 03 S.......0.1.0...
0020: 55 04 03 13 07 61 6A 2D 63 65 72 74 82 10 BE 43 U....aj-cert...C
0030: 09 77 97 76 6C AA 4A F9 DC 4F 0A 9B EB 62 .w.vl.J..O...b


]
  Algorithm: [MD5withRSA]
  Signature:
0000: 67 23 E0 8B 3D B0 2C 9D 8D B0 DA 55 70 D9 A7 AE g#..=.,....Up...
0010: C7 1A F2 05 43 0B 95 0C FA F9 CA 82 D9 FA B7 42 ....C..........B
0020: CF 88 CF 04 00 2A 08 ED 7D 7C AB 2F 2B E3 07 D3 .....*...../+...
0030: E5 4E 6F 38 35 B8 8F 77 9F 06 2F F8 ED 08 4B 5E .No85..w../...K^
0040: 7F 7F 0B D2 B2 03 56 09 E7 82 1B 69 2A 9E 46 E0 ......V....i*.F.
0050: 40 91 AA 3F 33 A6 F4 E0 69 0F 4B 12 56 54 2B 4E @..?3...i.K.VT+N
0060: 92 54 0C EF 6C B6 1A 6B B3 D3 EC 7B CB 40 37 3D .T..l..k.....@7=
0070: C0 19 B8 72 1B 90 FE 4F A3 31 64 25 4C 6E 36 28 ...r...O.1d%Ln6(

]
***
Found trusted certificate:
[
[
  Version: V3
  Subject: CN=aj-cert
  Signature Algorithm: MD5withRSA, OID = 1.2.840.113549.1.1.4

  Key: Sun RSA public key, 1024 bits
  modulus: 136681044873253699075218075320211068862424281882421768761631619303131241089317130259510057330712150821551051103403717999192665146330081144812780875864357549631096435866124471499932510506486896855140092040222659105925611946394695700894664860668204515986901190206235587067148894307622736257095571604305270129313
  public exponent: 65537
  Validity: [From: Sat Jan 01 00:30:00 IST 2000,
               To: Sat Jan 01 00:00:00 IST 2050]
  Issuer: CN=aj-cert
  SerialNumber: [ -41bcf688 68899355 b50623b0 f564149e]

Certificate Extensions: 2
[1]: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
]

[2]: ObjectId: 2.5.29.1 Criticality=false
Extension unknown: DER encoded OCTET string =
0000: 04 3C 30 3A 80 10 AC 17 4A 34 DF C8 CD E7 E8 99 .<0:....J4......
0010: 53 F4 D5 EF EE A0 A1 14 30 12 31 10 30 0E 06 03 S.......0.1.0...
0020: 55 04 03 13 07 61 6A 2D 63 65 72 74 82 10 BE 43 U....aj-cert...C
0030: 09 77 97 76 6C AA 4A F9 DC 4F 0A 9B EB 62 .w.vl.J..O...b


]
  Algorithm: [MD5withRSA]
  Signature:
0000: 67 23 E0 8B 3D B0 2C 9D 8D B0 DA 55 70 D9 A7 AE g#..=.,....Up...
0010: C7 1A F2 05 43 0B 95 0C FA F9 CA 82 D9 FA B7 42 ....C..........B
0020: CF 88 CF 04 00 2A 08 ED 7D 7C AB 2F 2B E3 07 D3 .....*...../+...
0030: E5 4E 6F 38 35 B8 8F 77 9F 06 2F F8 ED 08 4B 5E .No85..w../...K^
0040: 7F 7F 0B D2 B2 03 56 09 E7 82 1B 69 2A 9E 46 E0 ......V....i*.F.
0050: 40 91 AA 3F 33 A6 F4 E0 69 0F 4B 12 56 54 2B 4E @..?3...i.K.VT+N
0060: 92 54 0C EF 6C B6 1A 6B B3 D3 EC 7B CB 40 37 3D .T..l..k.....@7=
0070: C0 19 B8 72 1B 90 FE 4F A3 31 64 25 4C 6E 36 28 ...r...O.1d%Ln6(

]
[read] MD5 and SHA1 hashes: len = 531
0000: 0B 00 02 0F 00 02 0C 00 02 09 30 82 02 05 30 82 ..........0...0.
0010: 01 6E A0 03 02 01 02 02 10 BE 43 09 77 97 76 6C .n........C.w.vl
0020: AA 4A F9 DC 4F 0A 9B EB 62 30 0D 06 09 2A 86 48 .J..O...b0...*.H
0030: 86 F7 0D 01 01 04 05 00 30 12 31 10 30 0E 06 03 ........0.1.0...
0040: 55 04 03 13 07 61 6A 2D 63 65 72 74 30 1E 17 0D U....aj-cert0...
0050: 39 39 31 32 33 31 31 38 33 30 30 30 5A 17 0D 34 991231183000Z..4
0060: 39 31 32 33 31 31 38 33 30 30 30 5A 30 12 31 10 91231183000Z0.1.
0070: 30 0E 06 03 55 04 03 13 07 61 6A 2D 63 65 72 74 0...U....aj-cert
0080: 30 81 9F 30 0D 06 09 2A 86 48 86 F7 0D 01 01 01 0..0...*.H......
0090: 05 00 03 81 8D 00 30 81 89 02 81 81 00 C2 A3 E8 ......0.........
00A0: A6 E4 51 70 85 B4 A0 E8 93 EC 39 38 FB 94 89 02 ..Qp......98....
00B0: 23 76 21 1B A3 A1 57 AD 09 9E 2A 84 2E FB 11 A5 #v!...W...*.....
00C0: 23 71 73 7F 9F 75 2D 2E CB 87 C6 CE C3 55 AB ED #qs..u-......U..
00D0: 45 04 C3 AA D8 DF CA 7F 6F C4 DA 2E D5 A4 E9 B1 E.......o.......
00E0: 19 EE C6 7C 87 E5 AF 8B C3 83 5F AF 07 71 92 C0 .........._..q..
00F0: 2A 6E C9 33 65 89 0F E7 C9 CD 36 8D 40 FE 0F 40 *n.3e.....6.@..@
0100: 16 97 28 B5 55 DF 4D 51 02 11 89 2B 15 89 50 DF ..(.U.MQ...+..P.
0110: 7C D9 58 61 47 42 C6 40 27 3C 71 C2 A1 02 03 01 ..XaGB.@'<q.....
0120: 00 01 A3 5C 30 5A 30 13 06 03 55 1D 25 04 0C 30 ...\0Z0...U.%..0
0130: 0A 06 08 2B 06 01 05 05 07 03 01 30 43 06 03 55 ...+.......0C..U
0140: 1D 01 04 3C 30 3A 80 10 AC 17 4A 34 DF C8 CD E7 ...<0:....J4....
0150: E8 99 53 F4 D5 EF EE A0 A1 14 30 12 31 10 30 0E ..S.......0.1.0.
0160: 06 03 55 04 03 13 07 61 6A 2D 63 65 72 74 82 10 ..U....aj-cert..
0170: BE 43 09 77 97 76 6C AA 4A F9 DC 4F 0A 9B EB 62 .C.w.vl.J..O...b
0180: 30 0D 06 09 2A 86 48 86 F7 0D 01 01 04 05 00 03 0...*.H.........
0190: 81 81 00 67 23 E0 8B 3D B0 2C 9D 8D B0 DA 55 70 ...g#..=.,....Up
01A0: D9 A7 AE C7 1A F2 05 43 0B 95 0C FA F9 CA 82 D9 .......C........
01B0: FA B7 42 CF 88 CF 04 00 2A 08 ED 7D 7C AB 2F 2B ..B.....*...../+
01C0: E3 07 D3 E5 4E 6F 38 35 B8 8F 77 9F 06 2F F8 ED ....No85..w../..
01D0: 08 4B 5E 7F 7F 0B D2 B2 03 56 09 E7 82 1B 69 2A .K^......V....i*
01E0: 9E 46 E0 40 91 AA 3F 33 A6 F4 E0 69 0F 4B 12 56 .F.@..?3...i.K.V
01F0: 54 2B 4E 92 54 0C EF 6C B6 1A 6B B3 D3 EC 7B CB T+N.T..l..k.....
0200: 40 37 3D C0 19 B8 72 1B 90 FE 4F A3 31 64 25 4C @7=...r...O.1d%L
0210: 6E 36 28 n6(
*** ServerHelloDone
[read] MD5 and SHA1 hashes: len = 4
0000: 0E 00 00 00 ....
*** ClientKeyExchange, RSA PreMasterSecret, TLSv1
[write] MD5 and SHA1 hashes: len = 134
0000: 10 00 00 82 00 80 3B 79 B6 F1 2D 84 39 9F C7 D4 ......;y..-.9...
0010: B8 E5 4C 7B 27 C2 C3 16 C4 92 B4 ED DA 89 9F 82 ..L.'...........
0020: D7 F3 3B 71 B1 0A 9B F8 EB E3 3A 97 05 C3 79 AB ..;q......:...y.
0030: 2C 23 C6 97 4B CD 91 3A DB E6 84 9B AA CB F1 AB ,#..K..:...


( This report has more than 16,000 characters and has been truncated. )