Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-7096080

UTF8 update and new CESU-8 charset



    • Enhancement
    • Status: Closed
    • P3
    • Resolution: Fixed
    • 7
    • 8
    • core-libs
    • b14
    • generic
    • generic
    • Verified


      Unicode Standard added "Addition Constraints on conversion of ill-formed UTF-8"
      in version 5.1 [1] and updated in 6.0 again with further "clarification" [2] regarding
      how a "conformance" implementation should handle ill-formed UTF-8 byte
      sequence. Basically it says

       (1) the conversion process should not interpret any ill-formed code unit sequence
       (2) such process must not treat any adjacent well-formed code unit sequences
           as being part of those ill-formed code unit sequences
       (3) and recommend a "best practice" of "maximal valid subpart" for replacement

      The new UTF-8 charset implementation we put in JDK7 (and back-ported to previous
      release since then) follows the new constraints in most cases, except

      (1) The decoder still accepts "historical" 3 bytes surrogates and 6 bytes surrogate
      pair (the encoder never output such sequence). Unicode Standard "tightened" UTF-8
      definition in ver 3.2 [3], as

          "Most notable among the corrigenda to the Standard is a further tightening
           of the definition of UTF-8, to eliminate irregular UTF-8 and to bring the
           Unicode specification of UTF-8 more completely into line with other
           specifications of UTF-8."

           So the 3-byte/6-byte surrogates are defined as "ill-formed" code unit
           sequence, instead of "irregular" [5] in ver 3.1

      (2) While no longer accepting the "histrical" 5-byte, 6-byte UTF-8 byte sequence,
      the decoder treats these 5/6-byte sequence as ONE malformed unit. As a result
      these bytes get replaced by one replacement character, when "replace for mlaformed"
      is desirble (as in new String(bytes), for example). According the latest Unicode
      standard [2], however, because the leading byte of these 5/6-byte sequence is no
      longer an illegal appearance of the UTF-8, these bytes should be treated as 5-6
      individual ill-formed bytes.

      (3)Corner case like ill-fomred byte sequence ED 31 is not handled correctly/
      consistently, as described in #7082884 [6]

      The reason behind (1) and (2) is mostly the compatibility concern. As acknowledged
      in TR#26 [4] (in which it defines CESU-8, a separate UTF encoding scheme that
      uses 3-6-byte sequence for supplementary characters, instead of 4-byte sequence
      in UTF-8), there are apps/data over there that do use surrogates pair in "UTF-8"
      form. To change the UTF-8 charset to follow standard obviously will break
      someone's code when they migrate/upgrade from JDK/JRE N to N+1, something we
      really try hard to avoid.

      That said, gvien almost decade has passed and we are now Unicode 6, I think the
      possibility of breaking someone's code/date of upgrading UTF-8 to do the "right
      thing" is small/minor. So I proposed here

      (1) to upgrade the JDK8 UTF-8 implementation to strictly follow the standard to
          a) reject 3-byte surrogate/6-byte surrogate pair
          b) treats 5/6-byte surrogate as individual ill-formed bytes.
      (2) to add CESU-8 charset into JDK/JRE's charset repository (for those still
          prefer/work on 3-6 bytes surrogate, in "UTF-8" form)

      [1] http://www.unicode.org/versions/Unicode5.1.0/#Notable_Changes
      [2] http://www.unicode.org/versions/Unicode6.0.0/#Conformance_Changes
      [3] http://www.unicode.org/reports/tr28/tr28-3.html
      [4] http://unicode.org/reports/tr26/
      [5] http://unicode.org/versions/corrigendum1.html
      [6] http://mail.openjdk.java.net/pipermail/core-libs-dev/2011-September/007722.html


        Issue Links



              sherman Xueming Shen
              sherman Xueming Shen
              0 Vote for this issue
              0 Start watching this issue