Summary

Clarify the spec of java.util.StringTokenizer(String, String, boolean) constructor, as to the length of the returned delimiter.

Problem

The current constructor spec reads:

     * If the {@code returnDelims} flag is {@code true}, then
     * the delimiter characters are also returned as tokens. Each
     * delimiter is returned as a string of length one. If the flag is
     * {@code false}, the delimiter characters are skipped and only
     * serve as separators between tokens.

The "length one" part is not correct if the delimiter is a supplementary character. In that case, String.length() returns 2 for the delimiter.

Solution

Modify the part with a more correct description, accommodating supplementary delimiters as well.

Specification

Change the constructor description of java.util.StringTokenizer(String, String, boolean) as:

@@ -172,13 +172,14 @@
      * characters in the {@code delim} argument are the delimiters
      * for separating tokens.
      * <p>
      * If the {@code returnDelims} flag is {@code true}, then
      * the delimiter characters are also returned as tokens. Each
-     * delimiter is returned as a string of length one. If the flag is
-     * {@code false}, the delimiter characters are skipped and only
-     * serve as separators between tokens.
+     * delimiter is returned as a string consisting of a single
+     * <a href="../lang/Character.html#unicode">Unicode code point</a>
+     * of the delimiter (which may be one or two {@code char}s). If the
+     * flag is {@code false}, the delimiter characters are skipped
+     * and only serve as separators between tokens.
      * <p>
      * Note that if {@code delim} is {@code null}, this constructor does
      * not throw an exception. However, trying to invoke other methods on the
      * resulting {@code StringTokenizer} may result in a
      * {@code NullPointerException}.