String.getBytes() does not work on some strings larger than 16MB

Name: rmT116609 Date: 11/05/2003


FULL PRODUCT VERSION :
java version "1.4.2"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2-b28)
Java HotSpot(TM) Client VM (build 1.4.2-b28, mixed mode)



FULL OS VERSION :
Linux wks001 2.4.20-19.9 #1 Wed Jul 23 19:06:26 EDT 2003 i686 i686 i386 GNU/Linux
SunOS drip 5.8 Generic_108528-22 sun4u sparc SUNW,UltraAX-i2

A DESCRIPTION OF THE PROBLEM :
When a string gets over a certain length (16777216 characters), calling getBytes() on it will trigger a java.nio.BufferOverflowException for certain string lengths. Adding one character at a time, shows this to be 1 in every 4.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Create a file of at least 16777216 characters (this is the boundary at which the bug starts to occur). e.g.:

  dd if=/dev/zero of=/tmp/inputfile bs=1024 count=16384

Create a test program to read in this file to a string. Then repeatedly add a character to the string and call getBytes() on it. Each 4th character added will cause a java.nio.BufferOverflowException. See source code example for this.



EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
total length = 16777216
now at total length = 16777217
now at total length = 16777218
now at total length = 16777219
now at total length = 16777220
now at total length = 16777221
now at total length = 16777222
now at total length = 16777223
now at total length = 16777224
now at total length = 16777225
now at total length = 16777226
now at total length = 16777227
now at total length = 16777228
now at total length = 16777229
now at total length = 16777230
...etc...
ACTUAL -
total length = 16777216
now at total length = 16777217
Error at total length = 16777217
java.nio.BufferOverflowException
now at total length = 16777218
now at total length = 16777219
now at total length = 16777220
now at total length = 16777221
Error at total length = 16777221
java.nio.BufferOverflowException
now at total length = 16777222
now at total length = 16777223
now at total length = 16777224
now at total length = 16777225
Error at total length = 16777225
java.nio.BufferOverflowException
now at total length = 16777226
now at total length = 16777227
now at total length = 16777228
now at total length = 16777229
Error at total length = 16777229
java.nio.BufferOverflowException
now at total length = 16777230
...etc...

ERROR MESSAGES/STACK TRACES THAT OCCUR :
Exception in thread "main" java.nio.BufferOverflowException
        at java.nio.charset.CoderResult.throwException(CoderResult.java:259)
        at java.lang.StringCoding$CharsetSE.encode(StringCoding.java:340)
        at java.lang.StringCoding.encode(StringCoding.java:374)
        at java.lang.StringCoding.encode(StringCoding.java:380)
        at java.lang.String.getBytes(String.java:590)
        at TestBuffer2.main(TestBuffer2.java:21)


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.io.*;

class TestBuffer2 {

public static void main(String[] args) throws IOException {

StringBuffer output = new StringBuffer();

byte[] buf = new byte[102400];
FileInputStream fis = new FileInputStream("/tmp/inputfile");
long totalLength=0;
int bytes = 0;
while((bytes = fis.read(buf))>0) {
output.append(new String(buf,0,bytes));
totalLength+=bytes;
}
System.out.println("total length = "+totalLength);

for (int i = 0; i < 10000; i++) {
try {
byte bufferoverflow2[] = output.toString().getBytes();
} catch (Exception e) {
System.out.println("Error at total length = "+totalLength);
System.out.println(e);
}
output.append("a");
totalLength += 1;
System.out.println("now at total length = "+totalLength);
}
System.out.println("Done!\n\n");
}
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
If you are not concerned with the exact format of your output string (e.g. when using it for HTML or XML purposes), you can hack around the problem like this:

if (output.length() > 16777217 && output.length() % 4 == 1) {
    output.append("\n");
}
(Incident Review ID: 223082)
======================================================================
###@###.### 2004-11-09 00:34:44 GMT