FULL PRODUCT VERSION :
java version "1.8.0_77"
Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Darwin 15.4.0 Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64 x86_64


A DESCRIPTION OF THE PROBLEM :
String.getBytes calls StringCoding.encode, which in turns calls StringCoding.scale. The latter tries to allocate an array longer than MAX_ARRAY_SIZE. As a result, it throws an OOM which can causes the JVM to be killed.

The way StringCoding.scale is implemented, it is not aware of a potential array size limit smaller than Integer.MAX_VALUE. In contrast, in many other java standard library implementations, such limit is taken into consideration when allocating large arrays (For example, see ArrayList.MAX_ARRAY_SIZE)

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the program provided below with at least 8G heap space

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
it should either
* allocate a smaller array, try to do the conversion. If the array turns out to be long enough, great. Otherwise, either thow an exception that is not an OOM, or explicitly throw an OOMError (instead of triggering OOM from the VM).
* fail right away by either thowing an exception that is not an OOM, or explicitly throw an OOMError (instead of triggering OOM from the VM).
ACTUAL -
Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit

In turn, VM get killed if an OOM killer is configured.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------

        char[] c = new char[Integer.MAX_VALUE / 2];
        for (int i = 0; i < c.length; i++) {
            c[i] = 'A';
        }
        String s = new String(c);
        c = null;
        byte[] bytes = s.getBytes(StandardCharsets.UTF_8);
        System.out.println(bytes.length);
---------- END SOURCE ----------