FULL PRODUCT VERSION :
java version "1.8.0_40"
Java(TM) SE Runtime Environment (build 1.8.0_40-b25)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows [Version 6.2.9200]

A DESCRIPTION OF THE PROBLEM :
Trying to encode a character sequence which contains a large number of invalid code points (i.e. only high surrogates) sometime results in a StackOverflowError.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the example provided in "Source code for an executable test case".

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
No StackOverflowError
ACTUAL -
Exception in thread "main" java.lang.StackOverflowError
at java.util.HashMap.hash(HashMap.java:357)
at java.util.HashMap.getEntry(HashMap.java:462)
at java.util.HashMap.get(HashMap.java:417)
at java.nio.charset.CoderResult$Cache.get(CoderResult.java:209)
at java.nio.charset.CoderResult$Cache.access$200(CoderResult.java:195)
at java.nio.charset.CoderResult.malformedForLength(CoderResult.java:234)
at sun.nio.cs.Surrogate$Parser.parse(Surrogate.java:238)
at sun.nio.cs.UTF_8$Encoder.encodeArrayLoop(UTF_8.java:542)
at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:619)
at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:561)
at sun.nio.cs.StreamEncoder.flushLeftoverChar(StreamEncoder.java:242)
at sun.nio.cs.StreamEncoder.flushLeftoverChar(StreamEncoder.java:247)
at sun.nio.cs.StreamEncoder.flushLeftoverChar(StreamEncoder.java:247)
[...]

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.io.IOException;
import java.io.Writer;
import java.nio.channels.Channels;
import java.nio.channels.FileChannel;
import java.nio.channels.WritableByteChannel;
import java.nio.charset.Charset;
import java.nio.charset.CharsetEncoder;
import java.nio.charset.CodingErrorAction;
import java.nio.file.Files;
import java.util.Arrays;

public class EncoderTest
{
  public static void main(String[] args) throws IOException
  {
    //Create dummy channel
    WritableByteChannel channel = FileChannel.open(Files.createTempFile(null, null));

    //Create an UTF-8 encoder which ignores malformed input
    Charset charset = Charset.forName("UTF-8");
    CharsetEncoder encoder = charset.newEncoder();
    encoder.onMalformedInput(CodingErrorAction.IGNORE);

    //Create a char array which is filled with high surrogates
    char[] charArray = new char[1024 * 1024];
    Arrays.fill(charArray, '\uD834');

    //Create encoder, encode a single char, encode remaining chars.
    Writer enc = Channels.newWriter(channel, encoder, 1);
    enc.write(charArray, 0, 1);
    enc.write(charArray, 0, charArray.length); //BOOM!
  }
}
---------- END SOURCE ----------