-
Bug
-
Resolution: Won't Fix
-
P4
-
None
-
8, 9, 10
-
x86_64
-
generic
FULL PRODUCT VERSION :
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
ADDITIONAL OS VERSION INFORMATION :
Darwin MACC02PTNZ7G8WP 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64
A DESCRIPTION OF THE PROBLEM :
When using StreamSupport.stream(Spliterators.spliteratorUnknownSize(someLargeIterator, 0), true) the JVM crashes as it runs out of HEAP space. This is due to the spliterator buffering the iterated items. The buffer size grows and they items then take up the whole jvm HEAP.
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the following code with a small -Xmx20M or so.
Do a HEAP dump using visualvm and notice that the retained instances of Foo grows indefinitely. If you want you can set the sleep to 0ms to have the exception happen immediately.
In practice my Foo is much larger and my Xmx is also much larger.
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The buffer size would have an intelligent MAX, or allow us to pass a max. Thus allowing this code to run without eating the whole HEAP
ACTUAL -
HEAP runs out of space.
ERROR MESSAGES/STACK TRACES THAT OCCUR :
Exception in thread "main" java.lang.OutOfMemoryError
REPRODUCIBILITY :
This bug can be reproduced always.
---------- BEGIN SOURCE ----------
import java.util.Iterator;
import java.util.NoSuchElementException;
import java.util.Spliterators;
import java.util.stream.StreamSupport;
public class SpliteratorExample {
public static void main(String[] args) {
StreamSupport.stream(Spliterators.spliteratorUnknownSize(new LargeIterator(), 0), true)
.forEach(foo -> {
try {
// Pretend we are writing to something
Thread.sleep(2);
} catch (InterruptedException e) {
System.out.println(foo);
e.printStackTrace();
}
});
}
static class Foo {
int id;
String content;
public Foo(int id, String content) {
this.id = id;
this.content = content;
}
}
static class LargeIterator implements Iterator<Foo> {
private static final int TOTAL_FOOS = 2500000;
private int current = 0;
public boolean hasNext() {
return current < TOTAL_FOOS;
}
public Foo next() {
if (hasNext()) {
current++;
return new Foo(current, "some string " + current);
}
throw new NoSuchElementException();
}
}
}
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
Create my own implementation of spliterator that does not buffer the items.
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
ADDITIONAL OS VERSION INFORMATION :
Darwin MACC02PTNZ7G8WP 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64
A DESCRIPTION OF THE PROBLEM :
When using StreamSupport.stream(Spliterators.spliteratorUnknownSize(someLargeIterator, 0), true) the JVM crashes as it runs out of HEAP space. This is due to the spliterator buffering the iterated items. The buffer size grows and they items then take up the whole jvm HEAP.
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the following code with a small -Xmx20M or so.
Do a HEAP dump using visualvm and notice that the retained instances of Foo grows indefinitely. If you want you can set the sleep to 0ms to have the exception happen immediately.
In practice my Foo is much larger and my Xmx is also much larger.
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The buffer size would have an intelligent MAX, or allow us to pass a max. Thus allowing this code to run without eating the whole HEAP
ACTUAL -
HEAP runs out of space.
ERROR MESSAGES/STACK TRACES THAT OCCUR :
Exception in thread "main" java.lang.OutOfMemoryError
REPRODUCIBILITY :
This bug can be reproduced always.
---------- BEGIN SOURCE ----------
import java.util.Iterator;
import java.util.NoSuchElementException;
import java.util.Spliterators;
import java.util.stream.StreamSupport;
public class SpliteratorExample {
public static void main(String[] args) {
StreamSupport.stream(Spliterators.spliteratorUnknownSize(new LargeIterator(), 0), true)
.forEach(foo -> {
try {
// Pretend we are writing to something
Thread.sleep(2);
} catch (InterruptedException e) {
System.out.println(foo);
e.printStackTrace();
}
});
}
static class Foo {
int id;
String content;
public Foo(int id, String content) {
this.id = id;
this.content = content;
}
}
static class LargeIterator implements Iterator<Foo> {
private static final int TOTAL_FOOS = 2500000;
private int current = 0;
public boolean hasNext() {
return current < TOTAL_FOOS;
}
public Foo next() {
if (hasNext()) {
current++;
return new Foo(current, "some string " + current);
}
throw new NoSuchElementException();
}
}
}
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
Create my own implementation of spliterator that does not buffer the items.