Issue | Fix Version | Assignee | Priority | Status | Resolution | Resolved In Build |
---|---|---|---|---|---|---|
JDK-8084505 | emb-9 | Unassigned | P3 | Resolved | Fixed | team |
JDK-8086800 | 8u65 | Unassigned | P3 | Resolved | Fixed | b01 |
JDK-8073323 | 8u60 | Lev Priima | P3 | Resolved | Fixed | b06 |
JDK-8138221 | emb-8u65 | Unassigned | P3 | Resolved | Fixed | b01 |
JDK-8076877 | emb-8u60 | Lev Priima | P3 | Resolved | Fixed | team |
JDK-8074033 | 7u85 | Lev Priima | P3 | Resolved | Fixed | b02 |
java version "1.7.0_75"
OpenJDK Runtime Environment (IcedTea 2.5.4) (7u75-2.5.4-1~precise1)
OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)
ADDITIONAL OS VERSION INFORMATION :
Every OS, since this is portable Java code.
A DESCRIPTION OF THE PROBLEM :
As previously reported in bug 8011944, the function mergeCollapse in the TimSort class does not preserve the invariant runLen[i] > runLen[i+1]+runLen[i+2]. This leads to an ArrayIndexOutOfBoundsException in TimSort's pushRun method. We refer to 8011944 for more details.
The bug was marked as fixed, after an update in which the allocated length of runLen, as declared in the constructor, was increased. However, we found that the fix does not suffice, see the attached test case.
In particular, the declared length 40 of runLen which is chosen if the length of the input array is >= 119151, does not suffice.
We provide a full analysis of the bug in a paper, which is available at:
http://envisage-project.eu/?page_id=1412
REGRESSION. Last worked in version 6u43
ADDITIONAL REGRESSION INFORMATION:
Last worked in version 6u31, according to bug report 8011944
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the attached test case as follows:
java TestTimSort 67108864 (<=changed from paper to correct value)
or
java -Xmx8G TestTimSort 1073741824
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
No Exception occurs during sorting.
ACTUAL -
ArrayIndexOutOfBoundsException
ERROR MESSAGES/STACK TRACES THAT OCCUR :
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 40
at java.util.TimSort.pushRun(TimSort.java:386)
at java.util.TimSort.sort(TimSort.java:213)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at TestTimSort.main(TestTimSort.java:18)
REPRODUCIBILITY :
This bug can be reproduced always.
---------- BEGIN SOURCE ----------
import java.util.*;
public class TestTimSort {
private static final int MIN_MERGE = 32;
private static final Comparator<Object> NATURAL_ORDER =
new Comparator<Object>() {
@SuppressWarnings("unchecked")
public int compare(Object first, Object second) {
return ((Comparable<Object>)first).compareTo(second);
}
};
private final int minRun;
private final List<Long> runs = new ArrayList<Long>();
public static void main(String[] args) {
int length = Integer.parseInt(args[0]);
Arrays.sort(JDKWorstCase(length), NATURAL_ORDER);
}
private TestTimSort(int len) {
minRun = minRunLength(len);
}
private static int minRunLength(int n) {
assert n >= 0;
int r = 0; // Becomes 1 if any 1 bits are shifted off
while (n >= MIN_MERGE) {
r |= (n & 1);
n >>= 1;
}
return n + r;
}
/**
* Adds a sequence x_1, ..., x_n of run lengths to <code>runs</code> such that:<br>
* 1. X = x_1 + ... + x_n <br>
* 2. x_j >= minRun for all j <br>
* 3. x_1 + ... + x_{j-2} < x_j < x_1 + ... + x_{j-1} for all j <br>
* These conditions guarantee that TimSort merges all x_j's one by one
* (resulting in X) using only merges on the second-to-last element.
* @param X The sum of the sequence that should be added to runs.
*/
private void generateJDKWrongElem(long X) {
for(long newTotal; X >= 2*minRun+1; X = newTotal) {
//Default strategy
newTotal = X/2 + 1;
//Specialized strategies
if(3*minRun+3 <= X && X <= 4*minRun+1) {
// add x_1=MIN+1, x_2=MIN, x_3=X-newTotal to runs
newTotal = 2*minRun+1;
} else if(5*minRun+5 <= X && X <= 6*minRun+5) {
// add x_1=MIN+1, x_2=MIN, x_3=MIN+2, x_4=X-newTotal to runs
newTotal = 3*minRun+3;
} else if(8*minRun+9 <= X && X <= 10*minRun+9) {
// add x_1=MIN+1, x_2=MIN, x_3=MIN+2, x_4=2MIN+2, x_5=X-newTotal to runs
newTotal = 5*minRun+5;
} else if(13*minRun+15 <= X && X <= 16*minRun+17) {
// add x_1=MIN+1, x_2=MIN, x_3=MIN+2, x_4=2MIN+2, x_5=3MIN+4, x_6=X-newTotal to runs
newTotal = 8*minRun+9;
}
runs.add(0, X-newTotal);
}
runs.add(0, X);
}
/**
* Fills <code>runs</code> with a sequence of run lengths of the form<br>
* Y_n x_{n,1} x_{n,2} ... x_{n,l_n} <br>
* Y_{n-1} x_{n-1,1} x_{n-1,2} ... x_{n-1,l_{n-1}} <br>
* ... <br>
* Y_1 x_{1,1} x_{1,2} ... x_{1,l_1}<br>
* The Y_i's are chosen to satisfy the invariant throughout execution,
* but the x_{i,j}'s are merged (by <code>TimSort.mergeCollapse</code>)
* into an X_i that violates the invariant.
* @param X The sum of all run lengths that will be added to <code>runs</code>.
*/
private void runsJDKWorstCase(int length) {
long runningTotal = 0, Y=minRun+4, X=minRun;
while(runningTotal+Y+X <= length) {
runningTotal += X + Y;
generateJDKWrongElem(X);
runs.add(0,Y);
// X_{i+1} = Y_i + x_{i,1} + 1, since runs.get(1) = x_{i,1}
X = Y + runs.get(1) + 1;
// Y_{i+1} = X_{i+1} + Y_i + 1
Y += X + 1;
}
if(runningTotal + X <= length) {
runningTotal += X;
generateJDKWrongElem(X);
}
runs.add(length-runningTotal);
}
private Integer[] createArray(int length) {
Integer[] a = new Integer[length];
Arrays.fill(a, 0);
int endRun = -1;
for(long len : runs)
a[endRun+=len] = 1;
a[length-1]=0;
return a;
}
public static Integer[] JDKWorstCase(int length) {
TestTimSort t = new TestTimSort(length);
t.runsJDKWorstCase(length);
return t.createArray(length);
}
}
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
There are two options:
1. In the constructor of the TimSort class, change
int stackLen = (len < 120 ? 5 :
len < 1542 ? 10 :
len < 119151 ? 24 : 40);
to
int stackLen = (len < 120 ? 5 :
len < 1542 ? 10 :
len < 119151 ? 24 : 49);
2. Change the mergeCollapse method of TimSort to:
private void mergeCollapse() {
while (stackSize > 1) {
int n = stackSize - 2;
if ( n > 0 && runLen[n-1] <= runLen[n] + runLen[n+1]
|| n-1 > 0 && runLen[n-2] <= runLen[n] + runLen[n-1]) {
if (runLen[n - 1] < runLen[n + 1])
n--;
} else if (n<0 || runLen[n] > runLen[n + 1]) {
break; // Invariant is established
}
mergeAt(n);
}
}
- backported by
-
JDK-8073323 TimSort fails with ArrayIndexOutOfBoundsException on worst case long arrays
-
- Resolved
-
-
JDK-8074033 TimSort fails with ArrayIndexOutOfBoundsException on worst case long arrays
-
- Resolved
-
-
JDK-8076877 TimSort fails with ArrayIndexOutOfBoundsException on worst case long arrays
-
- Resolved
-
-
JDK-8084505 TimSort fails with ArrayIndexOutOfBoundsException on worst case long arrays
-
- Resolved
-
-
JDK-8086800 TimSort fails with ArrayIndexOutOfBoundsException on worst case long arrays
-
- Resolved
-
-
JDK-8138221 TimSort fails with ArrayIndexOutOfBoundsException on worst case long arrays
-
- Resolved
-
- blocks
-
JDK-8074149 (array) Review TimSort implementations
-
- Open
-
- relates to
-
JDK-8011944 Sort fails with ArrayIndexOutOfBoundsException
-
- Closed
-
-
JDK-8073959 java/util/Arrays/TimSortStackSize2.java fails with java.lang.OutOfMemoryError: Java heap space when running without CompressedOops
-
- Closed
-
-
JDK-8073124 Tune test and document TimSort runs stack size increase
-
- Closed
-