Broken JIT compiler optimization for loop unswitching

FULL PRODUCT VERSION :
java version " 1.7.0_25 "
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)


FULL OS VERSION :
Linux xerosv 3.2.0-49-generic #75-Ubuntu SMP Tue Jun 18 17:39:32 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Linux peugeot 2.6.18-238.el5 #1 SMP Thu Jan 13 15:51:15 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
Linux bellatrix 2.6.32-71.el6.x86_64 #1 SMP Fri May 20 03:51:51 BST 2011 x86_64 x86_64 x86_64 GNU/Linux
Microsoft Windows [Version 6.1.7601]

A DESCRIPTION OF THE PROBLEM :
The JIT compiler optimization leads to a SIGSEGV or an NullPointerException at a place it must not happen.

The demo code works well while having the JIT disabled (-Djava.compiler=NONE or -Xint), but it starts crashing as soon as the JIT is enabled. In production we've a little different situation where the service doesn't crash, but it raises an NullPointerException at a place where it is normally impossible.

THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: No

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
You can use this project to reproduce it, it is not from us, but makes it easy to reproduce the issue:

https://github.com/rholder/jvm-loop-unswitching-bug

The same issue is as well reported at other places and that is what we've got in our production environment:

https://issues.apache.org/jira/browse/HTTPCLIENT-1173

EXPECTED VERSUS ACTUAL BEHAVIOR :
Expected that it neither crashs with a SIGV nor that throws a NoSuchElementException.
ERROR MESSAGES/STACK TRACES THAT OCCUR :
Two possible result, the above code produces a SIGV:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007ff799074b27, pid=41781, tid=140701181404928
#
# JRE version: 7.0_25-b15
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J com.nokia.cmp.middleware.LoopUnswitchBugTest$1.run()V
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try " ulimit -c unlimited " before starting Java again
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
#


We've installed disassembler and it crashes here:


  0x00007ff799074b18: mov %r10d,0x10(%rsp)
  0x00007ff799074b1d: mov %ebx,%r8d
  0x00007ff799074b20: inc %r8d ;*iadd
                                                ; - java.util.ArrayList$Itr::next@53 (line 798)
                                                ; - org.apache.http.impl.cookie.BestMatchSpec::formatCookies@39 (line 189)
                                                ; - com.nokia.cmp.middleware.LoopUnswitchBugTest$1::run@53 (line 46)
  0x00007ff799074b23: mov %r8d,0xc(%r13) ;*putfield cursor
                                                ; - java.util.ArrayList$Itr::next@54 (line 798)
                                                ; - org.apache.http.impl.cookie.BestMatchSpec::formatCookies@39 (line 189)
                                                ; - com.nokia.cmp.middleware.LoopUnswitchBugTest$1::run@53 (line 46)
  0x00007ff799074b27: mov 0x8(%r12,%r10,8),%r10d ; implicit exception: dispatches to 0x00007ff799074f95
  0x00007ff799074b2c: cmp $0xefc9a08f,%r10d ; {oop('org/apache/http/impl/cookie/BasicClientCookie')}
  0x00007ff799074b33: jne 0x00007ff799074f4b ;*checkcast
                                                ; - org.apache.http.impl.cookie.BestMatchSpec::formatCookies@44 (line 189)
                                                ; - com.nokia.cmp.middleware.LoopUnswitchBugTest$1::run@53 (line 46)
  0x00007ff799074b39: mov %r8d,0x10(%r13) ;*putfield lastRet
                                                ; - java.util.ArrayList$Itr::next@61 (line 799)
                                                ; - org.apache.http.impl.cookie.BestMatchSpec::formatCookies@39 (line 189)
                                                ; - com.nokia.cmp.middleware.LoopUnswitchBugTest$1::run@53 (line 46)



In our production environment it leads to a NoSuchElementException at a place where it must not happen:


NokiaSSOService: FAILED due to an unexpected exception, failture put into local cache
exception:
java.util.NoSuchElementException
     at java.util.ArrayList$Itr.next(ArrayList.java:794)
   at org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:189)
    at org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
      at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:108)
   at org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:174)
      at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:452)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:941)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:919)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:910)
        at com.nokia.idm.noa.http.ApacheHttpClient.call(ApacheHttpClient.java:59)
     at com.nokia.idm.noa.api.AuthenticateImpl.retrieve(AuthenticateImpl.java:210)
 at com.nokia.idm.noa.api.DefaultNoaApi.retrieve(DefaultNoaApi.java:139)
       at com.nokia.cmp.upm.authentication.NokiaSSOService.authenticate(NokiaSSOService.java:213)
    at com.nokia.cmp.upm.Manager$2.onCall(Manager.java:296)
       at com.nokia.cmp.aio.Async$Cursor.invoke(Async.java:786)
      at com.nokia.cmp.aio.Async$Cursor.access$900(Async.java:645)
  at com.nokia.cmp.aio.Async$Worker.executeCursor(Async.java:1659)
      at com.nokia.cmp.aio.Async$Worker.executeNextGlobalCursor(Async.java:1719)
    at com.nokia.cmp.aio.Async$Worker.run(Async.java:1935)


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
The source code that we used to produce the SEGV including the hotspot.log (full disassembled code) and more details can be downloaded from here:

http://middleware.tst.cmp.nokia.com/file/logs.tgz

In fact we just used the code from here:

https://github.com/rholder/jvm-loop-unswitching-bug

  To try to reproduce as we have rather a NoSuchElementException. However, as our code is too complex and large to simply reproduce the problem the above is much better suited. For us in production NoSuchElementException happens in this code,as reported as well here:

https://issues.apache.org/jira/browse/HTTPCLIENT-1173


    public List<Header> formatCookies(final List<Cookie> cookies) {
        if (cookies == null) {
            throw new IllegalArgumentException( " List of cookie may not be null " );
        }
        int version = Integer.MAX_VALUE;
        boolean isSetCookie2 = true;
        for (Cookie cookie: cookies) { // <---- this is line 189
            if (!(cookie instanceof SetCookie2)) {
                isSetCookie2 = false;
            }
            if (cookie.getVersion() < version) {
                version = cookie.getVersion();
            }
        }
        if (version > 0) {
            if (isSetCookie2) {
                return getStrict().formatCookies(cookies);
            } else {
                return getObsoleteStrict().formatCookies(cookies);
            }
        } else {
            return getCompat().formatCookies(cookies);
        }
    }


As you can see it must be impossible that at line 189 a NoSuchElementException is raised as its a compiled loop.
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
The only workaround that works for us is to disable JIT, which is terrible in a production environment.