Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8329528

G1 does not update TAMS correctly when dropping retained regions during Concurrent Start pause

    XMLWordPrintable

Details

    • gc
    • b19
    • Verified

    Backports

      Description

        With JDK 22 we're starting to see various issues arising from jdk.internal.vm.FillerArray. It's not clear whether this is a duplicate of JDK-8319548 or not, so I'm filing this issue to track it.

        The issue we see is, "java.lang.ClassCastException: class Ljdk.internal.vm.FillerArray; cannot be cast to class ...", which seems to appear in different execution paths, and with different targets of the cast, e.g. java.nio.ByteBuffer, java.util.Map, or even a non-JDK type org.elasticsearch.index.engine.LiveVersionMap$VersionLookup.

        I originally thought that this was only occurring during heap dump, but I now think that it could occur at any time when the GC is under pressure?

        Here are some snippets of stacktraces that we see:

        java.lang.ClassCastException: class Ljdk.internal.vm.FillerArray; cannot be cast to class java.nio.ByteBuffer (Ljdk.internal.vm.FillerArray; and java.nio.ByteBuffer are in module java.base of loader 'bootstrap')
          at io.netty.buffer@4.1.107.Final/io.netty.buffer.PoolChunk.allocate(PoolChunk.java:354)
          at io.netty.buffer@4.1.107.Final/io.netty.buffer.PoolChunkList.allocate(PoolChunkList.java:108)
          at io.netty.buffer@4.1.107.Final/io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:204)
          ...

        In one particular case, I see hundreds of these, all appearing around the same time:

        java.lang.ClassCastException: class Ljdk.internal.vm.FillerArray; cannot be cast to class org.elasticsearch.index.engine.LiveVersionMap$VersionLookup (Ljdk.internal.vm.FillerArray; is in module java.base of loader 'bootstrap'; org.elasticsearch.index.engine.LiveVersionMap$VersionLookup is in module org.elasticsearch.server@8.14.0 of loader 'app')
          at org.elasticsearch.stateless@8.14.0/co.elastic.elasticsearch.stateless.engine.StatelessLiveVersionMapArchive.getRamBytesUsed(StatelessLiveVersionMapArchive.java:156)
          at org.elasticsearch.server@8.14.0/org.elasticsearch.index.engine.LiveVersionMap.ramBytesUsedForRefresh(LiveVersionMap.java:483)
          at org.elasticsearch.server@8.14.0/org.elasticsearch.index.engine.InternalEngine.getIndexBufferRAMBytesUsed(InternalEngine.java:2573)
          at org.elasticsearch.server@8.14.0/org.elasticsearch.index.shard.IndexShard.getIndexBufferRAMBytesUsed(IndexShard.java:2355)
          ...

        java.lang.IncompatibleClassChangeError: Class Ljdk.internal.vm.FillerArray; does not implement the requested interface java.util.Map
          at org.apache.lucene.core@9.10.0/org.apache.lucene.index.ReadersAndUpdates.getNumDVUpdates(ReadersAndUpdates.java:168)
          at org.apache.lucene.core@9.10.0/org.apache.lucene.index.ReaderPool.anyDocValuesChanges(ReaderPool.java:384)
          at org.apache.lucene.core@9.10.0/org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:5776)
          at org.apache.lucene.core@9.10.0/org.apache.lucene.index.StandardDirectoryReader.isCurrent(StandardDirectoryReader.java:455)
          at org.apache.lucene.core@9.10.0/org.apache.lucene.index.FilterDirectoryReader.isCurrent(FilterDirectoryReader.java:133)
          at org.elasticsearch.server@8.14.0/org.elasticsearch.index.engine.Engine.refreshNeeded(Engine.java:1093)
          ...

        Attachments

          1. gc.log
            10.12 MB
          2. gc-verifier.log
            5.83 MB
          3. gc-verifier2.log
            8.18 MB
          4. gc-verifier3-tail.log
            49.88 MB
          5. hs_err_pid1328.log
            506 kB
          6. hs_err_pid24410.log
            185 kB

          Issue Links

            Activity

              People

                tschatzl Thomas Schatzl
                chegar Chris Hegarty
                Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved: