Details
-
Bug
-
Resolution: Fixed
-
P1
-
22
-
b19
-
Verified
Backports
Issue | Fix Version | Assignee | Priority | Status | Resolution | Resolved In Build |
---|---|---|---|---|---|---|
JDK-8330772 | 22.0.2 | Thomas Schatzl | P1 | Closed | Fixed | b04 |
Description
With JDK 22 we're starting to see various issues arising from jdk.internal.vm.FillerArray. It's not clear whether this is a duplicate of JDK-8319548 or not, so I'm filing this issue to track it.
The issue we see is, "java.lang.ClassCastException: class Ljdk.internal.vm.FillerArray; cannot be cast to class ...", which seems to appear in different execution paths, and with different targets of the cast, e.g. java.nio.ByteBuffer, java.util.Map, or even a non-JDK type org.elasticsearch.index.engine.LiveVersionMap$VersionLookup.
I originally thought that this was only occurring during heap dump, but I now think that it could occur at any time when the GC is under pressure?
Here are some snippets of stacktraces that we see:
java.lang.ClassCastException: class Ljdk.internal.vm.FillerArray; cannot be cast to class java.nio.ByteBuffer (Ljdk.internal.vm.FillerArray; and java.nio.ByteBuffer are in module java.base of loader 'bootstrap')
at io.netty.buffer@4.1.107.Final/io.netty.buffer.PoolChunk.allocate(PoolChunk.java:354)
at io.netty.buffer@4.1.107.Final/io.netty.buffer.PoolChunkList.allocate(PoolChunkList.java:108)
at io.netty.buffer@4.1.107.Final/io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:204)
...
In one particular case, I see hundreds of these, all appearing around the same time:
java.lang.ClassCastException: class Ljdk.internal.vm.FillerArray; cannot be cast to class org.elasticsearch.index.engine.LiveVersionMap$VersionLookup (Ljdk.internal.vm.FillerArray; is in module java.base of loader 'bootstrap'; org.elasticsearch.index.engine.LiveVersionMap$VersionLookup is in module org.elasticsearch.server@8.14.0 of loader 'app')
at org.elasticsearch.stateless@8.14.0/co.elastic.elasticsearch.stateless.engine.StatelessLiveVersionMapArchive.getRamBytesUsed(StatelessLiveVersionMapArchive.java:156)
at org.elasticsearch.server@8.14.0/org.elasticsearch.index.engine.LiveVersionMap.ramBytesUsedForRefresh(LiveVersionMap.java:483)
at org.elasticsearch.server@8.14.0/org.elasticsearch.index.engine.InternalEngine.getIndexBufferRAMBytesUsed(InternalEngine.java:2573)
at org.elasticsearch.server@8.14.0/org.elasticsearch.index.shard.IndexShard.getIndexBufferRAMBytesUsed(IndexShard.java:2355)
...
java.lang.IncompatibleClassChangeError: Class Ljdk.internal.vm.FillerArray; does not implement the requested interface java.util.Map
at org.apache.lucene.core@9.10.0/org.apache.lucene.index.ReadersAndUpdates.getNumDVUpdates(ReadersAndUpdates.java:168)
at org.apache.lucene.core@9.10.0/org.apache.lucene.index.ReaderPool.anyDocValuesChanges(ReaderPool.java:384)
at org.apache.lucene.core@9.10.0/org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:5776)
at org.apache.lucene.core@9.10.0/org.apache.lucene.index.StandardDirectoryReader.isCurrent(StandardDirectoryReader.java:455)
at org.apache.lucene.core@9.10.0/org.apache.lucene.index.FilterDirectoryReader.isCurrent(FilterDirectoryReader.java:133)
at org.elasticsearch.server@8.14.0/org.elasticsearch.index.engine.Engine.refreshNeeded(Engine.java:1093)
...
The issue we see is, "java.lang.ClassCastException: class Ljdk.internal.vm.FillerArray; cannot be cast to class ...", which seems to appear in different execution paths, and with different targets of the cast, e.g. java.nio.ByteBuffer, java.util.Map, or even a non-JDK type org.elasticsearch.index.engine.LiveVersionMap$VersionLookup.
I originally thought that this was only occurring during heap dump, but I now think that it could occur at any time when the GC is under pressure?
Here are some snippets of stacktraces that we see:
java.lang.ClassCastException: class Ljdk.internal.vm.FillerArray; cannot be cast to class java.nio.ByteBuffer (Ljdk.internal.vm.FillerArray; and java.nio.ByteBuffer are in module java.base of loader 'bootstrap')
at io.netty.buffer@4.1.107.Final/io.netty.buffer.PoolChunk.allocate(PoolChunk.java:354)
at io.netty.buffer@4.1.107.Final/io.netty.buffer.PoolChunkList.allocate(PoolChunkList.java:108)
at io.netty.buffer@4.1.107.Final/io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:204)
...
In one particular case, I see hundreds of these, all appearing around the same time:
java.lang.ClassCastException: class Ljdk.internal.vm.FillerArray; cannot be cast to class org.elasticsearch.index.engine.LiveVersionMap$VersionLookup (Ljdk.internal.vm.FillerArray; is in module java.base of loader 'bootstrap'; org.elasticsearch.index.engine.LiveVersionMap$VersionLookup is in module org.elasticsearch.server@8.14.0 of loader 'app')
at org.elasticsearch.stateless@8.14.0/co.elastic.elasticsearch.stateless.engine.StatelessLiveVersionMapArchive.getRamBytesUsed(StatelessLiveVersionMapArchive.java:156)
at org.elasticsearch.server@8.14.0/org.elasticsearch.index.engine.LiveVersionMap.ramBytesUsedForRefresh(LiveVersionMap.java:483)
at org.elasticsearch.server@8.14.0/org.elasticsearch.index.engine.InternalEngine.getIndexBufferRAMBytesUsed(InternalEngine.java:2573)
at org.elasticsearch.server@8.14.0/org.elasticsearch.index.shard.IndexShard.getIndexBufferRAMBytesUsed(IndexShard.java:2355)
...
java.lang.IncompatibleClassChangeError: Class Ljdk.internal.vm.FillerArray; does not implement the requested interface java.util.Map
at org.apache.lucene.core@9.10.0/org.apache.lucene.index.ReadersAndUpdates.getNumDVUpdates(ReadersAndUpdates.java:168)
at org.apache.lucene.core@9.10.0/org.apache.lucene.index.ReaderPool.anyDocValuesChanges(ReaderPool.java:384)
at org.apache.lucene.core@9.10.0/org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:5776)
at org.apache.lucene.core@9.10.0/org.apache.lucene.index.StandardDirectoryReader.isCurrent(StandardDirectoryReader.java:455)
at org.apache.lucene.core@9.10.0/org.apache.lucene.index.FilterDirectoryReader.isCurrent(FilterDirectoryReader.java:133)
at org.elasticsearch.server@8.14.0/org.elasticsearch.index.engine.Engine.refreshNeeded(Engine.java:1093)
...
Attachments
Issue Links
- backported by
-
JDK-8330772 G1 does not update TAMS correctly when dropping retained regions during Concurrent Start pause
- Closed
- relates to
-
JDK-8330080 G1: Investigate refactoring aged out retained region dropping logic
- Open
- links to
-
Commit openjdk/jdk22u/ac9ca971
-
Commit openjdk/jdk/ff5c9a4d
-
Review openjdk/jdk22u/145
-
Review openjdk/jdk/18692
(1 links to)