Apache Lucene is still working on integrating project Panama for accessing memory mapped files.

We have an implementation for JDK 17 (4th incubator of Panama) now available which performs quite well for direct access to off-heap memory using VarHandles - you can get more details about the performance problems described here by looking at the benchmark results posted there: https://github.com/apache/lucene/pull/177

There is still the requirement to copy data from off-heap memory segments to standard Java arrays (and vice versa), so you can do normal processing on-heap with java language. With the vector APIs we will for sure go away with that more and more, but currently we often need to read data into byte[], float[] or long[] to process them with conventional code. Lucene's API to do this copying is looking like:

  private static void copySegmentToHeap(MemorySegment src, long srcOffset, byte[] target, int targetOffset, int len) {
    MemorySegment.ofArray(target).asSlice(targetOffset, len).copyFrom(src.asSlice(srcOffset, len));
  }

We analyzed the performance of this in our code (often it is called million of times with quite small byte arrays, but also with huge ones). To my knowledge of escape analysis the above code above should IMHO perfectly optimize and remove all allocation of HeapMemorySegment.OfByte instances, but it isn't!

The problem with this code is:
(1) it produces tons of heap allocations (we ran our benchmark with JFR enabled):

WARNING: Using incubator modules: jdk.incubator.foreign
PROFILE SUMMARY from 14415681 events (total: 5004408M)
  tests.profile.mode=heap
  tests.profile.count=30
  tests.profile.stacksize=1
  tests.profile.linenumbers=false
PERCENT HEAP SAMPLES STACK
55.11% 2757776M jdk.internal.foreign.HeapMemorySegmentImpl$OfByte#fromArray()
22.41% 1121632M jdk.internal.foreign.HeapMemorySegmentImpl$OfByte#dup()
20.85% 1043452M jdk.internal.foreign.MappedMemorySegmentImpl#dup()
0.71% 35428M jdk.internal.foreign.HeapMemorySegmentImpl$OfLong#dup()
0.18% 9206M jdk.internal.foreign.HeapMemorySegmentImpl$OfLong#fromArray()
0.12% 6150M org.apache.lucene.search.ExactPhraseMatcher$1$1#getImpacts()
0.09% 4729M org.apache.lucene.search.ExactPhraseMatcher$1#getImpacts()
0.09% 4620M org.apache.lucene.util.FixedBitSet#<init>()
0.06% 3099M java.util.AbstractList#iterator()
0.04% 1773M java.util.ArrayList#grow()
0.03% 1360M org.apache.lucene.codecs.lucene90.blocktree.SegmentTermsEnumFrame#<init>()
0.02% 1090M java.util.ArrayList#iterator()
0.02% 1085M org.apache.lucene.util.PriorityQueue#<init>()
0.02% 908M org.apache.lucene.util.ArrayUtil#growExact()
0.02% 796M org.apache.lucene.queryparser.charstream.FastCharStream#refill()
0.02% 787M org.apache.lucene.codecs.lucene90.Lucene90PostingsReader$BlockDocsEnum#<init>()
0.01% 577M org.apache.lucene.util.BytesRef#<init>()
0.01% 537M jdk.internal.misc.Unsafe#allocateUninitializedArray()
0.01% 517M org.apache.lucene.util.fst.ByteSequenceOutputs#read()
0.01% 394M org.apache.lucene.codecs.lucene90.blocktree.SegmentTermsEnum#getFrame()
0.01% 370M org.apache.lucene.codecs.lucene90.Lucene90PostingsReader#newTermState()
0.01% 366M org.apache.lucene.codecs.lucene90.Lucene90PostingsReader$BlockImpactsDocsEnum#<init>()
0.01% 352M org.apache.lucene.codecs.lucene90.blocktree.SegmentTermsEnum#<init>()
0.01% 349M org.apache.lucene.queryparser.charstream.FastCharStream#GetImage()
0.01% 322M org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnumFrame#load()
0.01% 317M org.apache.lucene.codecs.lucene90.ForUtil#<init>()
0.01% 295M org.apache.lucene.store.MemorySegmentIndexInput#buildSlice()
0.01% 261M org.apache.lucene.codecs.lucene90.Lucene90PostingsReader$BlockImpactsPostingsEnum#<init>()
0.00% 240M org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader$BlockState#document()
0.00% 237M java.util.Arrays#copyOfRange()

(2) the code is much slower than our old MappedByteBuffer code (which simply set position() on the byteBuffer and called the getBytes(byte[], into offset, int length) method)).

So it looks like there is some problem with escape analysis. It does not matter if we enable tiered compilation or not. The results are similar. With tiered compilation enabled, the garbage created seems to be smaller, but it still creates a lot of that. The JFR output of our old MappedByteBuffer#getBytes() code has the same heap statistics, except the first ones (HeapMemorySegment ) removed.

Nevertheless, we would like to have utility methods like the example above in MemoryAccess for all array types.

Utility methods could also be annotated with @ForecInline or similar, which cannot be done in our code.

In addition: There is a copy operation missing that takes the byte order and automatically swaps bytes. E.g., for copying from a MemorySegment to a long[] or float[] we have to first check that the endianness of platform fits the data in the file. Therefor it would be good to have a copy method with allows to specify the byte order of the source/target memory segment.

As mentioned about it is essential to have methods that take offset/length to prevent mess of wrapping. In short: Please add all combinations of System.arrayCopy variants for transferring slices from/to MemorySegments and on-heap arrays.

WORKAROUND: As workaround, I implemented the following hack:

  private static void copySegmentToHeap(MemorySegment src, long srcOffset, byte[] target, int targetOffset, int len) {
    Objects.checkFromIndexSize(srcOffset, len, src.byteSize());
    theUnsafe.copyMemory(null, src.address().toRawLongValue() + srcOffset,
        target, Unsafe.ARRAY_BYTE_BASE_OFFSET + targetOffset, len);
    //MemorySegment.ofArray(target).asSlice(targetOffset, len).copyFrom(src.asSlice(srcOffset, len));
  }

When using this code, the performance of our code is identical to the previous code. No object allocations! The JFR statistics on heap allocations look identical to our old code.

I also implemented another workaround that did a simple for-loop to copy instead of sun.misc.Unsafe. This worked well for small arrays, but was much slower for large arrays (because it does not seem to optimize to do 8-bytes-at-a-time copying).

Of course this code is not safe, as it does not use ScopedMemoryAccess, but it was enough for performance testing. I propose to add methods like the implementation I did (using ScopedMemoryAccess) to MemoryAccess class. And also add swapping as needed (to adapt byte order).

Thanks,
Uwe and Robert Muir (for the Lucene team)