ADDITIONAL SYSTEM INFORMATION :
This bug is generically present on all operating systems, but is mitigated by native file-locking mechanisms on Windows, so on Windows the actual effect cannot be reproduced, while on Linux it is 100 % reproducible with proper test setup.

I tested the behavior on Java 11 where I encountered the problem and validated that it is also still happening on Java 24.

A DESCRIPTION OF THE PROBLEM :
If you have at least 4 threads that try to lock the same region of the same file, and unlucky thread scheduling happens, it is possible to obtain an overlapping file lock which usually should be prevented.

Sketch of what happens:

*Thread 1*
- successfully acquires the lock
- adds list A with the lock to {{FileLockTable.lockMap}}

*Thread 2*
- fails to get the lock due to overlapping lock attempt recognized using {{FileLockTable.lockMap}}
- calls {{close}} on the {{Channel}}
- the channel close impl calls {{FileLockTable#removeAll}}
- {{FileLockTable#removeAll}} gets list A from the {{FileLockTable.lockMap}} in its second code line

*Thread 1*
- releases the lock, removing it from list A
- calls {{close}} on the {{Channel}} which removes list A from {{FileLockTable.lockMap}}

*Thread 3*
- successfully acquires the lock
- adds list B with the lock to {{FileLockTable.lockMap}}

*Thread 2*
- {{FileLockTable#removeAll}} continues execution and calls {{FileLockTable#removeKeyIfEmpty}}
- {{FileLockTable#removeKeyIfEmpty}} has an assert that checks that the current list for the file key is A, which would fail as the current list is B
- due to this failed assertion but A being empty, it now removes B with the valid used lock from Thread 3 from {{FileLockTable.lockMap}} which should not have happened

*Thread 4*
- successfully adds the {{FileLockImpl}} to the {{FileLockTable}} in new list C to {{FileLockTable.lockMap}} as there is no list stored anymore so the overlapping check passes
- {{FileChannelImpl#tryLock}} (for example) then calls {{nd.lock}} which calls the native {{lock0}} to get the file lock on the OS level
- Here Windows and Linux are different.
  On Windows the file is exclusively locked on the native filesystem and the {{lock0}} call returns {{-1}} which causes {{tryLock}} to return {{null}} which mitigates the problem on Windows.
  On Linux on the other hand, the {{lock0}} call is successful and so {{tryLock}} is returning an overlapping but seemingly "valid" lock object.

With the reproduction recipe below, this bug is 100% reproducible.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
- Put below code into a class file
- Add a breakpoint on the third line of {{sun.nio.ch.FileLockTable#removeAll}}, it is the line after {{list = lockMap.get(fileKey);}} but before {{synchronized (list) {}}, the line that does the {{null}}-check on {{list}}
- Make the breakpoint non-suspending
- Make the breakpoint conditional with condition {{Thread.currentThread().getName().equals("Thread 2")}}
- Add an "evaluate and log" statement to the breakpoint with code {{showcase.Test.syncPoint2.countDown(); showcase.Test.syncPoint4.await()}}
- Run the {{main}} method

On Windows you will see that first Thread 1 and then Thread 3 first increment, then decrement the counter and the FATAL line is not printed.
On Linux on the other hand you will see that Thread 1 increments, then decrements the counter, then Thread 3 increments the counter, then Thread 4 increments the counter, the FATAL line is logged, Thread 4 decrements the counter, and finally Thread 3 decrements the counter.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
No overlapping file lock is possible, but an {{OverlappingFileLockException}} being thrown on trying to do so.
ACTUAL -
An overlapping file lock is obtained.

---------- BEGIN SOURCE ----------
{code:java}
package showcase;

import java.io.IOException;
import java.nio.channels.FileChannel;
import java.nio.channels.FileLock;
import java.nio.channels.OverlappingFileLockException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicInteger;

import static java.nio.file.StandardOpenOption.WRITE;

public class Test {
    public static final CountDownLatch syncPoint1 = new CountDownLatch(1);
    public static final CountDownLatch syncPoint2 = new CountDownLatch(1);
    public static final CountDownLatch syncPoint3 = new CountDownLatch(1);
    public static final CountDownLatch syncPoint4 = new CountDownLatch(1);
    public static final CountDownLatch syncPoint5 = new CountDownLatch(1);
    public static final CountDownLatch syncPoint6 = new CountDownLatch(1);
    public static final CountDownLatch syncPoint7 = new CountDownLatch(1);

    public static void main(String[] args) throws IOException, InterruptedException {
        Path file = Files.createTempFile("", "");
        file.toFile().deleteOnExit();
        AtomicInteger count = new AtomicInteger(0);
        for (int i = 1; i <= 4; i++) {
            int finalI = i;
            new Thread("Thread " + i) {
                @Override
                public void run() {
                    try {
                        switch (finalI) {
                            case 2:
                                syncPoint1.await();
                                break;

                            case 3:
                                syncPoint1.await();
                                syncPoint2.await();
                                syncPoint3.await();
                                break;

                            case 4:
                                syncPoint1.await();
                                syncPoint2.await();
                                syncPoint3.await();
                                syncPoint4.await();
                                syncPoint5.await();
                                break;
                        }

                        FileChannel channel = FileChannel.open(file, WRITE);
                        boolean locked = false;
                        try {
                            FileLock lock = channel.tryLock();
                            if (lock != null) {
                                try {
                                    locked = true;
                                    System.out.println(Thread.currentThread() + ": increment counter");
                                    if (count.incrementAndGet() > 1) {
                                        System.out.println("FATAL: Acquired overlapping lock!");
                                    }
                                    switch (finalI) {
                                        case 1:
                                            syncPoint1.countDown();
                                            syncPoint2.await();
                                            break;

                                        case 3:
                                            syncPoint4.countDown();
                                            syncPoint6.await();
                                            break;
                                    }
                                } finally {
                                    lock.release();
                                }
                            }
                        } catch (OverlappingFileLockException e) {
                            // this is good, just end the thread
                        } finally {
                            channel.close();
                            if (locked) {
                                System.out.println(Thread.currentThread() + ": decrement counter");
                                count.decrementAndGet();
                            }
                            switch (finalI) {
                                case 1:
                                    syncPoint3.countDown();
                                    break;

                                case 2:
                                    syncPoint5.countDown();
                                    break;

                                case 3:
                                    syncPoint7.countDown();
                                    break;

                                case 4:
                                    syncPoint6.countDown();
                                    break;
                            }
                        }
                    } catch (Exception e) {
                        throw new RuntimeException(e);
                    }
                }
            }.start();
        }
        syncPoint7.await();
    }
}
{code}
---------- END SOURCE ----------