System.loadLibrary fails on Big Sur for libraries hidden from filesystem

ADDITIONAL SYSTEM INFORMATION :
MacBook Pro (15-inch, 2018)
MacOS Big Sur 11.6
OpenJDK 17, OpenJDK 11, Oracle JDK 17, Oracle JDK 18-ea

A DESCRIPTION OF THE PROBLEM :
OSX Big Sur no longer ships with copies of the libraries on the filesystem [1] and therefore attempts to load a native library via System.loadLibrary no longer works.

Details:
- The new dynamic linker cache introduced by OSX Big Sur creates problems with previous code which checks for the existence of a library file in the filesystem before attempting to load it with dlopen(...). According to the Big Sur release notes in [1],
   "Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache."

Tracing the execution of System.loadLibrary shows that the offending check happens at [2] (tested on OpenJDK 11 and 17, Oracle JDK 17 and 18-ea) where File.exists(...) is used to check whether the library file is present on the filesystem before attempting to load it. As a result, valid dynamic libraries that otherwise open fine via dlopen(...) fail with UnsatisfiedLinkError in current versions of the JDK.

Background:
- the way I discovered this issue was while attempting to get the OSX Accelerate LibVec BLAS implementation to load via Netlib in Spark; Netlib kept complaining that it can't find the native BLAS libraries. Inspecting the Netlib code [3] showed that it used System.loadLibrary to attempt to load the appropriately-configured BLAS library. From there, further tests on using System.loadLibrary via "jshell" showed the problem to be in its implementation. I have confirmed that the dlopen(...) call works with "hidden" libraries by creating a small C application that invoked dlopen and reported success or failure. Inspection of the JDK code for System.loadLibrary eventually led to the offending file existence check in [2]

[1] https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11_0_1-release-notes/#Kernel
[2] https://github.com/openjdk/jdk/blob/895e2bd7c0bded5283eca8792fbfb287bb75016b/src/java.base/share/classes/jdk/internal/loader/NativeLibraries.java#L163
[3] https://github.com/luhenry/netlib/blob/20ecbd98425ea7baae5ebd70d392c9eb206dfb26/blas/src/main/jdk17/dev/ludovic/netlib/blas/ForeignLinkerBLAS.java#L55

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
In a OSX Big Sur installation with XCode and developer tools installed install any Java version from the list in "Runtime information", then

capitanu@leo:~/Downloads/java/jdk-18.jdk/Contents/Home$ export JAVA_HOME=$(pwd)
capitanu@leo:~/Downloads/java/jdk-18.jdk/Contents/Home$ export LD_LIBRARY_PATH='/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A'
capitanu@leo:~/Downloads/java/jdk-18.jdk/Contents/Home$ export JAVA_LIBRARY_PATH='/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A'
capitanu@leo:~/Downloads/java/jdk-18.jdk/Contents/Home$ jshell
| Welcome to JShell -- Version 18-ea
| For an introduction type: /help intro

jshell> System.loadLibrary("blas")
| Exception java.lang.UnsatisfiedLinkError: no blas in java.library.path: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A:/Users/capitanu/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
| at ClassLoader.loadLibrary (ClassLoader.java:2429)
| at Runtime.loadLibrary0 (Runtime.java:818)
| at System.loadLibrary (System.java:1998)
| at (#1:1)

jshell> System.load("/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib")
| Exception java.lang.UnsatisfiedLinkError: Can't load library: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
| at ClassLoader.loadLibrary (ClassLoader.java:2393)
| at Runtime.load0 (Runtime.java:755)
| at System.load (System.java:1962)
| at (#2:1)

To prove that dlopen works with "hidden" libraries, create a file dlopen.c with contents:

#include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>

int main(int argc, char** argv)
{
    void *handle;

    if (argc != 2) {
        fprintf(stderr, "Usage: %s <lib_filename_or_full_path>\n", argv[0]);
        return EXIT_FAILURE;
    }

    printf("Attempting to load library '%s'...\n", argv[1]);

    handle = dlopen(argv[1], RTLD_LAZY);

    if (handle == NULL) {
fprintf(stderr, "Unable to load library!\n");
return EXIT_FAILURE;
    }

    printf("Library successfully loaded!\n");

    return dlclose(handle);
}

Then:
gcc -o dlopen dlopen.c
./dlopen libBLAS.dylib
and
./dlopen /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib



EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Expected for dynamic library to load correctly via System.loadLibrary and System.load even for "hidden" libraries that are not visible on the filesystem but can be successfully loaded via dlopen.

ACTUAL -
| Exception java.lang.UnsatisfiedLinkError: no blas in java.library.path: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A:/Users/capitanu/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
| at ClassLoader.loadLibrary (ClassLoader.java:2429)
| at Runtime.loadLibrary0 (Runtime.java:818)
| at System.loadLibrary (System.java:1998)
| at (#1:1)

---------- BEGIN SOURCE ----------
System.load("/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib")
---------- END SOURCE ----------

FREQUENCY : always