libfontconfig missing on AIX in many hs_err 'Dynamic libraries' lists and native stack incomplete

We looked into a number of libfontconfig related crashes on AIX.
After some investigation, it turned out that a shared home directory holding the font cache between different machines
(also big and little endian servers) did not work well on AIX with libfontconfig.
Setting XDG_CACHE_HOME to some local non-shared directory like
XDG_CACHE_HOME=/priv/localdir/fc-cache
made the crashes disappear.

In those crash cases observed, libfontconfig is loaded by dlopen from the jdk C codebase (Java_sun_font_FontConfigManager_getFontConfig),
and shortly afterwards the crash occured in libfontconfig.a::FcCompare .
However in most cases, libfontconfig was missing in the "Dynamic libraries:" list of the hs_err file. Additionally a lot of entries
(unknown module)::(unknown function)+?
showed up in the native stacktrace where symbols from libfontconfig should be.

Reason of the outdated lib and symbols is that the information about the loaded shared libs is currently cached on AIX and is outdated
in the described crash situation.
See LoadedLibraries::reload() in the HS codebase. A way to solve this would be to update the cache more often (e.g. after all or at least some important
dlopen calls from the JDK C codebase).


crash example with missing info:

#
# SIGBUS (0xa) at pc=0x090000000959f02c, pid=11993400, tid=5398
#


--------------- T H R E A D ---------------

Current thread (0x0000000117672190): JavaThread "MainThread" [_thread_in_native, id=5398, stack(0x0000000117680000,0x000000011788d888) (2102K)]

Stack: [0x0000000117680000,0x000000011788d888], sp=0x000000011788b850, free space=2094k
Native frame:
iar: 0x090000000959f02c (unknown module)::(unknown function)+?
lr: 0x09000000095a0b58 (unknown module)::(unknown function)+?
sp: 0x000000011788b850 (base - 0x2038)
rtoc: 0x09001000a086f0c8
|---stackaddr----| |----lrsave------|: <function name>
0x000000011788b930 - 0x09000000095a0b58 (unknown module)::(unknown function)+?
0x000000011788ba90 - 0x09000000095a1120 (unknown module)::(unknown function)+?
0x000000011788bb20 - 0x0900000017d7ecc0 libawt_headless.so::Java_sun_font_FontConfigManager_getFontConfig+0xa80 (C++ saves_cr saves_lr stores_bc gpr_saved:18 fixedparms:6 parmsonstk:1)
0x000000011788bd70 - 0x0a0001000741ef48 (unknown module)::(unknown function)+?
0x000000011788be90 - 0x0a000100074186a0 (unknown module)::(unknown function)+?
0x000000011788bf50 - 0x0a000100074186a0 (unknown module)::(unknown function)+?
0x000000011788c040 - 0x0a000100074182d8 (unknown module)::(unknown function)+?
0x000000011788c0f0 - 0x0a000100074182d8 (unknown module)::(unknown function)+?
0x000000011788c1b0 - 0x0a00010007418bb4 (unknown module)::(unknown function)+?
0x000000011788c240 - 0x0a000100074182d8 (unknown module)::(unknown function)+?
0x000000011788c2f0 - 0x0a000100074182d8 (unknown module)::(unknown function)+?
0x000000011788c380 - 0x0a000100074186a0 (unknown module)::(unknown function)+?
0x000000011788c410 - 0x0a000100074186a0 (unknown module)::(unknown function)+?
0x000000011788c4a0 - 0x0a000100074186a0 (unknown module)::(unknown function)+?
0x000000011788c540 - 0x0a000100074182d8 (unknown module)::(unknown function)+?
0x000000011788c5e0 - 0x0a000100074182d8 (unknown module)::(unknown function)+?
0x000000011788c680 - 0x0a000100074182d8 (unknown module)::(unknown function)+?
0x000000011788c750 - 0x0a000100074186a0 (unknown module)::(unknown function)+?
0x000000011788c7f0 - 0x0a000100074186a0 (unknown module)::(unknown function)+?
0x000000011788c890 - 0x0a000100074186a0 (unknown module)::(unknown function)+?
0x000000011788c920 - 0x0a000100074186a0 (unknown module)::(unknown function)+?
0x000000011788c9c0 - 0x0a000100074182d8 (unknown module)::(unknown function)+?
0x000000011788ca80 - 0x0a000100074182d8 (unknown module)::(unknown function)+?
0x000000011788cb30 - 0x0a000100074182d8 (unknown module)::(unknown function)+?
0x000000011788cbd0 - 0x0a00010007418bb4 (unknown module)::(unknown function)+?
0x000000011788cc80 - 0x0a000100074182d8 (unknown module)::(unknown function)+?
0x000000011788cd40 - 0x0a00010007418f80 (unknown module)::(unknown function)+?
0x000000011788ce00 - 0x0a000100074186a0 (unknown module)::(unknown function)+?
0x000000011788cea0 - 0x0a000100074108f4 (unknown module)::(unknown function)+?
0x000000011788d020 - 0x090000001807b340 libjvm.so::JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x378 (C++ uses_alloca saves_cr saves_lr stores_bc gpr_saved:8 fixedparms:4 parmsonstk:1)
0x000000011788d150 - 0x09000000181e5d90 libjvm.so::os::os_exception_wrapper(void (*)(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*), JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x3c (C++ uses_alloca saves_lr stores_bc gpr_saved:1 fixedparms:5 parmsonstk:1)
0x000000011788d1d0 - 0x0900000017ed15dc libjvm.so::JavaCalls::call_virtual(JavaValue*, Klass*, Symbol*, Symbol*, JavaCallArguments*, JavaThread*)+0x200 (C++ fp_present uses_alloca saves_lr stores_bc gpr_saved:5 fixedparms:6 parmsonstk:1)
0x000000011788d300 - 0x0900000017ed1348 libjvm.so::JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x78 (C++ uses_alloca saves_lr stores_bc gpr_saved:2 fixedparms:6 parmsonstk:1)
0x000000011788d400 - 0x0900000019033a90 libjvm.so::thread_entry(JavaThread*, JavaThread*)+0xe8 (C++ uses_alloca saves_lr stores_bc gpr_saved:3 fixedparms:2 parmsonstk:1)
0x000000011788d4e0 - 0x0900000017fbce70 libjvm.so::JavaThread::thread_main_inner()+0x1f8 (C++ uses_alloca saves_lr stores_bc gpr_saved:4 fixedparms:1 parmsonstk:1)
0x000000011788d5b0 - 0x0900000017fbabdc libjvm.so::JavaThread::run()+0x21c (C++ uses_alloca saves_lr stores_bc gpr_saved:5 fixedparms:1 parmsonstk:1)
0x000000011788d650 - 0x0900000017f16898 libjvm.so::Thread::call_run()+0x128 (C++ uses_alloca saves_lr stores_bc gpr_saved:3 fixedparms:1 parmsonstk:1)
0x000000011788d6e0 - 0x0900000017f15d94 libjvm.so::thread_native_entry(Thread*)+0x194 (C++ uses_alloca saves_lr stores_bc gpr_saved:9 fixedparms:1 parmsonstk:1)
0x000000011788d7a0 - 0x090000000056204c libpthreads.a::_pthread_body+0xec (C saves_lr stores_bc gpr_saved:1 fixedparms:1 )
0x000000011788d820 - 0x0000000000000000
*** end of backchain ***
-----------------------
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j sun.font.FontConfigManager.getFontConfig(Ljava/lang/String;Lsun/font/FontConfigManager$FontConfigInfo;[Lsun/font/FontConfigManager$FcCompFont;Z)V+0 java.desktop@22-internal
j sun.font.FontConfigManager.initFontConfigFonts(Z)V+178 java.desktop@22-internal
j sun.awt.FcFontManager.getDefaultPlatformFont()[Ljava/lang/String;+10 java.desktop@22-internal
j sun.font.SunFontManager$2.run()Ljava/lang/Void;+216 java.desktop@22-internal
j sun.font.SunFontManager$2.run()Ljava/lang/Object;+1 java.desktop@22-internal
j java.security.AccessController.executePrivileged(Ljava/security/PrivilegedAction;Ljava/security/AccessControlContext;Ljava/lang/Class;)Ljava/lang/Object;+29 java.base@22-internal
j java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;+5 java.base@22-internal
j sun.font.SunFontManager.<init>()V+208 java.desktop@22-internal
j sun.awt.FcFontManager.<init>()V+1 java.desktop@22-internal
j sun.awt.X11FontManager.<init>()V+1 java.desktop@22-internal
j sun.font.PlatformFontInfo.createFontManager()Lsun/font/FontManager;+4 java.desktop@22-internal
j sun.font.FontManagerFactory.getInstance()Lsun/font/FontManager;+21 java.desktop@22-internal
j sun.font.FontUtilities.getFontConfigFUIR(Ljava/lang/String;II)Ljavax/swing/plaf/FontUIResource;+12 java.desktop@22-internal
j javax.swing.plaf.nimbus.NimbusDefaults.<init>()V+78 java.desktop@22-internal
j javax.swing.plaf.nimbus.NimbusLookAndFeel.<init>()V+31 java.desktop@22-internal
j Test6741426.main([Ljava/lang/String;)V+4
j java.lang.invoke.LambdaForm$DMH+0x00000004000c1000.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;)V+10 java.base@22-internal
j java.lang.invoke.LambdaForm$MH+0x00000004000c3400.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+33 java.base@22-internal
j java.lang.invoke.LambdaForm$MH+0x00000004000c3800.invokeExact_MT(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+20 java.base@22-internal
j jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+55 java.base@22-internal
j jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+23 java.base@22-internal
j java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+102 java.base@22-internal
j com.sun.javatest.regtest.agent.MainWrapper$MainTask.run()V+134
j java.lang.Thread.runWith(Ljava/lang/Object;Ljava/lang/Runnable;)V+5 java.base@22-internal
j java.lang.Thread.run()V+19 java.base@22-internal
v ~StubRoutines::call_stub 0x0a000100074108f4


Dynamic libraries:
   text: 0x0000000100000000 - 0x0000000100017d56, data: 0x0000000110000ca8 - 0x000000011000a6d0 java
   text: 0x0900000000000d00 - 0x090000000048baa7, data: 0x09001000a0000500 - 0x09001000a01166b0 /usr/lib/libc.a(shr_64.o)
   text: 0x090000000048c9c0 - 0x09000000004b17e7, data: 0x09001000a01aa190 - 0x09001000a01b63c0 /usr/lib/libodm.a(shr_64.o)
   text: 0x09000000004b2220 - 0x09000000004b2c7e, data: 0x09001000a01177a8 - 0x09001000a0117930 /usr/lib/libcrypt.a(shr_64.o)
   text: 0x09000000004b3e80 - 0x09000000004dd27b, data: 0x09001000a01cf720 - 0x09001000a01d9784 /usr/lib/libcorcfg.a(shr_64.o)
   text: 0x090000000052e120 - 0x09000000005458bd, data: 0x09001000a01c0d50 - 0x09001000a01ced20 /usr/lib/libsrc.a(shr_64.o)
   text: 0x0900000000546fc0 - 0x090000000055ddc9, data: 0x09001000a01b7e58 - 0x09001000a01bf5f0 /usr/lib/libcfg.a(shr_64.o)
   text: 0x090000000055e000 - 0x09000000005a4fe1, data: 0x09001000a0118000 - 0x09001000a01a9194 /usr/lib/libpthreads.a(shr_xpg5_64.o)
   text: 0x09000000005d9b20 - 0x0900000000675d28, data: 0x09001000a024b940 - 0x09001000a026fc88 /usr/lib/libperfstat.a(shr_64.o)
   text: 0x0900000000676b00 - 0x090000000074796c, data: 0x09001000a0211058 - 0x09001000a024aa05 /usr/lib/liblvm.a(shr_64.o)
   text: 0x0900000000be5000 - 0x0900000000c04288, data: 0x09001000a02f4250 - 0x09001000a02f9a30 /usr/lib/security/LDAP64
   text: 0x0900000001010000 - 0x0900000001017ba6, data: 0x08001000a03e5100 - 0x08001000a03e58a0 /priv/jvmtests/output_openjdk22_dev_dbgU_rs6000_64/sapjvm_22/lib/libjimage.so
   text: 0x090000000108e700 - 0x09000000010dc07f, data: 0x09001000a02e6a38 - 0x09001000a02ef156 /usr/lib/libc++abi.a(libc++abi.so.1)
   text: 0x09000000010dd6e0 - 0x09000000010e8003, data: 0x09001000a02e3f96 - 0x09001000a02e5408 /usr/lib/libunwind.a(libunwind.so.1)
   text: 0x09000000011c1000 - 0x09000000011e92c6, data: 0x08001000a03e6504 - 0x08001000a03e9228 /priv/jvmtests/output_openjdk22_dev_dbgU_rs6000_64/sapjvm_22/lib/libjava.so
   text: 0x0900000001207000 - 0x09000000012072fb, data: 0x09001000a02f3000 - 0x09001000a02f3000 /usr/lib/libdl.a(shr_64.o)
   text: 0x0900000001208000 - 0x090000000122104e, data: 0x08001000a03ecf77 - 0x08001000a03ee930 /priv/jvmtests/output_openjdk22_dev_dbgU_rs6000_64/sapjvm_22/lib/libnio.so
   text: 0x0900000001222000 - 0x09000000012318fc, data: 0x08001000a03eadd0 - 0x08001000a03eb7d0 /priv/jvmtests/output_openjdk22_dev_dbgU_rs6000_64/sapjvm_22/lib/libnet.so
   text: 0x0900000001232000 - 0x090000000124b5fe, data: 0x08001000a03efe14 - 0x08001000a03f0cb8 /priv/jvmtests/output_openjdk22_dev_dbgU_rs6000_64/sapjvm_22/lib/libzip.so
   text: 0x090000000429de58 - 0x09000000042c3534, data: 0x09001000a02f0054 - 0x09001000a02f0078 /usr/lib/libc++.a(shr2_64.o)
   text: 0x09000000097ec400 - 0x09000000099535fe, data: 0x09001000a0831f30 - 0x09001000a084eb70 /usr/lib/libc++.a(libc++.so.1)
   text: 0x0900000017c7a000 - 0x0900000017d79c63, data: 0x08001000a03f1ffc - 0x08001000a043f3b8 /priv/jvmtests/output_openjdk22_dev_dbgU_rs6000_64/sapjvm_22/lib/libawt.so
   text: 0x0900000017d7a000 - 0x0900000017d82aca, data: 0x08001000a0440038 - 0x08001000a04408e8 /priv/jvmtests/output_openjdk22_dev_dbgU_rs6000_64/sapjvm_22/lib/libawt_headless.so
 * text: 0x0900000017ddc000 - 0x09000000198710ac, data: 0x08001000a00003a8 - 0x08001000a03e4378 /priv/jvmtests/output_openjdk22_dev_dbgU_rs6000_64/sapjvm_22/lib/server/libjvm.so
   text: 0x090000001bbaa000 - 0x090000001bcf1515, data: 0x08001000a0441ef0 - 0x08001000a045f060 /priv/jvmtests/output_openjdk22_dev_dbgU_rs6000_64/sapjvm_22/lib/libfontmanager.so
   text: 0x090000001bcf2000 - 0x090000001bdbcac2, data: 0x09001000a0802430 - 0x09001000a080be78 /priv/jvmtests/output_openjdk22_dev_dbgU_rs6000_64/sapjvm_22/lib/libfreetype.so


A 'good" crash would look like this and show the libfontconfig.a::FcCompare and other functions in the hs_err , and also libfontconfig in the 'Dynamic libraries:' list :

#
# SIGBUS (0xa) at pc=0x090000000959f02c, pid=14287142, tid=6427
#
# JRE version: OpenJDK Runtime Environment (22.0) (fastdebug build 22-internal-adhoc.openjdk.jdk-dev)
# Java VM: OpenJDK 64-Bit Server VM (fastdebug 22-internal-adhoc.openjdk.jdk-dev, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, aix-ppc64)
# Problematic frame:
# C [libfontconfig.a+0x3b62c] FcCompare+0x8c
#

--------------- T H R E A D ---------------

Current thread (0x00000001183dcd30): JavaThread "AWT-EventQueue-1" [_thread_in_native, id=6427, stack(0x0000000118620000,0x000000011882d888) (2102K)]

Stack: [0x0000000118620000,0x000000011882d888], sp=0x000000011882b2f0, free space=2092k
Native frame:
iar: 0x090000000959f02c libfontconfig.a::FcCompare+0x8c (C fp_present saves_lr stores_bc gpr_saved:1 fixedparms:4 parmsonstk:1)
lr: 0x09000000095a0b58 libfontconfig.a::FcFontSetSort+0x318 (C fp_present saves_lr stores_bc gpr_saved:1 fixedparms:7 parmsonstk:1)
sp: 0x000000011882b2f0 (base - 0x2598)
rtoc: 0x09001000a084b0c8
|---stackaddr----| |----lrsave------|: <function name>
0x000000011882b3d0 - 0x09000000095a0b58 libfontconfig.a::FcFontSetSort+0x318 (C fp_present saves_lr stores_bc gpr_saved:1 fixedparms:7 parmsonstk:1)
0x000000011882b530 - 0x09000000095a1120 libfontconfig.a::FcFontSort+0x140 (C saves_lr stores_bc gpr_saved:1 fixedparms:5 parmsonstk:1)
0x000000011882b5c0 - 0x0900000017d7ecc0 libawt_headless.so::Java_sun_font_FontConfigManager_getFontConfig+0xa80 (C++ saves_cr saves_lr stores_bc gpr_saved:18 fixedparms:6 parmsonstk:1)
0x000000011882b810 - 0x0a0001000741ef48 (unknown module)::(unknown function)+?
0x000000011882b930 - 0x0a000100074186a0 (unknown module)::(unknown function)+?
  ...