SA fails to properly discover system libraries for OSX core files

While working on JDK-8247515, which involved fixing issues that SA has with setting up symbol tables for OSX core files (and then doing lookups), I realized that the SA code that tries to discover all dylibs in the core file is broken.

The algorithm it uses is to look in each LC_SEGMENT_64 in the core file, checking the start of each to see if represents a dylib, which is indicated by MH_MAGIC_64 at the start. Once a mach-o file is found this way, it is searched for an LC_ID_DYLIB entry which provides the path to the dylib. Once one mach-0 file has been found in a segment, SA will then search the rest of the segment page by page looking for more mach-o files in the same segment.

This approach sort of works, and at least finds the JVM libs reliably. However, it has a few problem with system libs. The first is that some are in segments that don't start with a mach-o file, and therefore end up getting skipped. Changing the algorithm to search all segments fixed this problem, but is very slow as every page of the core file ends up getting paged in. It also seemed to find a few false positives (things that were not mach-o files) and also found dylibs that don't seem to be relevant. This was determined by looking at the output of lldb "image list", which showed about 239 libraries, yet the brute force SA approach found about 3x this number.

The other issue is that when there are multiple system libraries in the same segment, there appears to be no way to determine where each starts and ends within the segment. So SA just registers them all as having the same start address and same size (the address and size of the segment). This means when an address -> symbol lookup is done, SA just ends up using the first dylib it found in the segment, which chances are is the wrong one. The result of the lookup ends up being the very last symbol in the dylib, plus a very large offset.

Since JVM libs always seem to each be in there own segment and therefore are always found, I think we can continue with the approach for finding them. However, we need a better way for the system libs. For now as part of JDK-8247515 I am disabling the looking up of system libs. This is done by ignoring any LC_ID_DYLIB that is found that has a relative path. If you look for references to this CR in the macosx ps_core.c, you can see where the relevant changes were made.

As for how to properly fix this issue, I'm unsure. I did a lot of investigating into what is in the core file (just LC_THREAD and LC_SEGMENT_64 load commands) and what is in each LC_SEGMENT_64 (no sections, which was surprising). Basically I couldn't find anything that looked like it could be used to assemble a link map of sorts. But certainly it must be possible, as I was able to get lldb to provide a link map with the "image list" command. I therefore suggest looking at the lldb source for the answer.