I have verified that our dylibs can be successfully loaded using dlopen(..., RTLD_LAZY) and dlopen(...,RTLD_NOW) under OSX 10.11 even when SIP (System Integrity Protection) is enabled but sadly the same is not true for Java System.loadLibrary(). That seems to be broken when SIP is enabled.

With this test program:

$ cat loadLib.java
class loadLib
{
    public static final String libname = "ttJdbcCS";

    public static void main(String[] args)
    {
        System.loadLibrary( libname );
    }
}

Running it with DYLD_LIBRARY_PATH=/opt/TimesTen/tt1122/lib gives

$ java loadLib
Exception in thread "main" java.lang.UnsatisfiedLinkError: no ttJdbcCS in
java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1864)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at loadLib.main(loadLib.java:7)

And running it with an explicit java.library.path gives a different but
equally fatal error:

$ java -Djava.library.path=/opt/TimesTen/tt1122/lib loadLib
Exception in thread "main" java.lang.UnsatisfiedLinkError:
/opt/TimesTen/tt1122/lib/libttJdbcCS.dylib:
dlopen(/opt/TimesTen/tt1122/lib/libttJdbcCS.dylib, 1): Library not loaded:
/ade/vespa_112280_macos64/timesten/TTBuild/macos64_relopt/lib/libttclient.dyli
b
  Referenced from: /opt/TimesTen/tt1122/lib/libttJdbcCS.dylib
  Reason: image not found
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1938)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1854)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at loadLib.main(loadLib.java:7)

The problem is due to SIP 'stripping' the DYLD_* environment variables from
the environment of any process launched from a 'protected' location. These
include /bin, /sbin, /usr/bin, /usr/sbin and /System. Normally on OS X java
is invoked via /usr/bin/java which is a symlink to
/System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java
which is a universal binary that determines which java to invoke from the
available JVMs on the system.

A workaround for this is to either directly invoke Java from it's installed
location:

/Library/Java/JavaVirtualMachines/jdk1.x.y_z.jdk/Contents/Home/bin/java

or to setup a symlink - /usr/local/bin/java - pointing to that binary (the
default PATH in OS X has /usr/local/bin before /usr/bin).