non-deterministic canonicalization cache behavior under ${java.home}

[dep, 13Jul2011]

  We recently had a problem where the Solaris-packaged java help
  (delivered as an extension) didn't work when used with a strict
  security policy... but only on some machines. On machines that
  worked, java would load the java help jar from

    /usr/jdk/instances/jdk1.6.0/jre/lib/ext/jhall.jar

  On machines that did not, it would load the jar from

    /usr/jdk/packages/javax.help-2.0/lib/jhall.jar

  The former is a symlink to the latter. For correct application of
  the security policy, it is necessary for the extension to load from
  the extension directory, not from /usr/jdk/packages. A similar
  problem is discussed in (4141872).

  The problem here is slightly different. On these two machines, we
  are getting different results when Java canonicalizes the symlink.
  On one machine, we get the symlink, on the other, we get the target.
  The problem boils down to the implementation of the canonicalization
  cache, as illustrated by a test case that does nothing but call
  getCanonicalFile() on the above symlink:

    # On the functioning machine:

    $ java -Dsun.io.useCanonPrefixCache=true Test
    /usr/jdk/instances/jdk1.6.0/jre/lib/ext/jhall.jar
    $ java -Dsun.io.useCanonPrefixCache=false Test
    /usr/jdk/packages/javax.help-2.0/lib/jhall.jar

    # On the malfunctioning machine:

    $ java -Dsun.io.useCanonPrefixCache=true Test
    /usr/jdk/packages/javax.help-2.0/lib/jhall.jar
    $ java -Dsun.io.useCanonPrefixCache=false Test
    /usr/jdk/packages/javax.help-2.0/lib/jhall.jar

  It turns out that the two machines -- despite having identical java
  installations -- had one subtle difference in the installed bits: the
  raw contents of the /usr/jdk/instances/jdk1.6.0/jre/lib/ext directory
  are ordered differently (observable using "ls -1U"). This is
  important because when constructing the URLClassLoader for
  java.ext.dirs, sun.misc.Launcher just calls list() on each directory
  to enumerate the jars within.

  The canonicalization cache, meanwhile, will cache path prefixes found
  under ${java.home}, passing through all subsequent files found under
  a previously canonicalized directory. This means that the results
  from the canonicalization algorithm differ depending on the order
  lookups are performed. If the first file canonicalized in a
  directory is a real file, subsequently canonicalized symlinks will be
  returned unmodified -- i.e. as if they are real files. On the other
  hand, if the first file canonicalized in a directory is a symlink (or
  more precisely, all symlinks in a directory canonicalized before a
  regular file is canonicalized), it will be resolved to its target
  prior to canonicalization.

  To put all the pieces together, we were seeing failures on those
  machines where chance resulted in the jhall.jar symlink being the
  first .jar file in the extension directory. This resulted in an
  attempt to load the extension from outside the extension directory,
  which in turn was considered a violation of the security policy
  that was in place.


  It is unfortunate that we must rely on the canonicalization cache in
  the first place -- as discussed in 4141872, manually resolving
  symlinks before determining if something falls in your sandbox
  defeats one of the primary uses of symlinks in Unix. That said, if
  we must rely on the canonicalization cache, it needs to at least
  behave in a consistent fashion.