Netbeans crash on 32 bit Solaris: need to call enable_extended_FILE_stdio() in VM initialisation

32 bit Solaris (SPARC and x86) ABI has a legacy limitation of the stdio routines to
file descriptors <= 255. The underlying problem is the size of the file descriptor
data type in a struct. This means that any linkable library that uses fopen() etc
may fail if the application uses up all these file descriptors before the library
is invoked. Solaris library providers have avoided these functions, but the same
is not true of 3rd party libraries, particularly those developed on systems which
do not have this limitation. The case in point is the Gnome desktop libraries
developed primarily for Linux but now used on Solaris. The consequence of this is
that applications which use these libraries on Solaris may fail and since the
mustang JDK needs to use these libraries in its desktop code, it can affect Java
applications. The most notable Java application that fails is Netbeans. Working
around this in Netbeans is next to impossible. Getting the open source libraries
fixed and propagated also appears problematic and there is no present understanding
of the number of libraries affected. At the very least fontconfig (a font configuration
library used by all GTK/KDE apps) and Pango (a text services library again used
by GTK and KDE) and cups (Unix printing) are affected. Other libraries may be
affected too.

Fortunately a viable solution is on the horizon and will be backported to S10 :
http://sac.sfbay.sun.com/Archives/CaseLog/arc/PSARC/2006/162/materials/DESIGN.txt
We would add the patch containing the backports to required patch lists.

There are several ways to use this, but it seems that most likely,
to take advantage of it in the JDK what we need to is call the new
enable_extended_FILE_stdio(3C) function. We would need to use dlsym() to access
this function at runtime. It needs to be invoked before the <=255 fds are exhausted
and needs to work in embedded VM environments too, so it appears that the correct
place to call this code is in the hotspot VM runtime initialisation.

There is a small potential for this new mechanism to cause problems in apps
which misuse the stdio APIs by directly de-referencing FILE->_file.
The most likely cause of this is apps which were compiled on Solaris 2.6 or
earlier, where the stdio.h header file implemented fileno() as a macro.
This risk has already been assessed by the Solaris PS-ARC as acceptable, in
part because it is an opt-in API. In the case of JDK we would 'opt-in' for any
Java app. But whereas we know we have a serious problem now, its not known if
the new API would cause new problems. So it would be prudent to
1) File a CCC
2) Provide a -XX VM option and envt variable to opt back out of this behaviour.