JNI call to getdtablesize() returns hard limit instead of soft limit.
A corresponding C program however behaves as expected and returns the
soft limit, as documented in the man page.

The problem is easily reproducible:

1. Testcase
-----------

 The testcase consists of the following files:

% ls -l
total 8
-rw-r--r-- 1 tl15687 sun 772 Oct 9 15:22 Makefile
-rw-r--r-- 1 tl15687 sun 404 Oct 9 12:49 test.java
-rw-r--r-- 1 tl15687 sun 244 Oct 9 13:07 testImpl.c
-rw-r--r-- 1 tl15687 sun 127 Oct 9 13:24 tst.c
%

 "tst.c" is the C program, which returns the soft limit as expected.
 Java class "test" makes use of "testImpl.c", which translates into
 "libtst.so". The JNI call returns the hard limit unexpectedly.


% more tst.c
#include <stdio.h>
#include <unistd.h>

int main() {

 int i = getdtablesize();
 printf("getdtablesize: %d\n", i);
 exit(0);
}
%

% more test.java

public class test
{
  public native int getdtablesize();

  static {
      System.loadLibrary("tst");
  }

  public static void main(String[] args) {

    int i;
    test tst = new test();

    i = 0;

    System.out.print("-------- Before JNI: ");
    System.out.println("i= " + i);

    i = tst.getdtablesize();

    System.out.print("-------- After JNI: ");
    System.out.println("i= " + i);

  }
}
%

% more testImpl.c
#include <stdio.h>
#include <jni.h>
#include <unistd.h>
#include "test.h"

JNIEXPORT jint JNICALL Java_test_getdtablesize (JNIEnv * env, jobject obj)
{
  jint ci;

  printf("JNI get_dtablesize .... \n");

  ci = getdtablesize();
  return ci;
}
%


% more Makefile

CC=/opt/SUNWspro/bin/CC
C=/opt/SUNWspro/bin/cc
JAVA=/j2sdk1_3_1_08
JAVA=/j2sdk1.4.1_05
JAVA=/j2sdk1.4.2_01

LIBRARIES = -lc

INCJAVA = $(JAVA)/include
INCJAVASOLARIS = $(JAVA)/include/solaris

CPPFLAGS =-I$(INCJAVA) -I$(INCJAVASOLARIS)


CCFLAGS = -V

#all: gen sign comp run
all: gen comp run

gen:
        $(JAVA)/bin/javac test.java
        $(JAVA)/bin/javah -jni test
comp:
        $(CC) -G $(CCFLAGS) $(CPPFLAGS) testImpl.c -o libtst.so $(LIBRARIES)
        $(C) $(CCFLAGS) tst.c -o tst
run:
        ./tst
        ulimit -n
        $(JAVA)/bin/java test
        ulimit -Hn
sign:
        $(JAVA)/bin/javap -s -p test > signature.txt

clean:
        rm -rf libtst.so tst *.class *.h signature.txt
%


2. Compile
----------
% make clean
rm -rf libtst.so tst *.class *.h signature.txt
% make gen
/j2sdk1.4.2_01/bin/javac test.java
/j2sdk1.4.2_01/bin/javah -jni test
% make comp
/opt/SUNWspro/bin/CC -G -V -I/j2sdk1.4.2_01/include -I/j2sdk1.4.2_01/include/solaris testImpl.c -o libtst.so -lc
CC: Sun WorkShop 6 update 2 C++ 5.3 Patch 111685-15 2003/05/14
ccfe: Sun WorkShop 6 update 2 C++ 5.3 Patch 111685-15 2003/05/14
CClink: Sun WorkShop 6 update 2 C++ 5.3 Patch 111685-15 2003/05/14
CC: Sun WorkShop 6 update 2 C++ 5.3 Patch 111685-15 2003/05/14
/opt/SUNWspro/bin/../WS6U2/bin/c++filt: Sun WorkShop 6 update 2 C++ 5.3 2001/05/15
ld: Software Generation Utilities - Solaris Link Editors: 5.9-1.343
/opt/SUNWspro/bin/cc -V tst.c -o tst
cc: Sun WorkShop 6 update 2 C 5.3 Patch 111679-12 2003/05/18
acomp: Sun WorkShop 6 update 2 C 5.3 Patch 111679-12 2003/05/18
ld: Software Generation Utilities - Solaris Link Editors: 5.9-1.343
%


3. Run
------
3.1 Run on Solaris 7
--------------------
% setenv LD_LIBRARY_PATH .:/usr/lib
% make run
./tst
getdtablesize: 64
ulimit -n
64
/j2sdk1.4.2_01/bin/java test
-------- Before JNI: i= 0
JNI get_dtablesize ....
-------- After JNI: i= 1024
ulimit -Hn
1024
%


3.2 Run on Solaris 8
--------------------
% setenv LD_LIBRARY_PATH .:/usr/lib
% make run
./tst
getdtablesize: 256
ulimit -n
256
/net/cores/tsc/Applix/java/JDK/j2sdk1.4.2_01/bin/java test
-------- Before JNI: i= 0
JNI get_dtablesize ....
-------- After JNI: i= 1024
ulimit -Hn
1024
%


3.3 Run on Solaris 9
--------------------
% setenv LD_LIBRARY_PATH .:/usr/lib
% make run
./tst
getdtablesize: 256
ulimit -n
256
/net/cores/tsc/Applix/java/JDK/j2sdk1.4.2_01/bin/java test
-------- Before JNI: i= 0
JNI get_dtablesize ....
-------- After JNI: i= 65536
ulimit -Hn
65536
%


4. Compare to man page
----------------------
% man getdtablesize
Reformatting page. Please Wait... done

Standard C Library Functions getdtablesize(3C)

NAME
     getdtablesize - get the file descriptor table size

SYNOPSIS
     #include <unistd.h>

     int getdtablesize(void);

DESCRIPTION
     The getdtablesize() function is equivalent to getrlimit(2)
     with the RLIMIT_NOFILE option.

RETURN VALUES
     The getdtablesize() function returns the current soft limit
     as if obtained from a call to getrlimit() with the
     RLIMIT_NOFILE option.
[ ... ]