Some of our tests fail to find messages properly after running for a while. We use the ResourceBundle mechanism to look up messages in properties files. We have a lot of messages, so we have broken them up randomly among 50 properties
files. These files are in the package c8e.loc, and are named
m0_*.properties through m49*.properties.

The problem happens with several different tests. After they
have been running for a while, they start to get diffs that
look like this:

---
1349d1348
< Number of columns fetched=2
1353a1353
> XSAJ5.U : =2
---

What this means is that the code failed to get a ResourceBundle, and so it is just printing out the message key and its parameters.

The tests do not always fail, nor do they always fail the same
way. It is not always the same ResourceBundle that it fails to
find.

I have tried to reproduce this problem with a simple test program
that just gets the ResourceBundles over and over. So far, I have not been
able to get the test program to fail. I have tried running the program in limited memory, to exercise garbage collection.

I suspect it is related to garbage collection
because: 1) it acts like a cache is failing, 2) ResourceBundle in
JDK 1.3 uses a new type of cache with weak references, and
3) Cloudscape is a very large application that puts a lot of
stress on garbage collection.

I am attaching some of our code that loads the ResourceBundles.
Please look at the method getBundleForLocale() in MessageService.java.
FYI, there is a field called "finder" of type BundleFinder in
MessageService.java - the implementation of the BundleFinder
interface is BaseMonitor.java, which I am also attaching.

The obfuscated name of the MessageService class is c8e.g.d,
and the obfuscated name of BaseMonitor is c8e.ad.g.

The ResourceBundles are used for localized messages. Cloudscape has
so many messages that we broke up the messages into 50 different
bundles. You can see these in cloudscape.jar and cloudsync.jar -
in the package c8e.loc, there are 50 properties files for the English
locale. We have separate jar files for all the locales other than
English - for example, in cloudscapeLocale_de_DE.jar, there are
50 properties files for the German language and nationality.

Cloudscape uses a hash function on the message key to determine
which bundle to use for a message. For example, the following
message is in m0_en.properties in cloudscape.jar and cloudsync.jar:

42Y67=Schema ''{0}'' cannot be dropped.

When looking up the message with key 42Y67, Cloudscape runs the
String "42Y67" through a hash function, which returns 0. This
means Cloudscape will try to get the ResourceBundle c8e.loc.m0,
so in the U.S. English locale, it will read the properties
file c8e.loc.m0_en.properties.

There are some interesting things to note. First, this problem
only happens in jdk 1.3.0, and ResourceBundle was re-written
in this version. Second, the failures are intermittent, and
do not always happen to the same tests, or even in the same
places in the same tests. Third, my experiments have shown
that, once a VM fails to find a ResourceBundle, it will continue
to fail to find the same ResourceBundle. Fourth, I have tried
to reproduce the problem with a small test program, and
failed.

All of this leads me to suspect that there is a garbage
collection problem that affects the caching in ResourceBundle.
The caching is one of the things that was re-written in jdk 1.3.0.
I think it uses weak references,

java -fullversion returns:

java full version "1.3.0-C"

We have only been able to reproduce this problem on WinNT and not
Solaris.


fariba.alavi@eng 2000-09-14

I, Fariba recevied the following email from Informix which contains more information:

This is Jeff Lichtman at Informix. Mei asked whether we are getting MissingResourceBundle exceptions. I have verified that we are getting these exceptions, and I am including some stack traces in attachments.
I should mention that these stack traces are not from the same version of Cloudscape that you have. Your version is in the form of our released product,
with obfuscation and all classes in jar files, while my version
is from our internal development codeline, without obfuscation,
with classes in directories, and with sanity checking code compiled in.

I also made it print the free memory and total memory every time
it prints the MissingResourceBundle stack trace. The stack traces
that I am attaching are the first failure for each ResourceBundle
for each test. As I mentioned in mail to Mei, once we start getting
a MissingResourceBundle exception for a particular ResourceBundle,
we continue to get it for the same ResourceBundle for the life of
the VM. In other words, once the ResourceBundle is lost, it stays
lost.

In one test (the outerjoin test), two different ResourceBundles
eventually become inaccessible. I am attaching the first failure
for each ResourceBundle in different files.

I should also note again that this problem is intermittent. It
is fairly easy to reproduce, by running our runtimestatistics
test suite, but the problem doesn't happen in the same places
in different runs, nor does it happen to the same ResourceBundles.

I, Fariba have copied the stack traces that Jeff from Informix is referring to in home/faribaa/informix directory. The stack traces files are:

distinctFiltering.tmp
outerjoin2.tmp
zindexes.tmp
outerjoin.tmp
partdml.tmp
zindexesLevel1.tmp