Subject: tracking back code history for 6799919
From: "Daniel D. Daugherty" <[email-masked]>
To: Ron Durbin <[email-masked]>
Date: Fri, 15 Feb 2013 09:36:13 -0700

Ron,

Here are my rough notes on tracking back the code history behind the
race condition that is in:

6799919 Recursive calls to report_vm_out_of_memory are handled incorrectly

If at all possible, first you find something uniquely grep'able about
the code in question. In this case:

$ hg annotate src/share/vm/utilities/debug.cpp | grep first_time_here
    0:   bool first_time_here = Atomic::xchg(exiting, 
&_exiting_out_of_mem) != exiting;
    0:   if (first_time_here) {

the word 'first_time_here' is nice and uniquely grep'able. The "hg annotate"
cmd tags each line in the specified source file with the linear changeset
number that introduced or modified a line. You can see info about the
changeset like this:

$ hg log -r 0 src/share/vm/utilities/debug.cpp
changeset:   0:a61af66fc99e
tag:         jdk7-b24
user:        duke
date:        Sat Dec 01 00:00:00 2007 +0000
summary:     Initial load

So the code in question has been around since before we switched from
TeamWare to Mercurial. Fortunately, I keep a copy of the various old
code workspaces on frankenputer:

$ pwd
/work/shared/mirrors/src_clones/hotspot_express/12/main_baseline

The above path is for Main_Baseline from HSX-12 which is when the
HSX TeamWare workspaces were migrated to Mercurial repos.

$ sccs get -p -m src/share/vm/utilities/debug.cpp | grep first_time_here
1.185
1.168.1.1         bool first_time_here = Atomic::xchg(exiting, 
&_exiting_out_of_mem) != exiting;
1.168.1.1         if (first_time_here) {
936 lines
No id keywords (cm7)

The "sccs" cmd above probably brings back nightmares, but it is useful.
Here is the SCCS delta:

$ sp -r1.168.1.1 src/share/vm/utilities/debug.cpp
src/share/vm/utilities/SCCS/s.debug.cpp:

D 1.168.1.1 05/07/29 11:29:29 sbohne 339 337    00031/00000/00930
MRs:
COMMENTS:
5073464 improve error handling for create_itable_stub out of memory

Based on SCCS numbers above (00031/00000/00930), it looks like it is a
straight addition:

$ sccs sccsdiff -r1.168 -r1.168.1.1 src/share/vm/utilities/debug.cpp

------- debug.cpp -------
227a228,258
 > // Used by report_vm_out_of_memory to detect recursion.
 > static jint _exiting_out_of_mem = 0;
 >
 > // Just passing the flow to VMError to handle error
 > void report_vm_out_of_memory(const char* file_name, int line_no, 
size_t size, const char* message) {
 >   if (Debugging || assert_is_suppressed(file_name, line_no)) return;
 >
 >   // We try to gather additional information for the first out of memory
 >   // error only; gathering additional data might cause an allocation 
and a
 >   // recursive out_of_memory condition.
 >
 >   const jint exiting = 1;
 >   // If we succeed in changing the value, we're the first one in.
 >   bool first_time_here = Atomic::xchg(exiting, &_exiting_out_of_mem) 
!= exiting;
 >
 >   if (first_time_here) {
 >     Thread* thread = ThreadLocalStorage::get_thread_slow();
 >     VMError(thread, size, message, file_name, line_no).report_and_die();
 >   }
 >   vm_abort();
 > }
 >
 > void report_vm_out_of_memory_vararg(const char* file_name, int 
line_no, size_t size, const char* format, ...) {
 >   char buffer[256];
 >   va_list ap;
 >   va_start(ap, format);
 >   jio_vsnprintf(buffer, sizeof(buffer), format, ap);
 >   va_end(ap);
 >   report_vm_out_of_memory(file_name, line_no, size, buffer);
 > }
 >

Here is the bug that Steve B used:

5073464 Can we improve error handling for create_itable_stub allocation 
failure
     https://jbs.oracle.com/bugs/browse/JDK-5073464

Based on the BugTraq2 legacy data, this bug was fixed in JDK1.6.0-B47.

I found the summary putback message for JDK1.6.0-B47 in one of my email
folders:

<begin summary putback message>
Original workspace:     prt-web:/net/hyperion/space4/hotspot-bugs
Submitter:              sbohne
Archived data: 
/net/prt-data.east/archives/main/rt_baseline/2005/20050801170202.sbohne.hotspot-bugs/
Webrev: 
http://analemma.sfbay.sun.com/net/prt-data.east/archives/main/rt_baseline/2005/20050801170202.sbohne.hotspot-bugs/workspace/webrevs/webrev-2005.08.01/index.html

Fixed 5073464: Can we improve error handling for create_itable_stub 
allocation
Fixed 6302670: Eliminate Hotspot __fabsf dependency

5073464: Can we improve error handling for create_itable_stub allocation

Previously many CodeCache allocations would fatal() if no storage was
available.  Updated these sites to call vm_exit_out_of_memory() instead
so that the "Out of swap space?" warning will be displayed to encourage
user diagnosis.  Also updated vm_exit_out_of_memory and the vmError "out of
memory" path to display the file/line of the caller, like fatal() does
already.

6302670: Eliminate Hotspot __fabsf dependency

On Solaris 8 02/04 and later, usage of fabs() with a float operand will
cause a dependency on __fabsf to be generated.  Since this symbol
doesn't exist on earlier Solaris 8 this binary won't run there. This
fix eliminates usage of fabs(float) in Hotspot and adds a Makefile check
to ensure that __fabsf doesn't reappear as an undefined symbol.

Reviewed by: Alan Bateman, Paul Hohensee, Azeem Jiva, Ken Russell, Tom
Rodriguez, Vladimir Kozlov, Peter Kessler

Fix verified (y/n): y

Verification testing:
    5073464: Manually test crash handler by setting ReservedCodeCacheSize
             to simulate low-memory situation.
    6302670: Manual Solaris build tests.

Other testing:
    PRT
[end sbohne Mon Aug  1 20:28:14 2005 EDT]
<end summary putback message>

Unfortunately, those summary putback message chopped off the file
list. I did some more poking around on an old server in CA:

$ pwd
/net/jano2/export2/hotspot/ws/main/rt_baseline/Codemgr_wsdata

And I found the following entry in the 'history' file:

<begin history file entry>
Original workspace:     prt-web:/net/hyperion/space4/hotspot-bugs
Submitter:              sbohne
Archived data: 
/net/prt-data.east/archives/main/rt_baseline/2005/20050801170202.sbohne.hotspot-bugs/
Webrev: 
http://analemma.sfbay.sun.com/net/prt-data.east/archives/main/rt_baseline/2005/20050801170202.sbohne.hotspot-bugs/workspace/webrevs/webrev-2005.08.01/index.html

Fixed 5073464: Can we improve error handling for create_itable_stub 
allocation
Fixed 6302670: Eliminate Hotspot __fabsf dependency

5073464: Can we improve error handling for create_itable_stub allocation

Previously many CodeCache allocations would fatal() if no storage was
available.  Updated these sites to call vm_exit_out_of_memory() instead
so that the "Out of swap space?" warning will be displayed to encourage
user diagnosis.  Also updated vm_exit_out_of_memory and the vmError "out of
memory" path to display the file/line of the caller, like fatal() does
already.

6302670: Eliminate Hotspot __fabsf dependency

On Solaris 8 02/04 and later, usage of fabs() with a float operand will
cause a dependency on __fabsf to be generated.  Since this symbol
doesn't exist on earlier Solaris 8 this binary won't run there. This
fix eliminates usage of fabs(float) in Hotspot and adds a Makefile check
to ensure that __fabsf doesn't reappear as an undefined symbol.

Reviewed by: Alan Bateman, Paul Hohensee, Azeem Jiva, Ken Russell, Tom
Rodriguez, Vladimir Kozlov, Peter Kessler

Fix verified (y/n): y

Verification testing:
    5073464: Manually test crash handler by setting ReservedCodeCacheSize
             to simulate low-memory situation.
    6302670: Manual Solaris build tests.

Other testing:
    PRT
END COMMENT
update: build/solaris/makefiles/sparcWorks.make
update: src/share/vm/asm/assembler.cpp
update: src/share/vm/code/stubs.cpp
update: src/share/vm/code/vtableStubs.cpp
update: src/share/vm/gc_implementation/shared/gcUtil.cpp
update: src/share/vm/memory/allocation.inline.hpp
update: src/share/vm/memory/concurrentMarkSweepGeneration.cpp
update: src/share/vm/runtime/java.cpp
update: src/share/vm/runtime/java.hpp
update: src/share/vm/runtime/os.cpp
update: src/share/vm/runtime/sharedRuntimeTrans.cpp
update: src/share/vm/runtime/sharedRuntimeTrig.cpp
update: src/share/vm/runtime/stubRoutines.cpp
update: src/share/vm/utilities/debug.cpp
update: src/share/vm/utilities/debug.hpp
update: src/share/vm/utilities/globalDefinitions.hpp
update: src/share/vm/utilities/vmError.cpp
update: src/share/vm/utilities/vmError.hpp

Examined files: 3651

Contents Summary:
       18   update
     3633   no action (unchanged)
<end history file entry>

Because there are two bug fixes in this putback, it's not easy to
figure which files belong to which bugs. Looking in each files'
s-dot history, I found 5073464 mentioned in these:

src/share/vm/asm/SCCS/s.assembler.cpp
src/share/vm/code/SCCS/s.stubs.cpp
src/share/vm/code/SCCS/s.vtableStubs.cpp
src/share/vm/memory/SCCS/s.allocation.inline.hpp
src/share/vm/runtime/SCCS/s.java.cpp
src/share/vm/runtime/SCCS/s.java.hpp
src/share/vm/runtime/SCCS/s.stubRoutines.cpp
src/share/vm/utilities/SCCS/s.debug.cpp
src/share/vm/utilities/SCCS/s.debug.hpp
src/share/vm/utilities/SCCS/s.vmError.cpp
src/share/vm/utilities/SCCS/s.vmError.hpp

so it is looking like this bug was introduced as part of a fairly
big fix from Steve B. I took a quick look at vmError.cpp and it looks
like VMError::report_and_die() already handled the multi-threaded
case even back then. So it is not at all clear why Steve B. added
the first_time_here logic in report_vm_out_of_memory().

So that should do it for a history hunting example...

Dan