FULL PRODUCT VERSION :


FULL OS VERSION :
Does not matter.
(Seen on Linux tombot.sirrix.de 3.2.0-53-generic #81-Ubuntu SMP Thu Aug 22 21:01:03 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux)

EXTRA RELEVANT SYSTEM CONFIGURATION :
The VM is configured to use 80% of the systems memory using -Xmx and -Xms. Kernel option vm.overcommit_ratio = 100, vm.overcommit_memory = 2

A DESCRIPTION OF THE PROBLEM :
When calling fork/exec the process is duplicated by fork and only then the new copy is replaces usding exec. That leads to problems, when the JVM uses most of the memory and only a small programm is to be started.

The problem was solved for Process.start in 2009 (see link below), but was apparently not solved for execution of the "-XX:OnOutOfMemoryError" handler.

-XX:OnOutOfMemoryError=reboot did not work, even though the log showed it would be started. Looking in the openJDK 7u40 sources, I found that this code still uses fork/exec instead of vfork, which was the solution for the Process.start case....

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7034935

THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Did not try

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Configure the vm to use most of the system memory.
Disable overcommit in linux.
Start the JRE using -XX:OnOutOfMemoryError=reboot
Allocate memory in a loop until an OnOutOfMemoryExecption occurs.


EXPECTED VERSUS ACTUAL BEHAVIOR :
Expected: Reboot of the server

Actual: The log shows a message about -XX:OnOutOfMemoryError=reboot and starting reboot, but it does not reboot.
REPRODUCIBILITY :
This bug can be reproduced always.