hs_err files are currently divided into the sections "THREAD", "PROCESS" and "SYSTEM". The problem with this is that a lot of important information is hidden far down in the file. The VM Arguments are below a long list of Dynamic libraries, the OS version is printed in the last section, etc. Not only is it hard to get a quick overview of an hs_err file, but in many cases we won't even get the full hs_err file from customers or the webbugs system. It may have been truncated, or someone just copies parts of it.

Therefore, we should add a SUMMARY section at the top of the hs_err file. The SUMMARY section should mostly include one-liners, and should include everything needed to get a quick overview of the crash.

Suggested SUMMARY:

--------------- S U M M A R Y ---------------

time: Fri Oct 11 14:28:57 2013
elapsed time: 43 seconds

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=0 (SEGV0), si_addr=0x000002a50000624f

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.0-b48) for linux-amd64 JRE (1.8.0-ea-b106), built on Sep 4 2013 19:11:21 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

OS:Red Hat Enterprise Linux Server release 5.3 (Tikanga)

uname:Linux 2.6.18-128.el5 #1 SMP Wed Jan 21 08:45:05 EST 2009 x86_64
libc:glibc 2.5 NPTL 2.5
rlimit: STACK 10240k, CORE 0k, NPROC 38911, NOFILE 1024, AS infinity
load average:0.02 0.04 0.04

CPU:total 4 (2 cores per cpu, 1 threads per core) family 6 model 15 stepping 6, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, tsc
Memory: 4k page, physical 4047940k(639792k free), swap 4096564k(4096448k free)

VM Arguments: -Xmx32m
java_command: MyClass 1234
java_class_path (initial): .
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/localhome/java/jdk8
PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin
SHELL=/bin/bash
DISPLAY=localhost:10.0

VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None

Heap:
 PSYoungGen total 18944K, used 983K [0x00000000eb600000, 0x00000000ecb00000, 0x0000000100000000)
  eden space 16384K, 6% used [0x00000000eb600000,0x00000000eb6f5cc8,0x00000000ec600000)
  from space 2560K, 0% used [0x00000000ec880000,0x00000000ec880000,0x00000000ecb00000)
  to space 2560K, 0% used [0x00000000ec600000,0x00000000ec600000,0x00000000ec880000)
 ParOldGen total 41984K, used 0K [0x00000000c2200000, 0x00000000c4b00000, 0x00000000eb600000)
  object space 41984K, 0% used [0x00000000c2200000,0x00000000c2200000,0x00000000c4b00000)
 Metaspace total 4486K, used 2755K, reserved 132096K
  data space 4100K, used 2466K, reserved 1024K
  class space 386K, used 289K, reserved 131072K

---------------

This summary will give a lot of the information needed to quickly categorize a crash. Did it happen during start-up or after several days? Is it a supported platform? Did we run with any strange VM arguments? Is the heap close to full?

Note, that there are many other suggestions for information to be added to the hs_err file. Some of these should definitely go to the summary, like if we run on a hypervisor or not.

EDITED: There were some discussions about if we should duplicate the information in the summary. For example, the OS name should clearly be in the summary but it also makes sense to have it in the more detailed SYSTEM section. In that case, this information should be printed twice, but in many cases, just having the information in the SUMMARY section is enough. (Original suggestion was that everything should be both in the SUMMARY and in the detailed sections below).