Neil Richards adds:

Digging back into its history, I see that it all stems from running java
under 'nohup' (e.g. 'nohup java ProgramWithShutdownHooks &').

'nohup' prevents things being registered against SIGHUP, so trying to do
so causes an exception to be thrown.

When this happens, the Terminator code currently decides that it
shouldn't bother trying to register shutdown hooks for SIGINT or SIGTERM
either.

The result is that shutdown hooks aren't currently run when a java
process run using 'nohup' is sent a SIGINT or SIGTERM signal, because of
the code's (false) assumption that a failure to register a handler for
SIGHUP must mean that the VM is running in -Xrs mode.

I don't think the current observed behaviour in this scenario is either
expected or desirable.
Frank Ding reports:

  I found that in java.lang.Terminator, setup() method,
The following code of registering default signal handlers can be improved:
/ try {
            Signal.handle(new Signal("INT"), sh);
            Signal.handle(new Signal("TERM"), sh);
        } catch (IllegalArgumentException e) {
        }/
The revised code is illustrated below:
/ try {
            Signal.handle(new Signal("INT"), sh);
        } catch (IllegalArgumentException e) {
        }
        try {
            Signal.handle(new Signal("TERM"), sh);
        } catch (IllegalArgumentException e) {
        }
/The improved version makes more sense since exception thrown from first Signal.handle call does not affect subsequent calls. This is more consistent with its original intention.
A patch I made is available @
http://cr.openjdk.java.net/~youdwei/ojdk-430/webrev.00//