[?] perhaps CDS should print a warning message in this situation?

$ javac EatHeap.java
$ jar cf EatHeap.jar EatHeap.class
$ rm -f EatHeap.jsa
$ java -Xmx64m -cp EatHeap.jar -XX:ArchiveClassesAtExit=EatHeap.jsa -Xlog:cds EatHeap 100
...
[0.378s][info][cds] Shared file region (rw) 0: 69760 bytes, addr 0x0000000800e93000 file offset 0x00001000 crc 0x739f095c
[0.378s][info][cds] Shared file region (ro) 1: 73352 bytes, addr 0x0000000800ea5000 file offset 0x00013000 crc 0x7de5efa7
[0.378s][info][cds] Shared file region (bm) 2: 2248 bytes, addr 0x0000000000000000 file offset 0x00025000 crc 0x119a2778
[0.378s][info][cds] Verify After CDS dynamic dump
$ ls -l EatHeap.jsa
-r--r--r-- 1 iklam iklam 155648 Apr 17 15:14 EatHeap.jsa

$ java -Xmx64m -cp EatHeap.jar -XX:ArchiveClassesAtExit=EatHeapOOM.jsa -Xlog:cds EatHeap
...
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"

(Notice there's no log related to CDS dumping)
$ ls -l EatHeapOOM.jsa
ls: cannot access 'EatHeapOOM.jsa': No such file or directory

===============
public class EatHeap {
    EatHeap next;
    byte[] buff;
    static EatHeap head;

    EatHeap() {
        // Completely fill the heap with small objects
        next = head;
        for (int n = 1024; n > 0; n = n / 2) {
            try {
                buff = new byte[n];
                return;
            } catch (Throwable t) {}
        }
    }
    public static void main(String args[]) throws Throwable {
        int n = args.length == 0 ? 0x7fffffff : Integer.parseInt(args[0]);
        while (n-- > 0) {
            head = new EatHeap();
        }
    }
}