ADDITIONAL SYSTEM INFORMATION :
Windows 11 21H2 x86_64
Ubuntu 22.10 x86_64

openjdk version "20-ea" 2023-03-21
OpenJDK Runtime Environment (build 20-ea+24-1795)
OpenJDK 64-Bit Server VM (build 20-ea+24-1795, mixed mode, sharing)

openjdk version "19.0.1" 2022-10-18
OpenJDK Runtime Environment (build 19.0.1+10-21)
OpenJDK 64-Bit Server VM (build 19.0.1+10-21, mixed mode, sharing)


A DESCRIPTION OF THE PROBLEM :
Virtual threads fail with NoClassDefFoundError instead of StackOverflowError with methods that catch execptions.

When the stack of a virtual thread overflows and the methods called catch exceptions, the overflow is not handled by throwing a StackOverflowError. Instead, a NoClassDefFoundError with differing classes is thrown.

On Windows, this warning is also printed multiple times:

OpenJDK 64-Bit Server VM warning: Potentially dangerous stack overflow in ReservedStackAccess annotated method java.util.concurrent.locks.ReentrantLock$Sync.lock()V [1]

In the test case, the System.exit call on the virtual thread also does not work after the overflow on Windows.


When calling methods without exception handler a StackOverflowError is thrown as expected.


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Compile attach sample with --enable-preview and run.

The sample can also be started with "nohandler" as program argument. Then it works as expected.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The process exits with 0 and prints "Expected StackOverflowError".
ACTUAL -
The process exits with 1 and prints "UNEXPECTED EXCEPTION: java.lang.NoClassDefFoundError". On Windows, hotspot warnings are printed.

---------- BEGIN SOURCE ----------
public class VThreadExceptionHandlerOverflow {
    public static void main(String[] args) throws InterruptedException {
        if (args.length == 1 && args[0].equals("nohandler")) {
            System.err.println("Testing without exception handler");
            startAndCheck(VThreadExceptionHandlerOverflow::noHandler1);
        } else {
            System.err.println("Testing with exception handler");
            startAndCheck(VThreadExceptionHandlerOverflow::exceptionHandler1);
        }
        System.err.println("Virtual thread was terminated unexpectedly or System.exit did not work in uncaughtExceptionHandler.");
        System.exit(1);
    }

    public static void startAndCheck(Runnable runnable) throws InterruptedException {
        Thread.Builder.OfVirtual builder = Thread.ofVirtual().uncaughtExceptionHandler((t, e) -> {
            System.err.println("UNEXPECTED EXCEPTION: " + e.toString());
            System.exit(1);
        });
        builder.start(() -> {
            try {
                runnable.run();
            } catch (StackOverflowError t) {
                System.err.println("Expected StackOverflowError");
                System.exit(0);
                return;
            }
            throw new RuntimeException("should not reach");
        }).join();
    }

    public static void exceptionHandler1() {
        try {
            exceptionHandler2();
        } catch (Throwable t) {
            t.printStackTrace();
        }
    }

    private static void exceptionHandler2() {
        try {
            exceptionHandler1();
        } catch (Throwable t) {
            t.printStackTrace();
        }
    }

    public static void noHandler1() {
        noHandler2();
    }

    private static void noHandler2() {
        noHandler1();
    }
}

---------- END SOURCE ----------

FREQUENCY : always