Summary

Throw an IllegalStateException when attempting to create a large number of nested event loops.

Problem

A large number of nested event loops (say, more than 100) likely indicates that the application has entered a state of unchecked recursion. On macOS 15 more than 250 nested loops can lead to a crash due to an OS check on the number of nested CFRunLoopRun calls. On Windows more than 280 nested loops can lead to erratic behavior due to resource exhaustion.

Solution

Impose a limit on the number of nested event loop calls. Attempting to exceed this limit will result in an illegal state exception so the developer will get a Java stack trace to examine rather than an OS crash log. Make sure the limit is large enough to not interfere with normal operation.

--- a/modules/javafx.graphics/src/main/java/com/sun/javafx/application/PlatformImpl.java
+++ b/modules/javafx.graphics/src/main/java/com/sun/javafx/application/PlatformImpl.java
@@ -982,4 +982,8 @@ public class PlatformImpl {
         }
     }

+    /**
+     * The maximum number of nested event loops.
+     */
+    public static final int MAX_NESTED_EVENT_LOOPS = 200;
 }

If a new nested event loop would exceed the limit canStartNestedEventLoop returns false. When at the limit a call to enterNestedEventLoop will throw an exception. The exception text includes the maximum number of nested event loops.

--- a/modules/javafx.graphics/src/main/java/com/sun/javafx/tk/quantum/QuantumToolkit.java
+++ b/modules/javafx.graphics/src/main/java/com/sun/javafx/tk/quantum/QuantumToolkit.java
@@ -619,10 +619,17 @@ public final class QuantumToolkit extends Toolkit {
         return stage;
     }

+    private boolean maxNestedEventLoopsHit() {
+        if (eventLoopMap == null) {
+            return false;
+        }
+        return eventLoopMap.size() >= PlatformImpl.MAX_NESTED_EVENT_LOOPS;
+    }
+
     @Override public boolean canStartNestedEventLoop() {
         checkFxUserThread();

-        return inPulse == 0;
+        return inPulse == 0 && !maxNestedEventLoopsHit();
     }

     @Override public Object enterNestedEventLoop(Object key) {
@@ -633,7 +640,12 @@ public final class QuantumToolkit extends Toolkit {
         }

         if (!canStartNestedEventLoop()) {
-            throw new IllegalStateException("Cannot enter nested loop during animation or layout processing");
+            if (maxNestedEventLoopsHit()) {
+                throw new IllegalStateException("Exceeded limit on number of nested event loops (" +
+                    PlatformImpl.MAX_NESTED_EVENT_LOOPS + ")");
+            } else {
+                throw new IllegalStateException("Cannot enter nested loop during animation or layout processing");
+            }
         }

         if (eventLoopMap == null) {

Specification

The max number of nested event loops is not publicly documented.

Add a new exception to Platform.enterNestedEventLoop.

--- a/modules/javafx.graphics/src/main/java/javafx/application/Platform.java
+++ b/modules/javafx.graphics/src/main/java/javafx/application/Platform.java
@@ -302,6 +302,9 @@ public final class Platform {
      * @throws IllegalStateException if this method is called on a thread
      * other than the JavaFX Application Thread.
      *
+     * @throws IllegalStateException if this call would exceed the maximum
+     * number of nested event loops.
+     *
      * @return the value passed into the corresponding call to exitEventLoop
      *
      * @since 9

Update the documentation for Stage.showAndWait to also mention the exception.

--- a/modules/javafx.graphics/src/main/java/javafx/stage/Stage.java
+++ b/modules/javafx.graphics/src/main/java/javafx/stage/Stage.java
@@ -398,6 +398,8 @@ public class Stage extends Window {
      *     other than the JavaFX Application Thread.
      * @throws IllegalStateException if this method is called during
      *     animation or layout processing.
+     * @throws IllegalStateException if this call would exceed the maximum
+     *      number of nested event loops.
      * @throws IllegalStateException if this method is called on the
      *     primary stage.
      * @throws IllegalStateException if this stage is already showing.

This change would also affect the behavior of Platform.canStartNestedEventLoop; it will return 'false' if the next call to enterNestedEventLoop would exceed the nesting limit. The documentation for this call is generic and relies on the specification for enterNestedEventLoop for details.