We seem to run into native locks on Mac's AWT implementation, in combination with Swing and embedded JavaFX. Code which is working on Windows, and - as far as tested - on Linux, deadlocks on Mac. In these cases, a thread dump does not show a Java level deadlock, hence the idea about native deadlocks. This is especially the case with awt cursor methods. We have seen this also with drag-and-drop operations (failing on Mac only).

Here is a reproducible testcase: a jframe with button, button-click shows a jdialog with jfxpanel. The jfxpanel has a button to close the jdialog via invokeAndWait, using setVisible(false) and dispose. This works on Windows. On Mac, the application freezes when clicking the close button in the jdialog/jfxpanel. See comment below for the test code (attachment still don't seem to work). I am also adding comments with stackdumps.

About the "native deadlocks", here's some reasoning as to why we think that this might be the cause.

1. FX thread holds cocoa lock when dispatching.
2. FX thread calls SwingUtilities.invokeAndWait - waiting for EDT.
3. EDT tries to get cursor position and is blocked by cocoa lock.

static void perform_dispatch(JNIEnv *env, ... {
  JNF_COCOA_ENTER(env);
 ...

  JNF_COCOA_ENTER(blockEnv);
  ...
    // call the user's runnable
    JNFCallObjectMethod(blockEnv, [wrappedRunnable jObject], jm_run);

 ...
}

versus

JNIEXPORT jobject JNICALL
Java_sun_lwawt_macosx_CCursorManager_nativeGetCursorPosition(... {
  ...
  JNF_COCOA_ENTER(env);
  ...