After a fix for JDK-8005465, FX and AWT can both work in headful mode. A user app can start both toolkits in any order, and they all work fine. However, there's still an issue with the order of termination.

Consider the following test application (also, some more details at http://javafx-jira.kenai.com/browse/RT-20784):

import javafx.application.Application;
import javafx.event.ActionEvent;
import javafx.event.EventHandler;
import javafx.scene.Scene;
import javafx.scene.control.Button;
import javafx.scene.layout.StackPane;
import javafx.stage.Stage;
import javax.swing.JButton;
import javax.swing.JFrame;
import javax.swing.SwingUtilities;

public class test extends Application {

    @Override
        public void start(Stage primaryStage) {
            SwingUtilities.invokeLater(new Runnable() {
                @Override
                public void run() {
                    JFrame jFrame = new JFrame();
                    jFrame.setBounds(10, 10, 320, 240);
                    jFrame.getContentPane().add(new JButton("I am a Swing Button!"));
                    jFrame.setDefaultCloseOperation(JFrame.DISPOSE_ON_CLOSE);
                    jFrame.setVisible(true);
                }
            });
            Button btn = new Button();
            btn.setText("Say 'Hello World'");
            btn.setOnAction(new EventHandler<ActionEvent>() {
                @Override
                public void handle(ActionEvent event) {
                    System.out.println("Hello World!");
                }
            });

            StackPane root = new StackPane();
            root.getChildren().add(btn);

            Scene scene = new Scene(root, 300, 250);

            primaryStage.setTitle("Hello World!");
            primaryStage.setScene(scene);
            primaryStage.show();
        }

    public static void main(String[] args) throws Exception {
        launch(args);
    }
}

Here we initialize FX as the main (embedder) toolkit by calling launch() from main(). Later we create an AWT top-level window, and end up with both FX and AWT windows showing on the screen. AWT works in the embedded mode in this case. If we close the AWT window first, and then the FX window, then no problem occurs. However, if we revert the order and close the FX window first, the AWT will crash (full crash dump is attached):

Thread 0 Crashed:: AppKit Thread Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff90f4c212 __pthread_kill + 10
1 libsystem_c.dylib 0x00007fff875c6af4 pthread_kill + 90
2 libsystem_c.dylib 0x00007fff8760adce abort + 143
3 libjvm.dylib 0x00000001084c8dfd os::abort(bool) + 25
4 libjvm.dylib 0x00000001085cd506 VMError::report_and_die() + 2316
5 libjvm.dylib 0x00000001084ca68f JVM_handle_bsd_signal + 1073
6 libsystem_c.dylib 0x00007fff875b38ea _sigtramp + 26
7 JavaNativeFoundation 0x0000000129467b4f -[JNFJObjectWrapper jObjectWithEnv:] + 42
8 liblwawt.dylib 0x000000012d126ba3 -[CGLLayer drawInCGLContext:pixelFormat:forLayerTime:displayTime:] + 207
9 com.apple.QuartzCore 0x00007fff8b7c5ae6 CAOpenGLLayerDraw(CAOpenGLLayer*, double, CVTimeStamp const*, unsigned int) + 768
10 com.apple.QuartzCore 0x00007fff8b7c56d2 -[CAOpenGLLayer _display] + 397
11 com.apple.QuartzCore 0x00007fff8b6c7661 CA::Layer::display_if_needed(CA::Transaction*) + 593
12 com.apple.QuartzCore 0x00007fff8b6c6e7b CA::Layer::layout_and_display_if_needed(CA::Transaction*) + 35
13 com.apple.QuartzCore 0x00007fff8b6bc653 CA::Context::commit_transaction(CA::Transaction*) + 261
14 com.apple.QuartzCore 0x00007fff8b6bc423 CA::Transaction::commit() + 369
15 com.apple.QuartzCore 0x00007fff8b6bc23f CA::Transaction::observer_callback(__CFRunLoopObserver*, unsigned long, void*) + 63
16 com.apple.CoreFoundation 0x00007fff91d749b7 __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
17 com.apple.CoreFoundation 0x00007fff91d74921 __CFRunLoopDoObservers + 369
18 com.apple.CoreFoundation 0x00007fff91d4fe51 __CFRunLoopRun + 929
19 com.apple.CoreFoundation 0x00007fff91d4f6b2 CFRunLoopRunSpecific + 290
20 java 0x0000000107fbe402 CreateExecutionEnvironment + 871
21 java 0x0000000107fb8bac JLI_Launch + 1952
22 java 0x0000000107fbe75f main + 101
23 libdyld.dylib 0x00007fff890417e1 start + 1

This happens because after closing the FX window, FX decides to terminate the native event loop and detach the AppKit thread from the JVM. This is required because the AppKit thread is attached to JVM as a non-daemon thread when FX is the main toolkit. This is by design of the FX threading model. Now, the AWT code, still having an open top-level window, tries to render some content, and crashes since [CGLLayer drawInCGLContex] tries to call to Java from the AppKit thread which is already detached from the JVM.

This issue is similar to JDK-7166725 which we worked around previously by not setting busy observers when running in headless mode. However, currently we need to be able to run headful AWT, and hence a similar workaround for this problem isn't applicable anymore.

We plan to develop a mechanism allowing both toolkits to notify each other when it's OK to terminate in JDK 8. However, this crash is still an issue for 7u releases.