Modal dialogs shown from the event dispatching thread can deadlock if show() or setVisible(true) is called from a synchronized block or method.

Consider the following test case, which is minimal code modeled after sunw.hotjava.security.SecurityDialog to reproduce a problem originally reported in HotJava on JavaOS , (bug 4155998). While writing the test to reproduce the problem in JavaOS, I discovered that the same bug can be reproduced in JDK 1.2 and JDK 1.1.7.

It states in the JDK API:

"It is permissible to show modal dialogs from the event dispatching thread because the toolkit will ensure that another dispatching thread will run while the one which invoked show is blocked."

However, if a modal dialog is shown from within a synchronized block, and another modal dialog is created as a result of an event, the EventDispatchThread from the first dialog tried to dispatch the event and is blocked trying to enter the synchronized block. Since the EventDispatchThread is now blocked, the first modal dialog, which is blocked in show(), can't be dismissed because the button event to dismiss it can't be dispatched by the blocked EventDispatchThread. The end result is everything is locked up.

Here is a stack trace when the VM is deadlocked. You can see the EventDispatchThread "AWT-Dispatch-Proxy" is in Monitor Wait trying to enter the synchronized block, and the first modal blocked in show() waiting to be notified:

castlerock% java -version
java version "1.2fcs"
Classic VM (build JDK-1.2fcs-B, green threads, sunwjit)
castlerock% java tests.modalDialog.ModalLock
Dialog 2062288915 attempting lock
Dialog 2062288915 got lock
Dialog 2101872659 attempting lock

^\SIGQUIT

Full thread dump:
    "AWT-Modal" (TID:0xebcb5be0, sys_thread_t:0x31b178, state:CW) prio=6
at java.lang.Object.wait(Native Method)
at sun.awt.motif.MDialogPeer.pShow(Native Method)
at sun.awt.motif.ModalThread.run(Compiled Code)
    "AWT-Dispatch-Proxy" (TID:0xebcb5c00, sys_thread_t:0x315508, state:MW) prio=6
at tests.modalDialog.ModalDialog.askAuthorization(Compiled Code)
at tests.modalDialog.ModalLock.paint(Compiled Code)
at sun.awt.motif.MComponentPeer.handleEvent(Compiled Code)
at java.awt.Component.dispatchEventImpl(Compiled Code)
at java.awt.Container.dispatchEventImpl(Compiled Code)
at java.awt.Window.dispatchEventImpl(Compiled Code)
at java.awt.Component.dispatchEvent(Compiled Code)
at java.awt.EventQueue.dispatchEvent(Compiled Code)
at java.awt.EventDispatchThread.run(Compiled Code)
    "Screen Updater" (TID:0xebcb47d8, sys_thread_t:0x2c0308, state:CW) prio=4
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Compiled Code)
at sun.awt.ScreenUpdater.nextEntry(Compiled Code)
at sun.awt.ScreenUpdater.run(Compiled Code)
    "AWT-Motif" (TID:0xebcaf268, sys_thread_t:0x259050, state:CW) prio=5
at java.lang.Object.wait(Native Method)
at sun.awt.motif.MToolkit.run(Native Method)
at java.lang.Thread.run(Compiled Code)
    "AWT-Input" (TID:0xebcaf290, sys_thread_t:0x254d58, state:CW) prio=5
at sun.awt.motif.InputThread.run(Native Method)
    "AWT-EventQueue-0" (TID:0xebcaf000, sys_thread_t:0x237590, state:CW) prio=6
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Compiled Code)
at sun.awt.motif.MDialogPeer.show(Compiled Code)
at java.awt.Dialog.show(Compiled Code)
at tests.modalDialog.ModalDialog.askAuthorization(Compiled Code)
at tests.modalDialog.ModalLock.paint(Compiled Code)
at java.awt.Container.update(Compiled Code)
at sun.awt.motif.MComponentPeer.handleEvent(Compiled Code)
at java.awt.Component.dispatchEventImpl(Compiled Code)
at java.awt.Container.dispatchEventImpl(Compiled Code)
at java.awt.Window.dispatchEventImpl(Compiled Code)
at java.awt.Component.dispatchEvent(Compiled Code)
at java.awt.EventQueue.dispatchEvent(Compiled Code)
at java.awt.EventDispatchThread.run(Compiled Code)
    "Finalizer" (TID:0xebc98560, sys_thread_t:0x6fa78, state:CW) prio=8
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:113)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:128)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:128)
    "Reference Handler" (TID:0xebc98458, sys_thread_t:0x6bcc8, state:CW) prio=10
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:303)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:209)
    "Signal dispatcher" (TID:0xebc98328, sys_thread_t:0x6a660, state:R) prio=10
    "Thread-0" (TID:0xebcb43a8, sys_thread_t:0x28b08, state:CW) prio=5
Monitor Cache Dump:
    java.lang.Object@EBCB5288/EBD92460: owner "AWT-EventQueue-0" (0x237590) 1 entry
Waiting to enter:
"AWT-Dispatch-Proxy" (0x315508)
    java.lang.ref.ReferenceQueue$Lock@EBC98578/EBCCE6F0: <unowned>
Waiting to be notified:
"Finalizer" (0x6fa78)
    sun.awt.motif.ModalThread@EBCB5BE0/EBD96000: owner "AWT-Modal" (0x31b178) 1 entry
Waiting to be notified:
"AWT-EventQueue-0" (0x237590)
    sun.awt.ScreenUpdater@EBCB47D8/EBD8E1C0: <unowned>
Waiting to be notified:
"Screen Updater" (0x2c0308)
    java.lang.ref.Reference$Lock@EBC98308/EBCCE3D0: <unowned>
Waiting to be notified:
"Reference Handler" (0x6bcc8)
    sun.awt.motif.MToolkit@EBCAEE68/EBD65EC8: <unowned>
Waiting to be notified:
"AWT-Modal" (0x31b178)
"AWT-Motif" (0x259050)
Registered Monitor Dump:
    PCMap lock: <unowned>
    utf8 hash table: <unowned>
    JNI pinning lock: <unowned>
    JNI global reference lock: <unowned>
    BinClass lock: <unowned>
    Class linking lock: <unowned>
    System class loader lock: <unowned>
    Code rewrite lock: <unowned>
    Heap lock: <unowned>
    Monitor cache lock: <unowned>
    Dynamic loading lock: <unowned>
    Monitor IO lock: <unowned>
    User signal monitor: <unowned>
    Child death monitor: <unowned>
    I/O monitor: <unowned>
Waiting to be notified:
"AWT-Input" (0x254d58)
    Alarm monitor: <unowned>
Waiting to be notified:
"Internal clock" (0x2b9d0)
    Thread queue lock: <unowned>
Waiting to be notified:
"Thread-0" (0x28b08)
    Monitor registry: owner "Signal dispatcher" (0x6a660) 1 entry


Here is the test code:
package tests.modalDialog;

import java.awt.*;
import java.awt.event.*;

public class ModalLock extends Frame {
    int n = 1;
    public ModalLock () {
super("Modal lock test");
setSize(300,300);
setLocation(0,0);
    }

    public void paint(Graphics g) {
new ModalDialog(this, "Security Check " + n).askAuthorization();
n++;
    }
    
    public static void main(String args[]) {
ModalLock test = new ModalLock();
test.setVisible(true);
    }
}

class ModalDialog extends Dialog implements ActionListener {
    private static Object lock = new Object();
    Button b;
    
    ModalDialog(Frame dw, String title) {
        super(dw, title, true);

        b = new Button("OK");
b.addActionListener(this);
add("South", new Label("Click OK to continue..."));
add("Center", b);
        pack();
    }

    public void actionPerformed(ActionEvent event) {
if(event.getSource() == b) {
setVisible(false);
        }
    }
    
    public void askAuthorization() {
System.out.println("Dialog " + hashCode() + " attempting lock");
synchronized(lock) {
System.out.println("Dialog " + hashCode() + " got lock");
show();
dispose();
}
    }

}