FULL PRODUCT VERSION :
It seems to be broken in ALL java versions 1.8 and 1.7 I tested nearly every version of 1.8 and I tested all the "regression" versions listed on this form's regress dropdown list that were testable with macOS (that means I tested 1.7 update 4). I didn't care to test with Java 1.6 since Java 6 is a macOS software product and didn't become a Java product until 1.7 update 4.

As a side note, when I tested with 1.7 update 4, I had a different bug where the Help menu was removed from the menu bar. Weird but I'm not intending to support 1.7 so you don't have to bother considering that bug for my sake.


$ java -version
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)


$ javac -version
javac 1.8.0_161


ADDITIONAL OS VERSION INFORMATION :
Darwin MacBook-Pro 17.3.0 Darwin Kernel Version 17.3.0: Thu Nov 9 18:09:22 PST 2017; root:xnu-4570.31.3~1/RELEASE_X86_64 x86_64


A DESCRIPTION OF THE PROBLEM :
If I click a JMenuItem, its ActionListener's actionPerformed method calls JOptionPane.showConfirmDialog and as a result that JMenuItem is disabled. But after closing the dialog the JMenuItem should be enabled. But it is not. In fact, all the JMenuItems are disabled after the dialog is closed.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Create a real or virtual macOS High Sierra machine.
2. Download the two source code files, test.java and test.sh.
3. Open a terminal window
4. Install Java SE Development Kit 8u161
5. Run test.sh after editing it to make it refer to 8u161
6. Wait for the menu bar to display its Help menu.
7. Click the Help menu.
8. Click the Help menu's "Item..." item.

You should then see a dialog for the code executes JOptionPane.showConfirmDialog.

9. Click the dialog's close control
10. Click the Help menu

If you reproduce the bug, then the "Item..." item will be disabled. If you cannot reproduce the bug, then the "Item..." item will be enabled. The latter behavior is what I desire and expect.

11. Click the test -> "About test" menu item

You should see a dialog created by something other than my code. This will be true whether or not the bug manifests and is what I desire.

12. Click the dialog's close control.

13. Cick the Help menu.

You should now see that the "Item..." item is enabled.



EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Read my Steps to Reproduce for all my expectations.
ACTUAL -
Read my Steps to Reproduce for what I see and what I expect.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
$ cat test.java
import java.awt.event.*;
import javax.swing.*;

public class test {
    public test () {
        final JFrame me = new JFrame();
        JMenuBar menuBar = new JMenuBar();
        JMenu helpMenu = new JMenu();
        JMenuItem mi = new JMenuItem();

        helpMenu.setText("Help");

        mi.setText("Item...");

        mi.addActionListener(new ActionListener() {
            @Override
            public void actionPerformed(ActionEvent e) {
               // JOptionPane.showConfirmDialog(me, "a");
         SwingUtilities.invokeLater(new Runnable() {
             public void run() {
                JOptionPane.showConfirmDialog(me, "a");
             }
         });
            }
        });
        helpMenu.add(mi);
        menuBar.add(helpMenu);
        me.setJMenuBar(menuBar);
        me.setVisible(true);
    }
    public static void main (String[] args) {
        SwingUtilities.invokeLater(new Runnable() {
            public void run() {
                new test();
            }
        });
    }
}




$ cat test.sh
JDK=1.7.0.jdk
#JDK=jdk1.7.0_80.jdk
#JDK=jdk1.8.0_161.jdk
JAVA_HOME=/Library/Java/JavaVirtualMachines/$JDK/Contents/Home
export JAVA_HOME
PATH=$JAVA_HOME/bin:$PATH
javac -version
javac test.java

java -version
java -Dapple.laf.useScreenMenuBar=true test

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
There is no known workaround. You can see from the source code in test.java, that I have tried two methods for raising the dialog. Neither of them make the JMenuBar work correctly.