FULL PRODUCT VERSION :
openjdk version "1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-2ubuntu0.16.04.2-b14)
OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Ubuntu 16.04 64 bit

A DESCRIPTION OF THE PROBLEM :
If JDialog.setVisible(true) is shortly followed by JDialog.setVisible(false), the dialog window remains open.

This bug has been already reported a while ago but it was closed:

https://bugs.openjdk.java.net/browse/JDK-5109571

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
The example provided at
https://bugs.openjdk.java.net/browse/JDK-5109571
reproduces the bug for me
(also provided below)

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
No dialog windows should remain visible
ACTUAL -
Lots of (almost all) created dialog windows remain visible

REPRODUCIBILITY :
This bug can be reproduced often.

---------- BEGIN SOURCE ----------
import java.awt.*;
import java.awt.event.*;
import javax.swing.*;

public class Test extends JFrame implements ActionListener
{
   private int delay = 20;
   private int loopDelay = 50;
   private int xx;
   private int x;
   private int y;
   
   public Test()
   {
      JButton button = new JButton("Test JDialog");
      button.setActionCommand("test");
      button.addActionListener(this);

      getContentPane().add(button, BorderLayout.NORTH);
      setSize(new Dimension(400, 100));
   }

   public void actionPerformed(ActionEvent e)
   {
      if ("test".equals(e.getActionCommand()))
      {
         xx+=50;
         x = xx;
         y = 0;
         
         for (int i = 0; i < 100; i++)
         {
            System.out.println("Showing dialog with delay " + delay);
            JDialog dialog = new JDialog(this, "Test Dialog", false);
            dialog.setLocation(x, y);
            dialog.setVisible(true);

            try
            {
               Thread.sleep(delay);
            }
            catch (Exception exc) { }

            dialog.setVisible(false);

            x+=5;
            y+=5;
            delay -= 5;

            if (delay < 0)
               delay = 20;

            try
            {
               Thread.sleep(loopDelay);
            }
            catch (Exception exc) { }
         }
      }
   }

   private static void createAndShowGUI()
   {
      Test test = new Test();
      test.setVisible(true);
   }

   public static void main(String[] args)
   {
      javax.swing.SwingUtilities.invokeLater(
         new Runnable()
         {
            public void run()
            {
               createAndShowGUI();
            }
         });
   }
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
A possible workaround is to call JDialog.dispose() instead of (or in addition to) JDialog.setVisible(false), but then the dialog cannot be used any longer.