ADDITIONAL SYSTEM INFORMATION :
Reproduced on MacOS (10.13.6, 12.0.1), Ubuntu 18.04 and Debian 9. Works as expected on Windows 7 to 10.

Can be reproduced across java 8 to 19.

A DESCRIPTION OF THE PROBLEM :
On Linux and MacOS, when you launch two JDialog, one with setUndecorated(true) and another with setModal(true), the first one is able to be focused with a click and this modeless dialog goes to the front, blocking the view of the modal dialog. If the first/modeless dialog doesn't have undecorated set to true, this behaviour doesn't happen and works as expected, the modeless dialog never blocks the view of the modal.

On Windows it works as expected on all cases.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Click on a background undecorated dialog while showing a modal dialog.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The focus remains on the modal dialog, and the visibility does not change.
ACTUAL -
The undecorated dialog goes to the front view above the modal dialog, which cannot be displayed at the front again if you click on it.

---------- BEGIN SOURCE ----------
import java.awt.Dialog.ModalityType;
import javax.swing.JDialog;
import javax.swing.JLabel;

public class Test {

public static void main(String[] args) {
JDialog d1 = new JDialog(null, "undecorated dialog", ModalityType.MODELESS);
d1.add(new JLabel("undecorated dialog"));
d1.setUndecorated(true);
d1.setSize(600, 200);
d1.setLocationRelativeTo(null);
d1.setVisible(true);

JDialog d2 = new JDialog(null, "modal dialog", ModalityType.APPLICATION_MODAL);
d2.add(new JLabel("modal dialog"));
d2.setSize(200, 600);
d2.setLocationRelativeTo(null);
d2.setVisible(true);
}

}

---------- END SOURCE ----------

FREQUENCY : always