ADDITIONAL SYSTEM INFORMATION :
Darwin Myst.local 17.6.0 Darwin Kernel Version 17.6.0: Tue May 8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASE_X86_64 x86_64

A DESCRIPTION OF THE PROBLEM :
The client property `apple.awt.documentModalSheet` is supposed to change the behavior of a Dialog such that it becomes attached to the owner window. This behavior is in fact implemented in `sun.lwawt.macosx.CPlatformWindow`, where the client property is tested and a SHEET bitmask is applied. However, it appears that this bitmask is not respected by the native platform implementation.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Create a dialog with a parent frame that is visible and a modality of DOCUMENT_MODAL.
Apply "apple.awt.documentModalSheet" with a value of Boolean.TRUE to the dialog's root pane.
Make the dialog visible.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The dialog appears as a sheet attached to the owner.
ACTUAL -
The dialog appears as a new window.

---------- BEGIN SOURCE ----------
                JDialog dialog = new JDialog( frame, null, Dialog.ModalityType.DOCUMENT_MODAL );
                dialog.getRootPane().putClientProperty( "apple.awt.documentModalSheet", Boolean.TRUE );
                dialog.add( new JLabel( "Hello world!" ) );
                dialog.pack();
                dialog.setVisible( true );
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
The application needs to be executed under Apple's deprecated JDK 1.6 in order to restore correct behavior.

To observe the correct behavior, run this code using Apple's JDK as available at:
http://support.apple.com/downloads/DL1572/en_US/javaforosx.dmg / https://support.apple.com/kb/dl1572

FREQUENCY : always