Reported at http://mail.openjdk.java.net/pipermail/macosx-port-dev/2013-March/005447.html :

When using the Swing client property "apple.awt.documentModalSheet" on JDK
1.7.0_17 on a Mountain Lion Retina MacBook Pro, the dialog is the expected
size but the content is offset (moved up under the title bar) and
hidden/clipped. When this property is not set, the layout is as expected.

The "sheet" dialog behavior is not present as it was in Apple JDK 6. I
don't know if it is planned to fully implement it. However, it appears
that this property is still taken into account by Oracle JDK 7 to offset
the content, but because the title bar's still there, it leads to layout
issues.