There is a Swing Manual Test Swing_JWindow/Manual/JDialogLAFTests/DialogLAFTest4.
In it's 4th scenario there is a check that custom item could be set for JDialog.

However there's no icon for JDialoh in alt-tab list at all.

There is a related bug 6445278 (with code example) stating that issue is Windows specific. It's indeed not reproducible on Ubuntu for SE Java. But it's there for embedded java.

OS: Ubuntu 11.4
Platform: linux-i586
Build: ejavase 6u29 b09