if (gd.isFullScreenSupported()) {
gd.setFullScreenWindow(frame);
}
this code only maximizes window. This issue can be reproduced on a reparenting window manager.

From X11GD_SetFullscreenMode(), awt_GraphicsEnv.c:1793

    /*
     * Note: the Window passed to this method is typically the "content
     * window" of the top-level, but we need the actual shell window for
     * the purposes of constructing the XEvent. Therefore, we walk up the
     * window hierarchy here to find the true top-level.
     */

But since our frame window is not a direct child of a root window, we sending client event with wrong window.

So we should find another way to determine a "top-level" window.

[1] http://tronche.com/gui/x/icccm/sec-4.html#s-4.1.1