http://java.net/jira/browse/MACOSX_PORT-35 submitted 2011/04/27 by Artem Ananiev
Run the attached test. It creates a simple undecorated window and register a mouse listener. When clicked, the window is maximized with a call to setExtendedState(MAXIMIZED_BOTH). However, the maximization only works for decorated windows, while undecorated windows just move to the left bottom corner of the screen.
The test case:

import java.awt.*;
import java.awt.event.*;

public class CheckExtendedState {

    public static void main(String[] args) {

final Frame f = new Frame();
f.setUndecorated(true);
f.addMouseListener(new MouseAdapter() {
public void mouseClicked(MouseEvent e) {
f.setExtendedState(Frame.MAXIMIZED_BOTH);
}
});
f.addKeyListener(new KeyAdapter() {
            public void keyTyped(KeyEvent e) {
if (e.getKeyChar() == 'g') {
System.exit(0);
}
}
});
f.setSize(640, 480);
f.setLocationRelativeTo(null);
f.setVisible(true);
    }

}