this is a linux/solaris part of 6364741 (REG: Using frame's menu actions does not make swing menu disappear on WinXP, since Mustang-b53)
Here is a part of evaluation of that bug:
I've tested this on linux (under metacity) and found that grab makes impossible
to open system menu for toplevel.
App doesn't react to Alt-space at all
If I click mouse on system menu or min/max/close buttons we close swing's menu
and user needs to click second time to either open menu or minimize/maximaze/close
toplevel :(

It looks like uner Linux menus of all applications behave differently.
E.g. gaim behaves the same way as java.
Firefox handle keyboard correctly, but mouse works the same way as in java.
This make me think that this is a common problem under X.