In the test, Frame and parentless modal Dialog are shown, in that order. Frame is bigger.
After a click on the Frame outside of Dialog, Frame goes on top violating modality settings.

There is a bug in the test allowing it sometimes and even often to pass:
both Frame and Dialog are disposed _before_ colors are compared. If we let harness to dispose windows, test fails always on my OS X 10.9: 50 out of 50 times.

And it is just visible: Frame is on top after the click.

NB: I'm going to fix the test in JDK-8063104.