FULL PRODUCT VERSION :
8u121

ADDITIONAL OS VERSION INFORMATION :
macOS 10.12.6 Sierra

A DESCRIPTION OF THE PROBLEM :
When the mouse is pressed over a JDialog in an area where this dialog overlaps its owner JFrame, a mouse pressed event is fired *followed by a mouse exited event on the dialog, a mouse entered and a mouse exited event on the frame, and a final mouse entered event on the dialog.


REGRESSION. Last worked in version 8u121

ADDITIONAL REGRESSION INFORMATION:
Regression introduced in 8u121 (reproducible with 8u121, 8u131, 8u141, 8u144).
Last working version 8u112.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the attached test case.
Click the "Open" button.
Move the mouse over the opened JDialog into the area where the dialog overlaps its parent frame.
Press and hold the left mouse button without moving the mouse.
Check the output written to System.out for received mouse events.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The topmost component of the JDialog receives a mouse pressed event (and no other events). Moreover, no other component receives any events (especially not components under the dialog).
ACTUAL -
The topmost component of the JDialog receives a mouse pressed event followed by a mouse exited and a mouse entered event. In between, the topmost component of the JFrame receives a mouse entered and a mouse exited event.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.awt.AWTEvent;
import java.awt.Dimension;
import java.awt.EventQueue;
import java.awt.Toolkit;
import java.awt.event.AWTEventListener;
import java.awt.event.ActionEvent;
import java.awt.event.MouseAdapter;
import java.awt.event.MouseEvent;
import javax.swing.AbstractAction;
import javax.swing.JButton;
import javax.swing.JComponent;
import javax.swing.JDialog;
import javax.swing.JFrame;
import javax.swing.JPanel;

public class TestCase {
  public static void main( String[] args ) {
    EventQueue.invokeLater(new Runnable() {
      public void run() {
        TestCase.start();
      }
    });
  }

  static void start() {
    Toolkit.getDefaultToolkit().addAWTEventListener(new AWTEventListener() {
      public void eventDispatched( final AWTEvent event ) {
        Object src = event.getSource();
        String name = src.getClass().getName();
        if (src instanceof JComponent) {
          name = ((JComponent) src).getName();
        }
        System.out.println(asString(event.getID()) + ": " + name);
      }

      private String asString( final int id ) {
        switch (id) {
          case MouseEvent.MOUSE_CLICKED:
            return "clicked";
          case MouseEvent.MOUSE_ENTERED:
            return "entered";
          case MouseEvent.MOUSE_EXITED:
            return "exited";
          case MouseEvent.MOUSE_PRESSED:
            return "pressed";
          case MouseEvent.MOUSE_RELEASED:
            return "released";
          case MouseEvent.MOUSE_MOVED:
            return "moved";
          default:
            return "unknown event: " + id;
        }
      }

    }, AWTEvent.MOUSE_EVENT_MASK|AWTEvent.MOUSE_MOTION_EVENT_MASK);

    final JFrame frame = new JFrame(TestCase.class.getName());
    JButton jb = new JButton(new AbstractAction("Open") {
      public void actionPerformed( final ActionEvent e ) {
        final JPanel empty = new JPanel();
        empty.setPreferredSize(new Dimension(400, 400));
        empty.setName("panel_in_dialog");
        empty.addMouseListener(new MouseAdapter() {});

        final JDialog jd = new JDialog(frame);
        jd.setLocationRelativeTo(frame);
        jd.getContentPane().add(empty);
        jd.pack();
        jd.setVisible(true);
      }
    });
    jb.setPreferredSize(new Dimension(200, 200));
    jb.setName("button_in_frame");

    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
    frame.getContentPane().add(jb);
    frame.pack();
    frame.setLocationRelativeTo(null);
    frame.setVisible(true);
  }
}

---------- END SOURCE ----------