Specification for java.awt.Window::setShape(Shape) states:
"Only the parts that belong to the given Shape remain visible and clickable"
however this assertion is not followed for example by macOS JDK implementation - as extra decorations edging the shape remain visible - so you cannot strictly limit the visible area with the given Shape as the spec guarantees - some halo is painted and is visible outside of the given shape.

Please see the attached screenshot illustrating the problem - where a black "+"-shaped window is painted over another white window and given "+" shape has visible edging artifacts.