ADDITIONAL SYSTEM INFORMATION :
Java >8

Tested with
Java 8 / Linux
Java 11 / Windows, Linux
Java 17 / Linux
Java 23 / Linux


A DESCRIPTION OF THE PROBLEM :
JLayer delegates its setBorder() and getBorder() method calls to its view. As a side effect, its (JComponent.)paintBorder() paints the border of that child. This is because paintBorder() calls getBorder() instead of using this.border directly like most other places in JComponent does (there's two calls of getBorder() in JComponent compared to 13 uses of this.border). Then the child paints the border again in paintChildren().

Originally JLayers didn't have borders: calling setBorder() threw an exception and getBorder() was not defined. This was changed as a result of fix to bug 8054543 <https://github.com/openjdk/jdk/commit/08fb482e4122fcd5d14a9a14f5570df86a21960e> sometime during Java 9. In that commit the current implementation of those methods was introduced.

When the child component is opaque there's no visible issue, but with a non-opaque object whose border is translucent there is. The issue appears when the border is directly drawn with translucent color (alpha < 1) or if it is drawn with opaque color but composited with a translucent AlphaComposite. In both cases the result is darker than expected because JLayer already drew the border.

I have tested the example code with Java 8 (draws correctly) and with Java 11, 17, and 23, all of which do not work.

Maybe either replace getBorder() with this.border in JComponent, or implement a no-op paintBorder() for JLayer. There might be other inconsistencies in JLayer between getBorder() returning non-null and other methods working as if there no border, e.g. getInsets().


---------- BEGIN SOURCE ----------
import java.awt.AlphaComposite;
import java.awt.BorderLayout;
import java.awt.Color;
import java.awt.Composite;
import java.awt.Container;
import java.awt.Dimension;
import java.awt.Graphics;
import java.awt.Graphics2D;
import javax.swing.BorderFactory;
import javax.swing.JFrame;
import javax.swing.JLayer;
import javax.swing.JPanel;
import javax.swing.SwingUtilities;
import javax.swing.plaf.LayerUI;

/**
 * Demonstrate how JLayer's border drawing behavior breaks translucent borders.
 *
 * <p>The windows created by this program have a JLayer containing a bordered
 * JPanel ("outer") that contains another bordered JPanel ("inner"),
 * demonstrating how the outer border is drawn differently than the inner one
 * with versions later than Java 8. Java 8 draws them similarly. The details
 * of these three windows are:</p>
 *
 * <dl>
 * <dt>Border</dt>
 * <dd>Draws the borders with 0.5f alpha.</dd>
 * <dt>Paint</dt>
 * <dd>Uses AlphaComposite to draw both panels.</dd>
 * <dt>Workaround</dt>
 * <dd>Wraps the outer panel in a third panel ("borderless") with an empty
 * border.</dd>
 * </dl>
 */
public class JLayerBug {
    public static void main(String[] args) {
        JLayerBug program = new JLayerBug();
        SwingUtilities.invokeLater(program::gui);
    }

    public void gui() {
        Window border = new Window(Kind.BORDER_ALPHA);
        border.pack();
        border.setVisible(true);

        Window paint = new Window(Kind.PAINT_ALPHA);
        paint.pack();
        paint.setVisible(true);

        Window workaround = new Window(Kind.WORKAROUND);
        workaround.pack();
        workaround.setVisible(true);

    }

    enum Kind {
        BORDER_ALPHA, PAINT_ALPHA, WORKAROUND
    }

    private class Window extends JFrame {
        private final Kind kind;

        private Window(Kind kind) {
            this.kind = kind;
            setTitle(title());

            Container c = getContentPane();
            c.setLayout(new BorderLayout());
            LayerUI<JPanel> layerUi = new LayerUI<>();
            JLayer<JPanel> layer = new JLayer<>(contentPanel(), layerUi);
            c.add(layer);

            setDefaultCloseOperation(EXIT_ON_CLOSE);
        }

        private String title() {
            switch (this.kind) {
                case BORDER_ALPHA: return "Border";
                case PAINT_ALPHA: return "Paint";
                case WORKAROUND: return "Workaround";
            }
            return "wot";
        }

        private JPanel contentPanel() {
            Color borderColor = kind != Kind.PAINT_ALPHA
                    ? new Color(0, 0, 0, 0.5f)
                    : Color.BLACK;

            JPanel outer = new JPanel() {
                @Override
                public void paint(Graphics g) {
                    if (kind != Kind.PAINT_ALPHA) {
                        super.paint(g);
                        return;
                    }

                    Graphics2D g2d = (Graphics2D) g;
                    Composite c = g2d.getComposite();
                    AlphaComposite a = AlphaComposite.getInstance(AlphaComposite.SRC_OVER, 0.25f);
                    g2d.setComposite(a);
                    super.paint(g2d);
                    g2d.setComposite(c);
                }
            };
            outer.setOpaque(false);
            outer.setLayout(new BorderLayout());
            outer.setBorder(BorderFactory.createLineBorder(borderColor, 5));
            outer.setPreferredSize(new Dimension(400, 300));

            JPanel inner = new JPanel();
            inner.setOpaque(false);
            inner.setBorder(BorderFactory.createLineBorder(borderColor, 5));
            outer.add(inner);

            if (kind == Kind.WORKAROUND) {
                JPanel borderless = new JPanel();
                borderless.setLayout(new BorderLayout());
                borderless.add(outer);
                return borderless;
            } else {
                return outer;
            }
        }
    }
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Add an extra borderless panel to wrap the child.