FULL PRODUCT VERSION :
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b18)
Java HotSpot(TM) Client VM (build 25.66-b18, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows [Version 6.1.7601]

A DESCRIPTION OF THE PROBLEM :
When printing, using the java native print dialog and a custom paper size, the paper selected may not be the paper size used if the orientation is landscape.

This is dependent on the paper sizes that are available on the system

It seems that issue traces back to CustomMediaSizeName.findMedia.
The paper size is flipped into landscape mode which means the difference in the X and Y are larger in one direction in smaller in another, but the code squares the differences and the +/- is lost, giving you a net large difference. Since the difference is larger, smaller paper sizes appear to have less difference and are selected over the same paper in another orientation.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Create a custom paper size and set it as default in printer driver Print Preferences
For this I created a copy of A4 paper, called 'A4_Margin' with 2 inch margins on all sides and set it as default on printer

The printer I used is CutePDFWriter but the issue exists on other printers (physical and virtual)

Create a new PrintRequestAttributeSet

add DialogTypeSelection.NATIVE to attribute set
add OrientationRequested.LANDSCAPE to attribute set

Call PrinterJob.printDialog(PrintRequestAttributeSet)

Validate page format retrieved from PrinterJob, using PrintRequestAttributeSet

Call PrinterJob.setPrintable

Call PrinterJob.Print(PrintRequestAttributeSet)

Note that the attribute set media is not the same paper as was selected.

Note that the output paper is not the same paper size that was selected.


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The paper size selected is the paper size used, regardless of orientation
ACTUAL -
The paper size used may change from the paper size selected, depending on orientation

REPRODUCIBILITY :
This bug can be reproduced often.

---------- BEGIN SOURCE ----------
/*
 * Copyright 2016 Mentor Graphics Corporation
 * All Rights Reserved
 *
 * THIS WORK CONTAINS TRADE SECRET AND PROPRIETARY
 * INFORMATION WHICH IS THE PROPERTY OF MENTOR
 * GRAPHICS CORPORATION OR ITS LICENSORS AND IS
 * SUBJECT TO LICENSE TERMS.
 */

import javax.print.attribute.Attribute;
import javax.print.attribute.HashPrintRequestAttributeSet;
import javax.print.attribute.PrintRequestAttributeSet;
import javax.print.attribute.standard.DialogTypeSelection;
import javax.print.attribute.standard.Media;
import javax.print.attribute.standard.OrientationRequested;
import java.awt.print.PageFormat;
import java.awt.print.PrinterException;
import java.awt.print.PrinterJob;
import java.util.function.BiConsumer;

public class PrintingTestIssue2
{
    private static void showPrintDialog(int orientation)
    {
        PrinterJob pJob = PrinterJob.getPrinterJob();
        PrintRequestAttributeSet pras = new HashPrintRequestAttributeSet();
        pras.add(DialogTypeSelection.NATIVE);
        pras.add(orientation == PageFormat.LANDSCAPE ? OrientationRequested.LANDSCAPE : OrientationRequested.PORTRAIT);

        if (pJob.printDialog(pras)) {
            PageFormat pFormat = pJob.getPageFormat(pras);
            pJob.validatePage(pFormat);
            pJob.setPrintable((graphics, pageFormat, pageIndex) -> 1, pFormat);
            try {
                pJob.print(pras);
            } catch (PrinterException e) {
                e.printStackTrace();
            }
            reportUsedPaperSize(pras);
        }
    }

    private static void reportUsedPaperSize(PrintRequestAttributeSet pras)
    {
        Attribute mediaAttr = (pras != null && !pras.isEmpty()) ? pras.get(Media.class) : null;
        Media m = (mediaAttr instanceof Media) ? (Media)mediaAttr : null;
        String setMediaName = m != null ? m.toString() : null;
        System.out.println(setMediaName != null ? "Media Used :" + setMediaName : "Media Used : Unknown");
    }

    private static void issue() throws PrinterException
    {
        String paperNameSelected = "A4_Margin";
        BiConsumer<String, Integer> runPaperTest = (orientName, orient) -> {
            System.out.println("[ " + orientName + " ]");
            System.out.println("Media selected :" + paperNameSelected);
            showPrintDialog(orient);
            System.out.println();
        };

        runPaperTest.accept("Portrait", PageFormat.PORTRAIT);
        runPaperTest.accept("Landscape", PageFormat.LANDSCAPE);
    }

    public static void main(String[] args) throws PrinterException {
        // In this issue, the paper size is flipped into landscape mode which means the difference in the X and Y
        // are larger in one direction in smaller in another, but the code squares the differences and the +/- is lost,
        // giving you a net large difference. Since the difference is larger, smaller paper sizes appear to have less
        // difference and are selected over the same paper in another orientation.
        issue();
    }
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Set paper and orientation as default in Printer's printing preferences before starting java application.

After calling PrinterJob.Print(PrintRequestAttributeSet), clear PrintRequestAttributeSet