This issue is reproducible on Windows.

Reproducer:

import javax.print.*;
import javax.print.attribute.HashPrintRequestAttributeSet;

public class Test {
    public static void main(String[] args) {
        PrintService service = PrintServiceLookup.lookupDefaultPrintService();
        PrintService[] services = PrintServiceLookup.lookupPrintServices(null, null);
        HashPrintRequestAttributeSet attrs = new HashPrintRequestAttributeSet();
        ServiceUI.printDialog(null, 200, 200, services, service, DocFlavor.SERVICE_FORMATTED.PAGEABLE, attrs);
    }
}


Running this test shows a dialog that includes properties button for showing properties dialog. This button is disabled on Mac. It's enabled on Windows but clicking it doesn't show any dialog. According to JDK-4673406, the properties dialog isn't supported for direct uses with javax.print.ServiceUI.printDialog. It might make sense to disable the properties button in this case.