The Apple standard for hiding icons on a dock is to set LSUIElement to 1 in Info.plist (see https://developer.apple.com/library/mac/documentation/general/Reference/InfoPlistKeyReference/Articles/LaunchServicesKeys.html#//apple_ref/doc/uid/20001431-108256)

Setting this value fails to hide the dock icon when launching a main class that extends javafx.application.Application. As a workaround, I can launch a separate main class that initializes AWT before JavaFX and call the Application class which will successfully hide the dock icon.

For example, assuming I already have LSUIElement set to 1, running TestApp will successfully hide the icon. Running JavaFXApp will not hide the icon.

public class TestApp {
  public static void main(String[] args) {
    java.awt.Toolkit.getDefaultToolkit();

    JavaFXApp.doLaunch(args);
  }
}

public class JavaFXApp extends javafx.application.Application {
  public static void doLaunch(String[] args) {
    launch(args);
  }

  @Override
  public void start(Stage primaryStage) throws Exception {
    ...
  }
}