Take the following simple example, which gradually fades a rectangle onto a transparent canvas:

public class Test extends Application {

    @Override
    public void start(Stage primaryStage) {
        primaryStage.initStyle(StageStyle.TRANSPARENT);
        int width = 1920;
        int height = 1080;

        Rectangle rect = new Rectangle(width, height);
        rect.setFill(Color.SALMON);
        rect.setOpacity(0);
        StackPane scenePane = new StackPane();
        scenePane.getChildren().add(rect);
        primaryStage.setScene(new Scene(scenePane));
        primaryStage.setWidth(width);
        primaryStage.setHeight(height);
        primaryStage.show();
        FadeTransition ft = new FadeTransition(Duration.millis(10000), rect);
        ft.setToValue(1);
        ft.play();
    }

    public static void main(String[] args) {
        launch(args);
    }

}


When run with the VM args -Xms100m -Xmx100m, this works no problem at all. However, when I give the VM substantially more memory (such as -Xms1000m -Xmx1000m) it very quickly falls over:

java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
        at com.sun.prism.impl.BufferUtil.newByteBuffer(BufferUtil.java:90)
        at com.sun.prism.impl.BufferUtil.newIntBuffer(BufferUtil.java:121)
        at com.sun.javafx.tk.quantum.UploadingPainter.run(UploadingPainter.java:148)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at com.sun.javafx.tk.RenderJob.run(RenderJob.java:58)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:125)
        at java.lang.Thread.run(Thread.java:745)

More than one profiler I've tried shows the heap at as taking up barely any of the allocated space - but the process view in task manager shows it shooting through all the available memory in a couple of seconds.

The oddities don't end there however - it only seems to fall over for some values of width and height (a width of 1921 for instance means that the application executes fine, no errors in sight.)

This only seems to occur with a transparent stage. Without the transparent style set on the stage in the first line, all seems well (with all configurations I've tried, anyway.) Likewise, it only occurs on Java 8 (I'm using 8u20) - all is fine with Java 7 / JFX 2.x. I'm running Windows 7x64.