FULL PRODUCT VERSION :
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows [Version 6.1.7601]
Microsoft Windows [Version 10.0.15063]

A DESCRIPTION OF THE PROBLEM :
JavaFX shapes, like i.e. javafx.scene.shape.Path, are rendered using a "mask texture", which is stored in the maskTex variable of the BaseContext class.

If the BaseResourcePool runs out of available space (maxSize is reached), it performs a cleanup to remove dead/unused textures and free some space. But, if the cleanup is "desperate", it will remove also the texture referenced by the maskTex variable. After this moment, every GUI rendering causes a NullPointerException (triggered by the method BaseContext.flushMask()), meaning an endless loop causing a GUI freeze.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1) Run the attached program on a Windows environment, providing the -Dprism.maxvram=10M system property (e.g. java -Dprism.maxvram=10M MyTestProgram)
2) Look at console output, there should be a lot of NullPointerExceptions.

Please note that the system property -Dprism.maxvram=10M is needed for debug purposes in order to accelerate the error showing. Even with much higher settings (like 512M or 1G) the problem shows up, if the application uses lots of textures, effects, shapes, etc.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Unused resources are cleaned and the program continues running normally
ACTUAL -
The GUI freezes and NullPointerExceptions are thrown.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
java.lang.NullPointerException
at com.sun.prism.d3d.D3DTexture.getContext(D3DTexture.java:84)
at com.sun.prism.d3d.D3DTexture.update(D3DTexture.java:207)
at com.sun.prism.d3d.D3DTexture.update(D3DTexture.java:151)
at com.sun.prism.impl.BaseContext.flushMask(BaseContext.java:109)
at com.sun.prism.impl.BaseContext.drawQuads(BaseContext.java:118)
at com.sun.prism.impl.VertexBuffer.flush(VertexBuffer.java:98)
at com.sun.prism.impl.BaseContext.flushVertexBuffer(BaseContext.java:101)
at com.sun.prism.impl.ps.BaseShaderContext.setRenderTarget(BaseShaderContext.java:746)
at com.sun.prism.impl.BaseContext.setRenderTarget(BaseContext.java:131)
at com.sun.prism.impl.BaseGraphics.<init>(BaseGraphics.java:105)
at com.sun.prism.impl.ps.BaseShaderGraphics.<init>(BaseShaderGraphics.java:86)
at com.sun.prism.d3d.D3DGraphics.<init>(D3DGraphics.java:42)
at com.sun.prism.d3d.D3DGraphics.create(D3DGraphics.java:65)
at com.sun.prism.d3d.D3DSwapChain.createGraphics(D3DSwapChain.java:131)
at com.sun.javafx.tk.quantum.PresentingPainter.run(PresentingPainter.java:87)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at com.sun.javafx.tk.RenderJob.run(RenderJob.java:58)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:125)
at java.lang.Thread.run(Thread.java:748)

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
public class Main extends Application {

    public void start(Stage primaryStage) throws Exception {
        StackPane pane = new StackPane();
        pane.getChildren().add(new Path());
        ImageView view = new ImageView(new WritableImage(2000, 500));
        pane.getChildren().add(view);
        pane.getChildren().add(new ImageView(new WritableImage(2000, 500)));
        primaryStage.setScene(new Scene(pane));
        primaryStage.show();

        // Let's create a simple transition, in order to force re-rendering and texture update
        ScaleTransition scaleTransition = new ScaleTransition(Duration.seconds(2), view);
        scaleTransition.setByX(1);
        scaleTransition.setByY(1);
        scaleTransition.setToX(2);
        scaleTransition.setToY(2);
        scaleTransition.play();
    }
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Giving the application more VRAM than the default, by using for example -Dprism.maxvram=2G might help in some cases, where the graphic card supports that.

A proposed fix for the bug could be improving the method flushMask in the class com.sun.prism.impl.BaseContext, like the following. Please note the added try/catch/finally block:

    protected final void flushMask() {
        if (curMaskRow > 0 || curMaskCol > 0) {
            try {
                maskTex.lock();
                // assert !maskTex.isSurfaceLost();
                // since it was bound and unflushed...
                maskTex.update(maskBuffer, maskTex.getPixelFormat(),
                                           0, 0, 0, 0, highMaskCol, nextMaskRow,
                                           maskTex.getContentWidth(), true);
            } catch (Exception e) {
                System.err.println("Caught NPE while updating maskTex - Continuing...");
            } finally {
                maskTex.unlock();
                curMaskRow = curMaskCol = nextMaskRow = highMaskCol = 0;
            }
        }
    }