ADDITIONAL SYSTEM INFORMATION :
I believe the problem is still present in the latest OpenJFX for OpenJDK 11, but I was unable to find a build which includes the Monocle platform for any of my systems. The Intel "x86egl" compile target has been removed in OpenJFX 11, and there's no ARM build provided for OpenJDK 11. The upcoming Ubuntu 18.04 release does provide an ARM build for OpenJDK 11, but it requires Linux kernel version 3.2.0, and I'm stuck with Linux kernel version 2.6.35 on my only ARM device.

So meanwhile, I reproduced this problem on the three systems described below. The Java VM is Oracle JDK 8 Update 172 (an early-access build) with the OpenJFX 8 x86egl and armv7hf overlay bundles built from sources on April 10, 2018.

---------------------------------------------------------------------
1. A Kobo Touch N905C e-reader with 256 MB of RAM and an 800 MHz Freescale i.MX507 Multimedia Application Processor Model MCIMX507CVM8B (ARMv7-A architecture, ARM Cortex-A8 core). This is a 32-bit ARM system running Ubuntu 14.04.5 LTS as follows:

$ uname -a
Linux koboa 2.6.35.3-850-gbc67621+ #619 PREEMPT Thu Dec 22 15:29:00 CST 2016 armv7l armv7l armv7l GNU/Linux

$ ldd --version
ldd (Ubuntu EGLIBC 2.19-0ubuntu6.14) 2.19

$ getconf GNU_LIBPTHREAD_VERSION
NPTL 2.19

$ ~/opt/jdk1.8.0_172/bin/java -version
java version "1.8.0_172-ea"
Java(TM) SE Runtime Environment (build 1.8.0_172-ea-b03)
Java HotSpot(TM) Client VM (build 25.172-b03, mixed mode)

---------------------------------------------------------------------
2. A Dell Inspiron 1011 netbook with 1 GB of RAM and a 1.60 GHz Intel Atom N270 processor with hyper-threading enabled (2 logical processors). This is an 32-bit Intel system running Ubuntu 16.04.4 LTS as follows:

$ uname -a
Linux mini10v 4.13.0-38-generic #43~16.04.1-Ubuntu SMP Wed Mar 14 17:46:42 UTC 2018 i686 i686 i686 GNU/Linux

$ ldd --version
ldd (Ubuntu GLIBC 2.23-0ubuntu10) 2.23

$ getconf GNU_LIBPTHREAD_VERSION
NPTL 2.23

$ ~/opt/jdk1.8.0_172/bin/java -version
java version "1.8.0_172-ea"
Java(TM) SE Runtime Environment (build 1.8.0_172-ea-b03)
Java HotSpot(TM) Client VM (build 25.172-b03, mixed mode)

---------------------------------------------------------------------
3. A QEMU/KVM virtual machine guest with 1 GB of RAM and 1 logical Broadwell-IBRS processor running on a Dell Precision Tower 3420 workstation host with 16 GB of RAM and a 4-core 3.30 GHz Intel Xeon E3-1223 v5 processor. This is a 64-bit Intel guest system running Ubuntu 16.04.4 LTS as follows:

$ uname -a
Linux testjfx64 4.13.0-38-generic #43~16.04.1-Ubuntu SMP Wed Mar 14 17:48:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ ldd --version
ldd (Ubuntu GLIBC 2.23-0ubuntu10) 2.23

$ getconf GNU_LIBPTHREAD_VERSION
NPTL 2.23

$ ~/opt/jdk1.8.0_172/bin/java -version
java version "1.8.0_172-ea"
Java(TM) SE Runtime Environment (build 1.8.0_172-ea-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b03, mixed mode)


A DESCRIPTION OF THE PROBLEM :
The QuantumRenderer thread rewinds the frame buffer while it is in use by the JavaFX Application Thread. The problem occurs on Monocle platforms which implement the NativeScreen interface by allocating an non-direct buffer with ByteBuffer.allocate(int), rather than a direct buffer with ByteBuffer.allocateDirect(int), when creating the backing byte buffer for the Framebuffer. Among the current Monocle platforms, both the Headless and VNC platforms use a non-direct buffer and encounter this problem.

Specifically, the JavaFX Application Thread regularly calls the put(IntBuffer) method in java.nio.IntBuffer to copy its frame buffer, as shown below (with my BEGIN/END comments added):

---------------------------------------------------------------------
public IntBuffer put(IntBuffer src) {
    if (src == this)
        throw new IllegalArgumentException();
    if (isReadOnly())
        throw new ReadOnlyBufferException();

    // BEGIN critical section for buffer position

    int n = src.remaining();
    if (n > remaining())
        throw new BufferOverflowException();
    for (int i = 0; i < n; i++)
        put(src.get()); // Line #771 in stack trace below

    // END critical section for buffer position

    return this;
}
---------------------------------------------------------------------

While the JavaFX Application Thread is iterating through the loop in the method above, the QuantumRenderer can call the getPixels() method in com.sun.glass.ui.Pixels, which sets the buffer's position to zero by rewinding it.

---------------------------------------------------------------------
public final Buffer getPixels() {
    if (this.bytes != null) {
        this.bytes.rewind();
        return this.bytes;
    } else if (this.ints != null) {
        this.ints.rewind(); // Line #164 in stack trace below
        return this.ints;
    } else {
        throw new RuntimeException("Unexpected Pixels state.");
    }
}
---------------------------------------------------------------------

The QuantumRenderer calls the getPixels() method while trying to find a buffer that's not in use, yet in doing so it can inadvertently modify a buffer that's in use. The call stacks of both threads when the problem occurs are as follows:

"JavaFX Application Thread"
at java.nio.IntBuffer.put(IntBuffer.java:771)
at com.sun.glass.ui.monocle.Framebuffer.composePixels(Framebuffer.java:168)
at com.sun.glass.ui.monocle.HeadlessScreen.uploadPixels(HeadlessScreen.java:118)
at com.sun.glass.ui.monocle.MonocleView._uploadPixels(MonocleView.java:95)
at com.sun.glass.ui.View.uploadPixels(View.java:771)
at com.sun.prism.PresentableState.uploadPixels(PresentableState.java:295)
at com.sun.javafx.tk.quantum.SceneState.access$001(SceneState.java:40)
at com.sun.javafx.tk.quantum.SceneState.lambda$uploadPixels$0(SceneState.java:123)
at com.sun.javafx.tk.quantum.SceneState$$Lambda$107.23582905.run
at com.sun.glass.ui.monocle.RunnableProcessor.runLoop(RunnableProcessor.java:92)
at com.sun.glass.ui.monocle.RunnableProcessor.run(RunnableProcessor.java:51)
at java.lang.Thread.run(Thread.java:748)

"QuantumRenderer-0"
at com.sun.glass.ui.Pixels.getPixels(Pixels.java:164)
at com.sun.prism.impl.QueuedPixelSource.usesSameBuffer(QueuedPixelSource.java:103)
at com.sun.prism.impl.QueuedPixelSource.getUnusedPixels(QueuedPixelSource.java:129)
at com.sun.javafx.tk.quantum.UploadingPainter.run(UploadingPainter.java:147)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at com.sun.javafx.tk.RenderJob.run(RenderJob.java:58)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:125)
at java.lang.Thread.run(Thread.java:748)

I encountered this problem when using a non-direct byte buffer in my own implementation of the NativeScreen interface for an electronic paper display. The following playlist contains a one-minute video showing the problem and a 10-second video of the same program with my suggested fix applied:

OpenJFX QuantumRenderer Bug (playlist)
https://www.youtube.com/playlist?list=PLvQ68hmt_qBMH0Usq0D41z-ZOkxGZHKIh

Direct links to the two videos in the playlist are as follows:

Before: Waving and Jumping Duke (one minute)
https://youtu.be/mfSzSMDIEIM

After: Waving Duke (10 seconds)
https://youtu.be/pUXPL_Gm8Qw


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
The simple JavaFX application in the NetBeans project called "jfxtest" on GitLab (https://gitlab.com/openjfxepd/jfxtest) can reproduce the problem in two ways:

1. The Monocle VNC platform lets you see the problem on screen.

2. The Monocle Headless platform lets you trap the problem with the Java debugger.

This is a timing problem, so it can be difficult to reproduce on fast systems. For example, I usually see the problem using the Monocle VNC platform within just a few seconds when running the test on the slow 32-bit Intel Atom processor, but it can take several minutes or longer to see the problem on the fast 64-bit Intel Xeon processor, if at all.

Even without actually seeing the problem with a VNC client, though, you can usually trap the error rather quickly with the debugger on any system using the larger screen size of the Monocle Headless platform.

I used the following Bash script to run the tests:

---------------------------------------------------------------------
#!/bin/bash
# Runs the JavaFX application on a Monocle platform
export JDK_HOME=$HOME/opt/jdk1.8.0_172

# Monocle platform and screen size
# Linux: Use framebuffer size (sudo fbset)
# Headless: 1280 × 800 px
# VNC: 1024 × 600 px
# EPD: 800 × 600 px
monocle_platform=VNC
width=1024
height=600

java_ext_dirs=$HOME/lib/ext:$JDK_HOME/jre/lib/ext

$JDK_HOME/bin/java -version

printf "Starting application on $monocle_platform platform ($width × $height px) ...\n"
# To debug remotely, add:
# -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y
$JDK_HOME/bin/java \
    -Djava.ext.dirs=$java_ext_dirs \
    -Dglass.platform=Monocle \
    -Dmonocle.platform=$monocle_platform -Dprism.order=sw \
    -jar $HOME/lib/jfxtest.jar --width=$width --height=$height
---------------------------------------------------------------------

For the VNC platform, I connected directly with the Remmina Remote Desktop Client using a color depth of "True color (24 bit)" and a quality of "Poor (fastest)," and I disabled encryption.

For the Headless platform, I set the breakpoint defined below to trap the error and stop all threads. The debugger traps the error within a few seconds in my tests.

File: jar:file:/.../javafx-src.zip!/com/sun/glass/ui/Pixels.java
Line number: 164
Condition: ints.position() > 0 && ints.position() < (1280 * 800)
Suspend: All threads

To see the buffer position while debugging with the Monocle Headless platform, I set the following watch variable:

ints.toString()

When the program stops at the breakpoint "Pixels.java:164", I saw values for the watch variable like the following:

java.nio.DirectIntBufferU[pos=5172 lim=1024000 cap=1024000]
java.nio.DirectIntBufferU[pos=852 lim=1024000 cap=1024000]
java.nio.DirectIntBufferU[pos=711 lim=1024000 cap=1024000]

The value of "pos" shows the source buffer position being used by the JavaFX Application Thread to copy the buffer just before the QuantumRenderer rewinds it, setting its position back to zero.


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The animated GIF below is a screencast of the expected result captured with the Byzanz screencast creator (https://git.gnome.org/browse/byzanz/). Notice that Duke is always centered in the window.

Expected result shown in a normal JavaFX window
https://gitlab.com/openjfxepd/jfxpatch/blob/master/doc/images/default.gif

ACTUAL -
The animated GIF below is a screencast of the actual result using the Monocle VNC platform and the Remmina Remote Desktop Client. Notice that Duke jumps around in the window when the buffer's position is reset by the QuantumRenderer while being copied by the JavaFX Application Thread.

Actual result on VNC platform
https://gitlab.com/openjfxepd/jfxpatch/blob/master/doc/images/vnc.gif

The three images below are screenshots of the actual results using the Monocle Headless platform and captured with the debugger by calling the WavingApp.saveFrame(IntBuffer) static method directly from the debugging session. Notice that Duke is not centered in the frame.

Actual result #1 on Headless platform
https://gitlab.com/openjfxepd/jfxpatch/blob/master/doc/images/headless1.png

Actual result #2 on Headless platform
https://gitlab.com/openjfxepd/jfxpatch/blob/master/doc/images/headless2.png

Actual result #3 on Headless platform
https://gitlab.com/openjfxepd/jfxpatch/blob/master/doc/images/headless3.png


---------- BEGIN SOURCE ----------
The test case is a simple JavaFX application that displays an animated GIF of Duke waving at about 8.3 frames per second. You can find the source code in "WavingApp.java" on GitLab (https://gitlab.com/openjfxepd/jfxtest/blob/master/src/org/status6/jfxtest/WavingApp.java). You can build the application with the NetBeans project called "jfxtest" on GitLab (https://gitlab.com/openjfxepd/jfxtest).

The associated "waving.gif" file can be found below, including the Makefile that builds it:

Animated GIF for the WavingApp JavaFX application
https://gitlab.com/openjfxepd/jfxtest/tree/master/images

For completeness, I have also included the full source code below:

---------------------------------------------------------------------
package org.status6.jfxtest;

import java.awt.image.BufferedImage;
import static java.awt.image.BufferedImage.TYPE_INT_ARGB_PRE;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.nio.IntBuffer;
import java.util.Map;
import javafx.application.Application;
import javafx.application.Platform;
import javafx.scene.Scene;
import javafx.scene.image.Image;
import javafx.scene.image.ImageView;
import javafx.scene.layout.Pane;
import javafx.scene.layout.StackPane;
import javafx.stage.Stage;
import javax.imageio.ImageIO;

/**
 * A simple JavaFX application to display an animated GIF of Duke waving.
 *
 * @author John Neffenger
 */
public class WavingApp extends Application {

    private static final String TITLE = "Waving Duke";
    private static final String IMAGE = "waving.gif";

    private static final String FORMAT = "png";
    private static final String PATHNAME = "frame.png";

    private static final String WIDTH_KEY = "width";
    private static final String WIDTH_DEFAULT = "800";
    private static final String HEIGHT_KEY = "height";
    private static final String HEIGHT_DEFAULT = "600";

    private static int width = 0;
    private static int height = 0;

    /**
     * Captures a frame buffer and saves it in PNG format as the file
     * <i>frame.png</i>. Call this method to capture the frame buffer in a
     * headless environment while debugging.
     *
     * @param buffer the integer frame buffer to capture
     */
    public static void saveFrame(IntBuffer buffer) {
        try {
            BufferedImage awtImage = new BufferedImage(width, height, TYPE_INT_ARGB_PRE);
            for (int x = 0; x < width; x++) {
                for (int y = 0; y < height; y++) {
                    awtImage.setRGB(x, y, buffer.get(y * width + x));
                }
            }
            ImageIO.write(awtImage, FORMAT, new File(PATHNAME));
        } catch (IOException e) {
            System.err.println(e);
        }
    }

    /**
     * Initializes the JavaFX application. Reads in the values of the width and
     * height parameters, if present. This method terminates the JavaFX
     * application if the value of the width or height is not an integer.
     */
    @Override
    public void init() {
        Application.Parameters parameters = getParameters();
        Map<String, String> map = parameters.getNamed();
        String w = map.getOrDefault(WIDTH_KEY, WIDTH_DEFAULT);
        String h = map.getOrDefault(HEIGHT_KEY, HEIGHT_DEFAULT);
        try {
            width = Integer.parseInt(w);
            height = Integer.parseInt(h);
        } catch (NumberFormatException e) {
            System.err.println(e);
            Platform.exit();
        }
    }

    /**
     * Starts the JavaFX application.
     *
     * @param stage the primary stage
     */
    @Override
    public void start(Stage stage) {
        InputStream input = WavingApp.class.getResourceAsStream(IMAGE);
        Image image = new Image(input);
        ImageView view = new ImageView(image);

        Pane pane = new StackPane();
        pane.getChildren().add(view);
        Scene scene = new Scene(pane, width, height);

        stage.setTitle(TITLE);
        stage.setScene(scene);
        stage.show();
    }

    /**
     * Launches the JavaFX application.
     *
     * @param args the optional named parameters, <em>width</em> and
     * <em>height</em>, providing the scene width and height in pixels. The
     * defaults values are 800 × 600 px (--width=800 --height=600). Note that
     * the Monocle VNC screen is 1024 × 600 px, while the Monocle Headless
     * screen is 1280 × 800 px.
     */
    public static void main(String[] args) {
        launch(args);
    }
}
---------------------------------------------------------------------

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
The workaround is to use a port of OpenJFX whose NativeScreen implementation allocates a direct buffer for its Framebuffer instead a non-direct buffer. A direct byte buffer ends up calling the put(IntBuffer) method in java.nio.DirectIntBufferU shown below. Notice that this method does not have the same critical section for the buffer's position found in the non-direct buffer implementation of the method in java.nio.IntBuffer.

---------------------------------------------------------------------
public IntBuffer put(IntBuffer src) {
    if (src instanceof DirectIntBufferU) {
        if (src == this)
            throw new IllegalArgumentException();
        DirectIntBufferU sb = (DirectIntBufferU)src;

        int spos = sb.position(); // Get buffer position
        int slim = sb.limit();
        assert (spos <= slim);
        int srem = (spos <= slim ? slim - spos : 0);

        int pos = position();
        int lim = limit();
        assert (pos <= lim);
        int rem = (pos <= lim ? lim - pos : 0);

        if (srem > rem)
            throw new BufferOverflowException();
        unsafe.copyMemory(sb.ix(spos), ix(pos), (long)srem << 2);
        sb.position(spos + srem);
        position(pos + srem);
    } else if (src.hb != null) {
        // ...
    } else {
        // ...
    }
    return this;
}
---------------------------------------------------------------------

The following fix solved the problem for me:

Compare buffers without rewinding them
https://gitlab.com/openjfxepd/jfxpatch/commit/f7c341775e5258e790a049f3fdce4a956ef665c7

For completeness, I have also included the full patch below:

---------------------------------------------------------------------
From f7c341775e5258e790a049f3fdce4a956ef665c7 Mon Sep 17 00:00:00 2001
From: John Neffenger <john@status6.com>
Date: Tue, 1 Nov 2016 15:51:43 -0700
Subject: [PATCH] Compare buffers without rewinding them

The QuantumRenderer calls the getPixels() method while trying to find a
buffer that's not in use, yet in doing so it can inadvertently rewind a
buffer in use by the JavaFX Application Thread. The stack of the call to
rewind() is:

"QuantumRenderer-0"
    at com.sun.glass.ui.Pixels.getPixels
        (Pixels.java:164)
    at com.sun.prism.impl.QueuedPixelSource.usesSameBuffer
        (QueuedPixelSource.java:103)
    at com.sun.prism.impl.QueuedPixelSource.getUnusedPixels
        (QueuedPixelSource.java:129)
    ...

Create a new method, getBuffer(), that returns the same ByteBuffer or
IntBuffer as getPixels() but does not rewind it, and change the method
usesSameBuffer(Pixels, Pixels) to call it instead of getPixels().

Fixes #1
---
 src/com/sun/glass/ui/Pixels.java | 13 +++++++++++++
 src/com/sun/prism/impl/QueuedPixelSource.java | 2 +-
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/src/com/sun/glass/ui/Pixels.java b/src/com/sun/glass/ui/Pixels.java
index 36c6c3e..d96bb24 100644
--- a/src/com/sun/glass/ui/Pixels.java
+++ b/src/com/sun/glass/ui/Pixels.java
@@ -169,6 +169,19 @@ public abstract class Pixels {
     }
 
     /*
+ * Return the original pixels buffer without rewinding it.
+ */
+ public final Buffer getBuffer() {
+ if (this.bytes != null) {
+ return this.bytes;
+ } else if (this.ints != null) {
+ return this.ints;
+ } else {
+ throw new RuntimeException("Unexpected Pixels state.");
+ }
+ }
+
+ /*
      * Return a copy of pixels as bytes.
      */
     public final ByteBuffer asByteBuffer() {
diff --git a/src/com/sun/prism/impl/QueuedPixelSource.java b/src/com/sun/prism/impl/QueuedPixelSource.java
index 617e5b9..854dff4 100644
--- a/src/com/sun/prism/impl/QueuedPixelSource.java
+++ b/src/com/sun/prism/impl/QueuedPixelSource.java
@@ -100,7 +100,7 @@ public class QueuedPixelSource implements PixelSource {
     private boolean usesSameBuffer(Pixels p1, Pixels p2) {
         if (p1 == p2) return true;
         if (p1 == null || p2 == null) return false;
- return (p1.getPixels() == p2.getPixels());
+ return (p1.getBuffer() == p2.getBuffer());
     }
 
     /**
--
libgit2 0.27.0
---------------------------------------------------------------------


FREQUENCY : often