FULL PRODUCT VERSION :
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
--------------------------
java version "9"
Java(TM) SE Runtime Environment (build 9+181)
Java HotSpot(TM) 64-Bit Server VM (build 9+181, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
System 1:
uname -a
Linux <xxx> 4.4.85-22-default #1 SMP Fri Sep 1 14:21:21 UTC 2017 (0c39a1f) x86_64 x86_64 x86_64 GNU/Linux

System 2:
uname -a
Linux <xxx> 4.13.1-1.gc0b7e1f-default #1 SMP PREEMPT Wed Sep 13 06:24:42 UTC 2017 (c0b7e1f) x86_64 x86_64 x86_64 GNU/Linux


(Observed on OpenSuse 13.2, 42.2, 42.3 and Tumbleweed)

EXTRA RELEVANT SYSTEM CONFIGURATION :

System 1:
glxinfo |grep Open

Vendor: Intel Open Source Technology Center (0x8086)
OpenGL vendor string: Intel Open Source Technology Center
OpenGL renderer string: Mesa DRI Intel(R) Haswell Mobile
OpenGL core profile version string: 4.5 (Core Profile) Mesa 17.0.5
OpenGL core profile shading language version string: 4.50
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 3.0 Mesa 17.0.5
OpenGL shading language version string: 1.30
OpenGL context flags: (none)
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.1 Mesa 17.0.5
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.10

-----

System 2:
glxinfo | grep Open
 Vendor: Intel Open Source Technology Center (0x8086)
OpenGL vendor string: Intel Open Source Technology Center
OpenGL renderer string: Mesa DRI Intel(R) HD Graphics 520 (Skylake GT2)
OpenGL core profile version string: 3.3 (Core Profile) Mesa 11.2.2
OpenGL core profile shading language version string: 3.30
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 3.0 Mesa 11.2.2
OpenGL shading language version string: 1.30
OpenGL context flags: (none)
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.1 Mesa 11.2.2
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.10

----
The behavior is observed on at least 4 different computers.


A DESCRIPTION OF THE PROBLEM :
JavaFX leaks memory outside heap. JMC, JProfiler and other tools report low heap usage.
The memory leak grows until entire system becomes unresponsive. On some systems Xorg consumes a lot of CPU.

Profiling with jemalloc seems to indicates the problem is related to the use of XCB:

Jmalloc result running demo application:
https://2.bp.blogspot.com/-FUXDLi6W_lg/WcDeepQ0J-I/AAAAAAAAAyY/9AFBkv6isWgwkto_zYM_DvpSkwIYAdgggCLcBGAs/s1600/jeprof-java8-demo.gif

Test steps using jmalloc on demo application and real world application:
http://systout.blogspot.no/2017/09/export-ldpreloadusrlib64libjemalloc.html


Log from top running Demo:
TIME+ VIRT RES %CPU %MEM
0:30.63 5877012 1.733g 80.00 22.49
0:48.97 7220500 2.874g 93.33 37.30
1:06.84 8553588 4.017g 75.00 52.13
1:26.42 9935.1m 5.364g 94.12 69.61
1:27.86 9.851g 5.414g 0.000 70.26
---> System unresponsive.



STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run any JavaFX application that updates labels/controls/images frequently.

Running jmalloc on Demo:
- Save file to Demo.java
- javac Demo.java
- export LD_PRELOAD=/usr/lib64/libjemalloc.so.2
- export MALLOC_CONF=prof:true,lg_prof_interval:30,lg_prof_sample:17
- java Demo
- jeprof --show_bytes --gif /usr/local/java/jdk1.8.0_144/bin/javaws jeprof*.heap >jeprof-demo.gif

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
JavaFX application should run for days and weeks.
ACTUAL -
Crashes after a short time, and in some cases forces reboot of system.
We do not see this behavior on Windows or Mac.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import javafx.animation.Animation;
import javafx.animation.TranslateTransition;
import javafx.application.Application;
import javafx.scene.Node;
import javafx.scene.Scene;
import javafx.scene.control.Label;
import javafx.scene.layout.StackPane;
import javafx.stage.Stage;
import javafx.util.Duration;

public class Demo extends Application{
   //Borrowed from https://bugs.openjdk.java.net/browse/JDK-8161911

    @Override
    public void start(Stage primaryStage) throws Exception {
        Label l = new Label("A");
        StackPane sp = new StackPane(l);
        primaryStage.setScene(new Scene(sp, 200, 200));
        primaryStage.show();
        addTranslate(l);
    }

    private void addTranslate(Node target) {
        TranslateTransition tt = new TranslateTransition(Duration.millis(3000), target);
        tt.setByX(50);
        tt.setAutoReverse(true);
        tt.setCycleCount(Animation.INDEFINITE);
        tt.play();
    }
    
    public static void main(String[] args) {
        launch(args);
    }

}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
None. We rely on Java Web Start to distribute the application inside the organsation. Setting -Dprism.order=sw in JNLP file does not seem the be honored.

If the bug can not be solved, it should be considered to allow forcing software rendering from JNLP.