FULL PRODUCT VERSION :
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Linux 4.6.0-1-amd64 #1 SMP Debian 4.6.2-2 (2016-06-25) x86_64 GNU/Linux

EXTRA RELEVANT SYSTEM CONFIGURATION :
Linux 4.6.0, X.Org X Server 1.18.3, Intel driver 2.99.917-2 (kms).

Hardware: i5-3210M CPU, HD4000 integrated GPU, 12MB RAM.

A DESCRIPTION OF THE PROBLEM :
Multiple memory leaks, outside of JVM-allocated memory, when using JavaFX animations and transitions on Linux with hardware pipe. Using "-Dprism.order=sw" completely alleviates the problem.
Monitoring applications with visualvm shows minimal heap and metaspace usage (<0.5GB total), while process resident memory is >6GB.

Test cases:
1. Set progress indicator to indeterminate, as in JDK-8152426. That bug was marked as duplicate of JDK-8151165, but that is not the case! Running the test-case provided in 8151165 shows negligible leaks, while the one in 8152426 comes up to 4+ GB in less than a minute. Also, in the case of 8152426 the ProgressIndicator must be displayed - if it's not visible there is no leak. They are not the same bug.
2. Using any indefinite transition on a node, as described in JDK-8161911. This was closed as "not an issue" because the developer couldn't replicate it on the Oracle JVM. I have tried it on the Oracle JVM and the same thing happens - unreasonable memory is consumed, 4+GB in a minute, and doesn't stop until process is killed. again, visualvm shows neither the heap or the metaspace are accounting for the used memory. The same thing happens with many nodes, not just Labels. The bug is indeed an issue. Maybe developer ran with software pipe, which does not suffer from this issue, or maybe some configurations aren't affected.
3. The same problem has been reported in other places: https://ubuntuforums.org/showthread.php?t=2326006 - general inexplicable memory leaks with JavaFX Ensemble demo, on Ubuntu with AMD GPU. https://github.com/jfoenixadmin/JFoenix/issues/52 - Serious memory leak in SceneBuilder, again - with visualvm showing heap and metaspace not accountable for used memory.

Please do not dismiss this as "not an issue" or "duplicate" again, as this is a real issue that affects many people. If more information is needed please contact me (mail address is given with report).

This happens with both OpenJDK and Oracle versions.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Use Linux, make sure you are _not_ using software pipeline. Possibly use only Intel or AMD GPUs (confirmed affected versions are Intel HD4000 and AMD Radeon 6450).

Either place a ProgressIndicator with indeterminate progress, or apply an indefinite transition to any displayed node.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Memory usage should remain relatively low, similar to what happens when using software pipe.
ACTUAL -
Memory usage goes up very quickly, passing 4GB in a minute, and doesn't stop until animation is stopped or process is killed. The memory is not reported as either heap or metaspace memory in visualvm.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
See linked bug reports - both, I believe, wrongly closed.
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Don't use indefinite animations, don't use ProgressIndicator with indeterminate progress, or switch to using software pipeline (-Dprism.order=sw).