FULL PRODUCT VERSION :
java version "9.0.1"
Java(TM) SE Runtime Environment (build 9.0.1+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.1+11, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows (Version 6.1.7601)

A DESCRIPTION OF THE PROBLEM :
When removing javafx.scene.shape.Sphere-objects from a group or container with container.getChildren().clear() the memory is not freed by the garbage collector. This causes an out-of-memory-exception when a certain number of spheres is added.

The problem arises only when the spheres have different radii. If all spheres have the same radius the memory is freed in the proper way.

I have published this problem on stackoverflow.com, see: https://stackoverflow.com/questions/47518745/memory-leaks-when-removing-objects-in-javafx-3d)
  
One of the members has analyzed the issue and found the cause of the problem. It is most probably located in the javafx.scene.shape.PredefinedMeshManager-class. Each time a sphere (or another 3D shape) is created its associated TriangleMesh-object is added to a HashMap. For spheres the key of the HashMap is based on their radius and number of divisions. Thus, spheres with a different radius have different keys whereas spheres with the same radius have the same key (same number of divisions assumed). Therefore, adding spheres with different radii causes a continuous increase of the HashMap in contrast to spheres with identical radius. When the container is cleared by the user the PredefinedMeshManager DOES NOT CLEAR the HashMap as well. As a result the application is exhausting memory and finally crashs with an out-of-memory exception. In order to prevent this behavior the cache of the PredefinedMeshManager has to be freed when the container is cleared either automatically (i.e. triggered somewhere in the JavaFX code) or by the user via an appropriate method.




STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
- Run the code below
- Click the button several times and monitor the JVM-heap with an appropriate Java profiler e.g. Visual VM


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
- The garbage collector should free the allocated heap memory of all spheres no longer referenced
- No out-of-memory-exception should be thrown

ACTUAL -
- The memory of spheres no longer referenced is not released. Heap memory run out. An out-of-memory-exception is thrown.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
package application;

    import java.util.Random;
    import javafx.application.Application;
    import javafx.stage.Stage;
    import javafx.scene.Group;
    import javafx.scene.Scene;
    import javafx.scene.control.Button;
    import javafx.scene.layout.VBox;
    import javafx.scene.shape.Sphere;

    public class Main extends Application {
        @Override
        public void start(Stage primaryStage) {
            try {

                // --- User defined code -----------------
                VBox root = new VBox();
                Button btn = new Button("Add spheres...");
                Group container = new Group();
                root.getChildren().addAll(btn, container);
                btn.setOnAction(e -> {
                    createSpheres(container);
                });
                // ---------------------------------------
                Scene scene = new Scene(root,400,400);
                primaryStage.setScene(scene);
                primaryStage.show();
            } catch(Exception e) {
                e.printStackTrace();
            }
        }

        public static void main(String[] args) {
            launch(args);
        }

        // --- User defined code ------------------------------------------------------------------------
        // Each call increases the used memory although the container is cleared and the GC is triggered.
        // The problem does not occur when all spheres have the same radius.
        // ----------------------------------------------------------------------------------------------
        private void createSpheres(Group container) {
                container.getChildren().clear();
                Runtime.getRuntime().gc();
                Random random = new Random();
                for (int i = 0; i < 500; i++) {
                    //double d = 100; // OK
                    double d = 100 * random.nextDouble() + 1; // Problem
                    container.getChildren().add(new Sphere(d));
                }
                System.out.printf("Spheres added. Total number of spheres: %d. Used memory: %d Bytes of %d Bytes.\n",
                        container.getChildren().size(),
                        Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory(),
                        Runtime.getRuntime().maxMemory());
        }
        // ----------------------------------------------------------------------------------------------
    }
---------- END SOURCE ----------