If I add new children to a Pane and afterwards delete them one by one, some of them still stay in memory even if there is no reference to them

HBox main = new HBox();
Set<Button> buttons = new HashSet<Button>();

for (int i = 0; i < 100; i++) {
Button b = new Button();
buttons.add(b);
main.getChildren().add(b);
}

for (Button b : buttons) {
main.getChildren().remove(b);
}

buttons.clear();

in aforementioned code 20 from 100 Buttons stay uncleaned. If I use main.getChildren().clear() everything works fine.