If pressing on the ColorPicker, then a ColorPicker-menu shows up. At the bottom of this menu there is the option for creating custom colors.

The "custom color" dialog is always placed at the right of the ColorPicker-menu. If the menu is on the very right of the screen, then the "custom color" dialog is still painted right of the menu - so it is opened in an invisible area, outside the screen.

Because the dialog is a modal one - you cannot close it anymore. You have to close the JavaFX application through the task manager...