Spec says:
To remove a color from Custom Colors area, users can right click the color and choose Delete action from the pop-up menu or >simply press DELETE key when the color is focused<.

DELETE key pressing doesn't work.