There are a number of corner cases that need to be unit tests in the dialogs API. Here's a list of some areas that need testing (over and above the simple unit tests all API should get):

1) Tests to ensure that a forcibly closed dialog is handled correctly:
a) One, non-cancel button (dialog should close)
b) Multiple buttons, all non-cancel (dialog should not close)
c) Multiple buttons, with one cancel button (dialog should close)
d) Multiple buttons, with multiple cancel buttons (dialog should close, the first cancel button should be used)

2) Tests to ensure that a closed dialog returns the correct value, depending on its R type, including:
a) Test to see what result and showAndWait() return when R is Void
b) Test to see what result and showAndWait() return when R is ButtonType
c) Test to see what result and showAndWait() return when R is String, and no resultConverter has been specified (expect a class cast exception in the unit test)
d) Test to see what result and showAndWait() return when R is String, when a resultConverter has been specified that returns null (expect Optional.empty())
e) Test to see what result and showAndWait() return when R is String, when a resultConverter has been specified that returns a valid String

3) Test all cases above to ensure that non-blocking and blocking dialogs have the same result values given the same user input.