When attaching Listeners to the getSelectedItems() ObservableList, only additions to the selected items (ie. selecting a new item) are notified, never removals (ie. deselecting an item or changing from an item to another should throw a remove for the not-anymore-selected item, and then an add for the new one).
The following code illustrates the problem nicely:

import javafx.application.Application;
import javafx.collections.FXCollections;
import javafx.collections.ListChangeListener;
import javafx.collections.ObservableList;
import javafx.geometry.Rectangle2D;
import javafx.scene.Scene;
import javafx.scene.control.ListView;
import javafx.scene.control.SelectionMode;
import javafx.scene.layout.StackPane;
import javafx.scene.paint.Color;
import javafx.stage.Screen;
import javafx.stage.Stage;

public final class selectedItemsTest extends Application {
public static void main(final String[] args) {
// Launch the JavaFX application: do initialization and call start()
// when ready.
Application.launch(args);
}

@Override
public void start(final Stage primaryStage) throws Exception {
final StackPane stack = new StackPane();
final Rectangle2D screen = Screen.getPrimary().getVisualBounds();
final Scene rootScene = new Scene(stack, screen.getWidth(), screen.getHeight(), Color.GRAY);

final ObservableList<String> inputStreams = FXCollections.observableArrayList("1", "2", "3", "4", "5");

final ListView<String> streams = new ListView<>();
stack.getChildren().add(streams);

streams.setItems(inputStreams);
streams.getSelectionModel().setSelectionMode(SelectionMode.MULTIPLE);

final ObservableList<String> selectedInputStreams = streams.getSelectionModel().getSelectedItems();

selectedInputStreams.addListener(new ListChangeListener<String>() {
@Override
public void onChanged(final Change<? extends String> change) {
while (change.next()) {
for (final String removed : change.getRemoved()) {
System.out.println("Removed: " + removed);
}

for (final String added : change.getAddedSubList()) {
System.out.println("Added: " + added);
}
}

if (!change.getList().isEmpty()) {
for (final String inList : change.getList()) {
System.out.println("In list currently: " + inList);
}
}
}
});

primaryStage.setTitle("TEST");
primaryStage.setScene(rootScene);

primaryStage.show();
}
}

You'll notice that the list itself (getList()) is alwasy correct and reflects removals correctly, they just never get notified to the listeners, which makes it impossible to update other data-structures correctly from a listener.
The temporary workaround I'm using is to clear() and then addAll(change.getList()), since the content of the list is, as said above, always accurate.
This might be partially related to RT-24367, as in making getSelectedItems() a separate ObservableList in its own right from the getItems() one would solve RT-24367, and adding a Listener to remove items removed from getItems() from the getSelectedItems() list too, and add/remove from getSelectedItems() based on the UI selection, would keep behavior consistent and fix the above.
I'm on JDK 1.8.0-ea-b102, I hope to have selected the right version. ;-)