A TabPane preferred size can only grow, it shall be able to reduce when the content change.
Changing maxw and maxh to local variables in TabPaneSkin would solve the issue.

Sample code:

public class TabPaneSample extends Application{
Button growButton = new Button("Grow");
VBox vbox = new VBox(growButton);
Tab tab = new Tab("tab");
TabPane tabPane = new TabPane();

@Override
public void start(Stage primaryStage) throws Exception {
// Create Scene
// Stage -> TabPane -> Tab -> VBox -> Button
vbox.setPrefWidth(200);
tabPane.getTabs().add(tab);
tab.setContent(vbox);
primaryStage.setScene(new Scene(tabPane));

// Button's action
// growButton add a button that remove itself
growButton.setOnAction(e -> {
Button shrinkButton = new Button("Shrink");
shrinkButton.setOnAction(event -> {
vbox.getChildren().remove(shrinkButton);
tabPane.requestLayout();
primaryStage.sizeToScene();
});
vbox.getChildren().add(shrinkButton);
tabPane.requestLayout();
primaryStage.sizeToScene();
});

primaryStage.show();
}

public static void main(String[] args) {
launch(args);
}
}