Below are some examples of the code tweaks that will help in extending the class to add features. It actually applies to many other places by using protected methods if you think people wants to have their subclass for the actual implementation.

TableViewSkin.java: this change will allow developers to use their own TableHeaderRow subclass.

Current:
        tableHeaderRow = new TableHeaderRow(tableView, flow);

Proposed:
        tableHeaderRow = createTableHeaderRow(tableView, flow);

    protected TableHeaderRow createTableHeaderRow(TableView tableView, VirtualFlow flow) {
        return new TableHeaderRow(tableView, flow);
    }

TableHeaderRow.java

Current:
        header = new NestedTableColumnHeader(table, null);

Proposed:
        header = createRootHeader(table); // you can choose the name

    protected NestedTableColumnHeader createRootHeader(TableView<?> table) {
        return new NestedTableColumnHeader(table, null);
    }

NestedTableColumnHeader.java: change the package local constructor to public so that it can be subclassed.

Current:
        TableColumnHeader newCol = col.getColumns().isEmpty() ?
            new TableColumnHeader(getTableView(), col) :
            new NestedTableColumnHeader(getTableView(), col);

Proposed:
        TableColumnHeader newCol = createTableColumnHeader(col);

    protected TableColumnHeader createTableColumnHeader(TableColumn col) {
        return col.getColumns().isEmpty() ?
                new TableColumnHeader(getTableView(), col) :
                new NestedTableColumnHeader(getTableView(), col);
    }