Just a minor note: for now the user can add the same tag set multiple times, is it really necessary?

For example:

        TIFFImageReadParam param = new TIFFImageReadParam();
        TIFFTagSet ts = ExifGPSTagSet.getInstance();

        param.addAllowedTagSet(ts);
        // <some code ...>
        // accidentally, do it again
        param.addAllowedTagSet(ts);
        // <some code ...>
        param.removeAllowedTagSet(ts);
        
        // output: "true" - that may be confusing
        System.out.println(param.getAllowedTagSets().contains(ts));

checked with JDK9 b111