It appears the j.u.Deflater compression functionality is broken when

 (1) level/strategy is changed (to new value) + (2) reset,()

which is a normal combination/use scenario when the deflater is being cached and reused. The attached test case fails with 1.2.11 (with data size > the DeflaterOutputStream's internal buffer size).

My reading of the 1.2.11 changes suggests the direct trigger is that the internal state "deflate_state.high_water" is not being reset correctly/appropriately in deflateReset/deflateResetKeep/lm_init().

It triggers a regression from 1.2.8 to 1.2.11 in the scenario that the "strategy/levle is changed" AND a followup "reset" is called (withe the assumption that everything should be reset back to initial state and we start a new compression circle, with the new level and strategy. This is how it worked in 1.2.8). Because one of the condition check is changed from "strm->total_in!=0" to "s->high_water" (for the flush last buffer" branch) in deflate.c/deflateParams().

from (1.2.8's)

if ((strategy != s->strategy || func != configuration_table[level].func) &&
    strm->total_in != 0) {
    /* Flush the last buffer: */
    err = deflate(strm, Z_BLOCK);
    if (err == Z_BUF_ERROR && s->pending == 0)
        err = Z_OK;
}

to (1.2.11's)

if ((strategy != s->strategy || func != configuration_table[level].func) &&
    s->high_water) {
    /* Flush the last buffer: */
    int err = deflate(strm, Z_BLOCK);
    if (err == Z_STREAM_ERROR)
        return err;
    if (strm->avail_out == 0)
        return Z_BUF_ERROR;
}

Given the nature of "reset" it seems reasonable to assume the "high_water" should be reset to 0, its initial value as well, inside either deflateResetKeep() or lm_init().

I have logged an issue at

https://github.com/madler/zlib/issues/275

Given jdk9 has been changed to uses the system zlib for all platforms except the windows, this one currently only fails on windows (but I would assume it will fails if the underlying system zlib is 1.2.11)

-------------------------------- test case ----------------------------------------------------
   public static void main(final String[] args) throws IOException {
        Path p = Paths.get("jms.compression.data.large");
        int[] levels = new int[] {
            Deflater.DEFAULT_COMPRESSION,
            0,
            Deflater.BEST_SPEED,
            // 2, 3, 4, 5, 6, 7, 8,
            Deflater.BEST_COMPRESSION,
        };
        try {
byte[] src = Files.readAllBytes(p);
            if (src.length == 0) return;

            Deflater def = new Deflater(Deflater.DEFAULT_COMPRESSION); //, true);
            for (int level : levels) {
                ByteArrayOutputStream baos = new ByteArrayOutputStream();
                System.out.println(" -------------- level:" + level + " ------------------");

def.setLevel(level);
                def.reset();

try (DeflaterOutputStream defos = new DeflaterOutputStream(baos, def)) {
                    defos.write(src);
                }
                byte[] compressed = baos.toByteArray();

ByteArrayInputStream bais = new ByteArrayInputStream(compressed);
                Inflater inf = new Inflater();
                try (InflaterInputStream infis = new InflaterInputStream(bais, inf)) {
                    byte[] bytes = infis.readAllBytes();
                    if (!Arrays.equals(src, bytes)) {
                        System.out.println(" ==> NG");
                        new Exception().printStackTrace();
                    } else {
                        System.out.println(" ==> OK");
                    }
                } catch (Exception x) {
                    System.out.println(" ==> NG");
                    x.printStackTrace();
                }
            }
} catch (IOException x) {
            x.printStackTrace();
}
}