C2: Masked byte comparisons with large masks produce wrong result on x86

Whilst testing some fairly simple code in eXist-db which does VBE (Variable Byte Encoding) of java.lang.short values to/from byte[], we discovered that the tests would always pass on JDKs 8 through 11, but fail in what appeared to be a random intermittent manner on JDKs 12 through 15 (with HotSpot).

A deeper investigation, enabled me to extract the minimum reproducible code into a standalone test example. We have subsequently tested all adoptOpenJDK from 8 through 14 with both HotSpot and J9, we have also tested JDK 15 with HotSpot from java.net.

1. On JDK versions from 8 to 11 with either HotSpot or J9, the code always runs correctly.
2. On JDK 12+ HotSpot versions the code often fails in a non-deterministic manner, sometimes it fails on the 2nd, 3rd, or 4th iteration. It performs upto 10 iterations by default. Sometimes it will even pass.
3. The code always passes on JDK 12+ J9 versions.
4. Disabling C2 on HotSpot by setting `-XX:TieredStopAtLevel=3` causes the code to pass correctly.
Therefore we suspect a regression in HotSpot in Java 12, which has propagated to 13, 14, and now 15 also.

The test code itself is entirely deterministic - constant input values, no threads, no random, and no system time. The code with instructions for reproducing is available from: https://github.com/adamretter/vbe-test. The results of my testing against all the JDKs is visible here: https://github.com/adamretter/vbe-test#testing-results. There are also Travis-CI results here: https://travis-ci.com/github/adamretter/vbe-test.

We consider this to be a critical issue, as the VBE is used in the persistent storage layer of the eXist-db database. The failure of the tests indicate that data corruption will likely occur when it is run on Java HotSpot 12+.

Many thanks to the AdoptOpenJDK Slack group, especially Ben Evans, MBien, and Patrick Reinhart, who helped me pin-point the issue towards HotSpot and C2.