The code emitter for the cmpdi instruction (compare immediate) is called with an out-of-range immediate value. The instruction takes 16bit signed immediates.

The issue is revealed by jtreg test compiler/rtm/locking/TestRTMAbortRatio. It specifies RTM parameters such that the immediate value that goes into the failing instruction exceeds the allowed range. Specifically, the following setting triggers the assert failure:
  -XX:RTMLockingThreshold=<x> -XX:RTMTotalCountIncrRate=<y>
Any combination where <x>/<y> yields a value outside the range [-32768..32767] will fail. The faulty code is in macroAssembler_ppc.cpp (rtm_abort_ratio_calculation).

This issue will surface only on machines with Power8 or newer processors. In addition, the operating system must support RTM (Linux: kernel version > 4.1, AIX: Rel. >= 7.2).

There is a similar issue in x86 code (same function). It's only similar because it will not immediately cause an assert failure or the like. In addition, the above mentioned quotient must exceed +/-2**31. Then, the quotient (calculated as 64bit int) will be truncated to 32bit and that value be used for comparison.