Expanded Summary:

double a%b returns NaN for some (a,b) (|a| < inf, |b|>0)
But this has so far only been observed on Windows, with 1.6-ish 64-bit VMs, on particular Intel CPUs.
It has been not-observed across a wide range of other OSes, VMs, and CPUs, but since the cause is unknown, it seems premature to say "can not reproduce".

(From Jeff Hain)

While testing some mathematical treatments,
I had "%" operator on doubles return NaN in
some cases where the divisor is subnormal
and the dividend huge.

More precisely, with
D(x) = Double.longBitsToDouble(x),

I had a%b being NaN for:

- all (a,b) such as
  D(0x7FE0000000000000L) <= |a| <= D(0x7FEFFFFFFFFFFFFFL)
  and
  D(0x0000000000040000L) <= |b| <= D(0x000000000007FFFFL)

- plus for some values of |b| outside this range,
  when |a| is D(0x7FEFFFFFFFFFFFFFL), like
  b = D(0x000000007FFFFFFFL) (but not for b = D(0x000000007FFEFFFFL))
  b = 6.767486E-317
  b = 7.528725E-318

Only happens with 64-bit VMs, also happens with -Xint and various flags.
------------------------------------------------------------------------------------------------------------------------------
Observed/not-observed on:

On Windows 7/Core i7 980X, it happens with the following 64 bits versions of Java:
- 1.5.0_22-b03
- 1.6.0_21-ea-b05
- 1.6.0_29-ea-b08 (and also crashes at some point but this version seems to crash a lot, so might be unrelated)
- 1.6.0_38-b05 ("Java HotSpot(TM) 64-Bit Server VM")
but not with:
- 1.7.0_11-b21
- 1.8.0-ea-b73

On Windows XP/T2300, it does not happen with:
- 1.6.0_38
- 1.6.0_39
- 1.7.0_10
------------------------------------------------------------------------------------------------------------------------------
I have some more non-results, MacOS 10.8.2, Core i5, does not happen with:

java version "1.6.0_37"
Java(TM) SE Runtime Environment (build 1.6.0_37-b06-434-11M3909)
Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01-434, mixed mode)

java version "1.7.0_13"
Java(TM) SE Runtime Environment (build 1.7.0_13-b20)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

openjdk version "1.8.0-internal-fastdebug" (near-current jdk8/tl)
OpenJDK Runtime Environment (build 1.8.0-internal-fastdebug-dr2chase_2013_02_08_16_15-b00)
OpenJDK 64-Bit Server VM (build 25.0-b17-fastdebug, mixed mode)

Each above used corresponding javac to compiler ModNaN.java, and plain "java" to test.

Also tried 7 and 8, class files from 1.6 javac, running java with flags:

-XX:+TieredCompilation -XX:TieredStopAtLevel=1
-XX:+TieredCompilation -XX:TieredStopAtLevel=2

Also tried on MacOS 10.8.2, Core i7, does not happen there with:

java version "1.6.0_37"
Java(TM) SE Runtime Environment (build 1.6.0_37-b06-434-11M3909)
Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01-434, mixed mode)

------------------------------------------------------------------------------------------------------------------------------
Tried Windows 7, core i5, JDK1.7.0_21 and JDK1.6.0_43, with -Xcomp and -Xint and all combinations of
-XX:UseSSE={0,1,2,3,4,99} and -XX:UseAVX={0,1,2,3,4,99}
------------------------------------------------------------------------------------------------------------------------------

Test code w/ good and bad outputs in comments:

public class ModNaN {
    /* Windows 7/Core i7 980X/1.6.0_38 (64):
NaNs
8.98846567431158E307 % 1.295163E-318 = NaN
1.7976931348623157E308 % 2.59032E-318 = NaN
1.7976931348623157E308 % 1.060997895E-314 = NaN
1.7976931348623157E308 % 6.767486E-317 = NaN
1.7976931348623157E308 % 7.528725E-318 = NaN
non-NaNs
8.98846567431158E307 % 1.29516E-318 = 2.53E-321
1.7976931348623157E308 % 2.590327E-318 = 0.0
1.7976931348623157E308 % 1.060965516E-314 = 9.35818525E-315
     */
    /* Windows 7/Core i7 980X/1.7.0_11 (64):
NaNs
8.98846567431158E307 % 1.295163E-318 = 0.0
1.7976931348623157E308 % 2.59032E-318 = 2.57135E-318
1.7976931348623157E308 % 1.060997895E-314 = 5.31535078E-315
1.7976931348623157E308 % 6.767486E-317 = 1.142612E-317
1.7976931348623157E308 % 7.528725E-318 = 4.64634E-318
non-NaNs
8.98846567431158E307 % 1.29516E-318 = 2.53E-321
1.7976931348623157E308 % 2.590327E-318 = 0.0
1.7976931348623157E308 % 1.060965516E-314 = 9.35818525E-315
     */
    public static void main(String[] args) {
        System.out.println("NaNs");
        for (double[] ab : new double[][]{
                new double[]{Double.longBitsToDouble(0x7FE0000000000000L),Double.longBitsToDouble(0x0000000000040000L)},
                new double[]{Double.longBitsToDouble(0x7FEFFFFFFFFFFFFFL),Double.longBitsToDouble(0x000000000007FFFFL)},
                //
                new double[]{Double.longBitsToDouble(0x7FEFFFFFFFFFFFFFL),Double.longBitsToDouble(0x000000007FFFFFFFL)},
                new double[]{Double.longBitsToDouble(0x7FEFFFFFFFFFFFFFL),6.767486E-317},
                new double[]{Double.longBitsToDouble(0x7FEFFFFFFFFFFFFFL),7.528725E-318},
        }) {
            double a = ab[0];
            double b = ab[1];
            double mod = a % b;
            System.out.println(a+" % "+b+" = "+mod);
        }

        System.out.println("non-NaNs");
        for (double[] ab : new double[][]{
                new double[]{Double.longBitsToDouble(0x7FE0000000000000L),Double.longBitsToDouble(0x000000000003FFFFL)},
                new double[]{Double.longBitsToDouble(0x7FEFFFFFFFFFFFFFL),Double.longBitsToDouble(0x0000000000080000L)},
                //
                new double[]{Double.longBitsToDouble(0x7FEFFFFFFFFFFFFFL),Double.longBitsToDouble(0x000000007FFEFFFFL)},
        }) {
            double a = ab[0];
            double b = ab[1];
            double mod = a % b;
            System.out.println(a+" % "+b+" = "+mod);
        }
------------------------------------------------------------------------------------------------------------------------------