_rescan_task_size and _marking_task_size are initialized in CompactibleFreeListSpace constructor(hotspot/src/share/vm/gc/cms/compactibleFreeListSpace.cpp module):
CompactibleFreeListSpace::CompactibleFreeListSpace(BlockOffsetSharedArray* bs, MemRegion mr) :
...
  _rescan_task_size(CardTableModRefBS::card_size_in_words * BitsPerWord *
                    CMSRescanMultiple),
  _marking_task_size(CardTableModRefBS::card_size_in_words * BitsPerWord *
                    CMSConcMarkMultiple),
...

Big values for CMSRescanMultiple and CMSConcMarkMultiple can cause overflow. In this case _rescan_task_size and _marking_task_size can become zero. This can lead to division by zero since these fields are used in denominator.

This can be showed by CMSRescanMultiple:
java -XX:+UseConcMarkSweepGC -XX:CMSRescanMultiple=9223372036854775808 TestGCOld 10 10 10 10 10000
...
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGFPE (0x8) at pc=0x00007f4771c48a0b, pid=29407, tid=29412
#
# JRE version: Java(TM) SE Runtime Environment (9.0) (fastdebug build 9-internal+0-2016-03-10-011512.daholme.jdk9-hs-rt)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 9-internal+0-2016-03-10-011512.daholme.jdk9-hs-rt, mixed mode, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0x8b4a0b] CompactibleFreeListSpace::initialize_sequential_subtasks_for_rescan(int)+0x7b
#
...

hs_err_pid29407.log is attached.