ADDITIONAL SYSTEM INFORMATION :
Oracle JDK version is:

 java version "1.8.0_60"
 Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
 Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

A DESCRIPTION OF THE PROBLEM :
Please refer to the 'Pause GC Duration' graph in this GC log analysis report https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMTAvLS0xMS4zLjE4Ni4xMzUtZ2MtMjAyMS0wOS0wNl8xNi00My0wOC5sb2ctLTUtMjUtMTk=&channel=WEB

You can notice that Young GC pause time increasing cumulatively over a period of time (Sep' 10 midnight to 6am). Here is the GC log file during this problematic window: https://drive.google.com/file/d/1gjEmgMyfklm1-8ydL4mo2rR3HHKi9INL/view?usp=sharing

Even though 'sys' time is close to 0, only 'user' time keeps increasing.

Here are the arguments passed to the application:

-Xms30G
-Xmx30G
-Xmn20g
-server
-Xss256K
-XX:SurvivorRatio=8
-XX:+UseConcMarkSweepGC
-XX:+UseCMSCompactAtFullCollection
-XX:CMSInitiatingOccupancyFraction=70
-XX:+CMSParallelRemarkEnabled
-XX:SoftRefLRUPolicyMSPerMB=0
-XX:CMSMaxAbortablePrecleanTime=20
-XX:-OmitStackTraceInFastThrow
-XX:MaxDirectMemorySize=256G
-Dio.netty.maxDirectMemory=10737418240
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintReferenceGC
-XX:+PrintHeapAtGC
-Xloggc:/export/Logs/jmq4-server/gc-%t.log
-Dfile.encoding=UTF-8