java version "10-ea"
Java(TM) SE Runtime Environment (build 10-ea+26)
Java HotSpot(TM) 64-Bit Server VM (build 10-ea+26, mixed mode)

OS Version:
Oracle Linux Server release 7.0

This issue observed during SBR(Same Binary Run) of client sanity tests. Error trace from log file.

----------System.err:(4/223)----------
[xcb] Extra reply data still left in queue
[xcb] This is most likely caused by a broken X extension library
[xcb] Aborting, sorry about that.
java: xcb_io.c:576: _XReply: Assertion `!xcb_xlib_extra_reply_data_left' failed.

After this error this test exited with error code 134.
result: Failed. Unexpected exit from test [exit code: 134]

After this error all tests started to fail, 3 tests (WindowDemoTest, OptionPaneDemoTest, DialogDemoTest) failed with this error and remaining tests failed with deadlock.

Deadlock traces:
Running jstack on process 25601
2017-10-17 00:41:51
Full thread dump Java HotSpot(TM) 64-Bit Server VM (10-ea+26 mixed mode):

"Attach Listener" #20 daemon prio=9 os_prio=0 tid=0x00007f48e4001000 nid=0x6bf9 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"AWT-EventQueue-0" #18 prio=6 os_prio=0 tid=0x00007f48bc398000 nid=0x642a runnable [0x00007f48c0e54000]
   java.lang.Thread.State: RUNNABLE
at sun.awt.X11.XInputMethod.openXIMNative(java.desktop@10-ea/Native Method)
at sun.awt.X11.XInputMethod.openXIM(java.desktop@10-ea/XInputMethod.java:64)
at sun.awt.X11InputMethod.initXIM(java.desktop@10-ea/X11InputMethod.java:196)
- locked <0x00000006d951e700> (a sun.awt.X11.XInputMethod)
at sun.awt.X11InputMethod.<init>(java.desktop@10-ea/X11InputMethod.java:179)
at sun.awt.X11.XInputMethod.<init>(java.desktop@10-ea/XInputMethod.java:49)
at sun.awt.X11.XInputMethodDescriptor.createInputMethod(java.desktop@10-ea/XInputMethodDescriptor.java:38)
at sun.awt.im.InputContext.getInputMethodInstance(java.desktop@10-ea/InputContext.java:855)
at sun.awt.im.InputContext.getInputMethod(java.desktop@10-ea/InputContext.java:805)
- locked <0x00000006d95143b8> (a sun.awt.im.InputMethodContext)
at sun.awt.im.InputContext.dispatchEvent(java.desktop@10-ea/InputContext.java:240)
at sun.awt.im.InputMethodContext.dispatchEvent(java.desktop@10-ea/InputMethodContext.java:196)
at java.awt.Component.dispatchEventImpl(java.desktop@10-ea/Component.java:4903)
at java.awt.Container.dispatchEventImpl(java.desktop@10-ea/Container.java:2317)
at java.awt.Component.dispatchEvent(java.desktop@10-ea/Component.java:4793)
at java.awt.KeyboardFocusManager.redispatchEvent(java.desktop@10-ea/KeyboardFocusManager.java:1955)
at java.awt.DefaultKeyboardFocusManager.typeAheadAssertions(java.desktop@10-ea/DefaultKeyboardFocusManager.java:1007)
at java.awt.DefaultKeyboardFocusManager.dispatchEvent(java.desktop@10-ea/DefaultKeyboardFocusManager.java:631)
at java.awt.Component.dispatchEventImpl(java.desktop@10-ea/Component.java:4842)
at java.awt.Container.dispatchEventImpl(java.desktop@10-ea/Container.java:2317)
at java.awt.Component.dispatchEvent(java.desktop@10-ea/Component.java:4793)
at java.awt.EventQueue.dispatchEventImpl(java.desktop@10-ea/EventQueue.java:777)
at java.awt.EventQueue.access$600(java.desktop@10-ea/EventQueue.java:97)
at java.awt.EventQueue$4.run(java.desktop@10-ea/EventQueue.java:726)
at java.awt.EventQueue$4.run(java.desktop@10-ea/EventQueue.java:720)
at java.security.AccessController.doPrivileged(java.base@10-ea/Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(java.base@10-ea/ProtectionDomain.java:89)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(java.base@10-ea/ProtectionDomain.java:99)
at java.awt.EventQueue$5.run(java.desktop@10-ea/EventQueue.java:750)
at java.awt.EventQueue$5.run(java.desktop@10-ea/EventQueue.java:748)
at java.security.AccessController.doPrivileged(java.base@10-ea/Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(java.base@10-ea/ProtectionDomain.java:89)
at java.awt.EventQueue.dispatchEvent(java.desktop@10-ea/EventQueue.java:747)
at java.awt.EventDispatchThread.pumpOneEventForFilters(java.desktop@10-ea/EventDispatchThread.java:203)
at java.awt.EventDispatchThread.pumpEventsForFilter(java.desktop@10-ea/EventDispatchThread.java:124)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(java.desktop@10-ea/EventDispatchThread.java:113)
at java.awt.EventDispatchThread.pumpEvents(java.desktop@10-ea/EventDispatchThread.java:109)
at java.awt.EventDispatchThread.pumpEvents(java.desktop@10-ea/EventDispatchThread.java:101)
at java.awt.EventDispatchThread.run(java.desktop@10-ea/EventDispatchThread.java:90)

"AWT-Shutdown" #19 prio=5 os_prio=0 tid=0x00007f48bc397000 nid=0x6429 in Object.wait() [0x00007f48c0f59000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@10-ea/Native Method)
- waiting on <0x00000006ca8c2f88> (a java.lang.Object)
at java.lang.Object.wait(java.base@10-ea/Object.java:516)
at sun.awt.AWTAutoShutdown.run(java.desktop@10-ea/AWTAutoShutdown.java:291)
- waiting to re-lock in wait() <0x00000006ca8c2f88> (a java.lang.Object)
at java.lang.Thread.run(java.base@10-ea/Thread.java:844)

"AWT-XAWT" #16 daemon prio=6 os_prio=0 tid=0x00007f48bc2dc800 nid=0x6427 waiting on condition [0x00007f48c135b000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@10-ea/Native Method)
- parking to wait for <0x00000006ca6b16e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@10-ea/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@10-ea/AbstractQueuedSynchronizer.java:883)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@10-ea/AbstractQueuedSynchronizer.java:915)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@10-ea/AbstractQueuedSynchronizer.java:1237)
at java.util.concurrent.locks.ReentrantLock.lock(java.base@10-ea/ReentrantLock.java:267)
at sun.awt.SunToolkit.awtLock(java.desktop@10-ea/SunToolkit.java:192)
at sun.awt.X11.XToolkit.waitForEvents(java.desktop@10-ea/Native Method)
at sun.awt.X11.XToolkit.run(java.desktop@10-ea/XToolkit.java:588)
at sun.awt.X11.XToolkit.run(java.desktop@10-ea/XToolkit.java:552)
at java.lang.Thread.run(java.base@10-ea/Thread.java:844)

"Java2D Disposer" #15 daemon prio=10 os_prio=0 tid=0x00007f48bc2d8800 nid=0x6426 in Object.wait() [0x00007f48c145c000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@10-ea/Native Method)
- waiting on <0x00000006ca8e12c8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@10-ea/ReferenceQueue.java:151)
- waiting to re-lock in wait() <0x00000006ca8e12c8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@10-ea/ReferenceQueue.java:172)
at sun.java2d.Disposer.run(java.desktop@10-ea/Disposer.java:144)
at java.lang.Thread.run(java.base@10-ea/Thread.java:844)

"MainThread" #13 prio=5 os_prio=0 tid=0x00007f49743c5800 nid=0x6425 in Object.wait() [0x00007f48e90a7000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@10-ea/Native Method)
- waiting on <0x00000006d9126f80> (a org.netbeans.jemmy.QueueTool$1JemmyInvocationLock)
at java.lang.Object.wait(java.base@10-ea/Object.java:516)
at org.netbeans.jemmy.QueueTool.invokeAndWait(QueueTool.java:434)
- waiting to re-lock in wait() <0x00000006d9126f80> (a org.netbeans.jemmy.QueueTool$1JemmyInvocationLock)
at org.netbeans.jemmy.QueueTool.invokeSmoothly(QueueTool.java:370)
at org.netbeans.jemmy.ComponentSearcher.findComponent(ComponentSearcher.java:163)
at org.netbeans.jemmy.operators.ContainerOperator$1.actionProduced(ContainerOperator.java:302)
at org.netbeans.jemmy.operators.ContainerOperator$1.actionProduced(ContainerOperator.java:299)
at org.netbeans.jemmy.Waiter.actionProduced(Waiter.java:229)
at org.netbeans.jemmy.Waiter.waitAction(Waiter.java:203)
at org.netbeans.jemmy.operators.ContainerOperator.waitSubComponent(ContainerOperator.java:319)
at org.netbeans.jemmy.operators.JButtonOperator.<init>(JButtonOperator.java:74)
at org.netbeans.jemmy.operators.JButtonOperator.<init>(JButtonOperator.java:86)
at ButtonDemoTest.test(ButtonDemoTest.java:108)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@10-ea/Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@10-ea/NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@10-ea/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@10-ea/Method.java:564)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:639)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:821)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1131)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:124)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
at org.testng.TestRunner.privateRun(TestRunner.java:773)
at org.testng.TestRunner.run(TestRunner.java:623)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:357)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:352)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:310)
at org.testng.SuiteRunner.run(SuiteRunner.java:259)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1185)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1110)
at org.testng.TestNG.run(TestNG.java:1018)
at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:94)
at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:54)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@10-ea/Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@10-ea/NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@10-ea/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@10-ea/Method.java:564)
at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:115)
at java.lang.Thread.run(java.base@10-ea/Thread.java:844)

"Common-Cleaner" #11 daemon prio=8 os_prio=0 tid=0x00007f497437d800 nid=0x6424 in Object.wait() [0x00007f48e91aa000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@10-ea/Native Method)
- waiting on <0x00000006ca8c3140> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@10-ea/ReferenceQueue.java:151)
- waiting to re-lock in wait() <0x00000006ca8c3140> (a java.lang.ref.ReferenceQueue$Lock)
at jdk.internal.ref.CleanerImpl.run(java.base@10-ea/CleanerImpl.java:148)
at java.lang.Thread.run(java.base@10-ea/Thread.java:844)
at jdk.internal.misc.InnocuousThread.run(java.base@10-ea/InnocuousThread.java:122)

"Service Thread" #10 daemon prio=9 os_prio=0 tid=0x00007f49742f8800 nid=0x6422 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Sweeper thread" #9 daemon prio=9 os_prio=0 tid=0x00007f49741fc000 nid=0x6421 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f49741fa000 nid=0x6420 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f49741f8000 nid=0x641f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f49741f6000 nid=0x641e waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f49741f4000 nid=0x641d waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f49741ed800 nid=0x641c runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f49741dd000 nid=0x641b in Object.wait() [0x00007f492c42d000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@10-ea/Native Method)
- waiting on <0x00000006ca92ee98> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@10-ea/ReferenceQueue.java:151)
- waiting to re-lock in wait() <0x00000006ca92ee98> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@10-ea/ReferenceQueue.java:172)
at java.lang.ref.Finalizer$FinalizerThread.run(java.base@10-ea/Finalizer.java:216)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f49741d9000 nid=0x641a waiting on condition [0x00007f492c52e000]
   java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList(java.base@10-ea/Native Method)
at java.lang.ref.Reference.processPendingReferences(java.base@10-ea/Reference.java:174)
at java.lang.ref.Reference.access$000(java.base@10-ea/Reference.java:44)
at java.lang.ref.Reference$ReferenceHandler.run(java.base@10-ea/Reference.java:138)

"main" #1 prio=5 os_prio=0 tid=0x00007f4974010000 nid=0x6404 in Object.wait() [0x00007f497ba97000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@10-ea/Native Method)
- waiting on <0x00000006ca8b0220> (a java.lang.Thread)
at java.lang.Thread.join(java.base@10-ea/Thread.java:1353)
- waiting to re-lock in wait() <0x00000006ca8b0220> (a java.lang.Thread)
at java.lang.Thread.join(java.base@10-ea/Thread.java:1427)
at com.sun.javatest.regtest.agent.MainWrapper.main(MainWrapper.java:74)

"VM Thread" os_prio=0 tid=0x00007f49741d2800 nid=0x6419 runnable

"GC Thread#0" os_prio=0 tid=0x00007f4974028800 nid=0x6405 runnable

"GC Thread#1" os_prio=0 tid=0x00007f497402a000 nid=0x6406 runnable

"GC Thread#2" os_prio=0 tid=0x00007f497402b800 nid=0x6407 runnable

"GC Thread#3" os_prio=0 tid=0x00007f497402d800 nid=0x6408 runnable

"GC Thread#4" os_prio=0 tid=0x00007f497402f000 nid=0x6409 runnable

"GC Thread#5" os_prio=0 tid=0x00007f4974031000 nid=0x640a runnable

"GC Thread#6" os_prio=0 tid=0x00007f4974032800 nid=0x640b runnable

"GC Thread#7" os_prio=0 tid=0x00007f4974034000 nid=0x640c runnable

"G1 Main Marker" os_prio=0 tid=0x00007f4974082000 nid=0x640d runnable

"G1 Marker#0" os_prio=0 tid=0x00007f4974083800 nid=0x640e runnable

"G1 Marker#1" os_prio=0 tid=0x00007f4974085800 nid=0x640f runnable

"G1 Refine#0" os_prio=0 tid=0x00007f497416d800 nid=0x6417 runnable

"G1 Refine#1" os_prio=0 tid=0x00007f497416b800 nid=0x6416 runnable

"G1 Refine#2" os_prio=0 tid=0x00007f497416a000 nid=0x6415 runnable

"G1 Refine#3" os_prio=0 tid=0x00007f4974168000 nid=0x6414 runnable

"G1 Refine#4" os_prio=0 tid=0x00007f4974166800 nid=0x6413 runnable

"G1 Refine#5" os_prio=0 tid=0x00007f4974164800 nid=0x6412 runnable

"G1 Refine#6" os_prio=0 tid=0x00007f4974163000 nid=0x6411 runnable

"G1 Refine#7" os_prio=0 tid=0x00007f4974161000 nid=0x6410 runnable

"G1 Young RemSet Sampling" os_prio=0 tid=0x00007f497416f000 nid=0x6418 runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007f4974312800 nid=0x6423 waiting on condition

JNI global references: 107



Attached 2 jtr files for reference:
WindowDemoTest.jtr : Logs with assertion error
ButtonDemoTest.java: Logs with deadlock