The following tests:
  hotspot/jtreg/runtime/RedefineTests/RedefineFinalizer.java
  hotspot/jtreg/runtime/RedefineTests/RedefineRunningMethods.java
  hotspot/jtreg/runtime/RedefineTests/RedefinePreviousVersions.java
  hotspot/jtreg/runtime/RedefineTests/RedefineDoubleDelete.java
  hotspot/jtreg/runtime/RedefineTests/RedefineInterfaceMethods.java
  hotspot/jtreg/runtime/RedefineTests/RedefineInterfaceCall.java
  hotspot/jtreg/runtime/RedefineTests/ RedefineRunningMethodsWithBacktrace.java

fail with similar errors like below:

----------System.out:(1/134)----------
[1.584s][trace][redefine,class,nestmates] redefined class RedefineRunningMethods$B attribute change error: NestHost attribute removed
----------System.err:(17/1392)----------
java.lang.UnsupportedOperationException: class redefinition failed: attempted to change the class NestHost or NestMembers attribute
at java.instrument/sun.instrument.InstrumentationImpl.redefineClasses0(Native Method)
at java.instrument/sun.instrument.InstrumentationImpl.redefineClasses(InstrumentationImpl.java:190)
at RedefineClassHelper.redefineClass(RedefineClassHelper.java:60)
at RedefineClassHelper.redefineClass(RedefineClassHelper.java:50)
at RedefineRunningMethods.main(RedefineRunningMethods.java:108)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:115)
at java.base/java.lang.Thread.run(Thread.java:844)

JavaTest Message: Test threw exception: java.lang.UnsupportedOperationException: class redefinition failed: attempted to change the class NestHost or NestMembers attribute
JavaTest Message: shutting down test

The issue with these tests is that they are are using a class redefinition framework (RedefineClassHelper.redefineClass + InMemoryJavaCompiler.compile) that changes the NestHost and NestMembers attributes in the redefined versions of classes which is not allowed by the Nestmates-related JVMTI Spec update.

Below is full RedefineRunningMethods.jtr log:

#Test Results (version 2)
#Mon Mar 12 01:58:24 PDT 2018
#-----testdescription-----
$file=/scratch/sspitsyn/valhalla/open/test/hotspot/jtreg/runtime/RedefineTests/RedefineRunningMethods.java
$root=/scratch/sspitsyn/valhalla/open/test/hotspot/jtreg
keywords=bug8055008 othervm
library=/test/lib
modules=java.base/jdk.internal.misc java.compiler java.instrument jdk.jartool/sun.tools.jar
run=USER_SPECIFIED main RedefineClassHelper\nUSER_SPECIFIED main/othervm -javaagent\:redefineagent.jar -Xlog\:redefine+class+iklass+add\=trace,redefine+class+iklass+purge\=trace RedefineRunningMethods\n
source=RedefineRunningMethods.java
title=Redefine EMCP and non-EMCP methods that are running in an infinite loop

#-----environment-----

#-----testresult-----
description=file\:/scratch/sspitsyn/valhalla/open/test/hotspot/jtreg/runtime/RedefineTests/RedefineRunningMethods.java
elapsed=3815 0\:00\:03.815
end=Mon Mar 12 01\:58\:24 PDT 2018
environment=regtest
execStatus=Failed. Execution failed\: `main' threw exception\: java.lang.UnsupportedOperationException\: class redefinition failed\: attempted to change the class NestHost or NestMembers attribute
harnessLoaderMode=Classpath Loader
harnessVariety=Full Bundle
hostname=sca00jqu.us.oracle.com
javatestOS=Linux 4.1.12-112.14.13.el7uek.x86_64 (amd64)
javatestVersion=5.0
jtregVersion=jtreg 4.2 fcs b13
modules=java.base/jdk.internal.misc java.compiler java.instrument jdk.jartool/sun.tools.jar
script=com.sun.javatest.regtest.exec.RegressionScript
sections=script_messages build main build compile main
start=Mon Mar 12 01\:58\:20 PDT 2018
test=runtime/RedefineTests/RedefineRunningMethods.java
testJDK=/scratch/sspitsyn/valhalla/build/linux-x86_64-normal-server-release/images/jdk
totalTime=3817
user.name=sspitsyn
work=/scratch/sspitsyn/tst/valhalla/JTwork/runtime/RedefineTests

#section:script_messages
----------messages:(7/476)----------
JDK under test: /scratch/sspitsyn/valhalla/build/linux-x86_64-normal-server-release/images/jdk
java version "11-internal" 2018-03-20
Java(TM) SE Runtime Environment 18.9 (build 11-internal+0-adhoc.sspitsyn.open)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11-internal+0-adhoc.sspitsyn.open, mixed mode)
Library /test/lib; kind: packages
   source directory: /scratch/sspitsyn/valhalla/open/test/lib
   class directory: /scratch/sspitsyn/tst/valhalla/JTwork/classes/test/lib

#section:build
----------messages:(3/102)----------
command: build RedefineClassHelper
reason: Named class compiled on demand
elapsed time (seconds): 0.0
result: Passed. All files up to date

#section:main
----------messages:(5/344)----------
command: main RedefineClassHelper
reason: User specified action: run main RedefineClassHelper
Mode: othervm
Additional options from @modules: --add-modules java.base,java.compiler,java.instrument,jdk.jartool --add-exports java.base/jdk.internal.misc=ALL-UNNAMED --add-exports jdk.jartool/sun.tools.jar=ALL-UNNAMED
elapsed time (seconds): 0.84
----------configuration:(5/190)----------
Boot Layer
  add modules: java.base java.compiler java.instrument jdk.jartool
  add exports: java.base/jdk.internal.misc ALL-UNNAMED
               jdk.jartool/sun.tools.jar ALL-UNNAMED

----------System.out:(0/0)----------
----------System.err:(1/15)----------
STATUS:Passed.
----------rerun:(29/2543)*----------
cd /scratch/sspitsyn/tst/valhalla/JTwork/scratch && \\
DISPLAY=localhost:13.0 \\
HOME=/home/sspitsyn \\
JTREG_HOME=/java/re/jtreg/4.2/promoted/latest/binaries/jtreg \\
LANG=en_US.UTF-8 \\
PATH=/bin:/usr/bin \\
TZ=PST8PDT \\
CLASSPATH=/scratch/sspitsyn/valhalla/open/test/hotspot/jtreg/runtime/RedefineTests:/scratch/sspitsyn/tst/valhalla/JTwork/classes/test/lib:/scratch/sspitsyn/valhalla/open/test/lib:/net/scanas416.us.oracle.com/export/java_re2/misc/nightly/jtreg/jtreg4.2/binaries/jtreg/lib/javatest.jar:/net/scanas416.us.oracle.com/export/java_re2/misc/nightly/jtreg/jtreg4.2/binaries/jtreg/lib/jtreg.jar \\
    /scratch/sspitsyn/valhalla/build/linux-x86_64-normal-server-release/images/jdk/bin/java \\
        -Dtest.src=/scratch/sspitsyn/valhalla/open/test/hotspot/jtreg/runtime/RedefineTests \\
        -Dtest.src.path=/scratch/sspitsyn/valhalla/open/test/hotspot/jtreg/runtime/RedefineTests:/scratch/sspitsyn/valhalla/open/test/lib \\
        -Dtest.classes=/scratch/sspitsyn/tst/valhalla/JTwork/classes/runtime/RedefineTests/RedefineRunningMethods.d \\
        -Dtest.class.path=/scratch/sspitsyn/tst/valhalla/JTwork/classes/runtime/RedefineTests/RedefineRunningMethods.d:/scratch/sspitsyn/tst/valhalla/JTwork/classes/test/lib \\
        -Dtest.vm.opts='-Dtest.java.opts=-server -Xlog:redefine+class+nestmates=trace' \\
        -Dtest.tool.vm.opts='-J-Dtest.java.opts=-server -J-Xlog:redefine+class+nestmates=trace' \\
        -Dtest.compiler.opts= \\
        -Dtest.java.opts= \\
        -Dtest.jdk=/scratch/sspitsyn/valhalla/build/linux-x86_64-normal-server-release/images/jdk \\
        -Dcompile.jdk=/scratch/sspitsyn/valhalla/build/linux-x86_64-normal-server-release/images/jdk \\
        -Dtest.timeout.factor=1.0 \\
        -Dtest.modules='java.base/jdk.internal.misc java.compiler java.instrument jdk.jartool/sun.tools.jar' \\
        -Dtest.nativepath=/scratch/sspitsyn/valhalla/build/linux-x86_64-normal-server-release/images/test/hotspot/jtreg/native \\
        --add-modules java.base,java.compiler,java.instrument,jdk.jartool \\
        --add-exports java.base/jdk.internal.misc=ALL-UNNAMED \\
        --add-exports jdk.jartool/sun.tools.jar=ALL-UNNAMED \\
        -Dtest.java.opts=-server \\
        -Xlog:redefine+class+nestmates=trace \\
        -Djava.library.path=/scratch/sspitsyn/valhalla/build/linux-x86_64-normal-server-release/images/test/hotspot/jtreg/native \\
        com.sun.javatest.regtest.agent.MainWrapper /scratch/sspitsyn/tst/valhalla/JTwork/runtime/RedefineTests/RedefineRunningMethods.d/main.0.jta
result: Passed. Execution successful

#section:build
----------messages:(5/157)----------
command: build RedefineRunningMethods
reason: Named class compiled on demand
Test directory:
  compile: RedefineRunningMethods
elapsed time (seconds): 1.365
result: Passed. Build successful

#section:compile
----------messages:(5/418)----------
command: compile /scratch/sspitsyn/valhalla/open/test/hotspot/jtreg/runtime/RedefineTests/RedefineRunningMethods.java
reason: .class file out of date or does not exist
Additional options from @modules: --add-modules java.base,java.compiler,java.instrument,jdk.jartool --add-exports java.base/jdk.internal.misc=ALL-UNNAMED --add-exports jdk.jartool/sun.tools.jar=ALL-UNNAMED
Mode: othervm
elapsed time (seconds): 1.364
----------configuration:(10/624)----------
javac compilation environment
  add modules: java.base java.compiler java.instrument jdk.jartool
  add exports: java.base/jdk.internal.misc ALL-UNNAMED
               jdk.jartool/sun.tools.jar ALL-UNNAMED
  source path: /scratch/sspitsyn/valhalla/open/test/hotspot/jtreg/runtime/RedefineTests
               /scratch/sspitsyn/valhalla/open/test/lib
  class path: /scratch/sspitsyn/valhalla/open/test/hotspot/jtreg/runtime/RedefineTests
               /scratch/sspitsyn/tst/valhalla/JTwork/classes/runtime/RedefineTests/RedefineRunningMethods.d
               /scratch/sspitsyn/tst/valhalla/JTwork/classes/test/lib

----------rerun:(24/1809)*----------
cd /scratch/sspitsyn/tst/valhalla/JTwork/scratch && \\
DISPLAY=localhost:13.0 \\
HOME=/home/sspitsyn \\
JTREG_HOME=/java/re/jtreg/4.2/promoted/latest/binaries/jtreg \\
LANG=en_US.UTF-8 \\
PATH=/bin:/usr/bin \\
TZ=PST8PDT \\
    /scratch/sspitsyn/valhalla/build/linux-x86_64-normal-server-release/images/jdk/bin/javac \\
        -J-Dtest.java.opts=-server \\
        -J-Xlog:redefine+class+nestmates=trace \\
        -J-Dtest.src=/scratch/sspitsyn/valhalla/open/test/hotspot/jtreg/runtime/RedefineTests \\
        -J-Dtest.src.path=/scratch/sspitsyn/valhalla/open/test/hotspot/jtreg/runtime/RedefineTests:/scratch/sspitsyn/valhalla/open/test/lib \\
        -J-Dtest.classes=/scratch/sspitsyn/tst/valhalla/JTwork/classes/runtime/RedefineTests/RedefineRunningMethods.d \\
        -J-Dtest.class.path=/scratch/sspitsyn/tst/valhalla/JTwork/classes/runtime/RedefineTests/RedefineRunningMethods.d:/scratch/sspitsyn/tst/valhalla/JTwork/classes/test/lib \\
        -J-Dtest.vm.opts='-Dtest.java.opts=-server -Xlog:redefine+class+nestmates=trace' \\
        -J-Dtest.tool.vm.opts='-J-Dtest.java.opts=-server -J-Xlog:redefine+class+nestmates=trace' \\
        -J-Dtest.compiler.opts= \\
        -J-Dtest.java.opts= \\
        -J-Dtest.jdk=/scratch/sspitsyn/valhalla/build/linux-x86_64-normal-server-release/images/jdk \\
        -J-Dcompile.jdk=/scratch/sspitsyn/valhalla/build/linux-x86_64-normal-server-release/images/jdk \\
        -J-Dtest.timeout.factor=1.0 \\
        -J-Dtest.modules='java.base/jdk.internal.misc java.compiler java.instrument jdk.jartool/sun.tools.jar' \\
        -J-Dtest.nativepath=/scratch/sspitsyn/valhalla/build/linux-x86_64-normal-server-release/images/test/hotspot/jtreg/native \\
        @/scratch/sspitsyn/tst/valhalla/JTwork/runtime/RedefineTests/RedefineRunningMethods.d/compile.1.jta
----------System.out:(0/0)----------
----------System.err:(0/0)----------
result: Passed. Compilation successful

#section:main
----------messages:(5/582)----------
command: main -javaagent:redefineagent.jar -Xlog:redefine+class+iklass+add=trace,redefine+class+iklass+purge=trace RedefineRunningMethods
reason: User specified action: run main/othervm -javaagent:redefineagent.jar -Xlog:redefine+class+iklass+add=trace,redefine+class+iklass+purge=trace RedefineRunningMethods
Mode: othervm [/othervm specified]
Additional options from @modules: --add-modules java.base,java.compiler,java.instrument,jdk.jartool --add-exports java.base/jdk.internal.misc=ALL-UNNAMED --add-exports jdk.jartool/sun.tools.jar=ALL-UNNAMED
elapsed time (seconds): 1.608
----------configuration:(5/190)----------
Boot Layer
  add modules: java.base java.compiler java.instrument jdk.jartool
  add exports: java.base/jdk.internal.misc ALL-UNNAMED
               jdk.jartool/sun.tools.jar ALL-UNNAMED

----------System.out:(1/134)----------
[1.584s][trace][redefine,class,nestmates] redefined class RedefineRunningMethods$B attribute change error: NestHost attribute removed
----------System.err:(17/1392)----------
java.lang.UnsupportedOperationException: class redefinition failed: attempted to change the class NestHost or NestMembers attribute
at java.instrument/sun.instrument.InstrumentationImpl.redefineClasses0(Native Method)
at java.instrument/sun.instrument.InstrumentationImpl.redefineClasses(InstrumentationImpl.java:190)
at RedefineClassHelper.redefineClass(RedefineClassHelper.java:60)
at RedefineClassHelper.redefineClass(RedefineClassHelper.java:50)
at RedefineRunningMethods.main(RedefineRunningMethods.java:108)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:115)
at java.base/java.lang.Thread.run(Thread.java:844)

JavaTest Message: Test threw exception: java.lang.UnsupportedOperationException: class redefinition failed: attempted to change the class NestHost or NestMembers attribute
JavaTest Message: shutting down test

STATUS:Failed.`main' threw exception: java.lang.UnsupportedOperationException: class redefinition failed: attempted to change the class NestHost or NestMembers attribute
----------rerun:(31/2759)*----------
cd /scratch/sspitsyn/tst/valhalla/JTwork/scratch && \\
DISPLAY=localhost:13.0 \\
HOME=/home/sspitsyn \\
JTREG_HOME=/java/re/jtreg/4.2/promoted/latest/binaries/jtreg \\
LANG=en_US.UTF-8 \\
PATH=/bin:/usr/bin \\
TZ=PST8PDT \\
CLASSPATH=/scratch/sspitsyn/tst/valhalla/JTwork/classes/runtime/RedefineTests/RedefineRunningMethods.d:/scratch/sspitsyn/valhalla/open/test/hotspot/jtreg/runtime/RedefineTests:/scratch/sspitsyn/tst/valhalla/JTwork/classes/test/lib:/scratch/sspitsyn/valhalla/open/test/lib:/net/scanas416.us.oracle.com/export/java_re2/misc/nightly/jtreg/jtreg4.2/binaries/jtreg/lib/javatest.jar:/net/scanas416.us.oracle.com/export/java_re2/misc/nightly/jtreg/jtreg4.2/binaries/jtreg/lib/jtreg.jar \\
    /scratch/sspitsyn/valhalla/build/linux-x86_64-normal-server-release/images/jdk/bin/java \\
        -Dtest.src=/scratch/sspitsyn/valhalla/open/test/hotspot/jtreg/runtime/RedefineTests \\
        -Dtest.src.path=/scratch/sspitsyn/valhalla/open/test/hotspot/jtreg/runtime/RedefineTests:/scratch/sspitsyn/valhalla/open/test/lib \\
        -Dtest.classes=/scratch/sspitsyn/tst/valhalla/JTwork/classes/runtime/RedefineTests/RedefineRunningMethods.d \\
        -Dtest.class.path=/scratch/sspitsyn/tst/valhalla/JTwork/classes/runtime/RedefineTests/RedefineRunningMethods.d:/scratch/sspitsyn/tst/valhalla/JTwork/classes/test/lib \\
        -Dtest.vm.opts='-Dtest.java.opts=-server -Xlog:redefine+class+nestmates=trace' \\
        -Dtest.tool.vm.opts='-J-Dtest.java.opts=-server -J-Xlog:redefine+class+nestmates=trace' \\
        -Dtest.compiler.opts= \\
        -Dtest.java.opts= \\
        -Dtest.jdk=/scratch/sspitsyn/valhalla/build/linux-x86_64-normal-server-release/images/jdk \\
        -Dcompile.jdk=/scratch/sspitsyn/valhalla/build/linux-x86_64-normal-server-release/images/jdk \\
        -Dtest.timeout.factor=1.0 \\
        -Dtest.modules='java.base/jdk.internal.misc java.compiler java.instrument jdk.jartool/sun.tools.jar' \\
        -Dtest.nativepath=/scratch/sspitsyn/valhalla/build/linux-x86_64-normal-server-release/images/test/hotspot/jtreg/native \\
        --add-modules java.base,java.compiler,java.instrument,jdk.jartool \\
        --add-exports java.base/jdk.internal.misc=ALL-UNNAMED \\
        --add-exports jdk.jartool/sun.tools.jar=ALL-UNNAMED \\
        -Dtest.java.opts=-server \\
        -Xlog:redefine+class+nestmates=trace \\
        -Djava.library.path=/scratch/sspitsyn/valhalla/build/linux-x86_64-normal-server-release/images/test/hotspot/jtreg/native \\
        -javaagent:redefineagent.jar \\
        -Xlog:redefine+class+iklass+add=trace,redefine+class+iklass+purge=trace \\
        com.sun.javatest.regtest.agent.MainWrapper /scratch/sspitsyn/tst/valhalla/JTwork/runtime/RedefineTests/RedefineRunningMethods.d/main.2.jta
result: Failed. Execution failed: `main' threw exception: java.lang.UnsupportedOperationException: class redefinition failed: attempted to change the class NestHost or NestMembers attribute


test result: Failed. Execution failed: `main' threw exception: java.lang.UnsupportedOperationException: class redefinition failed: attempted to change the class NestHost or NestMembers attribute