FULL PRODUCT VERSION :
java -version'の結果
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux xxx 2.6.32-641.11.1.el6.x86_64 #1 SMP Wed Oct 26 10:25:23 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux


EXTRA RELEVANT SYSTEM CONFIGURATION :
Introduction.

Our system uses RMI, and some RMI stubs include original classes. (* 1)
After updating to Java8u121, this class was rejected to bind to rmiregistry as follows by registryFilter that is added .

java.io.ObjectInputStream filterCheck
INFO: ObjectInputFilter REJECTED: class <Target Class>, array length: -1, nRefs: 8, depth: 2, bytes: 507, ex: n/a

Therefore, in Java8u121, we could avoid this issue by setting registryFilter in the java.security file as follows.

sun.rmi.registry.registryFilter=<Target Class>

(*1) It uses the original InvocationHandler class which is not the java.rmi.server.RemoteObjectInvocationHandler class.

<Note>
The binding of another RMI stub that does not contain an original InvocationHandler class is not rejected.


A DESCRIPTION OF THE PROBLEM :
<Problem contents>
After updating to Java8u131, binding was rejected by registryFilter as follows when RMI stub containing classes that need to be set in registryFilter was bound to rmiregisty.

java.io.ObjectInputStream filterCheck
INFO: ObjectInputFilter REJECTED: null, array length: -1, nRefs: 16, depth: 6, bytes: 692, ex: n/a

Based on logs, we supposed that it was rejected because depth exceeded the maximum value of 5.
Therefore, based on JEP 290 (*1), the maxdepth value was added as 7 to the registryFilter.
Below setting image to java.security file.

sun.rmi.registry.registryFilter=<Target Class>;\
maxdepth=7

However, despite specifying maxdepth as 7, it was REJECTed at depth: 6, and it appeared to be ineffective.
There is the our initial investigation result.

<Our primary view>
There is the our initial investigation result.
Based on the source code, in the RegistryImpl class (*2), maxdepth of registryFilter is defined as 5 (*3) as a fixed value, so specifying maxdepth of registryFilter seems to have no effect in the java.security file.

(*1) : <http://openjdk.java.net/jeps/290>
(*2) : src/share/classes/sun/rmi/registry/RegistryImpl.java
 <http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/75f31e0bd829/src/share/classes/sun/rmi/registry/RegistryImpl.java>
(*3) : There are two point on RegistryImpl.java
 Line 99: private static int REGISTRY_MAX_DEPTH = 5;
 Line 397: if (filterInfo.depth() > REGISTRY_MAX_DEPTH) {

<Related issue>
After updating to Java8u131, the bind to rmiregistry is rejected by registryFilter even though registryFilter is set


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
In Java8u131, bind an RMI stub that requires setting registryFilter to rmiregistry.


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
If maxdepth is specified for registryFilter, maxdepth is checked with the specified value instead of 5.

ACTUAL -
Although maxdepth is specified for registryFilter, it seems that maxdepth is checked with a fixed value as 5.


ERROR MESSAGES/STACK TRACES THAT OCCUR :
<rmiregistry log (standard output)>
java.io.ObjectInputStream filterCheck
INFO: ObjectInputFilter REJECTED: null, array length: -1, nRefs: 16, depth: 6, bytes: 692, ex: n/a

<Part of stack trace>
Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
 java.io.InvalidClassException: filter status: REJECTED
 at sun.rmi.registry.RegistryImpl_Skel.dispatch(Unknown Source) [rt.jar:1.8.0_131]
 at sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:450) [rt.jar:1.8.0_131]
 at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294) [rt.jar:1.8.0_131]
 at sun.rmi.transport.Transport$1.run(Transport.java:200) [rt.jar:1.8.0_131]
 at sun.rmi.transport.Transport$1.run(Transport.java:197) [rt.jar:1.8.0_131]
 at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_131]
 at sun.rmi.transport.Transport.serviceCall(Transport.java:196) [rt.jar:1.8.0_131]
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) [rt.jar:1.8.0_131]
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) [rt.jar:1.8.0_131]
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) [rt.jar:1.8.0_131]
 at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_131]
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) [rt.jar:1.8.0_131]
 ... 3 more
Caused by: java.io.InvalidClassException: filter status: REJECTED
 at java.io.ObjectInputStream.filterCheck(ObjectInputStream.java:1244) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1664) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1515) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) [rt.jar:1.8.0_131]
 at sun.rmi.server.MarshalInputStream.readLocation(MarshalInputStream.java:313) [rt.jar:1.8.0_131]
 at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:189) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) [rt.jar:1.8.0_131]
 ... 15 more


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
Unfortunately we can not submit it now.
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Do not update to Java8u131.
Set registryFilter in Java8u121.