FULL PRODUCT VERSION :
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux xxx 2.6.32-641.11.1.el6.x86_64 #1 SMP Wed Oct 26 10:25:23 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux


EXTRA RELEVANT SYSTEM CONFIGURATION :
Introduction.

Our system uses RMI, and some RMI stubs include original classes. (* 1)
After updating to Java8u121, this class was rejected to bind to rmiregistry as follows by registryFilter that is added .

java.io.ObjectInputStream filterCheck
INFO: ObjectInputFilter REJECTED: class <Target Class>, array length: -1, nRefs: 8, depth: 2, bytes: 507, ex: n/a

Therefore, in Java8u121, we could avoid this issue by setting registryFilter in the java.security file as follows.

sun.rmi.registry.registryFilter=<Target Class>

(*1) It uses the original InvocationHandler class which is not the java.rmi.server.RemoteObjectInvocationHandler class.

<Note>
The binding of another RMI stub that does not contain an original InvocationHandler class is not rejected.



A DESCRIPTION OF THE PROBLEM :
<Problem contents>
After updating to Java8u131, binding was rejected by registryFilter as follows when RMI stub containing classes that need to be set in registryFilter was bound to rmiregisty.

java.io.ObjectInputStream filterCheck
INFO: ObjectInputFilter REJECTED: null, array length: -1, nRefs: 16, depth: 6, bytes: 692, ex: n/a

Although registryFilter was set as follows in the java.security file, bind was successful in Java8u121.

sun.rmi.registry.registryFilter=<Target Class>

<Our primary view>
There is the our initial investigation result.
Based on logs, we supposed that binding was rejected because the value of depth exceeded 5(*1) on registryFilter.
In our conjecture, the following implementation change of ObjectInputStream.java changed in Java8u131 may be affecting.

<http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/diff/8a2c97926e63/src/share/classes/java/io/ObjectInputStream.java>
8172299: Improve class processing
date: Thu, 09 Feb 2017 16:34:22 +0000 (3 months ago)
method : readNonProxyDesc
- desc.initProxy(cl, resolveEx, readClassDesc(false));
-
- // Call filterCheck on the definition
- filterCheck(desc.forClass(), -1);
+ try {
+ totalObjectRefs++;
+ depth++;
+ desc.initProxy(cl, resolveEx, readClassDesc(false));
+ } finally {
+ depth--;
+ }

Based on the following stack trace, there was a trace of passing through the incrementing logic of depth.

 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) [rt.jar:1.8.0_131]

(*1)The maximum value of depth is based on below.
src/share/classes/sun/rmi/registry/RegistryImpl.java
 <http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/75f31e0bd829/src/share/classes/sun/rmi/registry/RegistryImpl.java>
 Line 99: private static int REGISTRY_MAX_DEPTH = 5;

<Related issue>
Specifying maxdepth for rmiregistry added in Java8u121 seems to have no effect


REGRESSION. Last worked in version 8u121

ADDITIONAL REGRESSION INFORMATION:
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
In Java8u131, bind an RMI stub that requires setting registryFilter to rmiregistry.


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
If registryFilter is set, binding to rmiregistry will succeed without being REJECTED.

ACTUAL -
In the case of Java8u131, even if registryFilter is set, binding to rmiregistry is rejected by registryFilter.


ERROR MESSAGES/STACK TRACES THAT OCCUR :
<rmiregistry log (standard output)>
java.io.ObjectInputStream filterCheck
INFO: ObjectInputFilter REJECTED: null, array length: -1, nRefs: 16, depth: 6, bytes: 692, ex: n/a

<Part of stack trace>
Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
 java.io.InvalidClassException: filter status: REJECTED
 at sun.rmi.registry.RegistryImpl_Skel.dispatch(Unknown Source) [rt.jar:1.8.0_131]
 at sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:450) [rt.jar:1.8.0_131]
 at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294) [rt.jar:1.8.0_131]
 at sun.rmi.transport.Transport$1.run(Transport.java:200) [rt.jar:1.8.0_131]
 at sun.rmi.transport.Transport$1.run(Transport.java:197) [rt.jar:1.8.0_131]
 at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_131]
 at sun.rmi.transport.Transport.serviceCall(Transport.java:196) [rt.jar:1.8.0_131]
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) [rt.jar:1.8.0_131]
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) [rt.jar:1.8.0_131]
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) [rt.jar:1.8.0_131]
 at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_131]
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) [rt.jar:1.8.0_131]
 ... 3 more
Caused by: java.io.InvalidClassException: filter status: REJECTED
 at java.io.ObjectInputStream.filterCheck(ObjectInputStream.java:1244) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1664) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1515) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) [rt.jar:1.8.0_131]
 at sun.rmi.server.MarshalInputStream.readLocation(MarshalInputStream.java:313) [rt.jar:1.8.0_131]
 at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:189) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) [rt.jar:1.8.0_131]
 ... 15 more


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
Unfortunately we can not submit it now.

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Do not update to Java8u131.
Set registryFilter in Java8u121.