Summary

Throwing IllegalStateException for invalid values of jdk.serialFilter and jdk.serialFilterFactory properties prevents use of deserialization until the property is corrected.

Problem

The effects of invalid values of jdk.serialFilter and jdk.serialFilterFactory properties are incompletely specified. The behavior for invalid values of the properties is different and use an unconventional exception type, ExceptionInInitializerError and leave the OIF.Config class uninitialized.

When the value of the jdk.serialFilter or jdk.serialFilterFactory system property or security property is invalid, deserialization should not be possible and it should be clear in the specification.

Solution

The configured serial filter and filter factory are initialized when the java.io.ObjectInputFilter.Config class is initialized. Typically, this occurs when the methods ObjectInputFilter.Config.getSerialFilter or ObjectInputFilter.Config.getSerialFilterFactory are called from the application or by constructing an java.io.ObjectInputStream. The exceptions in the ObjectInputFilter.Config class initialization caused by invalid values of the two properties, either by system properties supplied on the command line or security properties are logged. The Config class marks either or both the filter and filter factory values as unusable and remembers the exception message.

Subsequent calls to the methods that get or set the filter or filter factory or create an ObjectInputStream throw java.lang.IllegalStateException with the remembered exception message. Constructing an ObjectInputStream calls both Config.getSerialFilter and Config.getSerialFilterFactory. The nature of the invalid property is reported as an IllegalStateException on first use.

Specification

The class javadoc of java.io.ObjectInputFilter.Config is updated to specify throwing java.lang.IllegalStateException whenjdk.serialFilter and jdk.serialFilterFactory properties are invalid.

@@ -523,12 +522,15 @@ public interface ObjectInputFilter {
      * {@systemProperty jdk.serialFilter}, its value is used to configure the filter.
      * If the system property is not defined, and the {@link java.security.Security} property
      * {@code jdk.serialFilter} is defined then it is used to configure the filter.
-     * The filter is created as if {@link #createFilter(String) createFilter} is called;
-     * if the filter string is invalid, an {@link ExceptionInInitializerError} is thrown
-     * and the initialization fails; subsequent attempts to use the configuration or
-     * serialization will fail with an implementation specific exception.
-     * If the system property {@code jdk.serialFilter} is not set on the command line
-     * it can be set with {@link #setSerialFilter(ObjectInputFilter) Config.setSerialFilter}.
+     * The filter is created as if {@link #createFilter(String) createFilter} is called,
+     * if the filter string is invalid the initialization fails and subsequent attempts to
+     * {@linkplain Config#getSerialFilter() get the filter}, {@link Config#setSerialFilter set a filter},
+     * or create an {@link ObjectInputStream#ObjectInputStream(InputStream) ObjectInputStream}
+     * throw {@link IllegalStateException}. Deserialization is not possible with an
+     * invalid serial filter.
+     * If the system property {@code jdk.serialFilter} or the {@link java.security.Security}
+     * property is not set the filter can be set with
+     * {@link #setSerialFilter(ObjectInputFilter) Config.setSerialFilter}.
      * Setting the {@code jdk.serialFilter} with {@link System#setProperty(String, String)
      * System.setProperty} <em>does not set the filter</em>.
      * The syntax for the property value is the same as for the
@@ -547,9 +549,12 @@ public interface ObjectInputFilter {
      * <p>The class must be public, must have a public zero-argument constructor, implement the
      * {@link BinaryOperator {@literal BinaryOperator<ObjectInputFilter>}} interface, provide its implementation and
      * be accessible via the {@linkplain ClassLoader#getSystemClassLoader() application class loader}.
-     * If the filter factory constructor is not invoked successfully, an {@link ExceptionInInitializerError}
-     * is thrown and subsequent use of the filter factory for deserialization fails with
-     * {@link IllegalStateException}.
+     * If the filter factory constructor is not invoked successfully subsequent attempts to
+     * {@linkplain Config#getSerialFilterFactory() get the factory},
+     * {@linkplain Config#setSerialFilterFactory(BinaryOperator) set the factory}, or create an
+     * {@link ObjectInputStream#ObjectInputStream(InputStream) ObjectInputStream}
+     * throw {@link IllegalStateException}. Deserialization is not possible with an
+     * invalid serial filter factory.
      * The filter factory configured using the system or security property during initialization
      * can NOT be replaced with {@link #setSerialFilterFactory(BinaryOperator) Config.setSerialFilterFactory}.
      * This ensures that a filter factory set on the command line is not overridden accidentally
@@ -698,8 +717,14 @@ public interface ObjectInputFilter {
* Returns the static JVM-wide deserialization filter or {@code null} if not configured.
*
* @return the static JVM-wide deserialization filter or {@code null} if not configured
+         * @throws IllegalStateException if the initialization of the filter from the
+         *      commandline property {@code jdk.serialFilter} or
+         *      the security property {@code jdk.serialFilter} fails.
          */
         public static ObjectInputFilter getSerialFilter() {
@@ -709,7 +734,9 @@ public interface ObjectInputFilter {
* @param filter the deserialization filter to set as the JVM-wide filter; not null
* @throws SecurityException if there is security manager and the
*       {@code SerializablePermission("serialFilter")} is not granted
-         * @throws IllegalStateException if the filter has already been set
+         * @throws IllegalStateException if the filter has already been set or the initialization
+         *       of the filter from the commandline property {@code jdk.serialFilter} or
+         *       the security property {@code jdk.serialFilter} fails.
          */
         public static void setSerialFilter(ObjectInputFilter filter) {
             Objects.requireNonNull(filter, "filter");

java.io.ObjectInputStream constructors specify IllegalStateException in the case of invalid serial filter or serial filter factory properties:

    @@ -375,6 +375,8 @@ public class ObjectInputStream
      * <p>The constructor initializes the deserialization filter to the filter returned
      * by invoking the {@link Config#getSerialFilterFactory()} with {@code null} for the current filter
      * and the {@linkplain Config#getSerialFilter() static JVM-wide filter} for the requested filter.
+     * If the serial filter or serial filter factory properties are invalid
+     * an {@link IllegalStateException} is thrown.
      *
      * <p>If a security manager is installed, this constructor will check for
      * the "enableSubclassImplementation" SerializablePermission when invoked
    @@ -387,6 +387,8 @@ public class ObjectInputStream
          * @throws  IOException if an I/O error occurs while reading stream header
      * @throws  SecurityException if untrusted subclass illegally overrides
      *          security-sensitive methods
+     * @throws  IllegalStateException if the initialization of {@link ObjectInputFilter.Config}
+     *          fails due to invalid serial filter or serial filter factory properties.
      * @throws  NullPointerException if {@code in} is {@code null}
      * @see     ObjectInputStream#ObjectInputStream()
      * @see     ObjectInputStream#readFields()
@@ -414,6 +416,8 @@ public class ObjectInputStream
      * <p>The constructor initializes the deserialization filter to the filter returned
      * by invoking the {@link Config#getSerialFilterFactory()} with {@code null} for the current filter
      * and the {@linkplain Config#getSerialFilter() static JVM-wide filter} for the requested filter.
+     * If the serial filter or serial filter factory properties are invalid
+     * an {@link IllegalStateException} is thrown.
      *
      * <p>If there is a security manager installed, this method first calls the
      * security manager's {@code checkPermission} method with the
 @@ -422,6 +424,8 @@ public class ObjectInputStream
      *          {@code checkPermission} method denies enabling
      *          subclassing.
      * @throws  IOException if an I/O error occurs while creating this stream
+     * @throws  IllegalStateException if the initialization of {@link ObjectInputFilter.Config}
+     *      fails due to invalid serial filter or serial filter factory properties.
      * @see SecurityManager#checkPermission
      * @see java.io.SerializablePermission
      */