The maven-javadoc-plugin has a flag to fail on warning, which means: if something is written to the stderr it'll fail the build.
There has been a change in 17-ea where normal output is pushed to the stderr instead of stdout.
Here's an example to show what has been written to the stderr, which should have been stdout:
[WARNING] Javadoc Warnings
[WARNING] Loading source file E:\java-workspace\apache-maven-plugins\plugins\maven-javadoc-plugin\target\it\MJAVADOC-538\src\main\java\foo\bar\MyClass.java...
[WARNING] Constructing Javadoc information...
[WARNING] Building index for all the packages and classes...
[WARNING] Standard Doclet version 17-ea+30-2618
[WARNING] Building tree for all the packages and classes...
[WARNING] Generating E:\java-workspace\apache-maven-plugins\plugins\maven-javadoc-plugin\target\it\MJAVADOC-538\target\site\apidocs\foo\bar\MyClass.html...
[WARNING] Generating E:\java-workspace\apache-maven-plugins\plugins\maven-javadoc-plugin\target\it\MJAVADOC-538\target\site\apidocs\foo\bar\package-summary.html...
[WARNING] Generating E:\java-workspace\apache-maven-plugins\plugins\maven-javadoc-plugin\target\it\MJAVADOC-538\target\site\apidocs\foo\bar\package-tree.html...
[WARNING] Generating E:\java-workspace\apache-maven-plugins\plugins\maven-javadoc-plugin\target\it\MJAVADOC-538\target\site\apidocs\foo\bar\class-use\MyClass.html...
[WARNING] Generating E:\java-workspace\apache-maven-plugins\plugins\maven-javadoc-plugin\target\it\MJAVADOC-538\target\site\apidocs\foo\bar\package-use.html...
[WARNING] Generating E:\java-workspace\apache-maven-plugins\plugins\maven-javadoc-plugin\target\it\MJAVADOC-538\target\site\apidocs\overview-tree.html...
[WARNING] Building index for all classes...
[WARNING] Generating E:\java-workspace\apache-maven-plugins\plugins\maven-javadoc-plugin\target\it\MJAVADOC-538\target\site\apidocs\allclasses-index.html...
[WARNING] Generating E:\java-workspace\apache-maven-plugins\plugins\maven-javadoc-plugin\target\it\MJAVADOC-538\target\site\apidocs\allpackages-index.html...
[WARNING] Generating E:\java-workspace\apache-maven-plugins\plugins\maven-javadoc-plugin\target\it\MJAVADOC-538\target\site\apidocs\index-all.html...
[WARNING] Generating E:\java-workspace\apache-maven-plugins\plugins\maven-javadoc-plugin\target\it\MJAVADOC-538\target\site\apidocs\index.html...
[WARNING] Generating E:\java-workspace\apache-maven-plugins\plugins\maven-javadoc-plugin\target\it\MJAVADOC-538\target\site\apidocs\help-doc.html...

The first line (Javadoc Warnings) comes from the plugin, the rest of the lines are captured from the stderr (!).
Most likely related to https://bugs.openjdk.java.net/browse/JDK-8268774