This has been observed once on Windows X64. I found a history bug was similar but resolved. See JDK-8159861. This is an intermittent failure.

Please refer to the test output below:
----------messages:(5/328)----------
command: main DefaultSignatureAlgorithm RSA 1024 SHA256withRSA
reason: User specified action: run main/othervm DefaultSignatureAlgorithm RSA 1024 SHA256withRSA
Mode: othervm [/othervm specified]
Additional options from @modules: --add-exports java.base/sun.security.tools.keytool=ALL-UNNAMED
elapsed time (seconds): 0.746
----------configuration:(3/79)----------
Boot Layer
  add exports: java.base/sun.security.tools.keytool ALL-UNNAMED

----------System.out:(1/192)----------
Command line args: [-keystore, ks, -storepass, changeit, -keypass, changeit, -alias, RSA10241484153942817, -keyalg, RSA, -keysize, 1024, -genkeypair, -dname, CN=RSA10241484153942817, -debug]
----------System.err:(1/16)----------
STATUS:Passed.
----------rerun:(29/3242)*----------
JDK8_HOME=c:/jenkins/workspace/mach5data/install/java/re/jdk/8/promoted/latest/bundles/windows-x64/jdk-8-windows-x64.tar.gz/jdk1.8.0 \\
JTREG_TIMEOUT_FACTOR=5 \\
JTREG_TIMEOUT_OPTION=-timeoutFactor:5 \\
JTREG_VERBOSE=fail,error,time \\
PATH='C:\\jenkins\\workspace\\mach5data\\install\\java\\re\\jdk\\8\\promoted\\latest\\bundles\\windows-x64\\jdk-8-windows-x64.tar.gz\\jdk1.8.0;C:\\jenkins\\workspace\\mach5data\\install\\jpg\\infra\\builddeps\\gnumake-cygwin_x86\\4.0+1.0\\gnumake-cygwin_x86-4.0+1.0.tar.gz\\cygwin\\bin;C:\\jenkins\\workspace\\mach5data\\install\\java\\re\\jtreg\\4.2\\promoted\\all\\b04\\bundles\\jtreg_bin-4.2.zip\\jtreg\\bin;C:\\cygwin\\usr\\local\\bin;C:\\cygwin\\bin;C:\\cygwin\\user\\bin;C:\\cygwin\\bin;.;C:\\ProgramData\\Oracle\\Java\\javapath;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0;C:\\Program Files\\Puppet Labs\\Puppet\\bin;C:\\Program Files (x86)\\Windows Kits\\8.1\\Windows Performance Toolkit;C:\\Program Files\\Microsoft SQL Server\\110\\Tools\\Binn' \\
SystemDrive=C: \\
SystemRoot='C:\\Windows' \\
TEMP='C:\\Temp' \\
buildid=BUILD_ID \\
windir='C:\\Windows' \\
CLASSPATH='C:\\jenkins\\workspace\\9-dev\\testresults\\windows-x64\\jdk_test.tier2\\jtreg\\JTwork\\classes\\4\\sun\\security\\tools\\keytool;C:\\jenkins\\workspace\\9-dev\\jdk\\test\\sun\\security\\tools\\keytool;C:\\jenkins\\workspace\\mach5data\\install\\java\\re\\jtreg\\4.2\\promoted\\all\\b04\\bundles\\jtreg_bin-4.2.zip\\jtreg\\lib\\javatest.jar;C:\\jenkins\\workspace\\mach5data\\install\\java\\re\\jtreg\\4.2\\promoted\\all\\b04\\bundles\\jtreg_bin-4.2.zip\\jtreg\\lib\\jtreg.jar' \\
    'C:\\jenkins\\workspace\\9-dev\\build\\jdk\\bin\\java' \\
        -Dtest.class.path.prefix='C:\\jenkins\\workspace\\9-dev\\testresults\\windows-x64\\jdk_test.tier2\\jtreg\\JTwork\\classes\\4\\sun\\security\\tools\\keytool;C:\\jenkins\\workspace\\9-dev\\jdk\\test\\sun\\security\\tools\\keytool' \\
        -Dtest.src='C:\\jenkins\\workspace\\9-dev\\jdk\\test\\sun\\security\\tools\\keytool' \\
        -Dtest.src.path='C:\\jenkins\\workspace\\9-dev\\jdk\\test\\sun\\security\\tools\\keytool' \\
        -Dtest.classes='C:\\jenkins\\workspace\\9-dev\\testresults\\windows-x64\\jdk_test.tier2\\jtreg\\JTwork\\classes\\4\\sun\\security\\tools\\keytool' \\
        -Dtest.class.path='C:\\jenkins\\workspace\\9-dev\\testresults\\windows-x64\\jdk_test.tier2\\jtreg\\JTwork\\classes\\4\\sun\\security\\tools\\keytool' \\
        -Dtest.vm.opts='-ea -esa' \\
        -Dtest.tool.vm.opts='-J-ea -J-esa' \\
        -Dtest.compiler.opts= \\
        -Dtest.java.opts= \\
        -Dtest.jdk='C:\\jenkins\\workspace\\9-dev\\build\\jdk' \\
        -Dcompile.jdk='C:\\jenkins\\workspace\\9-dev\\build\\jdk' \\
        -Dtest.timeout.factor=5.0 \\
        -Dtest.modules='java.base/sun.security.tools.keytool jdk.crypto.ec' \\
        --add-exports java.base/sun.security.tools.keytool=ALL-UNNAMED \\
        -ea \\
        -esa \\
        com.sun.javatest.regtest.agent.MainWrapper 'C:\\jenkins\\workspace\\9-dev\\testresults\\windows-x64\\jdk_test.tier2\\jtreg\\JTwork\\sun\\security\\tools\\keytool\\DefaultSignatureAlgorithm.d\\main.0.jta' RSA 1024 SHA256withRSA
result: Passed. Execution successful

#section:build
----------messages:(3/111)----------
command: build DefaultSignatureAlgorithm
reason: Named class compiled on demand
elapsed time (seconds): 0.0
result: Passed. All files up to date

#section:main
----------messages:(5/328)----------
command: main DefaultSignatureAlgorithm RSA 3072 SHA256withRSA
reason: User specified action: run main/othervm DefaultSignatureAlgorithm RSA 3072 SHA256withRSA
Mode: othervm [/othervm specified]
Additional options from @modules: --add-exports java.base/sun.security.tools.keytool=ALL-UNNAMED
elapsed time (seconds): 2.982
----------configuration:(3/79)----------
Boot Layer
  add exports: java.base/sun.security.tools.keytool ALL-UNNAMED

----------System.out:(1/192)----------
Command line args: [-keystore, ks, -storepass, changeit, -keypass, changeit, -alias, RSA30721484153943569, -keyalg, RSA, -keysize, 3072, -genkeypair, -dname, CN=RSA30721484153943569, -debug]
----------System.err:(1/16)----------
STATUS:Passed.
----------rerun:(29/3242)*----------
JDK8_HOME=c:/jenkins/workspace/mach5data/install/java/re/jdk/8/promoted/latest/bundles/windows-x64/jdk-8-windows-x64.tar.gz/jdk1.8.0 \\
JTREG_TIMEOUT_FACTOR=5 \\
JTREG_TIMEOUT_OPTION=-timeoutFactor:5 \\
JTREG_VERBOSE=fail,error,time \\
PATH='C:\\jenkins\\workspace\\mach5data\\install\\java\\re\\jdk\\8\\promoted\\latest\\bundles\\windows-x64\\jdk-8-windows-x64.tar.gz\\jdk1.8.0;C:\\jenkins\\workspace\\mach5data\\install\\jpg\\infra\\builddeps\\gnumake-cygwin_x86\\4.0+1.0\\gnumake-cygwin_x86-4.0+1.0.tar.gz\\cygwin\\bin;C:\\jenkins\\workspace\\mach5data\\install\\java\\re\\jtreg\\4.2\\promoted\\all\\b04\\bundles\\jtreg_bin-4.2.zip\\jtreg\\bin;C:\\cygwin\\usr\\local\\bin;C:\\cygwin\\bin;C:\\cygwin\\user\\bin;C:\\cygwin\\bin;.;C:\\ProgramData\\Oracle\\Java\\javapath;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0;C:\\Program Files\\Puppet Labs\\Puppet\\bin;C:\\Program Files (x86)\\Windows Kits\\8.1\\Windows Performance Toolkit;C:\\Program Files\\Microsoft SQL Server\\110\\Tools\\Binn' \\
SystemDrive=C: \\
SystemRoot='C:\\Windows' \\
TEMP='C:\\Temp' \\
buildid=BUILD_ID \\
windir='C:\\Windows' \\
CLASSPATH='C:\\jenkins\\workspace\\9-dev\\testresults\\windows-x64\\jdk_test.tier2\\jtreg\\JTwork\\classes\\4\\sun\\security\\tools\\keytool;C:\\jenkins\\workspace\\9-dev\\jdk\\test\\sun\\security\\tools\\keytool;C:\\jenkins\\workspace\\mach5data\\install\\java\\re\\jtreg\\4.2\\promoted\\all\\b04\\bundles\\jtreg_bin-4.2.zip\\jtreg\\lib\\javatest.jar;C:\\jenkins\\workspace\\mach5data\\install\\java\\re\\jtreg\\4.2\\promoted\\all\\b04\\bundles\\jtreg_bin-4.2.zip\\jtreg\\lib\\jtreg.jar' \\
    'C:\\jenkins\\workspace\\9-dev\\build\\jdk\\bin\\java' \\
        -Dtest.class.path.prefix='C:\\jenkins\\workspace\\9-dev\\testresults\\windows-x64\\jdk_test.tier2\\jtreg\\JTwork\\classes\\4\\sun\\security\\tools\\keytool;C:\\jenkins\\workspace\\9-dev\\jdk\\test\\sun\\security\\tools\\keytool' \\
        -Dtest.src='C:\\jenkins\\workspace\\9-dev\\jdk\\test\\sun\\security\\tools\\keytool' \\
        -Dtest.src.path='C:\\jenkins\\workspace\\9-dev\\jdk\\test\\sun\\security\\tools\\keytool' \\
        -Dtest.classes='C:\\jenkins\\workspace\\9-dev\\testresults\\windows-x64\\jdk_test.tier2\\jtreg\\JTwork\\classes\\4\\sun\\security\\tools\\keytool' \\
        -Dtest.class.path='C:\\jenkins\\workspace\\9-dev\\testresults\\windows-x64\\jdk_test.tier2\\jtreg\\JTwork\\classes\\4\\sun\\security\\tools\\keytool' \\
        -Dtest.vm.opts='-ea -esa' \\
        -Dtest.tool.vm.opts='-J-ea -J-esa' \\
        -Dtest.compiler.opts= \\
        -Dtest.java.opts= \\
        -Dtest.jdk='C:\\jenkins\\workspace\\9-dev\\build\\jdk' \\
        -Dcompile.jdk='C:\\jenkins\\workspace\\9-dev\\build\\jdk' \\
        -Dtest.timeout.factor=5.0 \\
        -Dtest.modules='java.base/sun.security.tools.keytool jdk.crypto.ec' \\
        --add-exports java.base/sun.security.tools.keytool=ALL-UNNAMED \\
        -ea \\
        -esa \\
        com.sun.javatest.regtest.agent.MainWrapper 'C:\\jenkins\\workspace\\9-dev\\testresults\\windows-x64\\jdk_test.tier2\\jtreg\\JTwork\\sun\\security\\tools\\keytool\\DefaultSignatureAlgorithm.d\\main.1.jta' RSA 3072 SHA256withRSA
result: Passed. Execution successful

#section:build
----------messages:(3/111)----------
command: build DefaultSignatureAlgorithm
reason: Named class compiled on demand
elapsed time (seconds): 0.0
result: Passed. All files up to date

#section:main
----------messages:(8/439)----------
command: main DefaultSignatureAlgorithm RSA 3073 SHA384withRSA
reason: User specified action: run main/othervm DefaultSignatureAlgorithm RSA 3073 SHA384withRSA
Mode: othervm [/othervm specified]
Additional options from @modules: --add-exports java.base/sun.security.tools.keytool=ALL-UNNAMED
Timeout information:
Could not find process id for the process that timed out.
Skipping timeout handling.
elapsed time (seconds): 600.005
----------configuration:(3/79)----------
Boot Layer
  add exports: java.base/sun.security.tools.keytool ALL-UNNAMED

----------System.out:(2/229)----------
Command line args: [-keystore, ks, -storepass, changeit, -keypass, changeit, -alias, RSA30731484153946530, -keyalg, RSA, -keysize, 3073, -genkeypair, -dname, CN=RSA30731484153946530, -debug]
Timeout signalled after 600 seconds
----------System.err:(0/0)----------
----------rerun:(29/3242)*----------
JDK8_HOME=c:/jenkins/workspace/mach5data/install/java/re/jdk/8/promoted/latest/bundles/windows-x64/jdk-8-windows-x64.tar.gz/jdk1.8.0 \\
JTREG_TIMEOUT_FACTOR=5 \\
JTREG_TIMEOUT_OPTION=-timeoutFactor:5 \\
JTREG_VERBOSE=fail,error,time \\
PATH='C:\\jenkins\\workspace\\mach5data\\install\\java\\re\\jdk\\8\\promoted\\latest\\bundles\\windows-x64\\jdk-8-windows-x64.tar.gz\\jdk1.8.0;C:\\jenkins\\workspace\\mach5data\\install\\jpg\\infra\\builddeps\\gnumake-cygwin_x86\\4.0+1.0\\gnumake-cygwin_x86-4.0+1.0.tar.gz\\cygwin\\bin;C:\\jenkins\\workspace\\mach5data\\install\\java\\re\\jtreg\\4.2\\promoted\\all\\b04\\bundles\\jtreg_bin-4.2.zip\\jtreg\\bin;C:\\cygwin\\usr\\local\\bin;C:\\cygwin\\bin;C:\\cygwin\\user\\bin;C:\\cygwin\\bin;.;C:\\ProgramData\\Oracle\\Java\\javapath;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0;C:\\Program Files\\Puppet Labs\\Puppet\\bin;C:\\Program Files (x86)\\Windows Kits\\8.1\\Windows Performance Toolkit;C:\\Program Files\\Microsoft SQL Server\\110\\Tools\\Binn' \\
SystemDrive=C: \\
SystemRoot='C:\\Windows' \\
TEMP='C:\\Temp' \\
buildid=BUILD_ID \\
windir='C:\\Windows' \\
CLASSPATH='C:\\jenkins\\workspace\\9-dev\\testresults\\windows-x64\\jdk_test.tier2\\jtreg\\JTwork\\classes\\4\\sun\\security\\tools\\keytool;C:\\jenkins\\workspace\\9-dev\\jdk\\test\\sun\\security\\tools\\keytool;C:\\jenkins\\workspace\\mach5data\\install\\java\\re\\jtreg\\4.2\\promoted\\all\\b04\\bundles\\jtreg_bin-4.2.zip\\jtreg\\lib\\javatest.jar;C:\\jenkins\\workspace\\mach5data\\install\\java\\re\\jtreg\\4.2\\promoted\\all\\b04\\bundles\\jtreg_bin-4.2.zip\\jtreg\\lib\\jtreg.jar' \\
    'C:\\jenkins\\workspace\\9-dev\\build\\jdk\\bin\\java' \\
        -Dtest.class.path.prefix='C:\\jenkins\\workspace\\9-dev\\testresults\\windows-x64\\jdk_test.tier2\\jtreg\\JTwork\\classes\\4\\sun\\security\\tools\\keytool;C:\\jenkins\\workspace\\9-dev\\jdk\\test\\sun\\security\\tools\\keytool' \\
        -Dtest.src='C:\\jenkins\\workspace\\9-dev\\jdk\\test\\sun\\security\\tools\\keytool' \\
        -Dtest.src.path='C:\\jenkins\\workspace\\9-dev\\jdk\\test\\sun\\security\\tools\\keytool' \\
        -Dtest.classes='C:\\jenkins\\workspace\\9-dev\\testresults\\windows-x64\\jdk_test.tier2\\jtreg\\JTwork\\classes\\4\\sun\\security\\tools\\keytool' \\
        -Dtest.class.path='C:\\jenkins\\workspace\\9-dev\\testresults\\windows-x64\\jdk_test.tier2\\jtreg\\JTwork\\classes\\4\\sun\\security\\tools\\keytool' \\
        -Dtest.vm.opts='-ea -esa' \\
        -Dtest.tool.vm.opts='-J-ea -J-esa' \\
        -Dtest.compiler.opts= \\
        -Dtest.java.opts= \\
        -Dtest.jdk='C:\\jenkins\\workspace\\9-dev\\build\\jdk' \\
        -Dcompile.jdk='C:\\jenkins\\workspace\\9-dev\\build\\jdk' \\
        -Dtest.timeout.factor=5.0 \\
        -Dtest.modules='java.base/sun.security.tools.keytool jdk.crypto.ec' \\
        --add-exports java.base/sun.security.tools.keytool=ALL-UNNAMED \\
        -ea \\
        -esa \\
        com.sun.javatest.regtest.agent.MainWrapper 'C:\\jenkins\\workspace\\9-dev\\testresults\\windows-x64\\jdk_test.tier2\\jtreg\\JTwork\\sun\\security\\tools\\keytool\\DefaultSignatureAlgorithm.d\\main.2.jta' RSA 3073 SHA384withRSA
result: Error. Program `C:\jenkins\workspace\9-dev\build\jdk\bin\java' timed out (timeout set to 600000ms, elapsed time was 600001ms).


test result: Error. Program `C:\jenkins\workspace\9-dev\build\jdk\bin\java' timed out (timeout set to 600000ms, elapsed time was 600001ms).