Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8282712

VMConnection.open() does not detect if VM failed to be created, resulting in NPE

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: P5 P5
    • 22
    • 19
    • core-svc
    • None
    • b12

      The following code in VMConnection.java is problematic:

          synchronized VirtualMachine open() {
              if (connector instanceof LaunchingConnector) {
                  vm = launchTarget();
              } else if (connector instanceof AttachingConnector) {
                  vm = attachTarget();
              } else if (connector instanceof ListeningConnector) {
                  vm = listenTarget();
              } else {
                  throw new InternalError("Invalid connect type");
              }
              vm.setDebugTraceMode(traceFlags);
      ...
          }

      If any of the xxxTarget() calls being made can fail and return null, this will result in an NPE when vm.SetDebugTraceMode() is called. This results in test failures that look like:

      java.io.IOException
      at jdk.jdi/com.sun.tools.jdi.VirtualMachineManagerImpl.createVirtualMachine(VirtualMachineManagerImpl.java:239)
      at jdk.jdi/com.sun.tools.jdi.AbstractLauncher.launch(AbstractLauncher.java:145)
      at jdk.jdi/com.sun.tools.jdi.SunCommandLineLauncher.launch(SunCommandLineLauncher.java:248)
      at VMConnection.launchTarget(VMConnection.java:314)
      at VMConnection.open(VMConnection.java:149)
      at TestScaffold.connect(TestScaffold.java:641)
      at TestScaffold.startUp(TestScaffold.java:365)
      at TestScaffold.startTo(TestScaffold.java:375)
      at TestScaffold.startToMain(TestScaffold.java:370)
      at Java_gTest.runTests(Java_gTest.java:117)
      at TestScaffold.startTests(TestScaffold.java:432)
      at Java_gTest.main(Java_gTest.java:102)
      at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
      at java.base/java.lang.reflect.Method.invoke(Method.java:577)
      at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
      at java.base/java.lang.Thread.run(Thread.java:1484)

       Unable to launch target VM.
      java.lang.NullPointerException: Cannot invoke "com.sun.jdi.VirtualMachine.setDebugTraceMode(int)" because "this.vm" is null
      at VMConnection.open(VMConnection.java:157)
      at TestScaffold.connect(TestScaffold.java:641)
      at TestScaffold.startUp(TestScaffold.java:365)
      at TestScaffold.startTo(TestScaffold.java:375)
      at TestScaffold.startToMain(TestScaffold.java:370)
      at Java_gTest.runTests(Java_gTest.java:117)
      at TestScaffold.startTests(TestScaffold.java:432)
      at Java_gTest.main(Java_gTest.java:102)
      at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
      at java.base/java.lang.reflect.Method.invoke(Method.java:577)
      at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
      at java.base/java.lang.Thread.run(Thread.java:1484)

      In mdash, the NPE will show up as the reason for the failure, rather than the IOE. I think the real issue is not the failure of open() to detect that vm == null, but the failure off the xxxTarget() methods to throw an exception. They shouldn't ever return null. For example:

          private VirtualMachine attachTarget() {
              AttachingConnector attacher = (AttachingConnector)connector;
              try {
                  return attacher.attach(connectorArgs);
              } catch (IOException ioe) {
                  ioe.printStackTrace();
                  System.err.println("\n Unable to attach to target VM.");
              } catch (IllegalConnectorArgumentsException icae) {
                  icae.printStackTrace();
                  System.err.println("\n Internal debugger error.");
              }
              return null; // Shuts up the compiler
          }

      All 3 of the xxxTarget() methods follow a similar pattern. They catch any exceptions thrown, print an error message, and return null. And the comment on the return is also indicative of this code being incorrect. If the code properly re-threw the exception, there would be no need for the return null to "shut up the compiler". Another option is for these methods to wrap the exception in a new exception.

      There are actually two copies of this VMConnection code. One is in jdb (src/jdk.jdi/share/classes/com/sun/tools/example/debug/tty/VMConnection.java) and one is in used for testing (open/test/jdk/com/sun/jdi/VMConnection.java). Both should be fixed, although I only noticed this code being a problem when running com/sun/jdi tests.

            cjplummer Chris Plummer
            cjplummer Chris Plummer
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: