Steps to reproduce:
1) download following files into local dir like c:\showdoc\html
http://rehte.cn.oracle.com:8080/wshost/new_framework/new_framework/tests/javafx/manual/HostServicesAPIs/html/TestFXShowdocumentInvalidFile.html
http://rehte.cn.oracle.com:8080/wshost/new_framework/new_framework/tests/javafx/manual/HostServicesAPIs/html/TestFXShowdocumentInvalidFile.jnlp
and following jar to dir c:\showdoc\classes
http://rehte.cn.oracle.com:8080/wshost/new_framework/new_framework/tests/javafx/manual/HostServicesAPIs/classes/showdocument.jar
2) set IE as default browser
3) disable pop-up blocker in IE
4) open local applet c:\showdoc\html\TestFXShowdocumentInvalidFile.html, it is expected to open a new window/tab to open a local file, though the local file doesn't exist. If it doesn't open a new tab/window, bug is reproduced.
A new tab can be opened on firefox.