I got the following assert when running some local testing:
# Internal Error (src/share/vm/oops/instanceKlass.hpp:1027), pid=11865, tid=12055
# assert(k != __null) failed: k should not be null

V [libjvm.so+0xae79ed] report_vm_error(char const*, int, char const*, char const*, ...)+0xdd
V [libjvm.so+0xc1486f] Exceptions::new_exception(Thread*, Symbol*, Symbol*, JavaCallArguments*, Handle, Handle)+0x48f
V [libjvm.so+0xc17db5] Exceptions::_throw_args(Thread*, char const*, int, Symbol*, Symbol*, JavaCallArguments*)+0x145
V [libjvm.so+0x145b6a2] invoke(InstanceKlass*, methodHandle const&, Handle, bool, objArrayHandle, BasicType, objArrayHandle, bool, Thread*)+0x1562
V [libjvm.so+0x145f665] Reflection::invoke_constructor(oopDesc*, objArrayHandle, Thread*)+0x1e5
V [libjvm.so+0xf632fe] JVM_NewInstanceFromConstructor+0xfe

This happens because in this line:
  InstanceKlass* klass = InstanceKlass::cast(SystemDictionary::resolve_or_fail(name, h_loader, h_protection_domain, true, thread));

resolve_or_fail can return NULL when we hit an OOME, and InstanceKlass:cast doesn't accept NULL:
  static const InstanceKlass* cast(const Klass* k) {
    assert(k != NULL, "k should not be null");

The root cause of this problem seems to be this patch:
changeset: 63843:c13acdcf0719
user: coleenp
date: Wed Mar 15 10:25:37 2017 -0400
summary: 8155672: Remove instanceKlassHandles and KlassHandles

diff --git a/hotspot/src/share/vm/utilities/exceptions.cpp b/hotspot/src/share/vm/utilities/exceptions.cpp
--- a/hotspot/src/share/vm/utilities/exceptions.cpp
+++ b/hotspot/src/share/vm/utilities/exceptions.cpp
@@ -265,11 +265,10 @@
   Handle h_exception;
 
   // Resolve exception klass
- Klass* ik = SystemDictionary::resolve_or_fail(name, h_loader, h_protection_domain, true, thread);
- instanceKlassHandle klass(thread, ik);
+ InstanceKlass* klass = InstanceKlass::cast(SystemDictionary::resolve_or_fail(name, h_loader, h_protection_domain, true, thread));
 
   if (!thread->has_pending_exception()) {
- assert(klass.not_null(), "klass must exist");
+ assert(klass != NULL, "klass must exist");
     // We are about to create an instance - so make sure that klass is initialized
     klass->initialize(thread);
     if (!thread->has_pending_exception()) {