JvmtiEnv::SetSystemProperty eventually calls PathString::set_value in arguments.cpp, which aborts the VM when it fails to allocate a string copy of the property value.

https://github.com/openjdk/jdk/blob/272d6531ef94534d2044377f126744b5139f7ae9/src/hotspot/share/runtime/arguments.cpp#L128-L141

bool PathString::set_value(const char *value) {
  if (_value != NULL) {
    FreeHeap(_value);
  }
  _value = AllocateHeap(strlen(value)+1, mtArguments );
  // should pass AllocFailStrategy::RETURN_NULL -----^
  assert(_value != NULL, "Unable to allocate space for new path value");

This should be fixed so that JvmtiEnv::SetSystemProperty can return JVMTI_ERROR_OUT_OF_MEMORY in case of OOM. See https://docs.oracle.com/en/java/javase/17/docs/specs/jvmti.html#SetSystemProperty