Paul reported a crash:

  And calling from Java i get a crash, after calling the call of the Java method but AFAICT before the next Java expression/statement is executed, i think in generated stub code trying to handle the returned value:

  # Internal Error (/Users/sandoz/Projects/jdk10/valhalla/hotspot/src/share/vm/runtime/mutex.cpp:1384), pid=75592, tid=5123
  # assert((!thread->is_Java_thread() || ((JavaThread *)thread)->thread_state() == _thread_in_vm) || rank() == Mutex::special) failed: wrong thread state for using locks
  ...
  V [libjvm.dylib+0xc9e274] VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x4d4
  V [libjvm.dylib+0xc9ea36] VMError::report_and_die(Thread*, char const*, int, char const*, char const*, __va_list_tag*)+0x4a
  V [libjvm.dylib+0x4cc421] report_vm_error(char const*, int, char const*, char const*, ...)+0xcd
  V [libjvm.dylib+0xa45f8b] Monitor::check_prelock_state(Thread*)+0x67
  V [libjvm.dylib+0xa45c39] Monitor::lock(Thread*)+0x97
  V [libjvm.dylib+0xbf4b4a] SystemDictionary::add_loader_constraint(Symbol*, Handle, Handle, Thread*)+0x138
  V [libjvm.dylib+0xbf4ec0] SystemDictionary::check_signature_loaders(Symbol*, Handle, Handle, bool, Thread*)+0xde
  V [libjvm.dylib+0x8e45eb] LinkResolver::check_method_loader_constraints(LinkInfo const&, methodHandle const&, char const*, Thread*)+0x283
  V [libjvm.dylib+0x8e3c07] LinkResolver::resolve_method(LinkInfo const&, Bytecodes::Code, Thread*)+0x48f
  V [libjvm.dylib+0x8e3752] LinkResolver::resolve_method_statically(Bytecodes::Code, constantPoolHandle const&, int, Thread*)+0x210
  V [libjvm.dylib+0x28ee39] Bytecode_invoke::static_target(Thread*)+0x7b
  V [libjvm.dylib+0xb666a2] SharedRuntime::store_value_type_fields_to_buf(JavaThread*, long)+0x1b2
  v ~RuntimeStub::store_value_type_fields_to_buf
  j UnsafeTest.main([Ljava/lang/String;)V+40

Fred suggested some improvements/cleanups:

One of the most significant issue when merging the two features
is the use of Universe::heap()->is_in_reserved() to determine if
a Klass pointer or an oop is passed in addition to field values.
With buffering, standalone values are not necessarily allocated
in the Java heap, so the method could return false even if the
argument is an oop pointing to a value type.

I'd like to propose another approach to this issue: instead
of testing the address itself, we could encode the type of
pointer with the address. Both oops (in Java heap or buffer)
and Klass pointers are word aligned, so the less significant
bit is always zero. Using this less significant bit could
tell us if the address is the address of a value or a Klass
instance. I'd prefer to keep the less significant bit to
zero for the oop (in case we have to hand it over the GC),
and set it to one when passing a Klass pointer. Another
advantage of this encoding is that the test of the nature
of the pointer is easy to implement in assembly code, which
could help us avoiding traps to the runtime (see below).

I’ve also noticed that the register mapping for the return
convention doesn’t seem cached, it is recomputed every time
the VM needs it (which also include a memory allocation
of a resource array). Because the return convention is
constant for a given value type on a given platform, it
could be cached in the ValueKlass instance (or even
precomputed when the ValueKlass instance is created).