The unit test for runtime installers (/open/test/jdk/tools/jpackage/share/RuntimePackageTest.java) is excluded on MacOS:
 * @comment Temporary disable for Linux and OSX until functionality implemented
 * @requires (os.family != "mac")
(note it is not disabled for linux as the comment says, and it is not because functionality is not implemented.

This is not surprising given bugs JDK-8237966, JDK-8237970 , and JDK-8237971 but the real problem is the following:

when specifying --runtime-image for an app-image or app package on macos, you can either specify the top level of a jdk (Containing the "Contents" directory, or the "Contents/Home" directory below it. So given:
jdk11/Contents/Home/<lib, bin, conf, include, ...>
you can either specify
"--runtime-image jdk11" or "--runtime-image jdk11/contents/Home"
and either will work. (this works because of code in MacAppImageBuilder.getRuntimeImageDir())

This is useful, since the jdk image in a build directory does not have the "Contents/Home", so if you set JDK_HOME=.../build/macosx-64/images/jdk you can then run:
$JDK_HOME/bin/jpackage ... --runtime-image $JDK_HOME

However you cannot do the same thing when building a runtime installer.
As above, if you use "jpackage --type pkg --runtime-image jdk11 ..." it will work, but if you use "jpackage --type pkg --runtime-image jdk11/Contents/Home ..." it will not work.
This means you cannot build runtime image from the jdk in the build dir (which is what the unit test does)
(note: you may need --name and --mac-package-identifier args due to related bugs)