When available swap space is low (less than the size of JVM process), Runtime.getRuntime().exec(string[]) call results in a hang. The size of the JVM process was 170MB and available swap was nearly 100MB. See following excerpt from pstack output -


----------------- lwp# 196 / thread# 196 --------------------
 fdf9b710 lwp_cond_wait (fdb960c0, fdb90120, 0)
 fe83f088 cond_wait_common (0, 0, 0, 0, 0, fe852000) + 148
 fe838b24 move_to_safe (fe85292c, 1, 30, c, 1, c) + 104
 fe838dc4 suspend_fork (fe8528b8, 1, ecf00000, fd000000, 0, 0) + 9c
 fe83ad44 _run_prefork (6fa58, fe8535d0, fe83ec60, 6f9d8, 6f9d8, fe8535d0) + 68
 fe83bdc0 fork1 (fe852000, fe853be8, fe852a40, fdfbc578, ecf00000, 0) + 40
 fd2687f8 Java_java_lang_UNIXProcess_forkAndExec (0, f8c520, fd28367c, 0, ffffff
ff, eceff480) + 564
 f940cc24 ???????? (eceff490, f506b730, 0, f94143e4, 0, eceff398)
 f94058b0 ???????? (eceff51c, 0, 0, f9415cf0, 1c, eceff418)
 f940042c ???????? (eceff5a8, eceff758, a, f5c5fdc8, 10, eceff4b0)
 fd544df8 void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments
*,Thread*) (eceff750, eceff658, eceff678, e895d8, e895d8, 6000) + 25c
 fd54d558 void jni_invoke(JNIEnv_*,JavaValue*,_jobject*,JNICallType,_jmethodID*,
JNI_ArgumentPusher*,Thread*) (0, eceff750, 6a49cc, 2, 6c0838, eceff734) + 390
 fd6d96d8 jni_NewObjectV (e8966c, 6a49c8, 6c0838, eceff818, fd9d6000, 0) + 2d0
 fd25c428 JNU_NewObjectByName (e8966c, fd26ff0c, fd26ff24, eceff910, 0, 0) + b0
 fd261340 Java_java_lang_Runtime_execInternal (e8966c, eceff914, eceff910, 0, 0,
 0) + 80

The process does not respond to kill -3, so I do not have java stack trace, but pstack output is attached.

Once the JVM reaches this stage, there is no significant CPU activity and truss just shows calls to lwp_cond_wait and some calls to lwp_mutex_lock.

Stopping other processes to make more memory (swap) available does not affect the process and eventually the only way to exit is kill -9

JRE version -

java version "1.4.0_02"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.0_02-b02)

This was while using server VM. (I haven't tried with client VM)

I would expect that this condition will result in java.io.IOException with the message "not enough space".

For more information, please refer to comments section.