Steve Goldman and I have found a deadlock in the nightly test
ThreadsUsePipedIO. It is caused by a circularity in the Heap_lock,
pending_list_lock and a condition variable for blocking compilation.
This deadlock is caused mostly with -Xcomp but it could happen less
frequently with other options. It also needs !BackgroundCompilation where
BackgroundCompilation is the default with C2, but not C1.

One thread has the new Heap_lock and is requesting the pending_list_lock
object:

  [5] _cmutex_lock(0x80e5644), at 0xdfb7a4a0
  [6] ObjectMonitor::enter2(0x80e5624, 0x8222780), at 0xdee152e7
  [7] ObjectMonitor::enter(0x80e5624, 0x8222780), at 0xdee155ba
  [8] ObjectSynchronizer::slow_enter(0x8222d04, 0xd4dc0794, 0x8222780), at 0xdee
3da93
  [9] ObjectSynchronizer::fast_enter(0x8222d04, 0xd4dc0794, 0x8222780), at 0xdee
3d8d8
  [10] VM_GC_Operation::acquire_pending_list_lock(), at 0xdee6f829
  [11] VM_GC_Operation::doit_prologue(0xd4dc0774), at 0xdee6f96b
  [12] VMThread::execute(0xd4dc0774), at 0xdee6ef5f
  [13] GenCollectedHeap::mem_allocate_work(0x80cf598, 0x14, 0x0, 0x0, 0x8222780)
...

One thread has the pending_list_lock in the method java.lang.ref.Reference.run()
and the compiler kicks in and waits for it to compile. There's a surprisingly
inefficient sequence of bytecodes in this function (see the invokestatic).

current thread: t@6
  [1] _swtch(0x0), at 0xdfb79297
  [2] cond_wait(0x82191ec, 0x82191d4), at 0xdfb780df
  [3] Monitor::wait(0x8219198, 0x0, 0x0), at 0xdee11267
  [4] CompileBroker::wait_for_completion(0x8219158), at 0xdee85ee9
  [5] CompileBroker::compile_method_base(0x80e5170, 0xffffffff, 0x80e4c00), at 0
xdee8564d
  [6] CompileBroker::compile_method(0x80e5170, 0xffffffff, 0x80e4c00), at 0xdee8
57a5
  [7] CallInfo::set(0xdd4a0a08, 0x80e5160, 0x80e5160, 0x80e5170, 0x80e5170, 0xff
ffffff, 0x80e4c00), at 0xdedfe902
  [8] CallInfo::set(0xdd4a0a08, 0x80e5160, 0x80e5170, 0x80e4c00), at 0xdedfe865
  [9] LinkResolver::resolve_static_call(0xdd4a0a08, 0x80e5160, 0x80e5164, 0x80e5
168, 0x80e516c, 0x1, 0x1, 0x80e4c00), at 0xdedfff4f
  [10] LinkResolver::resolve_invokestatic(0xdd4a0a08, 0x80e5158, 0x400, 0x80e4c0
0), at 0xdee01550
  [11] LinkResolver::resolve_invoke(0xdd4a0a08, 0x0, 0x80e5158, 0x400, 0xb8, 0x8
0e4c00), at 0xdee01379
  [12] InterpreterRuntime::resolve_invoke(), at 0xded996ff
  [13] 0xdb40deff(), at 0xdb40defe
  [14] 0xdb40016a(0xdd4a0b58, 0xdd4a0cd4, 0xa, 0xd9056cd0, 0xdb407fa0, 0xdd4a0c6


The compiler thread is trying to compile, but needs the Heap_lock to get
some more memory.

  [3] _cmutex_lock(0x804f39c), at 0xdfb7a4a0
  [4] Mutex::wait_for_lock_implementation(), at 0xdee10018
  [5] Mutex::lock(0x804dee8), at 0xdee0fd44
  [6] CompactingPermGen::allocate_array(0x804ffa8, 0x80ee548, 0xa, 0xb, 0x80ee00
...
  [16] ciBytecodeStream::get_constant(0xdb2e081c), at 0xdeed8315
  [17] GraphBuilder::load_constant(0xdb2e0954), at 0xdeea9595
  [18] GraphBuilder::connect_to_end(0xdb2e0954, 0x81764b4), at 0xdeead8bc
  [19] GraphBuilder::build_transitive_closure(0xdb2e0954, 0x81764b4), at 0xdeeaf
164

To reproduce, javac the .java file attached and run:

java -Xcomp ThreadsUsePipedIO

until it stops before "connection made" message.