Thread suspension is currently implemented as a safepoint VM operation. This is is from VMThread::execute(op)

     if (!concurrent) {
       // Wait for completion of request (non-concurrent)
       MutexLocker mu(VMOperationRequest_lock);
       while(t->vm_operation_completed_count() < ticket) {
         ->wait(!t->is_Java_thread());
       }
     } else {
       // Concurrent VM operations, just wait for the specified time
       if (is_compilation && BackgroundCompilationWaitThreshold > 0) {
         MutexLocker mu(VMOperationRequest_lock);
         (!t->is_Java_thread(), BackgroundCompilationWaitThreshold);
       }

Here is a scenario that leads to deadlock: a thread requesting any VM operation takes a safepoint inside VMOperationRequest_lock->wait() (but before the actual OS condition variable wait() is executed - in both Solaris and Win32 implementations of wait, we do a safepoint check before executing OS wait()).

This is bad, because the JavaThread enters a safepoint while holding
VMOperationRequest_lock. In fact, it may also hold other global locks such as Compile_lock or SystemDictionary_lock. If the VM operation that caused he current safepoint suspends our thread, the VM deadlocks. For example, the VM needs VMOperationRequest_lock to notify the thread that triggered this safepoint about the completion of the vm operation. However the lock cannot be obtained since the thread that is holding the lock is suspended.

This problem is common for both Win32 and Solaris. SunSoft ThreadSuspendSelf test can be used to trigger the deadlock in a special case of self-suspension (BugID: 4210441)