Name: sg39081 Date: 02/17/99


In our MFC application, we have been experiencing crashes following the dismissal of a modal Java dialog. The crash occurs because a zero is being written to memory overwriting a class object's function table pointer. The cause of this corruption has been traced to the winawt.dll's WmEndModal() routine in awt_Dialog.cpp. It has been found that this corruption occurs often in our application, though not always corrupting something that leads to a crash.

Our application creates a Java VM, which it uses at times to display modal Java dialogs. The modal Java dialog, as you would expect, calls setVisible(true) to display the dialog, and calls dispose() soon after the setVisible call returns. The memory corruption is occurring due to the following sequence of events within winawt.dll in the code for awt_Dialog.cpp.

1. WmEndModal() routine is called as part of the handling for a posted WM_AWT_DLG_ENDMODAL message. This occurs on the Toolkit window's thread. At the beginning of this routine, a stack trace for the VM thread shows it was last in winawt.dll at the SendMessage call in the sun_awt_windows_WComponentPeer__dispose() routine found in awt_Component.cpp. This seems to indicate that the dispose() call in the Java code has been reached and has begun processing. Within the WmEndModal() routine a SendMessage call is made to hide the dialog. I consistently see the AWT Toolkit thread lose execution focus within this SendMessage call, I assume because this thread's time slice has run out.

2. Next, it appears the VM thread is able to resume executing the dispose() handling, eventually executing the destructor for the dialog WmEndModal() is operating on. Because the m_modalWnd has not yet been cleared, the destructor makes a recursive call to WmEndModal(). After WmEndModal returns to the destructor, apparently without interruption, the dialog object's memory is freed, making that memory available for allocation to someone else.

3. Next, the SendMessage call from step 1 returns. However, the dialog object's memory has been freed by the destructor and the "this" pointer often points to memory that has been reallocated to someone else. Setting m_modalWnd to NULL at the end of the WmEndModal routine now ends up corrupting memory.

As a practical workaround, we have moved the setting of m_modalWnd to NULL to the top of the routine and made changes based on the assumption that the class object could get deleted during the routine's execution. This greatly reduces the likelihood of corruption occurring. However, this doesn't deal with the issue of the VM thread being able to cause the execution of the awt_Dialog destructor while the Toolkit thread is in the middle of the WmEndModal() routine. Hopefully a fix can be found that would avoid this.

All the changes we made are in the WmEndModal() routine found in awt_Dialog.cpp. The following code shows these changes.

// The following routine has been modified to deal with a reentrancy problem
// caused by m_modalWnd being set to NULL at the end of the routine. It is
// possible for the thread executing this routine to lose focus before m_modalWnd
// is set to NULL. Before focus resumes executing this thread, the AwtDialog
// destructor can get called, deleting the dialog object out from under this
// routine. Now, setting m_modalWnd to NULL writes a zero into memory it no longer
// owns. Setting m_modalWnd near the top of the routine should prevent the
// destructor from making a reentrant call to this routine.
MsgRouting AwtDialog::WmEndModal()
{
    // get local copies of needed window handles
    // Note: Use volatile to get compiler to keep code for this statement here
    volatile HWND hWnd = GetHWnd();
    HWND modalWnd = m_modalWnd;
    // clear window handle here
    m_modalWnd = NULL;

    ASSERT( ::GetCurrentThreadId() == AwtToolkit::MainThread() );
// ASSERT( ::IsWindowVisible( GetHWnd() ) );
    ASSERT( ::IsWindowVisible( hWnd ) );
// ASSERT( ::IsWindow(m_modalWnd) );
    ASSERT( ::IsWindow(modalWnd) );

    // re-enable top-level windows
// AwtDialog::ModalEnable( GetHWnd() );
    AwtDialog::ModalEnable( hWnd );
    
// HWND hWndParent = ::GetParent( GetHWnd() );
    HWND hWndParent = ::GetParent( hWnd );
    BOOL isEnabled = ::IsWindowEnabled(hWndParent);

    // Need to temporarily enable the dialog's owner window
    // or Windows could activate another application. This
    // happens when another app was activated previous to
    // a nested dialog being activated/closed (in which case the
    // owner wouldn't be re-enabled in ModalEnable).
    ::EnableWindow(hWndParent, TRUE);
    // hide the dialog
    ::SendMessage(hWnd,WM_AWT_COMPONENT_SHOW, SW_HIDE,0);
    // restore the owner's true enabled state
    ::EnableWindow(hWndParent, isEnabled);

    // bring the next window in the stack to the front
    AwtDialog::ModalNextWindowToFront( hWndParent );
// m_modalWnd = NULL;

    return mrConsume;
}
(Review ID: 54295)
======================================================================