Today, during the nmethod unlinking phase, the per-nmethod lock is held across first an is_unloading() call on the nmethod and then inline cache cleaning, which may take the nmethod locks of all nmethods referred to from the inline caches.
If care is not taken, an nmethod A can have an inline cache pointing at nmethod B, and B can have an inline cache pointing back at A. This could potentially cause a deadlock. Today it is subtly safe, because between calling is_unloading() and cleaning the inline caches, the nmethod entry barrier is disarmed, which causes an mfence in the patching code. This ensures that the racing threads do not enter a deadlock situation, because they will observe the is_unloading state that was published by the other thread in the race.
I would like to move the locks so that this becomes more robust.
If care is not taken, an nmethod A can have an inline cache pointing at nmethod B, and B can have an inline cache pointing back at A. This could potentially cause a deadlock. Today it is subtly safe, because between calling is_unloading() and cleaning the inline caches, the nmethod entry barrier is disarmed, which causes an mfence in the patching code. This ensures that the racing threads do not enter a deadlock situation, because they will observe the is_unloading state that was published by the other thread in the race.
I would like to move the locks so that this becomes more robust.