We received an issue about possible infinite loop in HotSpot
Server Compiler.

When a licensee launches their servlet in 1.3.1_03 (windows edition),
an infinite loop seems to occur in the VM and consume all resources.

Their system is too big to extract reasonable sized program for reproduction.
They sent the result of their investigation for this issue as an
alternatives.
The followings are their invatigation.

==========>
Investigation:

 - How to collect data
   a) To get CPU ratio of each thread by using Performance Monitor in Windows.
        They specified the following monitoring data.
          Object : Thread
          Counter : %processor Time, ID Process, ID Thread
          Instance : All the threads in issued process
          
   b) To launch "userdump.exe" (similar to "gcore" in UNIX system)
      several times every few seconds.
      
 According to the above data a), it turns out that a thread consumes most of
 CPU resource.
 We also looked into the dump data of above b) by the thread ID.
 Then we find out the thread is compiler thread in VM.
 Furthermore, we find out the very thread is always in the same member
 function when the issue happens.
 
***
- src/share/vm/opto/phaseX.cpp:
 ...
 Node *PhaseIterGVN::transform_old( Node *n ) {
    
   while( i ) {
 #ifndef PRODUCT
     debug_only( if( loop_count >= K ) i->dump(4); )
     assert(loop_count < K, "infinite loop in PhaseGVN::transform");
     debug_only( loop_count++; )
 #endif
     
   }
    
 }
 
***

 The compiler threads is always in the above "while" block.
 (The stack trace of compiler thread stopped either in the "while" block
  or in the function called from the "while" block.)
  
 We also look for the method name which the compiler thread is compiling
 and it is doing the following method in every dump file.
 
 Methods name:
   java/io/Win32FileSystem#resolve(Ljava/io/File;)Ljava/lang/String;
   
 
 Here, summarizing what we know,
   - A compiler thread consumes most of CPU resource.
   - The compiler thread is always in the "while" block and during
     compiling the same method in every dump file.
 
 We conclude the compiler thread is under infinite loop status.
 
 
Note1:
  According to our investigation, there are two senarios that the
  transform_old() function is called.(Please see the following chart.)
  In this case, the infinite loop occurs with the (1) path, which is,
  when PhaseIterGVN::optimize() calles PhaseIterGVN::transform_old(),
  the infinite loop occurs.

****
 PhaseIterGVN::transform_old() <----------------+
          | PhaseIterGVN::transform()
         (1) PhaseIdealLoop::is_counted_loop()
          | IdealLoopTree::counted_loop()
          | IdealLoopTree::counted_loop()
 PhaseIterGVN::optimize() PhaseIdealLoop::PhaseIdealLoop()
  | |
 Compile::Optimize()-----------(2)--------------+
 Compile::Compile()
 C2Compiler::compile_method()
 CompileBroker::invoke_compiler_on_method()
****

Note2:
 According to the information gotten out of the above note1,
 we set the .hotspot_compiler file which includes the line,
 
   exclude java/io/Win32FileSystem resolve
   
 under the current directory.
 Then the infinite loop issue does not occur.
 
Note3:
 The while block has some debug statements(#ifdef PRODUCT ) and it
 will terminate the execution as an assertion error when the couonter
 "i" is greater than a number.(1024 ?)
 Is the above effective as a workaround ?
<==========

****** Additinal information ******
The licensee modified the Hotspot(server)source(1.3.1_03) to get the Node
information and kept looking into its behavior.
Then they suspect that the infinite loop occurs in the following
senario.

+++++++ Licensee's Report +++++++

The below partial code is extracted from HotSpot 1.3.1_03
hotspot/src/share/vm/opto/memnode.cpp.

....
 Node *StoreNode::Ideal( PhaseGVN *phase, PhaseDefUse *du ) { <= line# 511
   ...
   if( mem->Opcode() == Op_MergeMem ) { <= line# 523
     const TypePtr *tp = t_adr->is_ptr();
     // TypeOopPtr::NOTNULL+any is an OOP with unknown offset - generally
     // means an array I have not precisely typed yet. Do not do any
     // alias stuff with it any time soon.
     const TypeOopPtr *toop = tp->isa_oopptr();
     if( tp->_base != Type::AnyPtr &&
         (!toop ||
         !toop->klass()->is_java_lang_Object() ||
          toop->_offset != Type::OffsetBot) ) {
       set_req( MemNode::Memory, <= line# 533
                mem->in(phase->C->get_alias_index(tp)), du );
       return this; <= line# 535
     }
   }
.........


Let us focus on the if-block from line#523 to line #535.

We find out there is a case which the Node(Node in "this") is not
changed in set_req() in line#533.
This case seems to happen in some internal status of VM.
As the result of no chnage of Node inforamtion, the statement "return this"
of line#535 always returns the same node information although update
is expected.

This guess is based on the Node information just after set_req()
from line# 2630 in the attached log file(sample1.log_hs131.log).
The information is specifying that Node information is not updated after
infinite loop occurs.


We have a similar case in the same file and consider that the above
mentioned guess is applicable for the following part of code.


 Node *LoadNode::Ideal( PhaseGVN *phase, PhaseDefUse *du ) { <= line# 166
   ...
   if( mem->Opcode() == Op_MergeMem ) { <= line# 188
     const TypePtr *tp = t_adr->is_ptr();
     // TypeOopPtr::NOTNULL+any is an OOP with unknown offset - generally
     // means an array I have not precisely typed yet. Do not do any
     // alias stuff with it any time soon.
     const TypeOopPtr *toop = tp->isa_oopptr();
     if( tp->_base != Type::AnyPtr &&
         (!toop ||
         !toop->klass()->is_java_lang_Object() ||
          toop->_offset != Type::OffsetBot) ) {
       set_req( MemNode::Memory, <= line# 198
                mem->in(phase->C->get_alias_index(tp)), du );
       return this; <= line# 200
     }
   }
  ...
  
  
WHAT WE DID :
  Wedeleted the if-block from source code of Hotspot 1.3.1_03
  because the deleting is done for the source code of Hotspot 1.4.1.
 
RESULT:
  The infinite loop has not occur so far.
  
REQUEST:
  1) Confirm whether our investigation is correct or not with using
     the attached Node information in log file(sample1.log_hs131.log).
  2) Review whether the deletion of if-clause is reasonable to avoid the
     infinite loop or not.
  3) if the deletion is reasonable,
       3-1) Review whether the workaround cause a side-effect or not
  4) Check whether there is any other similar or possible part
     to cause the loop other than StoreNode::Ideal() and LoadNode::Ideal()
  5) The condition which causes this loop on java user side
     what and how user do, this infinite loop occurs.
     

APPENDIX:
  1) We change the hotspot files as follows.
      - PhaseIterGVN::transform_old() in hotspot/src/share/vm/opto/phaseX.cpp
      - LoadNoad::Ideal() and StoreNode::Ideal()
                                      in hotspot/src/share/vm/opto/memnode.cpp
                                      
     Detail diffs, please see the attached "debug_vm.diff"
  
  2) "debug_vm.diff" includes the workaround also.
      Specifically, we can swicth the mode with "SkipIfOfMergeMem" variable
      which can be cahnge on/of outside the VM.
       
  3) Contents of log Data
     - JVM version inf
     - Method's name when the infinite loop happens
     - All the node information when loop_count is 1024
     - The log of i->dump(4) when loop_count is the number between in 1024
       and 1124.
     - Node information which StoreNode::Ideal() returns, or
       Node information which LoadNode::Ideal() returns when loop_count
       is the number between in 1024 and 1124.
  
     The attached log data occurs when the loop happens to StoreNode::Ideal().
     Althogh we do not provide the log data related to LoadNode::Ideal(),
     we know the loop occurs in LoadNode::Ideal().
      

***************************************************************************