nijiaben @ PerfMa reports:

Yesterday our colleague found a strange situation. After executing jstack, I saw multiple Attach Listener threads in the output. I was very confused at the time and felt that it was unlikely that the Attach Listener should theoretically exist only one.

So I followed the code of the hotspot, this situation may indeed exist. When we execute jstack operations concurrently on a process, it is likely that multiple Attach Listener threads are created. By default, the Attach Listener is not created when jvm is started. Instead, when we first trigger through the attach mechanism, the Attach Listener thread is created by the Signal Dispatcher thread.

static void signal_thread_entry(JavaThread* thread, TRAPS) {
  os::set_priority(thread, NearMaxPriority);
  while (true) {
    int sig;
    {
      // FIXME : Currently we have not decieded what should be the status
      // for this java thread blocked here. Once we decide about
      // that we should fix this.
      sig = os::signal_wait();
    }
    if (sig == os::sigexitnum_pd()) {
       // Terminate the signal thread
       return;
    }

    switch (sig) {
      case SIGBREAK: {
        // Check if the signal is a trigger to start the Attach Listener - in that
        // case don't print stack traces.
        if (!DisableAttachMechanism && AttachListener::is_init_trigger()) {
          continue;
        }
        ...
    }
}

bool AttachListener::is_init_trigger() {
  if (init_at_startup() || is_initialized()) {
    return false; // initialized at startup or already initialized
  }
  char fn[PATH_MAX+1];
  sprintf(fn, ".attach_pid%d", os::current_process_id());
  int ret;
  struct stat64 st;
  RESTARTABLE(::stat64(fn, &st), ret);
  if (ret == -1) {
    snprintf(fn, sizeof(fn), "%s/.attach_pid%d",
             os::get_temp_directory(), os::current_process_id());
    RESTARTABLE(::stat64(fn, &st), ret);
  }
  if (ret == 0) {
    // simple check to avoid starting the attach mechanism when
    // a bogus user creates the file
    if (st.st_uid == geteuid()) {
      init();
      return true;
    }
  }
  return false;
}

The Attach Listener thread is created in the AttachListener::init method above, but the _initialized property is used to pre-determine whether the thread needs to be created before the init method is executed, and _initialized is set to true in the attach_listener_thread_entry

static void attach_listener_thread_entry(JavaThread* thread, TRAPS) {
  os::set_priority(thread, NearMaxPriority);

  thread->record_stack_base_and_size();

  if (AttachListener::pd_init() != 0) {
    return;
  }
  AttachListener::set_initialized();
  ...
}

However, if more than one request signal is sent before _initialized=true is set, multiple Attach Listeners may be created because the Signal Dispatcher and Attach Listener threads are executed asynchronously.

We can zoom in on this problem by modifying the code in hotspot

static void attach_listener_thread_entry(JavaThread* thread, TRAPS) {
  os::set_priority(thread, NearMaxPriority);

  thread->record_stack_base_and_size();

  sleep(15);
  
  if (AttachListener::pd_init() != 0) {
    return;
  }
  AttachListener::set_initialized();
  ...
}

When the process is started, keep executing jstack <pid>, and you will eventually see that there are a lot of Attach Listener threads.

"Attach Listener" #16 daemon prio=9 os_prio=0 tid=0x00007f21a800e000 nid=0x35b8 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" #15 daemon prio=9 os_prio=0 tid=0x00007f21a800c000 nid=0x35a5 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" #14 daemon prio=9 os_prio=0 tid=0x00007f21a800a000 nid=0x3593 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" #13 daemon prio=9 os_prio=0 tid=0x00007f21a8008000 nid=0x3582 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" #12 daemon prio=9 os_prio=0 tid=0x00007f21a8006800 nid=0x3570 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" #11 daemon prio=9 os_prio=0 tid=0x00007f21a8004800 nid=0x355e runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" #10 daemon prio=9 os_prio=0 tid=0x00007f21a8002800 nid=0x354b runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" #9 daemon prio=9 os_prio=0 tid=0x00007f21a8001000 nid=0x3539 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
...

In general, the creation of the Attach Listener thread is created by the Signal Dispatcher thread, but the flag that determines whether the Signal Dispatcher can repeatedly create the Attach Listener thread is set in an Attach Listener thread. If the flag is not set in time, There may be cases where multiple Attach Listener threads are created.