Analyzing output of !threadpool and !threads in windbg

Question

I have generated dumps on four servers and am analyzing the output of !threadpool and !threads. I noticed the roughly consistent following output: ``` 0:024> !threadpool CPU utilization 0% Worker Thread: Total: 2 Running: 0 Idle: 2 MaxLimit: 200 MinLimit: 2 Work Request in Queue: 0 Number of Timers: 27 Completion Port Thread:Total: 2 Free: 0 MaxFree: 4 CurrentLimit: 2 MaxLimit: 200 MinLimit: 2 !threads -special ThreadCount: 32 UnstartedThread: 0 BackgroundThread: 19 PendingThread: 0 DeadThread: 13 Hosted Runtime: no ``` ` My questions are: 1)How can I determine what the source of those 27 timers are? 2)What does 13 dead threads mean? 3)One of my threads is marked as having a lock. If i switch to that thread and run !clrstack, i see the following - is it related to my timers? ``` 0:027> !clrstack OS Thread Id: 0x14cc (27) *** WARNING: Unable to verify checksum for System.ni.dll Child-SP RetAddr Call Site 000000000ca6e1a0 000007fef5c06477 System.Threading.WaitHandle.WaitAny(System.Threa…

Accepted Answer

1)How can I determine what the source of those 27 timers are? Try to find instances of `TimerCallback` (for Threading.Timer): ``` !dumpheap -type TimerCallback ``` Then dump the callback properties (where callback address is the "Address" from the dumpheap output): ``` !do ``` Then dump the `Value` address of the `_target` property: ``` !do <_target address> ``` That should spit out the object that holds a reference to the TimerCallback, which should lead you to where the timer was created. I recommend checking out Tess Ferrandez's debugging labs, if you haven't already. 2)What does 13 dead threads mean? My understanding is that a dead thread refers to a C++ thread which no longer has an active OS thread, but still has references and thus cannot be destroyed (C++ threads use ref counting). A C# thread holds a reference to a C++ thread, and if your managed code keeps a reference to a C# thread, then that could be your problem. This post on Yun Jin's blog might be of some interest to you. (wayback archive) 3)One of my threads is marked as having a lock. If i switch to that thread and run !clrstack, i see the following - is it related to my timers? That looks like a (System.Threading) timer thread waiting for its interval to elapse.

Analyzing output of !threadpool and !threads in windbg

Solution

Problem