Getting started in ASP.Net debugging with WinDbg

We’ve all faced the dreaded application is slow or eats all my RAM. You take a memory dump and recycle the AppPool.  But then what? Some of the people I work with are memory dump gurus, but the barrier to entry seemed pretty high. This is a quick primer on how to dive into a dump file. I’m not an expert, and by no means will you be an expert from reading this. “Advanced .Net debugging in under an hour” is how our PFE explained it.

Note: The “bitness” of the dump file is important throughout. A 64bit dump requires the 64bit debugger and the 64bit symbols.

First install WinDbg. It’s a good idea to put it in C:\WinDbgx86 or C:\WinDbgx64, since the default path is a bit unwieldy.

Next create a directory called c:\symbols. This will become your symbol cache.

Now run WinDbg.exe. It will be in the x86 or x64 directory.

In WinDbg, pick file → Symbol File Path and enter the following string

srv*c://symbols*http://msdl.microsoft.com/download/symbols

Note the forward slashes in the path.

Next you’ll need the symbol files. If you have the exact .Net version to match the dump, you can run

.loadby sos clr

The symbol file is included in the .Net install and will be imported automatically. If you version does not match, you can use psscor. There is psscor2 and passcor4, depending on the framework version. Search Google for the download, and save the dll file in C:\WinDbg(bit)\(bit) (ie c:\WinDbgx64\x64). Import psscor by running

.load passcor4

Now your debugging environment is ready. Open a dump file. I’ve got a (short) reference table below, but the way I work through it is this:

!ASPXPages → Will dump all ASPX requests and their corresponding threads. If the thread ID is “XXX” then it’s already completed and there is no stack.

!runaway → Shows all threads and their runtime. The slowest thread is on top. Once you’ve picked a thread ID, change to it’s context by executing:

~XXs → Where XX is the thread ID. ie: ~29s switches to thread 29 context. In a thread context run

!CLRStack → Lists the .Net call stack for the thread

!dso → Dump Stack Objects, will show all the objects on the stack.

That’s pretty well it. From here, you use !do <addr> to dump objects located at the specified address. Run !dso, take any object address, and run !do on it. You’ll get the idea.

Command Description
!sym noisey Print output about automatic symbol file fetching
!sym quiet Undo !sym noisey
~ List threads
~##s Switch to a thread where ## is the thread number. ie: ~10s switches to thread #10.
!threads Lists managed threads that is .Net threads
!ASPXPages List threads handing ASPX pages. If the return code is XXX then the thread is completed.
!runaway Shows how long threads have been executing. Look at the top entries for obvious slowdowns.
!CLRStack Once you’re in a thread context, CLRStack will show the call stack
!dso Dump Stack Objects – this is the good stuff. It shows all the objects in the stack
!do ADDR Dump an object
!da ADDR Dumps an array
!dumpheap [-stat] [-min ###] Dumps all the objects on the heap. Adding -stat will show a count by type and a sum of size, and help identify the large objects in the heap. Adding -min will filter out objects which are smaller than ###
!dae Dump All Exceptions
!pe Print exception, if any, for the current thread
!GCRoot ADDR Finds all the objects which reference the object specified at ADDR

Dumping an object

Consider the following:

You run !CLRStack and see the following:

000000001432d078 0000000077746eba [NDirectMethodFrameStandalone: 000000001432d078] <Module>.SNIReadSync(SNI_Conn*, SNI_Packet**, Int32)
000000001432d040 000007fee89669b7 DomainNeutralILStubClass.IL_STUB_PInvoke(SNI_Conn*, SNI_Packet**, Int32)
000000001432d120 000007fee894e9cf SNINativeMethodWrapper.SNIReadSync(System.Runtime.InteropServices.SafeHandle, IntPtr ByRef, Int32)
000000001432d190 000007fee894e6cb System.Data.SqlClient.TdsParserStateObject.ReadSni(System.Data.Common.DbAsyncResult, System.Data.SqlClient.TdsParserStateObject)
000000001432d230 000007fee894e587 System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()

The top three calls are native calls and can’t be debugged. They’re invoked by the .Net runtime to do some work. The last .Net call was SqlClient.TdsParserStateObject.ReadSni.

Now run !dso and scroll up to the top.

0:068> !dso
OS Thread Id: 0x5084 (68)
RSP/REG          Object           Name
000000001432CB00 000000024ff60b58 System.Reflection.RtFieldInfo
000000001432CBF8 000000024ff9ce90 System.Collections.Generic.Stack`1[[System.Byte[], mscorlib]]
000000001432CC40 000000016feb5318 System.Byte[][]
000000001432CC50 000000016feb0c48 System.Byte[]
000000001432CC60 000000012fd89b00 System.DefaultBinder
000000001432CC70 000000024ff9ce90 System.Collections.Generic.Stack`1[[System.Byte[], mscorlib]]
000000001432CC88 000000016feb0c48 System.Byte[]
000000001432CC90 000000016feb0c48 System.Byte[]
000000001432CCA0 000000024ff9ce70 System.Runtime.SynchronizedPool`1+GlobalPool[[System.Byte[], mscorlib]]
000000001432CCB0 000000024ff602f0 System.ServiceModel.Description.MessageDescription
000000001432CCB8 000000016feb0c48 System.Byte[]
000000001432CCC0 000000024ff9ce18 System.Runtime.SynchronizedPool`1[[System.Byte[], mscorlib]]
000000001432CCD8 000000024ff9ce70 System.Runtime.SynchronizedPool`1+GlobalPool[[System.Byte[], mscorlib]]
000000001432CCE0 00000000ffd40e40 System.Collections.Generic.Stack`1[[System.ServiceModel.Channels.TextMessageEncoderFactory+TextMessageEncoder+UTF8BufferedMessageData, System.ServiceModel]]
000000001432CD70 00000000ffd40e68 System.ServiceModel.Channels.TextMessageEncoderFactory+TextMessageEncoder+UTF8BufferedMessageData
000000001432D0C0 00000001dfefba18 System.Data.SqlClient.SqlCommand

Again, the top few lines are system level and can be ignored. Here you see the SqlClient SqlCommand child object.

Use !do to dump that object

!do 00000001dfefba18

From here the results will be specific to the object. It’s properties and values will be listed. There are two types of variables, reference and value. Value variables are almost always integers. The value column of the results will make it clear.

              MT    Field   Offset                 Type VT     Attr            Value Name
000007feeab15a48  40001e0        8        System.Object  0 instance 0000000000000000 __identity
000007feefb915c8  40002c3       10 ...ponentModel.ISite  0 instance 0000000000000000 site
000007feefb90280  40002c4       18 ....EventHandlerList  0 instance 0000000000000000 events
000007feeab15a48  40002c2      190        System.Object  0   shared           static EventDisposed
                                 >> Domain:Value  0000000001f7e030:NotInit  0000000003b89f20:000000010fe0b520 <<
000007feeab1c7d8  4001733       b0         System.Int32  1 instance           621735 ObjectID
000007feeab168f0  4001734       20        System.String  0 instance 000000020fe27720 _commandText
000007fee89a1818  4001735       b4         System.Int32  1 instance                4 _commandType
000007feeab1c7d8  4001736       b8         System.Int32  1 instance               30 _commandTimeout
000007fee8ea1668  4001737       bc         System.Int32  1 instance                3 _updatedRowSource
000007feeab1d608  4001738       d0       System.Boolean  1 instance                0 _designTimeInvisible
000007fee8ed46c0  4001739       28 ...ent.SqlDependency  0 instance 0000000000000000 _sqlDep
000007fee89a1b78  400173d       30 ...rameterCollection  0 instance 00000001dfefbaf8 _parameters

Use !do on the value to get the object referenced by that address. In the case of an SqlCommand object, _parameters and _commandText will be most insteresting

Advertisements