Getting started in ASP.Net debugging with WinDbg

We’ve all faced the dreaded application is slow or eats all my RAM. You take a memory dump and recycle the AppPool.  But then what? Some of the people I work with are memory dump gurus, but the barrier to entry seemed pretty high. This is a quick primer on how to dive into a dump file. I’m not an expert, and by no means will you be an expert from reading this. “Advanced .Net debugging in under an hour” is how our PFE explained it.

Note: The “bitness” of the dump file is important throughout. A 64bit dump requires the 64bit debugger and the 64bit symbols.

First install WinDbg. It’s a good idea to put it in C:\WinDbgx86 or C:\WinDbgx64, since the default path is a bit unwieldy.

Next create a directory called c:\symbols. This will become your symbol cache.

Now run WinDbg.exe. It will be in the x86 or x64 directory.

In WinDbg, pick file → Symbol File Path and enter the following string

srv*c://symbols*http://msdl.microsoft.com/download/symbols

Note the forward slashes in the path.

Next you’ll need the symbol files. If you have the exact .Net version to match the dump, you can run

.loadby sos clr

The symbol file is included in the .Net install and will be imported automatically. If you version does not match, you can use psscor. There is psscor2 and passcor4, depending on the framework version. Search Google for the download, and save the dll file in C:\WinDbg(bit)\(bit) (ie c:\WinDbgx64\x64). Import psscor by running

.load passcor4

Now your debugging environment is ready. Open a dump file. I’ve got a (short) reference table below, but the way I work through it is this:

!ASPXPages → Will dump all ASPX requests and their corresponding threads. If the thread ID is “XXX” then it’s already completed and there is no stack.

!runaway → Shows all threads and their runtime. The slowest thread is on top. Once you’ve picked a thread ID, change to it’s context by executing:

~XXs → Where XX is the thread ID. ie: ~29s switches to thread 29 context. In a thread context run

!CLRStack → Lists the .Net call stack for the thread

!dso → Dump Stack Objects, will show all the objects on the stack.

That’s pretty well it. From here, you use !do <addr> to dump objects located at the specified address. Run !dso, take any object address, and run !do on it. You’ll get the idea.

Command Description
!sym noisey Print output about automatic symbol file fetching
!sym quiet Undo !sym noisey
~ List threads
~##s Switch to a thread where ## is the thread number. ie: ~10s switches to thread #10.
!threads Lists managed threads that is .Net threads
!ASPXPages List threads handing ASPX pages. If the return code is XXX then the thread is completed.
!runaway Shows how long threads have been executing. Look at the top entries for obvious slowdowns.
!CLRStack Once you’re in a thread context, CLRStack will show the call stack
!dso Dump Stack Objects – this is the good stuff. It shows all the objects in the stack
!do ADDR Dump an object
!da ADDR Dumps an array
!dumpheap [-stat] [-min ###] Dumps all the objects on the heap. Adding -stat will show a count by type and a sum of size, and help identify the large objects in the heap. Adding -min will filter out objects which are smaller than ###
!dae Dump All Exceptions
!pe Print exception, if any, for the current thread
!GCRoot ADDR Finds all the objects which reference the object specified at ADDR

Dumping an object

Consider the following:

You run !CLRStack and see the following:

000000001432d078 0000000077746eba [NDirectMethodFrameStandalone: 000000001432d078] <Module>.SNIReadSync(SNI_Conn*, SNI_Packet**, Int32)
000000001432d040 000007fee89669b7 DomainNeutralILStubClass.IL_STUB_PInvoke(SNI_Conn*, SNI_Packet**, Int32)
000000001432d120 000007fee894e9cf SNINativeMethodWrapper.SNIReadSync(System.Runtime.InteropServices.SafeHandle, IntPtr ByRef, Int32)
000000001432d190 000007fee894e6cb System.Data.SqlClient.TdsParserStateObject.ReadSni(System.Data.Common.DbAsyncResult, System.Data.SqlClient.TdsParserStateObject)
000000001432d230 000007fee894e587 System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()

The top three calls are native calls and can’t be debugged. They’re invoked by the .Net runtime to do some work. The last .Net call was SqlClient.TdsParserStateObject.ReadSni.

Now run !dso and scroll up to the top.

0:068> !dso
OS Thread Id: 0x5084 (68)
RSP/REG          Object           Name
000000001432CB00 000000024ff60b58 System.Reflection.RtFieldInfo
000000001432CBF8 000000024ff9ce90 System.Collections.Generic.Stack`1[[System.Byte[], mscorlib]]
000000001432CC40 000000016feb5318 System.Byte[][]
000000001432CC50 000000016feb0c48 System.Byte[]
000000001432CC60 000000012fd89b00 System.DefaultBinder
000000001432CC70 000000024ff9ce90 System.Collections.Generic.Stack`1[[System.Byte[], mscorlib]]
000000001432CC88 000000016feb0c48 System.Byte[]
000000001432CC90 000000016feb0c48 System.Byte[]
000000001432CCA0 000000024ff9ce70 System.Runtime.SynchronizedPool`1+GlobalPool[[System.Byte[], mscorlib]]
000000001432CCB0 000000024ff602f0 System.ServiceModel.Description.MessageDescription
000000001432CCB8 000000016feb0c48 System.Byte[]
000000001432CCC0 000000024ff9ce18 System.Runtime.SynchronizedPool`1[[System.Byte[], mscorlib]]
000000001432CCD8 000000024ff9ce70 System.Runtime.SynchronizedPool`1+GlobalPool[[System.Byte[], mscorlib]]
000000001432CCE0 00000000ffd40e40 System.Collections.Generic.Stack`1[[System.ServiceModel.Channels.TextMessageEncoderFactory+TextMessageEncoder+UTF8BufferedMessageData, System.ServiceModel]]
000000001432CD70 00000000ffd40e68 System.ServiceModel.Channels.TextMessageEncoderFactory+TextMessageEncoder+UTF8BufferedMessageData
000000001432D0C0 00000001dfefba18 System.Data.SqlClient.SqlCommand

Again, the top few lines are system level and can be ignored. Here you see the SqlClient SqlCommand child object.

Use !do to dump that object

!do 00000001dfefba18

From here the results will be specific to the object. It’s properties and values will be listed. There are two types of variables, reference and value. Value variables are almost always integers. The value column of the results will make it clear.

              MT    Field   Offset                 Type VT     Attr            Value Name
000007feeab15a48  40001e0        8        System.Object  0 instance 0000000000000000 __identity
000007feefb915c8  40002c3       10 ...ponentModel.ISite  0 instance 0000000000000000 site
000007feefb90280  40002c4       18 ....EventHandlerList  0 instance 0000000000000000 events
000007feeab15a48  40002c2      190        System.Object  0   shared           static EventDisposed
                                 >> Domain:Value  0000000001f7e030:NotInit  0000000003b89f20:000000010fe0b520 <<
000007feeab1c7d8  4001733       b0         System.Int32  1 instance           621735 ObjectID
000007feeab168f0  4001734       20        System.String  0 instance 000000020fe27720 _commandText
000007fee89a1818  4001735       b4         System.Int32  1 instance                4 _commandType
000007feeab1c7d8  4001736       b8         System.Int32  1 instance               30 _commandTimeout
000007fee8ea1668  4001737       bc         System.Int32  1 instance                3 _updatedRowSource
000007feeab1d608  4001738       d0       System.Boolean  1 instance                0 _designTimeInvisible
000007fee8ed46c0  4001739       28 ...ent.SqlDependency  0 instance 0000000000000000 _sqlDep
000007fee89a1b78  400173d       30 ...rameterCollection  0 instance 00000001dfefbaf8 _parameters

Use !do on the value to get the object referenced by that address. In the case of an SqlCommand object, _parameters and _commandText will be most insteresting

Advertisements

Finding unused app pools with powershell

I’ve been tasked with cleaning up our UAT environment, and needed a way to find unused App Pools. To make things more exciting, we don’t have Microsoft.Web.Administration.ServerManager installed, so the native powershell way wasn’t going to work. This is my solution, by invoking appcmd and string parsing the results.

$command = "c:\windows\system32\inetsrv\appcmd.exe list app"
$apps = Invoke-Expression $command

$command = "c:\windows\system32\inetsrv\appcmd.exe list apppool"
$pools = Invoke-Expression $command
foreach ($pool in $pools)
{
    $used = $false
    foreach ($app in $apps)
    {
        $apppool = $app
        $apppool = $apppool.Substring($apppool.indexof(":")+1)
        $apppool = $apppool.Replace(")","")
        $apppool = $apppool.trim()
        if ($pool.contains($apppool))
        {
            $used = $true
            break
        }

    }
    echo "$used $pool"
}

Git for Windows

Git is the popular source control tool which has become the darling of the internet. Unlike subversion, git keeps a local copy of the repository, allowing offline commits. This makes it a lot harder to do a full Windows port. Luckily, there is a package called msygit, which has all the dependencies bundled with it.

The default is to run git in a BASH prompt, but I can’t imagine why you would want to do that. If I’m running on the windows command prompt for other stuff, I don’t want to switch out of it into some hackish BASH prompt to run git. I just switch that option and let the good times roll.

Line endings are a little more complication. I use the default “check out Windows style, check in Unix style”, but I’m thinking to fine tune that a bit. Any good Windows text editor can handle Unix line breaks. If I’m doing a module for MediaWiki or Drupal, I might force it to stay in Unix mode. Conversely, if I’m doing something very windows-centric like ADSI or WMI scripts in PHP (or even VBS), then I’ll want it Windows style on the remote side. The idea is that if someone were to download a ZIP of a repo from git hub, they should be able to view that file using the native console tool (cat or type), and not have some crazy line breaks. More information about handling it is in the GitHub help article “Dealing with line endings“.

One other small thing is that on Windows systems with an NTLM proxy (looking at you TMG), you need to specify the proxy settings in the environment variable http_proxy and https_proxy. Setting it in http.proxy didn’t seem to do the job. I found a bunch of posts dealing with this, and having something to do with git not calling –proxy-ntlm on cURL and it being a hassle to override. I wimped out and used the environment variable.

PHP, WMI and Microsoft NLB

I’ve been playing with my NLB cluster, and wanted to find a way to evict nodes which were failing some test. This functionality exists in most hardware load balancers, but isn’t something Microsoft NLB does natively. Controlling NLB nodes is done using WMI, and the classes are fairly well documented. I was also fishing for an excuse to try ReactPHP, which is an event driven non-blocking library for PHP.

The result was my MicrosoftNLB class and the simplewatcher.php script. I used wmi2php to generate the boiler plate, and write a wrapper class called NLBNode. There is still a lot of work to be done, but it’s achieved MVP for me. Simplewatcher.php which will get all the peers in an NLB cluster, challenge them for a specific URL, and evict any node which fails. This is good for taking a node out of load (rename nlb.php to nlb.oos), or for stopping a node if IIS stops responding. If a node fails at the network layer (ie hardware or OS failure), it will stop participating in the cluster automagically. It will not put a node back into load, you must do that manually.

 

This is also my first composer project, and you’ll need to use it to get ReactPHP installed. To use the node watcher, just edit the first few lines of examples\simplewatcher.php, and run it from the command line.

wmi2php

I’ve been playing with my NLB cluster and want to use WMI to control the nodes. The PHP COM library works great, and I’ve had some success. The properties of COM objects aren’t exposed to you, so if  you print_r($somevar), you get nothing. I’ve been using ScriptomaticV2 and the NLB WMI documentation to muddle through it, and I found I was doing a lot of copying and pasting. Well no more! I worked through the major methods in Scriptomatic and re-implemented them in PHP. Now I can export PHP class files for any WMI class.

Check it out in GitHub.

It should be noted that I’m not a huge fan of “getters and setters”, so this uses overrides and an array of valid properties. I’m planning to expand it to create more “native” classes.

So now I can:

wmi2php.php generate --namespace=root/CIMV2 --class=Win32_ComputerSystem

Enjoy.

Windows 2012 classic theme

Do you like the cyan color theme in Windows Server 2012? No you don’t, it’s hideous.

win2k12-default

Microsoft removed the ability to customize the color scheme, so you’re stuck with that awful look right? Wrong! The good news is you can export the Windows Classic theme from Windows 7.

  1. On your Windows 7 machine, right click the desktop and pick personalize.
  2. Under Basic and HIgh Contrast Themes, activate Windows Classic
  3. Now scroll back up to My Themes
  4. Right click on your current theme and pick “Save theme for sharing”
  5. Save it as classic.themepack (or whatever you want)
  6. On a Windows 2012 machine, double click the file. It will automatically be imported and set as your current theme. No more awful cyan!

win2k12-classic

 

Now if anyone knows how to get rid of the ridiculous 8 pixel wide border, please let me know!

Windows Server 2012 – Network Load Balancing

I’ve been wanting to play with a load balanced IIS cluster for a while. Microsoft Network Load Balancing is included as a component of Windows Server 2012. NLB uses a hearbeat to check if nodes are still online, but it is not application aware. My environment is as follows:

  • A domain controller hosting the contoso.local domain
  • A database server running SQL 2012
  • A load balancing cluster host
  • Two web servers, WEB1 and WEB2

Setting up NLB

Nodes in an NLB cluster should be dual-honed, with 1 NIC for management and the other for NLB. The NLB NIC will take on the MAC of the cluster IP, so you’ll lose control over that interface. I disabled Client for Microsoft Networks on that interface as well. You can also use the registry and turn off automatic DNS registration for that interface. All nodes need to have the NLB feature installed, which is done through server manager.

When the nodes are setup, configure the cluster host:

  1. Install the NLB feature and open the NLB manager (nlbmgr.exe)
  2. Right click on Network Load Balancing Clusters and pick New Cluster
  3. Enter the management IP for the first node in the cluster. Click Connect
  4. NLB manager will connect to that node and bring up a list of interfaces. Select the interface to be used for NLB binding
  5. Click next past host parameters
  6. Set a cluster IP address
  7. Click next past cluster parameters
  8. Setup port rules for HTTP (80) and HTTPS (443)
  9. Finish

If you’re using VirtualBox there are a few possible gotchas:

  • You need to enable promiscuous mode for the NICs in the VBox config
  • NLB had to be set for Multicast, which doesn’t make sense, but it worked

Thanks to Bearrito on pastebin for this help.

Some links about NLB and unicast