This is just a brief article to note the debugging work flow I used while recently developing the Associated Windows Shell Extension. Windows Shell Extensions can be a bit tricky to debug, especially when you use the live Windows Explorer to test your work, as I do. The work flow I use is not very complicated, and doesn’t require hacking the Windows Registry (as some techniques do). There are other work flows that run separate Explorer processes, and do not require elevated privileges, but this is the work flow I use.
It should be noted that some actions documented herein will likely require elevated privileges. I am always Administrator, so do not typically run into difficulties. If you do not operate as Administrator, you may need to do so in order to successfully accomplish tasks listed in this article.
[ Keep ‘Em Separated ]
While developing a Shell Extension, I like to set up a folder that will contain all the files that will ultimately be included in the distributed product. This is never the Visual Studio project’s debug or release folder. Aside from having issues compiling if you have a Shell Extension active in Explorer, you run the risk of having the files deleted by a project “Clean” or just by hand-deleting those folders.
You can place the folder to contain these files anywhere you like, but I like to create one under the target folder (e.g., “C:\Program Files (x86)\…”). This lets me see how the code will behave in its live folder as I develop it.
As I’m developing, I manually copy the build product (the Shell Extension DLL) into the target folder and install it into the running Explorer process from there. This, of course, requires command-line actions, but having a UN*X background, I almost always have a DOS window open all the time.
In any case, you should not use your project debug or release folder as your working folders. You will often find that you will want the in-development Shell Extension active and usable in Explorer while you are also tweaking and re-compiling the project. If you have the Shell Extension found in the project’s debug or release folders installed and active in Explore, your build process will be unable to overwrite it, and you will find yourself performing addition uninstalls each time you want to build again.
[ Adding Your Extension ]
Once you’ve built your working Shell Extension DLL, you’ll need to register it into the Explorer process. Registration is performed using the
regsvr32 command-line utility.
regsvr32 /s Associated_x64.dll
This call registers the Shell Extension with Explorer. The “/s” option merely suppresses the GUI confirmation dialog. It may be useful to call
regsvr32 without this switch just to confirm that your Extension registered successfully.
It should be noted that this call does not immediately load the Extension DLL into Explorer. Not until the Extension is used does the DLL actually get loaded and locked into Explorer’s process space. For example, a Context Menu Shell Extension would be loaded only when a right-click context menu is activated in Explorer. So, until the DLL is actually loaded for use by Explorer, it can be freely removed by another call to
regsvr32 which will unregister it:
regsvr32 /s /u Associated_x64.dll
If your Extension DLL is locked into Explorer’s process space, however, this call to unregister will not completely detach the shared library. It will remain locked in Explorer’s process space. Explorer must be restarted to completely release the shared library. Usually, this is done using a reboot; you’ll often see program installers do this when you uninstall applications that have Extensions locked in Explorer’s process space. With elevated privileges, however, you needn’t be that drastic.
[ Debugging the Bugger ]
Of course, you’ll only be able to actually debug an active Extension if you have a debug build capable of loading into Explorer. This means the shared library can find all of its dependent files, either in it’s working folder, or somewhere along the PATH. How this is configured is left as an exercise for the reader, but your Extension will fail to load into Explorer if any of its dependent files are not locatable by one of these methods.
After a successful registration of your debug build with Explorer, you’ll want to start the process of debugging it. This is accomplished by loading up the Extension project in Visual Studio (it probably already is), and setting break points within the code where you wish the Extension to stop during its execution. With the debug environment prepared, debugging begins when you attach your Visual Studio debugging session to the running Explorer process. This is done by selecting “Attach to Process…” from the Debug menu:
This opens a dialog that lists all the current processes, sorted alphabetically. Locate the “explorer.exe” process, and select “Attach”:
Your debug session is now active. You may now interact with your Extension in the active Explorer, and your breakpoints will hit just as though you were running the Extension as a stand-alone process.
When you are finished, you can simply detach from the running process (terminating your debug session will also work, but may kill the host process.)
[ Reset ]
As I’ve mentioned, once your Extension is loaded into Explorer’s process space, Explorer itself must be terminated in order to release the shared library. Rebooting Windows accomplishes this, but this sort of solution is horribly disruptive to the development work flow. Instead, you can manually terminate and restart Explorer, and in between, unregister your Shell Extension.
I use the following simple batch file to accomplish all these steps in one call:
taskkill /F /IM explorer.exe
regsvr32 /s /u %1
The steps simply terminate the running Explorer process, unregister the indicated Shell Extension file, and then restart Explorer. Of course, you will need elevated privileges in order to kill Explorer, so if you do not run with these by default, you may need to modify certain calls (like “taskkill”) with the “runas” option in order to request elevated privileges.
Once the shared library has been removed from Explorer, you are free to overwrite the Extension file with your updated build, and then re-register to star the fun all over again. 😉