1 / 44

Advanced Kernel Debugging

Advanced Kernel Debugging. Andre Vachon Software Development Lead Windows Product Feedback Microsoft Corporation. Goals, Non-Goals. Goals Give you insights into what the debugger can do Will be referring you to the docs often Cover important debugger features and changes Non-Goals

Télécharger la présentation

Advanced Kernel Debugging

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.


Presentation Transcript

  1. Advanced Kernel Debugging Andre Vachon Software Development Lead Windows Product FeedbackMicrosoft Corporation

  2. Goals, Non-Goals • Goals • Give you insights into what the debugger can do • Will be referring you to the docs often • Cover important debugger features and changes • Non-Goals • Teach you how to debug your component • Technology sessions and whitepapers have in depth discussions and information about troubleshooting various OS technologies • Provide details about specific debugger commands • The documentation is the best source of information of all the debugger commands and features • These slides should not be used as debugger reference material

  3. Session Outline • Debugger Installation and configuration • WinDbg UI • Remote debugging options • Dump Files • Dump file debugging • 64-bit debugging • Debugger core engine changes • Debugger commands • Debugger extensions

  4. Debugger Teams Current Focus • Reliability • Managed code debugging • User-mode only technology today • Symbol server • !analyze • Customers satisfaction • Tell us what you need

  5. Debugger Install • Always get the latest release • Latest releases are the most reliable, with most fixes • http://www.microsoft.com/whdc/ddk/debugging • The debugger in the DDK and SDK are out-of-date, only there for convenience • We support all target OS versions back to Windows NT 4.0 SP3 • We support all host OS versions back to Windows NT 4.0 SP3 • Both user mode and kernel mode debugging • Windows 9x kernel debugging is not supported • The package contains • Docs – debugger.chm – this is your friend! • Information about all debugger commands, extensions and tools • All the debuggers (WinDbg, CDB, NTSD, KD) and supporting DLLs • Extensions for all OS versions - right versions are loaded automatically • Variety of other debugger related tools • ADPlus, GFlags, KDbgCtrl, Logger, etc.

  6. Common Issues • Local variables are wrong • Turn off compiler optimizations • Breakpoint issues • Bp sets a breakpoint on an address; Bu sets a breakpoint on a symbol • Use the “bu” command if you want to save breakpoints in the workspace, or want them to work across module load\unload\load • “I found a debugger problem” • Report it! We don’t release until all known bugs are fixed • 1394 debugging won’t connect • Disable 1394 host controller on the target • COM port debugging won’t connect • Disable legacy USB support in the BIOS • USB 2.0 debugging • Initial trial implementation is now available - requires a special USB cable

  7. WinDbg Docked Windows • WinDbg windows can be floating or docked • Docked windows shrink and grow with the WinDbg frame they belong to • Docked window positions and sizes stay relative to each other as the frame changes • Undocked (a.k.a., floating) windows are always on top of the WinDbg window • WinDbg supports multiple docks • Individual WinDbg windows can be part of any dock • Can be moved to any dock at any time • New windows • We recommend all windows be docked • Use multiple docks on a multimon system • Ctrl-Tab iterates through all windows in all docks

  8. WinDbg Tabbed Windows • WinDbg windows that are part of a dock can be tabbed • Tabbed windows are overlaid • They are resized as a group when an edge is dragged • When a tabbed window is dragged to another location, it is moved individually

  9. WinDbg Windows Menus • Each window has its own menu • Menu is accessed by • Left click on the menu button (next to the close button) • Right click on the title bar of a window • Right click on the tab of a tabbed window • Disassembly Window menu • Highlight instruction from current source line – used to match all the assembly to a source line, providing the equivalent of a .cod file • Show source line \ source file for current instruction – display source information for each assembly line

  10. WinDbg Demo • Docked windows • Multiple docks • Tabbed source windows • Highlight assembly instructions for current source line

  11. Debugger Remoting • Very valuable if you have testing done in multiple remote locations • Provides a complete remoting experience • Can be enabled at startup or dynamically • -server, -remote, or .server to enable a remote client to connect • -premote to enable a process server the host can connect to • Works with mixed debugger: WinDbg connecting to CDB • See docs for extensive descriptions and details • Supports debugging across firewalls • Use debugger proxies

  12. Targets • Kernel mode • Target is the machine being debugged. Multiple “transports” supported • COM port or 1394 • Works on all platforms • USB 2.0 support in the works • Trial binaries available in the current release • USB 2.0 debugging will have similar limitation to 1394 debugging • BIOS, PCI bridge, etc issues • Additionally requires special USB device (peer-to-peer networking device) • Local KD • Only useful for basic data inspection • No breakpoint support (no single machine debugging) • User mode • Machine where the process being debugged runs • Most often, host and target are the same • To separate the host from the target, use a “process server”, dbgsrv.exe • See docs for complete details

  13. Host • Defined to be • The machine where the core debugger engine runs • The machine that needs access to the symbols • Windbg.exe, kd.exe, ntsd.exe, and cdb.exe can be host debuggers • They can also be configured to be remote debuggers • The host can connect to the target in many ways • windbg –k [other param] to connect to a kernel target • -k kdsrv: to attach to a KD connection server • windbg – p | pe | pn | pt to connect to a user mode target on the same machine • windbg –premote to connect to a process server

  14. Remote Client And Proxies • The host must enable remoting by using .server or -server • Can be started dynamically • Multiple protocol supported, including • Named Pipe, TCP, SSL • The remote side connects using –remote • Example: cdb –remote tcp:server=,port=1639 • All commands are sent from the remote to the host, executed on host, and output sent back to the target • WinDbg source file lookup (using .lscrpath) and other UI only commands (.cls, .open) are the only exception • Process server \ KD Connection server • Makes low-level target data remotely available for the host debugger • Proxy \ repeater • Machine running a proxy tool, sitting between a remote and a host, or between a host and target

  15. User Mode Debugging: Non-Invasive Attach • In a normal debugging scenario, the debugger “attaches” to a process • OS feature that connects the debugger and debuggee • OS sends events to the debugger • Module loads, exceptions, etc. • Attaches do not always work • A debugger may already be attached to the process • Loader lock hang can cause attach to fail • Non-invasive attach is the debugger controlling the process externally • All threads are suspended • All data can be inspected • No standard events will be sent (no breakpoints)

  16. User Mode Debugging: Multi-Process\System Debugging • The debugger can attach and debug multiple processes or systems simultaneously • .attach, .create can be used to attach to\create additional processes • Use ‘|s’ to switch between processes • .opendump can be used to open multiple dumps • .attach –premote can attach to a new process server • Multiple dumps or process servers are treated as different “systems” • Use ‘||s’ to switch between systems

  17. Kernel Dump Files • OS can generate 3 different types of dumps • Full (all physical memory) • Kernel (pages mapped in the kernel address space) • Minidump (64 K) • Header, basic OS information, PRCB • OS Module list (loaded and unloaded) • Faulting Process, Thread, Stack and context • Data pages pointed to by the context and bugcheck params • The kernel can also store additional data in minidumps by calling IoAddTriageDumpDataBlock() • Future addition of SMBIOS information • Kernel minidumps only store data – no image information • The debugger requires access to images to debug a minidump • OS also provides callbacks to drivers so they can store additional state (added to all dumps)

  18. Dump File Generation • Operating system generates the crash time at the time of the crash • Raised IRQL – no code can be paged in • Special path in the disk drivers to support writing dump files • Dump files get written to the page file – guaranteed, locked down sectors on disk • Type of dumps generated by the OS : • Windows XP client: minidump is the default • Windows Server 2003: kernel dump is the default • Default can be reconfigured by the end-user • When Full or Kernel dumps are generated by the OS, upon reboot a minidump is created from the original dump • Called “Dump conversion” • Equivalent to using “.dump /m” in the debugger • OCA error reports always contain the minidump

  19. Debugging A Minidump • Debuggers: • Kernel minidumps require using KD or WinDbg • Both WinDbg and VS supports debugging user mode minidumps • Step 1: Get the images • A minidump contains minimal data. The code streams are not included, except for the code page pointed to by eip or bugcheck params • All Microsoft kernel mode code for recent OS is on the Internet symbol server • You need to the latest debugger to get access to the images • Step 2: Debugger maps the images and merges the images with the data stored in the minidump • Debug info from the images is used to look for the symbols • If you have the wrong image, wrong symbols will be loaded • Step 3: Get symbols • Symbol server is again the best solution

  20. Symbol Loading • http://msdn.microsoft.com/download/symbols is the location where we try to get every public symbol file in Microsoft indexed • We work with many groups across Microsoft to get all the files in this unique location • We store compressed files on the server • The debugger needs to use a local cache to store the symbol files • The debugger automatically populates the local cache specified in the debugger • Full access to the Internet symbol server requires debugger 6.2.13 or later • New EULA must be accepted • Managing multiple versions of symbols and images can be hard • Store all your binaries in your own symbol server • Designed to store all versions of images and symbols you create. • Based on timestamps for images and GUID for PDBs • SymStore – tool to build a symbol server • Easily scripted • Run it as part of the build process

  21. Source Code Loading • KD \ NTSD \ CDB • Source code is loaded on the host machine • .scrpath to set the path • WinDbg • WinDbg can load its own version of the source code • Use .lsrcpath when connecting remotely to get local source access • Source code need not be loaded on the same machine • .srcpath | .lsrcpath \\<machine_name>\share\<root_of_source_tree> • The debugger engine does path matching to find the actual files in the source tree, if the paths are close to identical to where the source code was built • If the paths are significantly different, include each directory where source is located

  22. Source Server • New feature in the debugger to automatically load the correct version of the source code automatically • Set your srcpath to ‘srv*’ • How does it work? • The build process must runs tools to encode the version of the source code into a PDB • When debugging, the debugger extracts the source code information from the PDB • The debugger calls the source control system to extract a copy of that source file on your machine • Requires the debugger machine have the source control tools installed • Requires the machine and user account have source access • Further documentation in the debugger package

  23. !analyze • Debugger extension designed to categorize bugs • Automated analysis • Simplifies analysis of known problems • Understand various states of the OS • Provides starting point to analyze complex problems • Extracts commonly used debugging information • Applies common triaging techniques and well known analysis steps • Results of the analysis are • “Bucket ID” • Unique string representing the bug • An Owner for the problem, extracted from triage.ini • In verbose mode, using !analyze -v • Detailed list of all the data found during the analysis

  24. !analyze Algorithm • Use bugcheck parameters to extract basic information • Scan the stack for special functions such as Trap0E • Analyze frames on the final stack to determine most likely culprit • Different weights are assigned to routines • Highest weight frame found on the stack is treated as the culprit • If stack does not yield an interesting frame, analyze raw stack data • Check for presence of memory or pool corrupting drivers • Hardware problems are quite common – no way to find these from a dump file today • Check for corrupted code streams (possible bad RAM) • Check for other possible problems, such as invalid call sequences • Bad CPU, chipset problems • Bad disk

  25. 64-Bit Debugging – What’s The Same? • WinDbg UI is identical • All debugging features work on both 32-bit and 64-bit • All debugger commands and extensions work on both 32-bit and 64-bit • Exceptions are processor specific debugger commands and extensions • MSRs, !pcr • Stepping, breakpoints work identically • Dump files work identically • Symbols work identically • Symbols are specific to the compiler and linker version • All general OS data structures should be identical • Again processor specific data structures (CONTEXT, PTE) will differ • Some macros that use processor specific features are different

  26. 64-Bit Debugging – What’s Different ? • Pointers are 64 bits instead of 32 • Be careful when casting pointers. • User dq to dump quadwords • Assembly languages are unique to each processor • x64 assembly is very similar to x86 • IA64 is totally different • Instructions are bundled 3 at a time, predicates, etc … • Calling conventions are unique to each processor • x64 passes first 4 params in registers • IA64 supports rotating registers • Source level debugging should hide all these differences • The OS tries to abstract all processor and platform specific features • Looking at PTEs and other processor specific structures internal to the OS will expose differences

  27. KDbgCtrl.exe • New tool, works on Windows Server 2003 and above • Tool to manipulate kernel debugging options on the target machine; Examples • du - Disable kernel debugger user exception handling • d - Disable kernel debugger • sdb <size> - Set kernel DbgPrint buffer size • See docs for complete details • Windows Server 2003 and later • Newer features only work on the latest OS

  28. Core Debugger Changes • Unicode support • Changed dbghelp.dll and dbgeng.dll to be fully Unicode • Symbols and file names can be in Unicode • WinDbg UI is unicode (CDB, KD remain ANSI) • Improved stack walking • New support for “Debugger programs”

  29. Managed Code Support • Requires Windows Longhorn build • Symbols looked up by address • Correct source lines looked up by address • Managed frames display locals • Local variables are correct for primitive types and the raw objref for complex types • ?? can get the address of a local by name • Symbols will be resolved to addresses for jitted code and static members of loaded types • x can be used with managed modules and symbols • Managed exception info is displayed for second-chance CLR exceptions • F9 in WinDbg can be used to set breakpoints • sx now has a 'clr' option for controlling the exception used by the CLR for managed exceptions • Many more features to come in future version of WinDbg and longhorn

  30. Debugger Commands • Read the docs for information on all commands • .formats: Converts numbers to timestamps • bm: sets breakpoints on all commands that match the wildcard • bp /p <EPROCESS/PEB> • .enable_unicode: Show ushorts as chars • .enable_long_status: Show signed int as hex • .frame: Switch the current frame on a stack • .dump: Generates a dump file of the target machine by reading data from the target • Kernel full dumps should only be done over 1394 • .pcmd: allows addition to the debugger prompt • example .pcmd –s !thread

  31. Interesting Debugger Commands • kP – Shows all function parameters, each on one line • As accurate as DV • Different algorithm as kb 0:000> kP ChildEBP RetAddr 0006fb34 77f6492b ntdll!DbgBreakPoint 0006fc90 77f55349 ntdll!LdrpInitializeProcess( struct _CONTEXT * Context = 0x0006fd30, void * SystemDllBase = 0x77e60000, struct _UNICODE_STRING * UnicodeImageName = 0x0006fce8, unsigned char UseCOR = 0xb4 '', unsigned char ImageFileOptionsPresent = 0x00 '') +0xdf3 0006fd1c 77f75d87 ntdll!LdrpInitialize( struct _CONTEXT * Context = 0x01000108, void * SystemArgument1 = 0x001a0019, void * SystemArgument2 = 0x77f630de) +0x186 00000000 00000000 ntdll!KiUserApcDispatcher+0x7

  32. Interesting Commands • lm [options]: detailed information about a module (replaces !drivers) kd> lmv mntos* start end module name 00400000 00618080 ntoskrnl (private pdb symbols) c:\localsym\ntoskrnl.pdb\DD3B00F5D69B4D1D8DADA85166188F862\ntoskrnl.pdb Loaded symbol image file: ntoskrnl.exe Mapped memory image file: c:\localsym\ntoskrnl.exe\404D5566218080\ntoskrnl.exe Image path: \\winbuilds\release\XPSP2\2094\usa\x86fre\bin\ntoskrnl.exe Timestamp: Mon Mar 08 21:25:58 2004 (404D5566) Checksum: 0021AF90 ImageSize: 00218080 File version: 5.1.2600.2094 Product version: 5.1.2600.2094 File flags: 0 (Mask 3F) File OS: 40004 NT Win32 File type: 1.0 App File date: 00000000.00000000 Translations: 0409.04b0 CompanyName: Microsoft Corporation ProductName: Microsoft« Windows« Operating System InternalName: ntoskrnl.exe OriginalFilename: ntoskrnl.exe ProductVersion: 5.1.2600.2094 FileVersion: 5.1.2600.2094 (xpsp_sp2_rc1.040308-1920) FileDescription: NT Kernel & System LegalCopyright: Microsoft Corporation. All rights reserved.

  33. Interesting Commands • ln: Shows source and symbolic information about the specified code address kd> ln 4bf339 d:\xpsp_rc1\base\ntos\mm\sysload.c(459) Source Depot: depot:2003 //depot/xp/base/ntos/mm/sysload.c#1 (004bf339) ntoskrnl!MmLoadSystemImage | (004bf53f) ntoskrnl!CmpQueryKeyName Exact matches: ntoskrnl!MmLoadSystemImage(UNICODE_STRING*, _UNICODE_STRING*, UNICODE_STRING*, unsigned long, void**, void**) • !address: Shows general information about an address kd> !address 77f75a58 77f50000 : 77f51000 - 00070000 Type 01000000 MEM_IMAGE Protect 00000020 PAGE_EXECUTE_READ State 00001000 MEM_COMMIT Usage RegionUsageImage FullPath ntdll.dll

  34. Debugger Commands • .kdfiles • Use the debugger to transfer updated versions of drivers over debug port to replace the version currently installed on the target • On each OS Mod load operation, the target checks with the debugger if an updated driver is present • Normally only works for regular drivers (loaded by MmLoadSystemImage). You need to use the special boot loader with debugger enabled to change boot drivers • wt: single step trace of a function and subfunctions • Numerous improvements to make the output more useful • Option to control depth of tracing, increasing performance • .call: Execute functions in the target process\system • BE VERY CAREFUL with this command. It can easily cause deadlocks in the target process • .call should not be used as a replacement for debugger extension

  35. Aliases • Aliases are character strings that are automatically replaced with other character strings • User-named aliases are set and are named by the user • Fixed-name aliases are set by the user, and are named $u0, $u1, ..., $u9 • Commands to manipulate user named aliases • as sets an alias • ad clears an alias • al lists all aliases • Commands to manipulate fixed named aliases • r sets an alias • Aliases are always stored as strings • String replacement is done before any evaluation is performed on the command \ expression

  36. Debugger Expressions • Pseudo registers: Special symbols recognized by the debugger which contain specific, common values • Examples: $thread – current thread, $peb – current peb (user mode), $ra – return address • ?? *@$teb will dump the current TEB • r? can be used to assign types information to a pseudo register ($t0) • ?? : C++ expression evaluator • Supports fully typed expressions – pointers, classes, arrays • Special handling for some preprocessor tokens CONTAINING_RECORD 0:000> ?? LdrDataTableEntry struct _LDR_DATA_TABLE_ENTRY * 0x6fc28 0:000> ?? ((ntdll!_LDR_DATA_TABLE_ENTRY *)0x6fc28)->FullDllName struct _UNICODE_STRING +0x000 Length : 0x30 +0x002 MaximumLength : 0x7ffe +0x004 Buffer : 0x00010000 "=::=::\“ 0:000> ?? LdrDataTableEntry->FullDllName.Buffer[5] unsigned short 0x3a

  37. Debugger Command Programs • A debugger command program is a small program consisting of debugger commands along with control flow tokens such as .if, .for, and .while • This a type of scripting. NOTE: in the debugger docs the term “script” refers to list of commands stored in a file which are passed to the debugger • Lets you build complex commands without having to write a whole debugger extension • A debugger command program can be executed from the command prompt, or stored in a file • Complete documentation about all the supported tokens and programming rules can be found in the docs

  38. Example Command Program $$ Walking the LDR_DATA_TABLE_ENTRY List $$ Get module list LIST_ENTRY in $t0. r? $t0 = &@$peb->Ldr->InLoadOrderModuleList $$ Iterate over all modules in list. .for (r? $t1 = *(ntdll!_LDR_DATA_TABLE_ENTRY**)@$t0; (@$t1 != 0) & (@$t1 != @$t0); r? $t1 = (ntdll!_LDR_DATA_TABLE_ENTRY*)@$t1->InLoadOrderLinks.Flink) { $$ Get base address in $Base. as /x ${/v:$Base} @@c++(@$t1->DllBase) $$ Get full name into $Mod. as /msu ${/v:$Mod} @@c++(&@$t1->FullDllName) .block { .echo ${$Mod} at ${$Base} } ad ${/v:$Base} ad ${/v:$Mod} }

  39. Scripting • List of commands that can be passed as input to the debugger • Designed for common series of commands to be run in automation scenarios • Commands are read line by line, as input is needed by the debugger • Script don’t allow for control flow (loops branching) • Because of this, they do allow for embedded ‘g’ ‘p’ ‘t’ operations • To load and run a script • Use –c command line option • Use $<, $><, $$<, and $$>< • Example of Scripting • cdb –remote <params> –c “.remote_exit; .logopen foo;!analyze –v;.logclose;.shell copy foo \\share;.reboot”

  40. Debugger Extensions • New debugger extensions • New !address (still under development): Given an address, described what type of memory it is (pool, code, heap, etc.) • New !chkimg: Analyzes code streams for possible corruption • Requires access to the original image • Compares the in-memory image to the disk image • New 0x10 flag to !process to get accurate user mode symbols for all processes • Significant updates to the longhorn version of ndiskd • Numerous minor extensions fixes and changes based on developer feedback • FYI If you load your own debugger extensions • The version command shows the list of loaded debugger extensions • Commands are searched for in each DLL, in order • Be careful of duplicate command names

  41. Writing Debugger Extensions • Samples, headers, and documentation all part of the debugger package • Not installed by default. You must do a custom install and select the optional SDK component. • WDbgExts.h is the old debugger extension interface • Still widely used, but limited in functionality • DbgEng.h is the full-blown debugger API • Hundreds of methods to control all aspects of debugging • See documentation (debugext.chm) for full details • Separate presentation on writing WinDbg debugger extensions

  42. Call To Action • Read the debugger docs! • Send us feedback – about the tool or the docs • Look at your OCA failures • These are REAL customer problems • Fix your pool corruption problems • Tell us about the bugs you fix, so we can update !analyze and point customers to your driver updates • Attend the OCA Crash Analysis session, following this session • Attend the WinDbg Ask the Experts sessions

  43. Resources • Debugger URL and download site • http://www.microsoft.com/whdc/ddk/debugging • Debugger e-mail – for debugger bug reports and feature requests • We try to fix all the bugs people report • We do not provide general debugging support on this alias • Debugger newsgroup • Microsoft.public.windbg • Good place for general debugging issues • Windbgfb @ microsoft.com

More Related