writing your own windows debugger – overview

Debuggers are the apple of the hacker’s eye. We benefit a lot from the debugger, but few of us know the principle of it.

In the book Gray Hat Python , the author has constructed a simple debugger. However, it is too simple, it is only a machine language level debugger, and can only set basic breakpoints and show CPU register information. We also want to know how to

• Show source code
• Set breakpoint based on lines, not memory address
• Set Step In, Step Out, Step Over
• Show stack trace
• Show global and local variables

The so-called user mode debugger is to debug the program in user mode. Windows has provided a series of open API for debugging, and they can be devided into three categories:

• API for starting the debuggee program
• API for handling debug event during debug loop
• API for inspecing and modifying debuggee program

The first thing to do before debugging a program is to start it. On Windows, we use CreateProcess to start to program:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

STARTUPINFO startupinfo = { 0 };
startupinfo.cb = sizeof(startupinfo);
PROCESS_INFORMATION processinfo = { 0 };
unsigned int creationflags = DEBUG_ONLY_THIS_PROCESS | CREATE_NEW_CONSOLE;

if (CreateProcess(
    "L:\git_up\anotherDebugger\anotherDebugger\Debug\test.exe",
    
    NULL,
    NULL,
    NULL,
    FALSE,
    creationflags,
    NULL,
    NULL,
    &startupinfo,
    &processinfo) == FALSE)
{
    std::cout << "CreateProcess failed: " << GetLastError() << std::endl;
    return;
}

• DEBUG_ONLY_THIS_PROCESS means the subprocess of the debuggee will not be debugged. If you need subprocess, use DEBUG_PROCESS.
• CREATE_NEW_CONSOLE means the debuggee’s and debugger’s output will be separated in two consoles.
• If the debugger process exits, the debuggee will also exit.

The debugger loop is a bit like Windows GUI message loop, some operations and exceptions will stop the debuggee and send event to the debugger. We always use

1
2

DEBUG_EVENT debugEvent;
WaitForDebugEvent(&debugEvent, INFINITE)

to capture the debug event.

There are 9 debug event in total:

• CREATE_PROCESS_DEBUG_EVENT. Reports a create-process debugging event.
• CREATE_THREAD_DEBUG_EVENT. Reports a create-thread debugging event.
• EXCEPTION_DEBUG_EVENT. Reports an exception debugging event.
• EXIT_PROCESS_DEBUG_EVENT. Reports an exit-process debugging event.
• EXIT_THREAD_DEBUG_EVENT. Reports an exit-thread debugging event.
• LOAD_DLL_DEBUG_EVENT. Reports a load-dynamic-link-library (DLL) debugging event.
• OUTPUT_DEBUG_STRING_EVENT. Reports an output-debugging-string debugging event.
• RIP_EVENT. Reports a RIP-debugging event (system debugging error).
• UNLOAD_DLL_DEBUG_EVENT. Reports an unload-DLL debugging event.

If the debug event has been handled correctly, then

1

ContinueDebugEvent(debuggeeprocessID, debuggeethreadID, DBG_CONTINUE);

to continue the debuggee process. Let’s combine the above to construct the debug loop:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

while (WaitForDebugEvent(&debugEvent, INFINITE) == TRUE)
{
    debuggeeprocessID = debugEvent.dwProcessId;
    debuggeethreadID = debugEvent.dwThreadId;
    if (dispatchDebugEvent(debugEvent) == TRUE)
    {
        ContinueDebugEvent(debuggeeprocessID, debuggeethreadID, FLAG.continueStatus);
    }
    else {
        break;
    }
}

bool (const DEBUG_EVENT & debugEvent)
{
    switch (debugEvent.dwDebugEventCode)
    {
    case CREATE_PROCESS_DEBUG_EVENT:
        // TBD
        break;

    case CREATE_THREAD_DEBUG_EVENT:
        // TBD
        break;

    case EXCEPTION_DEBUG_EVENT:
        // TBD
        break;

    case EXIT_PROCESS_DEBUG_EVENT:
        // TBD
        break;

    case EXIT_THREAD_DEBUG_EVENT:
        // TBD
        break;

    case LOAD_DLL_DEBUG_EVENT:
        // TBD
        break;

    case UNLOAD_DLL_DEBUG_EVENT:
        // TBD
        break;

    case OUTPUT_DEBUG_STRING_EVENT:
        // TBD
        break;

    case RIP_EVENT:
        // TBD
        break;

    default:
        cout << "Unknown debug event." << endl;
        return false;
        break;
    }
}

In the next part of the series, I intend to give a brief introduction about the 9 debug events.