Operating System support for debuggers
Debuggers are user mode applications which means that they cannot execute privileged code. Interactions between debugger and the debugged application goes through the OS.
Different operating systems have different support for debuggers.
- Linux has ptrace() and wait(). pstrace can be called with different requests such as step, R/W of text or data blocks. The call is nonblocking as many requests can be fulfilled immediately by the OS (as it already has the information) without having to look into the application being debugged. For requests that take time to complete, wait() can be issued to wait for the next event notification
- Linux also has the /proc device driver interface that exposes running processes like files and directories. Standard APIs like open(), close(), read(), write(), ioctl() can be used on these to achieve debugging purposes. This allow debuggers to attach to processes already running instead of opening a new instance as is needed with ptrace()
- ioctl() can be used to register events of interest.
- While breakpoint is not a event for ioctl, we can register a notification to wait for a process to stop, and then modify the text section of the code to induce a stop (illegal op)
- The process will stop, and the debugger will be notified because it asked for notification
- Windows has the Win32 debug API
- Java has its own platform debugging architecture