GDB Debugging

Launching

• $ gdb [executable | executable process_id | executable core_file]
• $ gdb -tui usage with TUI instead of CLI

Is is not necessary use the prefix ./ to load an executable. The corresponding source files are loaded automatically according to information embedded inside the executable, manual loading is possible with the directory command.

Common information

Commands may always be truncated if the abbreviation is unambiguous.

Environment variables can be modified within GDB, which is useful to control behavior of programs.

• set environment_variable variable_name = value set environment variable

The handling of signals like SIGINT - which is often invoked by Ctrl+c - and other signals can be modified, shown and even send from within GDB.

• info signals or info handle print table of signals and how GDB handle each one
• handle signal keyword modify behavior of signal handling
• signal signal_name send a signal to program, either name or number of the signal

Forking and Threading

Process Forking

• GDB has no special support for debugging programs which create child processes using fork().
• GDB will continue debugging the parent process, if a breakpoint is set in code which the child executes, the child will get a SIGTRAP signal which will cause it to terminate - unless it catches the signal.
• As a workaround sleep() can be used in the child process if a self-defined environment variable is set. Use subsequently $ ps to get the process_id of the child and use a new instance of GDB to attach to it.

• However - on recent Linux systems it is possible to debug either the child or parent process after a fork() or both - while one process is held suspended.

  • set follow-fork-mode [parent | child] default is parent
  • show follow-fork mode
  • set detach-on-fork [on | off] default is on, with off both process remain under control of GDB, supported on Linux
  • show detach-on-fork
  • info inferiors
  • inferior inferior_number make inferior_number the current inferior

https://sourceware.org/gdb/current/onlinedocs/gdb/Forks.html

Multi Threading

There are two modes available the all-stop mode - which is the default - and the non-stop mode - when the target supports it. The mode preference only applies when GDB starts or connects to a program. In all-stop mode all threads stop execution when GDB halts the program and all threads continue execution when GDB continues execution - also when stepping. In non-stop mode all threads continue execution while only one or multiple threads are halted by GDB.

• set pagination [on | off] using CLI pagination breaks non-stop mode, set it to off before
• set non-stop [on | off] turn non-stop mode on or off
• show non-stop
• info threads displays a summary of all threads currently existing in the program.
• thread thread_number make thread_number the current thread
• show print thread-events show messages when a thread has started or exited
• set print thread-events [on | off] - turn messages about threads on or off, default is on

In multi-threaded programs breakpoints or some other events in one thread may cause blocking system calls in other threads to return prematurely. Therefore it is important to check and handle the return values of system calls appropriately - anyway a good programming style e.g. use the return of sleep() within a while loop.

https://sourceware.org/gdb/current/onlinedocs/gdb/Thread-Stops.html

Running

• run accepts same options like the executable and passes the command to the shell. pipes, redirections and wildcards are usable.
• quit exit debugger, ask for confirmation if a session is active
• attach process_id attach to running process
• detach detach from running process
• target switch between in debug targtes e.g. process inside GDB, external process and core dump
• file executable load or reload possibly modified executable
• directory load or reload sources from compilation directory or current directory

Remote debugging

Their are two possible modes remote and remote-extended where the later offers more comfortable features but is sometimes not available. The remote side is refered as target and the debugging side usually as host. While the executable on the target can be stripped, the executable on the host must be fully debuggable i.e. symbol tables and libraries are available. If necessary use:

• file specify location of executable on the host, if needed
• set sysroot specify location on the host of target libraries unless GDB uses correct sysroot anyway

How-to

1. on target
   • $ gdbserver hostname_of_host:port_number executable start program
   • $ gdbserver hostname_of_host:port_number --attach process_id attach to already running program
2. $ gdb on host
3. target [remote | extended-remote] hostname_of_target:port_number within gdb on host

It is also possible to use STDIO pipes:

• target remote | ssh -T hostname gdbserver - --attach process_id uses SSH without pseudo terminal and - tells gdbserver to use stdio pipes

https://sourceware.org/gdb/current/onlinedocs/gdb/Remote-Debugging.html

Linespec Location

Several commands accept a location as line_spec:

• line_number
• [-offset | +offset]
• function
• filename:line_number
• filename:function
• label within function

Symbols specific to C++ programming language - like namespaces, classes and scope operators - are usable e.g. [namespace::class::member_function]. When an expression is ambiguous a menu can be displayed, by default this feature is off and all possible matches are used. This behavior can be modified with set multiple-symbols mode.

Navigation

• list [line_spec | first_line last_line] print source lines, by default continues with the next ten lines
• backtrace shows the stack frames, summary of how the program got where it is. the current executing frame is frame zero, followed by its callers counting upward to the main function.
• frame nr_of_frame shows current selected frame and re-centers position for list
  • up one frame up
  • down one frame down

Stepping and Continuing

• next executes next source line
• step executes next source line and enters functions when debugging symbols are available
• continue [-a] resumes execution until program is halted by a breakpoint or signal. when multipe threads are stopped independently, the option -acontinues execution of all threads.
• until executes until target is reached
• finish executes up to end of current function
• interrupt [-a] stop the whole program i.e. all threads in non-stop mode
• jump line_spec continue execution at line_spec. this doesn't modify the stack frame, stack pointer, contents of any memory location or registers, therefore the results will be bizarre (depending on machine language) when jump is used to change the current executing function. jumps within the same function should be save.

Breakpoints and Watchpoints

• break [line_spec] [thread thread_id] [if condition] set a breakpoint. when several overloaded functions are possible, GDB will provide a menu.
  • if condition set a breakpoint with a condition, evaluate condition each time the breakpoint is reached
  • thread thread_id set a thread specific breakpoint which applies until the specfified thread exits
• watch expression set a watchpoint
• info [break | breakpoints | watchpoints] print a table of all breakpoints and watchpoints

Print, Modify and Evaluate

• print [variable | function() ] show, evaluate and manipulate data, this includes the manual execution of functions. dereference * and member access -> are usable.
• print variable = value manipulates data through assignment
  • Overloaded operators of class types are not directly usable, use explicit calls to operator functions e.g. print str.operator=("value")
• set variable = value maniuplates data through assignment but doesn't print the result. if the variable name is identical to a subcommand of set use the variable subcommand i.e. set variable variable_name = value instead.
• ptype show precise type definition
• x [count][type][size] memory_address print memory at given address, optionally with given count, type and size
  • count number of items
  • type o(octal), x(hex), d(decimal), u(unsigned decimal), t(binary), f(float), a(address), i(instruction), c(char), s(string) and z(hex, zero padded on the left)
  • size b(byte), h(halfword), w(word) and g(giant, 8 bytes)

General Informations

• info
• info program
• info args print the args of the selected frame or program
• info locals local variables
• info variables names and types of all variables execluding local variables
• info address variable exact location
• info frame info about current stack frame

Core dumps

• Memory dump of program including variables and fixed data
• Stored in working directory
• Usually turned off by default, turn on with ulimit -c unlimited within file .bashrc
• Program shall be compiled with -g

Patching executables

An executable file must be loaded writable and an address within the executable modified with set - afterwards GDB must be quit immediately.

How-to

1. set write on allows modification of binary files, likely causes messages like text file bussy
2. file executable
3. disas/r function show disassembly
4. set {type} addr = value where type is a variable type, addr is address within the executable - retrieved from the dissassembly and not the from the current program memory like a variable or pointer address.
5. quit leave gdb, file is written

A binary patch can be created with diff [-a | --text] and applied by patch but diff is not designed for binary patches. The resulting patch likely becomes bigger then the binaries without actual source lines to compare. Therefore dedicated binary patch tools are commended and perform appropriately. Git has also support for creating and applying binary patches. Binary patching also can be done manually with UNIX tools like dd and strings:
https://unix.stackexchange.com/a/214824

Reverse Debugging

Reverse debugging stores a record within a buffer, the record either stops when it reaches the limit of the buffer or overwrites the exiting record when working as circular buffer. * set record stop-at-limit [on | off] when off record log is used as circular/ring-buffer * show record stop-at-limit

• record starts record
• record-stop stops record and replay target, maybe program will terminated
• reverse-next reexecutes last source line
• reverse-continue reexecute until halted
• record-step reexecutes last source line, enters functions
• reverse-finish rexecutes up to the begin of the function

https://www.sourceware.org/gdb/wiki/ProcessRecord/Tutorial

How-to

1. breakpoint [line_spec] program must be running, set break at appropriate location e.g. main() or even _start
2. run
3. record
   • following commands can be used interchangeably now:
   • reverse-[next | step | finish | ...] - reverse execute
   • [next | step | finish | ...] - forward execute
   • and so on
   • when all breakpoints are disabled a continue will return the program to the end of the current record and halt
4. record stop

Workarounds

In case of issues with xsave or similiar set the environment variable LD_BIND_NOW to 1 before running the program.

• set environment_variable LD_BIND_NOW = 1

Quote from the man page:

If set to a nonempty string, causes the dynamic linker to resolve all symbols at program startup instead of deferring function call resolution to the point when they are first referenced. This is useful when using a debugger.

Resources and Links

https://sourceware.org/gdb/