Linux kernel/modules debugging in virtual environment
Dealing with OSes one of most challenging tasks is the debugging of the entire system.
According to the system being debugged a developer may choose among the following methods:
- use a hardware JTAG (suited for embedded systems - development boards);
- use a debug stub (i.e. Linux kernel offers KGDB and a number of tracing capabilities);
- use a virtual machine.
In this article this third mechanism will be explained along with a number of useful techniques.
The running system is a linux-based distribution.
In the last few years many virtualization solutions providers have endowed their products with a GDB stub.
Among the others VMWare Workstation now offers such capability.
This feature is a per-virtual-machine basis therefore the configuration file must be changed accordingly.
If the hosted OS is 32bit open the configuration file and add this line:
If the hosted OS is 64bit open the configuration file and add this line:
VMware Workstation will listen for debug connection on port 8832 (8864 for 64bit).
Run gdb on the host, reference it to the kernel with symbols and attach to the virtual machine:
(gdb) file vmlinux-vvvv.debug
(gdb) target remote localhost:8832
Since gdb starts in 32-bit mode by default, you may also need to switch it to x64-64 before connecting:
(gdb) set architecture x86-64
(gdb) target remote localhost:8864
One tipical scenario is the following: the developer has created an ad-hoc distribution along with the libraries and necessary user space applications and wants to debug the overall system.
The developer may also have created a toolchain along with a particular version of glibc (or elibc).
In this situation when the program being debugged loads a shared library, gdb will look
for the symbols for that library on its own local file system, which generally means that it will find the shared libraries (and symbols) for the host operating system, rather than those of the virtual guest operating system. Unless the host and guest OS are identical, this can lead to confusion.
To face this situation it is necessary to tell gdb where to load libraries (typically the directory specified in the prefix) in this way:
(gdb) set solib-absolute-prefix /home/username/target/toolchain
Another possibility is to cross-compile gdb such that it knows the "prefix" of the target system and where to load the libraries from.
To debug modules the developer may follow one of these steps.
Load the module into the running kernel.
Go to cd /sys/module/<mymodule>/sections.
By issuing "ls -A1" the module's various ELF sections loaded into kernel space will be printed.
cat .text .data .bss (the section addresses I care about)
0xffffffffa00f4000 (address of module's text section ...)
0xffffffffa00f4880 (... and data ...)
0xffffffffa00f5200 (... and BSS)
and we can now tell gdb all about those sections thusly:
(gdb) add-symbol-file .../<mymodule>.ko 0xffffffffa00f4000 \
-s .data 0xffffffffa00f4880 \
-s .bss 0xffffffffa00f5200
add symbol table from file ".../<mymodule>.ko" at
.text_addr = 0xffffffffa00f4000
.data_addr = 0xffffffffa00f4880
.bss_addr = 0xffffffffa00f5200
(y or n) y
Reading symbols from .../<mymodule>.ko...done.
Note carefully how you add a module's symbols to your current gdb session with add-symbol-file: you must add the address
of the module's text segment as the argument to the command, after which you need only add the addresses of whatever extra ELF sections you care about.
If there was nothing in the BSS section you wanted to print, you could have omitted that option.