Kernel debugging QEMU

Debugging the FreeBSD kernel in QEMU is really straight forward. Get yourself some kernel symbols (either from the build dir or for /usr/lib/debug/kernel* ), load them into gdb, kick of QEMU with -s -S and connect.

$ qemu-system-x86_64 ... -s -S ...

The -S flag causes QEMU to wait for the gdbstub to start execution rather than letting CPUs free immediately. So we need to connect with gdb to get going.

$ gdb kernel.debug
(gdb) target remote localhost:1234
(gbd) c

Usage is documented a little in the manual (as much as anything is ever gdb documented).

I got the steps from a FreeBSD 10 guide and other than the symbols file changing in between everything is about the same. It is nice when things don't change.

gdb is nice and all, but can we use a debugger that ships with FreeBSD?

The lldb documentation suggests that we can do something like this:

(lldb) platform list
Available platforms:
host: Local FreeBSD user platform plug-in.
remote-freebsd: Remote FreeBSD user platform plug-in.
remote-gdb-server: A platform that uses the GDB remote protocol as the communication transport.
qemu-user: Platform for debugging binaries under user mode qemu
(lldb) platform select remote-gdb-server
  Platform: remote-gdb-server
 Connected: no
(lldb) platform connect connect://localhost:1234
  Platform: remote-gdb-server
  Hostname: (null)
 Connected: yes

It will say everything is connected, but when we try to continue we get an error.

 (lldb) c
error: invalid target, create a target using the 'target create' command

From a stackoverflow question I found the gdb-remote command which does what the documentation don't:

(lldb) gdb-remote localhost:1234
Process 1 stopped
* thread #1, stop reason = signal SIGTRAP
    frame #0: 0x000000000000fff0
->  0xfff0: addb   %al, (%rax)
    0xfff2: addb   %al, (%rax)
    0xfff4: addb   %al, (%rax)
    0xfff6: addb   %al, (%rax)
(lldb) c
Process 1 resuming
Process 1 stopped
* thread #4, stop reason = signal SIGINT
    frame #0: 0xffffffff805ee459
->  0xffffffff805ee459: incl   %eax
    0xffffffff805ee45b: cmpl   0x812097(%rip), %eax
    0xffffffff805ee461: jl     0xffffffff805ee450
    0xffffffff805ee463: nopw   %cs:(%rax,%rax)
(lldb) list udp_output
error: Could not find function named: "udp_output"

Things are confused.

(lldb) image list
[  0] 7A0DEA14 0x0000000000200000 /sbin/init
      /usr/lib/debug/sbin/init.debug

(lldb) f
error: Command requires a process which is currently stopped.
(lldb) c
error: Process is running.  Use 'process interrupt' to pause execution.

Out of the box like this I think lldb needs some more configuration. Frustratingly as always the documentation around debugging is seriously lacking.

gdb isn't perfect either:

End of the file was already reached, use "list ." to list the current location again
(gdb) list .

Fatal signal: Segmentation fault
----- Backtrace -----
0x1350771 ???
0x14717f6 ???
0x1471f7f ???
0x82fbf457f handle_signal
        /home/pkgbuild/worktrees/main/lib/libthr/thread/thr_sig.c:300
0x82fbf3b3a thr_sighandler
        /home/pkgbuild/worktrees/main/lib/libthr/thread/thr_sig.c:243
0x8230ef2d2 ???
0x173da31 ???
0x137ac37 ???
0x1384a35 ???
0x17cb202 ???
0x1470faa ???
0x1471335 ???
0x14709a2 ???
0x8262127c2 ???
0x1471d8d ???
0x14705fc ???
0x17ff86f ???
0x1cab092 ???
0x1caab64 ???
0x1576a59 ???
0x1573c40 ???
0x1250760 ???
0x83000d839 __libc_start1
        /home/pkgbuild/worktrees/main/lib/libc/csu/libc_start1.c:157
0x125064f ???
---------------------
A fatal error internal to GDB has been detected, further
debugging is not possible.  GDB will now terminate.

This is a bug, please report it.  For instructions, see:
<https://www.gnu.org/software/gdb/bugs/>.

zsh: segmentation fault (core dumped)  gdb

https://people.freebsd.org/~avg/kyivbsd/KyivBSD2010.pdf

a links post? https://www.unitedbsd.com/blog/775-remote-debugging-the-running-openbsd-kernel https://forums.freebsd.org/threads/setting-up-a-bhyve-vm-for-kernel-development-and-debugging.90935/