FreeBSD kernel module debugging

I wanted to do kernel mode programming on FreeBSD and needed a setup to load a  kernel module and do kgdb remote debugging. I wanted the whole setup on my windows laptop. The following
details the steps needed to setup VMs connect them serially (virtual) for kgdb debugging and compile/debug a FreeBSD hello world kernel module.

Goal: To write a FreeBSD kernel module and be able to do remote source level debugging.
Setup needs two instances of Virtual Box running FreeBSD (hosted on windows),
one VM (Target VM) runs a test kernel module which needs to be debugged and the second VM (Dev VM) is used to run kgdb to remote debug the test kernel module loaded on target VM.
                       
    1) Download FreeBSD i386 iso
    2) Download Virtual Box
    3) Install standard FreeBSD with sources in a Virtual Box VM.
    4) Build a Debug FreeBSD kernel
            # cd /usr/src/sys/i386/ 
            # cp conf/GENERIC conf/MYDEBUGKERNEL
            Edit MYDEBUGKERNEL with more debug optins DEBUG -g (already set), OPTION KDB
           (already set),  OPTION DDB, OPTION GDB
            Compile the kernel
            # cd /usr/src/ 
            # make buildkernel KERNCONF=MYDEBUGKERNEL 
            # make installkernel KERNCONF=MYDEBUGKERNEL
    5) Clone the Virtual Box, So that we have two VMs Dev/Target debugging setup having the same  
        sources binaries and symbols.        
    6) setup serial communication  between VMs.
        Serial communication: through windows named pipe \\.\Pipe\<pipe name> and on
        Target VM, edit /boot/device.hints , "hint.uart.0.flags" to the value "0xC0" and reboot.
        Check VMs can communicate serially.
        Target VM: cat /dev/cuau0 (or some other file).
        Dev VM: echo "test string" >> /dev/cuau0 (or some other file) - you should see "test string"
        on Target VM. This is done so that kgdb can send commands from Dev VM to Target VM.
       
        one VM will create the named pipe and the other VM uses it, make sure when you start the
        VMs always start the VM which creates the  named pipe first
        
    7) setup Network communication between VMs.
        setup Internal Network, allow all with different mac address between VM. provide
        static ips to both the VMs.
        ifconfig em0 10.1.1.1 up on Dev machine
        ifconfig em0 10.1.1.2 up on Target machine.
        test it using ping 10.1.1.1 from 10.1.1.2 and arp -a.
       
    8) setup ftp or rsync server on Target machine and test it
        http://www.freebsd.org/doc/handbook/network-ftp.html
        I ftp my test FreeBSD .ko from Dev VM to target VM.
           
    9) Test remote debugging.
        Dev VM>
        cd /usr/obj/usr/src/sys/MYDEBUGKERNEL
        kgdb -r /dev/cuau0 ./kernel.debug
       
        Target VM>
        # sysctl debug.kdb.enter=1
        db> gdb 
        Step to enter the remote GDB backend. 
        db> c (continue)
           
        Try breaking (sysctl debug.kdb.enter=1) Target VM multiple times and continuing as above
        in target VM, till the Dev VM is able to grab the Target VM in kgdb.
       
        Dev VM: stack trace on DEV VM should confirm the remote debugging capability.
        kgdb> bt
        kgdb> c (continue)
               
        Now the whole setup is ready with Dev VM and Target VM with serial cable connected and
        also a network connected, also load the gdb macros in kgdb
        DEV VM:
        # cd /usr/obj/usr/src/sys/MYDEBUGKERNEL  (or name of kernel config)
        # make gdbinit (generates .gdbinit which is loaded when kgdb is started from the above
                                  directory).
       
        Reconnect Dev and Target with the gdb macros loaded in dev VM.
        DEV VM:
        kgdb> kldstat (gdb macro list all the modules loaded in Target VM, should be same as kldstat
                                 command when run in Target VM.)
       
        gdb macros is a nice way to figure out how the FreeBSD datastructures are laid out 
        in memory (checkout gdbinit.kernel, gdbinit.i386 in MYDEBUGKERNEL) along with
        cscope  and ctags for code browsing.
       
        Building and debugging "Hello world" kernel module: need two files Makefile and hello.c
        http://www.freesoftwaremagazine.com/articles/writing_a_kernel_module_for_freebsd
   
    Makefile:
        KMOD = hello
        SRCS = hello.c
   
        .include <bsd.kmod.mk>
   
    hello.c
   
    #include <sys/param.h>
    #include <sys/module.h>
    #include <sys/kernel.h>
    #include <sys/systm.h>
   
    void hello_debug_unload(void)  /* needed for testing remote debugging */
    {
        uprintf("Bye Bye hello_debug_unload\n");
    }
   
    /* The function called at load/unload. */
    static
    int event_handler(struct module *module, int event, void *arg) {
        int e = 0; /* Error, 0 for normal return status */
        switch (event) {
        case MOD_LOAD:
                uprintf("Hello Free Software Magazine Readers! \n");
                break;
        case MOD_UNLOAD:
                uprintf("Bye Bye FSM reader, be sure to check http://freesoftwaremagazine.com !\n");
                break;
        default:
                e = EOPNOTSUPP; /* Error, Operation Not Supported */
                break;
        }
      
        return(e);
   }
/* The second argument of DECLARE_MODULE. */
static moduledata_t hello_conf = {
    "hello",    /* module name */
     event_handler,  /* event handler */
     NULL            /* extra data */
};
DECLARE_MODULE(hello, hello_conf, SI_SUB_DRIVERS, SI_ORDER_MIDDLE);

Compiling kernel module:
    make DEBUG_FLAGS=-g3
    make install DEBUG_FLAGS=-g3  (copies hello.ko and  hello.ko.symbols to /boot/kernel)
   
loading kernel module   
    ftp hello.ko from Dev VM to Target VM, connect remote gdb.
    target VM > kldload ./hello.ko
    target VM> kldstat
   
    Figure out where the text section of hello_fsm kernel module starts in memory (module load
    address + module text section start).
    http://www.cz.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-kld.html
    objdump --section-headers hello.ko | grep text (text section start offset)
    kldstat would give the hello module load address.
    so kgdb on dev VM needs to be provided with the text section start of the loaded module(sum up
    both the addresses).
   
    On Dev VM break into Target VM and setup the new loaded kmod details.   
    Dev VM kgdb> add-symbol-file hello.ko 0xc0ae22d0
    Dev VM kgdb> b hello_debug_unload
    /* this gets hit when we do kunload on hello_fsm in target, check whether sources can be listed */
   
    target VM > kunload hello_fsm
    /* gets stuck in breakpoint */

    dev VM>  /* got control of  my code, happy digging */

 
Thanks to all the internet authors on FreeBSD kernel debugging.