Linux Post Mortem Analysis

No one likes being in this position - the Linux kernel crashes or hangs without any output on the console. Which is why you should check out the early printk page. This will enable you to see normal crash messages even before the kernel/serial device is set up all the way… Even if you are booting on the wrong machine type (booting a 537 image on a 548), you will get something printed on the next reset of the bootloader.

If you are unable to enable early_printk, there are still things to do. The first thing to attempt to get more information in such a situation is a dump of the log buffer - often the Linux kernel has already collected useful information in its console I/O buffer which just does not get printed because the kernel does not run until successful initialization of the console port.

Follow the below steps:

1. On your host, find out the virtual address of the log buffer:

   rgetz@test:~/checkouts/kernel/uClinux-dist> grep __log_buf ./linux-2.6.x/System.map
   00144d80 b __log_buf

2. Reset your board - do not power-cycle it! If your board does not have a reset button, you can still have things reset on a kernel crash by:

   bfin> set bootargs $(bootargs) panic=3
   bfin> set bootcmd

   The addition of panic=3 tells the kernel to reboot on panic, and clearing bootcmd will ensure that a new kernel does not boot on a reset.

3. Use U-Boot to print a memory dump of that memory area with the strings command (if you don't have strings command, just use memory dump md.b):

   bfin> strings 0x00144d80
   <5>Linux version2.6.16.11-ADI-2006R1 (rgetz@test) (gcc version3.4.5 (ADI cvs)) #1 Mon Aug 7 10:55:44 EDT 2006
   <6>Blackfin support (C) 2004-2006 Analog Devices, Inc
   <6>Compiled for ADSP-BF537 Rev. 0.2
   <6>Blackfin uClinux support by http://blackfin.uclinux.org/
   <6>Processor Speed: 500 MHz core clock and 100 Mhz System Clock
   .....

4. This buffer is a circular buffer, where the default size of this buffer is 16k, (this can be controlled by changing the configuration option by Kernel hacking → Kernel debugging → Kernel log buffer size). If the kernel has printed more information that 16k, and the buffer has wrapped, the start of the buffer can be found by looking at the pointer _log_start.

   rgetz@test:~/checkouts/kernel/uClinux-dist> grep _log_start ./linux-2.6.x/System.map
   00144cf0 b _log_start

   This tells us the start/end of the circular buffer is pointed to by the value in 0x144cf0.

   bfin> md.l 144cf0 0x1
   00144cf0: 00000b92    ....

   This tells us that 0x0b92 is the offset of the buffer.

5. Have a peek at the end of the buffer 0x0b92 + 0x144d80 to see the last message that prints out.

   bfin> md.w 0145912 0x100
   00145912: 303c 523e 7365 6174 7472 6e69 2067 7973    <0>Restarting sy
   00145922: 7473 6d65 0a2e 343c 2e3e 000a 0000 0000    stem..<4>.......
   00145932: 0000 0000 0000 0000 0000 0000 0000 0000    ................