world leader in high performance signal processing
Trace: » debugging_applications

Application Debugging

Sometimes the best way to debug an application is also the easy way. Here are some things to check before firing up gdb.

Stack Checking

Many application errors are the result of stack overflows. This is where data normally contained in the stack overflows into data in the bss section, or worse, into executable code space. To help debug this issue, the Blackfin compiler supports stack checking.

If the application's stack is overflowed, then an error will be generated and the application will be terminated.

FLAT Binaries

The default stack size for FLAT binaries is 4k (0x1000) bytes.

Checking

In order to properly check the stack at runtime, you must compile all the objects in question with a special compiler flag so that all the stack changes are instrumented with checks. This will add overhead to your application, but not a whole lot, so it shouldn't affect the normal execution.

The first (and preferred) method is to use the compiler flag -mstack-check-l1. This flag supports both non-threaded and threaded applications. The l1 part indicates that this flag utilizes a little bit of L1 scratch pad (but should not interfere with the normal running of things of course). Just add the flag to your compiling:

$ bfin-uclinux-gcc -mstack-check-l1 -c application.c
$ bfin-uclinux-gcc -elf2flt application.o -o application

The second method is to use the compiler flag -fstack-limit-symbol=_stack_start. This flag only supports non-threaded applications however. Again, just add the flag to your compiling:

$ bfin-uclinux-gcc -fstack-limit-symbol=_stack_start -c application.c
$ bfin-uclinux-gcc -elf2flt application.o -o application

Setting

Stack sizes can be increased on the build system using bfin-uclinux-flthdr or on the target with flthdr:

$ flthdr -s 8192 application
This sets the application stack size to 8k (8192 bytes).

You can also control the stack size while compiling your application:

$ bfin-uclinux-gcc -Wl,-elf2flt=”-s 8192” application.c
or
$ bfin-uclinux-gcc -Wl,-elf2flt=-s8192 application.c

You can also use flthdr to check the stack size:

$ bfin-uclinux-flthdr application
application:
    Magic:        bFLT
    Rev:          4
    Build Date:   Fri Mar 21 04:51:58 2008
    Entry:        0x44
    Data Start:   0x3540
    Data End:     0x3b10
    BSS End:      0x3df0
    Stack Size:   0x1000
    Reloc Start:  0x3b10
    Reloc Count:  0xba
    Flags:        0x1 ( Load-to-Ram )

FDPIC ELF Binaries

The default stack size for FDPIC ELF binaries is 128k (0x20000) bytes.

Checking

Just like FLAT binaries, you use the -mstack-check-l1 flag when compiling:

$ bfin-linux-uclibc-gcc -mstack-check-l1 application.c -o application

The -fstack-limit-symbol flag will not work for ELF binaries.

Setting

Stack sizes are generally controlled when linking the application by using the -Wl,--defsym,__stacksize=$SIZE (where you obviously replace $SIZE with the actual stack size you want). Setting the stack size on shared libraries does nothing as shared libraries are not the program entry point.

$ bfin-linux-uclibc-gcc -Wl,--defsym,__stacksize=$SIZE application.c -o application

You can view the current stack size by just using readelf and looking at the MemSiz part of the GNU_STACK program header.

$ bfin-linux-uclibc-readelf -l ./romfs/bin/busybox

Elf file type is EXEC (Executable file)
Entry point 0x7b18
There are 6 program headers, starting at offset 52

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  PHDR           0x000034 0x00000034 0x00000034 0x000c0 0x000c0 R E 0x4
  INTERP         0x0000f4 0x000000f4 0x000000f4 0x0000d 0x0000d R   0x1
      [Requesting program interpreter: /lib/ld-uClibc.so.0]
  LOAD           0x000000 0x00000000 0x00000000 0x47ac4 0x47ac4 R E 0x1000
  LOAD           0x047ac4 0x0004bac4 0x0004bac4 0x04dcc 0x0b5cc RW  0x1000
  DYNAMIC        0x0498f4 0x0004d8f4 0x0004d8f4 0x000b8 0x000b8 RW  0x4
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x20000 RWE 0x8    ←- this line!

 Section to Segment mapping:
  Segment Sections...
   00
   01     .interp
   02     .interp .hash .dynsym .dynstr .rel.got .rel.plt .init .plt .text .fini .rodata .rofixup
   03     .data .eh_frame .dynamic .ctors .dtors .jcr .got .bss
   04     .dynamic
   05     .interp .hash .dynsym .dynstr .rel.got .rel.plt .init .plt

Additionally, there is the fdpichdr utility (which can be found in the toolchain svn tree) which can change the field on the fly:

$ bfin-linux-uclibc-readelf -l test | grep GNU_STACK
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x20000 RWE 0x8
$ fdpichdr --stack-size 0x40000 test
$ bfin-linux-uclibc-readelf -l test | grep GNU_STACK
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x40000 RWE 0x8

Multi-threaded applications

The POSIX standard does not dictate the size of a thread's stack. This is implementation dependent and varies. The default size on Blackfin/uClibc is 16k. Exceeding the default stack limit is often very easy to do, with the usual results: program termination and/or corrupted data.

Stack checking with -mstack-check-l1 will catch application threads which exceed their thread stack, however, to set the size of the stack, it must be done in your code with the standard pthread stack setting functions:

If you want to put the stack into a specific location in memory, that can be done with the calls to:

The stack overflow message may not identify if it is a thread, or the main application which is causing the stack overflow. It is up to the programmer to determine where the problem is occurring.

Mudflap

mudflap is a new compiler option in GCC 4.x that detects memory-handling problems. It does this by looking for unsafe source-level pointer operations. Constructs are replaced with expressions that normally evaluate to the same value, but include parts that refer to libmudflap, the mudflap runtime. To evaluate memory accesses, the runtime maintains a database of valid memory objects.

Consider this code:

int a[10];
int b[10];

int main(void) {
   return a[11];
}

Tools like valgrind detects no error; the compiler allocates memory for a and b consecutively, and an access of a[11] will likely read b[1].

However, if you use the blackfin gcc compiler to compile the code:

$ bfin-linux-uclibc-gcc -o bug bug.c -g -fmudflap -lmudflap

Use -fmudflapth instead of -fmudflap to compile and -fmudflapth -lmudflapth to link if your program is multi-threaded.

When executed, the program fails and mudflap prints a warning.

$ ./mem_test
*******
mudflap violation 1 (check/read): time=1196719575.180687 ptr=0x80ca060 size=48
pc=0xb7e9626d location=`mem_test.c:6 (overflow_by_1)'
      /usr/lib/libmudflap.so.0(__mf_check+0x3d) [0xb7e9626d]
      ./mem_test(overflow_by_1+0x75) [0x80487b9]
      ./mem_test(main+0x16) [0x80487d6]
Nearby object 1: checked region begins 0B into and ends 8B after
mudflap object 0x80cb250: name=`mem_test.c:1 a'
bounds=[0x80ca060,0x80ca087] size=40 area=static check=3r/0w liveness=3
alloc time=1196719575.180434 pc=0xb7e95ccd
number of nearby objects: 1

mudflap can also print a list of memory leaks through its -print-leaks option. However, mudflap does not detect reads of uninitialized memory.

Mudflap instrumentation and runtime overhead do cost significant extra time and memory. At build time, the compiler needs to process the instrumentation code. When running, it takes time to perform the checks, and memory to represent the object database. The behavior of the application has a strong impact on the run-time slowdown, affecting the lookup cache hit rate, the overall number of checks, and the number of objects tracked in the database, and their rates of change.

For more information see the gcc Docs

Magic System Request Key

The magic system request is a 'magical' key combo you can hit which the kernel will respond to regardless of whatever else it is doing, unless it is completely locked up. To enable the magic SysRq key, you need to say “yes” to 'Magic SysRq key (CONFIG_MAGIC_SYSRQ)' when configuring the kernel.

When running a kernel with SysRq compiled in, /proc/sys/kernel/sysrq controls the functions allowed to be invoked via the System Request key. By default the file contains 1 which means that every possible SysRq request is allowed. Check the kernel documentation for more information.

To use the magic System request key, you send a BREAK, then within 5 seconds a command key. Sending BREAK twice is interpreted as a normal BREAK. To send a BREAK:

  • in minicom, do a Control-A then f;
  • in Kermit, type the escape character (normally Ctrl-\ ) then B

Common commands are:

  • b - Will immediately reboot the system without syncing or unmounting your disks.
  • o - Will shut your system off (if configured and supported).
  • s - Will attempt to sync all mounted filesystems.
  • p - Will dump the current registers and flags to your console.
  • t - Will dump a list of current tasks and their information to your console.
  • m - Will dump current memory info to your console.
  • e - Send a SIGTERM to all processes, except for init.
  • i - Send a SIGKILL to all processes, except for init.
  • h - Will display help

Analyzing Traces

See the Analyzing Traces page.

Using GDB

See the debuggers page.