world leader in high performance signal processing
Trace: » adeos

Xenomai/Adeos porting for Blackfin

Real-time on Linux

There are usually two approaches to make Linux real-time.

  • Using a sencond kernel to schedule real-time tasks: solutions include Xenomai/ADEOS and RTLinux, RTAI, etc.
  • Improve Linux kernel itself with regards to preemption, low latency, etc. See the real-time preemption patch: http://people.redhat.com/~mingo/realtime-preempt/.

Adeos and Xenomai

Adeos is an event pipe line. The purpose of Adeos is to provide a flexible environment for sharing hardware resources among multiple operating systems, or among multiple instances of a single OS. It has been ported to blackfin-uclinux since 2005R4 release. ADEOS is also known as “I-pipe”, it delievers system events (interrupts, execptions, system calls) in a timely and prioritized manner, along a “pipe line” of domains.

Xenomai is a real-time development framework cooperating with the Linux kernel. Xenomai implements a micro-kernel (nucleus), with real-time scheduler. Xenomai's real-time nucleus and Linux kernel are in two ADEOS domains. Xenomai runs in a higher priority domain than Linux kernel. Xenomai also implements different APIs providing real-time services, like creating real-time tasks, timers, semaphores.

ADEOS Makes Blackfin uClinux Real-time

Currently the Linux “real-time preemption patch” does not work on uClinux. So when you want to implement hard real-time application, Adoes may be the choice, especially:

  • When your real-time application makes use of existing Linux services, like network stack, existing Linux device drivers, Adeos can be used. Both Adeos and Xenomai work as part of the Linux kernel. The real-time task can call Linux service as long as it does not break the Linux kernel. But please take care when calling Linux services:
    • Critical region in Linux can be preempted by RT task if HW irq is not disabled
    • RT task cannot sleep in Linux Domain
  • When your real-time application does not want to be disturbed by Linux, you can run the application in a real-time Xenomai task, or even protect the critical job by turning off the HW interrupts. Adeos provides the nice feature of “virtual irq”, making it possible for Linux kernel to work normally after being preempted for a long period of time.
  • Xenomai simulates several APIs for existing RTOS systems, including rtai, vrtx, vxworks and POSIX real-time interface, which makes it possible to port and run existing RT application to Linux.

This chapter will guide you through the steps to run Xenomai on Blackfin.

Enable Adeos and Xenomai on Blackfin

Since 2012R2 release, xenomai was added to buildroot, You just need to enable the xenomai, as following.

 -*- Xenomai Userspace
()    Custom Xenomai version
[ ]   Enable SMP support
[*]   Install testsuite
[ ]   RTCan utilities
[ ]   Analogy libs and utils
[*]   Native skin library
[*]   POSIX skin library
[ ]   VX-Works skin library
[ ]   PSOS skin library
[ ]   RTAI skin library
[ ]   uiTron skin library
[ ]   VRTX skin library  

And manually specify the path of adeos patch for 2012R2 release, board/AnalogDevices/blackfin/patches/adeos/ipipe-core-3.5.7-blackfin-3.patch, in make menuconfig → kernel → Linux Kernel Extentions, as following.

[*] Adeos/Xenomai Real-time patch
(board/AnalogDevices/blackfin/patches/adeos/ipipe-core-3.5.7-blackfin-3.patch) Path for Adeos patch file
[ ] Open Cryptographic Framework (OCF)
[ ] RTAI Real-time patch

For releases between 2008R1-RC8 and 2010R1, xenomai was added to uclinux-dist. You just need to selected and build kernel as bellow. The makefile system will patch the kernel.

 Miscellaneous Applications  --->
      [ ] C++ Test Applications
   --- realtime
      [*] xenomai

By default the “latency” test will be installed to rootfs.

If you are using older release, please follow steps bellow. Otherwise, please skip to next section.

Prepare kernel source, toolchain and patches

  1. Download Xenomai. We use the most recent release v2.4.0.
  2. Download blackfin-uclinux source. We use the 2008R1 release.
  3. Download blackfin-uclinux toolchain. We use the 2008R1 release. Add the install path of toolchain to the PATH environment variable.
  4. Which adeos patch to use?

We need to patch the Linux kernel with adeos blackfin patch. There are two adeos patches available:

  • Adeos patch from Xenomai: located in [xenomai-src]/ksrc/arch/blackfin/patch/. This patch is created against the vanilla kernel tree (linux-2.6.23 at the time of writing) by Xenomai/Adeos maintainer.
  • Adeos patch in blackfin uclinux: located in uclinux-dist/bfin_patch/adeos_patch. Since formal bfin-uclinux release may not be updated to use latest stable vanilla linux kernel (e.g, blackfin-uclinux 2008R1 version is still based on linux-2.6.22). So we keep an adeos kernel patch for the blackfin uclinux release.

Here we uses the adeos patch in bfin-uclinux.

Apply Adeos and Xenomai kernel patch to the blackfin-uclinux kernel

In the following example, $xenomai-root is the path of the xenomai.

$adeos-patch-dir is the path of the Adeos patch.

$uClinux-dist is the path of of blackfin-uclinux release, for example: /home/test/uClinux-dist

$linux-tree is the path of the blackfin-uclinux kernel source, for example: /home/test/uClinux-dist/linux-2.6.x

  • Run the Xenomai configuration script to apply the kernel patch:
    adam@linux> $xenomai-root/scripts/prepare-kernel.sh --arch=blackfin \
                --adeos=$adeos-patch-dir/adeos-ipipe-2.6.<...>.patch \
                --linux=$linux-tree
  • Configure and build kernel
    adam@linux> cd $uClinux-dist
    adam@linux> make menuconfig

Select BF537-STAMP and save.

    Vendor/Product Selection  ---> 
      [*](BF537-STAMP) AnalogDevices Products
  • After exiting from “make menuconfig”, select the default configuration for Xenomai (just return at each prompt).
    *
    * Real-time sub-system
    *
    Xenomai (XENOMAI) [Y/n] (NEW) Y
      Nucleus (XENO_OPT_NUCLEUS) [Y/n] (NEW) Y
    ... ...
  adam@linux> cd $linux-tree
  adam@linux> make menuconfig

Keep the default configuration as:

  [*] Interrupt pipeline (NEW)
  [*] Xenomai                                                                                                                                     
  <*>   Nucleus                                                                                                                                   
  [*]     Pervasive real-time support in user-space                                                                                               
  [ ]       Interrupt shield support                                                                                                              
  [ ]       Disable priority coupling                                                                                                             
  [*]       Enforce syscall access control                                                                                                        
  [*]     Optimize as pipeline head                                                                                                               
  (32)    Number of pipe devices                                                                                                                  
  (512)   Number of registry slots                                                                                                                
  (128)   Size of the system heap (Kb)                                                                                                            
  [*]     Statistics collection                                                                                                                   
  [ ]     Debug support                                                                                                                           
  [ ]     Watchdog support
           Timing  --->                                                                                  
           Scalability  --->          
           Shared interrupts  --->     
           Machine  --->                                                                                                                            
           Interfaces  --->                                                                                                                         
           Drivers  --->      

Build image and run test cases

  • Build uclinux kernel and applications:
    adam@linux> cd $uClinux-dist
    adam@linux> make
  • Build Xenomai applications and install the applications to uclinux romfs directory. This will build xenomai in FDPIC format.

$build-root is a directory to build xenomai application.

    adam@linux> mkdir $build-root 
    adam@linux> cd $build-root
    adam@linux> $xenomai-root/configure --host=bfin-linux-uclibc
    adam@linux> make install DESTDIR=$uClinux-dist/romfs
    adam@linux> cd $uClinux-dist
    adam@linux> make image

It is possible there would be error ”.couldn't allocate a block (no free space)”. Need to edit $uClinux-dist/vendors/AnalogDevices/BF537-STAMP/Makefile, changing “BLOCKS” from 4096 to 8192 then make again. Or you can remove ”/usr/xemomai/share and /usr/xemomai/include from romfs”

image is created as $uClinux-dist/images/linux

  • Download the image to BF537 STAMP and boot
  • Run the sample test case on BF537 STAMP:
    root:/> cd /usr/bin
    root:/usr/bin> ls                                                               
    clocktest      irqloop        loudly         switchtest     xeno-test           
    cyclictest     latency        switchbench    workload       xeno-test_new      
    root:/usr/bin> ./latency -h

Details about the latency test case

This “latency” example program features a periodic real-time thread measuring its scheduling latency over periods of 100 us. Each second, results are posted by the sampling thread to a display thread, which in turn outputs the latency figures to the screen using standard Linux services. This illustrates the seamless migration of Xenomai's real-time threads between the Xenomai and Linux schedulers, in order to perform every system call issued by xeno-enabled Linux tasks from the proper execution context.

  • The source code of this “latency” test case is in: src/testsuite/latency/latency.c
  • The source code of “timer benchmark driver” is in: ksrc/drivers/benchmark/timerbench.c
  • The Scheduling Latency option affects the latency result. When programing a timer, The Scheduling Latency value will be substracted from the timer expire period forcing the timer to wake up early in an attempt to elimiate the fixed latency value:
    • XENO_OPT_TIMING_SCHEDLAT: Here we set it as 1 ns, that is, ignore it.
              "Scheduling latency (ns)"
              default 0
              Scheduling latency is the time between the termination of an
              interrupt handler and the execution of the first instruction
              of the real-time thread this handler resumes. A default value
              of 0 (recommended) will cause a pre-calibrated value to be
              used.

Three test modes

This test has three test modes, depending on the ”-t” option.

  • -t0 Run the User Space task
  • -t1 Run the Kernel Space test
  • -t2 Run the Timer Interrupt test
User space task

Real-time thread is created in user space, using the rt_task_create() function.

latency.c

     rt_task_create(&latency_task,"sampling",0,99,T_FPU); // Create a real-time task named "sampling", with priority 99
                                                          // and default stack size 
     rt_task_start(&latency_task,&latency,NULL); // start the real-time task, calling the "latency()" routine 

Xenomai provides several “skins” for real-time applcation, including native, posix, psos+, rtai, rtdm, vxworks, uition, vrtx. The API above is “native” skin, which is specific to Xenomai. You can find definition of the native API in the online document or document generated while builing Xenomai.

The “sampling” thread calls

int rt_task_set_periodic  (RT_TASK * task, RTIME idate, RTIME period)

to “Make a task periodic by programing its first release point and its period in the processor time line”. In this case, by default, the timer expires every 100us.

Then “sampling” thread repeatly calls:

int rt_task_wait_period(unsigned long * overruns_r) 

“Make the current task wait for the next periodic release point in the processor time line.” The latency is measured as the difference between expected “release point” and the actual time when the real-time thread returns from rt_wait_period(). The latency involves:

  1. Handling timer interrupt
  2. Current thread is scheduled to run

Run in mode 0 for 60 sec:

root:/usr/bin> ./latency -t0 -T60 -h 
== Sampling period: 100 us
== Test mode: periodic user-mode task
== All results in microseconds
warming up...            
RTT|  00:00:01  (periodic user-mode task, 100 us period, priority 99)
RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst
RTD|      2.922|      4.298|     17.914|       0|     0|      2.922|     17.914
RTD|      3.746|      4.324|     18.338|       0|     0|      2.922|     18.338
...
---|--param|----range-|--samples
HSD|    min|   2 -  3 |        1
HSD|    min|   3 -  4 |       58
---|--param|----range-|--samples
HSD|    avg|   2 -  3 |        1
HSD|    avg|   3 -  4 |    26537
HSD|    avg|   4 -  5 |   541602
HSD|    avg|   5 -  6 |    14843
HSD|    avg|   6 -  7 |       76
HSD|    avg|   7 -  8 |      175
HSD|    avg|   8 -  9 |    11097
HSD|    avg|   9 - 10 |      987
HSD|    avg|  10 - 11 |     2247
HSD|    avg|  11 - 12 |     1259
HSD|    avg|  12 - 13 |      317
HSD|    avg|  13 - 14 |      167
HSD|    avg|  14 - 15 |      290
HSD|    avg|  15 - 16 |      172
HSD|    avg|  16 - 17 |      123
HSD|    avg|  17 - 18 |       32
HSD|    avg|  18 - 19 |       37
HSD|    avg|  19 - 20 |        6
HSD|    avg|  20 - 21 |        3
---|--param|----range-|--samples
HSD|    max|  16 - 17 |       19
HSD|    max|  17 - 18 |        5
HSD|    max|  18 - 19 |       27
HSD|    max|  19 - 20 |        6
HSD|    max|  20 - 21 |        2
HSH|--param|--samples-|--average--|---stddev--
HSS|    min|        59|      2.983|      0.130
HSS|    avg|    599971|      4.120|      0.909
HSS|    max|        59|     17.441|      1.149
---|-----------|-----------|-----------|--------|------|-------------------------
RTS|      2.922|      4.312|     20.258|       0|     0|    00:01:00/00:01:00

It is interesting that this real-time task in user space switches between “primary” and “secondary” domains when running. When a RT task is scheduled by the Xenomai scheduler, it is said to be in “primary” domain, when it is scheduled by Linux kernel, it is in “secondary” domain. When the RT task calls Linux system calls, it may be switched to secondary domain, and vice versa. This feature makes a RT task in user space execute in a similar way to a normal Linux task. It can call Linux services like “printf” and switch back to Xenomai domain seamlessly.

Kernel space task

To enable this test mode, please first of all build timerbench driver into kernel:

Xenomai --->
  drivers ---> 
    <*> Timer benchmark driver 

This mode differs from the “user space task” mode in that, the real-time thread is created and running in kernel space. The thread is created in a kernel driver: timer benchmark driver:

timerbench.c:

int rtdm_task_init(rtdm_task_t *task, const char *name,
                   rtdm_task_proc_t task_proc, void *arg,
                   int priority, uint64_t period)

This is another set of API called RTDM (Real Time Driver Model). RTDM is designed to be a portable interface for real-time device drivers in different real-time systems. It is suggested that you write real-time device drivers using this interface instead of native Adeos/Xenomai interface. There is a sample RS232 real-time serial driver in Xenomai source using RTDM. The RTDM wraps native Adeos/Xenomai services, like interrupt handling, RT task creation.

The real-time thread repeatly calls:

rtdm_task_sleep_until(uint64_t wakeup_time)

“Sleep until a specified absolute time”. The latency measured is the the difference between the expected timer expire time and the actual time when the thread returns from “rtdm_sleep_until”. The latency includes:

  1. Handling timer interrupt
  2. Current thread is schedule to run

Run in mode 1 for 60 sec:

root:/usr/bin> 
root:/usr/bin> ./latency -t1 -T60 -h 

The kernel space RT task can use Linux kernel services, like kmalloc(). But please note that since the RT task can preempt the Linux kernel at any moment when it is scheduled, even when Linux kernel is holding a lock, the kernel shared resource locking may be broken.

Timer interrupt

To enable this test mode, please first of all build the timerbench driver into kernel:

Xenomai --->
  drivers ---> 
    <*> Timer benchmark driver 

In this test mode, the test case measures the timer interrupt latency. timerbench.c:

xntimer_init(&ctx->timer, timer_proc, ctx); // initialize the timer, set up the interrup handler: timer_proc
      xntimer_start(...); // start the timer

In the timer interrupt handle, the latency is measured as the difference between the expected timer expire time and the time when the interrupt handler is actually called.

Run in mode 2 for 60 sec:

root:/usr/bin> ./latency -t2 -T60 -h

Other Test Cases

switchbench

This case measures context switch latency of RT task.

Two RT task are created. One task waits on a semaphore by calling “rt_sem_p()”, another task wakes it up using “rt_sem_broadcast()”. The switch latency is measured as the interval between entering “rt_sem_broadcast()” and returning from “rt_sem_p()”.

Here is sample output on a BF537-STAMP board. Time is measured in “microsecond (10^-6 Second)”. The average context switch latency is about “15 us”.

usage: switch [options]                                                         
        -h             - enable histogram                                       
        -p <period_us> - timer period                                           
        -n <samples>   - number of samples to collect                           
        -i <samples>   - number of _first_ samples to ignore                    

root:/> switchbench -n 600000 -i 10000 -h                                       
== Sampling period: 100 us                                                      
== Do not interrupt this program                                                
RTH|     lat min|     lat avg|     lat max|        lost                         
RTD|      14.014|      15.232|      23.424|           0                         
---|---range-|---samples                                                        
HSD| 14 - 15 |      18667                                                       
HSD| 15 - 16 |     541601                                                       
HSD| 16 - 17 |         54                                                       
HSD| 17 - 18 |         89                                                       
HSD| 18 - 19 |      14814                                                       
HSD| 19 - 20 |      11606                                                       
HSD| 20 - 21 |       3030                                                       
HSD| 21 - 22 |         57                                                       
HSD| 22 - 23 |         61                                                       
HSD| 23 - 24 |         22                                                       
HSS|    590001|     15.150|      0.831                                          

switchtest

This case tests how many task switches could happen every second. It also demos writing RT application using POSIX interface.

The Xenomai RT thread is created the same way as normal pthread:

err = pthread_create(&param->thread, rt_attr,
                      task_routine[param->type], param);

However normal pthread is created in different way in xenomai/adeos:

err = __real_pthread_create(&param->thread, sleeper_attr,
                   sleeper, param);
root:/> switchtest --help                                                       
Usage:                                                                             
"Create threads of various types and attempt to switch context between these                
threads, printing the count of context switches every second.                   
                                                                                
Available options are:                                                          
--help or -h, cause this program to print this help string and exit;            
--lines <lines> or -l <lines> print headers every <lines> lines.                
--quiet or -q, prevent this program from printing every second the count of     
context switches;                                                               
--timeout <duration> or -T <duration>, limit the test duration to <duration>    
seconds;                                                                        
--nofpu or -n, disables any use of FPU instructions.                            
                                                                                
Each 'threadspec' specifies the characteristics of a thread to be created:      
threadspec = (rtk|rtup|rtus|rtuo)(_fp|_ufpp|_ufps)*[0-9]*                       
rtk for a kernel-space real-time thread;                                        
rtup for a user-space real-time thread running in primary mode,                 
rtus for a user-space real-time thread running in secondary mode,               
rtuo for a user-space real-time thread oscillating between primary and          
secondary mode,                                                                 
                                                                                
_fp means that the created thread will have the XNFPU bit armed (only valid for 
rtk),                                                                           
_ufpp means that the created thread will use the FPU when in primary mode       
(invalid for rtus),                                                             
_ufps means that the created thread will use the FPU when in secondary mode     
(invalid for rtk and rtup),                                                     
                                                                                
[0-9]* specifies the ID of the CPU where the created thread will run, 0 if      
unspecified.                                                                    
                                                                                
Passing no 'threadspec' is equivalent to running:                               
switchtest rtk0 rtk0 rtk_fp0 rtk_fp0 rtk_fp_ufpp0 rtk_fp_ufpp0 rtup0 rtup0 rtup0
                                                                                
Passing only the --nofpu or -n argument is equivalent to running:               
switchtest rtk0 rtk0 rtup0 rtup0 rtus0 rtus0 rtuo0 rtuo0             

Be sure the enable “xeno_switchtest” driver in kernel before running this test:

Real-time sub-system  --->
    [*] xenomai
       [*] Nuclues
            Drivers --->
               Testing Drivers --->
                   <*> Context switch unit testing driver

Sample output on BF537-STAMP. Each line displays how many context switches happened in one second (2250 or 2259 in this test).

root:/> switchtest
== Testing FPU check routines...
== FPU check routines: unimplemented, skipping FPU switches tests.
== Threads: sleeper-0 rtk-1 rtk-2 rtup-3 rtup-4 rtus-5 rtus-6 rtuo-7 rtuo-8
RTT|  00:00:01
RTH|ctx switches|-------total
RTD|        6615|        6615
RTD|        6617|       13232
RTD|        6638|       19870
RTD|        6617|       26487
RTD|        6617|       33104
RTD|        6638|       39742
RTD|        6617|       46359
RTD|        6617|       52976
RTD|        6638|       59614
RTD|        6617|       66231
RTD|        6617|       72848
RTD|        6638|       79486
RTD|        6619|       86105
RTD|        6620|       92725
RTD|        6633|       99358

cyclictest

Test high resolution timers. It is ported from http://rt.wiki.kernel.org/index.php/Cyclictest. It measures the latency between expected timer expire time and actual expire time.

Here is output from BF537-STAMP:

* Test case: clock_nanosleep(TIME_ABSTIME), Interval 10000 microseconds,. 10000 loops, no load.

root:/> cyclictest -t 1 -p 80 -n -i 10000 -l 10000
0.00 0.00 0.00 1/22 248
T: 0 (  248) P:80 I:   10000 C:   10000 Min:       9 Act:      22 Avg:      16 Max:      32

The meaning of output. T: thread_cound (thread_id) P: thread priority I: interval (us) C: loop count Min: minimum timer latency (us) Act: actual latency Avg: average latency Max: max latency

* Test case: POSIX interval timer, Interval 10000 micro seconds,. 10000 loops, no load.

root:/> cyclictest -t 1 -p 80 -i 10000 -l 10000
0.00 0.00 0.00 1/22 245
T: 0 (  245) P:80 I:   10000 C:   10000 Min:       8 Act:      21 Avg:      15 Max:      30

clocktest

For each CPU, it repeatedly prints a time offset (compared to the reference gettimeofday()), a drift value, the number of warps and the maximum warp in microseconds.

A snapshot for BF537-STAMP:

usage: clocktest [options]
  [-C <clock_id>]              # tested clock, default=0 (CLOCK_REALTIME)
  [-T <test_duration_seconds>] # default=0, so ^C to end
root:/> ./clocktest 
== Tested clock: 0 (CLOCK_REALTIME)
CPU      ToD offset [us] ToD drift [us/s]      warps max delta [us]
--- -------------------- ---------------- ---------- --------------
  0   1167601205919116.0            0.009          0            0.0

IPIPE Tracer

IPIPE Tracer is a tracing tool to analyze ADEOS. IPIPE tracer doc has an introduction. Here are steps to configure ipipe-tracer on Blackfin.

The ipipe-tracer for Blackfin is part of bfin-adeos patch.

  • Configure kernel

Kernel configuration looks like:

 --- Kernel hacking                                                                                                                          
   [*]   I-pipe debugging  
   [*]     Check for illicit cross-domain calls                                                                                                                      
   [*]     Latency tracing                                                                                                                        
   [ ]       Enable tracing on boot                                                                                                                
   [*]       Intrument function entries                                                                                                            
   [*]       Trace IRQs-off times                                                                                                                 
   (14)      Depth of trace log (14 => 16Kpoints, 15 => 32Kpoints)                                                                                 
   [ ]       Use vmalloc'ed trace buffer
   [ ]       Enable panic back traces

Build kernel and boot. Also build “latency” test case introduced above.

  • Configure ipipe-trace and begin trace

After kenel boot, configure ipipe-tracer via the ”/proc/ipipe/tracer”:

# Enable ipipe-tracer
root:~> echo 1 > /proc/ipipe/trace/enable

# Enable verbose output 
root:~> echo 1 > /proc/ipipe/trace/verbose

# We may need more back trace point for analyze 
root:~> echo 200 > /proc/ipipe/trace/back_trace_points
  • Run the “latency” test case for a while and get trace.
# Make the test run in "Timer interrupt mode", with timer period = 200us, freeze the maxinum interrupt path
root:~> ./latency -t2 -p200 -f

root:~> cat /proc/ipipe/trace/frozen
I-pipe frozen back-tracing service on 2.6.16.27-ADI-2006R2/ipipe-1.5-00
------------------------------------------------------------
Freeze: 144879102832 cycles, Trace Points: 200 (+50)

 +----- Hard IRQs ('|': locked)
 |+---- <unused>
 ||+--- <unused>
 |||+-- Xenomai
 ||||+- Linux ('*': domain stalled, '+': current, '#': current+stalled)
 |||||                        +---------- Delay flag ('+': > 1 us, '!': > 10 us)
 |||||                        |        +- NMI noise ('N')
 |||||                        |        |
<snip>

:|  # func                 -48    0.922  ___ipipe_grab_irq+0x22 (__common_int_entry+0x72)

/* Begin to handle IRQ6 */
:|  # begin   0x00000006   -47+   1.820  ___ipipe_grab_irq+0x34 (__start+0xfffff000)
:|  # func                 -45+   1.530  ___ipipe_handle_irq+0x1e (___ipipe_grab_irq+0x3c)
:|  # func                 -43+   1.168  _rthal_timer_ack+0x8 (___ipipe_handle_irq+0x178)
:|  # func                 -42+   1.102  ___ipipe_dispatch_wired+0x14 (___ipipe_handle_irq+0x190)
:|  # end     0x00000006   -41+   2.010  ___ipipe_grab_irq+0x42 (__start+0xfffff000)
:   # func                 -39+   1.090  _ipipe_unstall_pipeline_head+0x8 (_xnshadow_harden+0xce)
:|  # begin   0x80000001   -38+   2.308  _ipipe_unstall_pipeline_head+0x18 (__start+0xfffff000)
:|  + func                 -36+   1.892  ___ipipe_sync_stage+0xe (_ipipe_unstall_pipeline_head+0x74)
:|  # func                 -34+   1.580  _xnintr_clock_handler+0x10 (___ipipe_sync_stage+0x186)
:|  # func                 -32+   1.460  _xnintr_irq_handler+0x18 (_xnintr_clock_handler+0x18)
:|  # func                 -31+   1.728  _xnpod_announce_tick+0xc (_xnintr_irq_handler+0x2a)
:|  # func                 -29+   2.442  _xntimer_do_tick_aperiodic+0x14 (_xnpod_announce_tick+0x16)
:|  # func                 -27+   1.590  _timer_proc+0xc (_xntimer_do_tick_aperiodic+0x1b2)
:|  # func                 -25+   1.880  _xnpod_get_time+0xa (_timer_proc+0x10)
:|  # func                 -23+   1.598  ___muldi3+0xc (_xnpod_get_time+0x52)
:|  # func                 -21+   2.432  ___muldi3+0xc (_xnpod_get_time+0x6e)
:|  # func                 -19+   4.130  ___div64_32+0x10 (_xnpod_get_time+0x1f0)
:|  # func                 -15+   6.770  ___div64_32+0x10 (_xnpod_get_time+0x1e4)
:|  # func                  -8+   1.808  _eval_inner_loop+0xe (_timer_proc+0x24)
:|  # func                  -6+   1.432  _ipipe_trace_frozen_reset+0xa (_eval_inner_loop+0xe0)
:|  # func                  -5+   4.400  ___ipipe_global_path_lock+0xa (_ipipe_trace_frozen_reset+0xe)

/* ipipe_trace_freeze() called here */
<|  # freeze  0x00010ba8     1    2.370  _eval_inner_loop+0xe6 (__start+0xfffff000)
 |  # func                   3    1.308  _xnpod_get_time+0xa (_timer_proc+0x2a)
 |  # func                   4    1.482  ___muldi3+0xc (_xnpod_get_time+0x52)
 
 <snip>

In the log above, there is a trace point:

:|  # begin   0x80000001   -38+   2.308  _ipipe_unstall_pipeline_head+0x18 (__start+0xfffff000)

This is a result of calling:

include/asm-blackfin/system.h:

#define local_irq_save_hw(x)                    \
do {                                            \
        __save_and_cli_hw(x);                   \
        if (local_test_iflag_hw(x))             \
                ipipe_trace_begin(0x80000001);  \
} while (0)

So we can use ipipe-tracer to trace the turn on/off of HW interrupts.

And we can use ipipe-tracer to analyze what happens when the maxium interrupt happens:

/* A Core Timer interrupt (IRQ6) is catched by Adeos */
/* Here we also recorded the current IMASK, ILAT, IPEND regisers */
:|  # func    0x0000001f, 0x00001000, 0x00008040
  -65    0.940  ___ipipe_grab_irq+0x22 (__common_int_entry+0x72)
:|  # begin   0x00000006   -64+   2.300  ___ipipe_grab_irq+0x34 (__common_int_entry+0x72)
:|  # func    0x0000001f, 0x00001000, 0x00008040
  -61+   1.650  ___ipipe_handle_irq+0x1e (___ipipe_grab_irq+0x3c)
:|  # func    0x0000001f, 0x00001000, 0x00008040
  -60+   1.508  _rthal_timer_ack+0x8 (___ipipe_handle_irq+0x178)
:|  # func    0x0000001f, 0x00001000, 0x00008040
  -58+   1.382  ___ipipe_dispatch_wired+0x14 (___ipipe_handle_irq+0x190)
:|  # end     0x00000006   -57+   3.060  ___ipipe_grab_irq+0x42 (__common_int_entry+0x72)

/* Before the IRQ6 handler registered in Xenomai domain is invoked, another interrupt (0x1a) enters Adeos */

:|  # func    0x0000001f, 0x00000000, 0x00009000
  -54+   2.670  ___ipipe_grab_irq+0x22 (__common_int_entry+0x72)
:|  # begin   0x0000001a   -51+   1.940  ___ipipe_grab_irq+0x34 (__common_int_entry+0x72)
:|  # func    0x0000001f, 0x00000000, 0x00009000
  -49+   3.540  ___ipipe_handle_irq+0x1e (___ipipe_grab_irq+0x3c)
:|  # func    0x0000001f, 0x00000000, 0x00009000
  -46+   2.310  ___ipipe_ack_irq+0x10 (___ipipe_handle_irq+0xee)
:|  # func    0x0000001f, 0x00000000, 0x00009000
  -43+   1.240  ___ipipe_walk_pipeline+0x18 (___ipipe_handle_irq+0x148)
:|  # end     0x0000001a   -42+   2.118  ___ipipe_grab_irq+0x42 (__common_int_entry+0x72)
:   # func    0x0000ffff, 0x00000000, 0x00008000
  -40+   1.242  _ipipe_unstall_pipeline_head+0x8 (_xnshadow_harden+0xce)
:|  # begin   0x80000001   -39+   2.870  _ipipe_unstall_pipeline_head+0x18 (_xnshadow_harden+0xce)
:|  + func    0x0000001f, 0x00000000, 0x00008000
  -36+   2.120  ___ipipe_sync_stage+0xe (_ipipe_unstall_pipeline_head+0x74)

/* When acknowledged the IRQ, the IRQ6 handler is called, however, after some delay by above handling of IRQ 0x1a */

:|  # func    0x0000001f, 0x00000000, 0x00008000
  -34+   1.360  _xnintr_clock_handler+0x10 (___ipipe_sync_stage+0x186)
:|  # func    0x0000001f, 0x00000000, 0x00008000
  -32+   1.710  _xnintr_irq_handler+0x18 (_xnintr_clock_handler+0x18)

As we can see, ipipe-tracer is helpful to analyze the call path.

Links to Adeos and Xenomai

Other resources