Inter Core Communication Introduction
Use Cases
- Compute Accelerator
- Real Time Task
Shared Memory based Inter-core Communication Protocol
Inter-core communication Framework design for Linux and bare metal application
Wrap ICC framework by MCAPI
Other implementations

Inter Core Communication Introduction

Parallel Processing refers to the concept of speeding-up the execution of a program by dividing the program into multiple fragments that can execute simultaneously, each on its own processor. A program being executed across n processors might execute n times faster than it would using a single processor.

Traditionally, multiple processors were provided within a specially designed “parallel computer”; along these lines, Linux now supports SMP systems in which multiple processors share a single memory and bus interface within a single computer. It is also possible for a group of computers (for example, a group of PCs each running Linux) to be interconnected by a network to form a parallel-processing cluster. The third alternative for parallel computing using Linux is to use the multimedia instruction extensions (i.e., MMX) to operate in parallel on vectors of integer data. Finally, it is also possible to use a Linux system as a “host” for a specialized attached parallel processing compute engine. All these approaches are discussed in detail in the Parallel-Processing How To, and the 4^th, (a specialized attached parallel processing compute engine) will be described here.

This document describes some design/architecture ideas on how to make Core B easier to use within a Linux framework. This initial implementation can be found in blackfin kernel and uClinux-dist source repositories under folder named icc. If you would like to provide feedback, please do on the fourms.

The classical trade-off between system performance and ease of programming is one of the primary differentiators between general purpose operating system (GPOS) and real-time operating systems (RTOS).

GPOSes tend to provide a higher degree of resource abstraction. This improves application portability, ease of development and increases system robustness through software modularity and isolation of resources. This makes a GPOS ideal for addressing general purpose system components such as networking, user interface and display management.

However, this abstraction sacrifices the fine-grained control of system resources required to meet the performance goals of computationally intensive algorithms such as signal processing code. For this level of control, developers typically turn to a real-time operating system (RTOS), or program directly on bare metal.

Use Cases

There are various use cases for wanting to be able to load bare metal applications or appications under real time OS into Core B, and use it like a hardware accelerator.

Compute Accelerator

There are various things that can be done to accelerate some task which normally runs under the Linux kernel.

Video Accelerator

Running an optimized H.264 or MPEG or WMV video codec on Core B, with mplayer running on Core A.

mplayer runs in Linux on Core A
the Linux kernel manages 100% of the peripherals (including LCD)
H.264 decoder on core B
The two pass the raw bitstream, and decoded video through the CPU→DSP framework.

The decoder does nothing except for decoding video stream into some frame buffers. Mplayer open a h.264 bit stream from either a file on disk or a connection over network.

Video

mplayer runs in Linux on Core A
the Linux kernel manages some of the peripherals (not including LCD)
H.264 decoder on core B
The mplayer passes the raw bitstream to the decoder, and decoded video is passed directly to the PPI from Core B.
The DSP code should negotiate a proper DMA/IRQ/GPIO/DRAM resource allocation with Linux kernel through the blackfin DSP framework.

Crypto Accelerator

Crypto_API_(Linux) offers hardware acceleration support.

Real Time Task

There are times where the hard real time performance offered by the Linux kernel or by ADEOS are not enough for the application. In those select times, you can still use a thin RTOS (VDK, uCos, etc.) on CORE B, and Linux on Core A.

Shared Memory based Inter-core Communication Protocol

This section is intended to define the communication well enough that different implementations can successfully communicate.

Shared Memory

There are a fixed number of shared variables with sizes and addresses known to each processor.

There is a shared variable for the basic message queue. Protocols that use this queue may require additional shared variables or may require individual processors to have a pool of shareable memory from which buffers can be allocated.

If processors have different word sizes and address maps then addresses of shared buffers and the size of addressable units could differ, and the protocol would need to define a common address representation and addressable unit. We also define types that have at least 16 bits and 32 bits with size larger than or equal to the smallest addressable unit.

We will use the data types:

typedef 'some unsigned integer' sm_unit_t;   // defined in specifics
typedef 'some unsigned integer' sm_uint16_t; // defined in specifics
typedef 'some unsigned integer' sm_uint32_t; // defined in specifics
typedef 'some integral type' sm_address_t;   // defined in specifics

Specifics for Blackfin

Both cores on BF561 and BF60x have the same address space and are byte addressed.

typedef uint8_t sm_unit_t;
typedef uint16_t sm_uint16_t;
typedef uint32_t sm_uint32_t;
typedef void *sm_address_t;

Cache policy

One of the assumptions of the MCAPI/ICC protocol is that the payload buffer received on one core is located in the memory region managed(owned) by the other cores.

Core 0 should set up write through CPLB entries for the memory region managed by core 1. So, The invalidate instruction on core 0 doesn’t flush dummy data in cache back to the MCAPI payload buffer sent by core 1 or drop unrelated data in the same cache line near the MCAPI payload boundary. What CPLB entries (WT/WB) are set up for the same memory region on core 1 doesn’t matter, because core 1 should flush the MCAPI payload buffer before sending.

For example:

BF609 mem addr	Owner	Core0 cache	Core1 cache
0~0x3FFFFF	Core0	WB	WT
0x400000~0x800000	Core1	WT	WB

Atomic access

The specific part defines a type which may be read and written atomically. Two operations are defined on the type: Read and Write, and it may hold values of sm_uint16_t.

// defined at specifics
typedef 'some type' sm_atomic_t;
sm_uint16_t sm_read_atomic(volatile sm_atomic_t *);
void sm_write_atomic(volatile sm_atomic_t *, sm_uint16_t);

Atomic means that if one core writes a variable and another core reads it, the value read is either the value before the write or the value after it and not some third value because the write had only half completed when the read occurred.

Atomic operations on a single core must also be ordered with respect to each other so the following logic holds.

// initial values
sm_atomic_t a = 0, b = 0;

on processor 0:
  sm_write_atomic(&a, 1);
  while (sm_read_atomic(&b) == 0)
    ;

on processor 1:
  sm_write_atomic(&b, 1);
  x = sm_read_atomic(&a);
  assert(x == 1);
  // because read(b) must follow write(a) on processor 0

Specifics for Blackfin

On BF561 and BF60x both cores use the same bus to L2 and the EBUI. L2 memory is 64-bits wide and memory attached to the EBUI is at least 16-bits wide. So an uncached 16-bit write to L2 or L3 is atomic.

typedef uint16_t sm_atomic_t;
inline void sm_atomic_write(volatile sm_atomic_t *a, sm_uint16_t v) {
  *a = v;
}
inline sm_uint16_t sm_atomic_read(volatile sm_atomic_t *a) {
  return *a;
}

Interrupts

Each processor must be able to raise interrupts on the other processor. We use one interrupt on each core which indicates some action is required. The interrupt handler works out what the action is from the channel state. So all modifications to shared data should be visible to both processors by the time the interrupt handler is entered.

The mechanism for initialising interrupt handlers and clearing the interrupt source is necessarily processor and environment specific. The initialisation sequence described below requires the interrupt to be initially masked, which is usually the case.

Specifics for Blakcfin

CPU	master core ICC interrupt	slave core ICC interrupt
BF561	core supplemental interrupt 0	core supplemental interrupt 0
BF60x	SEC soft interrupt 0	SEC soft interrupt 1

The protocol does not define the core interrupt vectors used to handle this interrupts or whether they are shared with other interrupt sources, as this is a decision local to the environment running on the core.

Modifications to shared data are made visible to the other core before raising the interrupt by

ensuring any cached writes are flushed from cached
executing an SSYNC instruction to flush the write buffer.

The interrupted core is responsible for ensuring the initial reads of shared data are not from cache.

Message passing

The protocol is for two way communication between two processors. If there are more processors in the system then the protocol could be used for separate two way channels between each pair of processors.

There are four message queues. Two in each direction, one for high priority messages and the other for standard priority.

Message queues are circular buffers containing SM_MSGQ_LEN fixed size messages.

  typedef struct {
    sm_atomic_t sent;
    sm_atomic_t received;
    sm_msg_t buf[SM_MSGQ_LEN];
  } sm_msgq_t;

The size and content of sm_msg_t is defined in Part 2 below.

SM_MSGQ_LEN is a constant. For efficiency it should be a power of two.

The message queue uses a lockless protocol. The sender always writes a message at sent % SM_MSGQ_LEN and then increments sent, and the receiver always reads from received % SM_MSGQ_LEN and then increments received. The number of messages in the queue is (sm_uint16_t)(sent - received). The counters are unsigned so, due to the wonders of modulo arithmetic, this is true even if received > sent because sent has wrapped round.

Before sending a message the sender checks that there is space available in the buffer. If space is available the sender writes the message to buf[sent % SM_MSGQ_LEN], increments sent, then raises the 'Action Required' interrupt on the other processor. If no space is available in the buffer the calling process must block.

The handler for the 'Action Required' interrupt causes a receiver for both high and standard priority queues to run. Whether the receivers execute within the handler or are just scheduled to run once it returns is environment dependent.

The receiver checks the number of messages in the buffer. If there are any it reads the message at buf[received % SM_MSGQ_LEN] and tries to deliver it. If successful it increments received and raises the 'Action Required' interrupt on the sending core.

The interrupt handler also checks whether space has come available in the queues there are processes blocked on.

The definition of a process, the mechanism for blocking a process, and the method of dealing with the race condition between the sender blocking and the receiver raising the interrupt is processor and environment specific and outside the scope of the protocol. For example in a bare metal environment there is only one “process” and it can block by spinning on a variable that is set by the interrupt handler whereas other environments would use operating system primitives.

A message channel between a pair of processors is composed of 4 message queues.

  typedef struct {
    volatile sm_msgq_t msgq[2][2];
  } sm_channel_t;

The processor of large cpu_id is on the contrary. It receives on msgq[priority][1] and sends on mesgq[priority][0]. The message queue id can be identified according to current cpu, destination cpu and cpu which sends the inter processor interrupt.

  if (cur_cpuid == ipi_src_cupid || cur_cpuid == des_cpuid)
      BUG();

  recv_msgq_id = cur_cpuid < ipi_src_cpuid ? 0 : 1;
  send_msgq_id = cur_cpuid < des_cpuid ? 1 : 0;

Each processor receives high priority on the queue msgq[0][recv_msgq_id], and the standard priority messages on msgq[1][recv_msgq_id]. So do the queue to send message.

If there are N processors in the architecture, there should be a channel array of (N - 1) * N / 2 channels. The channel arrays exist at a known location of the shared memory. The channel id can be identified according to current cpu and remote cpu.

Message Channel ID Table

Processor ID	0	1	2	3
0	NA	0	1	2
1	NA	NA	3	4
2	NA	NA	NA	5
3	NA	NA	NA	NA

  #define CPU_NUM 4
  sm_channel_t channels[(CPU_NUM - 1) * CPU_NUM / 2];
  int8 channel_table[CPU_NUM][CPU_NUM] = {
   {-1, 0, 1, 2}, 
   {-1,-1, 3, 4},
   {-1,-1,-1, 5},
   {-1,-1,-1,-1},
  };
  
  channel_id = channel_table[cur_cpuid, remote_cpuid];
  if (channel_id < 0 || channel_id >= CPU_NUM)
      BUG();
  channel = channels[channel_id];

Each message queue is statically initialised with received and sent containing the value 0. When a processor starts running the 'Action Required' interrupt is masked before attempting to sending the first message a handler is installed and the interrupt is unmasked. A message queue can be written to before the receiver has initialized its interrupts. If it fills up the 'Action Required' signal is raised but not serviced until the receiver unmasks the interrupt.

Specifics for Blackfin

A single block of four message queues is held in the shared variable at a known address msgq. Its start address should be at a fixed position known to code running on all processors.

  #define MSGQ_START_ADDR	0xFEB00000	// in BF561 and BF60x L2 SRAM

  typedef struct {
    volatile sm_msgq_t msgq[2][2];
  } sm_channel_t;

  static dsp_channel_t *sm_ch = (sm_channel_t *)MSGQ_START_ADDR;

Core A receives messages on sm_ch→msgq[priority][0]
Core B receives messages on sm_ch→msgq[priority][1]

Message Format

  typedef sm_unint16_t sm_endpoint_t;

  typedef struct {
    sm_endpoint_t dst_ep, src_ep;
    sm_uint32_t type;
    sm_uint32_t length;
    sm_address_t payload;
  } sm_msg_t;

The fields dst_ep and src_ep denote endpoints. The meaning of an endpoint is application dependent. The receiver should inspect the dst_ep field to decide how to process the message. The src_ep field indicates the sender which may be meaningful to the receiving endpoint.

type is an an unsigned 32 bit integer value that indicates a message type defined in one of the higher level protocols and is mainly interpreted by the endpoint.

The top eight bits of the value indicates the protocol and the low 24 bits the subtype.

  // compose type enumeration value from protocol & subtype
  #define SM_MSG_TYPE(protocol, subtype) (((protocol)<<24)|(subtype))

  // extract subtype from type enumeration value
  #define SM_MSG_SUBTYPE(type) ((type)&0xffffff)

  // extract protocol from type enumeration value
  #define SM_MSG_PROTOCOL(type) (((type)>>24)&0xff)

An endpoint may recognise more than one protocol. The receiver must know the protocols recognised by each endppoint. When dst_ep has the value 0xffff the message is broadcast to every endpoint which recognises the protocol encoded in the type field.

The meaning of length and payload is dependent on the value of type and interpreted by the endpoint.

Specifics for BF561

On BF561 message reads and writes to a queue in L2 is more efficient if 'sm_msg_t' is aligned on a 64 bit boundary. An aligned access should take 14 rather than 21 cycles.

The type declaration for the VisualDSP compiler should use pragma align:

typedef struct {
#pragma align 8
  ...
} sm_msg_t;

All protocol Types

All protocol types

enum {
        SP_GENERAL = 0,
        SP_CORE_CONTROL,
        SP_TASK_MANAGER,
        SP_RES_MANAGER,
        SP_PACKET,
        SP_SESSION_PACKET,
        SP_SCALAR,
        SP_SESSION_SCALAR,
        SP_MAX,
};

Standard Message Types

All protocols should recognise the standard message types.

A couple of common error conditions are covered by standard messages.

These are sent with the same priority as the message to which they responding.

SM_BAD_ENDPOINT

All endpoints should recognise the message:

SM_BAD_ENDPOINT = SM_MSG_TYPE(0, 0)

This may be sent in response to a message sent by this endpoint to indicate the dst_ep field was invalid.

The SM_BAD_ENDPOINT message has its src_ep field set to the invalid endpoint id, its length to 0, and its payload to the type value of the original message.

The SM_BAD_ENDPOINT message may not be sent in all environments. If endpoints can be created dynamically it may be more appropriate to queue the message until the endpoint is created.