path: root/doc/networking-profile.md

# ENEA LINUX NETWORKING PROFILE

Technology trends show that Linux-based Operating Systems have increased their
presence in the area of high-performance networking applications. While there
are no standardized ways of programatically accessing hardware offload
capabilities, several paradigms co-exist in Linux ecosystem to address this
specific need (e.g. USDPAA, DPDK, ODP etc.) Networking Profile in Enea Linux is
a framework for anyone attempting to implement high-performance networking
applications on various hardware platforms. It aims to bring in place all
necessary building blocks which facilitate efficient development of Linux-based
solutions on top of network accelerated hardware platforms. As different
hardware platforms have distinct data-path acceleration solutions, Networking
Profile implementation is very dependent on underlying hardware capabilities.

This document tries to describe the implementation details, changes, additions,
kernel configurations and tunings Enea made in order to achieve highly
optimized Linux distributions for networking applications.

The following paragraphs focus on Enea Linux Networking Profile on DPAA-based
QorIQ platforms, illustrating the implementation and changes on NXP P2041rdb
target.

	Table of Content
	-------------------------------------------
	1. Supported Targets
	2. Real-Time Performance
	------- 2.1 Kernel Modifications
	------- 2.2 CPU-Isolation with partrt
	------- 2.3 Latency Benchmarks
	3. USDPAA Usage
	------- 3.1 Packages
	------- 3.2 Prepare Target
	------- 3.3 Device Trees
	------- 3.4 Boot Parameters
	------- 3.5 Boot Instructions P2041RDB
	------- 3.6 Run Reflector
	------- 3.7 Run SRA
	------- 3.8 Throughput using USDPAA

## 1. Supported Targets
Enea Linux Networking Profile has initially been tested on p2041rdb.
<!--
        Table 1.1 Functionally verified targets
| Target   | Reflector App | SRIO | SRIO RCW available?
| ---      | ---           | ---  | ---                |
| p2041rdb | OK            | Yes  | Yes*

\* RCW that supports SRIO has to be created in code warrior or copied from the
  SRA User Guide [NXP-SDK], see section 3.1.
-->
## 2. Improving Real-Time Performance
<!--
FIXME/WIP! need to add nohz and test.
-->

### 2.1 Kernel Modifications
When modifying a kernel for high-performance and low-latency applications there
are several aspects to take into consideration. In the [Enea Linux Real-Time
Guide](http://linuxrealtime.org/index.php/Main_Page) a thorough investigation
and explanation of how to optimize Linux for low latency is given. Below is a
short description of kernel features added specifically to Enea Linux
Networking Profile in order to enhance real-time performance.

            Table 2.1 Added kernel features and their properties.
| Change                                          | Reason
| ---                                             | ---      |
| RCU priority boosting -> cfg/rcu_boost.cfg      | Give low priority readers a higher priority to keep them from blocking tasks of higher prority. [RCU] |
| Offload RCU callback Processing -> cfg/rcu_nocb | To reduce OS jitter, enable offloading of RCU callback processing to kernel threads. The rcu_nocbs boot parameter is used to define the set of CPUs to be offloaded.    |
| Hotplug CPU -> cfg/hotplug_cpu.cfg              | Allows CPUs to be added to/removed from a live kernel. [HOTPLUG] |
<!-- Adaptive-ticks CPU - cfg/nohz.cfg              Avoid scheduling clock interruptd for CPUs running a single task. [NOHZ] -->

#### 2.1.1 Boot Parameters
<!--
**TBD/FIXME:** Boot parameters for nohz_full and isolcpus.
-->
**From [KERN-PARA]:**
<!--
nohz=           [KNL] Boottime enable/disable dynamic ticks
                Valid arguments: on, off
                Default: on
-->
```

rcu_nocbs=      [KNL]
                In kernels built with CONFIG_RCU_NOCB_CPU=y, set
                the specified list of CPUs to be no-callback CPUs.
                Invocation of these CPUs' RCU callbacks will
                be offloaded to "rcuox/N" kthreads created for
                that purpose, where "x" is "b" for RCU-bh, "p"
                for RCU-preempt, and "s" for RCU-sched, and "N"
                is the CPU number.  This reduces OS jitter on the
                offloaded CPUs, which can be useful for HPC and
                real-time workloads.  It can also improve energy
                efficiency for asymmetric multiprocessors.

isolcpus=       [KNL,SMP] Isolate CPUs from the general scheduler.
                Format:
                <cpu number>,...,<cpu number>
                or
                <cpu number>-<cpu number>
                (must be a positive range in ascending order)
                or a mixture
                <cpu number>,...,<cpu number>-<cpu number>

                This option can be used to specify one or more CPUs
                to isolate from the general SMP balancing and scheduling
                algorithms. You can move a process onto or off an
                "isolated" CPU via the CPU affinity syscalls or cpuset.
                <cpu number> begins at 0 and the maximum value is
                "number of CPUs in system - 1".

                This option is the preferred way to isolate CPUs. The
                alternative -- manually setting the CPU mask of all
                tasks in the system -- can cause problems and
                suboptimal load balancer performance.
```

### 2.2 Cpu-isolation with partrt
A tool called partrt is included in the networking profile to divide an SMP
Linux system into partitions. A description of the tool can be found in the
[Linux Real-Time
Guide](http://linuxrealtime.org/index.php/Improving_the_Real-Time_Properties#The_CPU_Partitioning_Tool_-_partrt).

### 2.3 Latency Benchmarks
The cyclictest suite [CYCLIC] is a measurement of system latency used in many
projects. As a comparison the measurement was applied to Enea Linux 6.0
Standard, and to Enea Linux Networking profile to investigate the impact of the
changes to the system.

Below *cyclictest* is tested on the two different systems, average and
maximum latency are presented in the tables below, first the test on the
standard profile and after the results on the networking profile are shown. It
is also combined with *stress* [STRESS] to show system performance under
different type of loads.

<!--
FIXME!
 **Enea Linux 6.0 Standard System Info**
 - Kernel size:
 - Root-fs size:

**Enea Linux 6.0 Networking Profile System Info**
 - Kernel size:
 - Root-fs size:
-->

        Command: Cyclictest with no stress

|  CPU \#  |  P  |  I   |  C_std  | Avg_std (us) | Max_std (us)| C_net       | Avg_net (us)| Max_net (us)
| ---      | --- | ---  | ---    | ---      | ---     | ---    | ---     | --- |
|  0       | 99  | 1000 | 100000 | 9        | 24      | 100000 | 6       | 11  |
|  1       | 99  | 1500 | 66817  | 9        | 21      | 66812  | 6       | 12  |
|  2       | 99  | 2000 | 50208  | 9        | 23      | 50106  | 6       | 18  |
|  3       | 99  | 2500 | 40083  | 9        | 14      | 40082  | 6       | 10  |

        Command: cyclictest with hdd stress:
        # stress -d 4 --hdd-bytes 1M &

|  CPU \#  |  P  |  I   |  C_std  | Avg_std (us) | Max_std (us)| C_net       | Avg_net (us)| Max_net (us)
| ---      | --- | ---  | ---    | ---      | ---     | ---    | ---     | --- |
|  0       | 99  | 1000 | 100000 | 14       | 223     | 100000 | 11      | 77  |
|  1       | 99  | 1500 | 66820  | 14       | 231     | 66745  | 11      | 100 |
|  2       | 99  | 2000 | 50109  | 14       | 186     | 50055  | 11      | 76  |
|  3       | 99  | 2500 | 40083  | 14       | 176     | 40041  | 11      | 81  |

        Command: cyclictest with vm stress:
        # stress -m 4 --vm-bytes 4096 &

|  CPU \#  |  P  |  I   |  C_std  | Avg_std (us) | Max_std (us)| C_net       | Avg_net (us)| Max_net (us)
| ---      | --- | ---  | ---     | ---      | ---     | ---    | ---     | --- |
|  0       | 99  | 1000 | 100000  | 5        | 20      | 100000 | 3       | 15  |
|  1       | 99  | 1500 | 66818   | 6        | 14      | 66739  | 3       |   7 |
|  2       | 99  | 1500 | 50109   | 6        | 14      | 50103  | 3       |   9 |
|  3       | 99  | 1500 | 40079   | 6        | 14      | 40081  | 3       |   6 |

        Command: cyclictest with full stress trial 1:
        # stress -c 4 -i 4 -m 4 --vm-bytes 4096 -d 4 --hdd-bytes 4096 &

|  CPU \#  |  P  |  I   |  C_std  | Avg_std (us) | Max_std (us)| C_net       | Avg_net (us)| Max_net (us)
| ---      | --- | ---  | ---    | ---      | ---     | ---   | ---     | --- |
|  0       | 99  | 1000 | 99808  |  7       | 93      | 99815 | 6       | 58  |
|  1       | 99  | 1500 | 66733  |  9       | 105     | 66739 | 6       | 54  |
|  2       | 99  | 2000 | 50039  | 9        | 79      | 50039 | 7       | 61  |
|  3       | 99  | 2500 | 40016  | 10       | 83      | 40032 | 6       | 57  |

        Command: cyclictest with full stress trial 2:
        # stress -c 4 -i 4 -m 4 --vm-bytes 4096 -d 4 --hdd-bytes 1M &

|  CPU \#  |  P  |  I   |  C_std  | Avg_std (us) | Max_std (us)| C_net       | Avg_net (us)| Max_net (us)
| ---      | --- | ---  | ---     | ---      | ---     | ---     | ---     | --- |
|  0       | 99  | 1000 | 100000  | 13       | 201     | 100000  | 10      | 87  |
|  1       | 99  | 1500 | 66646   | 11       | 186     | 66685   | 9       | 85  |
|  2       | 99  | 2000 | 49969   | 10       | 195     | 49998   | 10      | 70  |
|  3       | 99  | 2500 | 39960   | 11       | 112     | 39992   | 10      | 90  |

## 3. USDPAA Usage
The need for predictive and good performance for networking systems is
critical. One way of achieving greater performance is for user-space to avoid
interactions with the kernel. The kernel is responsible for hardware
acceleration allocation and scheduling. By using frameworks such as
USDPAA[NXP-SDK] and DPDK[DPDK] control over certain hardware can be given
to user-space. USDPAA is specific to NXP/Freescale's QoriQ platforms, for more
information please see their guide to USDPAA [NXP-SDK].

In the guides below, an example of how to prepare a p2041rdb target with SRIO
and ethernet through USDPAA is given.


### 3.1 Packages
The networking profile supports and includes all packages necessary for
software support of USDPAA. If another image is created the below listed
packages are relevant to include (all available in meta-fsl-ppc and
meta-freescale) in order to add support for USDPAA and some example applications.

        * usdpaa
        * usdpaa-apps
        * fmc
        * fmlib
        * flib
        * eth-config

### 3.2 Prepare Target
The SRIO application needs us to boot with a RCW and board configuration that
allows usage of the PCI extender port. The below examples are specific to
p2041rdb, but similar steps can be taken for other targets where SRIO is not
enabled by default.

#### 3.2.1 Reset Control Word (RCW)
The reset control word must configure the serial-deserializer (SERDES) bus for
SRIO. This can be done by either a predefined binary/setting, or can be created
in Code Warrior [CW].

The RCW used in this example was given in the NXP/Freescale SRA User Guide of
[NXP-SDK].

To program the RCW to target from u-boot follow the steps below:

```
=> tftp 1000000 <path-2-file>/RR_RS_0x02.bin

=> md 0xec000000
ec000000: aa55aa55 010e0100 12600000 00000000 .U.U.....`......
ec000010: 241c0000 00000000 248e40c0 c3c02000 $.......$.@... .
ec000020: de800000 40000000 00000000 00000000 ....@...........
ec000030: 00000000 d0030f07 00000000 00000000 ................
ec000040: 00000000 00000000 091380c0 000009c4 ................
ec000050: 09000010 00000000 091380c0 000009c4 ................
ec000060: 09000014 00000000 091380c0 000009c4 ................
ec000070: 09000018 81d00000 091380c0 000009c4 ................
ec000080: 890b0050 00000002 091380c0 000009c4 ...P............
ec000090: 890b0054 00000002 091380c0 000009c4 ...T............
ec0000a0: 890b0058 00000002 091380c0 000009c4 ...X............
ec0000b0: 890b005c 00000002 091380c0 000009c4 ...\............
ec0000c0: 890b0090 00000002 091380c0 000009c4 ................
ec0000d0: 890b0094 00000002 091380c0 000009c4 ................
ec0000e0: 890b0098 00000002 091380c0 000009c4 ................
ec0000f0: 890b009c 00000002 091380c0 000009c4 ................

=> protect off 0xec000000 +$filesize
Un-Protected 1 sectors
=> erase 0xec000000 +$filesize

. done
Erased 1 sectors

=> cp.b 1000000 0xec000000 $filesize
Copy to Flash... 9done

=> protect on 0xec000000 +$filesize
Protected 1 sectors

=> md 0xec000000
ec000000: aa55aa55 010e0100 12600000 00000000 .U.U.....`......
ec000010: 241c0000 00000000 088040c0 c3c02000 $.........@... .
ec000020: de800000 40000000 00000000 00000000 ....@...........
ec000030: 00000000 d0030f07 00000000 00000000 ................
ec000040: 00000000 00000000 091380c0 000009c4 ................
ec000050: 09000010 00000000 091380c0 000009c4 ................
ec000060: 09000014 00000000 091380c0 000009c4 ................
ec000070: 09000018 81d00000 091380c0 000009c4 ................
ec000080: 890b0050 00000002 091380c0 000009c4 ...P............
ec000090: 890b0054 00000002 091380c0 000009c4 ...T............
ec0000a0: 890b0058 00000002 091380c0 000009c4 ...X............
ec0000b0: 890b005c 00000002 091380c0 000009c4 ...\............
ec0000c0: 890b0090 00000002 091380c0 000009c4 ................
ec0000d0: 890b0094 00000002 091380c0 000009c4 ................
ec0000e0: 890b0098 00000002 091380c0 000009c4 ................
ec0000f0: 890b009c 00000002 091380c0 000009c4 ................
=>
```

In order to obtain specific hardware settings, some multiplexers need to be
set. Descriptions of these can be obtained by typing **cpld -h** in u-boot,
information about the peripherals are also available in the board's respective
user guide. For SRIO on the p2041rdb target the following settings are
necessary.

```
cpld_cmd lane_mux 6 0
cpld_cmd lane_mux a 0
cpld_cmd lane_mux c 0
cpld_cmd lane_mux d 0
```
### 3.3 Device Trees
USDPAA enabled targets have very specific device trees, this is because instead
of handing over hardware to the linux kernel, it is managed by the DPAA
framework.

The networking profile includes a custom device-tree
(<machine>-usdpaa-enea.dtb), currently defined for p2041rdb. It gives one
ethernet interface to the kernel while the rest belongs to DPAA.

Available and tested device-trees for p2041rdb:

* uImage-p2041rdb-usdpaa-enea.dtb       EL custom interface, gives one interface to the linux kernel and remaining to DPAA.

### 3.4 Boot Parameters
USDPAA demands some custom boot arguments. If not given, or if given improperly
the USDPAA applications will not be usable. The NXP/Freescale manual covers these arguments, however might be misdirecting since the documentation in several places are, rather than target agnostic, specific instructions that are only applicable to certain targets. If unsure, one can consult the benchmarking
chapter of the NXP/Freescale SDK documentation that include more exact steps per
tested targets.

For our purposes of testing SRIO and the reflector application we only need the
'usdpaa_mem' boot argument. If the reserved memory is too large, it will cause
segmentation faults.

        Table 4.1 'usdpaa_mem=?'

| TARGET    | usdpaa_mem
| ---       | ---       |
| p2041rdb  |  =< 64M   |

### 3.5 Boot instructions P2041RDB
Below are instructions on how to boot a p2041rdb board with usdpaa enabled.

#### 3.5.1 Boot over NFS server
```
tftp 1000000 uImage-p2041rdb.bin
tftp c00000 uImage-p2041rdb-usdpaa-enea.dtb

setenv bootargs root=/dev/nfs
nfsroot=172.21.3.8:/unix/enea_linux_rootfs/<folder-path> rw ip=dhcp
console=ttyS0,115200 memmap=16M$0xf7000000 mem=4080M max_addr=f6ffffff
usdpaa_mem=64M

bootm 1000000 - c00000
```
#### 3.5.2 RAM Boot
```
tftp 1000000 uImage-p2041rdb.bin
tftp 2000000 uImage-p2041rdb-usdpaa.dtb
tftp 5000000 enea-image-networking-p2041rdb.ext2.gz.u-boot

setenv bootargs root=/dev/ram rw console=ttyS0,115200 ramdisk_size=10000000
log_buf_len=128K usdpaa_mem=64M

bootm 0x1000000 0x5000000 0x2000000
```

### 3.6 Run Reflector
Reflector is a demo application from NXP/Freescale that through ethernet
recieves a package and sends it back to with switched source-destination IP
addresses switched.

In order to test reflector an ethernet cable must be connected between either
two p2041rdb targets or between a work-PC and the p2041rdb target.

Connect two targets by ethernet and do the following steps to test connection
with usdpaa:

1. Boot Board A with uImage-p2041rdb-uspdaa-enea.dtb
2. Boot Board B with uImage-p2041rdb.dtb
3. On board A do the following:
```
        # Configure what ethernet ports that should be used by dpaa
        $ vi config-p2041rdb.xml
        <cfgdata>
          <config>
            <engine name="fm0">
              <port type="MAC" number="2"
policy="hash_ipsec_src_dst_spi_policy1"/>
              <port type="MAC" number="3"
policy="hash_ipsec_src_dst_spi_policy2"/>
              <port type="MAC" number="4"
policy="hash_ipsec_src_dst_spi_policy3"/>
              <port type="MAC" number="5"
policy="hash_ipsec_src_dst_spi_policy4"/>
            </engine>
          </config>
        </cfgdata>

        $ fmc -c config-p2041rdb.xml -p /usr/etc/usdpaa_policy_hash_ipv4.xml -a

        # start reflector and check the ethernet ports:
        $ reflector
        reflector> ifconfig
```
4. On board B do the following:
```
        # Configure the network interface to which you connected the ethernet
          cable, by choosing an ip- and MAC address.
        $ ifconfig -a
        $ ifconfig <eth-x> 192.168.0.10 netmask 255.255.255.0 up

        # Configure to connect to the hw address
        $ arp -s 192.168.0.11 <hw-address> -i <eth-x>

        # Ping Board A
        $ ping 192.168.0.11
```
### 3.7 Run SRA
The user space drivers from NXP/Freescale support usage of SRIO from linux user
space. The SRA application is a demo application from NXP/Freescale that can
implement writing from one SRIO interface to another, avioding kernel
interaction by using DMA (direct memory access) memory management. More
information on the drivers can be found in the NXP/freescale SDK [NXP-SDK].

In the test run below two boards are prepared with SRIO interfaces. Both boards
are initialized with a memory setting that sets the different SRIO memory
spaces to different values. Then data from Board A's *Write-prepapration space*
is written to Board B's *Map space*.

```
        # start the srio application
        $ sra

        # setup board B (receiver)
        sra> sra -attr port1 win_attr 1 nwrite nread

        # set local memory to data predefined in 0x100000
        sra> sra -op port1 1 0 0 s 0x100000

        # read to view what is written to port 1
        sra> sra -op port1 1 0 0 p 0x100000


        # setup board A (transmitter)
        sra> sra -attr port1 win_attr 1 nwrite nread

        # set local memory and then read to confirm
        sra> sra -op port1 1 0 0 s 0x100000
        sra> sra -op port1 1 0 0 p 0x100000

        # write what is in 'write preparing space' to outbound
        sra> sra -op port1 1 0 0 w 0x100000

        # confirm on board B that 'write preparation space' from board A is written in
        # 'map space'
        sra> sra -op port1 1 0 0 p 0x100000
```

```
          Board A                                 Board B
        +-------------------+                   +-------------------+
        | Map space         |           +------>| Map space         |
        +-------------------+           |       +-------------------+
        | Read data space   |           |       | Read data space   |
        +-------------------+  outbound |       +-------------------+
        | write preparation |-----------+       | write preparation |
        | space             |                   | space             |
        +-------------------+                   +-------------------+
        | reserved space    |                   | reserved space    |
        +-------------------+                   +-------------------+

Image 3.1 SRIO memory space
```

### 3.8 Throughput using USDPAA
To show some of the power of using the USDPAA framework a test of the
throughput over a 10G ethernet link was tested and the results are presented
below.

The tool used for performance measuring was Spirent Test Center which is used as
a packet generator along with the “Spirent Test Center” application, version
4.33. The test targets are connected to the Spirent Test Center through 10G
Ethernet ports(XAUI-RISER card). On Enea Linux 6.0 a the USDPAA reflector was
used as a packet forwarding application. The resulting Throughput performance measured with spirent can be seen in image 3.2 below.

<!--![alt text](./throughput_p2041rdb_networking.png =250x)-->
<img src="img/throughput_p2041rdb_networking.png" alt="Drawing" style="width: 500px;"/>

	Image 3.2 Throughput on p2041rdb over a 10G ethernet port using USDPAA.
	x-axis shows the frame size in bytes, and y-axis the aggregated throughput in
	megabits per second.



--------------
--------------
[NXP-SDK] QorIQ SDK 1.9 Documentation -
https://freescale.sdlproducts.com/LiveContent/web/pub.xql?c=t&action=home&pub=QorIQ_SDK_1.9&lang=en-US#addHistory=true&filename=GUID-81837065-81AD-449B-8572-E96C3EED636F.xml&docid=GUID-81837065-81AD-449B-8572-E96C3EED636F&inner_id=&tid=&query=&scope=&resource=&toc=false&eventType=lcContent.loadHome

[KERN-PARA] Kernel Parameters, https://www.kernel.org/doc/Documentation/kernel-parameters.txt

[RCU] Paul McKenney, *Priority-Boosting RCU Read-Side Critical Section*, https://lwn.net/Articles/220677/
<!--
[NOHZ] https://www.kernel.org/doc/Documentation/timers/NO_HZ.txt
-->

[HOTPLUG] https://www.kernel.org/doc/Documentation/cpu-hotplug.txt

[DPDK] http://dpdk.org/

[CYCLIC] https://rt.wiki.kernel.org/index.php/Cyclictest

[STRESS] http://linux.die.net/man/1/stresshttp://linux.die.net/man/1/stress

[CW] http://www.nxp.com/products/software-and-tools/software-development-tools/codewarrior-development-tools:CW_HOME