Enea NFV Access comes with two standard SDKs which can be used to develop user-applications and kernel modules for both host and guest images. A standard SDK consists of: Cross-Development Toolchain: cross-compiler and cross-debugger Libraries, Headers and Symbols that are specific to the image Environment Setup Script which defines the environment variables To install the SDK on your host development machine, there are two installation scripts available under the Download section on portal.enea.com. They have self explanatory names. inteld1521/sdk/enea-glibc-x86_64-enea-image-virtualization-host-sdkcorei7-64-toolchain-7.0.sh - for host applications. qemux86-64/sdk/enea-glibc-x86_64-enea-image-virtualization-guestsdk-core2-64-toolchain-7.0.sh - for guest applications. After installing the SDK, a developer will be able to compile and generate executables for the preferred target machine. Cross-gdb (x86_64-enea-linux-gdb) is created by the Cross-Development toolchain. It can be used to debug applications on the target platform from the development workstation. For kernel debugging, ftrace and kgdb are enabled on the host sdk image. Various user-space tools helpful in the development process, are also provided. The tools include LatencyTop, Perf, CrossTap, OProfile, Lttng-ust and GDBserver.

Before cross-compiling applications for your target, you need to install the corresponding toolchain on your workstation. To do that, simply run the installer and follow the steps included with it: $ ./enea-glibc-x86_64-enea-image-virtualization-guest-sdk-core2-64-toolchain-7.0.shWhen prompted, select to install the toolchain in the desired directory, referred to as sdkdir. A default path where the toolchain will be installed will be shown in the prompt. The installer unpacks the environment setup script in sdkdir and the toolchain under sdkdir/sysroots. Choose a unique directory for each toolchain. Installing a second toolchain of any type in the same directory as a previously installed one will break the $PATH variable of the first one. Setup the toolchain environment for your target by sourcing the environment-setup script. Example: $ source sdkdir/environment-setup-core2-64-enea-linux

Once the environment-setup script is sourced, you can make your applications as per usual and get them compiled for your target. Below you see how to cross-compile from command line. Create a Makefile for your application. Example: a simple Makefile and application: helloworld:helloworld.o $(CC) -o helloworld helloworld.o clean: rm -f *.o helloworld #include stdio.h; int main(void) { printf("Hello World\n"); return 0; } Run make to cross-compile your application according to the environment set up: $ make Deploy the helloworld program to your target and run it: # ./helloworld hello world

Before cross-compiling kernle modules, you need to make sure the installed toolchain includes the kernel source tree, which should be available at: sdkdir/sysroots/targetarch-enea-linux/usr/src/kernel. Once the environment-setup script is sourced, you can make your kernel modules as usual and get them compiled for your target. Below you see how to cross-compile a kernel module. Create a Makefile for the kernel module. Example: a simple Makefile and kernel module: obj-m := hello.o PWD := $(shell pwd) KERNEL_SRC := full path to kernel source tree all: scripts $(MAKE) -C $(KERNEL_SRC) M=$(PWD) LDFLAGS="" modules scripts: $(MAKE) -C $(KERNEL_SRC) scripts clean: $(MAKE) -C $(KERNEL_SRC) M=$(PWD) LDFLAGS="" clean #include linux/module.h /* Needed by all modules */ #include linux/kernel.h /* Needed for KERN_INFO */ #include linux/init.h /* Needed for the macros */ static int __init hello_start(void) { printk(KERN_INFO "Loading hello module...\n"); printk(KERN_INFO "Hello, world\n"); return 0; } static void __exit hello_end(void) { printk(KERN_INFO "Goodbye, world\n"); } module_init(hello_start); module_exit(hello_end); MODULE_LICENSE("GPL"); Run make to cross-compile your kernel module according to the environment set up: $ make Deploy the kernel module hello.ko to your target and install/remove it: # insmod hello.ko # rmmod hello.ko # dmesg [...] Loading hello module... [...] Hello, world [...] Goodbye, world

You can use ssh to deploy your artifacts on the host target. For this you will need a network connection to the target, to use scp to copy to the desired location. # scp helloworld root@<target_ip_address>:/tmp

You can deploy your artifacts onto the guest VM running on the target, by using TAP Networking. You can use the /etc/qemu-ifup script to create the tap interface on the host and attach it to the existing virtual bridge (virbr0). This bridge interface is created by the libvirt library and can be used to connect to the outside network. To be able to transfer files to the guest via scp, port forwarding should be enabled on the host. The script sets iptables rules to forward traffic from a host port to the guest default SSH port (22). Follow the steps below to create this setup: On the host, run the qemu-ifup script located in /etc directory: # /etc/qemu-ifup –t tap0 –a 192.168.122.10 –p 1050 –g 22 tap0 - the tap interface name which will be created and added to the virtual bridge (virbr0). 192.168.122.10 - the IP address of the guest virtual network device. It has to be in the same network with the IP address of the virbr0 interface. 1050 - the host port which is set to forward traffic from the host to the guest. 22 - the default SSH guest port used in port forwarding. Launch the virtual machine specifying the newly created tap interface: -device e1000,netdev=net0 \ -netdev tap,id=net0,ifname=tap0,script=no,downscript=no On the guest, after logging, configure the virtual network device and set the default gateway as the virbr0 ip address: ip addr add 192.168.122.10/24 dev enp0s2 ip link set enp0s2 up ip route add default via 192.168.122.1 dev enp0s2 Now you can use scp from your development machine to deploy your artifacts on the guest, by giving the host port for forwarding as a command parameter: scp -P 1050 helloworld root@target_ip_address:/tmp On the host, after finishing the deployment session and stopping the virtual machine, you can use the qemu-ifdown script to clean up the configuration on host. The following command will remove the tap interface and all the iptables rules for the specific ip address: # /etc/qemu-ifdown –t tap0 –a 192.168.122.10If we need to remove only a particular port forwarding rule from iptables, this should be run: # /etc/qemu-ifdown –t tap0 –a 192.168.122.10 –p 1050 –g 22

The cross-debugger (x86_64-enea-linux-gdb) is created when installing the SDK on the development machine. It is helpful for debugging both the kernel and user-applications. In order to perform this task, we need the following tools to be available on the target machine: Kgdb – for kernel cross-debugging GDBServer – for application cross-debugging The Host Development image provides both of these tools and has to be booted on the target machine for cross-debugging sessions.

To debug a user-application on host, a TCP connection has to be established between the host and development machines. GDBserver is the program which runs on the target machine and allows you to run GDB on your workstation. Below you can find how a simple helloworld application can be debugged using GDBServer and cross-gdb. On target, launch the GDBServer, specifying how to communicate with GDB and the name of your program:# gdbserver :<port_no> /tmp/helloworldThe target will now be listening on the port given as a parameter to the gdbserver. On the development machine, from the <sdkdir>, start the cross-gdb:# x86_64-enea-linux-gdb <path_to_the_program>/helloworldConnect the GDB to the target: (gdb) target remote <target_ip_address>:<port_no>Now remote debugging is started and the GDB commands are available to debug your program from the target machine.

To debug a user-application on guest, a TCP connection has to be established between the host and development machines. Similarly, as when deploying artifacts on guest, for a cross-debugging session, TAP Networking is required. A tap interface should be added to the existing virtual bridge (virbr0), along with port forwarding rules created in iptables. In order to do this, the script /etc/qemu-ifup can pe used: On the host, run the script qemu-ifup located in /etc directory: # /etc/qemu-ifup –t tap0 –a 192.168.122.10 –p 1051 –g 1025 tap0 - the tap interface name which will be created and added to virtual bridge (virbr0). 192.168.122.10 - the IP address of the guest virtual network device. It has to be in the same network with the IP address of the virbr0 interface. 1051 – the host port which is set to forward traffic from the host to the guest and is used by gdb target remote command. 1025– the port used by GDBServer on guest for listening. Launch the virtual machine, specifying the newly created tap interface:-device e1000,netdev=net0 \ -netdev tap,id=net0,ifname=tap0,script=no,downscript=no On the guest, after logging, configure the virtual network device and set the default gateway to virbr0 ip address:ip addr add 192.168.122.10/24 dev enp0s2 ip link set enp0s2 up ip route add default via 192.168.122.1 dev enp0s2 GDBserver is the program which runs on the guest VM and allows you to run GDB on your workstation. On the guest, launch GBDserver specifying how to communicate with GDB and the name of your program: # gdbserver :1025 /tmp/helloworldThe guest is now listening on port 1025, given as a parameter to the gdbserver. On the development machine, from the <sdkdir>, start the cross-gdb:# x86_64-enea-linux-gdb <path_to_the_program>/helloworld Connect GDB to the target:(gdb) target remote <target_ip_address>:1051Now remote debugging is started and the GDB commands are available to debug your program from the guest VM. On the host, after finishing the cross-debugging session and stopping the virtual machine, you can use the qemu-ifdown script to clean up the configuration on host:# /etc/qemu-ifdown -t tap0 -a 192.168.122.10

In order to debug the kernel, a serial connection is required between the development and target machines. Debugging commands will be sent from your workstation to the target machine via a serial port. The KGDB kernel options are enabled in the Enea NFV Access Host SDK image and the tool can be used in the following way: On target, once serial communication is established, configure kgdboc after the kernel boots:# echo ttyS0,115200 > /sys/module/kgdboc/parameters/kgdboc In order to connect to gdb via kgdboc, the kernel must first be stopped:# echo g > /proc/sysrq-trigger Close the console to the target, eg.: Ctrl + ] for a telnet session. On your development machine, start cross-gdb using the vmlinux kernel image as a parameter. The image is located in <sdkdir>/sysroots/corei7-64-enea-linux/boot/ and should be the same as the image found in the /boot directory from the target.# x86_64-enea-linux-gdb / ./sysroots/corei7-64-enea-linux/boot/vmlinux-4.9.30-intel-pk-standard Connect GDB to the target machine using target command and the serial device:(gdb) set remotebaud 115200 (gdb) target remote /dev/ttyS0 The kernel can now be debugged in a similar manner as an application program.