path: root/doc/book-enea-nfv-access-guide/doc/using_nfv_access_sdks.xml

<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
<chapter id="workflow">
  <title>Using NFV Access SDKs</title>

  <para>Enea NFV Access comes with two standard SDKs which can be used to
  develop user-applications and kernel modules for both host and guest
  images.</para>

  <para>A standard SDK consists of:</para>

  <itemizedlist>
    <listitem>
      <para>Cross-Development Toolchain: cross-compiler and
      cross-debugger</para>
    </listitem>

    <listitem>
      <para>Libraries, Headers and Symbols that are specific to the
      image</para>
    </listitem>

    <listitem>
      <para>Environment Setup Script which defines the environment
      variables</para>
    </listitem>
  </itemizedlist>

  <para>To install the SDK on your host development machine, there are two
  installation scripts available under the Download section on <emphasis
  role="bold">portal.enea.com</emphasis>. They have self explanatory
  names.</para>

  <itemizedlist>
    <listitem>
      <para><literal>inteld1521/sdk/enea-glibc-x86_64-enea-image-virtualization-host-sdkcorei7-64-toolchain-7.0.sh</literal>
      - for host applications.</para>
    </listitem>

    <listitem>
      <para><literal>qemux86-64/sdk/enea-glibc-x86_64-enea-image-virtualization-guestsdk-core2-64-toolchain-7.0.sh</literal>
      - for guest applications.</para>
    </listitem>
  </itemizedlist>

  <para>After installing the SDK, a developer will be able to compile and
  generate executables for the preferred target machine. Cross-gdb
  (<filename>x86_64-enea-linux-gdb</filename>) is created by the
  Cross-Development toolchain. It can be used to debug applications on the
  target platform from the development workstation. For kernel debugging,
  <command>ftrace</command> and <command>kgdb</command> are enabled on the
  host sdk image.</para>

  <para>Various user-space tools helpful in the development process, are also
  provided. The tools include <emphasis role="bold">LatencyTop</emphasis>,
  <emphasis role="bold">Perf</emphasis>, <emphasis
  role="bold">CrossTap</emphasis>, <emphasis role="bold">OProfile</emphasis>,
  <emphasis role="bold">Lttng-ust</emphasis> and <emphasis
  role="bold">GDBserver</emphasis>.</para>

  <section id="install-crosscomp">
    <title>Installing the Cross-Compilation Toolchain</title>

    <para>Before cross-compiling applications for your target, you need to
    install the corresponding toolchain on your workstation. To do that,
    simply run the installer and follow the steps included with it:</para>

    <orderedlist>
      <listitem>
        <para><programlisting>$ ./enea-glibc-x86_64-enea-image-virtualization-guest-sdk-core2-64-toolchain-7.0.sh</programlisting>When
        prompted, select to install the toolchain in the desired directory,
        referred to as <literal>sdkdir</literal>.</para>

        <para>A default path where the toolchain will be installed will be
        shown in the prompt. The installer unpacks the environment setup
        script in <literal>sdkdir</literal> and the toolchain under
        <literal>sdkdir/sysroots</literal>.</para>

        <note>
          <para>Choose a unique directory for each toolchain. Installing a
          second toolchain of any type in the same directory as a previously
          installed one will break the <literal>$PATH</literal> variable of
          the first one.</para>
        </note>
      </listitem>

      <listitem>
        <para>Setup the toolchain environment for your target by sourcing the
        environment-setup script. Example: <programlisting>$ source sdkdir/environment-setup-core2-64-enea-linux</programlisting></para>
      </listitem>
    </orderedlist>
  </section>

  <section id="crosscomp-apps">
    <title>Cross-Compiling Applications from Command Line</title>

    <para>Once the environment-setup script is sourced, you can make your
    applications as per usual and get them compiled for your target. Below you
    see how to cross-compile from command line.</para>

    <orderedlist>
      <listitem>
        <para>Create a Makefile for your application. Example: a simple
        Makefile and application:</para>

        <programlisting>helloworld:helloworld.o
        $(CC) -o helloworld helloworld.o
clean:
        rm -f *.o helloworld
#include stdio.h;
int main(void) {
        printf("Hello World\n");
        return 0;
}</programlisting>
      </listitem>

      <listitem>
        <para>Run <command>make</command> to cross-compile your application
        according to the environment set up:</para>

        <programlisting>$ make</programlisting>
      </listitem>

      <listitem>
        <para>Deploy the helloworld program to your target and run it:</para>

        <programlisting># ./helloworld
hello world</programlisting>
      </listitem>
    </orderedlist>
  </section>

  <section id="crosscomp-kern-mod">
    <title>Cross-Compiling Kernel Modules</title>

    <para>Before cross-compiling kernle modules, you need to make sure the
    installed toolchain includes the kernel source tree, which should be
    available at:
    <literal>sdkdir/sysroots/targetarch-enea-linux/usr/src/kernel</literal>.</para>

    <para>Once the environment-setup script is sourced, you can make your
    kernel modules as usual and get them compiled for your target. Below you
    see how to cross-compile a kernel module.</para>

    <orderedlist>
      <listitem>
        <para>Create a Makefile for the kernel module. Example: a simple
        Makefile and kernel module:</para>

        <programlisting>obj-m := hello.o
PWD := $(shell pwd)

KERNEL_SRC := full path to kernel source tree

all: scripts
    $(MAKE) -C $(KERNEL_SRC) M=$(PWD) LDFLAGS="" modules
scripts:
    $(MAKE) -C $(KERNEL_SRC) scripts
clean:
    $(MAKE) -C $(KERNEL_SRC) M=$(PWD) LDFLAGS="" clean
#include linux/module.h     /* Needed by all modules */
#include linux/kernel.h     /* Needed for KERN_INFO */
#include linux/init.h       /* Needed for the macros */

static int __init hello_start(void)
{
    printk(KERN_INFO "Loading hello module...\n");
    printk(KERN_INFO "Hello, world\n");
    return 0;
}

static void __exit hello_end(void)
{
    printk(KERN_INFO "Goodbye, world\n");
}

module_init(hello_start);
module_exit(hello_end);

MODULE_LICENSE("GPL");</programlisting>
      </listitem>

      <listitem>
        <para>Run <command>make</command> to cross-compile your kernel module
        according to the environment set up:</para>

        <programlisting>$ make</programlisting>
      </listitem>

      <listitem>
        <para>Deploy the kernel module <literal>hello.ko</literal> to your
        target and install/remove it:</para>

        <programlisting># insmod hello.ko
# rmmod hello.ko
# dmesg
[...] Loading hello module...
[...] Hello, world
[...] Goodbye, world</programlisting>
      </listitem>
    </orderedlist>
  </section>

  <section id="deploy-artifacts">
    <title>Deploying your artifacts</title>

    <section id="deploy_onhost">
      <title>Deploying on host</title>

      <para>You can use <literal>ssh</literal> to deploy your artifacts on the
      host target. For this you will need a network connection to the target,
      to use <literal>scp</literal> to copy to the desired location.</para>

      <para><programlisting># scp helloworld root@&lt;target_ip_address&gt;:/tmp</programlisting></para>
    </section>

    <section id="deploy_onguest">
      <title>Deploying on guest</title>

      <para>You can deploy your artifacts onto the guest VM running on the
      target, by using TAP Networking. You can use the
      <filename>/etc/qemu-ifup</filename> script to create the tap interface
      on the host and attach it to the existing virtual bridge
      (<literal>virbr0</literal>). This bridge interface is created by the
      <filename>libvirt</filename> library and can be used to connect to the
      outside network. To be able to transfer files to the guest via
      <literal>scp</literal>, port forwarding should be enabled on the host.
      The script sets iptables rules to forward traffic from a host port to
      the guest default SSH port (22).</para>

      <para>Follow the steps below to create this setup:</para>

      <orderedlist>
        <listitem>
          <para>On the host, run the <literal>qemu-ifup</literal> script
          located in <literal>/etc</literal> directory:</para>

          <para><programlisting># /etc/qemu-ifup &ndash;t tap0 &ndash;a 192.168.122.10 &ndash;p 1050 &ndash;g 22</programlisting></para>

          <itemizedlist>
            <listitem>
              <para><emphasis role="bold">tap0</emphasis> - the tap interface
              name which will be created and added to the virtual bridge
              (virbr0).</para>
            </listitem>

            <listitem>
              <para><emphasis role="bold">192.168.122.10</emphasis> - the IP
              address of the guest virtual network device. It has to be in the
              same network with the IP address of the virbr0 interface.</para>
            </listitem>

            <listitem>
              <para><emphasis role="bold">1050</emphasis> - the host port
              which is set to forward traffic from the host to the
              guest.</para>
            </listitem>

            <listitem>
              <para><emphasis role="bold">22</emphasis> - the default SSH
              guest port used in port forwarding.</para>
            </listitem>
          </itemizedlist>
        </listitem>

        <listitem>
          <para>Launch the virtual machine specifying the newly created tap
          interface:</para>

          <para><programlisting>-device e1000,netdev=net0 \
-netdev tap,id=net0,ifname=tap0,script=no,downscript=no</programlisting></para>
        </listitem>

        <listitem>
          <para>On the guest, after logging, configure the virtual network
          device and set the default gateway as the <literal>virbr0</literal>
          ip address:</para>

          <para><programlisting>ip addr add 192.168.122.10/24 dev enp0s2
ip link set enp0s2 up
ip route add default via 192.168.122.1 dev enp0s2</programlisting></para>
        </listitem>

        <listitem>
          <para>Now you can use <literal>scp</literal> from your development
          machine to deploy your artifacts on the guest, by giving the host
          port for forwarding as a command parameter: <programlisting>scp -P 1050 helloworld root@target_ip_address:/tmp</programlisting></para>
        </listitem>

        <listitem>
          <para>On the host, after finishing the deployment session and
          stopping the virtual machine, you can use the
          <literal>qemu-ifdown</literal> script to clean up the configuration
          on host. The following command will remove the tap interface and all
          the iptables rules for the specific ip address: <programlisting># /etc/qemu-ifdown &ndash;t tap0 &ndash;a 192.168.122.10</programlisting>If
          we need to remove only a particular port forwarding rule from
          iptables, this should be run: <programlisting># /etc/qemu-ifdown &ndash;t tap0 &ndash;a 192.168.122.10 &ndash;p 1050 &ndash;g 22</programlisting></para>
        </listitem>
      </orderedlist>
    </section>
  </section>

  <section id="crossdebugging">
    <title>Cross-Debugging on Enea NFV Access</title>

    <para>The cross-debugger (<literal>x86_64-enea-linux-gdb</literal>) is
    created when installing the SDK on the development machine. It is helpful
    for debugging both the kernel and user-applications. In order to perform
    this task, we need the following tools to be available on the target
    machine:</para>

    <itemizedlist>
      <listitem>
        <para>Kgdb &ndash; for kernel cross-debugging</para>
      </listitem>

      <listitem>
        <para>GDBServer &ndash; for application cross-debugging</para>
      </listitem>
    </itemizedlist>

    <para>The Host Development image provides both of these tools and has to
    be booted on the target machine for cross-debugging sessions.</para>

    <section id="ua_debug_host">
      <title>User-application Cross-Debugging on Host</title>

      <para>To debug a user-application on host, a TCP connection has to be
      established between the host and development machines. GDBserver is the
      program which runs on the target machine and allows you to run GDB on
      your workstation. Below you can find how a simple
      <filename>helloworld</filename> application can be debugged using
      GDBServer and cross-gdb.</para>

      <para>On target, launch the GDBServer, specifying how to communicate
      with GDB and the name of your program:<programlisting># gdbserver :&lt;port_no&gt; /tmp/helloworld</programlisting>The
      target will now be listening on the port given as a parameter to the
      gdbserver. On the development machine, from the
      <literal>&lt;sdkdir&gt;</literal>, start the cross-gdb:<programlisting># x86_64-enea-linux-gdb &lt;path_to_the_program&gt;/helloworld</programlisting>Connect
      the GDB to the target: <programlisting>(gdb) target remote &lt;target_ip_address&gt;:&lt;port_no&gt;</programlisting>Now
      remote debugging is started and the GDB commands are available to debug
      your program from the target machine.</para>
    </section>

    <section id="us_debug_guest">
      <title>User-application Cross-Debugging on Guest</title>

      <para>To debug a user-application on guest, a TCP connection has to be
      established between the host and development machines. Similarly, as
      when deploying artifacts on guest, for a cross-debugging session, TAP
      Networking is required.</para>

      <para>A tap interface should be added to the existing virtual bridge
      (<literal>virbr0</literal>), along with port forwarding rules created in
      iptables. In order to do this, the script
      <filename>/etc/qemu-ifup</filename> can pe used:</para>

      <orderedlist>
        <listitem>
          <para>On the host, run the script <literal>qemu-ifup</literal>
          located in <literal>/etc</literal> directory: <programlisting># /etc/qemu-ifup &ndash;t tap0 &ndash;a 192.168.122.10 &ndash;p 1051 &ndash;g 1025</programlisting></para>

          <itemizedlist>
            <listitem>
              <para><emphasis role="bold">tap0</emphasis> - the tap interface
              name which will be created and added to virtual bridge
              (virbr0).</para>
            </listitem>

            <listitem>
              <para><emphasis role="bold">192.168.122.10</emphasis> - the IP
              address of the guest virtual network device. It has to be in the
              same network with the IP address of the virbr0 interface.</para>
            </listitem>

            <listitem>
              <para><emphasis role="bold">1051</emphasis> &ndash; the host
              port which is set to forward traffic from the host to the guest
              and is used by gdb target remote command.</para>
            </listitem>

            <listitem>
              <para><emphasis role="bold">1025</emphasis>&ndash; the port used
              by GDBServer on guest for listening.</para>
            </listitem>
          </itemizedlist>
        </listitem>

        <listitem>
          <para>Launch the virtual machine, specifying the newly created tap
          interface:<programlisting>-device e1000,netdev=net0 \
-netdev tap,id=net0,ifname=tap0,script=no,downscript=no</programlisting></para>
        </listitem>

        <listitem>
          <para>On the guest, after logging, configure the virtual network
          device and set the default gateway to virbr0 ip
          address:<programlisting>ip addr add 192.168.122.10/24 dev enp0s2
ip link set enp0s2 up
ip route add default via 192.168.122.1 dev enp0s2</programlisting></para>
        </listitem>

        <listitem>
          <para>GDBserver is the program which runs on the guest VM and allows
          you to run GDB on your workstation. On the guest, launch GBDserver
          specifying how to communicate with GDB and the name of your program:
          <programlisting># gdbserver :1025 /tmp/helloworld</programlisting>The
          guest is now listening on port 1025, given as a parameter to the
          gdbserver.</para>
        </listitem>

        <listitem>
          <para>On the development machine, from the<filename>
          &lt;sdkdir&gt;</filename>, start the cross-gdb:<programlisting># x86_64-enea-linux-gdb &lt;path_to_the_program&gt;/helloworld</programlisting></para>
        </listitem>

        <listitem>
          <para>Connect GDB to the target:<programlisting>(gdb) target remote &lt;target_ip_address&gt;:1051</programlisting>Now
          remote debugging is started and the GDB commands are available to
          debug your program from the guest VM.</para>
        </listitem>

        <listitem>
          <para>On the host, after finishing the cross-debugging session and
          stopping the virtual machine, you can use the
          <filename>qemu-ifdown</filename> script to clean up the
          configuration on host:<programlisting># /etc/qemu-ifdown -t tap0 -a 192.168.122.10</programlisting></para>
        </listitem>
      </orderedlist>
    </section>

    <section id="kernel_crossdebug">
      <title>Kernel Cross-Debugging</title>

      <para>In order to debug the kernel, a serial connection is required
      between the development and target machines. Debugging commands will be
      sent from your workstation to the target machine via a serial
      port.</para>

      <para>The KGDB kernel options are enabled in the Enea NFV Access Host
      SDK image and the tool can be used in the following way:</para>

      <itemizedlist>
        <listitem>
          <para>On target, once serial communication is established, configure
          <literal>kgdboc</literal> after the kernel boots:<programlisting># echo ttyS0,115200 &gt; /sys/module/kgdboc/parameters/kgdboc</programlisting></para>
        </listitem>

        <listitem>
          <para>In order to connect to gdb via kgdboc, the kernel must first
          be stopped:<programlisting># echo g &gt; /proc/sysrq-trigger</programlisting></para>
        </listitem>

        <listitem>
          <para>Close the console to the target, eg.: <command>Ctrl +
          ]</command> for a telnet session.</para>
        </listitem>

        <listitem>
          <para>On your development machine, start cross-gdb using the vmlinux
          kernel image as a parameter. The image is located in
          <filename>&lt;sdkdir&gt;/sysroots/corei7-64-enea-linux/boot/</filename>
          and should be the same as the image found in the
          <literal>/boot</literal> directory from the target.<programlisting># x86_64-enea-linux-gdb /
./sysroots/corei7-64-enea-linux/boot/vmlinux-4.9.30-intel-pk-standard</programlisting></para>
        </listitem>

        <listitem>
          <para>Connect GDB to the target machine using target command and the
          serial device:<programlisting>(gdb) set remotebaud 115200
(gdb) target remote /dev/ttyS0</programlisting></para>
        </listitem>
      </itemizedlist>

      <para>The kernel can now be debugged in a similar manner as an
      application program.</para>
    </section>
  </section>
</chapter>