doc: Updated ARM Hypervisor Virtualization chapter

Signed-off-by: Dragos Motrea <Dragos.Motrea@enea.com>

1 files changed, 124 insertions, 105 deletions

diff --git a/doc/book-enea-nfv-access-guide/doc/hypervisor_virtualization.xml b/doc/book-enea-nfv-access-guide/doc/hypervisor_virtualization.xml
index 9733544..76a2568 100644
--- a/doc/book-enea-nfv-access-guide/doc/hypervisor_virtualization.xml
+++ b/doc/book-enea-nfv-access-guide/doc/hypervisor_virtualization.xml
@@ -18,23 +18,23 @@
 
     <para>QEMU can make use of KVM when running a target architecture that is
     the same as the host architecture. For instance, when running
-    qemu-system-x86_64 on an x86-64 compatible processor (containing
-    virtualization extensions Intel VT or AMD-V), you can take advantage of
+    qemu-system-aarch64 on an aarch64 compatible processor (containing
+    virtualization extensions AMD-V or Intel VT), you can take advantage of
     the KVM acceleration, giving you benefit for your host and your guest
     system.</para>
 
-    <para>Enea NFV Access includes an optimizied version of QEMU with KVM-only
+    <para>Enea NFV Access includes an optimized version of QEMU with KVM-only
     support. To use KVM pass<command> --enable-kvm</command> to QEMU.</para>
 
     <para>The following is an example of starting a guest:</para>
 
-    <programlisting>taskset -c 0,1 qemu-system-x86_64 \
--cpu host -M q35 -smp cores=2,sockets=1 \
+    <programlisting>taskset -c 0,1 qemu-system-aarch64 \
+-cpu host -machine virt,gic_version=3 -smp cores=2,sockets=1 \
 -vcpu 0,affinity=0 -vcpu 1,affinity=1  \
 -enable-kvm -nographic \
--kernel bzImage \
--drive file=enea-nfv-access-guest-qemux86-64.ext4,if=virtio,format=raw \
--append 'root=/dev/vda console=ttyS0,115200' \
+-kernel Image \
+-drive file=enea-nfv-access-guest-qemuarm64.ext4,if=virtio,format=raw \
+-append 'root=/dev/vda console=ttyAMA0,115200' \
 -m 4096 \
 -object memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on \
 -numa node,memdev=mem -mem-prealloc</programlisting>
@@ -97,9 +97,7 @@
 
     <para>If the hardware does not have an IOMMU (known as "Intel VT-d" on
     Intel-based machines and "AMD I/O Virtualization Technology" on AMD-based
-    machines), it will not be possible to assign devices in KVM.
-    Virtualization Technology features (VT-d, VT-x, etc.) must be enabled from
-    BIOS on the host target before starting a virtual machine.</para>
+    machines), it will not be possible to assign devices in KVM. </para>
   </section>
 
   <section id="net_in_guest">
@@ -156,19 +154,10 @@
       to appear and behave as if they were physically attached to the guest
       operating system.</para>
 
-      <para>Preparing an Intel system for PCI passthrough:</para>
+      <para>Preparing a system for PCI passthrough:</para>
 
       <itemizedlist>
         <listitem>
-          <para>Enable the Intel VT-d extensions in BIOS</para>
-        </listitem>
-
-        <listitem>
-          <para>Activate Intel VT-d in the kernel by using
-          <literal>intel_iommu=on</literal> as a kernel boot parameter</para>
-        </listitem>
-
-        <listitem>
           <para>Allow unsafe interrupts in case the system doesn't support
           interrupt remapping. This can be done using
           <literal>vfio_iommu_type1.allow_unsafe_interrupts=1</literal> as a
@@ -179,15 +168,17 @@
       <para>Create guest with direct passthrough via VFIO framework like
       so:</para>
 
-      <programlisting>-device vfio-pci,host=0000:03:10.2 \</programlisting>
+      <programlisting>-device vfio-pci,host=0001:01:00.1 \</programlisting>
+
+      <para>On the host, there are Virtual Functions for each physical
+      Ethernet port. The primary VF has to be used for a guest network to
+      access, before starting QEMU:</para>
 
-      <para>On the host, one or more VirtualFunctions (VFs) must be created in
-      order to be allocated for a guest network to access, before starting
-      QEMU:</para>
+      <programlisting>$ modprobe vfio_pci
+$ dpdk-devbind.py --bind=vfio-pci 0001:01:00.1</programlisting>
 
-      <programlisting>$ echo 2 &gt; /sys/class/net/eno3/device/sriov_numvfs
-$ modprobe vfio_pci
-$ dpdk-devbind.py --bind=vfio-pci 0000:03:10.2</programlisting>
+      <para>For more details related to Virtual Functions, see: <ulink
+      url="http://dpdk.org/doc/guides/nics/thunderx.html">http://dpdk.org/doc/guides/nics/thunderx.html</ulink>.</para>
     </section>
 
     <section id="multiqueue">
@@ -215,7 +206,7 @@ where vectors is calculated as: 2 + 2 * queues number.</programlisting>
 DPDK Virtio PMD
 $ testpmd -c 0x7 -- -i --rxq=2 --txq=2 --nb-cores=2 ...</programlisting>
 
-        <para>For QEMU documentation please see: <ulink
+        <para>For QEMU documentation, see: <ulink
         url="https://qemu.weilnetz.de/doc/qemu-doc.html">https://qemu.weilnetz.de/doc/qemu-doc.html</ulink>.</para>
       </section>
     </section>
@@ -340,12 +331,12 @@ $ testpmd -c 0x7 -- -i --rxq=2 --txq=2 --nb-cores=2 ...</programlisting>
       following example:</para>
 
       <programlisting>&lt;os&gt;
-  &lt;kernel&gt;bzImage&lt;/kernel&gt;
+  &lt;kernel&gt;Image&lt;/kernel&gt;
 &lt;/os&gt;
 &lt;devices&gt;
   &lt;disk type='file' device='disk'&gt;
     &lt;driver name='qemu' type='raw' cache='none'/&gt;
-    &lt;source file='enea-nfv-access-guest-qemux86-64.ext4'/&gt;
+    &lt;source file='enea-nfv-access-guest-qemuarm64.ext4'/&gt;
     &lt;target dev='vda' bus='virtio'/&gt;
   &lt;/disk&gt;
 &lt;/devices&gt;</programlisting>
@@ -356,7 +347,7 @@ $ testpmd -c 0x7 -- -i --rxq=2 --txq=2 --nb-cores=2 ...</programlisting>
 
       <para>Command <command>virsh create</command> starts a guest:</para>
 
-      <programlisting>virsh create example-guest-x86.xml</programlisting>
+      <programlisting>virsh create example-guest-arm.xml</programlisting>
 
       <para>If further configurations are needed before the guest is reachable
       through <literal>ssh</literal>, a console can be started using command
@@ -370,7 +361,7 @@ $ testpmd -c 0x7 -- -i --rxq=2 --txq=2 --nb-cores=2 ...</programlisting>
       the guest XML file:</para>
 
       <programlisting>&lt;os&gt;
-  &lt;cmdline&gt;console=ttyS0,115200&lt;/cmdline&gt;
+  &lt;cmdline&gt;console=ttyAMA0,115200&lt;/cmdline&gt;
 &lt;/os&gt;
 &lt;devices&gt;
   &lt;console type='pty'&gt;
@@ -477,14 +468,14 @@ $ testpmd -c 0x7 -- -i --rxq=2 --txq=2 --nb-cores=2 ...</programlisting>
           now, only bridged networks are supported (those where the user has
           to manually create the bridge).</para>
 
-          <programlisting>    &lt;interface type='vhostuser'&gt;
-      &lt;mac address='00:00:00:00:00:01'/&gt;
-      &lt;source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/&gt;
-       &lt;model type='virtio'/&gt;
-      &lt;driver queues='1'&gt;
-        &lt;host mrg_rxbuf='off'/&gt;
-      &lt;/driver&gt;
-    &lt;/interface&gt;</programlisting>
+          <programlisting>    &lt;qemu:commandline&gt;
+      &lt;qemu:arg value='-chardev'/&gt;
+      &lt;qemu:arg value='socket,id=charnet0,path=/var/run/openvswitch/vhost-user1'/&gt;
+      &lt;qemu:arg value='-netdev'/&gt;
+      &lt;qemu:arg value='type=vhost-user,id=hostnet0,chardev=charnet0'/&gt;
+      &lt;qemu:arg value='-device'/&gt;
+      &lt;qemu:arg value='virtio-net-pci,netdev=hostnet0,id=net0,mac=00:00:00:00:00:01,bus=pcie.0,addr=0x2'/&gt;
+    &lt;/qemu:commandline&gt;</programlisting>
         </listitem>
 
         <listitem>
@@ -497,40 +488,68 @@ $ testpmd -c 0x7 -- -i --rxq=2 --txq=2 --nb-cores=2 ...</programlisting>
           PCI devices to appear and behave as if they were physically attached
           to the guest operating system.</para>
 
-          <para>Preparing an Intel system for PCI passthrough is done like
-          so:</para>
+          <para>Preparing a system for PCI passthrough is done like so:</para>
 
           <itemizedlist>
             <listitem>
-              <para>Enable the Intel VT-d extensions in BIOS</para>
+              <para>Allow unsafe interrupts in case the system doesn't support
+              interrupt remapping. This can be done using
+              <literal>vfio_iommu_type1.allow_unsafe_interrupts=1</literal> as
+              a boot kernel parameter.</para>
             </listitem>
 
             <listitem>
-              <para>Activate Intel VT-d in the kernel by using
-              <literal>intel_iommu=on</literal> as a kernel boot
-              parameter</para>
+              <para>Change the owner of the
+              <literal>/dev/vfio/&lt;group&gt;</literal> to qemu and edit
+              <literal>/etc/libvirt/qemu.conf</literal> to explicitly allow
+              permission to it:</para>
+
+              <para><programlisting>$ chown qemu:qemu /dev/vfio/66
+$ ls -l /dev/vfio/66
+crw------- 1 qemu qemu 244, 0 Dec  4 08:24 /dev/vfio/66
+
+$ cat /etc/libvirt/qemu.conf
+...
+cgroup_device_acl = [
+    "/dev/null", "/dev/full", "/dev/zero",
+    "/dev/random", "/dev/urandom",
+    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
+    "/dev/rtc","/dev/hpet", "/dev/vfio/vfio",
+    <command>"/dev/vfio/66"</command>,
+]
+...</programlisting></para>
             </listitem>
 
             <listitem>
-              <para>Allow unsafe interrupts in case the system doesn't support
-              interrupt remapping. This can be done using
-              <literal>vfio_iommu_type1.allow_unsafe_interrupts=1</literal> as
-              a boot kernel parameter.</para>
+              <para>Increase the locked memory limits within the libvirtd
+              service file :</para>
+
+              <para><programlisting>$ cat /lib/systemd/system/libvirtd.service
+...
+[Service]
+Type=forking
+PIDFile=/var/run/libvirtd.pid
+EnvironmentFile=-/etc/sysconfig/libvirtd
+<command>LimitMEMLOCK=infinity</command>
+ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS
+ExecReload=/bin/kill -HUP $MAINPID
+KillMode=process
+Restart=on-failure
+# Override the maximum number of opened files
+#LimitNOFILE=2048
+...</programlisting></para>
             </listitem>
           </itemizedlist>
 
           <para>VFs must be created on the host before starting the
           guest:</para>
 
-          <programlisting>$ echo 2 &gt; /sys/class/net/eno3/device/sriov_numvfs
-$ modprobe vfio_pci
-$ dpdk-devbind.py --bind=vfio-pci 0000:03:10.0
-   &lt;interface type='hostdev' managed='yes'&gt;
-     &lt;source&gt;
-       &lt;address type='pci' domain='0x0' bus='0x03' slot='0x10' function='0x0'/&gt;
-     &lt;/source&gt;
-     &lt;mac address='52:54:00:6d:90:02'/&gt;
-   &lt;/interface&gt;</programlisting>
+          <programlisting>$ modprobe vfio_pci
+$ dpdk-devbind.py --bind=vfio-pci 0001:01:00.1
+   &lt;qemu:commandline&gt;
+     &lt;qemu:arg value='-device'/&gt;
+     &lt;qemu:arg value='vfio-pci,host=0001:01:00.1'/&gt;
+   &lt;/qemu:commandline&gt;</programlisting>
         </listitem>
 
         <listitem>
@@ -542,8 +561,8 @@ $ dpdk-devbind.py --bind=vfio-pci 0000:03:10.0
           <programlisting>    &lt;interface type='bridge'&gt;
       &lt;mac address='52:54:00:71:b1:b6'/&gt;
       &lt;source bridge='ovsbr0'/&gt;
+      &lt;model type='rtl8139'/&gt;
       &lt;virtualport type='openvswitch'/&gt;
-      &lt;address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/&gt;
     &lt;/interface&gt;</programlisting>
 
           <para>For further details on the network XML format, see <ulink
@@ -558,58 +577,48 @@ $ dpdk-devbind.py --bind=vfio-pci 0000:03:10.0
       <section id="guest-config-vhost-user-interface">
         <title>Guest configuration with vhost-user interface</title>
 
-        <programlisting>&lt;domain type='kvm'&gt;
+        <programlisting>&lt;domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'&gt;
   &lt;name&gt;vm_vhost&lt;/name&gt;
-  &lt;uuid&gt;4a9b3f53-fa2a-47f3-a757-dd87720d9d1d&lt;/uuid&gt;
+  &lt;uuid&gt;ed204646-1ad5-11e7-93ae-92361f002671&lt;/uuid&gt;
   &lt;memory unit='KiB'&gt;4194304&lt;/memory&gt;
   &lt;currentMemory unit='KiB'&gt;4194304&lt;/currentMemory&gt;
   &lt;memoryBacking&gt;
     &lt;hugepages&gt;
-      &lt;page size='1' unit='G' nodeset='0'/&gt;
+      &lt;page size='512' unit='M' nodeset='0'/&gt;
     &lt;/hugepages&gt;
   &lt;/memoryBacking&gt;
-  &lt;vcpu placement='static'&gt;2&lt;/vcpu&gt;
-  &lt;cputune&gt;
-    &lt;shares&gt;4096&lt;/shares&gt;
-    &lt;vcpupin vcpu='0' cpuset='4'/&gt;
-    &lt;vcpupin vcpu='1' cpuset='5'/&gt;
-    &lt;emulatorpin cpuset='4,5'/&gt;
-  &lt;/cputune&gt;
   &lt;os&gt;
-    &lt;type arch='x86_64' machine='pc'&gt;hvm&lt;/type&gt;
-    &lt;kernel&gt;/mnt/qemu/bzImage&lt;/kernel&gt;
-    &lt;cmdline&gt;root=/dev/vda console=ttyS0,115200&lt;/cmdline&gt;
+    &lt;type arch='aarch64' machine='virt,gic_version=3'&gt;hvm&lt;/type&gt;
+    &lt;kernel&gt;/mnt/qemu/Image&lt;/kernel&gt;
+    &lt;cmdline&gt;root=/dev/vda console=ttyAMA0,115200n8&lt;/cmdline&gt;
     &lt;boot dev='hd'/&gt;
   &lt;/os&gt;
   &lt;features&gt;
     &lt;acpi/&gt;
     &lt;apic/&gt;
   &lt;/features&gt;
+  &lt;vcpu placement='static'&gt;2&lt;/vcpu&gt;
   &lt;cpu mode='host-model'&gt;
     &lt;model fallback='allow'/&gt;
-    &lt;topology sockets='2' cores='1' threads='1'/&gt;
+    &lt;topology sockets='1' cores='2' threads='1'/&gt;
     &lt;numa&gt;
-      &lt;cell id='0' cpus='0-1' memory='4194304' unit='KiB' memAccess='shared'/&gt;
+      &lt;cell id='0' cpus='0' memory='4194304' unit='KiB' memAccess='shared'/&gt;
     &lt;/numa&gt;
   &lt;/cpu&gt;
+  &lt;cputune&gt;
+     &lt;vcpupin vcpu="0" cpuset="4"/&gt;
+     &lt;vcpupin vcpu="1" cpuset="5"/&gt;
+  &lt;/cputune&gt;
   &lt;on_poweroff&gt;destroy&lt;/on_poweroff&gt;
   &lt;on_reboot&gt;restart&lt;/on_reboot&gt;
   &lt;on_crash&gt;destroy&lt;/on_crash&gt;
   &lt;devices&gt;
-    &lt;emulator&gt;/usr/bin/qemu-system-x86_64&lt;/emulator&gt;
+    &lt;emulator&gt;/usr/bin/qemu-system-aarch64&lt;/emulator&gt;
     &lt;disk type='file' device='disk'&gt;
       &lt;driver name='qemu' type='raw' cache='none'/&gt;
-      &lt;source file='/mnt/qemu/enea-nfv-access-guest-qemux86-64.ext4'/&gt;
+      &lt;source file='/mnt/qemu/enea-nfv-access-guest-qemuarm64.ext4'/&gt;
       &lt;target dev='vda' bus='virtio'/&gt;
     &lt;/disk&gt;
-    &lt;interface type='vhostuser'&gt;
-      &lt;mac address='00:00:00:00:00:01'/&gt;
-      &lt;source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/&gt;
-       &lt;model type='virtio'/&gt;
-      &lt;driver queues='1'&gt;
-        &lt;host mrg_rxbuf='off'/&gt;
-      &lt;/driver&gt;
-    &lt;/interface&gt;
     &lt;serial type='pty'&gt;
       &lt;target port='0'/&gt;
     &lt;/serial&gt;
@@ -617,33 +626,41 @@ $ dpdk-devbind.py --bind=vfio-pci 0000:03:10.0
       &lt;target type='serial' port='0'/&gt;
     &lt;/console&gt;
   &lt;/devices&gt;
+  &lt;qemu:commandline&gt;
+     &lt;qemu:arg value='-chardev'/&gt;
+     &lt;qemu:arg value='socket,id=charnet0,path=/var/run/openvswitch/vhost-user1'/&gt;
+     &lt;qemu:arg value='-netdev'/&gt;
+     &lt;qemu:arg value='type=vhost-user,id=hostnet0,chardev=charnet0'/&gt;
+     &lt;qemu:arg value='-device'/&gt;
+     &lt;qemu:arg value='virtio-net-pci,netdev=hostnet0,id=net0,mac=00:00:00:00:00:01,bus=pcie.0,addr=0x2'/&gt;
+  &lt;/qemu:commandline&gt;
 &lt;/domain&gt;</programlisting>
       </section>
 
       <section id="guest-config-pci-passthrough">
         <title>Guest configuration with PCI passthrough</title>
 
-        <programlisting>&lt;domain type='kvm'&gt;
-  &lt;name&gt;vm_sriov1&lt;/name&gt;
-  &lt;uuid&gt;4a9b3f53-fa2a-47f3-a757-dd87720d9d1d&lt;/uuid&gt;
+        <programlisting>&lt;domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'&gt;
+  &lt;name&gt;vm_sriov&lt;/name&gt;
+  &lt;uuid&gt;ed204646-1ad5-11e7-93ae-92361f002671&lt;/uuid&gt;
   &lt;memory unit='KiB'&gt;4194304&lt;/memory&gt;
   &lt;currentMemory unit='KiB'&gt;4194304&lt;/currentMemory&gt;
   &lt;memoryBacking&gt;
     &lt;hugepages&gt;
-      &lt;page size='1' unit='G' nodeset='0'/&gt;
+      &lt;page size='512' unit='M' nodeset='0'/&gt;
     &lt;/hugepages&gt;
   &lt;/memoryBacking&gt;
-  &lt;vcpu&gt;2&lt;/vcpu&gt;
   &lt;os&gt;
-    &lt;type arch='x86_64' machine='q35'&gt;hvm&lt;/type&gt;
-    &lt;kernel&gt;/mnt/qemu/bzImage&lt;/kernel&gt;
-    &lt;cmdline&gt;root=/dev/vda console=ttyS0,115200&lt;/cmdline&gt;
+    &lt;type arch='aarch64' machine='virt,gic_version=3'&gt;hvm&lt;/type&gt;
+    &lt;kernel&gt;/mnt/qemu/Image&lt;/kernel&gt;
+    &lt;cmdline&gt;root=/dev/vda console=ttyAMA0,115200n8&lt;/cmdline&gt;
     &lt;boot dev='hd'/&gt;
   &lt;/os&gt;
   &lt;features&gt;
     &lt;acpi/&gt;
     &lt;apic/&gt;
   &lt;/features&gt;
+  &lt;vcpu placement='static'&gt;2&lt;/vcpu&gt;
   &lt;cpu mode='host-model'&gt;
     &lt;model fallback='allow'/&gt;
     &lt;topology sockets='1' cores='2' threads='1'/&gt;
@@ -651,22 +668,20 @@ $ dpdk-devbind.py --bind=vfio-pci 0000:03:10.0
       &lt;cell id='0' cpus='0' memory='4194304' unit='KiB' memAccess='shared'/&gt;
     &lt;/numa&gt;
   &lt;/cpu&gt;
+  &lt;cputune&gt;
+     &lt;vcpupin vcpu="0" cpuset="4"/&gt;
+     &lt;vcpupin vcpu="1" cpuset="5"/&gt;
+  &lt;/cputune&gt;
   &lt;on_poweroff&gt;destroy&lt;/on_poweroff&gt;
   &lt;on_reboot&gt;restart&lt;/on_reboot&gt;
   &lt;on_crash&gt;destroy&lt;/on_crash&gt;
   &lt;devices&gt;
-    &lt;emulator&gt;/usr/bin/qemu-system-x86_64&lt;/emulator&gt;
+    &lt;emulator&gt;/usr/bin/qemu-system-aarch64&lt;/emulator&gt;
     &lt;disk type='file' device='disk'&gt;
       &lt;driver name='qemu' type='raw' cache='none'/&gt;
-      &lt;source file='/mnt/qemu/enea-nfv-access-guest-qemux86-64.ext4'/&gt;
+      &lt;source file='/mnt/qemu/enea-nfv-access-guest-qemuarm64.ext4'/&gt;
       &lt;target dev='vda' bus='virtio'/&gt;
     &lt;/disk&gt;
-   &lt;interface type='hostdev' managed='yes'&gt;
-     &lt;source&gt;
-       &lt;address type='pci' domain='0x0' bus='0x03' slot='0x10' function='0x0'/&gt;
-     &lt;/source&gt;
-     &lt;mac address='52:54:00:6d:90:02'/&gt;
-   &lt;/interface&gt;
     &lt;serial type='pty'&gt;
       &lt;target port='0'/&gt;
     &lt;/serial&gt;
@@ -674,6 +689,10 @@ $ dpdk-devbind.py --bind=vfio-pci 0000:03:10.0
       &lt;target type='serial' port='0'/&gt;
     &lt;/console&gt;
   &lt;/devices&gt;
+  &lt;qemu:commandline&gt;
+     &lt;qemu:arg value='-device'/&gt;
+     &lt;qemu:arg value='vfio-pci,host=0001:01:00.1'/&gt;
+  &lt;/qemu:commandline&gt;
 &lt;/domain&gt;</programlisting>
       </section>
 
@@ -723,12 +742,12 @@ $ dpdk-devbind.py --bind=vfio-pci 0000:03:10.0
       &lt;driver name='qemu' type='raw' cache='none'/&gt;
       &lt;source file='/mnt/qemu/enea-nfv-access-guest-qemux86-64.ext4'/&gt;
       &lt;target dev='vda' bus='virtio'/&gt;
-    &lt;/disk&gt;
+    &lt;/disk&gt;vi
     &lt;interface type='bridge'&gt;
       &lt;mac address='52:54:00:71:b1:b6'/&gt;
       &lt;source bridge='ovsbr0'/&gt;
+      &lt;model type='rtl8139'/&gt;
       &lt;virtualport type='openvswitch'/&gt;
-      &lt;address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/&gt;
     &lt;/interface&gt;
     &lt;serial type='pty'&gt;
       &lt;target port='0'/&gt;
@@ -741,4 +760,4 @@ $ dpdk-devbind.py --bind=vfio-pci 0000:03:10.0
       </section>
     </section>
   </section>
-</chapter>
\ No newline at end of file
+</chapter>