1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
|
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
<chapter id="ovs">
<title>Open Virtual Switch</title>
<para>Open vSwitch (OVS) is an open-source multilayer virtual switch
designed to be used in virtualized environments to forward traffic between
different VMs on the same host, and also between VMs and the physical
network.</para>
<para>Native OVS forwarding is handled by two major components: a user-space
daemon called <literal>ovs-vswitchd</literal> and a
<literal>fastpath</literal> kernel module used to accelerate the data path.
The fastpath kernel module will handle packets received on the NIC by simply
consulting a flow table with corresponding action rules (e.g to forward the
packet or modify its headers). If no matching entry is found in the flow
table, the packet is copied to the user-space and sent to the ovs-vswitchd
deamon which determines how it should be handled ("slowpath").</para>
<para>The packet is then passed back to the kernel module together with the
desired action and the flow table is updated, so that subsequent packets in
the same flow can be handled in fastpath without any user-space interaction.
In this way, OVS eliminates a lot of the context switching between
kernel-space and user-space, but the throughput is still limited by the
capacity of the Linux kernel stack.</para>
<section id="ovs-dpdk">
<title>OVS-DPDK</title>
<para>To improve performance, OVS supports integration with Intel DPDK
libraries to operate entirely in user-space (OVS-DPDK). DPDK Poll Mode
Drivers (PMDs) enable direct transfers of packets between the physical NIC
and user-space, thereby eliminating the overhead of interrupt handling and
Linux kernel network stack processing. OVS-DPDK provides DPDK-backed
vhost-user ports as the primary way to connect guests to this datapath.
The vhost-user interfaces are transparent to the guest.</para>
</section>
<section id="ovs-commands">
<title>OVS commands</title>
<para>OVS provides a rich set of command line management tools, most
importantly:</para>
<itemizedlist>
<listitem>
<para>ovs-vsctl: Used to manage and inspect switch configurations,
e.g. to create bridges and to add/remove ports.</para>
</listitem>
<listitem>
<para>ovs-ofctl: Used to configure and monitor flows.</para>
</listitem>
</itemizedlist>
<para>For more information about Open vSwitch, see <ulink
url="http://openvswitch.org">http://openvswitch.org</ulink>.</para>
</section>
<section id="config-ovs-dpdk">
<title>Configuring OVS-DPDK for improved performance</title>
<section id="dpdk-lcore-mask">
<title>dpdk-lcore-mask</title>
<para>Specifies the CPU core affinity for DPDK lcore threads. The lcore
threads are used for DPDK library tasks. For performance it is best to
set this to a single core on the system, and it should not overlap the
pmd-cpu-mask, as seen in the example below.</para>
<para>Example: To use core 1:</para>
<programlisting>ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x1</programlisting>
</section>
<section id="pmd-cpu-mask">
<title>pmd-cpu-mask</title>
<para>The DPDK PMD threads polling for incoming packets are CPU bound
and should be pinned to isolated cores for optimal performance.</para>
<para>If OVS-DPDK receives traffic on multiple ports, for example when
DPDK and vhost-user ports are used for bi-directional traffic, the
performance can be significantly improved by creating multiple PMD
threads and affinitizing them to separate cores in order to share the
workload, by each being responsible for an individual port. The cores
should not be hyperthreads on the same CPU.</para>
<para>The PMD core affinity is specified by setting an appropriate core
mask. Example: using cores 2 and 3:</para>
<programlisting>ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0xc</programlisting>
</section>
</section>
<section id="setup-ovs-dpdk">
<title>How to set up OVS-DPDK</title>
<para>The DPDK must be configured prior to setting up OVS-DPDK. See
[FIXME] for DPDK setup instructions, then follow these steps:</para>
<orderedlist>
<listitem>
<para>Clean up the environment:</para>
<programlisting>killall ovsdb-server ovs-vswitchd
rm -f /var/run/openvswitch/vhost-user*
rm -f /etc/openvswitch/conf.db</programlisting>
</listitem>
<listitem>
<para>Start the ovsdb-server:</para>
<programlisting>export DB_SOCK=/var/run/openvswitch/db.sock
ovsdb-tool create /etc/openvswitch/conf.db /usr/share/openvswitch/vswitch.ovsschema
ovsdb-server --remote=punix:$DB_SOCK /
--remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach</programlisting>
</listitem>
<listitem>
<para>Start ovs-vswitchd with DPDK support enabled:</para>
<programlisting>ovs-vsctl --no-wait init
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x1
ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0xc
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
ovs-vswitchd unix:$DB_SOCK --pidfile --detach /
--log-file=/var/log/openvswitch/ovs-vswitchd.log</programlisting>
</listitem>
<listitem>
<para>Create the OVS bridge and attach ports:</para>
<programlisting>ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev
ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk /
:dpdk-devargs=<PCI device></programlisting>
</listitem>
<listitem>
<para>Add DPDK vhost-user ports:</para>
<programlisting>ovs-vsctl add-port ovsbr0 vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser</programlisting>
<para>This command creates a socket at
<literal>/var/run/openvswitch/vhost-user1</literal>, which can be
provided to the VM on the QEMU command line. See [FIXME] for
details.</para>
</listitem>
<listitem>
<para>Define flows:</para>
<programlisting>ovs-ofctl del-flows ovsbr0
ovs-ofctl show ovsbr0
ovs-ofctl add-flow ovsbr0 in_port=1,action=output:2
ovs-ofctl add-flow ovsbr0 in_port=2,action=output:1</programlisting>
</listitem>
</orderedlist>
</section>
</chapter>
|
