Create new version of NFV Core 1.0 Installation Guide

USERDOCAP-240

Signed-off-by: Miruna Paun <Miruna.Paun@enea.com>

1 files changed, 794 insertions, 0 deletions

diff --git a/book-enea-nfv-core-installation-guide/doc/high_availability.xml b/book-enea-nfv-core-installation-guide/doc/high_availability.xml
new file mode 100644
index 0000000..e489101
--- /dev/null
+++ b/book-enea-nfv-core-installation-guide/doc/high_availability.xml
@@ -0,0 +1,794 @@
+<?xml version="1.0" encoding="ISO-8859-1"?>
+<chapter id="high_availability">
+  <title>High Availability Guide</title>
+
+  <para>ENEA NFV Core 1.0 has been designed to provide high availability
+  characteristics that are needed for developing and deploying telco-grade NFV
+  solutions on top of our OPNFV based platform.</para>
+
+  <para>The High Availability subject in general is very wide and still an
+  important focus in both opensource communities and independent/proprietary
+  solutions market. ENEA NFV Core 1.0 aims to initially leverage the efforts
+  in the upstream OPNFV and OpenStack opensource projects, combining solutions
+  from both worlds in an effort to provide flexibility and a wide enough use
+  case coverage. ENEA has a long time expertise and proprietary solutions
+  addressing High Availability for telco applications, which are subject to
+  integrating with the NFV based solutions, however the initial scope for ENEA
+  NFV Core is to leverage as much as possible the OPNFV Reference Platform and
+  open source projects in general, such as it will be seen further ahead in
+  this chapter.</para>
+
+  <section id="levels">
+    <title>High Availability Levels</title>
+
+    <para>The base for the feature set in ENEA NFV Core is divided into three
+    levels:</para>
+
+    <itemizedlist>
+      <listitem>
+        <para>Hardware Fault</para>
+      </listitem>
+
+      <listitem>
+        <para>NFV Platform HA</para>
+      </listitem>
+
+      <listitem>
+        <para>VNF High Availability</para>
+      </listitem>
+    </itemizedlist>
+
+    <para>The same division of levels of fault management can be seen in the
+    scope of the High Availability for OPNFV (Availability) project. OPNFV
+    also hosts the Doctor Project which is a fault management and maintenance
+    project to develop and realize the consequent implementation for the OPNFV
+    reference platform.</para>
+
+    <para>These two projects complement each other.</para>
+
+    <para>The Availability project addresses HA requirement and solutions from
+    the perspective of the three levels mentioned above and produces high
+    level requirements and API definitions for High Availability of OPNFV, HA
+    Gap Analysis Report for OpenStack and more recently works on optimizing
+    existing OPNFV test frameworks, such as Yardstick, and develops test cases
+    which realize HA specific use cases and scenarios such as derived from the
+    HA requirements.</para>
+
+    <para>The Doctor Project on the other hand aims to build fault management
+    and maintenance framework for high availability of Network Services on top
+    of virtualized infrastructure; the key feature is immediate notification
+    of unavailability of virtualized resources from VIM, to process recovery
+    of VNFs on them. The Doctor project has also collaborated with the
+    Availability project on identifying gaps in upstream project, mainly
+    OpenStack but not exclusive, and has worked towards implementing missing
+    features or improving the functionality, one good example being the Aodh
+    event based alarms, which allows for fast notifications when certain
+    predefined events occur. The Doctor project also produced an architecture
+    design and a reference implementation based on opensource components,
+    which will be presented later on in this document.</para>
+  </section>
+
+  <section id="doctor_arch">
+    <title>Doctor Architecture</title>
+
+    <para>The Doctor documentation shows the detailed architecture for Fault
+    Management and NFVI Maintenance . The two are very similar so we will
+    focus on the Fault Management.</para>
+
+    <para>The architecture specifies a set of functional blocks:</para>
+
+    <itemizedlist>
+      <listitem>
+        <para>Monitor - monitors the virtualized infrastructure capturing
+        fault events in the Software and Hardware; for this particular
+        component we chose Zabbix which is integrated into the platform by
+        means of the Fuel Zabbix Plugin, available upstream.</para>
+      </listitem>
+
+      <listitem>
+        <para>Inspector - this component is able to receive notifications from
+        Monitor components and also OpenStack core components, which allows it
+        to create logic relationships between entities, identify affected
+        resources when faults occur, and communicates with Controllers to
+        update the states of the virtual and physical resources. For this
+        component ENEA NFV Core 1.0 makes use of Vitrage , an OpenStack
+        related project used for Root Cause Analysis, which has been adapted
+        to server as a Doctor Inspector. The integration into the platform is
+        realized with the help of a Fuel Plugin which has been developed
+        internally by ENEA.</para>
+      </listitem>
+
+      <listitem>
+        <para>Controller - OpenStack core components act as Controllers, which
+        are responsible for maintaining the resource map between physical and
+        virtual resources, they accept update requests from the Inspector and
+        are responsible for sending failure event notifications to the
+        Notifier. Components such as Nova, Neutron, Glance, Heat act as
+        Controllers in the Doctor Architecture.</para>
+      </listitem>
+
+      <listitem>
+        <para>Notifier - the focus of this component is on selecting and
+        aggregating failure events received from the controller based on
+        policies mandated by the Consumer. The role of the Notifier is
+        accomplished by the Aodh component in OpenStack.</para>
+      </listitem>
+    </itemizedlist>
+
+    <para>Besides the Doctor components there are a couple other blocks
+    mentioned in the architecture:</para>
+
+    <itemizedlist>
+      <listitem>
+        <para>Administrator - this represents the human role of administrating
+        the platform by means of dedicated interfaces, either visual
+        dashboards, like OpenStack Horizon or Fuel Dashboard, or via CLI
+        tools, like the OpenStack unified CLI that can be accessed
+        traditionally from one of the servers that act as OpenStack Controller
+        nodes. In the case of ENEA NFV Core 1.0, the Administrator can also
+        access the Zabbix dashboard for doing further configurations. The same
+        applies for the Vitrage tool, which comes with its own Horizon
+        dashboard which enables the user to visually inspect the faults
+        reported by the monitoring tools and also creates visual
+        representations of the virtual and physical resources, the
+        relationships between them and the fault correlation. For Vitrage,
+        users will usually want to configure additional usecases and describe
+        relationships between components, via template files written in yaml
+        format. More information about using Vitrage will be presented in a
+        following section.</para>
+      </listitem>
+
+      <listitem>
+        <para>Consumer - this block is vaguely described in the Doctor
+        Architecture and it's out of its scope. Doctor only deals with fault
+        detection and management, making sure faults are handled as soon as
+        possible after detection, identifies affected virtual resources and
+        updates the states of them, but since the actual VNFs are managed,
+        according to the ETSI architecture, by a different entity, Doctor does
+        not deal with recovery actions of the VNFs. The role of the Consumer
+        thus falls in the task of a VNF Manager and Orchestrator. ENEA NFV
+        Core 1.0 provides VNF management capabilities using Tacker, which is
+        an OpenStack project that implements a generic VNF Manager and
+        Orchestrator according to the ETSI MANO Architectural
+        Framework.</para>
+      </listitem>
+    </itemizedlist>
+
+    <para>The functional blocks overview in the picture below has been
+    complemented to show the components used for realizing the Doctor
+    Architecture:</para>
+
+    <mediaobject>
+      <imageobject role="fo">
+        <imagedata contentwidth="600" fileref="images/functional_blocks.svg"
+                   format="SVG" />
+      </imageobject>
+    </mediaobject>
+
+    <section id="dr_fault_mg">
+      <title>Doctor Fault Management</title>
+
+      <para>The architecture described in the Doctor project has been
+      demonstrated in various PoCs and demos, but always using sample
+      components for either the consumer or the monitor. ENEA has worked with
+      upstream projects, Doctor and Vitrage, to realize the goals of the
+      Doctor project by using real components, as described before.</para>
+
+      <para>The two pictures below show a typical fault management scenario,
+      as described in the Doctor documentation.</para>
+
+      <mediaobject>
+        <imageobject>
+          <imagedata contentwidth="600" fileref="images/dr_fault_mg.svg" />
+        </imageobject>
+      </mediaobject>
+
+      <mediaobject>
+        <imageobject>
+          <imagedata contentwidth="600" fileref="images/dr_fault_mg_2.svg" />
+        </imageobject>
+      </mediaobject>
+
+      <para>ENEA NFV Core 1.0 uses the same approach described above, but it's
+      worth going through each step and detail them.</para>
+
+      <orderedlist>
+        <listitem>
+          <para>When creating a VNF, the user will have to enable the
+          monitoring capabilities of Tacker, by passing a template which
+          specifies that an alarm will be created when the VM represented by
+          this VNF changes state. The support for alarm monitoring in Tacker
+          is captured in the Alarm Monitoring Framework spec in OpenStack
+          documentation. In a few words, Tacker should be able to create a VNF
+          and then create an Aodh alarm of type event which triggers when the
+          instance is in state ERROR. The action to take when this event
+          triggers is to perform an HTTP call, to an URL managed by Tacker. As
+          a result of this action, Tacker can detect when an instance has
+          failed (for whatever reasons) and will respawn it somewhere
+          else.</para>
+        </listitem>
+
+        <listitem>
+          <para>The subscribe response in this case is an empty operation, the
+          Notifier (Aodh) only has to confirm that the alarm has been
+          created.</para>
+        </listitem>
+
+        <listitem>
+          <para>The NFVI sends monitoring events for resources the VIM has
+          been subscribed to. Note: this subscription message exchange between
+          the VIM and NFVI is not shown in this message flow. This steps is
+          related to Vitrage's capability of receiving notifications from
+          OpenStack services, at this moment Vitrage supports notifications
+          from nova.host, nova.instances, nova.zone, cinder.volume,
+          neutron.network, neutron.port and heat.stack OpenStack
+          datasources.</para>
+        </listitem>
+
+        <listitem>
+          <para>This steps describes faults being detected by Zabbix which are
+          sent to the Inspector (Vitrage) as soon as detected, using a push
+          approach by means of sending an AMQP message to a dedicated message
+          queue managed by Vitrage. For example, if nova-compute fails on one
+          of the compute nodes, Zabbix will format a message specifying all
+          the needed details needed for processing the fault, e.g. a
+          timestamp, what host failed, what event occurred and others.</para>
+        </listitem>
+
+        <listitem>
+          <para>Database lookup to find the virtual resources affected by the
+          detected fault. In this step Vitrage will perform various
+          calculations to detect what virtual resources are affected by the
+          raw failure presented by Zabbix. Vitrage can be configured via
+          templates to correlate instances with the physical hosts they are
+          running on, so that if a compute node fails, then instances running
+          on that host will be affected. A typical usecase is to mark the
+          compute node down (a.k.a mark_host_down) and update the states of
+          all instances running on them, by issuing Nova API calls for each of
+          these instances. Step 5c) shows the Controller (Nova in this case)
+          acting upon the state change of the instance and issues an event
+          alarm to Aodh.</para>
+        </listitem>
+
+        <listitem>
+          <para>The Notifier will acknowledge the alarm event request from
+          Nova and will trigger the alarm(s) created by Tacker in step 1).
+          Since Tacker has configured the alarm to send an HTTP request, Aodh
+          will perform that HTTP call at the URL managed by Tacker.</para>
+        </listitem>
+
+        <listitem>
+          <para>The Consumer (Tacker) will react to the HTTP call and perform
+          the action configured by the user (e.g. respawn the VNF).</para>
+        </listitem>
+
+        <listitem>
+          <para>The action is sent to the Controller (Nova) so that the VNF is
+          recreated.</para>
+        </listitem>
+      </orderedlist>
+
+      <note>
+        <para>The ENEA NFV Core 1.0 Pre-Release fully covers the required
+        Doctor functionality only for the Vitrage and Zabbix
+        components.</para>
+      </note>
+    </section>
+
+    <section id="zabbix">
+      <title>Zabbix Configuration for Push Notifications</title>
+
+      <para>Vitrage supports Zabbix datasource by means of regularly polling
+      the Zabbix agents, which need to be configured in advance. The Vitrage
+      plugin developed internally by ENEA can automatically configure Zabbix
+      so that everything works as expected.</para>
+
+      <para>However, polling is not fast enough for a telco usecase, so it is
+      necessary to configure pushed notifications for Zabbix . This requires
+      manual configuration on one of the controller nodes, since Zabbix uses a
+      centralized database which makes the configuration available on all the
+      other nodes.</para>
+
+      <para>The Zabbix configuration dashboard is available at the same IP
+      address where OpenStack can be reached, e.g.
+      http://&lt;vip__zbx_vip_mgmt&gt;/zabbix.</para>
+
+      <para>To forward zabbix events to Vitrage a new media script needs to be
+      created and associated with a user. Follow the steps below as a Zabbix
+      Admin user:</para>
+
+      <orderedlist>
+        <listitem>
+          <para>Create a new media type [Admininstration Media Types Create
+          Media Type]</para>
+
+          <itemizedlist>
+            <listitem>
+              <para>Name: Vitrage Notifications</para>
+            </listitem>
+
+            <listitem>
+              <para>Type: Script</para>
+            </listitem>
+
+            <listitem>
+              <para>Script name: zabbix_vitrage.py</para>
+            </listitem>
+          </itemizedlist>
+        </listitem>
+
+        <listitem>
+          <para>Modify the Media for the Admin user [Administration
+          Users]</para>
+
+          <itemizedlist>
+            <listitem>
+              <para>Type: Vitrage Notifications</para>
+            </listitem>
+
+            <listitem>
+              <para>Send to: rabbit://rabbit_user:rabbit_pass@127.0.0.1:5672/
+              --- Vitrage message bus url (you need to search for this in
+              /etc/vitrage/vitrage.conf or /etc/nova/nova.conf
+              transport_url)</para>
+            </listitem>
+
+            <listitem>
+              <para>When active: 1-7,00:00-24:00</para>
+            </listitem>
+
+            <listitem>
+              <para>Use if severity: (all)</para>
+            </listitem>
+
+            <listitem>
+              <para>Status: Enabled</para>
+            </listitem>
+          </itemizedlist>
+        </listitem>
+
+        <listitem>
+          <para>Configure Action [Configuration Actions Create Action
+          Action]</para>
+
+          <itemizedlist>
+            <listitem>
+              <para>Name: Forward to Vitrage</para>
+            </listitem>
+
+            <listitem>
+              <para>Default Subject: {TRIGGER.STATUS}</para>
+            </listitem>
+
+            <listitem>
+              <para>Default Message: host={HOST.NAME1} hostid={HOST.ID1}
+              hostip={HOST.IP1} triggerid={TRIGGER.ID}
+              description={TRIGGER.NAME} rawtext={TRIGGER.NAME.ORIG}
+              expression={TRIGGER.EXPRESSION} value={TRIGGER.VALUE}
+              priority={TRIGGER.NSEVERITY} lastchange={EVENT.DATE}
+              {EVENT.TIME}</para>
+            </listitem>
+          </itemizedlist>
+        </listitem>
+
+        <listitem>
+          <para>To send events add under the Conditions tab: 'Maintenance
+          status not in 'maintenance'".</para>
+        </listitem>
+
+        <listitem>
+          <para>Finally, add an operation:</para>
+
+          <itemizedlist>
+            <listitem>
+              <para>Send to Users: Admin</para>
+            </listitem>
+
+            <listitem>
+              <para>Send only to: Vitrage Notifications</para>
+            </listitem>
+          </itemizedlist>
+        </listitem>
+      </orderedlist>
+
+      <para>Using these instructions, Zabbix will call the zabbix_vitrage.py
+      script, which is made readily available by the Fuel Vitrage Plugin,
+      passing the arguments described in step 3). The zabbix_vitrage.py script
+      will then interpret the parameters and format an AMQP message will be
+      sent to the vitrage.notifications queue, which is managed by the
+      vitrage-graph service.</para>
+    </section>
+
+    <section id="vitrage_config">
+      <title>Vitrage Configuration</title>
+
+      <para>The Vitrage team has been collaborating with OPNFV Doctor Project
+      in order to support Vitrage as an Inspector Component. The Doctor
+      usecase for Vitrage is described in an OpenStack blueprint .
+      Additionally, ENEA NFV Core has complemented Vitrage with the capability
+      of setting states of failed instances by implementing an action type in
+      Vitrage which calls Nova APIs to set instances in error state. There is
+      also an action type which allows fencing failed hosts.</para>
+
+      <para>In order to make use of these features, Vitrage supports
+      additional configurations via yaml templates that must be placed in
+      /etc/vitrage/templates on the nodes have the Vitrage role.</para>
+
+      <para>The example below shows how to program Vitrage to mark failed
+      compute hosts as down and then to change the state of the instances to
+      Error, by creating Vitrage deduced alarms.</para>
+
+      <programlisting>metadata:
+ name: test_nova_mark_instance_err
+ description: test description
+definitions:
+ entities:
+  - entity:
+     category: ALARM
+     type: zabbix
+     rawtext: Nova Compute process is not running on {HOST.NAME}
+     template_id:  zabbix_alarm
+  - entity:
+     category: RESOURCE
+     type: nova.host
+     template_id: host
+  - entity:
+     category: RESOURCE
+     type: nova.instance
+     template_id: instance
+ relationships:
+  - relationship:
+     source: zabbix_alarm
+     relationship_type: on
+     target: host
+     template_id: nova_process_not_running
+  - relationship:
+      source: host
+      target: instance
+      relationship_type: contains
+      template_id : host_contains_instance
+scenarios:
+ - scenario:
+    condition: nova_process_not_running and host_contains_instance
+    actions:
+     - action:
+        action_type: mark_down
+        action_target:
+         target: host
+     - action:
+        action_type: set_instance_state
+        action_target:
+         target: instance
+     - action:
+        action_type: set_state
+        action_target:
+         target: instance
+        properties:
+         state: ERROR</programlisting>
+
+      <para>For the action type of fencing a similar action item must be
+      added:</para>
+
+      <programlisting>- scenario:
+    condition: critical_problem_on_host
+    actions:
+     - action:
+        action_type: fence
+        action_target:
+         target: host</programlisting>
+
+      <para>After a template is configured, it is required to restart the
+      vitrage-api and vitrage-graph services:</para>
+
+      <programlisting>root@node-6:~# systemctl restart vitrage-api
+root@node-6:~# systemctl restart vitrage-graph</programlisting>
+    </section>
+
+    <section id="vitrage_custom">
+      <title>Vitrage Customizations</title>
+
+      <para>ENEA NFV Core 1.0 has added custom features for Vitrage which
+      allow two kinds of action:</para>
+
+      <orderedlist>
+        <listitem>
+          <para>Perform actions Northbound of the VIM</para>
+
+          <itemizedlist>
+            <listitem>
+              <para>Nova force host down on compute</para>
+            </listitem>
+
+            <listitem>
+              <para>Setting instance state to error in nova; this is used in
+              conjunction with an alarm created by Tacker, as described
+              before, should allow Tacker to detect when an instance is
+              affected and take proper actions.</para>
+            </listitem>
+          </itemizedlist>
+        </listitem>
+
+        <listitem>
+          <para>Perform actions Southbound of the VIM.</para>
+
+          <para>Vitrage templates allow us to program fencing actions for
+          hosts with failed services. In the event of that systemd is unable
+          to recover from a critical process or other type of sofware error
+          ocurs on Hardware supporting them, we can program a fencing of that
+          Node which will perform a reboot thus attempting to recover a failed
+          node.</para>
+        </listitem>
+      </orderedlist>
+    </section>
+  </section>
+
+  <section id="pm_high_avail">
+    <title>Pacemaker High Availability</title>
+
+    <para>Many of the OpenStack solutions which offer High Availability
+    characteristics employ pacemaker for achieving highly available OpenStack
+    services. Traditionally pacemaker has been used for managing only the
+    control plane services, so it can effectively provide redundancy and
+    recovery for the Controller nodes only. One reason for this is that
+    Controller nodes and Compute nodes essentially have very different High
+    Availability requirements that need to be considered. Typically, for
+    Controller nodes, the services that run on them are stateless, with few
+    exceptions, where only one instance of a given service is allowed, but for
+    which redundancy is still desired, one good example being an AMQP service
+    (e.g. RabbitMQ). Compute nodes HA requirements depend on the type of
+    services that run on them, but typically it is desired that failures on
+    these nodes is detected as soon as possible so that the instances that run
+    on them can be either migrated, resurrected or restarted. One other aspect
+    is that sometimes failures on the physical hosts do not necessarily cause
+    a failure on the services (VNFs), but having these services incapacitated
+    can prevent accessing and controlling the services.</para>
+
+    <para>So Controller High Availability is one subject which is in general
+    well understood and experimented with, and the base of achieving this is
+    Pacemaker using Corosync underneath.</para>
+
+    <para>Extending the use of pacemaker to Compute nodes was thought as a
+    possible solution for providing VNF high availability, but this turns out
+    to be a problem which is not easy to solve. On one hand pacemaker as a
+    clustering tool can only scale properly up to limited number of nodes,
+    usually less than 128. This poses a problem for large scale deployments
+    where hundreds of compute nodes are required. On the other hand, Compute
+    node HA requires other considerations and calls for specially designed
+    solutions.</para>
+
+    <section id="pm_remote">
+      <title>Pacemaker Remote</title>
+
+      <para>As mentioned earlier, pacemaker and corosync do not scale well
+      over a large cluster, because each node has to talk to everyone,
+      essentially creating a mesh configuration. Some solution to this problem
+      could be partitioning the cluster into smaller groups, but this solution
+      has its limitation and it's generally difficult to manage.</para>
+
+      <para>A better solution is using pacemaker-remote, a feature of
+      pacemaker which allows extending the cluster beyond the usual limits by
+      using the pacemaker monitoring capabilities, essentially creating a new
+      type of resource which enables adding light weight nodes to the cluster.
+      More information about pacemaker-remote can be found on the official
+      clusterlabs website.</para>
+
+      <para>Please note that at this moment pacemaker remote must be
+      configured manually after deployment. Here are the manual steps for
+      doing so:</para>
+
+      <orderedlist>
+        <listitem>
+          <para>Logon to the Fuel Master using the default credentials if not
+          changed (root/r00tme)</para>
+        </listitem>
+
+        <listitem>
+          <para>Type fuel node to obtain the list of nodes, their roles and
+          the IP addresses</para>
+
+          <programlisting>[root@fuel ~]# fuel node
+id | status | name             | cluster | ip        | mac               | roles    /
+                 | pending_roles | online | group_id
+---+--------+------------------+---------+-----------+-------------------+----------/
+-----------------+---------------+--------+---------
+ 1 | ready  | Untitled (8c:d4) |       1 | 10.20.0.4 | 68:05:ca:46:8c:d4 | ceph-osd,/
+ controller      |               |      1 |        1
+ 4 | ready  | Untitled (8c:c2) |       1 | 10.20.0.6 | 68:05:ca:46:8c:c2 | ceph-osd,/
+ compute         |               |      1 |        1
+ 5 | ready  | Untitled (8c:c9) |       1 | 10.20.0.7 | 68:05:ca:46:8c:c9 | ceph-osd,/
+ compute         |               |      1 |        1
+ 2 | ready  | Untitled (8b:64) |       1 | 10.20.0.3 | 68:05:ca:46:8b:64 | /
+controller, mongo, tacker |               |      1 |        1
+ 3 | ready  | Untitled (8c:45) |       1 | 10.20.0.5 | 68:05:ca:46:8c:45 | /
+controller, vitrage       |               |      1 |        1</programlisting>
+        </listitem>
+
+        <listitem>
+          <para>Each controller has a unique pacemaker authkey, we need to
+          keep one an propagate it to the other servers. Assuming node-1,
+          node-2 and node-3 are the controllers, execute the following from
+          the Fuel console:</para>
+
+          <programlisting>[root@fuel ~]# scp node-1:/etc/pacemaker/authkey .
+[root@fuel ~]# scp authkey node-2:/etc/pacemaker/
+[root@fuel ~]# scp authkey node-3:/etc/pacemaker/
+[root@fuel ~]# scp authkey node-3:/etc/pacemaker/
+[root@fuel ~]# scp authkey node-4:~
+[root@fuel ~]# scp authkey node-5:~</programlisting>
+        </listitem>
+
+        <listitem>
+          <para>For each compute node, log on to it using the corresponding
+          IP.</para>
+        </listitem>
+
+        <listitem>
+          <para>Install the required packages:</para>
+
+          <programlisting>root@node-4:~# apt-get install pacemaker-remote resource-agents crmsh</programlisting>
+        </listitem>
+
+        <listitem>
+          <para>Copy the authkey from the Fuel master and make sure the right
+          permissions are set:</para>
+
+          <programlisting>[root@node-4:~]# cp authkey /etc/pacemaker
+[root@node-4:~]# chown root:haclient /etc/pacemaker/authkey</programlisting>
+        </listitem>
+
+        <listitem>
+          <para>Add iptables rule for the default port (3121). Also save it to
+          /etc/iptables/rules.v4 to make it persistent:</para>
+
+          <programlisting>root@node-4:~# iptables -A INPUT -s 192.168.0.0/24 -p tcp -m multiport /
+--dports 3121 -m comment --comment "pacemaker_remoted from 192.168.0.0/24" -j ACCEPT </programlisting>
+        </listitem>
+
+        <listitem>
+          <para>Start the pacemaker-remote service</para>
+
+          <programlisting>[root@node-4:~]# systemctl start pacemaker-remote.service</programlisting>
+        </listitem>
+
+        <listitem>
+          <para>Log on one of the controller nodes and configure the
+          pacemaker-remote resources:</para>
+
+          <programlisting>[root@node-1:~]# pcs resource create node-4.domain.tld remote
+[root@node-1:~]# pcs constraint location node-4.domain.tld prefers /
+node-1.domain.tld=100 node-2.domain.tld=100 node-3.domain.tld=100
+[root@node-1:~]# pcs constraint location node-4.domain.tld avoids node-5.domain.tld
+[root@node-1:~]# pcs resource create node-5.domain.tld remote
+[root@node-1:~]# pcs constraint location node-5.domain.tld prefers /
+node-1.domain.tld=100 node-2.domain.tld=100 node-3.domain.tld=100
+[root@node-1:~]# pcs constraint location node-5.domain.tld avoids node-4.domain.tld</programlisting>
+        </listitem>
+
+        <listitem>
+          <para>Remote nodes should now appear online:</para>
+
+          <programlisting>[root@node-1:~]# pcs status
+Cluster name: OpenStack
+Last updated: Thu Aug 24 12:00:21 2017          Last change: Thu Aug 24 11:57:32 2017 /
+by root via cibadmin on node-1.domain.tld
+Stack: corosync
+Current DC: node-1.domain.tld (version 1.1.14-70404b0) - partition with quorum
+5 nodes and 78 resources configured
+
+Online: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld ]
+RemoteOnline: [ node-4.domain.tld node-5.domain.tld ]</programlisting>
+        </listitem>
+      </orderedlist>
+    </section>
+
+    <section id="pm_fencing">
+      <title>Pacemaker Fencing</title>
+
+      <para>ENEA NFV Core 1.0 makes use of the fencing capabilities of
+      Pacemaker to isolate faulty nodes and trigger recovery actions by means
+      of power cycling the failed nodes. Fencing is configured by creating
+      STONITH type resources for each of the servers in the cluster, both
+      Controller nodes and Compute nodes. The STONITH adapter for fencing the
+      nodes is fence_ipmilan, which makes use of the IPMI capabilities of the
+      Cavium ThunderX servers.</para>
+
+      <para>Here are the steps for enabling fencing capabilities in the
+      cluster:</para>
+
+      <orderedlist>
+        <listitem>
+          <para>Logon to the Fuel Master using the default credentials if not
+          changed (root/r00tme).</para>
+        </listitem>
+
+        <listitem>
+          <para>Type fuel node to obtain the list of nodes, their roles and
+          the IP addresses:</para>
+
+          <programlisting>[root@fuel ~]# fuel node
+id | status | name             | cluster | ip        | mac               | roles    /
+                 | pending_roles | online | group_id
+---+--------+------------------+---------+-----------+-------------------+----------/
+-----------------+---------------+--------+---------
+ 1 | ready  | Untitled (8c:d4) |       1 | 10.20.0.4 | 68:05:ca:46:8c:d4 | ceph-osd,/
+ controller      |               |      1 |        1
+ 4 | ready  | Untitled (8c:c2) |       1 | 10.20.0.6 | 68:05:ca:46:8c:c2 | ceph-osd,/
+ compute         |               |      1 |        1
+ 5 | ready  | Untitled (8c:c9) |       1 | 10.20.0.7 | 68:05:ca:46:8c:c9 | ceph-osd,/
+ compute         |               |      1 |        1
+ 2 | ready  | Untitled (8b:64) |       1 | 10.20.0.3 | 68:05:ca:46:8b:64 | /
+controller, mongo, tacker |               |      1 |        1
+ 3 | ready  | Untitled (8c:45) |       1 | 10.20.0.5 | 68:05:ca:46:8c:45 | /
+controller, vitrage       |               |      1 |        1
+</programlisting>
+        </listitem>
+
+        <listitem>
+          <para>Logon to each server to install additional packages:</para>
+
+          <programlisting>[root@node-1:~]# apt-get install fence-agents ipmitool</programlisting>
+        </listitem>
+
+        <listitem>
+          <para>Configure pacemaker fencing resources; this needs to be done
+          once on one of the controllers. The parameters will vary, depending
+          on the BMC addresses of each node and credentials.</para>
+
+          <programlisting>[root@node-1:~]# crm configure primitive ipmi-fencing-node-1 /
+stonith::fence_ipmilan params pcmk_host_list="node-1.domain.tld" /
+ipaddr=10.0.100.151 login=ADMIN passwd=ADMIN op monitor interval="60s"
+[root@node-1:~]# crm configure primitive ipmi-fencing-node-2 /
+stonith::fence_ipmilan params pcmk_host_list="node-2.domain.tld" /
+ipaddr=10.0.100.152 login=ADMIN passwd=ADMIN op monitor interval="60s"
+[root@node-1:~]# crm configure primitive ipmi-fencing-node-3 /
+stonith::fence_ipmilan params pcmk_host_list="node-3.domain.tld" /
+ipaddr=10.0.100.153 login=ADMIN passwd=ADMIN op monitor interval="60s"
+[root@node-1:~]# crm configure primitive ipmi-fencing-node-4 /
+stonith::fence_ipmilan params pcmk_host_list="node-4.domain.tld" /
+ipaddr=10.0.100.154 login=ADMIN passwd=ADMIN op monitor interval="60s"
+[root@node-1:~]# crm configure primitive ipmi-fencing-node-5 /
+stonith::fence_ipmilan params pcmk_host_list="node-5.domain.tld" /
+ipaddr=10.0.100.155 login=ADMIN passwd=ADMIN op monitor interval="60s"</programlisting>
+        </listitem>
+
+        <listitem>
+          <para>Activate fencing by enabling stonith property in pacemaker (by
+          default it is disabled); this also needs to be done only once, on
+          one of the controllers.</para>
+
+          <programlisting>[root@node-1:~]# pcs property set stonith-enabled=true</programlisting>
+        </listitem>
+      </orderedlist>
+    </section>
+  </section>
+
+  <section id="ops_resources_agents">
+    <title>OpenStack Resource Agents</title>
+
+    <para>The OpenStack community has been working for some time on
+    identifying possible solutions for enabling High Availability for Compute
+    nodes, although initially the subject of HA on compute node was very
+    controversial as not being something that should concern the cloud
+    platform. Over time it became obvious that even on a true cloud platform,
+    where services are designed to run without being affected by the
+    availability of the cloud platform, fault management and recovery is still
+    very important and desirable. This is very much the case for NFV
+    applications, where, in the good tradition of telecom applications, the
+    operators must have complete engineering control over the resources it
+    owns and manages.</para>
+
+    <para>The work for compute node high availability is captured in an
+    OpenStack user story and documented upstream, showing proposed solutions,
+    summit talks and presentations.</para>
+
+    <para>A number of these solutions make use of OpenStack Resource Agents,
+    which are basically a set of specialized pacemaker resources which are
+    capable of identifying failures in compute nodes and can perform automatic
+    evacuation of the instances affected by these failures.</para>
+
+    <para>ENEA NFV Core 1.0 aims to validate and integrate this work and to
+    make this feature available in the platform to be used as an alternative
+    to the Doctor framework, where simple, autonomous recovery of the running
+    instances is desired.</para>
+  </section>
+</chapter>
\ No newline at end of file