path: root/book-enea-nfv-core-installation-guide/doc/high_availability.xml

<?xml version="1.0" encoding="ISO-8859-1"?>
<chapter id="high_availability">
  <title>High Availability Guide</title>

  <para>Enea NFV Core <xi:include
  href="../../s_docsrc_common/pardoc-distro.xml"
  xmlns:xi="http://www.w3.org/2001/XInclude"
  xpointer="element(EneaNFV_REL_VER/1)" /> has been designed to provide high
  availability characteristics that are needed for developing and deploying
  telco-grade NFV solutions on top of our OPNFV based platform. The High
  Availability subject in general is very wide and still an important focus in
  both opensource communities and the independent/proprietary solutions
  market.</para>

  <para>Enea NFV Core <xi:include
  href="../../s_docsrc_common/pardoc-distro.xml"
  xmlns:xi="http://www.w3.org/2001/XInclude"
  xpointer="element(EneaNFV_REL_VER/1)" /> aims to initially leverage the
  efforts in the upstream OPNFV and OpenStack opensource projects, combining
  solutions from both worlds in an effort to provide flexibility and use-case
  coverage. Enea has long term expertise and proprietary solutions addressing
  High Availability for telco applications, which are subject to integration
  with the NFV based solutions.</para>

  <section id="levels">
    <title>High Availability Levels</title>

    <para>The foundation for the feature set available in Enea NFV Core is
    divided into three levels:</para>

    <itemizedlist>
      <listitem>
        <para>Hardware Fault</para>
      </listitem>

      <listitem>
        <para>NFV Platform H.A.</para>
      </listitem>

      <listitem>
        <para>VNF High Availability</para>
      </listitem>
    </itemizedlist>

    <para>The same division of levels for fault management can be seen in the
    scope of the High Availability for OPNFV ("Availability") project. OPNFV
    also hosts Doctor, a fault management and maintenance project designed to
    develop and perform the consequent implementation of the OPNFV reference
    platform. These two projects complement each other.</para>

    <para>The Availability project addresses H.A. requirements and solutions
    from the perspective of the three levels mentioned above. It produces high
    level requirements and API definitions for High Availability for OPNFV, a
    H.A. Gap Analysis Report for OpenStack and more recently, works on
    optimizing existing OPNFV test frameworks, such as Yardstick, developing
    test cases which realize H.A.-specific use-cases and scenarios derived
    from the H.A. requirements.</para>

    <para>The Doctor project aims to build fault management and maintenance
    framework for the high availability of Network Services, on top of a
    virtualized infrastructure. The key feature is immediate notification of
    unavailability of virtualized resources from VIM, to process recovery of
    VNFs on them.</para>

    <para>The Doctor project has also collaborated with the Availability
    project on identifying gaps in upstream projects, such as but not
    exclusively OpenStack. It has also worked towards implementing missing
    features and improving functionality, with a good example being the Aodh
    event based alarms, which allow for fast notifications when certain
    predefined events occur.</para>

    <para>The Doctor project also produced an architectural design and a
    reference implementation based on opensource components, which will be
    presented later on in this document.</para>
  </section>

  <section id="doctor_arch">
    <title>Doctor Architecture</title>

    <para>The Doctor project documentation shows the detailed architecture for
    Fault Management and NFVI Maintenance. Being quite similar with each
    other, the focus in the following sections shall remain on Fault
    Management.</para>

    <para>The architecture specifies a set of functional blocks:</para>

    <itemizedlist>
      <listitem>
        <para><emphasis role="bold">Monitor</emphasis> - monitors the
        virtualized infrastructure, capturing fault events in software and
        hardware. For this component we choose <emphasis
        role="bold">Zabbix</emphasis> which is integrated into the platform
        through the Fuel Zabbix Plugin, available upstream.</para>
      </listitem>

      <listitem>
        <para><emphasis role="bold">Inspector</emphasis> - this component
        receives notifications from Monitor components and OpenStack core
        components, allowing it to create logical relationships between
        entities, identify affected resources when faults occur, and to
        communicate with Controllers in order to update the states of the
        virtual and physical resources.</para>

        <para>For this component Enea NFV Core <xi:include
        href="../../s_docsrc_common/pardoc-distro.xml"
        xmlns:xi="http://www.w3.org/2001/XInclude"
        xpointer="element(EneaNFV_REL_VER/1)" /> makes use of Vitrage, an
        OpenStack related project used for Root Cause Analysis. The
        integration into the platform is done with the help of a Fuel Plugin
        which has been developed internally by Enea.</para>
      </listitem>

      <listitem>
        <para><emphasis role="bold">Controller - </emphasis>OpenStack core
        components act as Controllers responsible for maintaining the resource
        map between physical and virtual resources. They accept update
        requests from the Inspector and are responsible for sending failure
        event notifications to the Notifier. Components such as Nova, Neutron,
        Glance, and Heat, act as Controllers in the Doctor
        Architecture.</para>
      </listitem>

      <listitem>
        <para><emphasis role="bold">Notifier</emphasis> - the focus of this
        component is on selecting and aggregating failure events received from
        the controller, based on policies mandated by the Consumer. The role
        of the Notifier is filled by the Aodh component in OpenStack.</para>
      </listitem>
    </itemizedlist>

    <para>Alongside the Doctor components, there are a few other blocks
    mentioned:</para>

    <itemizedlist>
      <listitem>
        <para><emphasis role="bold">Administrator</emphasis> - this represents
        the human role of administrating the platform by means of dedicated
        interfaces. These can be visual dashboards like OpenStack Horizon or
        Fuel Dashboard, or via CLI tools like the OpenStack unified CLI, that
        can be accessed from one of the servers that act as OpenStack
        Controller nodes.</para>

        <para>In Enea NFV Core <xi:include
        href="../../s_docsrc_common/pardoc-distro.xml"
        xmlns:xi="http://www.w3.org/2001/XInclude"
        xpointer="element(EneaNFV_REL_VER/1)" /> the Administrator can also
        access the Zabbix dashboard to perform supplementary configurations.
        The same applies for the Vitrage tool, which comes with its own
        Horizon dashboard, enabling the user to visually inspect the faults
        reported by the monitoring tools through visual representations of the
        virtual and physical resources, the relationships between them and the
        fault correlation.</para>

        <para>For Vitrage, users will usually want to configure additional
        use-cases and describe relationships between components via template
        files written in <literal>yaml</literal> format.</para>
      </listitem>

      <listitem>
        <para><emphasis role="bold">Consumer</emphasis> - this block is
        vaguely described in the Doctor Architecture and is out of its current
        scope. Doctor only deals with fault detection and management, but
        since the actual VNFs are managed, according to the ETSI architecture,
        by a different entity, Doctor does not deal with recovery actions of
        the VNFs. The role of the Consumer thus falls to that of a VNF Manager
        and Orchestrator.</para>

        <para>Enea NFV Core <xi:include
        href="../../s_docsrc_common/pardoc-distro.xml"
        xmlns:xi="http://www.w3.org/2001/XInclude"
        xpointer="element(EneaNFV_REL_VER/1)" /> provides VNF management
        capabilities using Tacker, which is an OpenStack project that
        implements a generic VNF Manager and Orchestrator, according to the
        ETSI MANO Architectural Framework.</para>
      </listitem>
    </itemizedlist>

    <para>The functional blocks overview in the picture below has been
    complemented to show the components used for realizing the Doctor
    Architecture:</para>

    <figure>
      <title>Functional Blocks Overview</title>

      <mediaobject>
        <imageobject>
          <imagedata align="center" fileref="images/functional_blocks.png"
                     scale="70" />
        </imageobject>
      </mediaobject>
    </figure>

    <section id="dr_fault_mg">
      <title>Doctor Fault Management</title>

      <para>The architecture described in the Doctor project has been
      demonstrated in various PoCs and demos, but always using sample
      components for either the consumer or the monitor. Enea has worked with
      upstream projects Doctor and Vitrage, to realize the goals of the Doctor
      project by using real components as described above.</para>

      <para>The two pictures below show a typical fault management
      scenario:</para>

      <figure>
        <title>Fault Management scenario (A)</title>

        <mediaobject>
          <imageobject>
            <imagedata align="center" fileref="images/dr_fault_mg.png"
                       scale="70" />
          </imageobject>
        </mediaobject>
      </figure>

      <figure>
        <title>Fault Management scenario (B)</title>

        <mediaobject>
          <imageobject>
            <imagedata align="center" fileref="images/dr_fault_mg_2.png"
                       scale="70" />
          </imageobject>
        </mediaobject>
      </figure>

      <para>Enea NFV Core <xi:include
      href="../../s_docsrc_common/pardoc-distro.xml"
      xmlns:xi="http://www.w3.org/2001/XInclude"
      xpointer="element(EneaNFV_REL_VER/1)" /> uses the same approach
      described above:</para>

      <orderedlist>
        <listitem>
          <para>When creating a VNF, the user will have to enable the
          monitoring capabilities of Tacker by passing a template, which
          specifies that an alarm will be created when the VM represented by
          this VNF changes state. The support for alarm monitoring in Tacker
          is detailed in the Alarm Monitoring Framework spec in the OpenStack
          documentation.</para>

          <para>Tacker should be able to create a VNF and then an Aodh alarm
          of type event, triggerable when the instance is in a state of ERROR.
          When this event is triggered perform an HTTP call to a URL managed
          by Tacker. As a result of this action, Tacker can detect when an
          instance has failed (for whatever reason) and will respawn it
          somewhere else.</para>
        </listitem>

        <listitem>
          <para>The subscribed response in this case is an empty operation,
          the Notifier (Aodh) only has to confirm that the alarm has been
          created.</para>
        </listitem>

        <listitem>
          <para>The NFVI sends monitoring events for the resources the VIM has
          been subscribed to.</para>

          <note>
            <para>This subscription message exchange between the VIM and NFVI
            is not shown in this message flow. This step is related to
            Vitrage's capability of receiving notifications from OpenStack
            services. At this moment Vitrage supports notifications from
            <literal>nova.host</literal>, <literal>nova.instances</literal>,
            <literal>nova.zone</literal>, <literal>cinder.volume</literal>,
            <literal>neutron.network</literal>,
            <literal>neutron.port</literal> and <literal>heat.stack</literal>
            OpenStack datasources.</para>
          </note>
        </listitem>

        <listitem>
          <para>This step describes faults detected by Zabbix which are sent
          to the Inspector (Vitrage) as soon as detected. This is done using a
          push approach by means of sending an AMQP message to a dedicated
          message queue managed by Vitrage. For example, if
          <literal>nova-compute</literal> fails on one of the compute nodes,
          Zabbix will format a message specifying all the needed details
          required for processing the fault: a timestamp, what host failed,
          what event occurred etc.</para>
        </listitem>

        <listitem>
          <para>This step shows the database lookup geared to find the virtual
          resources affected by the detected fault. Vitrage will perform
          various calculations to detect what virtual resources are affected
          by the raw failure presented by Zabbix.</para>

          <para>Vitrage can be configured via templates to correlate instances
          with the physical hosts they are running on, so that if a compute
          node fails, then instances running on that host will be affected. A
          typical use-case is to mark the compute node down
          (<literal>mark_host_down</literal>) and update the states of all
          instances running on them. This is done by issuing Nova API calls
          for each of these instances.</para>

          <para>Step 5c. shows the Controller (Nova in this case) acting upon
          the state change of the instance and issuing an event alarm to
          Aodh.</para>
        </listitem>

        <listitem>
          <para>The Notifier will acknowledge the alarm event request from
          Nova and will trigger the alarm(s) created by Tacker in step 1.
          Since Tacker has configured the alarm to send an HTTP request, Aodh
          will perform that HTTP call at the URL managed by Tacker.</para>
        </listitem>

        <listitem>
          <para>The Consumer (Tacker) will react to the HTTP call and perform
          the action configured by the user (e.g. respawn the VNF).</para>
        </listitem>

        <listitem>
          <para>The action is sent to the Controller (Nova) so that the VNF is
          recreated.</para>
        </listitem>
      </orderedlist>

      <note condition="hidden">
        <para>The ENEA NFV Core <xi:include
        href="../../s_docsrc_common/pardoc-distro.xml"
        xmlns:xi="http://www.w3.org/2001/XInclude"
        xpointer="element(EneaNFV_REL_VER/1)" /> Pre-Release fully covers the
        required Doctor functionality only for the Vitrage and Zabbix
        components.</para>
      </note>
    </section>

    <section id="zabbix">
      <title>Zabbix Configuration for Push Notifications</title>

      <para>Vitrage supports Zabbix datasource by regularly polling the Zabbix
      agents, which need to be configured in advance.</para>

      <para>The Vitrage plugin developed internally by Enea can automatically
      configure Zabbix so that everything works as expected. Polling however,
      is not fast enough for a telco use-case, so it is necessary to configure
      push notifications for Zabbix . This requires manual configuration on
      one of the controller nodes, which will be used by the centralized
      database making the configuration available on all the other
      nodes.</para>

      <para>The Zabbix configuration dashboard is available at the same IP
      address where OpenStack can be reached, e.g.
      <literal>http://10.0.101.42/zabbix</literal>. Default username and
      password are <literal>admin</literal> and <literal>zabbix</literal>
      respectively.</para>

      <figure>
        <title>Zabbix Dashboard</title>

        <mediaobject>
          <imageobject>
            <imagedata align="center" fileref="images/Zabbix_Home.png"
                       scale="70" />
          </imageobject>
        </mediaobject>
      </figure>

      <para>To forward Zabbix events to Vitrage, a new media type needs to be
      created and associated with a Zabbix Admin user.</para>

      <para><emphasis role="bold">Creating the Media Type which calls
      zabbix_vitrage.py</emphasis></para>

      <orderedlist>
        <listitem>
          <para>Create a new media type by selecting <literal>Media
          Types</literal> under the <literal>Admininstration</literal> tab and
          choosing <literal>Create Media Type</literal>.</para>

          <para>Now fill each parameter field with the values given below,
          before clicking <literal>add</literal>:</para>

          <itemizedlist>
            <listitem>
              <para>Name: Vitrage Notifications</para>
            </listitem>

            <listitem>
              <para>Type: Script</para>
            </listitem>

            <listitem>
              <para>Script name: <filename>zabbix_vitrage.py</filename></para>
            </listitem>
          </itemizedlist>

          <figure>
            <title>Creating the Media Type</title>

            <mediaobject>
              <imageobject>
                <imagedata align="center"
                           fileref="images/Zabbix_MediaType.png" scale="70" />
              </imageobject>
            </mediaobject>
          </figure>
        </listitem>

        <listitem>
          <para>Under <literal>Administration</literal>, in the
          <literal>Users</literal> tab (next to <literal>Media
          Types</literal>), click on the drop-down menu <literal>User
          groups</literal> and select <literal>Users</literal>, to display the
          list of existing users. Choose from this list, the
          <literal>admin</literal> alias:</para>

          <figure>
            <title>Existing Users</title>

            <mediaobject>
              <imageobject>
                <imagedata align="center"
                           fileref="images/Zabbix_AdminSelectUsers.png"
                           scale="90" />
              </imageobject>
            </mediaobject>
          </figure>

          <figure>
            <title>Choosing an Alias</title>

            <mediaobject>
              <imageobject>
                <imagedata align="center"
                           fileref="images/Zabbix_AdminUser.png" scale="70" />
              </imageobject>
            </mediaobject>
          </figure>
        </listitem>

        <listitem>
          <para>Inside the <literal>User</literal> tab for the
          <literal>admin</literal> alias, click <literal>Add</literal> next to
          the <literal>Groups</literal> list:</para>

          <figure>
            <title>Adding Groups</title>

            <mediaobject>
              <imageobject>
                <imagedata align="center" fileref="images/Zabbix_User.png"
                           scale="70" />
              </imageobject>
            </mediaobject>
          </figure>
        </listitem>

        <listitem>
          <para>Once the <literal>User groups</literal> popup menu
          appears:</para>

          <itemizedlist>
            <listitem>
              <para>Check the Zabbix Administrators checkbox</para>
            </listitem>

            <listitem>
              <para>Click <literal>Select</literal></para>
            </listitem>

            <listitem>
              <para>Click <literal>Update</literal></para>
            </listitem>
          </itemizedlist>

          <figure>
            <title>Updating the User Groups</title>

            <mediaobject>
              <imageobject>
                <imagedata align="center"
                           fileref="images/Zabbix_AdminUserSelectGroup.png"
                           scale="70" />
              </imageobject>
            </mediaobject>
          </figure>
        </listitem>

        <listitem>
          <para>Reselect the <literal>admin</literal> alias to enter the
          <literal>Configuration of Users</literal> screen once again.</para>

          <para>Select the <literal>Media</literal> tab and click
          <literal>Add</literal> to create and modify a new media type. A
          popup menu will appear for <literal>New Media</literal>. Fill each
          field with the values below:</para>

          <itemizedlist>
            <listitem>
              <para>Type: Vitrage Notifications</para>
            </listitem>

            <listitem>
              <para>Send to: <literal>e.g.
              rabbit://nova:Q1Vt9iocA8OpXKOPoUYOLKie@192.168.0.4:5673/</literal></para>

              <para>This url can be found in either
              <literal>/etc/vitrage/vitrage.conf or in
              /etc/nova/nova.conf</literal>. Only one url is needed and it is
              vital that the url contains the ending " / ".</para>
            </listitem>

            <listitem>
              <para>When active: 1-7, 00:00-24:00</para>
            </listitem>

            <listitem>
              <para>Use if severity: (all)</para>
            </listitem>

            <listitem>
              <para>Status: Enabled</para>
            </listitem>

            <listitem>
              <para>Click <literal>Add</literal> and then
              <literal>Update</literal> to save your changes.</para>
            </listitem>
          </itemizedlist>

          <figure>
            <title>Creating the Vitrage Notifications</title>

            <mediaobject>
              <imageobject>
                <imagedata align="center"
                           fileref="images/Zabbix_AdminUserMediaAdd.png"
                           scale="70" />
              </imageobject>
            </mediaobject>
          </figure>
        </listitem>

        <listitem>
          <para>To configure an action, under the
          <literal>Configuration</literal> tab select
          <literal>Actions</literal>. Then, under the
          <literal>Action</literal> tab, fill in the parameter fields with the
          values below:</para>

          <itemizedlist>
            <listitem>
              <para>Name: Forward to Vitrage</para>
            </listitem>

            <listitem>
              <para>Default Subject: {TRIGGER.STATUS}</para>
            </listitem>

            <listitem>
              <para>Default Message:</para>

              <programlisting>host={HOST.NAME1}
hostid={HOST.ID1}
hostip={HOST.IP1}
triggerid={TRIGGER.ID}
description={TRIGGER.NAME}
rawtext={TRIGGER.NAME.ORIG}
expression={TRIGGER.EXPRESSION}
value={TRIGGER.VALUE}
priority={TRIGGER.NSEVERITY}
lastchange={EVENT.DATE} {EVENT.TIME}</programlisting>

              <figure>
                <title>Configuring an action</title>

                <mediaobject>
                  <imageobject>
                    <imagedata align="center"
                               fileref="images/Zabbix_Action.png" scale="70" />
                  </imageobject>
                </mediaobject>
              </figure>
            </listitem>
          </itemizedlist>
        </listitem>

        <listitem>
          <para>To send events, in the <literal>Conditions</literal> tab shown
          below, make sure the label <literal>Maintenance status not in
          "maintenance"</literal> is added and remove the label
          <literal>Trigger value = PROBLEM</literal>, before saving your
          changes.</para>

          <figure>
            <title>Configuring Conditions</title>

            <mediaobject>
              <imageobject>
                <imagedata align="center"
                           fileref="images/Zabbix_ActionConditions.png"
                           scale="70" />
              </imageobject>
            </mediaobject>
          </figure>
        </listitem>

        <listitem>
          <para>In the <literal>Operations</literal> tab still under
          <literal>Configuration</literal> and <literal>Actions</literal>,
          configure the users you wish to send to.</para>

          <itemizedlist>
            <listitem>
              <para>In <literal>Operations Details</literal>, set:</para>

              <itemizedlist>
                <listitem>
                  <para>Operation Type: Send message</para>
                </listitem>

                <listitem>
                  <para>Send to Users: add <literal>admin</literal></para>
                </listitem>

                <listitem>
                  <para>Send only to: Vitrage Notifications</para>
                </listitem>
              </itemizedlist>

              <figure>
                <title>Configuring Operations</title>

                <mediaobject>
                  <imageobject>
                    <imagedata align="center"
                               fileref="images/Zabbix_ActionOperations.png"
                               scale="70" />
                  </imageobject>
                </mediaobject>
              </figure>
            </listitem>
          </itemizedlist>
        </listitem>

        <listitem>
          <para>In the <literal>Configuration</literal> tab, under
          <literal>Templates</literal>, look for and select the template named
          <filename>Template App Zabbix Agent</filename> and click on
          <literal>Items</literal>.</para>
        </listitem>

        <listitem>
          <para>Select the item <literal>Agent ping</literal> to open and edit
          it:</para>

          <figure>
            <title>Configuring Items</title>

            <mediaobject>
              <imageobject>
                <imagedata align="center"
                           fileref="images/Zabbix_ConfigTemplatesZabbixAgent.png"
                           scale="70" />
              </imageobject>
            </mediaobject>
          </figure>
        </listitem>

        <listitem>
          <para>Set the value in the field <literal>Update Interval (is
          sec)</literal> to 10 seconds, and click
          <literal>Update</literal>:</para>

          <figure>
            <title>Setting a new Update Interval</title>

            <mediaobject>
              <imageobject>
                <imagedata align="center"
                           fileref="images/Zabbix_ConfigTemplateZabbixAgentPing.png"
                           scale="70" />
              </imageobject>
            </mediaobject>
          </figure>
        </listitem>

        <listitem>
          <para>After returning to the <literal>Configuration of
          templates</literal> screen, select<literal> Triggers </literal>to
          open and edit existing triggers.</para>
        </listitem>

        <listitem>
          <para>Select <literal>Zabbix agent on {HOST.NAME} is unreachable for
          5 minutes</literal> to open and edit the trigger:</para>

          <figure>
            <title>Selecting a Trigger to edit</title>

            <mediaobject>
              <imageobject>
                <imagedata align="center"
                           fileref="images/Zabbix_ConfigTemplateZabbixAgentTrigger1.png"
                           scale="70" />
              </imageobject>
            </mediaobject>
          </figure>
        </listitem>

        <listitem>
          <para>Edit the <literal>Name</literal> and
          <literal>Expression</literal> fields for this trigger to check the
          condition every 20 seconds. Set the severity to
          <literal>HIGH</literal> and click <literal>Update</literal>:</para>

          <figure>
            <title>Configuring Trigger Settings</title>

            <mediaobject>
              <imageobject>
                <imagedata align="center"
                           fileref="images/Zabbix_ConfigTemplateZabbixAgentTrigger2.png"
                           scale="70" />
              </imageobject>
            </mediaobject>
          </figure>
        </listitem>
      </orderedlist>

      <para>Using these instructions, Zabbix will call the
      <literal>zabbix_vitrage.py</literal> script, made readily available by
      the Fuel Vitrage Plugin, to pass the arguments described above. This
      script will then interpret the parameters and format an AMQP message to
      be sent to the <literal>vitrage.notifications</literal> queue, managed
      by the vitrage-graph service.</para>
    </section>

    <section id="vitrage_config">
      <title>Vitrage Configuration</title>

      <para>The Vitrage team has been collaborating with the OPNFV Doctor
      project in order to support Vitrage as an Inspector Component. The
      Doctor use-case for Vitrage is described in an OpenStack blueprint. Enea
      NFV Core has complemented Vitrage with the ability to set the states of
      failed instances by implementing an action type in Vitrage. This action
      calls Nova APIs to set instances in error state. An action type which
      allows fencing failed hosts also exists.</para>

      <para>In order to make use of these features, Vitrage supports
      additional configurations via <literal>yaml</literal> templates that
      must be placed in <literal>/etc/vitrage/templates</literal> on the node
      that have the Vitrage role.</para>

      <figure>
        <title>Available Additional Templates</title>

        <mediaobject>
          <imageobject>
            <imagedata align="center"
                       fileref="images/Vitrage_ConfigurationRole.png"
                       scale="70" />
          </imageobject>
        </mediaobject>
      </figure>

      <para>The example below shows how to program Vitrage to mark failed
      compute hosts as down and then to change the state of the instances to
      ERROR, by creating Vitrage deduced alarms.</para>

      <programlisting>metadata:
  name: test_mark_inst_err
  description: test description
definitions:
  entities:
    - entity:
        category: ALARM
        type: zabbix
        severity: HIGH
        template_id: zabbix_alarm
    - entity:
        category: RESOURCE
        type: nova.host
        template_id: host
    - entity:
        category: RESOURCE
        type: nova.instance
        template_id: instance
  relationships:
    - relationship:
        source: zabbix_alarm
        relationship_type: on
        target: host
        template_id : critical_problem_on_host
    - relationship:
        source: host
        target: instance
        relationship_type: contains
        template_id : host_contains_instance
scenarios:
  - scenario:
      condition: critical_problem_on_host and host_contains_instance
      actions:
        - action:
            action_type: mark_down
            action_target:
              target: host
        - action:
            action_type: set_state
            action_target:
              target: instance
            properties:
              state: ERROR
        - action:
            action_type: set_instance_state
            action_target:
              target: instance</programlisting>

      <para>For the action type of fencing, a similar action item must be
      added:</para>

      <programlisting>- scenario:
    condition: critical_problem_on_host
    actions:
     - action:
        action_type: fence
        action_target:
         target: host</programlisting>

      <para>After a template is configured, a restart of the
      <literal>vitrage-api</literal> and <literal>vitrage-graph</literal>
      services is needed:</para>

      <programlisting>root@node-6:~# systemctl restart vitrage-api
root@node-6:~# systemctl restart vitrage-graph</programlisting>
    </section>

    <section id="vitrage_custom">
      <title>Vitrage Customizations</title>

      <para>Enea NFV Core <xi:include
      href="../../s_docsrc_common/pardoc-distro.xml"
      xmlns:xi="http://www.w3.org/2001/XInclude"
      xpointer="element(EneaNFV_REL_VER/1)" /> has added custom features for
      Vitrage which allow two kinds of actions:</para>

      <orderedlist>
        <listitem>
          <para>Perform actions Northbound of the VIM:</para>

          <itemizedlist>
            <listitem>
              <para>Nova force host down on compute</para>
            </listitem>

            <listitem>
              <para>Setting instance state to ERROR in nova. This is used in
              conjunction with an alarm created by Tacker, as described
              before, and should allow Tacker to detect when an instance is
              affected and take proper actions.</para>
            </listitem>
          </itemizedlist>
        </listitem>

        <listitem>
          <para>Perform actions Southbound of the VIM:</para>

          <para>Vitrage templates allow us to program fencing actions for
          hosts with failed services. In the event that
          <literal>systemd</literal> is unable to recover from a critical
          process or a type of sofware error ocurs on the Hardware supporting
          them, the fencing of Node can be programmed, and it in turn will
          perform a reboot, attempting to recover the failed node.</para>
        </listitem>
      </orderedlist>
    </section>
  </section>

  <section condition="hidden" id="pm_high_avail">
    <title>Pacemaker High Availability</title>

    <para>Many of the OpenStack solutions which offer High Availability
    characteristics employ Pacemaker for achieving highly available OpenStack
    services. Traditionally Pacemaker has been used for managing only the
    control plane services, so it can effectively provide redundancy and
    recovery for the Controller nodes only. A reason for this is that
    Controller nodes and Compute nodes essentially have very different High
    Availability requirements that need to be considered.</para>

    <para>Typically, for Controller nodes, the services that run on them are
    stateless, with few exceptions, where only one instance of a given service
    is allowed, but for which redundancy is still desired. A good example
    would be an AMQP service (e.g. RabbitMQ). Compute nodes H.A. requirements
    depend on the type of services that run on them, but typically it is
    desired that failures on these nodes be detected as soon as possible so
    that the instances that run on them can be either migrated, resurrected or
    restarted. Sometimes failures on the physical hosts do not necessarily
    cause a failure on the services (VNFs), but having these services
    incapacitated can prevent access to and controlling the services.</para>

    <para>Controller High Availability is thus a subject generally well
    understood and experimented with, and the basis for this is Pacemaker
    using Corosync underneath.</para>

    <para>Extending the use of Pacemaker to Compute nodes was thought as a
    possible solution for providing VNF high availability, but the problem
    turned out to be more complicated. On one hand, Pacemaker as a clustering
    tool, can only scale properly up to a limited number of nodes, usually
    less than 128. This poses a problem for large scale deployments where
    hundreds of compute nodes are required. On the other hand, Compute node
    H.A. requires other considerations and calls for specially designed
    solutions.</para>

    <section id="pm_remote">
      <title>Pacemaker Remote</title>

      <para>As mentioned earlier, Pacemaker and corosync do not scale well
      over a large cluster, since each node has to talk to every other,
      essentially creating a mesh configuration. A solution to this problem
      could be partitioning the cluster into smaller groups, but this has its
      limitations and it is generally difficult to manage.</para>

      <para>A better solution is using <literal>pacemaker-remote</literal>, a
      feature of Pacemaker, which allows for extending the cluster beyond the
      usual limits by using the Pacemaker monitoring capabilities. It
      essentially creates a new type of resource which enables adding light
      weight nodes to the cluster. More information about pacemaker-remote can
      be found on the official clusterlabs website.</para>

      <para>Please note that at this moment Pacemaker remote must be
      configured manually after deployment. Here are the manual steps for
      doing so:</para>

      <orderedlist>
        <listitem>
          <para>Log onto the Fuel Master using the default credentials, if
          they have not been changed (root/r00tme).</para>
        </listitem>

        <listitem>
          <para>Type fuel node to obtain the list of nodes, their roles and
          the IP addresses.</para>

          <programlisting>[root@fuel ~]# fuel node
id | status | name             | cluster | ip        | mac               | roles    /
                 | pending_roles | online | group_id
---+--------+------------------+---------+-----------+-------------------+----------/
-----------------+---------------+--------+---------
 1 | ready  | Untitled (8c:d4) |       1 | 10.20.0.4 | 68:05:ca:46:8c:d4 | ceph-osd,/
 controller      |               |      1 |        1
 4 | ready  | Untitled (8c:c2) |       1 | 10.20.0.6 | 68:05:ca:46:8c:c2 | ceph-osd,/
 compute         |               |      1 |        1
 5 | ready  | Untitled (8c:c9) |       1 | 10.20.0.7 | 68:05:ca:46:8c:c9 | ceph-osd,/
 compute         |               |      1 |        1
 2 | ready  | Untitled (8b:64) |       1 | 10.20.0.3 | 68:05:ca:46:8b:64 | /
controller, mongo, tacker |               |      1 |        1
 3 | ready  | Untitled (8c:45) |       1 | 10.20.0.5 | 68:05:ca:46:8c:45 | /
controller, vitrage       |               |      1 |        1</programlisting>
        </listitem>

        <listitem>
          <para>Each controller has a unique Pacemaker authkey. One needs to
          be kept and propagated to the other servers. Assuming node-1, node-2
          and node-3 are the controllers, execute the following from the Fuel
          console:</para>

          <programlisting>[root@fuel ~]# scp node-1:/etc/pacemaker/authkey .
[root@fuel ~]# scp authkey node-2:/etc/pacemaker/
[root@fuel ~]# scp authkey node-3:/etc/pacemaker/
[root@fuel ~]# scp authkey node-3:/etc/pacemaker/
[root@fuel ~]# scp authkey node-4:~
[root@fuel ~]# scp authkey node-5:~</programlisting>
        </listitem>

        <listitem>
          <para>For each compute node, log on to it using the corresponding
          IP</para>
        </listitem>

        <listitem>
          <para>Install the required packages:</para>

          <programlisting>root@node-4:~# apt-get install pacemaker-remote resource-agents crmsh</programlisting>
        </listitem>

        <listitem>
          <para>Copy the authkey from the Fuel Master and make sure the right
          permissions are set:</para>

          <programlisting>[root@node-4:~]# cp authkey /etc/pacemaker
[root@node-4:~]# chown root:haclient /etc/pacemaker/authkey</programlisting>
        </listitem>

        <listitem>
          <para>Add an iptables rule for the default port (3121). Save it also
          to <literal>/etc/iptables/rules.v4</literal> to make it
          persistent:</para>

          <programlisting>root@node-4:~# iptables -A INPUT -s 192.168.0.0/24 -p tcp -m multiport /
--dports 3121 -m comment --comment "pacemaker_remoted from 192.168.0.0/24" -j ACCEPT</programlisting>
        </listitem>

        <listitem>
          <para>Start the pacemaker-remote service:</para>

          <programlisting>[root@node-4:~]# systemctl start pacemaker-remote.service</programlisting>
        </listitem>

        <listitem>
          <para>Log onto one of the controller nodes and configure the
          pacemaker-remote resources:</para>

          <programlisting>[root@node-1:~]# pcs resource create node-4.domain.tld remote
[root@node-1:~]# pcs constraint location node-4.domain.tld prefers /
node-1.domain.tld=100 node-2.domain.tld=100 node-3.domain.tld=100
[root@node-1:~]# pcs constraint location node-4.domain.tld avoids node-5.domain.tld
[root@node-1:~]# pcs resource create node-5.domain.tld remote
[root@node-1:~]# pcs constraint location node-5.domain.tld prefers /
node-1.domain.tld=100 node-2.domain.tld=100 node-3.domain.tld=100
[root@node-1:~]# pcs constraint location node-5.domain.tld avoids node-4.domain.tld</programlisting>
        </listitem>

        <listitem>
          <para>Remote nodes should now appear online:</para>

          <programlisting>[root@node-1:~]# pcs status
Cluster name: OpenStack
Last updated: Thu Aug 24 12:00:21 2017		Last change: Thu Aug 24 11:57:32 2017 /
by root via cibadmin on node-1.domain.tld
Stack: corosync
Current DC: node-1.domain.tld (version 1.1.14-70404b0) - partition with quorum
5 nodes and 78 resources configured

Online: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld ]
RemoteOnline: [ node-4.domain.tld node-5.domain.tld ]</programlisting>
        </listitem>
      </orderedlist>
    </section>

    <section id="pm_fencing">
      <title>Pacemaker Fencing</title>

      <para>ENEA NFV Core <xi:include
      href="../../s_docsrc_common/pardoc-distro.xml"
      xmlns:xi="http://www.w3.org/2001/XInclude"
      xpointer="element(EneaNFV_REL_VER/1)" /> makes use of the fencing
      capabilities of Pacemaker to isolate faulty nodes and trigger recovery
      actions by means of power cycling the failed nodes. Fencing is
      configured by creating <literal>STONITH</literal> type resources for
      each of the servers in the cluster, both Controller nodes and Compute
      nodes. The <literal>STONITH</literal> adapter for fencing the nodes is
      <literal>fence_ipmilan</literal>, which makes use of the IPMI
      capabilities of the ThunderX servers.</para>

      <para>Here are the steps for enabling fencing capabilities on a
      cluster:</para>

      <orderedlist>
        <listitem>
          <para>Log onto the Fuel Master using the default credentials, if not
          they have not been changed (root/r00tme).</para>
        </listitem>

        <listitem>
          <para>Type fuel node to obtain the list of nodes, their roles and
          the IP addresses:</para>

          <programlisting>[root@fuel ~]# fuel node
id | status | name             | cluster | ip        | mac               | roles    /
                 | pending_roles | online | group_id
---+--------+------------------+---------+-----------+-------------------+----------/
-----------------+---------------+--------+---------
 1 | ready  | Untitled (8c:d4) |       1 | 10.20.0.4 | 68:05:ca:46:8c:d4 | ceph-osd,/
 controller      |               |      1 |        1
 4 | ready  | Untitled (8c:c2) |       1 | 10.20.0.6 | 68:05:ca:46:8c:c2 | ceph-osd,/
 compute         |               |      1 |        1
 5 | ready  | Untitled (8c:c9) |       1 | 10.20.0.7 | 68:05:ca:46:8c:c9 | ceph-osd,/
 compute         |               |      1 |        1
 2 | ready  | Untitled (8b:64) |       1 | 10.20.0.3 | 68:05:ca:46:8b:64 | /
controller, mongo, tacker |               |      1 |        1
 3 | ready  | Untitled (8c:45) |       1 | 10.20.0.5 | 68:05:ca:46:8c:45 | /
controller, vitrage       |               |      1 |        1</programlisting>
        </listitem>

        <listitem>
          <para>Log onto each server to install additional packages:</para>

          <programlisting>[root@node-1:~]# apt-get install fence-agents ipmitool</programlisting>
        </listitem>

        <listitem>
          <para>Configure Pacemaker fencing resources. This needs to be done
          once on one of the controllers. The parameters will vary, depending
          on the BMC addresses of each node and credentials.</para>

          <programlisting>[root@node-1:~]# crm configure primitive ipmi-fencing-node-1 /
stonith::fence_ipmilan params pcmk_host_list="node-1.domain.tld" /
ipaddr=10.0.100.151 login=ADMIN passwd=ADMIN op monitor interval="60s"
[root@node-1:~]# crm configure primitive ipmi-fencing-node-2 /
stonith::fence_ipmilan params pcmk_host_list="node-2.domain.tld" /
ipaddr=10.0.100.152 login=ADMIN passwd=ADMIN op monitor interval="60s"
[root@node-1:~]# crm configure primitive ipmi-fencing-node-3 /
stonith::fence_ipmilan params pcmk_host_list="node-3.domain.tld" /
ipaddr=10.0.100.153 login=ADMIN passwd=ADMIN op monitor interval="60s"
[root@node-1:~]# crm configure primitive ipmi-fencing-node-4 /
stonith::fence_ipmilan params pcmk_host_list="node-4.domain.tld" /
ipaddr=10.0.100.154 login=ADMIN passwd=ADMIN op monitor interval="60s"
[root@node-1:~]# crm configure primitive ipmi-fencing-node-5 /
stonith::fence_ipmilan params pcmk_host_list="node-5.domain.tld" /
ipaddr=10.0.100.155 login=ADMIN passwd=ADMIN op monitor interval="60s"</programlisting>
        </listitem>

        <listitem>
          <para>Activate fencing by enabling the <literal>stonith</literal>
          property in Pacemaker (disabled by default). This also needs to be
          done only once, on one of the controllers.</para>

          <programlisting>[root@node-1:~]# pcs property set stonith-enabled=true</programlisting>
        </listitem>
      </orderedlist>
    </section>
  </section>

  <section condition="hidden" id="ops_resources_agents">
    <title>OpenStack Resource Agents</title>

    <para>The OpenStack community has been working for some time on
    identifying possible solutions for enabling High Availability for Compute
    nodes, after a period of belief that this subject was not something that
    should concern the cloud platform. Over time it became obvious that even
    on a true cloud platform, where services are designed to run without being
    affected by the availability of the cloud platform, fault management and
    recovery are still very important and desirable. This is also the case for
    NFV applications, where in the good tradition of telecom applications, the
    operators must have complete engineering control over the resources they
    own and manage.</para>

    <para>The work for Compute node High Availability is captured in an
    OpenStack user story and documented upstream, showing proposed solutions,
    summit talks and presentations. A number of these solutions make use of
    OpenStack Resource Agents, which are a set of specialized Pacemaker
    resources capable of identifying failures in compute nodes and can perform
    automatic evacuation of the instances affected by these failures.</para>

    <para>ENEA NFV Core <xi:include
    href="../../s_docsrc_common/pardoc-distro.xml"
    xmlns:xi="http://www.w3.org/2001/XInclude"
    xpointer="element(EneaNFV_REL_VER/1)" /> aims to validate and integrate
    this work and to make this feature available in the platform aimed as an
    alternative to the Doctor framework, where simple, autonomous recovery of
    running instances is desired.</para>
  </section>
</chapter>