High Availability GuideEnea NFV Core has been designed to provide high
availability characteristics that are needed for developing and deploying
telco-grade NFV solutions on top of our OPNFV based platform. The High
Availability subject in general is very wide and still an important focus in
both opensource communities and the independent/proprietary solutions
market.Enea NFV Core aims to initially leverage the
efforts in the upstream OPNFV and OpenStack opensource projects, combining
solutions from both worlds in an effort to provide flexibility and use-case
coverage. Enea has long term expertise and proprietary solutions addressing
High Availability for telco applications, which are subject to integration
with the NFV based solutions.High Availability LevelsThe foundation for the feature set available in Enea NFV Core is
divided into three levels:Hardware FaultNFV Platform H.A.VNF High AvailabilityThe same division of levels for fault management can be seen in the
scope of the High Availability for OPNFV ("Availability") project. OPNFV
also hosts Doctor, a fault management and maintenance project designed to
develop and perform the consequent implementation of the OPNFV reference
platform. These two projects complement each other.The Availability project addresses H.A. requirements and solutions
from the perspective of the three levels mentioned above. It produces high
level requirements and API definitions for High Availability for OPNFV, a
H.A. Gap Analysis Report for OpenStack and more recently, works on
optimizing existing OPNFV test frameworks, such as Yardstick, developing
test cases which realize H.A.-specific use-cases and scenarios derived
from the H.A. requirements.The Doctor project aims to build fault management and maintenance
framework for the high availability of Network Services, on top of a
virtualized infrastructure. The key feature is immediate notification of
unavailability of virtualized resources from VIM, to process recovery of
VNFs on them.The Doctor project has also collaborated with the Availability
project on identifying gaps in upstream projects, such as but not
exclusively OpenStack. It has also worked towards implementing missing
features and improving functionality, with a good example being the Aodh
event based alarms, which allow for fast notifications when certain
predefined events occur.The Doctor project also produced an architectural design and a
reference implementation based on opensource components, which will be
presented later on in this document.Doctor ArchitectureThe Doctor project documentation shows the detailed architecture for
Fault Management and NFVI Maintenance. Being quite similar with each
other, the focus in the following sections shall remain on Fault
Management.The architecture specifies a set of functional blocks:Monitor - monitors the
virtualized infrastructure, capturing fault events in software and
hardware. For this component we choose Zabbix which is integrated into the platform
through the Fuel Zabbix Plugin, available upstream.Inspector - this component
receives notifications from Monitor components and OpenStack core
components, allowing it to create logical relationships between
entities, identify affected resources when faults occur, and to
communicate with Controllers in order to update the states of the
virtual and physical resources.For this component Enea NFV Core makes use of Vitrage, an
OpenStack related project used for Root Cause Analysis. The
integration into the platform is done with the help of a Fuel Plugin
which has been developed internally by Enea.Controller - OpenStack core
components act as Controllers responsible for maintaining the resource
map between physical and virtual resources. They accept update
requests from the Inspector and are responsible for sending failure
event notifications to the Notifier. Components such as Nova, Neutron,
Glance, and Heat, act as Controllers in the Doctor
Architecture.Notifier - the focus of this
component is on selecting and aggregating failure events received from
the controller, based on policies mandated by the Consumer. The role
of the Notifier is filled by the Aodh component in OpenStack.Alongside the Doctor components, there are a few other blocks
mentioned:Administrator - this represents
the human role of administrating the platform by means of dedicated
interfaces. These can be visual dashboards like OpenStack Horizon or
Fuel Dashboard, or via CLI tools like the OpenStack unified CLI, that
can be accessed from one of the servers that act as OpenStack
Controller nodes.In Enea NFV Core the Administrator can also
access the Zabbix dashboard to perform supplementary configurations.
The same applies for the Vitrage tool, which comes with its own
Horizon dashboard, enabling the user to visually inspect the faults
reported by the monitoring tools through visual representations of the
virtual and physical resources, the relationships between them and the
fault correlation.For Vitrage, users will usually want to configure additional
use-cases and describe relationships between components via template
files written in yaml format.Consumer - this block is
vaguely described in the Doctor Architecture and is out of its current
scope. Doctor only deals with fault detection and management, but
since the actual VNFs are managed, according to the ETSI architecture,
by a different entity, Doctor does not deal with recovery actions of
the VNFs. The role of the Consumer thus falls to that of a VNF Manager
and Orchestrator.Enea NFV Core provides VNF management
capabilities using Tacker, which is an OpenStack project that
implements a generic VNF Manager and Orchestrator, according to the
ETSI MANO Architectural Framework.The functional blocks overview in the picture below has been
complemented to show the components used for realizing the Doctor
Architecture:Doctor Fault ManagementThe architecture described in the Doctor project has been
demonstrated in various PoCs and demos, but always using sample
components for either the consumer or the monitor. Enea has worked with
upstream projects Doctor and Vitrage, to realize the goals of the Doctor
project by using real components as described above.The two pictures below show a typical fault management
scenario:Enea NFV Core uses the same approach
described above:When creating a VNF, the user will have to enable the
monitoring capabilities of Tacker by passing a template, which
specifies that an alarm will be created when the VM represented by
this VNF changes state. The support for alarm monitoring in Tacker
is detailed in the Alarm Monitoring Framework spec in the OpenStack
documentation.Tacker should be able to create a VNF and then an Aodh alarm
of type event, triggerable when the instance is in a state of ERROR.
When this event is triggered perform an HTTP call to a URL managed
by Tacker. As a result of this action, Tacker can detect when an
instance has failed (for whatever reason) and will respawn it
somewhere else.The subscribed response in this case is an empty operation,
the Notifier (Aodh) only has to confirm that the alarm has been
created.The NFVI sends monitoring events for the resources the VIM has
been subscribed to.This subscription message exchange between the VIM and NFVI
is not shown in this message flow. This step is related to
Vitrage's capability of receiving notifications from OpenStack
services. At this moment Vitrage supports notifications from
nova.host, nova.instances,
nova.zone, cinder.volume,
neutron.network,
neutron.port and heat.stack
OpenStack datasources.This step describes faults detected by Zabbix which are sent
to the Inspector (Vitrage) as soon as detected. This is done using a
push approach by means of sending an AMQP message to a dedicated
message queue managed by Vitrage. For example, if
nova-compute fails on one of the compute nodes,
Zabbix will format a message specifying all the needed details
required for processing the fault: a timestamp, what host failed,
what event occurred etc.This step shows the database lookup geared to find the virtual
resources affected by the detected fault. Vitrage will perform
various calculations to detect what virtual resources are affected
by the raw failure presented by Zabbix.Vitrage can be configured via templates to correlate instances
with the physical hosts they are running on, so that if a compute
node fails, then instances running on that host will be affected. A
typical use-case is to mark the compute node down
(mark_host_down) and update the states of all
instances running on them. This is done by issuing Nova API calls
for each of these instances.Step 5c. shows the Controller (Nova in this case) acting upon
the state change of the instance and issuing an event alarm to
Aodh.The Notifier will acknowledge the alarm event request from
Nova and will trigger the alarm(s) created by Tacker in step 1.
Since Tacker has configured the alarm to send an HTTP request, Aodh
will perform that HTTP call at the URL managed by Tacker.The Consumer (Tacker) will react to the HTTP call and perform
the action configured by the user (e.g. respawn the VNF).The action is sent to the Controller (Nova) so that the VNF is
recreated.The ENEA NFV Core Pre-Release fully covers the
required Doctor functionality only for the Vitrage and Zabbix
components.Zabbix Configuration for Push NotificationsVitrage supports Zabbix datasource by regularly polling the Zabbix
agents, which need to be configured in advance.The Vitrage plugin developed internally by Enea can automatically
configure Zabbix so that everything works as expected. Polling however,
is not fast enough for a telco use-case, so it is necessary to configure
push notifications for Zabbix . This requires manual configuration on
one of the controller nodes, which will be used by the centralized
database making the configuration available on all the other
nodes.The Zabbix configuration dashboard is available at the same IP
address where OpenStack can be reached, e.g.
http://10.0.101.42/zabbix. Default username and
password are admin and zabbix
respectively.To forward Zabbix events to Vitrage, a new media type needs to be
created and associated with a Zabbix Admin user.Creating the Media Type which calls
zabbix_vitrage.pyCreate a new media type by selecting Media
Types under the Admininstration tab and
choosing Create Media Type.Now fill each parameter field with the values given below,
before clicking add:Name: Vitrage NotificationsType: ScriptScript name: zabbix_vitrage.pyUnder Administration, in the
Users tab (next to Media
Types), click on the drop-down menu User
groups and select Users, to display the
list of existing users. Choose from this list, the
admin alias:Inside the User tab for the
admin alias, click Add next to
the Groups list:Once the User groups popup menu
appears:Check the Zabbix Administrators checkboxClick SelectClick UpdateReselect the admin alias to enter the
Configuration of Users screen once again.Select the Media tab and click
Add to create and modify a new media type. A
popup menu will appear for New Media. Fill each
field with the values below:Type: Vitrage NotificationsSend to: e.g.
rabbit://nova:Q1Vt9iocA8OpXKOPoUYOLKie@192.168.0.4:5673/This url can be found in either
/etc/vitrage/vitrage.conf or in
/etc/nova/nova.conf. Only one url is needed and it is
vital that the url contains the ending " / ".When active: 1-7, 00:00-24:00Use if severity: (all)Status: EnabledClick Add and then
Update to save your changes.To configure an action, under the
Configuration tab select
Actions. Then, under the
Action tab, fill in the parameter fields with the
values below:Name: Forward to VitrageDefault Subject: {TRIGGER.STATUS}Default Message:host={HOST.NAME1}
hostid={HOST.ID1}
hostip={HOST.IP1}
triggerid={TRIGGER.ID}
description={TRIGGER.NAME}
rawtext={TRIGGER.NAME.ORIG}
expression={TRIGGER.EXPRESSION}
value={TRIGGER.VALUE}
priority={TRIGGER.NSEVERITY}
lastchange={EVENT.DATE} {EVENT.TIME}To send events, in the Conditions tab shown
below, make sure the label Maintenance status not in
"maintenance" is added and remove the label
Trigger value = PROBLEM, before saving your
changes.In the Operations tab still under
Configuration and Actions,
configure the users you wish to send to.In Operations Details, set:Operation Type: Send messageSend to Users: add adminSend only to: Vitrage NotificationsIn the Configuration tab, under
Templates, look for and select the template named
Template App Zabbix Agent and click on
Items.Select the item Agent ping to open and edit
it:Set the value in the field Update Interval (is
sec) to 10 seconds, and click
Update:After returning to the Configuration of
templates screen, select Triggers to
open and edit existing triggers.Select Zabbix agent on {HOST.NAME} is unreachable for
5 minutes to open and edit the trigger:Edit the Name and
Expression fields for this trigger to check the
condition every 20 seconds. Set the severity to
HIGH and click Update:Using these instructions, Zabbix will call the
zabbix_vitrage.py script, made readily available by
the Fuel Vitrage Plugin, to pass the arguments described above. This
script will then interpret the parameters and format an AMQP message to
be sent to the vitrage.notifications queue, managed
by the vitrage-graph service.Vitrage ConfigurationThe Vitrage team has been collaborating with the OPNFV Doctor
project in order to support Vitrage as an Inspector Component. The
Doctor use-case for Vitrage is described in an OpenStack blueprint. Enea
NFV Core has complemented Vitrage with the ability to set the states of
failed instances by implementing an action type in Vitrage. This action
calls Nova APIs to set instances in error state. An action type which
allows fencing failed hosts also exists.In order to make use of these features, Vitrage supports
additional configurations via yaml templates that
must be placed in /etc/vitrage/templates on the node
that have the Vitrage role.The example below shows how to program Vitrage to mark failed
compute hosts as down and then to change the state of the instances to
ERROR, by creating Vitrage deduced alarms.metadata:
name: test_mark_inst_err
description: test description
definitions:
entities:
- entity:
category: ALARM
type: zabbix
severity: HIGH
template_id: zabbix_alarm
- entity:
category: RESOURCE
type: nova.host
template_id: host
- entity:
category: RESOURCE
type: nova.instance
template_id: instance
relationships:
- relationship:
source: zabbix_alarm
relationship_type: on
target: host
template_id : critical_problem_on_host
- relationship:
source: host
target: instance
relationship_type: contains
template_id : host_contains_instance
scenarios:
- scenario:
condition: critical_problem_on_host and host_contains_instance
actions:
- action:
action_type: mark_down
action_target:
target: host
- action:
action_type: set_state
action_target:
target: instance
properties:
state: ERROR
- action:
action_type: set_instance_state
action_target:
target: instanceFor the action type of fencing, a similar action item must be
added:- scenario:
condition: critical_problem_on_host
actions:
- action:
action_type: fence
action_target:
target: hostAfter a template is configured, a restart of the
vitrage-api and vitrage-graph
services is needed:root@node-6:~# systemctl restart vitrage-api
root@node-6:~# systemctl restart vitrage-graphVitrage CustomizationsEnea NFV Core has added custom features for
Vitrage which allow two kinds of actions:Perform actions Northbound of the VIM:Nova force host down on computeSetting instance state to ERROR in nova. This is used in
conjunction with an alarm created by Tacker, as described
before, and should allow Tacker to detect when an instance is
affected and take proper actions.Perform actions Southbound of the VIM:Vitrage templates allow us to program fencing actions for
hosts with failed services. In the event that
systemd is unable to recover from a critical
process or a type of sofware error ocurs on the Hardware supporting
them, the fencing of Node can be programmed, and it in turn will
perform a reboot, attempting to recover the failed node.Pacemaker High AvailabilityMany of the OpenStack solutions which offer High Availability
characteristics employ Pacemaker for achieving highly available OpenStack
services. Traditionally Pacemaker has been used for managing only the
control plane services, so it can effectively provide redundancy and
recovery for the Controller nodes only. A reason for this is that
Controller nodes and Compute nodes essentially have very different High
Availability requirements that need to be considered.Typically, for Controller nodes, the services that run on them are
stateless, with few exceptions, where only one instance of a given service
is allowed, but for which redundancy is still desired. A good example
would be an AMQP service (e.g. RabbitMQ). Compute nodes H.A. requirements
depend on the type of services that run on them, but typically it is
desired that failures on these nodes be detected as soon as possible so
that the instances that run on them can be either migrated, resurrected or
restarted. Sometimes failures on the physical hosts do not necessarily
cause a failure on the services (VNFs), but having these services
incapacitated can prevent access to and controlling the services.Controller High Availability is thus a subject generally well
understood and experimented with, and the basis for this is Pacemaker
using Corosync underneath.Extending the use of Pacemaker to Compute nodes was thought as a
possible solution for providing VNF high availability, but the problem
turned out to be more complicated. On one hand, Pacemaker as a clustering
tool, can only scale properly up to a limited number of nodes, usually
less than 128. This poses a problem for large scale deployments where
hundreds of compute nodes are required. On the other hand, Compute node
H.A. requires other considerations and calls for specially designed
solutions.Pacemaker RemoteAs mentioned earlier, Pacemaker and corosync do not scale well
over a large cluster, since each node has to talk to every other,
essentially creating a mesh configuration. A solution to this problem
could be partitioning the cluster into smaller groups, but this has its
limitations and it is generally difficult to manage.A better solution is using pacemaker-remote, a
feature of Pacemaker, which allows for extending the cluster beyond the
usual limits by using the Pacemaker monitoring capabilities. It
essentially creates a new type of resource which enables adding light
weight nodes to the cluster. More information about pacemaker-remote can
be found on the official clusterlabs website.Please note that at this moment Pacemaker remote must be
configured manually after deployment. Here are the manual steps for
doing so:Log onto the Fuel Master using the default credentials, if
they have not been changed (root/r00tme).Type fuel node to obtain the list of nodes, their roles and
the IP addresses.[root@fuel ~]# fuel node
id | status | name | cluster | ip | mac | roles /
| pending_roles | online | group_id
---+--------+------------------+---------+-----------+-------------------+----------/
-----------------+---------------+--------+---------
1 | ready | Untitled (8c:d4) | 1 | 10.20.0.4 | 68:05:ca:46:8c:d4 | ceph-osd,/
controller | | 1 | 1
4 | ready | Untitled (8c:c2) | 1 | 10.20.0.6 | 68:05:ca:46:8c:c2 | ceph-osd,/
compute | | 1 | 1
5 | ready | Untitled (8c:c9) | 1 | 10.20.0.7 | 68:05:ca:46:8c:c9 | ceph-osd,/
compute | | 1 | 1
2 | ready | Untitled (8b:64) | 1 | 10.20.0.3 | 68:05:ca:46:8b:64 | /
controller, mongo, tacker | | 1 | 1
3 | ready | Untitled (8c:45) | 1 | 10.20.0.5 | 68:05:ca:46:8c:45 | /
controller, vitrage | | 1 | 1Each controller has a unique Pacemaker authkey. One needs to
be kept and propagated to the other servers. Assuming node-1, node-2
and node-3 are the controllers, execute the following from the Fuel
console:[root@fuel ~]# scp node-1:/etc/pacemaker/authkey .
[root@fuel ~]# scp authkey node-2:/etc/pacemaker/
[root@fuel ~]# scp authkey node-3:/etc/pacemaker/
[root@fuel ~]# scp authkey node-3:/etc/pacemaker/
[root@fuel ~]# scp authkey node-4:~
[root@fuel ~]# scp authkey node-5:~For each compute node, log on to it using the corresponding
IPInstall the required packages:root@node-4:~# apt-get install pacemaker-remote resource-agents crmshCopy the authkey from the Fuel Master and make sure the right
permissions are set:[root@node-4:~]# cp authkey /etc/pacemaker
[root@node-4:~]# chown root:haclient /etc/pacemaker/authkeyAdd an iptables rule for the default port (3121). Save it also
to /etc/iptables/rules.v4 to make it
persistent:root@node-4:~# iptables -A INPUT -s 192.168.0.0/24 -p tcp -m multiport /
--dports 3121 -m comment --comment "pacemaker_remoted from 192.168.0.0/24" -j ACCEPTStart the pacemaker-remote service:[root@node-4:~]# systemctl start pacemaker-remote.serviceLog onto one of the controller nodes and configure the
pacemaker-remote resources:[root@node-1:~]# pcs resource create node-4.domain.tld remote
[root@node-1:~]# pcs constraint location node-4.domain.tld prefers /
node-1.domain.tld=100 node-2.domain.tld=100 node-3.domain.tld=100
[root@node-1:~]# pcs constraint location node-4.domain.tld avoids node-5.domain.tld
[root@node-1:~]# pcs resource create node-5.domain.tld remote
[root@node-1:~]# pcs constraint location node-5.domain.tld prefers /
node-1.domain.tld=100 node-2.domain.tld=100 node-3.domain.tld=100
[root@node-1:~]# pcs constraint location node-5.domain.tld avoids node-4.domain.tldRemote nodes should now appear online:[root@node-1:~]# pcs status
Cluster name: OpenStack
Last updated: Thu Aug 24 12:00:21 2017 Last change: Thu Aug 24 11:57:32 2017 /
by root via cibadmin on node-1.domain.tld
Stack: corosync
Current DC: node-1.domain.tld (version 1.1.14-70404b0) - partition with quorum
5 nodes and 78 resources configured
Online: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld ]
RemoteOnline: [ node-4.domain.tld node-5.domain.tld ]Pacemaker FencingENEA NFV Core makes use of the fencing
capabilities of Pacemaker to isolate faulty nodes and trigger recovery
actions by means of power cycling the failed nodes. Fencing is
configured by creating STONITH type resources for
each of the servers in the cluster, both Controller nodes and Compute
nodes. The STONITH adapter for fencing the nodes is
fence_ipmilan, which makes use of the IPMI
capabilities of the ThunderX servers.Here are the steps for enabling fencing capabilities on a
cluster:Log onto the Fuel Master using the default credentials, if not
they have not been changed (root/r00tme).Type fuel node to obtain the list of nodes, their roles and
the IP addresses:[root@fuel ~]# fuel node
id | status | name | cluster | ip | mac | roles /
| pending_roles | online | group_id
---+--------+------------------+---------+-----------+-------------------+----------/
-----------------+---------------+--------+---------
1 | ready | Untitled (8c:d4) | 1 | 10.20.0.4 | 68:05:ca:46:8c:d4 | ceph-osd,/
controller | | 1 | 1
4 | ready | Untitled (8c:c2) | 1 | 10.20.0.6 | 68:05:ca:46:8c:c2 | ceph-osd,/
compute | | 1 | 1
5 | ready | Untitled (8c:c9) | 1 | 10.20.0.7 | 68:05:ca:46:8c:c9 | ceph-osd,/
compute | | 1 | 1
2 | ready | Untitled (8b:64) | 1 | 10.20.0.3 | 68:05:ca:46:8b:64 | /
controller, mongo, tacker | | 1 | 1
3 | ready | Untitled (8c:45) | 1 | 10.20.0.5 | 68:05:ca:46:8c:45 | /
controller, vitrage | | 1 | 1Log onto each server to install additional packages:[root@node-1:~]# apt-get install fence-agents ipmitoolConfigure Pacemaker fencing resources. This needs to be done
once on one of the controllers. The parameters will vary, depending
on the BMC addresses of each node and credentials.[root@node-1:~]# crm configure primitive ipmi-fencing-node-1 /
stonith::fence_ipmilan params pcmk_host_list="node-1.domain.tld" /
ipaddr=10.0.100.151 login=ADMIN passwd=ADMIN op monitor interval="60s"
[root@node-1:~]# crm configure primitive ipmi-fencing-node-2 /
stonith::fence_ipmilan params pcmk_host_list="node-2.domain.tld" /
ipaddr=10.0.100.152 login=ADMIN passwd=ADMIN op monitor interval="60s"
[root@node-1:~]# crm configure primitive ipmi-fencing-node-3 /
stonith::fence_ipmilan params pcmk_host_list="node-3.domain.tld" /
ipaddr=10.0.100.153 login=ADMIN passwd=ADMIN op monitor interval="60s"
[root@node-1:~]# crm configure primitive ipmi-fencing-node-4 /
stonith::fence_ipmilan params pcmk_host_list="node-4.domain.tld" /
ipaddr=10.0.100.154 login=ADMIN passwd=ADMIN op monitor interval="60s"
[root@node-1:~]# crm configure primitive ipmi-fencing-node-5 /
stonith::fence_ipmilan params pcmk_host_list="node-5.domain.tld" /
ipaddr=10.0.100.155 login=ADMIN passwd=ADMIN op monitor interval="60s"Activate fencing by enabling the stonith
property in Pacemaker (disabled by default). This also needs to be
done only once, on one of the controllers.[root@node-1:~]# pcs property set stonith-enabled=trueOpenStack Resource AgentsThe OpenStack community has been working for some time on
identifying possible solutions for enabling High Availability for Compute
nodes, after a period of belief that this subject was not something that
should concern the cloud platform. Over time it became obvious that even
on a true cloud platform, where services are designed to run without being
affected by the availability of the cloud platform, fault management and
recovery are still very important and desirable. This is also the case for
NFV applications, where in the good tradition of telecom applications, the
operators must have complete engineering control over the resources they
own and manage.The work for Compute node High Availability is captured in an
OpenStack user story and documented upstream, showing proposed solutions,
summit talks and presentations. A number of these solutions make use of
OpenStack Resource Agents, which are a set of specialized Pacemaker
resources capable of identifying failures in compute nodes and can perform
automatic evacuation of the instances affected by these failures.ENEA NFV Core aims to validate and integrate
this work and to make this feature available in the platform aimed as an
alternative to the Doctor framework, where simple, autonomous recovery of
running instances is desired.