1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
|
<?xml version="1.0" encoding="ISO-8859-1"?>
<chapter id="high_availability">
<title>High Availability Guide</title>
<para>Enea NFV Core 1.0 has been designed to provide high availability
characteristics that are needed for developing and deploying telco-grade NFV
solutions on top of our OPNFV based platform. The High Availability subject
in general is very wide and still an important focus in both opensource
communities and the independent/proprietary solutions market.</para>
<para>Enea NFV Core 1.0 aims to initially leverage the efforts in the
upstream OPNFV and OpenStack opensource projects, combining solutions from
both worlds in an effort to provide flexibility and use-case coverage. Enea
has long term expertise and proprietary solutions addressing High
Availability for telco applications, which are subject to integration with
the NFV based solutions.</para>
<section id="levels">
<title>High Availability Levels</title>
<para>The foundation for the feature set available in Enea NFV Core is
divided into three levels:</para>
<itemizedlist>
<listitem>
<para>Hardware Fault</para>
</listitem>
<listitem>
<para>NFV Platform H.A.</para>
</listitem>
<listitem>
<para>VNF High Availability</para>
</listitem>
</itemizedlist>
<para>The same division of levels for fault management can be seen in the
scope of the High Availability for OPNFV ("Availability") project. OPNFV
also hosts Doctor, a fault management and maintenance project designed to
develop and perform the consequent implementation of the OPNFV reference
platform. These two projects complement each other.</para>
<para>The Availability project addresses H.A. requirements and solutions
from the perspective of the three levels mentioned above. It produces high
level requirements and API definitions for High Availability for OPNFV, a
H.A. Gap Analysis Report for OpenStack and more recently, works on
optimizing existing OPNFV test frameworks, such as Yardstick, developing
test cases which realize H.A.-specific use-cases and scenarios derived
from the H.A. requirements.</para>
<para>The Doctor project aims to build fault management and maintenance
framework for the high availability of Network Services, on top of a
virtualized infrastructure. The key feature is immediate notification of
unavailability of virtualized resources from VIM, to process recovery of
VNFs on them.</para>
<para>The Doctor project has also collaborated with the Availability
project on identifying gaps in upstream projects, such as but not
exclusively OpenStack. It has also worked towards implementing missing
features and improving functionality, with a good example being the Aodh
event based alarms, which allow for fast notifications when certain
predefined events occur.</para>
<para>The Doctor project also produced an architectural design and a
reference implementation based on opensource components, which will be
presented later on in this document.</para>
</section>
<section id="doctor_arch">
<title>Doctor Architecture</title>
<para>The Doctor project documentation shows the detailed architecture for
Fault Management and NFVI Maintenance. Being quite similar with each
other, the focus in the following sections shall remain on Fault
Management.</para>
<para>The architecture specifies a set of functional blocks:</para>
<itemizedlist>
<listitem>
<para><emphasis role="bold">Monitor</emphasis> - monitors the
virtualized infrastructure, capturing fault events in software and
hardware. For this component we choose <emphasis
role="bold">Zabbix</emphasis> which is integrated into the platform
through the Fuel Zabbix Plugin, available upstream.</para>
</listitem>
<listitem>
<para><emphasis role="bold">Inspector</emphasis> - this component
receives notifications from Monitor components and OpenStack core
components, allowing it to create logical relationships between
entities, identify affected resources when faults occur, and to
communicate with Controllers in order to update the states of the
virtual and physical resources.</para>
<para>For this component Enea NFV Core 1.0 makes use of Vitrage, an
OpenStack related project used for Root Cause Analysis. The
integration into the platform is done with the help of a Fuel Plugin
which has been developed internally by Enea.</para>
</listitem>
<listitem>
<para><emphasis role="bold">Controller - </emphasis>OpenStack core
components act as Controllers responsible for maintaining the resource
map between physical and virtual resources. They accept update
requests from the Inspector and are responsible for sending failure
event notifications to the Notifier. Components such as Nova, Neutron,
Glance, and Heat, act as Controllers in the Doctor
Architecture.</para>
</listitem>
<listitem>
<para><emphasis role="bold">Notifier</emphasis> - the focus of this
component is on selecting and aggregating failure events received from
the controller, based on policies mandated by the Consumer. The role
of the Notifier is filled by the Aodh component in OpenStack.</para>
</listitem>
</itemizedlist>
<para>Alongside the Doctor components, there are a few other blocks
mentioned:</para>
<itemizedlist>
<listitem>
<para><emphasis role="bold">Administrator</emphasis> - this represents
the human role of administrating the platform by means of dedicated
interfaces. These can be visual dashboards like OpenStack Horizon or
Fuel Dashboard, or via CLI tools like the OpenStack unified CLI, that
can be accessed from one of the servers that act as OpenStack
Controller nodes.</para>
<para>In Enea NFV Core 1.0 the Administrator can also access the
Zabbix dashboard to perform supplementary configurations. The same
applies for the Vitrage tool, which comes with its own Horizon
dashboard, enabling the user to visually inspect the faults reported
by the monitoring tools through visual representations of the virtual
and physical resources, the relationships between them and the fault
correlation.</para>
<para>For Vitrage, users will usually want to configure additional
use-cases and describe relationships between components via template
files written in <literal>yaml</literal> format.</para>
</listitem>
<listitem>
<para><emphasis role="bold">Consumer</emphasis> - this block is
vaguely described in the Doctor Architecture and is out of its current
scope. Doctor only deals with fault detection and management, but
since the actual VNFs are managed, according to the ETSI architecture,
by a different entity, Doctor does not deal with recovery actions of
the VNFs. The role of the Consumer thus falls to that of a VNF Manager
and Orchestrator.</para>
<para>Enea NFV Core 1.0 provides VNF management capabilities using
Tacker, which is an OpenStack project that implements a generic VNF
Manager and Orchestrator, according to the ETSI MANO Architectural
Framework.</para>
</listitem>
</itemizedlist>
<para>The functional blocks overview in the picture below has been
complemented to show the components used for realizing the Doctor
Architecture:</para>
<mediaobject>
<imageobject role="fo">
<imagedata contentwidth="600" fileref="images/functional_blocks.svg"
format="SVG" />
</imageobject>
</mediaobject>
<section id="dr_fault_mg">
<title>Doctor Fault Management</title>
<para>The architecture described in the Doctor project has been
demonstrated in various PoCs and demos, but always using sample
components for either the consumer or the monitor. Enea has worked with
upstream projects Doctor and Vitrage, to realize the goals of the Doctor
project by using real components as described above.</para>
<para>The two pictures below show a typical fault management
scenario:</para>
<mediaobject>
<imageobject>
<imagedata contentwidth="600" fileref="images/dr_fault_mg.svg" />
</imageobject>
</mediaobject>
<mediaobject>
<imageobject>
<imagedata contentwidth="600" fileref="images/dr_fault_mg_2.svg" />
</imageobject>
</mediaobject>
<para>Enea NFV Core 1.0 uses the same approach described above:</para>
<orderedlist>
<listitem>
<para>When creating a VNF, the user will have to enable the
monitoring capabilities of Tacker by passing a template, which
specifies that an alarm will be created when the VM represented by
this VNF changes state. The support for alarm monitoring in Tacker
is detailed in the Alarm Monitoring Framework spec in the OpenStack
documentation.</para>
<para>Tacker should be able to create a VNF and then an Aodh alarm
of type event, triggerable when the instance is in a state of ERROR.
When this event is triggered perform an HTTP call to a URL managed
by Tacker. As a result of this action, Tacker can detect when an
instance has failed (for whatever reason) and will respawn it
somewhere else.</para>
</listitem>
<listitem>
<para>The subscribed response in this case is an empty operation,
the Notifier (Aodh) only has to confirm that the alarm has been
created.</para>
</listitem>
<listitem>
<para>The NFVI sends monitoring events for the resources the VIM has
been subscribed to.</para>
<note>
<para>This subscription message exchange between the VIM and NFVI
is not shown in this message flow. This step is related to
Vitrage's capability of receiving notifications from OpenStack
services. At this moment Vitrage supports notifications from
<literal>nova.host</literal>, <literal>nova.instances</literal>,
<literal>nova.zone</literal>, <literal>cinder.volume</literal>,
<literal>neutron.network</literal>,
<literal>neutron.port</literal> and <literal>heat.stack</literal>
OpenStack datasources.</para>
</note>
</listitem>
<listitem>
<para>This step describes faults detected by Zabbix which are sent
to the Inspector (Vitrage) as soon as detected. This is done using a
push approach by means of sending an AMQP message to a dedicated
message queue managed by Vitrage. For example, if
<literal>nova-compute</literal> fails on one of the compute nodes,
Zabbix will format a message specifying all the needed details
required for processing the fault: a timestamp, what host failed,
what event occurred etc.</para>
</listitem>
<listitem>
<para>This step shows database lookup geared to find the virtual
resources affected by the detected fault. Vitrage will perform
various calculations to detect what virtual resources are affected
by the raw failure presented by Zabbix.</para>
<para>Vitrage can be configured via templates to correlate instances
with the physical hosts they are running on, so that if a compute
node fails, then instances running on that host will be affected. A
typical use-case is to mark the compute node down
(<literal>mark_host_down</literal>) and update the states of all
instances running on them. This is done by issuing Nova API calls
for each of these instances.</para>
<para>Step 5c. shows the Controller (Nova in this case) acting upon
the state change of the instance and issuing an event alarm to
Aodh.</para>
</listitem>
<listitem>
<para>The Notifier will acknowledge the alarm event request from
Nova and will trigger the alarm(s) created by Tacker in step 1.
Since Tacker has configured the alarm to send an HTTP request, Aodh
will perform that HTTP call at the URL managed by Tacker.</para>
</listitem>
<listitem>
<para>The Consumer (Tacker) will react to the HTTP call and perform
the action configured by the user (e.g. respawn the VNF).</para>
</listitem>
<listitem>
<para>The action is sent to the Controller (Nova) so that the VNF is
recreated.</para>
</listitem>
</orderedlist>
<note condition="hidden">
<para>The ENEA NFV Core 1.0 Pre-Release fully covers the required
Doctor functionality only for the Vitrage and Zabbix
components.</para>
</note>
</section>
<section id="zabbix">
<title>Zabbix Configuration for Push Notifications</title>
<para>Vitrage supports Zabbix datasource by means of regularly polling
the Zabbix agents, which need to be configured in advance. The Vitrage
plugin developed internally by Enea can automatically configure Zabbix
so that everything works as expected. Polling however, is not fast
enough for a telco use-case, so it is necessary to configure push
notifications for Zabbix . This requires manual configuration on one of
the controller nodes, since Zabbix uses a centralized database which
makes the configuration available on all the other nodes.</para>
<para>The Zabbix configuration dashboard is available at the same IP
address where OpenStack can be reached, e.g.
<literal>http://10.0.6.42/zabbix</literal>.</para>
<para>To forward Zabbix events to Vitrage, a new media script needs to
be created and associated with a user. Follow the steps below as a
Zabbix Admin user:</para>
<orderedlist>
<listitem>
<para>Create a new media type [Admininstration Media Types >
Create Media Type]</para>
<itemizedlist>
<listitem>
<para>Name: Vitrage Notifications</para>
</listitem>
<listitem>
<para>Type: Script</para>
</listitem>
<listitem>
<para>Script name: <filename>zabbix_vitrage.py</filename></para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para>Modify the Media for the Admin user [Administration
Users]</para>
<itemizedlist>
<listitem>
<para>Type: Vitrage Notifications</para>
</listitem>
<listitem>
<para>Send to:
<literal>rabbit://rabbit_user:rabbit_pass@127.0.0.1:5672/
---</literal> Vitrage message bus url (you need to search for
this in <literal>/etc/vitrage/vitrage.conf or
/etc/nova/nova.conf transport_url</literal>)</para>
</listitem>
<listitem>
<para>When active: 1-7, 00:00-24:00</para>
</listitem>
<listitem>
<para>Use if severity: (all)</para>
</listitem>
<listitem>
<para>Status: Enabled</para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para>Configure Action [Configuration > Actions > Create
Action > Action]</para>
<itemizedlist>
<listitem>
<para>Name: Forward to Vitrage</para>
</listitem>
<listitem>
<para>Default Subject: {TRIGGER.STATUS}</para>
</listitem>
<listitem>
<para>Default Message:</para>
<programlisting>host={HOST.NAME1}
hostid={HOST.ID1}
hostip={HOST.IP1}
triggerid={TRIGGER.ID}
description={TRIGGER.NAME}
rawtext={TRIGGER.NAME.ORIG}
expression={TRIGGER.EXPRESSION}
value={TRIGGER.VALUE}
priority={TRIGGER.NSEVERITY}
lastchange={EVENT.DATE} {EVENT.TIME}</programlisting>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para>To send events add under the <literal>Conditions</literal>
tab: "Maintenance status not in "maintenance"".</para>
</listitem>
<listitem>
<para>Finally, add an operation:</para>
<itemizedlist>
<listitem>
<para>Send to Users: Admin</para>
</listitem>
<listitem>
<para>Send only to: Vitrage Notifications</para>
</listitem>
</itemizedlist>
</listitem>
</orderedlist>
<para>Using these instructions, Zabbix will call the
<literal>zabbix_vitrage.py</literal> script, made readily available by
the Fuel Vitrage Plugin, to pass the arguments described in step 3. The
<literal>zabbix_vitrage.py</literal> script will then interpret the
parameters and format an AMQP message to be sent to the
<literal>vitrage.notifications</literal> queue, managed by the
vitrage-graph service.</para>
</section>
<section id="vitrage_config">
<title>Vitrage Configuration</title>
<para>The Vitrage team has been collaborating with the OPNFV Doctor
project in order to support Vitrage as an Inspector Component. The
Doctor use-case for Vitrage is described in an OpenStack blueprint. Enea
NFV Core has complemented Vitrage with the ability to set the states of
failed instances by implementing an action type in Vitrage. This action
calls Nova APIs to set instances in error state. An action type which
allows fencing failed hosts also exists.</para>
<para>In order to make use of these features, Vitrage supports
additional configurations via <literal>yaml</literal> templates that
must be placed in <literal>/etc/vitrage/templates</literal> on the nodes
have the Vitrage role.</para>
<para>The example below shows how to program Vitrage to mark failed
compute hosts as down and then to change the state of the instances to
ERROR, by creating Vitrage deduced alarms.</para>
<programlisting>metadata:
name: test_nova_mark_instance_err
description: test description
definitions:
entities:
- entity:
category: ALARM
type: zabbix
rawtext: Nova Compute process is not running on {HOST.NAME}
template_id: zabbix_alarm
- entity:
category: RESOURCE
type: nova.host
template_id: host
- entity:
category: RESOURCE
type: nova.instance
template_id: instance
relationships:
- relationship:
source: zabbix_alarm
relationship_type: on
target: host
template_id: nova_process_not_running
- relationship:
source: host
target: instance
relationship_type: contains
template_id : host_contains_instance
scenarios:
- scenario:
condition: nova_process_not_running and host_contains_instance
actions:
- action:
action_type: mark_down
action_target:
target: host
- action:
action_type: set_instance_state
action_target:
target: instance
- action:
action_type: set_state
action_target:
target: instance
properties:
state: ERROR</programlisting>
<para>For the action type of fencing, a similar action item must be
added:</para>
<programlisting>- scenario:
condition: critical_problem_on_host
actions:
- action:
action_type: fence
action_target:
target: host</programlisting>
<para>After a template is configured, a restart of the
<literal>vitrage-api</literal> and <literal>vitrage-graph</literal>
services is needed:</para>
<programlisting>root@node-6:~# systemctl restart vitrage-api
root@node-6:~# systemctl restart vitrage-graph</programlisting>
</section>
<section id="vitrage_custom">
<title>Vitrage Customizations</title>
<para>Enea NFV Core 1.0 has added custom features for Vitrage which
allow two kinds of actions:</para>
<orderedlist>
<listitem>
<para>Perform actions Northbound of the VIM:</para>
<itemizedlist>
<listitem>
<para>Nova force host down on compute</para>
</listitem>
<listitem>
<para>Setting instance state to ERROR in nova. This is used in
conjunction with an alarm created by Tacker, as described
before, and should allow Tacker to detect when an instance is
affected and take proper actions.</para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para>Perform actions Southbound of the VIM:</para>
<para>Vitrage templates allow us to program fencing actions for
hosts with failed services. In the event that
<literal>systemd</literal> is unable to recover from a critical
process or a type of sofware error ocurs on the Hardware supporting
them, the fencing of Node can be programmed, and it in turn will
perform a reboot, attempting to recover the failed node.</para>
</listitem>
</orderedlist>
</section>
</section>
<section id="pm_high_avail">
<title>Pacemaker High Availability</title>
<para>Many of the OpenStack solutions which offer High Availability
characteristics employ Pacemaker for achieving highly available OpenStack
services. Traditionally Pacemaker has been used for managing only the
control plane services, so it can effectively provide redundancy and
recovery for the Controller nodes only. A reason for this is that
Controller nodes and Compute nodes essentially have very different High
Availability requirements that need to be considered.</para>
<para>Typically, for Controller nodes, the services that run on them are
stateless, with few exceptions, where only one instance of a given service
is allowed, but for which redundancy is still desired. A good example
would be an AMQP service (e.g. RabbitMQ). Compute nodes H.A. requirements
depend on the type of services that run on them, but typically it is
desired that failures on these nodes be detected as soon as possible so
that the instances that run on them can be either migrated, resurrected or
restarted. Sometimes failures on the physical hosts do not necessarily
cause a failure on the services (VNFs), but having these services
incapacitated can prevent access to and controlling the services.</para>
<para>Controller High Availability is thus a subject generally well
understood and experimented with, and the basis for this is Pacemaker
using Corosync underneath.</para>
<para>Extending the use of Pacemaker to Compute nodes was thought as a
possible solution for providing VNF high availability, but the problem
turned out to be more complicated. On one hand, Pacemaker as a clustering
tool, can only scale properly up to a limited number of nodes, usually
less than 128. This poses a problem for large scale deployments where
hundreds of compute nodes are required. On the other hand, Compute node
H.A. requires other considerations and calls for specially designed
solutions.</para>
<section id="pm_remote">
<title>Pacemaker Remote</title>
<para>As mentioned earlier, Pacemaker and corosync do not scale well
over a large cluster, since each node has to talk to every other,
essentially creating a mesh configuration. A solution to this problem
could be partitioning the cluster into smaller groups, but this has its
limitations and it is generally difficult to manage.</para>
<para>A better solution is using <literal>pacemaker-remote</literal>, a
feature of Pacemaker, which allows for extending the cluster beyond the
usual limits by using the Pacemaker monitoring capabilities. It
essentially creates a new type of resource which enables adding light
weight nodes to the cluster. More information about pacemaker-remote can
be found on the official clusterlabs website.</para>
<para>Please note that at this moment Pacemaker remote must be
configured manually after deployment. Here are the manual steps for
doing so:</para>
<orderedlist>
<listitem>
<para>Log onto the Fuel Master using the default credentials, if
they have not been changed (root/r00tme).</para>
</listitem>
<listitem>
<para>Type fuel node to obtain the list of nodes, their roles and
the IP addresses.</para>
<programlisting>[root@fuel ~]# fuel node
id | status | name | cluster | ip | mac | roles /
| pending_roles | online | group_id
---+--------+------------------+---------+-----------+-------------------+----------/
-----------------+---------------+--------+---------
1 | ready | Untitled (8c:d4) | 1 | 10.20.0.4 | 68:05:ca:46:8c:d4 | ceph-osd,/
controller | | 1 | 1
4 | ready | Untitled (8c:c2) | 1 | 10.20.0.6 | 68:05:ca:46:8c:c2 | ceph-osd,/
compute | | 1 | 1
5 | ready | Untitled (8c:c9) | 1 | 10.20.0.7 | 68:05:ca:46:8c:c9 | ceph-osd,/
compute | | 1 | 1
2 | ready | Untitled (8b:64) | 1 | 10.20.0.3 | 68:05:ca:46:8b:64 | /
controller, mongo, tacker | | 1 | 1
3 | ready | Untitled (8c:45) | 1 | 10.20.0.5 | 68:05:ca:46:8c:45 | /
controller, vitrage | | 1 | 1</programlisting>
</listitem>
<listitem>
<para>Each controller has a unique Pacemaker authkey. One needs to
be kept and propagated to the other servers. Assuming node-1, node-2
and node-3 are the controllers, execute the following from the Fuel
console:</para>
<programlisting>[root@fuel ~]# scp node-1:/etc/pacemaker/authkey .
[root@fuel ~]# scp authkey node-2:/etc/pacemaker/
[root@fuel ~]# scp authkey node-3:/etc/pacemaker/
[root@fuel ~]# scp authkey node-3:/etc/pacemaker/
[root@fuel ~]# scp authkey node-4:~
[root@fuel ~]# scp authkey node-5:~</programlisting>
</listitem>
<listitem>
<para>For each compute node, log on to it using the corresponding
IP</para>
</listitem>
<listitem>
<para>Install the required packages:</para>
<programlisting>root@node-4:~# apt-get install pacemaker-remote resource-agents crmsh</programlisting>
</listitem>
<listitem>
<para>Copy the authkey from the Fuel Master and make sure the right
permissions are set:</para>
<programlisting>[root@node-4:~]# cp authkey /etc/pacemaker
[root@node-4:~]# chown root:haclient /etc/pacemaker/authkey</programlisting>
</listitem>
<listitem>
<para>Add an iptables rule for the default port (3121). Save it also
to <literal>/etc/iptables/rules.v4</literal> to make it
persistent:</para>
<programlisting>root@node-4:~# iptables -A INPUT -s 192.168.0.0/24 -p tcp -m multiport /
--dports 3121 -m comment --comment "pacemaker_remoted from 192.168.0.0/24" -j ACCEPT</programlisting>
</listitem>
<listitem>
<para>Start the pacemaker-remote service:</para>
<programlisting>[root@node-4:~]# systemctl start pacemaker-remote.service</programlisting>
</listitem>
<listitem>
<para>Log onto one of the controller nodes and configure the
pacemaker-remote resources:</para>
<programlisting>[root@node-1:~]# pcs resource create node-4.domain.tld remote
[root@node-1:~]# pcs constraint location node-4.domain.tld prefers /
node-1.domain.tld=100 node-2.domain.tld=100 node-3.domain.tld=100
[root@node-1:~]# pcs constraint location node-4.domain.tld avoids node-5.domain.tld
[root@node-1:~]# pcs resource create node-5.domain.tld remote
[root@node-1:~]# pcs constraint location node-5.domain.tld prefers /
node-1.domain.tld=100 node-2.domain.tld=100 node-3.domain.tld=100
[root@node-1:~]# pcs constraint location node-5.domain.tld avoids node-4.domain.tld</programlisting>
</listitem>
<listitem>
<para>Remote nodes should now appear online:</para>
<programlisting>[root@node-1:~]# pcs status
Cluster name: OpenStack
Last updated: Thu Aug 24 12:00:21 2017 Last change: Thu Aug 24 11:57:32 2017 /
by root via cibadmin on node-1.domain.tld
Stack: corosync
Current DC: node-1.domain.tld (version 1.1.14-70404b0) - partition with quorum
5 nodes and 78 resources configured
Online: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld ]
RemoteOnline: [ node-4.domain.tld node-5.domain.tld ]</programlisting>
</listitem>
</orderedlist>
</section>
<section id="pm_fencing">
<title>Pacemaker Fencing</title>
<para>ENEA NFV Core 1.0 makes use of the fencing capabilities of
Pacemaker to isolate faulty nodes and trigger recovery actions by means
of power cycling the failed nodes. Fencing is configured by creating
<literal>STONITH</literal> type resources for each of the servers in the
cluster, both Controller nodes and Compute nodes. The
<literal>STONITH</literal> adapter for fencing the nodes is
<literal>fence_ipmilan</literal>, which makes use of the IPMI
capabilities of the ThunderX servers.</para>
<para>Here are the steps for enabling fencing capabilities on a
cluster:</para>
<orderedlist>
<listitem>
<para>Log onto the Fuel Master using the default credentials, if not
they have not been changed (root/r00tme).</para>
</listitem>
<listitem>
<para>Type fuel node to obtain the list of nodes, their roles and
the IP addresses:</para>
<programlisting>[root@fuel ~]# fuel node
id | status | name | cluster | ip | mac | roles /
| pending_roles | online | group_id
---+--------+------------------+---------+-----------+-------------------+----------/
-----------------+---------------+--------+---------
1 | ready | Untitled (8c:d4) | 1 | 10.20.0.4 | 68:05:ca:46:8c:d4 | ceph-osd,/
controller | | 1 | 1
4 | ready | Untitled (8c:c2) | 1 | 10.20.0.6 | 68:05:ca:46:8c:c2 | ceph-osd,/
compute | | 1 | 1
5 | ready | Untitled (8c:c9) | 1 | 10.20.0.7 | 68:05:ca:46:8c:c9 | ceph-osd,/
compute | | 1 | 1
2 | ready | Untitled (8b:64) | 1 | 10.20.0.3 | 68:05:ca:46:8b:64 | /
controller, mongo, tacker | | 1 | 1
3 | ready | Untitled (8c:45) | 1 | 10.20.0.5 | 68:05:ca:46:8c:45 | /
controller, vitrage | | 1 | 1</programlisting>
</listitem>
<listitem>
<para>Log onto each server to install additional packages:</para>
<programlisting>[root@node-1:~]# apt-get install fence-agents ipmitool</programlisting>
</listitem>
<listitem>
<para>Configure Pacemaker fencing resources. This needs to be done
once on one of the controllers. The parameters will vary, depending
on the BMC addresses of each node and credentials.</para>
<programlisting>[root@node-1:~]# crm configure primitive ipmi-fencing-node-1 /
stonith::fence_ipmilan params pcmk_host_list="node-1.domain.tld" /
ipaddr=10.0.100.151 login=ADMIN passwd=ADMIN op monitor interval="60s"
[root@node-1:~]# crm configure primitive ipmi-fencing-node-2 /
stonith::fence_ipmilan params pcmk_host_list="node-2.domain.tld" /
ipaddr=10.0.100.152 login=ADMIN passwd=ADMIN op monitor interval="60s"
[root@node-1:~]# crm configure primitive ipmi-fencing-node-3 /
stonith::fence_ipmilan params pcmk_host_list="node-3.domain.tld" /
ipaddr=10.0.100.153 login=ADMIN passwd=ADMIN op monitor interval="60s"
[root@node-1:~]# crm configure primitive ipmi-fencing-node-4 /
stonith::fence_ipmilan params pcmk_host_list="node-4.domain.tld" /
ipaddr=10.0.100.154 login=ADMIN passwd=ADMIN op monitor interval="60s"
[root@node-1:~]# crm configure primitive ipmi-fencing-node-5 /
stonith::fence_ipmilan params pcmk_host_list="node-5.domain.tld" /
ipaddr=10.0.100.155 login=ADMIN passwd=ADMIN op monitor interval="60s"</programlisting>
</listitem>
<listitem>
<para>Activate fencing by enabling the <literal>stonith</literal>
property in Pacemaker (disabled by default). This also needs to be
done only once, on one of the controllers.</para>
<programlisting>[root@node-1:~]# pcs property set stonith-enabled=true</programlisting>
</listitem>
</orderedlist>
</section>
</section>
<section condition="hidden" id="ops_resources_agents">
<title>OpenStack Resource Agents</title>
<para>The OpenStack community has been working for some time on
identifying possible solutions for enabling High Availability for Compute
nodes, after a period of belief that this subject was not something that
should concern the cloud platform. Over time it became obvious that even
on a true cloud platform, where services are designed to run without being
affected by the availability of the cloud platform, fault management and
recovery are still very important and desirable. This is also the case for
NFV applications, where in the good tradition of telecom applications, the
operators must have complete engineering control over the resources they
own and manage.</para>
<para>The work for Compute node High Availability is captured in an
OpenStack user story and documented upstream, showing proposed solutions,
summit talks and presentations. A number of these solutions make use of
OpenStack Resource Agents, which are a set of specialized Pacemaker
resources capable of identifying failures in compute nodes and can perform
automatic evacuation of the instances affected by these failures.</para>
<para>ENEA NFV Core 1.0 aims to validate and integrate this work and to
make this feature available in the platform aimed as an alternative to the
Doctor framework, where simple, autonomous recovery of running instances
is desired.</para>
</section>
</chapter>
|