2 files changed, 626 insertions, 0 deletions

diff --git a/doc/book-enea-nfv-access-getting-started/doc/advanced_configurations.xml b/doc/book-enea-nfv-access-getting-started/doc/advanced_configurations.xml
index 1b04c40..74ff1dd 100644
--- a/doc/book-enea-nfv-access-getting-started/doc/advanced_configurations.xml
+++ b/doc/book-enea-nfv-access-getting-started/doc/advanced_configurations.xml
@@ -420,4 +420,630 @@ node0.1048576kB = 3 </programlisting>
       </note>
     </section>
   </section>
+
+  <section id="high_availability_ig">
+    <title>Installing the Enea uCPE Manager in High Availability Mode</title>
+
+    <para>The following describes the setup needed for running the Enea uCPE
+    Manager in High Availabilty (HA) mode, with a MariaDB database cluster.
+    The desired setup is depicted in the following diagram:</para>
+
+    <figure>
+      <title>The High Availability setup</title>
+
+      <mediaobject>
+        <imageobject>
+          <imagedata align="center" contentwidth="600"
+                     fileref="images/high_av_setup.png" />
+        </imageobject>
+      </mediaobject>
+    </figure>
+
+    <section id="ha_reqs">
+      <title>Requirements for High Availability</title>
+      
+      <para>The following hardware is needed for deploying the base configuration:</para>
+
+      <itemizedlist>
+        <listitem>
+          <para>Machines running the Enea uCPE Manager and MariaDB:</para>
+
+          <itemizedlist spacing="compact">
+            <listitem>
+              <para>4 CPU cores</para>
+            </listitem>
+
+            <listitem>
+              <para>12 - 16 GB memory</para>
+            </listitem>
+
+            <listitem>
+              <para>256 - 512 GB hard disk</para>
+            </listitem>
+          </itemizedlist>
+        </listitem>
+
+        <listitem>
+          <para>Machines running only MariaDB:</para>
+
+          <itemizedlist spacing="compact">
+            <listitem>
+              <para>2 CPU cores</para>
+            </listitem>
+
+            <listitem>
+              <para>8 GB memory</para>
+            </listitem>
+
+            <listitem>
+              <para>256 - 512 GB hard disk</para>
+            </listitem>
+          </itemizedlist>
+        </listitem>
+
+        <listitem>
+          <para>The Enea uCPE Manager machines should run CentOS 7, this is
+          the only currently supported version.</para>
+        </listitem>
+
+        <listitem>
+          <para>All machines should be on the same subnet. For geographically
+          distributed servers, a VPN can be used.</para>
+        </listitem>
+
+        <listitem>
+          <para>All VCPE devices will typically connect to the external IP
+          (WAN) address (exported by the Big-IP firewall).</para>
+        </listitem>
+
+        <listitem>
+          <para>WAN traffic will be HTTPS, whereas internal communication will
+          be through HTTP.</para>
+        </listitem>
+
+        <listitem>
+          <para>External clients (browsers using the GUI as well as clients
+          using the REST API) will connect to the external (WAN)
+          address.</para>
+        </listitem>
+      </itemizedlist>
+    </section>
+
+    <section id="firewall_rules">
+      <title>Firewall Rules</title>
+
+      <para>The following firewall configuration is needed:</para>
+
+      <orderedlist>
+        <listitem>
+          <para>Disable <literal>SELINUX</literal> on all database servers by
+          editing <literal>/etc/sysconfig/selinux</literal> and changing the
+          following:</para>
+
+          <programlisting>SELINUX=disabled
+SELINUXTYPE=targeted</programlisting>
+        </listitem>
+
+        <listitem>
+          <para>Reboot the server:</para>
+
+          <programlisting>[root@localhost ~]# sudo shutdown -r now</programlisting>
+        </listitem>
+      </orderedlist>
+
+      <para>The following ports should be opened in the local firewall (not
+      Big-IP), for each Enea uCPE Manager machine:</para>
+
+      <table>
+        <title>Ports for Enea uCPE Manager Machines</title>
+
+        <tgroup cols="2">
+          <colspec align="left" />
+
+          <thead>
+            <row>
+              <entry align="center">Port (Protocol)</entry>
+
+              <entry align="center">Usage</entry>
+            </row>
+          </thead>
+
+          <tbody>
+            <row>
+              <entry>80 (TCP)</entry>
+
+              <entry>HTTP (used by Big-IP firewall)</entry>
+            </row>
+
+            <row>
+              <entry>443 (TCP)</entry>
+
+              <entry>HTTPS</entry>
+            </row>
+
+            <row>
+              <entry>54327 (UDP)</entry>
+
+              <entry>Cluster multicasting (Hazelcast)</entry>
+            </row>
+
+            <row>
+              <entry>5701 - 5708 (TCP)</entry>
+
+              <entry>Hazelcast communications</entry>
+            </row>
+
+            <row>
+              <entry>4334 (TCP)</entry>
+
+              <entry>NETCONF call-home</entry>
+            </row>
+
+            <row>
+              <entry>7000 - 7009 (TCP)</entry>
+
+              <entry>Reverse SSH connection pool</entry>
+            </row>
+          </tbody>
+        </tgroup>
+      </table>
+
+      <para>For each MariaDB machine, the following firewall configuration is needed:</para>
+
+      <table>
+        <title>Ports for MariaDB Machines</title>
+
+        <tgroup cols="2">
+          <colspec align="left" />
+
+          <thead>
+            <row>
+              <entry align="center">Port (Protocol)</entry>
+
+              <entry align="center">Usage</entry>
+            </row>
+          </thead>
+
+          <tbody>
+            <row>
+              <entry>3306 (TCP)</entry>
+
+              <entry>Client connections</entry>
+            </row>
+
+            <row>
+              <entry>4567 (UDP/TCP)</entry>
+
+              <entry>Galera cluster replication with multicasting</entry>
+            </row>
+
+            <row>
+              <entry>4568 (TCP)</entry>
+
+              <entry>Incremental state transfer</entry>
+            </row>
+
+            <row>
+              <entry>4444 (TCP)</entry>
+
+              <entry>State snapshot transfer</entry>
+            </row>
+          </tbody>
+        </tgroup>
+      </table>
+
+      <para>The following ports should be accessible externally and translated
+      to the Virtual IP side as shown below (by the Big-IP firewall):</para>
+
+      <table>
+        <title>Ports for Virtual IP</title>
+
+        <tgroup cols="3">
+          <colspec align="left" />
+
+          <thead>
+            <row>
+              <entry align="center">External Port (Protocol)</entry>
+
+              <entry align="center">Usage</entry>
+
+              <entry align="center">Local Port (Protocol)</entry>
+            </row>
+          </thead>
+
+          <tbody>
+            <row>
+              <entry>443 (TCP)</entry>
+
+              <entry>HTTPS to/back HTTP</entry>
+
+              <entry>80 (TCP)</entry>
+            </row>
+
+            <row>
+              <entry>4334 (TCP)</entry>
+
+              <entry>NETCONF call-home</entry>
+
+              <entry>4334 (TCP)</entry>
+            </row>
+
+            <row>
+              <entry>7000 - 7009 (TCP)</entry>
+
+              <entry>Reverse SSH connection pool</entry>
+
+              <entry>7000 - 7009 (TCP)</entry>
+            </row>
+          </tbody>
+        </tgroup>
+      </table>
+    </section>
+
+    <section id="ha_installation">
+      <title>Installing High Availability</title>
+
+      <para>The Enea uCPE Manager can be installed in High Availability mode with 
+      a MariaDB database cluster by performing the following steps. The mandatory 
+      Java configuration is also detailed.</para>
+
+      <section id="ha_mariadb">
+        <title>Installing and configuring the MariaDB cluster</title>
+
+        <para>Install the latest MariaDB packages on all servers.</para>
+
+        <note>
+          <para>The setup was tested using MariaDB 10.5.8, built for CentOS
+          7.</para>
+        </note>
+
+        <para><emphasis role="bold">How to install MariaDB</emphasis></para>
+
+        <orderedlist>
+          <listitem>
+            <para>Make sure the following packages are installed:</para>
+
+            <programlisting>MariaDB-compat-10.5.8-1.el7.centos.x86_64
+MariaDB-common-10.5.8-1.el7.centos.x86_64
+MariaDB-server-10.5.8-1.el7.centos.x86_64
+MariaDB-client-10.5.8-1.el7.centos.x86_64
+galera-4-26.4.6-1.el7.centos.x86_64</programlisting>
+
+            <para>These provide the MariaDB server, client and the Galera
+            <literal>wsrep</literal> provider library.</para>
+          </listitem>
+
+          <listitem>
+            <para>Copy the <literal>wsrep</literal> template:</para>
+
+            <programlisting>[root@localhost ~]# cp /usr/share/mysql/wsrep.cnf /etc/my.cnf.d</programlisting>
+          </listitem>
+
+          <listitem>
+            <para>Change the following configuration:</para>
+
+            <programlisting># Full path to wsrep provider library or 'none'
+wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so
+
+# Provider specific configuration options
+#wsrep_provider_options=
+
+# Logical cluster name. Should be the same for all nodes.
+wsrep_cluster_name="ucpemanager"
+
+# Group communication system handle
+wsrep_cluster_address="gcomm://192.168.10.11,192.168.10.12,..,192.168.10.16"
+
+# Human-readable node name (non-unique). Hostname by default.
+wsrep_node_name=Node1  
+# current node's name. set node name for each server in the cluster
+
+# Base replication &lt;address|hostname&gt;[:port] of the node.
+# The values supplied will be used as defaults for state transfer receiving,
+# listening ports and so on. Default: address of the first network interface.
+wsrep_node_address=192.168.10.11 
+#current node's interface IP . must be set for each node in the cluster</programlisting>
+
+            <note>
+              <para>Steps 2 and 3 must be performed for each MariaDB node in
+              the cluster.</para>
+            </note>
+          </listitem>
+
+          <listitem>
+            <para>Bootstrap the first node in the cluster (identified by
+            <literal>Node1</literal> for example), by running:</para>
+
+            <programlisting>[root@localhost ~]# galera_new_cluster</programlisting>
+
+            <para>This script passes the
+            <literal>--wsrep-new-cluster</literal> to
+            <literal>mysqld</literal> which tells the node that there is no
+            pre-existing cluster to connect to. The node will create a new
+            UUID to identify the new cluster.</para>
+
+            <note>
+              <para>Do not execute this script when connecting to an existing
+              cluster. It will create a new UUID to identify the cluster
+              again, and the node won't reconnect to the old cluster.</para>
+            </note>
+          </listitem>
+
+          <listitem>
+            <para>Go to <literal>Node1</literal> and start the service:</para>
+
+            <programlisting>[root@localhost ~]# systemctl start mariadb</programlisting>
+
+            <para>Subsequently, start the service on the other servers.</para>
+          </listitem>
+
+          <listitem>
+            <para>Verify that the nodes have entered the cluster:</para>
+
+            <programlisting>[root@localhost ~]# mysql --host=localhost --user=root -p
+MariaDB [(none)]&gt; show status like 'wsrep_cluster_conf_%';
++-----------------------+-------+
+| Variable_name         | Value |
++-----------------------+-------+
+| wsrep_cluster_conf_id | 3     |
++-----------------------+-------+
+1 row in set (0.001 sec)</programlisting>
+          </listitem>
+
+          <listitem>
+            <para>Run the initial configuration script (only once, on one of
+            the machines in the cluster):</para>
+
+            <programlisting>[root@localhost ~]# mysql_secure_installation
+
+Switch to unix_socket authentication [Y/n] Y
+Enabled successfully!
+Reloading privilege tables..
+ ... Success!
+…
+Change the root password? [Y/n] Y
+New password: 
+Re-enter new password: 
+Password updated successfully!
+Reloading privilege tables..
+ ... Success!
+…
+Remove anonymous users? [Y/n] Y
+ ... Success!
+…
+Disallow root login remotely? [Y/n] Y
+ ... Success!
+…
+Remove test database and access to it? [Y/n] Y (optional)
+ - Dropping test database...
+ ... Success!
+ - Removing privileges on test database...
+ ... Success!
+Reload privilege tables now? [Y/n] Y
+ ... Success!
+
+Cleaning up...
+
+All done!  If you've completed all of the above steps, your MariaDB
+installation should now be secure.
+
+Thanks for using MariaDB!</programlisting>
+          </listitem>
+
+          <listitem>
+            <para>Create the initial database and grant access to it:</para>
+
+            <programlisting>[root@localhost application]# mysql --host=localhost --user=root -p
+MariaDB [(none)]&gt; CREATE DATABASE ucpemanager CHARACTER SET='utf8' \
+COLLATE='utf8_bin';
+Query OK, 1 row affected (0.004 sec)
+
+MariaDB [(none)]&gt; GRANT ALL PRIVILEGES ON ucpemanager.* \
+TO 'enea'@'%' IDENTIFIED BY 'somepassword' WITH GRANT OPTION;</programlisting>
+          </listitem>
+        </orderedlist>
+      </section>
+
+      <section id="ha_java_sdk_install">
+        <title>Installing the Java SDK</title>
+
+        <para>The following steps describe the installation of Java 11 SDK on
+        the CentOS 7 machines that will run the Enea uCPE Manager
+        installation:</para>
+
+        <orderedlist>
+          <listitem>
+            <para>Install the following packages:</para>
+
+            <programlisting>java-11-openjdk-devel-11.0.10.0.9-0.el7_9.x86_64
+java-11-openjdk-11.0.10.0.9-0.el7_9.x86_64</programlisting>
+          </listitem>
+
+          <listitem>
+            <para>Check that java points to the current JRE:</para>
+
+            <programlisting>root@localhost ~]# java -version
+openjdk version "11.0.10" 2021-01-19 LTS
+OpenJDK Runtime Environment 18.9 (build 11.0.10+9-LTS)
+OpenJDK 64-Bit Server VM 18.9 (build 11.0.10+9-LTS, mixed mode, sharing)</programlisting>
+
+            <para>If it doesn't, then check the alternatives, and make sure
+            that java points to the JDK11 installation:</para>
+
+            <programlisting>[root@localhost ~]# alternatives --config java</programlisting>
+          </listitem>
+
+          <listitem>
+            <para>Set the <literal>JAVA_HOME</literal> environment variable
+            and update paths:</para>
+
+            <programlisting>export JAVA_HOME=$(dirname $(dirname $(readlink $(readlink $(which javac)))))
+export PATH=$PATH:$JAVA_HOME/bin
+export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar</programlisting>
+
+            <para>As an alternative, the variables can be written into the
+            <filename>.bashrc</filename> file, so that they load every time a
+            console is opened. To enable these settings for all users, add the
+            variables to <literal>/etc/environment</literal>.</para>
+          </listitem>
+
+          <listitem>
+            <para>The <literal>JAVA_HOME</literal> variable should point
+            to:</para>
+
+            <programlisting>[root@localhost ~]# echo $JAVA_HOME
+/usr/lib/jvm/java-11-openjdk-11.0.10.0.9-0.el7_9.x86_64</programlisting>
+          </listitem>
+        </orderedlist>
+      </section>
+
+      <section id="ha_ucpe_mg">
+        <title>Installing the Enea uCPE Manager in High Availabilty
+        mode</title>
+
+        <para>These steps must be taken on each of the CentOS 7 machines that
+        will host the Enea uCPE Manager.</para>
+
+        <para>As the root user, go to the distribution folder of the Enea uCPE
+        Manager, and run:</para>
+
+        <programlisting>[root@localhost distro]#./install.sh /opt/ \
+Enea_NFV_Access_uCPEManager_2.3.0-build23.tar.gz
+This will install uCPEManager into /opt/ucpemanager folder.
+Select the following options, while asked by the installation script:
+Are you using the embedded PostgreSQL database? [Y/N]: N
+External database selected, getting user information ...
+Press 1 for PostgreSQL, 2 for MariaDB, 3 for SQL Server, 4 for Oracle and 5 \
+for MySQL: 2
+Specify database server name(s) or IP Address(es): \
+192.168.10.11,192.168.10.12,…,192.168.10.16 *(see note)
+Specify database ID (or name) [ucpemanager]: 
+Specify database server port [3306]: 
+Specify database user name [root]: enea
+Specify database password [root]: somepassword
+Specify database startup thread pool size [1]: 
+Creating database configuration file \
+/opt//ucpemanager/application/config/databaseConfig.xml ...
+Done .
+…
+Installing ucpemanager service ..
+Specify service username [ucpemanager]: 
+Specify service password [ucpemanager]: somepassword
+…
+Specify the IP address of the local interface: 192.168.10.11
+Is this server part of a cluster? [Y/N]: Y
+Specify the name of the cluster [ucpemanager]: 
+Specify the shared (virtual) cluster IP address: 192.168.10.10
+Specify the netmask for the cluster IP address [255.255.255.0]: 
+HA Configuration files modified successfully.
+Configuration complete.</programlisting>
+
+        <note>
+          <para>For each Enea uCPE Manager installation, place the local
+          interface IP first in the list of IPs. This will optimize database
+          communication, since the Enea uCPE Manager uses the list of IPs
+          sequentially, therefore using the internal loopback interface for
+          communicating with the database.</para>
+        </note>
+
+        <para>Once the servers are up and running, log into the <emphasis
+        role="bold">Primary</emphasis> and go to <emphasis
+        role="bold">System</emphasis> and select <emphasis role="bold">Cluster
+        View</emphasis>. The list of Enea uCPE Managers should be displayed,
+        with one listed as Primary and the rest as Backup.</para>
+      </section>
+    </section>
+
+    <section id="ha_upgrade">
+      <title>Upgrading a High Availability Deployment</title>
+
+      <para>Upgrading a High Availabilty deployment is a highly complex,
+      multi-step process that requires care to ensure both consistency and
+      high-availability. Some steps need to be done manually.</para>
+
+      <orderedlist>
+        <listitem>
+          <para>We start with the assumption that
+          <literal>ucpeManager-1</literal>is the "PRIMARY" server.</para>
+        </listitem>
+
+        <listitem>
+          <para>Shut down database services on one side of the network, for
+          example: MariaDB-4, MariaDB-5 and MariaDB-6.</para>
+        </listitem>
+
+        <listitem>
+          <para>Disconnect the network interfaces towards the VPN for machines
+          MariaDB-4, MariaDB-5 and MariaDB-6. This will prevent any attempts
+          at failover/synchronization.</para>
+        </listitem>
+
+        <listitem>
+          <para>Run the upgrade process on <literal>ucpeManager-3</literal>
+          and <literal>ucpeManager-4</literal>. This will upgrade the service
+          to the current release. Once the upgrade process completes, shutdown
+          the Enea uCPE Manager service on both machines.</para>
+        </listitem>
+
+        <listitem>
+          <para>Disconnect the <literal>ucpeManager-2</literal> machine from
+          the network (which will take MariaDB-2 offline as well). At this
+          point, only the "PRIMARY" server is running, this is the start of
+          the interval when we are susceptible to single-server
+          failure.</para>
+        </listitem>
+
+        <listitem>
+          <para>Shutdown the MariaDB-2 process and run the Enea uCPE Manager
+          upgrade process on <literal>ucpeManager-2</literal>. This will
+          upgrade the service to the current release. Once the upgrade process
+          completes, shutdown the uCPE Manager service on the machine.</para>
+        </listitem>
+
+        <listitem>
+          <para>Reconnect the network interfaces towards the VPN for MariaDB-4
+          (<literal>ucpeManager-3</literal>), MariaDB-5
+          (<literal>ucpeManager-4</literal>) and MariaDB-6
+          (<literal>ucpeManager-2</literal>). Restart database services on
+          MariaDB-2, MariaDB-4, MariaDB-5 and MariaDB-6. This will allow
+          database services on all machines to synchronize, any data that has
+          been modified during the upgrade process will be made
+          consistent.</para>
+        </listitem>
+
+        <listitem>
+          <para>Shutdown the "Primary" server
+          (<literal>ucpeManager-1</literal>). At this point, the service is no
+          longer available.</para>
+        </listitem>
+
+        <listitem>
+          <para>Start the Enea uCPE Manager services on
+          <literal>ucpeManager-2</literal>. This machine will come up as the
+          new "PRIMARY" with the upgraded software. As part of the startup
+          process, it will upgrade the database and perform any other
+          upgrade-related functionality.</para>
+        </listitem>
+
+        <listitem>
+          <para>At this point (once startup completes), service is available.
+          However, we are still susceptible to single-server failure.</para>
+        </listitem>
+
+        <listitem>
+          <para>Start the Enea uCPE Manager services on
+          <literal>ucpeManager-3</literal> and
+          <literal>ucpeManager-4</literal>. At this point, we are in
+          highly-available mode.</para>
+        </listitem>
+
+        <listitem>
+          <para>Upgrade the Enea uCPE Manager on
+          <literal>ucpeManager-1</literal> (the one that has been shut down).
+          Once that upgrade is complete and the service restarts, the entire
+          setup has been upgraded to the new version.</para>
+        </listitem>
+      </orderedlist>
+    </section>
+  </section>
 </chapter>
\ No newline at end of file
diff --git a/doc/book-enea-nfv-access-getting-started/doc/images/high_av_setup.png b/doc/book-enea-nfv-access-getting-started/doc/images/high_av_setup.png
new file mode 100644
index 0000000..e2edd67
--- /dev/null
+++ b/doc/book-enea-nfv-access-getting-started/doc/images/high_av_setup.png