Merge pull request #546 from advancedtelematic/Backport/sumo/secondaries-and-CI-improvement

Backport/sumo/secondaries and ci improvement

17 files changed, 120 insertions, 87 deletions

diff --git a/README.adoc b/README.adoc
index 5c4e5bc..8711e86 100644
--- a/README.adoc
+++ b/README.adoc
@@ -19,6 +19,20 @@ toc::[]
 
 If you don't already have a Yocto project that you want to add OTA to, you can use the https://docs.atsgarage.com/quickstarts/raspberry-pi.html[HERE OTA Connect Quickstart] project to rapidly get up and running on a Raspberry Pi. It takes a standard https://www.yoctoproject.org/tools-resources/projects/poky[poky] distribution, and adds OTA and OSTree capabilities.
 
+=== Dependencies
+
+In addition to the link:https://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#required-packages-for-the-build-host[standard Yocto dependencies], meta-updater generally requires a few additional dependencies, depending on your use case and target platform. To install these additional packages on Debian/Ubuntu, run this:
+
+....
+sudo apt install cpu-checker default-jre parted
+....
+
+To build for https://github.com/advancedtelematic/meta-updater-minnowboard[Minnowboard] with GRUB, you will also need to install https://github.com/tianocore/tianocore.github.io/wiki/OVMF[TianoCore's ovmf] package on your host system. On Debian/Ubuntu, you can do so with this command:
+
+....
+sudo apt install ovmf
+....
+
 === Adding meta-updater capabilities to your build
 
 If you already have a Yocto-based project and you want to add atomic filesystem updates to it, you just need to do three things:
@@ -72,7 +86,7 @@ If your board isn't supported yet, you can add board integration code yourself.
 
 You may take a look into https://github.com/advancedtelematic/meta-updater-minnowboard[Minnowboard] or https://github.com/advancedtelematic/meta-updater-raspberrypi[Raspberry Pi] integration layers for examples.
 
-Although we have used U-Boot so far, other boot loaders can be configured work with OSTree as well.
+Although we have focused on U-Boot and GRUB so far, other bootloaders can be configured to work with OSTree as well.
 
 Your images will also need network connectivity to be able to reach an actual OTA backend. Our 'poky-sota' distribution does not mandate or install a default network manager but our supported platforms use the `virtual/network-configuration` recipe, which can be used as a starting example.
 
@@ -89,7 +103,7 @@ Your images will also need network connectivity to be able to reach an actual OT
 * `SOTA_DEPLOY_CREDENTIALS` - when set to '1' (default value), deploys credentials to the built image. Override it in `local.conf` to built a generic image that can be provisioned manually after the build.
 * `SOTA_CLIENT_PROV` - which provisioning method to use. Valid options are `aktualizr-shared-prov`, `aktualizr-device-prov`, and `aktualizr-device-prov-hsm`. For more information on these provisioning methods, see the https://docs.ota.here.com/client-config/client-provisioning-methods.html[OTA Connect documentation]. The default is `aktualizr-shared-prov`. This can also be set to an empty string to avoid using a provisioning recipe.
 * `SOTA_CLIENT_FEATURES` - extensions to aktualizr. The only valid options are `hsm` (to build with HSM support) and `secondary-network` (to set up a simulated 'in-vehicle' network with support for a primary node with a DHCP server and a secondary node with a DHCP client).
-* `SOTA_SECONDARY_CONFIG_DIR` - a directory containing JSON configuration files for virtual secondaries on the host. These will be installed into `/etc/sota/ecus` on the device and automatically provided to aktualizr.
+* `SOTA_SECONDARY_CONFIG` - a file containing JSON configuration for secondaries. It will be installed into `/etc/sota/ecus` on the device and automatically provided to aktualizr. See link:https://github.com/advancedtelematic/aktualizr/blob/master/docs/posix-secondaries-bitbaking.adoc[here] for more details.
 * `SOTA_HARDWARE_ID` - a custom hardware ID that will be written to the aktualizr config. Defaults to MACHINE if not set.
 * `RESOURCE_xxx_pn-aktualizr` - controls maximum resource usage of the aktualizr service, when `aktualizr-resource-control` is installed on the image. See <<aktualizr service resource control>> for details.
 * `SOTA_POLLING_SEC` - sets polling interval for aktualizr to check for updates if aktualizr-polling-sec is included in the image.
@@ -216,11 +230,7 @@ IMAGE_INSTALL_append = " dropbear "
 
 3. Some tests require that `SOTA_PACKED_CREDENTIALS` is set in your `conf/local.conf`. See the <<sota-related-variables-in-localconf,SOTA-related variables in local.conf>> section.
 
-4. To be able to build an image for the grub tests, you will need to install https://github.com/tianocore/tianocore.github.io/wiki/OVMF[TianoCore's ovmf] package on your host system. On Debian-like systems, you can do so with this command:
-+
-```
-sudo apt install ovmf
-```
+4. To be able to build an image for the GRUB tests, you will need to install the ovmf package as described in the <<Dependencies,dependencies>>.
 
 5. Run oe-selftest:
 +
@@ -259,3 +269,9 @@ Provisioning procedure depends on your provisioning recipe, i.e. the value of `S
 ** put URL to the server backend (together with protocol prefix and port number) at `/var/sota/gateway.url`. If you're using HERE OTA Connect, you can find the URL in the `autoprov.url` file in your credentials archive.
 ** put root CA certificate (for the *server*, not for the *device*) at `/var/sota/import/root.crt`.
 ** put client certificate and private key to slots 1 and 2 of the PKCS#11-compatible device.
+
+== License
+
+This code is licensed under the link:COPYING.MIT[MIT license], a copy of which can be found in this repository. All code is copyright HERE Europe B.V., 2016-2019.
+
+We require that contributors accept the terms of Linux Foundation's link:https://developercertificate.org/[Developer Certificate of Origin]. Please see the https://github.com/advancedtelematic/aktualizr/blob/master/CONTRIBUTING.md[contribution instructions of aktualizr] for more information.
diff --git a/conf/include/bblayers/sota_raspberrypi3.inc b/conf/include/bblayers/sota_raspberrypi3.inc
index 03f8f44..42d5eb1 100644
--- a/conf/include/bblayers/sota_raspberrypi3.inc
+++ b/conf/include/bblayers/sota_raspberrypi3.inc
@@ -1,3 +1,4 @@
 BBLAYERS += "${METADIR}/meta-openembedded/meta-python"
 BBLAYERS += "${METADIR}/meta-updater-raspberrypi"
 BBLAYERS += "${METADIR}/meta-raspberrypi"
+BBLAYERS += "${METADIR}/meta-openembedded/meta-networking"
diff --git a/conf/local.conf.sample.append b/conf/local.conf.sample.append
new file mode 100644
index 0000000..4588ec3
--- /dev/null
+++ b/conf/local.conf.sample.append
@@ -0,0 +1,35 @@
+
+#
+# meta-updater configuration, see README.adoc and aktualizr's
+# documentation for more options and detailed documentation
+#
+
+MACHINE = "##MACHINE##"
+DISTRO = "poky-sota-systemd"
+
+# General SOTA setup
+#SOTA_CLIENT_PROV = "aktualizr-auto-prov"
+#SOTA_PACKED_CREDENTIALS = "/path/to/credentials.zip"
+
+# Uncomment this line to start an ssh server at boot automatically
+#IMAGE_FEATURES += "ssh-server-dropbear"
+
+# Uncomment this line to set the log level of aktualizr to 'debug' (from 'info'
+# by default)
+#IMAGE_INSTALL_append += " aktualizr-log-debug"
+
+# Store systemd logs in persistent storage
+#
+# It greatly helps diagnosing issues on testing devices but should be
+# carefully weighted against file system usage and flash device wear for
+# production systems. Please refer to systemd's docs for more details
+IMAGE_INSTALL_append += " systemd-journald-persistent"
+
+# Set resource limits for aktualizr service
+#
+# Uncomment these lines to change the default parameters.
+#
+#RESOURCE_CPU_WEIGHT_pn-aktualizr = "100"
+#RESOURCE_MEMORY_HIGH_pn-aktualizr = "100M"
+#RESOURCE_MEMORY_MAX_pn-aktualizr = "80%"
+IMAGE_INSTALL_append += " aktualizr-resource-control"
diff --git a/lib/oeqa/selftest/cases/updater_qemux86_64.py b/lib/oeqa/selftest/cases/updater_qemux86_64.py
index 80d135f..2b4726c 100644
--- a/lib/oeqa/selftest/cases/updater_qemux86_64.py
+++ b/lib/oeqa/selftest/cases/updater_qemux86_64.py
@@ -358,7 +358,7 @@ class IpSecondaryTests(OESelftestTestCase):
             self._test_ctx.append_config('SOTA_CLIENT_PROV = " aktualizr-shared-prov "')
 
         def is_ecu_registered(self, ecu_id):
-            max_number_of_tries = 20
+            max_number_of_tries = 40
             try_counter = 0
 
             # aktualizr-info is not always able to load ECU serials from DB
diff --git a/recipes-sota/aktualizr/aktualizr_git.bb b/recipes-sota/aktualizr/aktualizr_git.bb
index 28c51d9..ddc0de1 100644
--- a/recipes-sota/aktualizr/aktualizr_git.bb
+++ b/recipes-sota/aktualizr/aktualizr_git.bb
@@ -6,7 +6,7 @@ LICENSE = "MPL-2.0"
 LIC_FILES_CHKSUM = "file://${S}/LICENSE;md5=9741c346eef56131163e13b9db1241b3"
 
 DEPENDS = "boost curl openssl libarchive libsodium sqlite3 asn1c-native"
-DEPENDS_append = "${@bb.utils.contains('PTEST_ENABLED', '1', ' coreutils-native ostree-native aktualizr-native ', '', d)}"
+DEPENDS_append = "${@bb.utils.contains('PTEST_ENABLED', '1', ' coreutils-native net-tools-native ostree-native aktualizr-native ', '', d)}"
 RDEPENDS_${PN}_class-target = "aktualizr-configs lshw"
 RDEPENDS_${PN}-host-tools = "aktualizr aktualizr-repo aktualizr-cert-provider ${@bb.utils.contains('PACKAGECONFIG', 'sota-tools', 'garage-deploy garage-push', '', d)}"
 
@@ -31,7 +31,7 @@ SRC_URI = " \
 SRC_URI[md5sum] = "e104ccd4f32e52571a5fc0e5042db050"
 SRC_URI[sha256sum] = "c590be1a57523bfe097af82279eda5c97cf40ae47fb27162cf33c469702c8a9b"
 
-SRCREV = "fce5854ff10e7efd52d69bbaf68dc2af990d5746"
+SRCREV = "9c592cf9d8dfcd995d47753f2be7bd1a2b56c7da"
 BRANCH ?= "master"
 
 S = "${WORKDIR}/git"
@@ -106,16 +106,6 @@ do_install_append () {
         printf "[provision]\nprimary_ecu_hardware_id = ${SOTA_HARDWARE_ID}\n" > ${D}${libdir}/sota/conf.d/40-hardware-id.toml
     fi
 
-    if [ -n "${SOTA_SECONDARY_CONFIG_DIR}" ]; then
-        if [ -d "${SOTA_SECONDARY_CONFIG_DIR}" ]; then
-            install -m 0700 -d ${D}${sysconfdir}/sota/ecus
-            install -m 0644 "${SOTA_SECONDARY_CONFIG_DIR}"/* ${D}${sysconfdir}/sota/ecus/
-            printf "[uptane]\nsecondary_configs_dir = /etc/sota/ecus/\n" > ${D}${libdir}/sota/conf.d/30-secondary-configs-dir.toml
-        else
-            bbwarn "SOTA_SECONDARY_CONFIG_DIR is set to an invalid directory (${SOTA_SECONDARY_CONFIG_DIR})"
-        fi
-    fi
-
     install -m 0755 -d ${D}${systemd_unitdir}/system
     aktualizr_service=${@bb.utils.contains('SOTA_CLIENT_FEATURES', 'serialcan', '${WORKDIR}/aktualizr-serialcan.service', '${WORKDIR}/aktualizr.service', d)}
     install -m 0644 ${aktualizr_service} ${D}${systemd_unitdir}/system/aktualizr.service
diff --git a/recipes-support/python-canonicaljson/python-canonicaljson.bb b/recipes-support/python-canonicaljson/python-canonicaljson.bb
deleted file mode 100644
index d8a0728..0000000
--- a/recipes-support/python-canonicaljson/python-canonicaljson.bb
+++ /dev/null
@@ -1,18 +0,0 @@
-DESCRIPTION = "python-canonicaljson recipe"
-
-LICENSE = "Apache-2.0"
-LIC_FILES_CHKSUM = "file://${S}/LICENSE;md5=2ee41112a44fe7014dce33e26468ba93"
-
-SRCREV = "92e2c06871cc275c2a8b8e3e899141a212aae0e8"
-SRC_URI = "git://github.com/matrix-org/python-canonicaljson.git"
-S = "${WORKDIR}/git"
-
-# Generate with:
-#   git describe --tags | cut -b2-
-PV = "1.0.0"
-inherit setuptools
-
-RDEPENDS_${PN} = "\
-        python-simplejson \
-        python-frozendict \
-        "
diff --git a/recipes-support/python-frozendict/python-frozendict.bb b/recipes-support/python-frozendict/python-frozendict.bb
deleted file mode 100644
index 79fe5c8..0000000
--- a/recipes-support/python-frozendict/python-frozendict.bb
+++ /dev/null
@@ -1,12 +0,0 @@
-DESCRIPTION = "python-frozendict recipe"
-
-LICENSE = "BSD"
-LIC_FILES_CHKSUM = "file://${S}/LICENSE.txt;md5=f4da037a49c09b456fdbbc7a5bd36132"
-
-SRCREV = "c5d16bafcca7b72ff3e8f40d3a9081e4c9233f1b"
-SRC_URI = "git://github.com/slezica/python-frozendict.git"
-S = "${WORKDIR}/git"
-
-PV = "1.2"
-inherit setuptools
-
diff --git a/recipes-support/python-petname/python-petname.bb b/recipes-support/python-petname/python-petname.bb
deleted file mode 100644
index 9abd5d9..0000000
--- a/recipes-support/python-petname/python-petname.bb
+++ /dev/null
@@ -1,16 +0,0 @@
-DESCRIPTION = "python-petname recipe"
-
-LICENSE = "Apache-2.0"
-LIC_FILES_CHKSUM = "file://${S}/LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
-
-SRCREV = "d0b767cdb1567defb104f29c3fd022239a7f231e"
-SRC_URI = "git://github.com/dustinkirkland/python-petname.git"
-S = "${WORKDIR}/git"
-
-PV = "2.2"
-inherit setuptools
-RDEPENDS_${PN} = " python-setuptools \
-                   python-argparse \
-                   "
-
-FILES_${PN} = "${libdir} ${bindir}/petname"
diff --git a/recipes-support/systemd-journald-persistent/files/10-persistent-journal.conf b/recipes-support/systemd-journald-persistent/files/10-persistent-journal.conf
new file mode 100644
index 0000000..a56527c
--- /dev/null
+++ b/recipes-support/systemd-journald-persistent/files/10-persistent-journal.conf
@@ -0,0 +1,3 @@
+[Journal]
+Storage=persistent
+SystemMaxUse=64M
diff --git a/recipes-support/systemd-journald-persistent/systemd-journald-persistent.bb b/recipes-support/systemd-journald-persistent/systemd-journald-persistent.bb
new file mode 100644
index 0000000..1715fe1
--- /dev/null
+++ b/recipes-support/systemd-journald-persistent/systemd-journald-persistent.bb
@@ -0,0 +1,20 @@
+SUMMARY = "Configuration for systemd-journald"
+DESCRIPTION = "Provides configuration for systemd-journald, so that logs are \
+stored on persistent storage"
+LICENSE = "MPL-2.0"
+LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MPL-2.0;md5=815ca599c9df247a0c7f619bab123dad"
+
+PACKAGE_ARCH = "${MACHINE_ARCH}"
+
+SRC_URI_append = " file://10-persistent-journal.conf"
+PR = "r1"
+
+S = "${WORKDIR}"
+
+FILES_${PN} = "${systemd_unitdir}/journald.conf.d/*"
+
+do_install() {
+    install -d ${D}/${systemd_unitdir}/journald.conf.d
+    install -m 0644 ${WORKDIR}/10-persistent-journal.conf ${D}/${systemd_unitdir}/journald.conf.d
+}
+
diff --git a/recipes-test/demo-config/shared-conf.inc b/recipes-test/demo-config/shared-conf.inc
index ce2bb44..c5ab598 100644
--- a/recipes-test/demo-config/shared-conf.inc
+++ b/recipes-test/demo-config/shared-conf.inc
@@ -2,4 +2,4 @@ SECONDARY_IP ?= "10.0.3.2"
 SECONDARY_PORT ?= "9050"
 PRIMARY_IP ?= "10.0.3.1"
 PRIMARY_PORT ?= "9040"
-PRIMARY_WAIT_TIMEOUT ?= "120"
+PRIMARY_WAIT_TIMEOUT ?= "240"
diff --git a/recipes-test/demo-network-config/files/26-multihomed-client.network b/recipes-test/demo-network-config/files/26-multihomed-client.network
new file mode 100644
index 0000000..f1e6cc6
--- /dev/null
+++ b/recipes-test/demo-network-config/files/26-multihomed-client.network
@@ -0,0 +1,9 @@
+[Match]
+Name=@IFNAME@
+
+[Network]
+Description=Multihomed network. DHCP-assigned IP for Primary<->Backend. Statically assigned IP for Primary<->Secondary
+DHCP=yes
+
+[Address]
+Address=@ADDR@
diff --git a/recipes-test/demo-network-config/network-config.inc b/recipes-test/demo-network-config/network-config.inc
new file mode 100644
index 0000000..ed623d4
--- /dev/null
+++ b/recipes-test/demo-network-config/network-config.inc
@@ -0,0 +1,17 @@
+SRC_URI_append = "\
+    file://26-${CONF_TYPE}-client.network \
+    "
+
+SECONDARY_INTERFACE ?= "${@ 'eth0' if d.getVar('MACHINE') == 'raspberrypi3' else 'enp0s5'}"
+
+do_install_append() {
+    bbnote "Network configuration type to be applied: ${CONF_TYPE}"
+    install -d ${D}/usr/lib/systemd/network
+    install -m 0644 ${WORKDIR}/26-${CONF_TYPE}-client.network ${D}/usr/lib/systemd/network/
+    sed -i -e 's|@ADDR@|${IP_ADDR}|g' \
+           -e 's|@IFNAME@|${SECONDARY_INTERFACE}|g' \
+           ${D}/usr/lib/systemd/network/26-${CONF_TYPE}-client.network
+
+}
+
+# vim:set ts=4 sw=4 sts=4 expandtab:
diff --git a/recipes-test/demo-network-config/primary-network-config.bb b/recipes-test/demo-network-config/primary-network-config.bb
index c7daa15..d840a95 100644
--- a/recipes-test/demo-network-config/primary-network-config.bb
+++ b/recipes-test/demo-network-config/primary-network-config.bb
@@ -18,8 +18,10 @@ do_install() {
 }
 
 PRIMARY_IP ?= "10.0.3.1"
+
 IP_ADDR = "${PRIMARY_IP}"
+CONF_TYPE ?= "${@ 'multihomed' if d.getVar('MACHINE') == 'raspberrypi3' and d.getVar('RPI_WIFI_ENABLE') != '1' else 'static'}"
 
-require static-network-config.inc
+require network-config.inc
 
 # vim:set ts=4 sw=4 sts=4 expandtab:
diff --git a/recipes-test/demo-network-config/secondary-network-config.bb b/recipes-test/demo-network-config/secondary-network-config.bb
index c70d88a..b1d70f1 100644
--- a/recipes-test/demo-network-config/secondary-network-config.bb
+++ b/recipes-test/demo-network-config/secondary-network-config.bb
@@ -23,7 +23,8 @@ do_install() {
 
 SECONDARY_IP ?= "10.0.3.2"
 IP_ADDR = "${SECONDARY_IP}"
+CONF_TYPE = "static"
 
-require static-network-config.inc
+require network-config.inc
 
 # vim:set ts=4 sw=4 sts=4 expandtab:
diff --git a/recipes-test/demo-network-config/static-network-config.inc b/recipes-test/demo-network-config/static-network-config.inc
deleted file mode 100644
index e64675e..0000000
--- a/recipes-test/demo-network-config/static-network-config.inc
+++ /dev/null
@@ -1,16 +0,0 @@
-SRC_URI_append = "\
-    file://26-static-client.network \
-    "
-
-SECONDARY_INTERFACE ?= "enp0s5"
-
-do_install_append() {
-    install -d ${D}/usr/lib/systemd/network
-    install -m 0644 ${WORKDIR}/26-static-client.network ${D}/usr/lib/systemd/network/
-    sed -i -e 's|@ADDR@|${IP_ADDR}|g' \
-           -e 's|@IFNAME@|${SECONDARY_INTERFACE}|g' \
-           ${D}/usr/lib/systemd/network/26-static-client.network
-
-}
-
-# vim:set ts=4 sw=4 sts=4 expandtab:
diff --git a/scripts/envsetup.sh b/scripts/envsetup.sh
index e912824..5827bc2 100755
--- a/scripts/envsetup.sh
+++ b/scripts/envsetup.sh
@@ -25,11 +25,12 @@ METADIR=${METADIR:-${SOURCEDIR}/../..}
 
 if [[ ! -f "${BUILDDIR}/conf/local.conf" ]]; then
   source "$METADIR/poky/oe-init-build-env" "$BUILDDIR"
+
   echo "METADIR  := \"\${@os.path.abspath('${METADIR}')}\"" >> conf/bblayers.conf
   cat "${METADIR}/meta-updater/conf/include/bblayers/sota.inc" >> conf/bblayers.conf
   cat "${METADIR}/meta-updater/conf/include/bblayers/sota_${MACHINE}.inc" >> conf/bblayers.conf
-  echo "MACHINE = \"${MACHINE}\"" >> conf/local.conf
-  echo "DISTRO = \"poky-sota-systemd\"" >> conf/local.conf
+
+  sed "s/##MACHINE##/$MACHINE/g" "${METADIR}/meta-updater/conf/local.conf.sample.append" >> conf/local.conf
 else
   source "$METADIR/poky/oe-init-build-env" "$BUILDDIR"
 fi