Container Bundling and Cross-Architecture Deployment ===================================================== This document describes how to bundle containers into Yocto images at build time using the container-cross-install system. This enables deploying Docker and Podman containers from x86_64 build systems to ARM64/x86_64 targets without requiring container runtimes on the build host. Prerequisites ------------- Enable the vcontainer distro feature in local.conf: DISTRO_FEATURES:append = " virtualization vcontainer" Enable multiconfig for blob builds: BBMULTICONFIG = "vruntime-aarch64 vruntime-x86-64" Choosing How to Bundle Containers --------------------------------- There are two ways to bundle containers into a host image: 1. **BUNDLED_CONTAINERS variable** (simpler, no extra recipe needed) # In local.conf or image recipe BUNDLED_CONTAINERS = "container-base:docker myapp-container:docker:autostart" 2. **container-bundle packages** (more flexible) # Create a bundle recipe, then add to IMAGE_INSTALL inherit container-bundle CONTAINER_BUNDLES = "myapp-container:autostart" ### Decision Guide Use Case | BUNDLED_CONTAINERS | Bundle Recipe --------------------------------------------|--------------------|-------------- Simple: containers in one host image | recommended | overkill Reuse containers across multiple images | repetitive | recommended Remote containers (docker.io/library/...) | not supported | required Package versioning and dependencies | not supported | supported Distribute pre-built container set | not supported | supported **For most single-image use cases, BUNDLED_CONTAINERS is simpler:** - No bundle recipe needed - Dependencies auto-generated at parse time - vrunner batch-import runs once for all containers **Use container-bundle.bbclass when you need:** - Remote container fetching via skopeo - A distributable/versioned package of containers - To share the same bundle across multiple different host images Component Relationships ----------------------- To bundle a local container like "myapp:autostart", three recipe types work together: 1. **Application Recipe** (builds the software) recipes-demo/myapp/myapp_1.0.bb - Compiles application binaries - Creates installable package (myapp) 2. **Container Image Recipe** (creates OCI image containing the app) recipes-demo/images/myapp-container.bb - inherit image image-oci - IMAGE_INSTALL = "myapp" - Produces: ${DEPLOY_DIR_IMAGE}/myapp-container-latest-oci/ 3. **Bundle Recipe** (packages container images for deployment) - OPTIONAL recipes-demo/bundles/my-bundle_1.0.bb - inherit container-bundle - CONTAINER_BUNDLES = "myapp-container:autostart" - Creates installable package with OCI data ### Flow Diagram myapp_1.0.bb myapp-container.bb (application) (container image) │ │ │ IMAGE_INSTALL="myapp" │ inherit image-oci └──────────────┬────────────────┘ │ ▼ myapp-container-latest-oci/ (OCI directory in DEPLOY_DIR_IMAGE) │ │ BUNDLED_CONTAINERS or CONTAINER_BUNDLES ▼ container-image-host (target host image with containers pre-installed) ### Concrete Example (from meta-virtualization) - Application: `recipes-demo/autostart-test/autostart-test_1.0.bb` - Container image: `recipes-demo/images/autostart-test-container.bb` - Usage: `BUNDLED_CONTAINERS = "autostart-test-container:docker:autostart"` Using BUNDLED_CONTAINERS ------------------------ Set in local.conf or image recipe: BUNDLED_CONTAINERS = "container-base:docker myapp-container:podman:autostart" ### Format name:runtime[:autostart][:external] - **name**: Container image recipe name or OCI directory name - **runtime**: `docker` or `podman` - **autostart**: Optional - `autostart`, `always`, `unless-stopped`, `on-failure` - **external**: Optional - skip dependency generation for third-party blobs ### Examples # Yocto-built containers (dependencies auto-generated) BUNDLED_CONTAINERS = "container-base:docker" BUNDLED_CONTAINERS = "myapp-container:podman:autostart" # Third-party blobs (no dependency generated) BUNDLED_CONTAINERS = "vendor-image:docker:external" # Legacy format (still supported) BUNDLED_CONTAINERS = "container-base-latest-oci:docker" Using container-bundle.bbclass ------------------------------ Create a bundle recipe: # recipes-demo/bundles/my-bundle_1.0.bb inherit container-bundle CONTAINER_BUNDLES = "\ myapp-container:autostart \ mydb-container \ docker.io/library/redis:7 \ " # Required for remote containers: CONTAINER_DIGESTS[docker.io_library_redis_7] = "sha256:..." To get the digest for a remote container, use skopeo: skopeo inspect docker://docker.io/library/redis:7 | jq -r '.Digest' Install in your host image: IMAGE_INSTALL:append:pn-container-image-host = " my-bundle" Container Autostart ------------------- Containers can be configured to start automatically on boot: | Policy | Description | |--------|-------------| | `autostart` | Alias for unless-stopped (recommended) | | `always` | Always restart container | | `unless-stopped` | Restart unless manually stopped | | `on-failure` | Restart only on non-zero exit code | **Generated files:** - Docker: `/lib/systemd/system/container-.service` - Podman: `/etc/containers/systemd/.container` (Quadlet format) vdkr and vpdmn - Virtual Container Runtimes =========================================== vdkr (virtual docker) and vpdmn (virtual podman) are tools that execute container commands inside a QEMU-emulated environment. They enable cross-architecture container operations during Yocto builds. | Tool | Runtime | State Directory | |------|---------|-----------------| | `vdkr` | Docker (dockerd + containerd) | `~/.vdkr//` | | `vpdmn` | Podman (daemonless) | `~/.vpdmn//` | Quick Start ----------- # Build and install the standalone SDK MACHINE=qemux86-64 bitbake vcontainer-tarball ./tmp/deploy/sdk/vcontainer-standalone.sh -d /tmp/vcontainer -y source /tmp/vcontainer/init-env.sh # List images vdkr images # Import an OCI container vdkr vimport ./my-container-oci/ myapp:latest # Export storage for deployment vdkr --storage /tmp/docker-storage.tar vimport ./container-oci/ myapp:latest # Clean persistent state vdkr clean Architecture Selection ---------------------- vdkr/vpdmn detect target architecture automatically. Override with: | Method | Example | Priority | |--------|---------|----------| | `--arch` / `-a` flag | `vdkr -a aarch64 images` | Highest | | Executable name | `vdkr-x86_64 images` | 2nd | | `VDKR_ARCH` env var | `export VDKR_ARCH=aarch64` | 3rd | | Config file | `~/.config/vdkr/arch` | 4th | | Host architecture | `uname -m` | Lowest | Memory Resident Mode -------------------- Keep QEMU VM running for fast command execution (~1s vs ~30s): vdkr vmemres start # Start daemon vdkr images # Fast! vdkr pull alpine:latest # Fast! vdkr vmemres stop # Stop daemon Commands -------- ### Docker-Compatible images, run, import, load, save, pull, tag, rmi, ps, rm, logs, start, stop, exec ### Extended (vdkr-specific) | Command | Description | |---------|-------------| | `vimport [name:tag]` | Import OCI directory or tarball | | `vrun [opts] [cmd]` | Run with entrypoint cleared | | `clean` | Remove persistent state | | `vmemres start/stop/status` | Memory resident VM control | How It Works ============ The container-cross-install system uses QEMU to run container tools (Docker/Podman) for the target architecture during the build. This solves the "pseudo problem" where container tools fail under Yocto's fakeroot. Architecture: ┌─────────────────────────────────────────────────────────────────┐ │ QEMU VM (target architecture) │ │ ┌───────────────────────────────────────────────────────────┐ │ │ │ rootfs.img (squashfs, ~100-130MB) │ │ │ │ - Docker or Podman tools │ │ │ │ - Processes OCI containers │ │ │ │ - Outputs storage tar via console │ │ │ └───────────────────────────────────────────────────────────┘ │ │ │ │ Drive Layout: │ │ /dev/vda = rootfs.img (tools) │ │ /dev/vdb = input disk (OCI containers) │ │ /dev/vdc = state disk (Docker/Podman storage) │ └─────────────────────────────────────────────────────────────────┘ Blobs are built via multiconfig and deployed to: tmp-vruntime-aarch64/deploy/images/qemuarm64/vdkr/ tmp-vruntime-x86-64/deploy/images/qemux86-64/vdkr/ Testing ======= cd /opt/bruce/poky/meta-virtualization # Run container-cross-install tests pytest tests/test_container_cross_install.py -v # Run vdkr/vpdmn CLI tests pytest tests/test_vdkr.py tests/test_vpdmn.py -v --vdkr-dir /tmp/vcontainer Quick Reference Commands ======================== Build host image with bundled containers: cd /opt/bruce/poky source oe-init-build-env build # Ensure local.conf has: # DISTRO_FEATURES:append = " virtualization vcontainer" # BBMULTICONFIG = "vruntime-aarch64 vruntime-x86-64" # BUNDLED_CONTAINERS = "container-base:docker autostart-test-container:docker:autostart" MACHINE=qemux86-64 bitbake container-image-host Build the vdkr/vpdmn SDK tarball: # Build blobs for desired architectures (sequentially to avoid deadlocks) bitbake mc:vruntime-x86-64:vdkr-initramfs-create mc:vruntime-x86-64:vpdmn-initramfs-create bitbake mc:vruntime-aarch64:vdkr-initramfs-create mc:vruntime-aarch64:vpdmn-initramfs-create # Build SDK tarball MACHINE=qemux86-64 bitbake vcontainer-tarball # Output: tmp/deploy/sdk/vcontainer-standalone.sh Install and test SDK: ./tmp/deploy/sdk/vcontainer-standalone.sh -d /tmp/vcontainer -y source /tmp/vcontainer/init-env.sh vdkr images vdkr vimport /path/to/container-base-latest-oci/ test:latest vdkr vmemres start vdkr images vdkr vmemres stop vdkr clean See Also ======== - `classes/container-cross-install.bbclass` - Main bundling class - `classes/container-bundle.bbclass` - Package-based bundling - `recipes-containers/vcontainer/README.md` - vdkr/vpdmn detailed documentation - `tests/README.md` - Test documentation