summaryrefslogtreecommitdiffstats
path: root/recipes-containers/vcontainer
Commit message (Collapse)AuthorAgeFilesLines
* vcontainer-tarball: unset OECORE_NATIVE_SYSROOT in CITim Orling2026-06-131-0/+2
| | | | | | | | | | | Add 'unset OECORE_NATIVE_SYSROOT' to the end of environment-setup-ci for the same reasons it needed to be in environment-setup-none. This fixes issues seen on AutoBuilder workers which use buildtools-tarball and also usage of oe-run-native (e.g. for skopeo-native or cosign-native). Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer-tarball: track rootfs hash via explicit mcdependsBruce Ashfield2026-06-121-0/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | vcontainer-tarball bundles vdkr and vpdmn rootfs images alongside the helper scripts into a single-shot SDK tarball. Those rootfs images live in the vruntime-x86-64 / vruntime-aarch64 multiconfigs and are pulled into the tarball at do_populate_sdk time from tmp-<mc>/deploy/images/<machine>/<tool>/<arch>/rootfs.img. For sstate to behave correctly, the tarball task's hash must reflect the actual rootfs content. In practice it didn't. A rootfs content change — adding netavark to the image, switching the runtime from iptables to nftables — left consumers of the SDK with a stale runtime even though their config asked for the new behaviour. Cleansstate of vcontainer-tarball alone didn't help: the path through the cached intermediates was serving the old image back. The recipe was relying on the chain rootfs-image:do_build -> initramfs-create:do_compile -> initramfs-create:do_deploy -> mcdepends -> tarball:do_populate_sdk to propagate the rootfs hash. The break is at do_deploy: it reads its rootfs.img out of DEPLOY_DIR, which is a known sstate sharp edge. do_deploy can sstate-hit on its own hash while the file it deploys came from a stale prior run, after which the downstream tarball task hash is computed against the stale content and also hits cache. Add a direct mcdepends from do_populate_sdk to {vdkr,vpdmn}-rootfs-image:do_image_complete for each enabled vruntime multiconfig. The rootfs-image task's hash now feeds the tarball hash directly, independent of the intermediate do_deploy step. If a rootfs content change moves the rootfs-image hash, the tarball is forced to rebuild. When the chain was already healthy this dep adds nothing new and costs nothing. Verified end-to-end: bitbake vcontainer-tarball -c cleansstate bitbake vcontainer-tarball sha256sum tmp/deploy/sdk/vcontainer-standalone.sh # -> f8e75f947a506202d88a6eb196a3616f5088b11247207beab589e91fb6f2bba7 echo 'IMAGE_INSTALL:append:pn-vpdmn-rootfs-image = " bash"' \ >> conf/local.conf bitbake vcontainer-tarball sha256sum tmp/deploy/sdk/vcontainer-standalone.sh # -> 1e89554500b2a4aaabc149a01f85fad2e78ba35796977469e7fcb1ecf15a856f Hashes differ — the rootfs content change correctly invalidated the tarball's sstate hash and produced a fresh build. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vrunner: warn on shutdown-escalation that state disk may be corruptedBruce Ashfield2026-06-121-0/+14
| | | | | | | | | | | | | | | | | | | | | | | | | When the guest doesn't exit within the daemon_stop() poll window, the graceful_shutdown() didn't complete and the state disk's ext4 journal may have unflushed writes from this session. Any subsequent escalation (QMP quit, SIGTERM, SIGKILL) leaves the disk image at risk of corruption: layer files end up with correct inode metadata but partially-unwritten data extents, surfacing on the next session as Error: reading blob sha256:<hash>: EOF Error: reading blob sha256:<hash>: file integrity checksum failed for "<file>" These errors are silent today — the operator hits them later in another session with no way to correlate back to the original cause. Emit a WARN as soon as the poll window expires, naming the symptom and pointing at the remediation (`memres restart --clean`) so the next session starts from a known-good state instead of inheriting a corrupted image. This is purely diagnostic — no behavior change to the shutdown sequence itself. The escalation paths (QMP quit, SIGTERM, SIGKILL) still run unchanged. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vrunner: use QMP quit before SIGTERM in daemon_stop escalationBruce Ashfield2026-06-121-1/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | When the SHUTDOWN-via-virtio-serial poll window expires and the QEMU process is still alive, send a QMP quit before SIGTERMing. Functionally near-equivalent to SIGTERM at the process level (both converge on qemu_system_killed() and a block-layer flush) but goes through QEMU's monitor interface, which is the same mechanism hv_idle_shutdown() already uses for idle-timeout shutdowns. Unifying the two escalation paths makes the code easier to reason about — one channel for "ask QEMU to exit cleanly," one fallback for "the monitor isn't responding." QMP quit is NOT a replacement for the virtio-serial ===SHUTDOWN=== command: it stops QEMU without giving the guest a chance to run its own graceful_shutdown() (which is what cleanly unmounts the state disk). It's only an improvement over SIGTERM as the *fallback* when the guest hasn't responded to the SHUTDOWN command within the poll window. The full architectural fix — QMP system_powerdown driving an ACPI handler in the guest that calls graceful_shutdown() — needs guest-side acpid wiring and is tracked separately. SIGTERM and SIGKILL remain as further fallbacks for cases where QMP itself is unresponsive (e.g. older configs without a QMP socket, or QEMU hung in a state where the monitor stops servicing commands). Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vrunner: poll for clean VM exit before SIGTERM in daemon_stopBruce Ashfield2026-06-121-3/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The daemon_stop() flow sent ===SHUTDOWN=== over virtio-serial and then slept exactly 2 seconds before unconditionally SIGTERMing QEMU. The guest's graceful_shutdown(), which is what the SHUTDOWN command ultimately triggers, does: sync umount /var/lib/containers/storage (ext4 journal commit) sync blockdev --flushbufs (per disk) sync sleep 2 reboot -f Under load this routinely takes 5-30 seconds — the ext4 journal commit on the state disk after a vimport that just wrote tens of MB of layer blobs is the slow step. A fixed 2-second wait followed by SIGTERM kills the guest mid-umount and leaves the state disk's journal half-committed: layer files have correct inode metadata but partially-unwritten data extents. The next memres session remounts that disk and reads the apparently-OK files. Tar-split reassembly during podman save / podman push then hits the unwritten extents and produces: Error: reading blob sha256:<hash>: EOF Error: reading blob sha256:<hash>: file integrity checksum failed for "<file>" Reported via yocto-patches as a workaround in run-push-containers (`<runtime> image rm --all` before push) on the autobuilder. The "first push works, subsequent fail" pattern in that report comes from the first push hitting a clean session and subsequent pushes inheriting the corrupted state disk from the prior SIGTERM-truncated shutdown. Replace the fixed sleep with a poll loop that watches for the QEMU process to exit, up to 60 seconds (120 * 0.5s). 60s is generous enough to cover heavy ext4 journal commits; short enough that a truly hung guest doesn't block the caller indefinitely. The existing SIGTERM and SIGKILL escalation paths remain as fallbacks for the genuinely-stuck case. Reproducer: vimport an OCI image as testimg:latest, save it, memres restart, re-vimport, save again. Without this fix the second save fails 100% of the time on the same blob digest with EOF or CRC error. With this fix six consecutive vimport+save cycles across three restart rounds complete cleanly. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vpdmn: switch to nftables and add kernel config for netavarkBruce Ashfield2026-06-041-1/+1
| | | | | | | | | | | | | | | | | | | | Netavark dropped iptables support entirely and now uses nft directly for firewall rules (masquerade, DNAT, connection tracking). The vpdmn rootfs had iptables installed but the vruntime kernel had CONFIG_NF_TABLES disabled, causing netavark to fail at runtime with "Protocol not supported" when trying to initialize the nftables Netlink socket. Replace iptables with nftables in vpdmn-rootfs-image IMAGE_INSTALL and add container-net.cfg kernel config fragment enabling NF_TABLES, NFT_MASQ, NFT_NAT, NFT_CT, NFT_FIB and related options required by netavark's nftables firewall driver. The fragment is gated on the vcontainer DISTRO_FEATURE so it only applies to vruntime/vcontainer kernels, not arbitrary builds that happen to include meta-virtualization. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* podman: switch from CNI to netavark as sole network backendBruce Ashfield2026-06-041-6/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | Podman's vendored containers/common library removed CNI support entirely in commit 8d1f636e40 (March 2026). The NetworkBackend() function now unconditionally rejects any backend other than netavark, making network_backend = "cni" in containers.conf a runtime error. The vpdmn rootfs originally used CNI because netavark's dependency chain (nmap -> libpcap -> bluez5 -> cairo) was too heavy for the vruntime BBMASK environment. That dependency chain no longer exists — netavark v1.16.0+ uses a built-in Rust connection tester and has no heavyweight runtime dependencies. Recipe changes: - Clear BUILDTAGS_EXTRA default (was CNI build tag, now a no-op) - Set VIRTUAL-RUNTIME_container_networking and _container_dns with strong assignments so podman always RDEPENDS on netavark and aardvark-dns regardless of distro config - Remove ${sysconfdir}/cni from FILES vpdmn rootfs changes: - Replace cni with netavark + aardvark-dns in IMAGE_INSTALL - Update containers.conf network_backend from cni to netavark Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer-common: support nested OCI layout and fix vimport shell errorsBruce Ashfield2026-05-101-31/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | The multi-arch OCI functions (is_oci_image_index, get_oci_platforms, select_platform_manifest) only checked index.json directly for platform information. With the skopeo-compatible nested OCI layout — where index.json references a single image index blob that in turn contains the per-platform manifests — the functions failed to detect multi-arch images because index.json no longer contains platform entries. Add _resolve_oci_platform_file() helper that handles both layouts: - Flat: platform info directly in index.json (legacy/simple case) - Nested: index.json → image index blob → platform manifests All three multi-arch functions now use this single helper, eliminating the layout resolution logic that would otherwise be duplicated in each. Also fixes two issues in the vimport case block: - 'local' keyword used outside a function (bash error on line 1879). The vimport handler is in a case statement in the main script body, not inside a function, so 'local' is invalid. The original multi-arch code was written assuming it would be inside a function. - OCI_SELECTED_PLATFORM was blank in output because select_platform_manifest sets it inside a $() subshell, where variable assignments are lost. Use normalize_arch_to_oci directly for the display message instead. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: add --config / VDKR_CONFIG for docker/podman auth credentialsTim Orling2026-04-296-0/+392
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add a VDKR_CONFIG / VPDMN_CONFIG env var and a matching --config <path> CLI flag that passes an existing docker config.json / podman auth.json into the QEMU-hosted container runtime so pulls from private registries work without having to retype --registry-user / --registry-pass on every command. Security posture (defence in depth): - Host-side pre-flight validation in vrunner.sh (validate_auth_config): reject symlinks, non-regular files, missing / unreadable files, files smaller than 2 bytes (minimum "{}") or larger than 1 MiB, and any permissions other than 0400 / 0600 / 0200. WARN if not owned by the invoking user. - Stage the file into a dedicated per-invocation directory under $TEMP_DIR at mode 0400 inside a 0700 parent; auto-cleanup rides the existing EXIT/INT/TERM trap. - Expose the staged file over a *separate* read-only virtio-9p tag ("${TOOL_NAME}_auth") so credentials cannot leak into the general /mnt/share input/output directory or into storage.tar outputs. - Only a boolean flag ("${CMDLINE_PREFIX}_auth=1") is appended to the kernel cmdline - never the path, the env var name, or the contents. - Guest mounts /mnt/auth ro,nosuid,nodev,noexec, copies to the runtime's canonical path, then unmounts immediately so neither the runtime nor user workloads keep a reference to the host staging directory. vrunner.sh: - Initialise AUTH_CONFIG from $VDKR_CONFIG / $VPDMN_CONFIG - Parse --config <path> (overrides the env vars) - Add validate_auth_config() and setup_auth_share() with the rules above - Call setup_auth_share in both the daemon start path and the non-daemon / batch-import path vcontainer-init-common.sh: - Default RUNTIME_AUTH="0" and parse ${VCONTAINER_RUNTIME_PREFIX}_auth=* from the kernel cmdline - Define mount_auth_share() / unmount_auth_share() using the per-runtime "${VCONTAINER_RUNTIME_NAME}_auth" 9p tag, mounted at /mnt/auth with ro,nosuid,nodev,noexec vdkr-init.sh: - install_auth_config() copies /mnt/auth/config.json to /root/.docker/config.json (mode 0600; parent dir 0700) - Called after install_registry_ca in main flow so --config takes precedence over --registry-user / --registry-pass; logs a NOTE when both mechanisms are supplied - Unmounts /mnt/auth after copy vpdmn-init.sh: - install_auth_config() copies to /run/containers/0/auth.json (the rootful podman canonical path) and exports REGISTRY_AUTH_FILE so the creds are picked up regardless of podman's search order - Mode 0600 on the file, 0700 on the containing directory - Unmounts /mnt/auth after copy vcontainer-common.sh: - Honour $VDKR_CONFIG / $VPDMN_CONFIG, parse --config, and forward AUTH_CONFIG to vrunner.sh via --config in build_runner_args - Document the flag and env vars in show_usage README.md: - New "Passing an existing docker/podman auth file (--config)" section with examples for both runtimes, a table of target paths, and the full security model AI-Generated: Claude Cowork Opus 4.7 Signed-off-by: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer-common: fix vstorage commands with --state-dirBruce Ashfield2026-04-291-7/+15
| | | | | | | | | | | | | vstorage list/df/clean scanned DEFAULT_STATE_DIR (~/.vpdmn/) for arch subdirectories, ignoring --state-dir. On CI where tests use --state-dir ~/.vpdmn-test/x86_64, the default directory doesn't exist so vstorage list reports "(no storage directories found)" and test_vstorage_shows_memres_status fails. Derive VSTORAGE_ROOT from the parent of STATE_DIR when --state-dir is set, so all vstorage subcommands scan the correct storage root. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer-tarball: add CI-safe environment script for autobuilderBruce Ashfield2026-04-281-2/+27
| | | | | | | | | | | | | The existing environment-setup-* script uses BASH_SOURCE to derive VCONTAINER_DIR, which is empty when parsed by yocto-autobuilder-helper's enable_tools_tarball() since it doesn't evaluate shell expressions. Generate a separate environment-setup-ci with flat export lines using baked-in absolute paths from ${SDKPATH}/${SDKPATHNATIVE}. The AB parser picks these up directly. SDK relocation rewrites the paths at install time. The interactive bash script is unchanged. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* vcontainer-initramfs-create: fix kernel deploy dependency via do_buildBruce Ashfield2026-04-281-11/+10
| | | | | | | | | | | | | | | | | | | | | On sstate-accelerated builds, the kernel binary (bzImage/Image) was missing from MC_DEPLOY because do_compile depended on the image recipes' do_image_complete, which runs before do_build. The kernel deploy dependency (virtual/kernel:do_deploy) is attached to do_build in image.bbclass, so depending on do_image_complete cut the chain short and virtual/kernel:do_deploy was never guaranteed to have run. Fix by depending on do_build instead of do_image_complete. The image artifacts (cpio.gz, squashfs) are already in DEPLOY_DIR_IMAGE after do_image_complete, so they remain available. do_build additionally ensures virtual/kernel:do_deploy has completed, placing the kernel in MC_DEPLOY for our do_compile to copy. This avoids adding an explicit virtual/kernel:do_deploy dependency which would couple this recipe to the kernel and prevent use cases where the kernel is provided externally. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: detach background-process stdio from memres start callerBruce Ashfield2026-04-223-7/+35
| | | | | | | | | | | | | | | | | | | | | | | | The memres start operation spawns long-running background processes (host-side idle watchdog and Xen domain monitor) that persist beyond the vrunner.sh script. These processes inherited file descriptors 0/1/2 from the parent shell without redirection. When invoked through a harness capturing output via pipes—such as pytest's subprocess.run(..., capture_output=True)—the inherited pipe write-ends kept the caller's read/communicate() operations blocked until memres stop executed, potentially for up to 30 minutes (IDLE_TIMEOUT default). The fix fully detaches stdio from three background spawners: - vrunner.sh: Watchdog subshell now redirects stdin from /dev/null, stdout/stderr to /dev/null, and uses disown - vrunner-backend-qemu.sh: Adds stdin redirection from /dev/null to existing log file redirections - vrunner-backend-xen.sh: Applies same detachment plus disown for daemon mode; redirects stdin for ephemeral-mode console reader From: Tim Orling <tim.orling@konsulko.com> Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer-initramfs-create: fix kernel path for multiconfig buildsBruce Ashfield2026-04-141-8/+12
| | | | | | | | | | | | | | | | | | | | The kernel was copied from DEPLOY_DIR_IMAGE which resolves to the main build config's deploy directory — not the multiconfig's. On autobuilder setups with sstate, this caused the kernel to be missing from the vdkr blob directory because: 1. The mcdepends (mc:<mc>::virtual/kernel:do_deploy) depended on the main config's kernel, which may be a different architecture 2. DEPLOY_DIR_IMAGE pointed to the main deploy dir, not the multiconfig's tmp-vruntime-<arch>/deploy/images/<machine>/ Fix by using MC_DEPLOY (the multiconfig's deploy directory) for the kernel path, consistent with how initramfs and rootfs are already copied. Remove the cross-multiconfig mcdepends since the kernel is built as a transitive dependency of the rootfs image within the same multiconfig. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: fix registry login and TLS certificate chainBruce Ashfield2026-04-133-3/+58
| | | | | | | | | | | | | | | | | | | | | | | vcontainer-common.sh: - Fix docker/podman login hanging silently in daemon mode. The login command requires interactive stdin for the password prompt, which the non-interactive daemon-send channel cannot provide. Use daemon-interactive mode (same as vshell/exec -it) which provides TTY passthrough via the virtio-serial + socat + expect channel. ca-certificates bbappend: - Add Let's Encrypt E8 (ECDSA) and R11 (RSA) intermediate certificates via a ca-certificates bbappend. Certs are fetched at build time from letsencrypt.org and installed into the standard CA certificate path. update-ca-certificates incorporates them into the system CA bundle. - Some registries (e.g., registry.yocto.io) don't send the full certificate chain, causing TLS verification to fail with Go programs (Docker, skopeo, podman) even though the root CAs are present. vdkr-rootfs-image.bb: - Add ca-certificates to IMAGE_INSTALL Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer-tarball: fix PACKAGE_ARCH for SPDX do_create_spdxTim Orling2026-04-121-1/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | PACKAGE_ARCH must be "${SDK_ARCH}-${SDKPKGSUFFIX}" (e.g. x86_64-nativesdk) to match the pattern explicitly listed in SSTATE_ARCHS in meta/classes-global/sstate.bbclass. do_create_recipe_spdx writes the static SPDX document under DEPLOY_DIR_SPDX/${SSTATE_PKGARCH}/static/ where SSTATE_PKGARCH = PACKAGE_ARCH. do_create_spdx then calls find_jsonld() which searches only the arch directories listed in SPDX_MULTILIB_SSTATE_ARCHS (derived from SSTATE_ARCHS via all_multilib_tune_values). The previous value "${SDK_ARCH}_${SDK_OS}" evaluates to "x86_64_linux" which is NOT listed in SSTATE_ARCHS, so the static document was written to a directory never searched, causing the fatal: ERROR: Could not find a static SPDX document named static-vcontainer-tarball The correct value "${SDK_ARCH}-${SDKPKGSUFFIX}" evaluates to "x86_64-nativesdk" which IS listed in SSTATE_ARCHS as the literal "${SDK_ARCH}-${SDKPKGSUFFIX}" entry, and matches the pattern used by buildtools-tarball.bb (the reference for host-only SDK tarballs with TARGET_ARCH = "none" / TARGET_OS = "none"). Signed-off-by: Tim Orling <tim.orling@konsulko.com>
* vruntime, vrunner, conftest: fix multiconfig and batch import issuesBruce Ashfield2026-04-063-11/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | vruntime.conf: - Reset all VIRTUAL-RUNTIME_container_* variables to prevent CONTAINER_PROFILE selections from leaking into vruntime multiconfigs (e.g., podman profile pulling netavark into vruntime builds) - Disable ptest for glib-2.0 — its -ptest RDEPENDS chain (python3-dbusmock -> python3-pygobject -> cairo -> fontconfig) pulls the entire graphics stack which is masked in vruntime - OE-core commit 159148f4de2 replaced DISTRO_FEATURES_BACKFILL_CONSIDERED with DISTRO_FEATURES_OPTED_OUT. The old variable no longer has any effect, which meant ptest, gobject-introspection-data, wayland, and other features were no longer being blocked in vruntime builds. This caused glib-2.0's ptest RDEPENDS to pull in the cairo → fontconfig → freetype graphics stack, which is masked by the vruntime BBMASK. - Set PREFERRED_PROVIDER_virtual/runc with strong assignment to ensure the unified runc recipe is used vrunner.sh: - Fix batch import exit code handling: wrap import chain in subshell and make the images listing best-effort. The previous '&& podman images' caused false failures when podman images couldn't initialize its network backend. Using 'exit' was also wrong as the command runs inside PID 1 init's eval — exit kills init causing kernel panic. vpdmn-rootfs-image.bb: - Switch from netavark to CNI networking — netavark's dependency chain (nmap -> libpcap -> bluez5 -> python3-pygobject -> cairo) cannot be built under the vruntime BBMASK environment - Add nsswitch.conf override (files-only backend) to prevent libnss_systemd segfaults — the vruntime VM uses busybox init with no systemd running, but libnss_systemd.so is pulled in as a dependency and segfaults on NSS resolution vdkr-rootfs-image.bb: - Document skopeo requirement for batch import conftest.py: - Add --k3s-timeout option and k3s/multinode markers for upcoming K3s test suite Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer-tarball: set S to UNPACKDIR for do_qa_unpack checkBruce Ashfield2026-03-161-0/+3
| | | | | | | | | The recipe only has file:// SRC_URI entries which unpack directly into UNPACKDIR, not a ${BP} subdirectory. The new do_qa_unpack QA check in insane.bbclass warns when S doesn't exist after unpack. Set S explicitly to satisfy the check. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: fix daemon mode missing shared directory for 9pBruce Ashfield2026-02-261-1/+6
| | | | | | | | | | | DAEMON_SHARE_DIR was referenced in the CA certificate copy and idle watchdog paths but never assigned, causing 'cp: cannot create regular file /ca.crt: Permission denied' when starting the daemon. Create the share directory under DAEMON_SOCKET_DIR and register it as a 9p mount, matching the path expected by daemon_run(). Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vxn: add host-side OCI image cache and fix Docker iptables conflictBruce Ashfield2026-02-261-6/+268
| | | | | | | | | | | | | | | | | | | Add a host-side OCI image cache at ~/.vxn/images/ for the vdkr/vpdmn standalone Xen path. Images pulled via skopeo are stored in a content-addressed layout (refs/ symlinks + store/ OCI dirs) so subsequent runs hit the cache without network access. New commands on Xen: pull, images, rmi, tag, inspect, image <subcmd>. The run path is unchanged — cache integration into hv_prepare_container is deferred to a follow-up. Also fix Docker iptables conflict: when docker-moby and vxn-docker-config coexist on Dom0, Docker's default FORWARD DROP policy blocks DHCP for Xen DomU vifs on xenbr0. Adding "iptables": false to daemon.json prevents Docker from modifying iptables since VM-based containers manage their own network stack. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vxn: add Docker/Podman integration and CLI frontendsBruce Ashfield2026-02-264-40/+167
| | | | | | | | | Add vdkr/vpdmn as Dom0 target packages with Xen auto-detection, native Docker/Podman config sub-packages, and OCI runtime fixes for Docker compatibility (JSON logging, root.path, kill --all, monitor PID lifecycle). Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vxn: add controlling terminal and clean up interactive outputBruce Ashfield2026-02-261-9/+8
| | | | | | | | | | | | | Use setsid -c to establish a controlling terminal for the container shell, fixing "can't access tty; job control turned off" and enabling Ctrl-C signal delivery. Run in a subshell so setsid() succeeds without forking (PID 1 is already a session leader). Remove [vxn] diagnostic markers from interactive output now that terminal mode is working. Suppress mount warning on read-only input disk. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vxn: fix terminal mode hang and enable interactive container supportBruce Ashfield2026-02-264-17/+67
| | | | | | | | | | | | | | | | | | | | | | | | The containerd shim's Create RPC hung indefinitely because go-runc captures the OCI runtime's stdout via a pipe, and cmd.Wait() blocks until all holders of the pipe's write end close it. The background monitor subshell inherited this pipe fd and held it open, preventing the shim from ever proceeding to ReceiveMaster() or calling Start. Fix by closing inherited stdout/stderr in the terminal-mode monitor with exec >/dev/null before entering the domain poll loop. Non-terminal mode is unaffected because the shim configures IO via FIFO dup2, where cmd.Wait() only waits for process exit. Additional changes for terminal mode support: - vxn-sendtty: set PTY to raw mode (cfmakeraw) before sending fd - vxn-oci-runtime: wait up to 5s for xenconsoled PTY, capture sendtty return code, write persistent debug file to /root/vxn-tty-debug, log every runtime invocation, remove stale debug logging - vxn-init.sh: add [vxn] diagnostic markers for terminal visibility, suppress kernel console messages early in interactive mode - vcontainer-preinit.sh: suppress kernel messages in quiet mode Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: inject vxn-init.sh into vdkr and vpdmn rootfs imagesBruce Ashfield2026-02-262-0/+8
| | | | | | | | | Install vxn-init.sh alongside the existing init scripts in both vdkr and vpdmn rootfs images. The Xen backend selects it at boot via the vcontainer.init=/vxn-init.sh kernel command line parameter. Add file-checksums tracking so rootfs rebuilds when the script changes. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: add bundle command for OCI runtime bundle creationBruce Ashfield2026-02-261-0/+118
| | | | | | | | | | Add 'bundle' command to the vcontainer CLI for creating OCI runtime bundles from container images. Pulls the image via skopeo, extracts layers into rootfs/, resolves entrypoint/cmd/env from OCI config, and generates config.json. Supports command override via -- separator. Only available on the Xen (vxn) backend. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: generalize init scripts for pluggable hypervisor backendsBruce Ashfield2026-02-263-15/+22
| | | | | | | | | | | Make preinit and guest init scripts hypervisor-agnostic: - vcontainer-preinit.sh: add vcontainer.init= cmdline parameter for init script selection and vcontainer.blk= for block device prefix (QEMU uses /dev/vda, Xen uses /dev/xvda) - vdkr-init.sh, vpdmn-init.sh: use NINE_P_TRANSPORT variable for 9p mount transport (virtio for QEMU, xen for Xen) Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: add QEMU hypervisor backend and register in recipesBruce Ashfield2026-02-263-1/+267
| | | | | | | | | | Add vrunner-backend-qemu.sh implementing the hypervisor interface for QEMU (arch setup, KVM detection, disk/network/9p options, VM lifecycle, QMP control). Register backend scripts in vcontainer-native and vcontainer-tarball recipes so they are available in both build-time and standalone tarball contexts. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vxn: add containerd OCI runtime integrationBruce Ashfield2026-02-265-0/+782
| | | | | | | | | | | | | | | | | | | | | | | | | | Add shell-based OCI runtime (vxn-oci-runtime) that enables containerd to manage Xen DomU containers through the standard runc shim. Non-terminal container output flows back to ctr via the shim's pipe mechanism. New files: - vxn-oci-runtime: OCI runtime (create/start/state/kill/delete/features/logs) - vxn-sendtty.c: SCM_RIGHTS helper for terminal mode PTY passing - containerd-shim-vxn-v2: PATH trick wrapper for runc shim coexistence - containerd-config-vxn.toml: CRI config (vxn default, runc fallback) - vctr: convenience wrapper injecting --runtime io.containerd.vxn.v2 Key design: - Monitor subprocess uses wait on xl console (not sleep-polling) for instant reaction when domain dies, then extracts output markers and writes to stdout (shim pipe -> containerd FIFO -> ctr client) - cmd_state checks monitor PID liveness (not domain status) to prevent premature cleanup race that killed monitor before output - cmd_delete always destroys remnant domains (no --force needed) - Coexists with runc: /usr/libexec/vxn/shim/runc symlink + PATH trick Verified: vctr run --rm, vctr run -d, vxn standalone, vxn daemon mode. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vxn: add per-container DomU lifecycle and memres persistent DomUBruce Ashfield2026-02-264-154/+922
| | | | | | | | | | | | | | | | | | | | | | | | Per-container DomU lifecycle: - run -d: per-container DomU with daemon loop and PTY-based IPC - ps: show Running vs Exited(code) via ===STATUS=== PTY query - exec/stop/rm: send commands to per-container DomU - logs: retrieve entrypoint output from running DomU - Entrypoint death detection with configurable grace period - Graceful error messages for ~25 unsupported commands - Command quoting fix: word-count+cut preserves internal spaces Memres (persistent DomU for fast container dispatch): - vxn memres start/stop/status/list for persistent DomU management - vxn run auto-dispatches to memres via xl block-attach + RUN_CONTAINER - Guest daemon loop handles ===RUN_CONTAINER===: mount hot-plugged xvdb, extract OCI rootfs, chroot exec entrypoint, unmount, report - Falls back to ephemeral mode when memres is occupied (PING timeout) - Xen-specific memres list shows xl domains and orphan detection Tested: vxn memres start + vxn run --rm alpine echo hello + vxn run --rm hello-world both produce correct output. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vxn: fix non-interactive mode for clean container outputBruce Ashfield2026-02-262-10/+41
| | | | | | | | | | | | | | | | | | | | Fix several issues preventing non-interactive mode (vxn --no-daemon run) from showing clean container output: - Fix console capture: check DAEMON_MODE instead of DAEMON_SOCKET in Xen backend so ephemeral runs use xl console capture instead of the daemon socat bridge (DAEMON_SOCKET is always set, DAEMON_MODE is only "start" for actual daemon launches) - Fix race condition: add post-loop marker detection after VM exits, with 2s delay for xl console to flush its buffer - Add stdbuf -oL to xl console for line-buffered output - Suppress mke2fs stdout (was only redirecting stderr) - Suppress kernel console messages during VM lifecycle in non-verbose mode - Fix grep -P (Perl regex) for BusyBox compatibility in exit code parsing - Preserve temp directory on failure for debugging - Fix hardcoded "QEMU" in error messages to "VM" Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vxn: add Xen DomU container runtime with OCI image supportBruce Ashfield2026-02-265-311/+1403
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | vxn runs OCI containers as Xen DomU guests — the VM IS the container. No Docker/containerd runs inside the guest; the init script directly mounts the container rootfs and execs the entrypoint via chroot. Host-side (Dom0): - vxn.sh: Docker-like CLI wrapper (sets HYPERVISOR=xen) - vrunner-backend-xen.sh: Xen xl backend for vrunner - hv_prepare_container(): pulls OCI images via skopeo, resolves entrypoint from OCI config using jq on host - xl create for VM lifecycle (PVH on aarch64, PV on x86_64) - Bridge networking with iptables DNAT for port forwards - Console capture via xl console for ephemeral mode Guest-side (DomU): - vxn-init.sh: mounts container rootfs from input disk, extracts OCI layers, execs entrypoint via chroot - Supports containers with or without /bin/sh - grep/sed fallback for OCI config parsing (no jq needed) - Daemon mode with command loop on hvc1 - vcontainer-init-common.sh: hypervisor detection, head -n fix - vcontainer-preinit.sh: init selection via vcontainer.init= Build system: - vxn-initramfs-create.inc: assembles boot blobs from vruntime multiconfig, injects vxn-init.sh into rootfs squashfs - vxn_1.0.bb: Dom0 package with scripts + blobs - nostamp on install/package chain (blobs from DEPLOY_DIR are untracked by sstate) - vxn.cfg: Xen PV kernel config fragment Tested: vxn -it --no-daemon run --rm hello-world Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer-tarball: remove parse-time bannerBruce Ashfield2026-02-111-34/+3
| | | | | | | | | | | The anonymous python function prints a banner unconditionally at parse time, which means it appears when building any recipe (e.g. xen-image-minimal), not just vcontainer-tarball. Remove the parse-time banner since the post-build banner in do_populate_sdk:append() already provides the same information and only fires when actually building the tarball. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: fix yocto-check-layer mcdepends parse errorBruce Ashfield2026-02-092-10/+20
| | | | | | | | | | | | | | | | | | Fix yocto-check-layer failure: ERROR: Multiconfig dependency mc::vruntime-x86-64:vpdmn-initramfs-create:do_deploy depends on nonexistent multiconfig configuration named configuration vruntime-x86-64 Several recipes and classes declared static mcdepends referencing vruntime-aarch64 and vruntime-x86-64 multiconfigs. When parsed without BBMULTICONFIG set (e.g. yocto-check-layer), BitBake validates these and fails because the referenced multiconfigs don't exist. Move mcdepends into anonymous python functions and only set them when the target multiconfig exists in BBMULTICONFIG, following the pattern established in meta/classes-recipe/kernel-fit-image.bbclass. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: add secure registry support with virtio-9p CA transportBruce Ashfield2026-02-094-6/+246
| | | | | | | | | | | | | | | | | | | | | | | | | Enable vdkr/vcontainer to pull from TLS-secured registries by transporting the CA certificate via virtio-9p shared folder. vcontainer-common.sh: Add --secure-registry, --ca-cert, --registry-user, --registry-password CLI options. Auto-detect bundled CA cert at registry/ca.crt in the tarball and enable secure mode automatically. vrunner.sh: Copy CA cert to the virtio-9p shared folder for both daemon and non-daemon modes. Fix daemon mode missing _9p=1 kernel cmdline parameter which prevented the init script from mounting the shared folder. vdkr-init.sh: Read CA cert from /mnt/share/ca.crt (virtio-9p) instead of base64-decoding from kernel cmdline (which caused truncation for large certificates). Install cert to /etc/docker/certs.d/{host}/ca.crt for Docker TLS verification. Support optional credential passing for authenticated registries. vcontainer-tarball.bb: Add script files to SRC_URI for proper file tracking and rebuild triggers. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: enable incremental builds by defaultBruce Ashfield2026-02-094-10/+38
| | | | | | | | | | | | | | | | | | | Previously, vcontainer recipes had [nostamp] flags that forced all tasks to rebuild on every bitbake invocation, even when nothing changed. This was added as a workaround for dependency tracking issues but caused slow rebuild times. Changes: - Make [nostamp] conditional on VCONTAINER_FORCE_BUILD variable - Default to normal stamp-based caching for faster incremental builds - file-checksums on do_rootfs still tracks init script changes - Add VCONTAINER_FORCE_BUILD status to the tarball build banner To enable the old always-rebuild behavior (for debugging dependency issues), set in local.conf: VCONTAINER_FORCE_BUILD = "1" Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer-tarball: build all architectures via single bitbake commandBruce Ashfield2026-02-091-6/+40
| | | | | | | | | | | | | | | | | | | | Previously, building vcontainer-tarball required multiple bitbake invocations or complex command lines to build both x86_64 and aarch64 blobs. This was a usability issue. Changes: - mcdepends now triggers builds for BOTH architectures automatically - VCONTAINER_ARCHITECTURES defaults to "x86_64 aarch64" (was auto-detect) - Add informational banner at parse time showing what will be built - Fix duplicate sanity check messages when multiconfig is active Usage is now simply: bitbake vcontainer-tarball To build only one architecture, set in local.conf: VCONTAINER_ARCHITECTURES = "x86_64" Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vdkr-init: improve Docker daemon startup logging and error handlingBruce Ashfield2026-02-091-5/+17
| | | | | | | | | | | Improve debugging capabilities when Docker daemon fails to start: - Log dockerd output to /var/log/docker.log instead of /dev/null - Capture docker info exit code and output for diagnostics - Show docker info error on every 10th iteration while waiting - Include last docker info output and docker.log tail on failure - Extend sleep on failure from 2s to 5s for log review Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: add host-side idle timeout with QMP shutdownBruce Ashfield2026-02-093-20/+154
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Implement reliable idle timeout for vmemres daemon mode using host-side monitoring with QMP-based shutdown, and container-aware idle detection via virtio-9p shared file. Host-side changes (vrunner.sh): - Add -no-reboot flag to QEMU for clean exit semantics - Spawn background watchdog when daemon starts - Watchdog monitors activity file timestamp - Check interval scales to idle timeout (timeout/5, clamped 10-60s) - Read container status from shared file (guest writes via virtio-9p) - Only shutdown if no containers are running - Send QMP "quit" command for graceful shutdown - Watchdog auto-exits if QEMU dies (no zombie processes) - Touch activity file in daemon_send() for user activity tracking Config changes (vcontainer-common.sh): - Add idle-timeout to build_runner_args() so it's always passed Guest-side changes (vcontainer-init-common.sh): - Add watchdog that writes container status to /mnt/share/.containers_running - Host reads this file instead of socket commands (avoids output corruption) - Close inherited virtio-serial fd 3 in watchdog subshell to prevent leaks - Guest-side shutdown logic preserved but disabled (QMP more reliable) - Handle Yocto read-only-rootfs volatile directories (/var/volatile) The shared file approach avoids sending container check commands through the daemon socket, which previously caused output corruption on the single-stream virtio-serial channel. The idle timeout is configurable via: vdkr vconfig idle-timeout <secs> Default: 1800 seconds (30 minutes) Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: consolidate initramfs-create recipesBruce Ashfield2026-02-093-80/+33
| | | | | | | | | | | | | | | | | | | Update vcontainer-initramfs-create.inc to use the image-based approach: - Depend on tiny-initramfs-image for cpio.gz (replaces file extraction) - Depend on rootfs-image for squashfs (unchanged) - Remove DEPENDS on squashfs-tools-native (no longer extracting files) Update recipe files to use the consolidated inc: - vdkr-initramfs-create_1.0.bb - vpdmn-initramfs-create_1.0.bb Boot flow remains unchanged: QEMU boots kernel + tiny initramfs -> preinit mounts rootfs.img from /dev/vda -> switch_root into rootfs.img -> vdkr-init.sh or vpdmn-init.sh runs Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: add tiny initramfs image infrastructureBruce Ashfield2026-02-095-3/+142
| | | | | | | | | | | | | | | | | | | | | Add proper Yocto image recipes for the tiny initramfs used by vdkr/vpdmn in the switch_root boot flow: - vcontainer-tiny-initramfs-image.inc: Shared image configuration - vcontainer-preinit_1.0.bb: Preinit script package (shared) - vdkr-tiny-initramfs-image.bb: Tiny initramfs for vdkr - vpdmn-tiny-initramfs-image.bb: Tiny initramfs for vpdmn The tiny initramfs contains only busybox and a preinit script that: 1. Mounts devtmpfs, proc, sysfs 2. Mounts the squashfs rootfs.img from /dev/vda 3. Creates tmpfs overlay for writes 4. Performs switch_root to the real rootfs This replaces ad-hoc file extraction with proper image-based builds, improving reproducibility and maintainability. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer-tarball: add nativesdk-expect dependencyBruce Ashfield2026-02-091-0/+1
| | | | | | | Add expect to the vcontainer SDK toolchain for interactive testing and automation scripts. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: fix runc/crun conflict in multiconfig buildsBruce Ashfield2026-02-092-10/+6
| | | | | | | | | | | | | | | | | | | | | | The vruntime distro is used for multiconfig builds of both vdkr (Docker/runc) and vpdmn (Podman/crun) images. When CONTAINER_PROFILE or VIRTUAL-RUNTIME_container_runtime is set, containerd and podman pull their preferred runtime via RDEPENDS, causing package conflicts. Fix by having vruntime distro NOT participate in CONTAINER_PROFILE: - Set VIRTUAL-RUNTIME_container_runtime="" to prevent automatic runtime selection - Explicitly install runc in vdkr-rootfs-image.bb - Explicitly install crun in vpdmn-rootfs-image.bb This allows both images to be built in the same multiconfig without conflicts, while standard container-host images continue to use CONTAINER_PROFILE normally. Also add kernel-modules to vdkr-rootfs-image for overlay filesystem support. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: add multi-arch OCI supportBruce Ashfield2026-02-091-4/+239
| | | | | | | | | | | | | | | | | | | | | Add functions to detect and handle multi-architecture OCI Image Index format with automatic platform selection during import. Also add oci-multiarch.bbclass for build-time multi-arch OCI creation. Runtime support (vcontainer-common.sh): - is_oci_image_index() - detect multi-arch OCI images - get_oci_platforms() - list available platforms - select_platform_manifest() - select manifest for target architecture - extract_platform_oci() - extract single platform to new OCI dir - normalize_arch_to_oci/from_oci() - architecture name mapping - Update vimport to auto-select platform from multi-arch images Build-time support (oci-multiarch.bbclass): - Create OCI Image Index from multiconfig builds - Collect images from vruntime-aarch64, vruntime-x86-64 - Combine blobs and create unified manifest list Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: add virtio-9p fast path for batch importsBruce Ashfield2026-02-094-50/+284
| | | | | | | | | | | | | Add virtio-9p filesystem support for faster storage output during batch container imports, replacing slow base64-over-console method. - Add --timeout option for configurable import timeouts - Mount virtio-9p share in batch-import mode - Parse _9p=1 kernel parameter for 9p availability - Write storage.tar directly to shared filesystem - Reduces import time from ~600s to ~11s for large containers Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: add vshell command for VM debug accessBruce Ashfield2026-02-091-0/+20
| | | | | | | | | | | | | | | | | | | | Add vshell command to vdkr/vpdmn for interactive shell access to the QEMU VM where Docker/Podman runs. This is useful for debugging container issues directly inside the virtual environment. Usage: vdkr vmemres start vdkr vshell # Now inside VM, can run docker commands directly docker images docker inspect <image> exit The vshell command requires the memory-resident daemon to be running (vmemres start). It opens an interactive shell via the daemon's --daemon-interactive mode. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vdkr: add registry configuration and pull fallbackBruce Ashfield2026-02-095-5/+493
| | | | | | | | | | | Add registry support to vdkr: - vconfig registry command for persistent config - --registry flag for one-off usage - Registry-first, Docker Hub fallback for pulls - Baked-in registry config via CONTAINER_REGISTRY_URL - Image commands (inspect, history, rmi, images) work without transform Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: fix ps -q to suppress port forward displayBruce Ashfield2026-02-091-1/+9
| | | | | | | | When using `ps -q` or `ps --quiet`, only container IDs should be output. The port forward registry display was being included, which broke cleanup code that expected just container IDs. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vrunner: update static port forwarding for bridge networkingBruce Ashfield2026-02-091-5/+8
| | | | | | | | | | | | | Update the static port forwarding (used at QEMU startup) to match the dynamic QMP port forwarding change. With bridge networking: - QEMU forwards host:port -> VM:port - Docker's iptables handles VM:port -> container:port Previously the static port forward went directly to the container port (host:8080 -> VM:80), which doesn't work with bridge networking where Docker needs to handle the final hop. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* recipes: fix multiconfig build commentsBruce Ashfield2026-02-091-2/+2
| | | | | | | | Fix the bitbake multiconfig commands in rootfs recipe comments. The multiconfig names are vruntime-aarch64 and vruntime-x86-64, not vdkr-*/vpdmn-*. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* vcontainer: update CLI for bridge networkingBruce Ashfield2026-02-091-12/+39
| | | | | | | | | | | | | | | | | | | Update the CLI wrapper to work with Docker bridge networking: - qmp_add_hostfwd(): Change QEMU port forward from host_port->guest_port to host_port->host_port. With bridge networking, Docker's iptables handles the final hop (VM:host_port -> container:guest_port). - Default network mode: Remove --network=host default. Docker's bridge is now the default, giving each container its own IP. Users can still explicitly use --network=host for legacy behavior. - Update help text to document the new bridge networking behavior. Port forwarding flow is now: Host:8080 -> QEMU -> VM:8080 -> Docker iptables -> Container:80 Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2026-09-04 07:37:59 +0000