vrunner: poll for clean VM exit before SIGTERM in daemon_stop

The daemon_stop() flow sent ===SHUTDOWN=== over virtio-serial and then
slept exactly 2 seconds before unconditionally SIGTERMing QEMU. The
guest's graceful_shutdown(), which is what the SHUTDOWN command
ultimately triggers, does:

  sync
  umount /var/lib/containers/storage   (ext4 journal commit)
  sync
  blockdev --flushbufs (per disk)
  sync
  sleep 2
  reboot -f

Under load this routinely takes 5-30 seconds — the ext4 journal commit
on the state disk after a vimport that just wrote tens of MB of layer
blobs is the slow step. A fixed 2-second wait followed by SIGTERM kills
the guest mid-umount and leaves the state disk's journal half-committed:
layer files have correct inode metadata but partially-unwritten data
extents.

The next memres session remounts that disk and reads the apparently-OK
files. Tar-split reassembly during podman save / podman push then hits
the unwritten extents and produces:

  Error: reading blob sha256:<hash>: EOF
  Error: reading blob sha256:<hash>: file integrity checksum failed
         for "<file>"

Reported via yocto-patches as a workaround in run-push-containers
(`<runtime> image rm --all` before push) on the autobuilder. The
"first push works, subsequent fail" pattern in that report comes from
the first push hitting a clean session and subsequent pushes inheriting
the corrupted state disk from the prior SIGTERM-truncated shutdown.

Replace the fixed sleep with a poll loop that watches for the QEMU
process to exit, up to 60 seconds (120 * 0.5s). 60s is generous enough
to cover heavy ext4 journal commits; short enough that a truly hung
guest doesn't block the caller indefinitely. The existing SIGTERM and
SIGKILL escalation paths remain as fallbacks for the genuinely-stuck
case.

Reproducer: vimport an OCI image as testimg:latest, save it, memres
restart, re-vimport, save again. Without this fix the second save
fails 100% of the time on the same blob digest with EOF or CRC error.
With this fix six consecutive vimport+save cycles across three restart
rounds complete cleanly.

Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>

1 files changed, 23 insertions, 3 deletions

diff --git a/recipes-containers/vcontainer/files/vrunner.sh b/recipes-containers/vcontainer/files/vrunner.sh
index 2fd61655..92db55d2 100755
--- a/recipes-containers/vcontainer/files/vrunner.sh
+++ b/recipes-containers/vcontainer/files/vrunner.sh
@@ -607,13 +607,33 @@ daemon_stop() {
     local pid=$(cat "$DAEMON_PID_FILE")
     log "INFO" "Stopping daemon (PID: $pid)..."
 
-    # Send shutdown command via socket
+    # Send shutdown command via socket, then poll until the VM exits.
+    #
+    # The guest's graceful_shutdown() does sync + umount of
+    # /var/lib/containers/storage + blockdev --flushbufs + sync + sleep 2
+    # + reboot -f. Under load (e.g. tens of MB of just-imported layer
+    # blobs awaiting ext4 journal commit) this routinely takes 5-30
+    # seconds. A fixed 2-second wait followed by SIGTERM kills the
+    # guest mid-umount and leaves the state disk's ext4 journal
+    # half-committed: layer files have correct inode metadata but
+    # partially-unwritten data extents, and the next session's reads
+    # hit EOF or CRC failures during tar-split layer reassembly:
+    #
+    #   Error: reading blob sha256:<hash>: EOF
+    #   Error: reading blob sha256:<hash>: file integrity checksum
+    #          failed for "<file>"
     if [ -S "$DAEMON_SOCKET" ]; then
         echo "===SHUTDOWN===" | socat - "UNIX-CONNECT:$DAEMON_SOCKET" 2>/dev/null || true
-        sleep 2
+        # Poll up to 60s (120 * 0.5s). Generous enough to cover heavy
+        # ext4 journal commits; short enough that a truly hung guest
+        # doesn't block the caller indefinitely.
+        for _i in $(seq 1 120); do
+            kill -0 "$pid" 2>/dev/null || break
+            sleep 0.5
+        done
     fi
 
-    # If still running, kill it
+    # If still running after the graceful window, escalate.
     if kill -0 "$pid" 2>/dev/null; then
         log "INFO" "Sending SIGTERM..."
         kill "$pid" 2>/dev/null || true