linux/meta-virtualization.git/scripts, branch master

oe-go-mod-fetcher: fix stale clone cache in pseudo-version expansion

2026-05-30T04:22:21+00:00

resolve_pseudo_version_commit() expands a 12-character pseudo-version short hash to its full 40-character form by cloning the upstream repo into a cache directory and running 'git log --since=...'. The cache is reused on subsequent invocations and refreshed via 'git fetch --all'. The initial clone used 'git clone --bare', which does NOT install a remote.origin.fetch refspec. As a result, the later 'git fetch --all' exits 0 but actually fetches nothing — the cache stays frozen at the state of the very first clone, no matter how much time passes. For any pseudo-version whose timestamp is newer than the first time the cache was populated, the 'git log --since=' search returns no commits, expansion silently returns None, and the 12-char hash propagates verbatim into the generated SRC_URI .inc files. bitbake then refuses to parse the recipe with an unhelpful "Unable to resolve '<12 chars>' in upstream git repository in git ls-remote output for " error, because its git fetcher requires a full 40-char SHA. Hit on a nerdctl v2.2.1 -> v2.3.1-tip bump: the repo's self-reference module 'github.com/containerd/nerdctl/v2' at pseudo-version v2.0.0-20260528082953-89ecd85071c4 (timestamped 2026-05-28) failed to expand because the cached clone of github.com/containerd/nerdctl was created on 2026-03-19 and never advanced. Two changes: - clone with --mirror instead of --bare. --mirror sets up the +refs/*:refs/* refspec, so 'git fetch --all' actually pulls new refs on every subsequent run. - print a non-VERBOSE warning when expansion leaves any 12-char hash unresolved, listing the affected modules and pointing at the cache directory. The previous failure message was gated behind VERBOSE_MODE, so an exhausted cache produced a silent fall-through and the next visible error was bitbake's confusing parse failure. Signed-off-by: Bruce Ashfield

oe-go-mod-fetcher: encode Go module paths in LIC_FILES_CHKSUM URIs

2026-05-29T17:40:22+00:00

bitbake's do_populate_lic walks pkg/mod/@/ at build time, where uses Go's filesystem-safe encoding for ASCII uppercase letters: every 'A'-'Z' becomes '!' followed by the lowercase form, so `github.com/HdrHistogram/...` lives at `github.com/!hdr!histogram/...` on disk. The license-scan writer was emitting canonical-form paths ("github.com/HdrHistogram/..."), so do_populate_lic couldn't find the LICENSE file and failed with "invalid file" QA errors for every uppercase-bearing imported module — 6 errors in k3s's case. The same canonical-vs-encoded gap also lived in the filter input. _get_unpacked_modules walks GOMODCACHE and returned the encoded form, while modules.json (the loop driver in scan_module_licenses) uses the canonical form, so the `key in selected_set` test for uppercase modules silently failed — those modules were dropped from the filter output entirely. That's why cosign/docker-compose/incus's committed go-mod-licenses.inc files don't have any uppercase entries: the build passed do_populate_lic because the entries were absent, not because they were correct. Normalize to canonical form internally and encode only at the output boundary: - _decode_go_module_path: '!x' -> 'X' for ASCII lowercase x. Used by _get_unpacked_modules so all three filter sources return canonical-keyed sets that compare cleanly against modules.json. - _encode_go_module_path: 'X' -> '!x'. Used by write_license_inc when constructing the file:// URI's pkg/mod/ portion. - Version (the @v... suffix) is not encoded — Go doesn't apply EscapeVersion to the on-disk layout in this position. Backwards-compatible for recipes whose imported modules happen to be all-lowercase (cosign's committed sidecars are unchanged on a real regen). For recipes whose imports include uppercase paths (k3s, likely others), regeneration is needed to fill in the previously- dropped entries; do_populate_lic will pass cleanly thereafter. Signed-off-by: Bruce Ashfield

oe-go-mod-fetcher + go-mod-discovery: add --build-target license-scan filter

2026-05-29T13:37:32+00:00

The license-scan filter previously had two tiers — walk GOMODCACHE for unpacked module dirs, falling back to `go list -m all` (the MVS set). For recipes whose do_compile builds multiple binaries spanning a wider import graph than the discovery step's single GO_MOD_DISCOVERY_BUILD_TARGET (incus: 8 cmd/* binaries vs the single ./cmd/incus-migrate used by discovery), the GOMODCACHE walk under-includes and the MVS fallback over-includes — producing go-mod-licenses.inc entries that point to modules bitbake never unpacks, which then trip do_populate_lic's "invalid file" QA gate. Add a tier-0 filter: when a recipe declares its full set of build targets via the new GO_MOD_DISCOVERY_LICENSE_TARGETS variable, the generator runs `go list -deps` on those targets and uses the union of their imported modules as the filter set. This is the most accurate signal for "what bitbake will unpack at build time" — equivalent to what go build would resolve before bitbake's do_unpack runs. Mechanism: - oe-go-mod-fetcher.py: new --build-target flag (repeatable) and _get_imported_modules() helper. Repeats over `action='append'` so the per-target paths survive shell word-splitting in the bbclass invocation without quoting tricks. - go-mod-discovery.bbclass: when GO_MOD_DISCOVERY_LICENSE_TARGETS is set, loop over its values and emit --build-target for each. The list is space-separated in the recipe; the bbclass loop keeps each target as its own shell word. Tier order is now: 1. --build-target (this commit): `go list -deps` on declared targets 2. GOMODCACHE walk (existing): modules unpacked by the discovery build 3. `go list -m all` MVS-selected set (existing): full module graph 4. No filter (existing): all go.sum entries Recipes without GO_MOD_DISCOVERY_LICENSE_TARGETS see no behavior change — verified against cosign via a real bitbake -c discover_and_generate run (zero diff vs HEAD on its sidecars). Incus's go-mod-licenses.inc drops from 418 (MVS fallback) to 176 lines once the new tier is wired up, matching the build's actual ~170 unpacked modules. Signed-off-by: Bruce Ashfield

oe-go-mod-fetcher: filter license scan to actually-unpacked modules

2026-05-28T02:23:28+00:00

The --scan-licenses output (go-mod-licenses.inc) was iterating every module in modules.json, which mirrors go.sum and includes every version Go fetched for hash verification — including unselected indirect-dep versions and test-/tool-only deps. bitbake's do_populate_lic validates each LIC_FILES_CHKSUM entry against an unpacked module dir at pkg/mod/@/. Modules that go.sum lists but go build does not import never get unpacked there, so the QA check fails with "invalid file" errors — hundreds at a time on a busy project like cosign. Add a tiered filter to scan_module_licenses(): 1. Walk GOMODCACHE for *@v* directories. This is exactly the set the discovery step's `go build` populated, and matches 1:1 what bitbake will unpack at build time from the SRC_URI entries we generate. 2. Fall back to `go list -m all` (the MVS-selected set). Smaller than go.sum but larger than the unpacked set; useful when GOMODCACHE isn't yet populated. Set GOPROXY explicitly so the helper can download a newer toolchain if go.mod's directive requires one (do_generate_modules's env has GOPROXY=off). 3. No filter — original behavior, kept for safety. For cosign 3.0.6-tip, this trims go-mod-licenses.inc from 1320 to 265 lines (1068 unselected entries pruned) and resolves all do_populate_lic QA errors. Signed-off-by: Bruce Ashfield

oe-go-mod-fetcher: add license scanning for Go module dependencies

2026-04-28T18:49:19+00:00

Add --scan-licenses to oe-go-mod-fetcher.py which scans Go module zips for license files and generates go-mod-licenses.inc with LICENSE and LIC_FILES_CHKSUM entries matching OE-core's go-mod-update-modules format. License detection uses OE-core's glob patterns and MD5 + crunched MD5 matching against known SPDX licenses. The hash database resolves from: 1. --common-license-dir (explicit path) 2. Auto-detected poky tree common-licenses 3. Bundled scripts/data/license-hashes.csv (offline fallback) New files: - scripts/generate-license-hashes.py: regenerate bundled CSV - scripts/data/license-hashes.csv: pre-computed hash DB (704 entries) bbclass changes: - go-mod-discovery: pass --scan-licenses during do_generate_modules - GO_MOD_DISCOVERY_SKIP_LICENSES variable to bypass scanning - do_update_license_hashes task to refresh bundled CSV Co-Authored-By: Claude Opus 4.6 (1M context)

oe-go-mod-fetcher: improve error messages with recipe-ready fix snippets

2026-04-28T13:45:07+00:00

When the fetcher encounters unverifiable commits or modules with no repository metadata, the error messages now show two equivalent options: Option 1: For bitbake users (most common) — add GO_MOD_VCS_EXCLUDE and a gomod:// SRC_URI entry to the recipe, fetching the module via the Go module proxy instead of git. Option 2: For direct oe-go-mod-fetcher.py invocation — pass --inject-commit / --set-repo on the command line with a complete example showing all required arguments. Both options produce the same result. The previous messages only suggested --inject-commit and --set-repo without context on where or how to use them. Signed-off-by: Bruce Ashfield

scripts: add QEMU VM launcher and k3s multi-node helper

2026-04-07T16:05:50+00:00

run-qemu-vm.sh: - Architecture-aware QEMU launcher (x86-64, arm64) - Finds native QEMU binary and libraries from build sysroots - Supports KVM auto-detection, socket networking, custom rootfs - Reusable by both humans and the pytest test suite run-k3s-multinode.sh: - Launches server or agent VMs for k3s multi-node testing - Passes k3s.role, k3s.server, k3s.token, k3s.node-ip, k3s.node-name via kernel cmdline for automatic guest-side configuration - Creates rootfs copy for agent VM (can't share ext4 read-write) - Prints usage instructions when run without arguments Usage: Terminal 1: ./scripts/run-k3s-multinode.sh server Server VM: k3s-get-token Terminal 2: ./scripts/run-k3s-multinode.sh agent --token Server VM: kubectl get nodes Signed-off-by: Bruce Ashfield

go-mod-discovery/oe-go-mod-fetcher: add GO_MOD_VCS_EXCLUDE for deleted repos

2026-03-19T15:56:06+00:00

Some upstream Go module repositories get deleted from GitHub (e.g., github.com/vtolstov/go-ioctl). While the Go module proxy still serves cached archives, VCS mode cannot git clone a deleted repo. This causes both do_fetch failures and generator verification failures. Add GO_MOD_VCS_EXCLUDE recipe variable (space-separated module path prefixes) and corresponding --exclude-module CLI flag. Excluded modules are filtered out before verification and SRC_URI generation. Recipes must provide a gomod:// SRC_URI entry for excluded modules as fallback. Usage in recipe: SRC_URI += "gomod://example.com/deleted-repo;version=v1.0.0;sha256sum=..." GO_MOD_VCS_EXCLUDE = "example.com/deleted-repo" Signed-off-by: Bruce Ashfield

oe-go-mod-fetcher: fix pseudo-version tag assignment in SRC_URI

2026-03-18T17:33:43+00:00

Go's module cache .info files store Origin.Ref as the "nearest tag" used to derive pseudo-versions (e.g., v0.0.0-20190215142949-d0b11bdaac8a stores Ref: "refs/tags/v0.3.0"). This ref is NOT a tag pointing to the pseudo-version's actual commit - it's just metadata about the base version. The generator was blindly using this ref as tag=v0.3.0;shallow=1 in SRC_URI entries. BitBake resolves the tag to one specific commit, finds it doesn't match the rev= parameter, and fails with: FetchError("The revision the git tag 'v0.3.0' resolved to didn't match the SRCREV in use...") This caused multiple SRC_URI entries for the same repo (e.g., go.googlesource.com/sys) to all claim tag=v0.3.0 but with different rev= values - only one could possibly be correct. Fix by detecting pseudo-versions via parse_pseudo_version_tag() and clearing the ref_hint so these entries use nobranch=1 without a tag, falling back to full clone by commit hash. Tagged versions (real releases) correctly retain their tag= parameter for shallow clones. The docker-compose .inc files are regenerated with the fix applied. Signed-off-by: Bruce Ashfield

go-mod-fetcher: fix shallow clone handling, duplicates, and discovery workflow

2026-01-04T16:40:15+00:00

oe-go-mod-fetcher.py: - Remove BB_GIT_SHALLOW_EXTRA_REFS generation - refs must be present in ALL repositories which isn't the case for module dependencies. Instead, use tag= parameter in individual SRC_URI entries. - Add tag= to SRC_URI when ref is a tag, allowing BitBake's shallow clone to include the necessary tag (with BB_GIT_SHALLOW=1) - Remove premature _ref_points_to_commit() check that was clearing ref_hints before repos were fetched, preventing tag= from being added - Fix pseudo-version verification: only use shallow fetch for actual tags (refs/tags/...), not branch refs. Pseudo-versions with branch refs (refs/heads/...) now correctly use unshallow path to reach historical commits that aren't fetchable with depth=1 oe-go-mod-fetcher-hybrid.py: - Fix duplicate SRC_URI entries when multiple modules share the same git repo/commit (e.g., errdefs and errdefs/pkg). Track added vcs_hashes to skip duplicates. - Add --discovery-cache option to calculate module sizes from discovery cache .zip files, enabling size recommendations during discover_and_generate go-mod-discovery.bbclass: - Add automatic hybrid mode recommendations after generate_modules, showing module sizes and suggested --git prefixes for conversion - Add GO_MOD_DISCOVERY_SKIP_VERIFY variable to skip commit verification on retries (useful after fixing verification issues) - Pass --discovery-cache to hybrid script for accurate size calculations Signed-off-by: Bruce Ashfield