| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
resolve_pseudo_version_commit() expands a 12-character pseudo-version
short hash to its full 40-character form by cloning the upstream repo
into a cache directory and running 'git log --since=...'. The cache is
reused on subsequent invocations and refreshed via 'git fetch --all'.
The initial clone used 'git clone --bare', which does NOT install a
remote.origin.fetch refspec. As a result, the later 'git fetch --all'
exits 0 but actually fetches nothing — the cache stays frozen at the
state of the very first clone, no matter how much time passes.
For any pseudo-version whose timestamp is newer than the first time the
cache was populated, the 'git log --since=<timestamp - 1 day>' search
returns no commits, expansion silently returns None, and the 12-char
hash propagates verbatim into the generated SRC_URI .inc files. bitbake
then refuses to parse the recipe with an unhelpful "Unable to resolve
'<12 chars>' in upstream git repository in git ls-remote output for
<repo>" error, because its git fetcher requires a full 40-char SHA.
Hit on a nerdctl v2.2.1 -> v2.3.1-tip bump: the repo's self-reference
module 'github.com/containerd/nerdctl/v2' at pseudo-version
v2.0.0-20260528082953-89ecd85071c4 (timestamped 2026-05-28) failed to
expand because the cached clone of github.com/containerd/nerdctl was
created on 2026-03-19 and never advanced.
Two changes:
- clone with --mirror instead of --bare. --mirror sets up the
+refs/*:refs/* refspec, so 'git fetch --all' actually pulls new
refs on every subsequent run.
- print a non-VERBOSE warning when expansion leaves any 12-char hash
unresolved, listing the affected modules and pointing at the cache
directory. The previous failure message was gated behind
VERBOSE_MODE, so an exhausted cache produced a silent fall-through
and the next visible error was bitbake's confusing parse failure.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
bitbake's do_populate_lic walks pkg/mod/<path>@<version>/ at build time,
where <path> uses Go's filesystem-safe encoding for ASCII uppercase
letters: every 'A'-'Z' becomes '!' followed by the lowercase form, so
`github.com/HdrHistogram/...` lives at `github.com/!hdr!histogram/...`
on disk. The license-scan writer was emitting canonical-form paths
("github.com/HdrHistogram/..."), so do_populate_lic couldn't find the
LICENSE file and failed with "invalid file" QA errors for every
uppercase-bearing imported module — 6 errors in k3s's case.
The same canonical-vs-encoded gap also lived in the filter input.
_get_unpacked_modules walks GOMODCACHE and returned the encoded form,
while modules.json (the loop driver in scan_module_licenses) uses the
canonical form, so the `key in selected_set` test for uppercase
modules silently failed — those modules were dropped from the filter
output entirely. That's why cosign/docker-compose/incus's committed
go-mod-licenses.inc files don't have any uppercase entries: the
build passed do_populate_lic because the entries were absent, not
because they were correct.
Normalize to canonical form internally and encode only at the output
boundary:
- _decode_go_module_path: '!x' -> 'X' for ASCII lowercase x. Used
by _get_unpacked_modules so all three filter sources return
canonical-keyed sets that compare cleanly against modules.json.
- _encode_go_module_path: 'X' -> '!x'. Used by write_license_inc
when constructing the file:// URI's pkg/mod/<path> portion.
- Version (the @v... suffix) is not encoded — Go doesn't apply
EscapeVersion to the on-disk layout in this position.
Backwards-compatible for recipes whose imported modules happen to be
all-lowercase (cosign's committed sidecars are unchanged on a real
regen). For recipes whose imports include uppercase paths (k3s,
likely others), regeneration is needed to fill in the previously-
dropped entries; do_populate_lic will pass cleanly thereafter.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The license-scan filter previously had two tiers — walk GOMODCACHE for
unpacked module dirs, falling back to `go list -m all` (the MVS set).
For recipes whose do_compile builds multiple binaries spanning a wider
import graph than the discovery step's single GO_MOD_DISCOVERY_BUILD_TARGET
(incus: 8 cmd/* binaries vs the single ./cmd/incus-migrate used by
discovery), the GOMODCACHE walk under-includes and the MVS fallback
over-includes — producing go-mod-licenses.inc entries that point to
modules bitbake never unpacks, which then trip do_populate_lic's
"invalid file" QA gate.
Add a tier-0 filter: when a recipe declares its full set of build
targets via the new GO_MOD_DISCOVERY_LICENSE_TARGETS variable, the
generator runs `go list -deps` on those targets and uses the union
of their imported modules as the filter set. This is the most accurate
signal for "what bitbake will unpack at build time" — equivalent to
what go build would resolve before bitbake's do_unpack runs.
Mechanism:
- oe-go-mod-fetcher.py: new --build-target flag (repeatable) and
_get_imported_modules() helper. Repeats over `action='append'`
so the per-target paths survive shell word-splitting in the bbclass
invocation without quoting tricks.
- go-mod-discovery.bbclass: when GO_MOD_DISCOVERY_LICENSE_TARGETS is
set, loop over its values and emit --build-target for each. The
list is space-separated in the recipe; the bbclass loop keeps
each target as its own shell word.
Tier order is now:
1. --build-target (this commit): `go list -deps` on declared targets
2. GOMODCACHE walk (existing): modules unpacked by the discovery build
3. `go list -m all` MVS-selected set (existing): full module graph
4. No filter (existing): all go.sum entries
Recipes without GO_MOD_DISCOVERY_LICENSE_TARGETS see no behavior change —
verified against cosign via a real bitbake -c discover_and_generate run
(zero diff vs HEAD on its sidecars). Incus's go-mod-licenses.inc drops
from 418 (MVS fallback) to 176 lines once the new tier is wired up,
matching the build's actual ~170 unpacked modules.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The --scan-licenses output (go-mod-licenses.inc) was iterating every module
in modules.json, which mirrors go.sum and includes every version Go fetched
for hash verification — including unselected indirect-dep versions and
test-/tool-only deps.
bitbake's do_populate_lic validates each LIC_FILES_CHKSUM entry against an
unpacked module dir at pkg/mod/<module>@<version>/. Modules that go.sum
lists but go build does not import never get unpacked there, so the QA
check fails with "invalid file" errors — hundreds at a time on a busy
project like cosign.
Add a tiered filter to scan_module_licenses():
1. Walk GOMODCACHE for *@v* directories. This is exactly the set the
discovery step's `go build` populated, and matches 1:1 what bitbake
will unpack at build time from the SRC_URI entries we generate.
2. Fall back to `go list -m all` (the MVS-selected set). Smaller than
go.sum but larger than the unpacked set; useful when GOMODCACHE
isn't yet populated. Set GOPROXY explicitly so the helper can
download a newer toolchain if go.mod's directive requires one
(do_generate_modules's env has GOPROXY=off).
3. No filter — original behavior, kept for safety.
For cosign 3.0.6-tip, this trims go-mod-licenses.inc from 1320 to 265
lines (1068 unselected entries pruned) and resolves all do_populate_lic
QA errors.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add --scan-licenses to oe-go-mod-fetcher.py which scans Go module zips
for license files and generates go-mod-licenses.inc with LICENSE and
LIC_FILES_CHKSUM entries matching OE-core's go-mod-update-modules format.
License detection uses OE-core's glob patterns and MD5 + crunched MD5
matching against known SPDX licenses. The hash database resolves from:
1. --common-license-dir (explicit path)
2. Auto-detected poky tree common-licenses
3. Bundled scripts/data/license-hashes.csv (offline fallback)
New files:
- scripts/generate-license-hashes.py: regenerate bundled CSV
- scripts/data/license-hashes.csv: pre-computed hash DB (704 entries)
bbclass changes:
- go-mod-discovery: pass --scan-licenses during do_generate_modules
- GO_MOD_DISCOVERY_SKIP_LICENSES variable to bypass scanning
- do_update_license_hashes task to refresh bundled CSV
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When the fetcher encounters unverifiable commits or modules with no
repository metadata, the error messages now show two equivalent options:
Option 1: For bitbake users (most common) — add GO_MOD_VCS_EXCLUDE
and a gomod:// SRC_URI entry to the recipe, fetching the module via
the Go module proxy instead of git.
Option 2: For direct oe-go-mod-fetcher.py invocation — pass
--inject-commit / --set-repo on the command line with a complete
example showing all required arguments.
Both options produce the same result. The previous messages only
suggested --inject-commit and --set-repo without context on where
or how to use them.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
run-qemu-vm.sh:
- Architecture-aware QEMU launcher (x86-64, arm64)
- Finds native QEMU binary and libraries from build sysroots
- Supports KVM auto-detection, socket networking, custom rootfs
- Reusable by both humans and the pytest test suite
run-k3s-multinode.sh:
- Launches server or agent VMs for k3s multi-node testing
- Passes k3s.role, k3s.server, k3s.token, k3s.node-ip, k3s.node-name
via kernel cmdline for automatic guest-side configuration
- Creates rootfs copy for agent VM (can't share ext4 read-write)
- Prints usage instructions when run without arguments
Usage:
Terminal 1: ./scripts/run-k3s-multinode.sh server
Server VM: k3s-get-token
Terminal 2: ./scripts/run-k3s-multinode.sh agent --token <TOKEN>
Server VM: kubectl get nodes
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some upstream Go module repositories get deleted from GitHub (e.g.,
github.com/vtolstov/go-ioctl). While the Go module proxy still serves
cached archives, VCS mode cannot git clone a deleted repo. This causes
both do_fetch failures and generator verification failures.
Add GO_MOD_VCS_EXCLUDE recipe variable (space-separated module path
prefixes) and corresponding --exclude-module CLI flag. Excluded modules
are filtered out before verification and SRC_URI generation. Recipes
must provide a gomod:// SRC_URI entry for excluded modules as fallback.
Usage in recipe:
SRC_URI += "gomod://example.com/deleted-repo;version=v1.0.0;sha256sum=..."
GO_MOD_VCS_EXCLUDE = "example.com/deleted-repo"
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Go's module cache .info files store Origin.Ref as the "nearest tag"
used to derive pseudo-versions (e.g., v0.0.0-20190215142949-d0b11bdaac8a
stores Ref: "refs/tags/v0.3.0"). This ref is NOT a tag pointing to the
pseudo-version's actual commit - it's just metadata about the base version.
The generator was blindly using this ref as tag=v0.3.0;shallow=1 in
SRC_URI entries. BitBake resolves the tag to one specific commit, finds
it doesn't match the rev= parameter, and fails with:
FetchError("The revision the git tag 'v0.3.0' resolved to didn't match
the SRCREV in use...")
This caused multiple SRC_URI entries for the same repo (e.g.,
go.googlesource.com/sys) to all claim tag=v0.3.0 but with different
rev= values - only one could possibly be correct.
Fix by detecting pseudo-versions via parse_pseudo_version_tag() and
clearing the ref_hint so these entries use nobranch=1 without a tag,
falling back to full clone by commit hash. Tagged versions (real
releases) correctly retain their tag= parameter for shallow clones.
The docker-compose .inc files are regenerated with the fix applied.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
oe-go-mod-fetcher.py:
- Remove BB_GIT_SHALLOW_EXTRA_REFS generation - refs must be present in
ALL repositories which isn't the case for module dependencies. Instead,
use tag= parameter in individual SRC_URI entries.
- Add tag=<tagname> to SRC_URI when ref is a tag, allowing BitBake's
shallow clone to include the necessary tag (with BB_GIT_SHALLOW=1)
- Remove premature _ref_points_to_commit() check that was clearing
ref_hints before repos were fetched, preventing tag= from being added
- Fix pseudo-version verification: only use shallow fetch for actual
tags (refs/tags/...), not branch refs. Pseudo-versions with branch
refs (refs/heads/...) now correctly use unshallow path to reach
historical commits that aren't fetchable with depth=1
oe-go-mod-fetcher-hybrid.py:
- Fix duplicate SRC_URI entries when multiple modules share the same
git repo/commit (e.g., errdefs and errdefs/pkg). Track added vcs_hashes
to skip duplicates.
- Add --discovery-cache option to calculate module sizes from discovery
cache .zip files, enabling size recommendations during discover_and_generate
go-mod-discovery.bbclass:
- Add automatic hybrid mode recommendations after generate_modules,
showing module sizes and suggested --git prefixes for conversion
- Add GO_MOD_DISCOVERY_SKIP_VERIFY variable to skip commit verification
on retries (useful after fixing verification issues)
- Pass --discovery-cache to hybrid script for accurate size calculations
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
| |
The main go-mod discovery fetcher had stronger duplicate detection
than the hybrid mode converter. We synchronize the two to avoid
getting dups in our generate SRC_URIs.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Introduce the ability to have hybrid gomod:// and git:// repositories.
This allows SRCREV bumping when fixing bugs, and using the git archiver
for some, all or none of the modules in a go mod project.
Example: k3s Hybrid Conversion
1. Ensure VCS mode works first
bitbake k3s
2. Get recommendations
bitbake k3s -c go_mod_recommend
3. Convert with recommended prefixes (keep containerd, k8s.io as git://)
python3 ./meta-virtualization/scripts/oe-go-mod-fetcher-hybrid.py \
--recipedir ./meta-virtualization/recipes-containers/k3s/ \
--git "github.com/containerd,k8s.io,sigs.k8s.io,github.com/rancher"
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
| |
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Use dereferenced tag (^{}) to get the actual commit hash
For annotated tags, ref_hint returns the tag object hash, not the commit
Example: refs/tags/v1.0.1 -> c49ff274 (tag object)
refs/tags/v1.0.1^{} -> 37c8de36 (actual commit)
current_tag_commit = git_ls_remote(vcs_url, f"{ref_hint}^{{}}")
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add the oe-go-mod-fetcher.py tool and supporting files for resolving
Go module dependencies via git repositories instead of module proxies.
oe-go-mod-fetcher.py:
- Parses go.mod and go.sum to identify required modules
- Resolves module paths to git repositories (handles vanity URLs)
- Maps module versions to git commits
- Generates SRC_URI entries for bitbake fetcher
- Creates go-mod-git.inc and go-mod-cache.inc files
- Supports monorepo detection and nested module handling
- Caches resolution results for performance
extract-discovered-modules.py:
- Helper script to extract module information from discovery cache
- Used by go-mod-discovery.bbclass during build
Also adds .gitignore to exclude runtime caches from version control.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- With wic plugins rename on OE-Core now can be imported.
See OE-Core revs,
afa1b5c9f6ed17c021e37a54d0d6abee50a60bf9
2de444fc3ef450f45f8f93403544e8f7461657b0
16c8251e5272510ad96613b8c6623550c5a72a34
- Drop the custom helper to find BootimgPcbiosPlugin plus adapt the code
removing all custom calls and references.
- Finally rename bootimg-biosxen to allow be imported.
Tested with xen-image-minimal and testimage.
Signed-off-by: Anibal Limon <anibal@limonsoftware.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
| |
When testing the OE core unpackdir fixups, it was found that the
x86-64 xen images wouldn't assemble due to wic plugin issues.
These tweaks get the images building, but runtime testing is
still pending.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Some of the git repositories for depedencies can be quite large.
The large files never seem to be related to build (as they would
be too large to be pure go modules).
To make things faster, update our rsync copy to exclude any
directories bigger than 500M, we can adjust the limit or make
it something a recipe can specify in the future, but for now
this helps long build times.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
| |
We need spaces around our SRCREV assignements to avoid
QA errors.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
| |
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
| |
I cases of failure to lookup a go.mod -> src repository allow
a mapping to be specified as part of the tool. This allows
us to avoid manually modifying .cache files and keep a
generation running.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
| |
The template code for writing SRC_URI entries contained
commented lines with "%s", but those are picked up as
replacement markers. As such, we failed to write a SRC_URI
at all
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
| |
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
| |
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
| |
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
| |
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
| |
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
| |
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
| |
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
oe-go-mod-autogen.py is a helper script for go mod recipes. It follows
Bruce's initiative about how to deal with go mod recipes in OE.
Example:
cmd: <path_to>/meta-virtualization/scripts/oe-go-mod-autogen.py \
--repo https://github.com/docker/compose --rev v2.20.3
output: src_uri.inc, relocation.inc, modules.txt
Copy these three generated files to replace the original ones,
then we only need update PV and SRCREV, and docker-compose is upgraded.
Below are some technical details.
* get module's repo from module name
This script checks the following two URLs to determine the module's repo.
1. https://<module_name_tweaked>?=go-get=1
2. https://pkg.go.dev/<module_name_tweaked>
The module_name_tweaked is derived from module_name, with the last components
removed one by one. Let me use two examples to explain this.
For module_name sigs.k8s.io/json, the sigs.k8s.io/json is first used as
module_name_tweaked for searching. And we can correctly get the repo URL, so
the search stops.
For module_name github.com/k3s-io/etcd/api/v3, the following ones are used
as module_name_tweaked:
github.com/k3s-io/etcd/api/v3
github.com/k3s-io/etcd/api
github.com/k3s-io/etcd
And when searching 'github.com/k3s-io/etcd', we get the repo URL, so the search
stops.
* determine the srcdir:destdir mapping in 'vendor' creation
To correctly form the 'vendor' directory, the mapping is critical.
This script makes use of tag matching and path matching to determine
the subpath in the repo for the module.
* avoid subpath being overriden by parent path
We need to avoid subpath being overriden by parent path. This is needed
for both SRC_URI ordering in src_uri.inc and the sites mapping ordering
in relocation.inc. This script simply uses the length as the ordering key,
simply for the reason that if a path is a subpath of another path, it must
be longer.
* the .git suffix is removed to sync with each other
Unlike normal recipes, go mod recipe usually have many SRC_URIs. This script
remove the '.git' suffix from repo URL so that the repo URLs are in sync
with each.
* basic directory hierarchy and caching mechanism
<cwd>/repos: hold the repos downloaded and checked
<cwd>/wget-contents: hold the contents to determine the module's repo
<cwd>/wget-contents/<module_name>.repo_url.cache: the repo value cache
This is to avoid unnecessary URL fetching and repo cloning.
* the ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE switch in script
The script must get the correct repo_url, fullsrc_rev and subpath for
each required module in go.mod to correctly generate the src_uri.inc and
relocation.inc files. If this process fails for any required module, this
script stop immediately, as I deliberately set ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE
to True in this script. The purpose is to encourage people to report
problems to meta-virt so that we can improve this script according to
these feedbacks. But this variable can set to False, then the script
only records the failed modules in self.modules_unhandled with reasons
added, people can modify the generated src_uri.inc and relocation.inc
to manually handle these unhandled modules if they are urgent to
add/upgrade some go mod recipes.
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
|
|
New bootimg-biosxen wic plugin to populate a boot partition for
launching Xen and dom0.
Includes example kickstart wks files to generate disk images to boot
into Xen from PC BIOS.
eg: wic create directdisk-xen -e xen-image-minimal
and write the resulting image file to a disk for boot.
Signed-off-by: Christopher Clark <christopher.clark6@baesystems.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
