| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The license-scan filter previously had two tiers — walk GOMODCACHE for
unpacked module dirs, falling back to `go list -m all` (the MVS set).
For recipes whose do_compile builds multiple binaries spanning a wider
import graph than the discovery step's single GO_MOD_DISCOVERY_BUILD_TARGET
(incus: 8 cmd/* binaries vs the single ./cmd/incus-migrate used by
discovery), the GOMODCACHE walk under-includes and the MVS fallback
over-includes — producing go-mod-licenses.inc entries that point to
modules bitbake never unpacks, which then trip do_populate_lic's
"invalid file" QA gate.
Add a tier-0 filter: when a recipe declares its full set of build
targets via the new GO_MOD_DISCOVERY_LICENSE_TARGETS variable, the
generator runs `go list -deps` on those targets and uses the union
of their imported modules as the filter set. This is the most accurate
signal for "what bitbake will unpack at build time" — equivalent to
what go build would resolve before bitbake's do_unpack runs.
Mechanism:
- oe-go-mod-fetcher.py: new --build-target flag (repeatable) and
_get_imported_modules() helper. Repeats over `action='append'`
so the per-target paths survive shell word-splitting in the bbclass
invocation without quoting tricks.
- go-mod-discovery.bbclass: when GO_MOD_DISCOVERY_LICENSE_TARGETS is
set, loop over its values and emit --build-target for each. The
list is space-separated in the recipe; the bbclass loop keeps
each target as its own shell word.
Tier order is now:
1. --build-target (this commit): `go list -deps` on declared targets
2. GOMODCACHE walk (existing): modules unpacked by the discovery build
3. `go list -m all` MVS-selected set (existing): full module graph
4. No filter (existing): all go.sum entries
Recipes without GO_MOD_DISCOVERY_LICENSE_TARGETS see no behavior change —
verified against cosign via a real bitbake -c discover_and_generate run
(zero diff vs HEAD on its sidecars). Incus's go-mod-licenses.inc drops
from 418 (MVS fallback) to 176 lines once the new tier is wired up,
matching the build's actual ~170 unpacked modules.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add --scan-licenses to oe-go-mod-fetcher.py which scans Go module zips
for license files and generates go-mod-licenses.inc with LICENSE and
LIC_FILES_CHKSUM entries matching OE-core's go-mod-update-modules format.
License detection uses OE-core's glob patterns and MD5 + crunched MD5
matching against known SPDX licenses. The hash database resolves from:
1. --common-license-dir (explicit path)
2. Auto-detected poky tree common-licenses
3. Bundled scripts/data/license-hashes.csv (offline fallback)
New files:
- scripts/generate-license-hashes.py: regenerate bundled CSV
- scripts/data/license-hashes.csv: pre-computed hash DB (704 entries)
bbclass changes:
- go-mod-discovery: pass --scan-licenses during do_generate_modules
- GO_MOD_DISCOVERY_SKIP_LICENSES variable to bypass scanning
- do_update_license_hashes task to refresh bundled CSV
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some upstream Go module repositories get deleted from GitHub (e.g.,
github.com/vtolstov/go-ioctl). While the Go module proxy still serves
cached archives, VCS mode cannot git clone a deleted repo. This causes
both do_fetch failures and generator verification failures.
Add GO_MOD_VCS_EXCLUDE recipe variable (space-separated module path
prefixes) and corresponding --exclude-module CLI flag. Excluded modules
are filtered out before verification and SRC_URI generation. Recipes
must provide a gomod:// SRC_URI entry for excluded modules as fallback.
Usage in recipe:
SRC_URI += "gomod://example.com/deleted-repo;version=v1.0.0;sha256sum=..."
GO_MOD_VCS_EXCLUDE = "example.com/deleted-repo"
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
oe-go-mod-fetcher.py:
- Remove BB_GIT_SHALLOW_EXTRA_REFS generation - refs must be present in
ALL repositories which isn't the case for module dependencies. Instead,
use tag= parameter in individual SRC_URI entries.
- Add tag=<tagname> to SRC_URI when ref is a tag, allowing BitBake's
shallow clone to include the necessary tag (with BB_GIT_SHALLOW=1)
- Remove premature _ref_points_to_commit() check that was clearing
ref_hints before repos were fetched, preventing tag= from being added
- Fix pseudo-version verification: only use shallow fetch for actual
tags (refs/tags/...), not branch refs. Pseudo-versions with branch
refs (refs/heads/...) now correctly use unshallow path to reach
historical commits that aren't fetchable with depth=1
oe-go-mod-fetcher-hybrid.py:
- Fix duplicate SRC_URI entries when multiple modules share the same
git repo/commit (e.g., errdefs and errdefs/pkg). Track added vcs_hashes
to skip duplicates.
- Add --discovery-cache option to calculate module sizes from discovery
cache .zip files, enabling size recommendations during discover_and_generate
go-mod-discovery.bbclass:
- Add automatic hybrid mode recommendations after generate_modules,
showing module sizes and suggested --git prefixes for conversion
- Add GO_MOD_DISCOVERY_SKIP_VERIFY variable to skip commit verification
on retries (useful after fixing verification issues)
- Pass --discovery-cache to hybrid script for accurate size calculations
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
| |
When building multiple packages (./... or multiple targets), go build
requires the output to be a directory. Create the directory and use it.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
When updating a SRCREV and then running discovery, we expect that
some patches will fail. We don't want that to block discovery as
these patches are not normally ADDING dependencies that would be
found.
So we move discovery before patch and patch failures can be dealt
with on a full build.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The discovery repository can be in an incosistent state
after a build. By restoring the go.* files, we ensure a stable
start point.
Add automatic removal of go.sum files from git-fetched
dependencies in vcs_cache during do_create_module_cache.
This prevents checksum mismatch errors caused by stale
go.sum files in dependencies having different checksums
than the git-sourced modules.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
To make it easier to run individual components of the go module
discovery and generation process, we add invidual tasks as well
as a combined task:
AVAILABLE TASKS:
bitbake <recipe> -c discover_modules
Build project and download modules from proxy.golang.org
This populates the discovery cache but does NOT extract or generate
bitbake <recipe> -c extract_modules
Extract module metadata from discovery cache to modules.json
Requires: discover_modules to have been run first
bitbake <recipe> -c generate_modules
Generate go-mod-git.inc and go-mod-cache.inc from modules.json
Requires: extract_modules to have been run first
bitbake <recipe> -c discover_and_generate
Run all three steps: discover -> extract -> generate
This is the "do everything" convenience task
bitbake <recipe> -c show_upgrade_commands
Show copy-pasteable command lines without running anything
bitbake <recipe> -c clean_discovery
Remove the persistent discovery cache
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
|
|
Add two new bbclass files that enable building Go applications using
git-based module resolution instead of network proxy fetches:
go-mod-vcs.bbclass:
- Provides do_create_module_cache task to build GOMODCACHE from git sources
- Implements pure Python h1: hash calculation with go-dirhash-native fallback
- Creates properly structured module zip files and hash files
- Handles module path transformations and case encoding
go-mod-discovery.bbclass:
- Runs module discovery using the oe-go-mod-fetcher tool
- Generates go-mod-git.inc and go-mod-cache.inc files
- Supports bootstrap mode for initial recipe conversion
Together these classes enable fully offline, reproducible Go builds
by fetching module sources via git and constructing the module cache
during the build rather than relying on network access to module proxies.
Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
|
