summaryrefslogtreecommitdiffstats
path: root/scripts
Commit message (Collapse)AuthorAgeFilesLines
* go-mod-fetcher: fix shallow clone handling, duplicates, and discovery workflowBruce Ashfield2026-01-042-22/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | oe-go-mod-fetcher.py: - Remove BB_GIT_SHALLOW_EXTRA_REFS generation - refs must be present in ALL repositories which isn't the case for module dependencies. Instead, use tag= parameter in individual SRC_URI entries. - Add tag=<tagname> to SRC_URI when ref is a tag, allowing BitBake's shallow clone to include the necessary tag (with BB_GIT_SHALLOW=1) - Remove premature _ref_points_to_commit() check that was clearing ref_hints before repos were fetched, preventing tag= from being added - Fix pseudo-version verification: only use shallow fetch for actual tags (refs/tags/...), not branch refs. Pseudo-versions with branch refs (refs/heads/...) now correctly use unshallow path to reach historical commits that aren't fetchable with depth=1 oe-go-mod-fetcher-hybrid.py: - Fix duplicate SRC_URI entries when multiple modules share the same git repo/commit (e.g., errdefs and errdefs/pkg). Track added vcs_hashes to skip duplicates. - Add --discovery-cache option to calculate module sizes from discovery cache .zip files, enabling size recommendations during discover_and_generate go-mod-discovery.bbclass: - Add automatic hybrid mode recommendations after generate_modules, showing module sizes and suggested --git prefixes for conversion - Add GO_MOD_DISCOVERY_SKIP_VERIFY variable to skip commit verification on retries (useful after fixing verification issues) - Pass --discovery-cache to hybrid script for accurate size calculations Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* oe-go-mod-fetcher-hybrid: improve duplicate detectionBruce Ashfield2026-01-031-1/+5
| | | | | | | | The main go-mod discovery fetcher had stronger duplicate detection than the hybrid mode converter. We synchronize the two to avoid getting dups in our generate SRC_URIs. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* go-mod: introduce hybrid modeBruce Ashfield2025-12-083-0/+855
| | | | | | | | | | | | | | | | | | | | | | Introduce the ability to have hybrid gomod:// and git:// repositories. This allows SRCREV bumping when fixing bugs, and using the git archiver for some, all or none of the modules in a go mod project. Example: k3s Hybrid Conversion 1. Ensure VCS mode works first bitbake k3s 2. Get recommendations bitbake k3s -c go_mod_recommend 3. Convert with recommended prefixes (keep containerd, k8s.io as git://) python3 ./meta-virtualization/scripts/oe-go-mod-fetcher-hybrid.py \ --recipedir ./meta-virtualization/recipes-containers/k3s/ \ --git "github.com/containerd,k8s.io,sigs.k8s.io,github.com/rancher" Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* docs: add QUICKSTART for go-mod-vcsBruce Ashfield2025-12-081-0/+408
| | | | Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* oe-go-mod-fetcher: use dereferenced tagsBruce Ashfield2025-12-081-1/+17
| | | | | | | | | | | Use dereferenced tag (^{}) to get the actual commit hash For annotated tags, ref_hint returns the tag object hash, not the commit Example: refs/tags/v1.0.1 -> c49ff274 (tag object) refs/tags/v1.0.1^{} -> 37c8de36 (actual commit) current_tag_commit = git_ls_remote(vcs_url, f"{ref_hint}^{{}}") Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* scripts: add oe-go-mod-fetcher for Go module VCS resolutionBruce Ashfield2025-12-084-0/+5094
| | | | | | | | | | | | | | | | | | | | | | Add the oe-go-mod-fetcher.py tool and supporting files for resolving Go module dependencies via git repositories instead of module proxies. oe-go-mod-fetcher.py: - Parses go.mod and go.sum to identify required modules - Resolves module paths to git repositories (handles vanity URLs) - Maps module versions to git commits - Generates SRC_URI entries for bitbake fetcher - Creates go-mod-git.inc and go-mod-cache.inc files - Supports monorepo detection and nested module handling - Caches resolution results for performance extract-discovered-modules.py: - Helper script to extract module information from discovery cache - Used by go-mod-discovery.bbclass during build Also adds .gitignore to exclude runtime caches from version control. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* scripts: wic plugin bootimg-biosxen drop helper to reuse bootimg_pcbiosAnibal Limon2025-07-311-59/+12
| | | | | | | | | | | | | | | | | | - With wic plugins rename on OE-Core now can be imported. See OE-Core revs, afa1b5c9f6ed17c021e37a54d0d6abee50a60bf9 2de444fc3ef450f45f8f93403544e8f7461657b0 16c8251e5272510ad96613b8c6623550c5a72a34 - Drop the custom helper to find BootimgPcbiosPlugin plus adapt the code removing all custom calls and references. - Finally rename bootimg-biosxen to allow be imported. Tested with xen-image-minimal and testimage. Signed-off-by: Anibal Limon <anibal@limonsoftware.com> Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* wic: fix bios plugin name referencesBruce Ashfield2025-06-251-2/+2
| | | | | | | | | | When testing the OE core unpackdir fixups, it was found that the x86-64 xen images wouldn't assemble due to wic plugin issues. These tweaks get the images building, but runtime testing is still pending. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* scripts: adjust relocation.inc to not copy large directoriesBruce Ashfield2025-04-181-4/+29
| | | | | | | | | | | | | Some of the git repositories for depedencies can be quite large. The large files never seem to be related to build (as they would be too large to be pure go modules). To make things faster, update our rsync copy to exclude any directories bigger than 500M, we can adjust the limit or make it something a recipe can specify in the future, but for now this helps long build times. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* scripts: add space around assignmentsBruce Ashfield2025-04-161-1/+1
| | | | | | | We need spaces around our SRCREV assignements to avoid QA errors. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* scripts/oe-go-mod-autogen: verbosely log exceptions and continue processingBruce Ashfield2025-03-241-7/+15
| | | | Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* scripts/oe-go-mod-autogen: allow repository mappingBruce Ashfield2024-07-151-3/+62
| | | | | | | | | I cases of failure to lookup a go.mod -> src repository allow a mapping to be specified as part of the tool. This allows us to avoid manually modifying .cache files and keep a generation running. Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* oe-go-mod-autogen: fix src_uri generationBruce Ashfield2024-07-111-5/+15
| | | | | | | | | The template code for writing SRC_URI entries contained commented lines with "%s", but those are picked up as replacement markers. As such, we failed to write a SRC_URI at all Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* scripts: adjust to syhead go source locationBruce Ashfield2024-06-041-1/+4
| | | | Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* oe-go-mod-autogen: clarify need to update main SRCREVBruce Ashfield2023-11-061-5/+11
| | | | Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* oe-go-mod-autogen: add extra information to duplicate module pathsBruce Ashfield2023-11-062-1/+34
| | | | Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* oe-go-mod-autogen: add README and extended --help informationBruce Ashfield2023-09-152-3/+178
| | | | Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* oe-go-mod-autogen: add -v alias for -dBruce Ashfield2023-09-151-0/+7
| | | | Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* oe-go-mod: enhance help messageBruce Ashfield2023-09-151-2/+18
| | | | Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* oe-go-mod-autogen: whitespace cleanup, license and copyrightBruce Ashfield2023-09-151-17/+37
| | | | Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* oe-go-mod-autogen.py: add script to help adding/upgrading go mod recipesChen Qi2023-09-151-0/+663
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | oe-go-mod-autogen.py is a helper script for go mod recipes. It follows Bruce's initiative about how to deal with go mod recipes in OE. Example: cmd: <path_to>/meta-virtualization/scripts/oe-go-mod-autogen.py \ --repo https://github.com/docker/compose --rev v2.20.3 output: src_uri.inc, relocation.inc, modules.txt Copy these three generated files to replace the original ones, then we only need update PV and SRCREV, and docker-compose is upgraded. Below are some technical details. * get module's repo from module name This script checks the following two URLs to determine the module's repo. 1. https://<module_name_tweaked>?=go-get=1 2. https://pkg.go.dev/<module_name_tweaked> The module_name_tweaked is derived from module_name, with the last components removed one by one. Let me use two examples to explain this. For module_name sigs.k8s.io/json, the sigs.k8s.io/json is first used as module_name_tweaked for searching. And we can correctly get the repo URL, so the search stops. For module_name github.com/k3s-io/etcd/api/v3, the following ones are used as module_name_tweaked: github.com/k3s-io/etcd/api/v3 github.com/k3s-io/etcd/api github.com/k3s-io/etcd And when searching 'github.com/k3s-io/etcd', we get the repo URL, so the search stops. * determine the srcdir:destdir mapping in 'vendor' creation To correctly form the 'vendor' directory, the mapping is critical. This script makes use of tag matching and path matching to determine the subpath in the repo for the module. * avoid subpath being overriden by parent path We need to avoid subpath being overriden by parent path. This is needed for both SRC_URI ordering in src_uri.inc and the sites mapping ordering in relocation.inc. This script simply uses the length as the ordering key, simply for the reason that if a path is a subpath of another path, it must be longer. * the .git suffix is removed to sync with each other Unlike normal recipes, go mod recipe usually have many SRC_URIs. This script remove the '.git' suffix from repo URL so that the repo URLs are in sync with each. * basic directory hierarchy and caching mechanism <cwd>/repos: hold the repos downloaded and checked <cwd>/wget-contents: hold the contents to determine the module's repo <cwd>/wget-contents/<module_name>.repo_url.cache: the repo value cache This is to avoid unnecessary URL fetching and repo cloning. * the ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE switch in script The script must get the correct repo_url, fullsrc_rev and subpath for each required module in go.mod to correctly generate the src_uri.inc and relocation.inc files. If this process fails for any required module, this script stop immediately, as I deliberately set ERROR_OUT_ON_FETCH_AND_CHECKOUT_FAILURE to True in this script. The purpose is to encourage people to report problems to meta-virt so that we can improve this script according to these feedbacks. But this variable can set to False, then the script only records the failed modules in self.modules_unhandled with reasons added, people can modify the generated src_uri.inc and relocation.inc to manually handle these unhandled modules if they are urgent to add/upgrade some go mod recipes. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
* wic: add support for bootable pcbios partition with Xen hypervisorChristopher Clark2020-02-271-0/+212
New bootimg-biosxen wic plugin to populate a boot partition for launching Xen and dom0. Includes example kickstart wks files to generate disk images to boot into Xen from PC BIOS. eg: wic create directdisk-xen -e xen-image-minimal and write the resulting image file to a disk for boot. Signed-off-by: Christopher Clark <christopher.clark6@baesystems.com> Signed-off-by: Bruce Ashfield <bruce.ashfield@gmail.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2026-02-13 20:07:40 +0000