linux/poky.git/meta/lib/oe/patch.py, branch 5.0_M1

lib/oe/patch: handle creating patches for CRLF sources

2023-12-06T22:55:49+00:00

Using devtool to patch CRLF based sources creates patch files which have mixed end of lines : LF for headers and CRLF for source context and modified lines. Python open(..., newline=None) (default for newline arg)does detect end-of-line in this mixed file but only outputs LF EOL data. This result in patch files that does not apply on the original sources. Switching to open(..., newline='') allows to detect end-of-line but keep the original end-of-line intact. This generate correct patches for CRLF based sources. Fixes [YOCTO #15285] (From OE-Core rev: 58f845499c0277a2b8069eefa235430b5f5f7661) Signed-off-by: Yoann Congal Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

devtool: add support for git submodules

2023-12-01T11:48:25+00:00

Adding the support of submodules required a lot of changes on the internal data structures: * initial_rev/startcommit used as a starting point for looking at new / updated commits was replaced by a dictionary where the keys are the submodule name ("." for main repo) and the values are the initial_rev/startcommit * the extractPatches function now extracts patch for the main repo and for all submodules and stores them in a hierarchical way describing the submodule path * store initial_rev/commit also for all submodules inside the recipe bbappend file * _export_patches now returns dictionaries that contains the 'patchdir' parameter (if any). This parameter is used to add the correct 'patchdir=' parameter on the recipe Also, recipe can extract a secondary git tree inside the workdir. By default, at the end of the do_patch function, there is a hook in devtool that commits everything that was modified to have a clean repository. It uses the command: "git add .; git commit ..." The issue here is that, it adds the secondary git tree as a submodule but in a wrong way. Doing "git add " declares a submodule but do not adds a url associated to it, and all following "git submodule foreach" commands will fail. So detect that a git tree was extracted inside S and correctly add it using "git submodule add ", so that it will be considered as a regular git submodule (From OE-Core rev: 900129cbdf25297a42ab5dbd02d1adbea405c935) Signed-off-by: Julien Stephan Signed-off-by: Alexandre Belloni Signed-off-by: Richard Purdie

lib/oe/patch: ensure os.chdir restoring always happens

2023-11-22T14:08:48+00:00

If we chdir(), do the chdir back to the original directory in a finally block so they always run. (From OE-Core rev: cdc40292818683b6df1c814498c7589450a163fa) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

patch.py: use --absolute-git-dir instead of --show-toplevel to retrieve gitdir

2023-08-29T08:41:07+00:00

[YOCTO #14141] Currently the gitdir is manually constructed using `git rev-parse --show-toplevel` and appending `.git`. This is most of the time correct but not always: `.git` can be a file with the following content: gitdir: This is the case for submodules, so when using devtool modify on a recipe using submodules *and* patching files inside one of the submodules, do_patch fails with the following error: ERROR: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: 0001: *** 0002:patch_do_patch(d) 0003: File: '<..>/poky/meta/classes-global/patch.bbclass', lineno: 157, function: patch_do_patch 0153: except Exception as exc: 0154: bb.utils.remove(process_tmpdir, True) 0155: bb.fatal("Importing patch '%s' with striplevel '%s'\n%s" % (parm['patchname'], parm['striplevel'], repr(exc).replace("\\n", "\n"))) 0156: try: *** 0157: resolver.Resolve() 0158: except bb.BBHandledException as e: 0159: bb.utils.remove(process_tmpdir, True) 0160: bb.fatal("Applying patch '%s' on target directory '%s'\n%s" % (parm['patchname'], patchdir, repr(e).replace("\\n", "\n"))) 0161: File: '<..>/poky/meta/lib/oe/patch.py', lineno: 769, function: Resolve 0765: def Resolve(self): 0766: olddir = os.path.abspath(os.curdir) 0767: os.chdir(self.patchset.dir) 0768: try: *** 0769: self.patchset.Push() 0770: except Exception: 0771: import sys 0772: os.chdir(olddir) 0773: raise File: '<..>/poky/meta/lib/oe/patch.py', lineno: 274, function: Push 0270: else: 0271: next = 0 0272: 0273: bb.note("applying patch %s" % self.patches[next]) *** 0274: ret = self._applypatch(self.patches[next], force) 0275: 0276: self._current = next 0277: return ret 0278: File: '<..>/poky/meta/lib/oe/patch.py', lineno: 556, function: _applypatch 0552: if os.path.lexists(hooks_dir_backup): 0553: raise Exception("Git hooks backup directory already exists: %s" % hooks_dir_backup) 0554: if os.path.lexists(hooks_dir): 0555: shutil.move(hooks_dir, hooks_dir_backup) *** 0556: os.mkdir(hooks_dir) 0557: commithook = os.path.join(hooks_dir, 'commit-msg') 0558: applyhook = os.path.join(hooks_dir, 'applypatch-msg') 0559: with open(commithook, 'w') as f: 0560: # NOTE: the formatting here is significant; if you change it you'll also need to Exception: NotADirectoryError: [Errno 20] Not a directory: '<..>/build/tmp/work/core2-64-poky-linux/vulkan-samples/git/devtooltmp-n87_zx1i/workdir/git/third_party/spdlog/.git/hooks' Using `git rev-parse --absolute-git-dir` instead of `git rev-parse --show-toplevel` ensure we get the correct gitdir (From OE-Core rev: f74879dd95b19504ce8a8554636d2310d0336806) Signed-off-by: Julien Stephan Signed-off-by: Luca Ceresoli Signed-off-by: Richard Purdie

patch.py: Use shlex instead of deprecated pipe

2023-04-14T15:44:24+00:00

The pipe library is deprecated in Python 3.11 and will be removed in Python 3.13. pipe.quote is just an import of shlex.quote anyway. Clean up imports while we're at it. (From OE-Core rev: 5f33c7b99a991c380d1813da8248ba5470ca4d4e) Signed-off-by: Ola x Nilsson Signed-off-by: Luca Ceresoli Signed-off-by: Richard Purdie

patch: support of git patches when the source uri contained subpath parameter

2023-04-01T10:36:26+00:00

This is for a specific case where: - A recipe use a subpath on a git repo (e.g. git://repo.git/projects;subpath=subproject) - The recipe contains a patch to apply - a devtool modify is used on this recipe With these conditions, the patch cannot be applied at all. GitApplyTree class is used for handling patch under devtool, but when subpath is present in SRC_URI, the resulting git tree is dirty (every files and directories which was not in subpath are suppressed) and so "git am" refuse to apply patches. That would not be an issue since the GitApplyTree have a fallback to PatchTree in case of error, but during this error management, there is a "git reset --hard HEAD" call which suppress the subpath operation and finally prevents the patch to be applied even with PatchTree. When devtool is not involved, only PatchTree class is used and the above problem is irrelevant. To support git patching during devtool, the presence of subpath and the dirtyness of the repo are checked. If both conditions are met, we directly call PatchTree like it was already done in case of error during git apply. (From OE-Core rev: d86cac2759cf7e91f4ff240833385e28e729ab79) Signed-off-by: Frederic Martinsons Signed-off-by: Richard Purdie

lib: Add copyright statements to files without one

2022-08-12T11:00:43+00:00

Where there isn't a copyright statement, add one to make it explicit. Also add license identifiers as MIT if there isn't one. (From OE-Core rev: bb731d1f3d2a1d50ec0aed864dbca54cf795b040) Signed-off-by: Richard Purdie

patch: handle if S points to a subdirectory of a git repo

2022-07-14T22:22:09+00:00

If PATCHTOOL = "git", SRC_URI fetches from a git repo and S points to a subdirectory of the checked out sources, then we were erroneously initialising the subdirectory as its own git repo. Check if the returned top-level repo directory is a subdirectory of WORKDIR and do not run initialise the source directory if that is the case. (This was a regression introduced with OE-Core revision 6184b56a7a0fc6f5d19fdfb81e7453667f7da940, however we didn't have a test that verified the behaviour.) (From OE-Core rev: 9cca53a2bcbf6809615ce5626c86c6ee481a7a76) Signed-off-by: Paul Eggleton Signed-off-by: Richard Purdie

patch.py: make sure that patches/series file exists before quilt pop

2022-05-31T14:24:18+00:00

* Since quilt upgrade to 0.67 some recipes sometimes fail in do_patch with errors like: ERROR: Applying patch 'GPLv2.patch' on target directory '/OE/build/oe-core/tmp-glibc/work/qemux86_64-oe-linux/keymaps/1.0-r31' CmdError('quilt --quiltrc /OE/build/oe-core/tmp-glibc/work/qemux86_64-oe-linux/keymaps/1.0-r31/recipe-sysroot-native/etc/quiltrc push', 0, 'stdout: stderr: File series fully applied, ends at patch GPLv2.patch ') * It affects only recipes with S = "${WORKDIR}", which wipe only ${S}/patches, because in other cases whole ${S} is wiped when do_unpack is re-executed. * It was originally added in: https://git.openembedded.org/openembedded-core/commit/?id=5fe5e6a15f26f23f0c5b863fafad7a0d382a55e2 since then it was extended to wipe whole ${S} when ${S} != ${WORKDIR} in: https://git.openembedded.org/openembedded-core/commit/?id=5fe5e6a15f26f23f0c5b863fafad7a0d382a55e2 https://git.openembedded.org/openembedded-core/commit/?id=eccae514b71394ffaed8fc45dea7942152a334a1 this is now causing issues to quilt-0.67 because it checks that ${S}/patches/series exists during 'quilt pop -a -f' which we call from QuiltTree.Clean to undo patches possibly already applied in ${S} in previous do_patch execution. * There are couple recipes affected by this e.g. keymaps (.patch already removed in oe-core), makedevs (.patch removal sent to ML yesterday https://lists.openembedded.org/g/openembedded-core/message/166172), devmem2 (https://lists.openembedded.org/g/openembedded-devel/message/97270), but there are other recipes with S = "${WORKDIR}" where you can trigger this e.g. by having a .patch file in DISTRO layer .bbappend (e.g. tzdata with webOS https://github.com/webosose/meta-webosose/blob/06e5298d9f5c47679b679081d9930f8d1c776142/meta-webos/recipes-extended/tzdata/tzdata.bbappend#L10) This do_patch issue is caused by: https://git.savannah.nongnu.org/cgit/quilt.git/commit/?id=8b39a960afcf45cd4f5804ae62b6b0656bdb191d introduced in kirkstone with: https://git.openembedded.org/openembedded-core/commit/?h=kirkstone&id=fa71afcee9ab42198c619333b77a15bd2ae02b20 The shortest sequence to reproduce this is just bitbake keymaps -c patch bitbake keymaps -c unpack -f bitbake keymaps -c patch with https://git.openembedded.org/openembedded-core/commit/?id=17d981005a0c0c97702ad88602b7181b69bcc9eb reverted. And the change in quilt behavior is causing QuiltTree.Clean (quilt pop -a -f) in: https://git.openembedded.org/openembedded-core/tree/meta/lib/oe/patch.py?id=17d981005a0c0c97702ad88602b7181b69bcc9eb#n601 to silently fail with "No series file found" before undoing the patches in ${S} and then quilt push failing, because all the patches are _still_ applied in ${S}. Removing ".pc" doesn't help, because we really need quilt's help to undo the patches (in this case to delete COPYING file from WORKDIR before applying the .patch which tries to add it again), because do_unpack cannot just wipe S and start over (because S == WORKDIR) - nor selectively removing the files listed in SRC_URI, because COPYING file isn't listed there. Using skip_series_check in 'quilt pop' (partially reverting the change from upstream) does fix this as well and it's simple one line patch (just adding skip_series_check=1 in pop.in), but might be difficult to upstream, because it's this strange OE specific behavior that we remove 'patches' directory and then still need quilt pop to work. (From OE-Core rev: c9d36882044b1c633d8611a77df54cd68c9bee25) Signed-off-by: Martin Jansa Signed-off-by: Luca Ceresoli Signed-off-by: Richard Purdie

patch.py: Prevent git repo reinitialization

2022-02-25T12:41:23+00:00

There were few bugs in the _isInitialized() function which might trigger git repo to be reinitialized and patches failing to apply. (From OE-Core rev: 80500ecda4c1bc8812e6e078b6b0db5ec46624de) Signed-off-by: Pavel Zhukov Signed-off-by: Richard Purdie