summaryrefslogtreecommitdiffstats
path: root/meta/lib/oe/path.py
Commit message (Collapse)AuthorAgeFilesLines
* lib/oe/path: Add canonicalize()Peter Kjellerstedt2021-02-031-0/+21
| | | | | | | | | | | | | | | oe.path.canonicalize() is used to canonicalize paths (i.e., remove symbolic links and "..", and make them absolute). It takes a string with paths separated by commas, and returns the canonicalized path in the same format. (From OE-Core rev: 1b557583deb53462e1ed0f3afbe157a795e6c51c) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 282b19c0e27488ec119f00fb2542ffdc1af54e2a) Signed-off-by: Steve Sakoman <steve@sakoman.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/path: try hardlinking instead of guessing when it might failHenning Schild2020-02-041-6/+18
| | | | | | | | | | | | | | | | The comparison of the stat st_dev is not enough to judge whether hardlinking will work. One example would be where you try and hardlink across two bind-mounts of a directory. The st_dev will be the same and the operation will still fail. Instead of implementing a check to try and figure out hardlink support just try hardlinking and fall back to a copy when running into an exception. (From OE-Core rev: f5571bda8327f927feb23b167ab4594b7d0c95bc) Signed-off-by: Henning Schild <henning.schild@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/lib+scripts: Convert to SPDX license headersRichard Purdie2019-05-091-0/+4
| | | | | | | | | | | | | | | | | | | | | | | This adds SPDX license headers in place of the wide assortment of things currently in our script headers. We default to GPL-2.0-only except for the oeqa code where it was clearly submitted and marked as MIT on the most part or some scripts which had the "or later" GPL versioning. The patch also drops other obsolete bits of file headers where they were encoountered such as editor modelines, obsolete maintainer information or the phrase "All rights reserved" which is now obsolete and not required in copyright headers (in this case its actually confusing for licensing as all rights were not reserved). More work is needed for OE-Core but this takes care of the bulk of the scripts and meta/lib directories. The top level LICENSE files are tweaked to match the new structure and the SPDX naming. (From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe.path: Add copyhardlink() helper functionPaul Barker2019-05-071-1/+12
| | | | | | | | | | | | | | This function creates hard links if possible, falling back to copying the file if the destination is on a different volume to the source. The docstring for copyhardlinktree() is also updated to make the difference between the two functions a little clearer. (From OE-Core rev: 5437efa16f9bec914e417c6c939a39c247084f52) Signed-off-by: Paul Barker <paul@betafive.co.uk> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/lib.oe.path: Ensure file sparseness is preservedRichard Purdie2018-08-151-2/+2
| | | | | | | | | Files when restored from sstate were missing their sparseness. Fix up various functions to preserve this and make things more deterministic. (From OE-Core rev: 055402e5504f041c346571e243c7cf0894955cad) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe.path: add which_wild functionChristopher Larson2018-07-041-0/+34
| | | | | | | | | | | | | | This is a function much like shutil.which or bb.utils.which, retaining shutil.which-like function semantics, bb.utils.which's support for returning available candidates for signatures, and most importantly, supports wildcards, returning only the first occurrance of each found pathname in the search path. (From OE-Core rev: ca276fe139129eec383d77768ba91b808c462b04) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/path: implement is_path_parent()Paul Eggleton2018-03-011-0/+22
| | | | | | | | | | | | | | | | | In a few places we have checks to see path B is the parent of path A, by adding / to the end of the path B and then seeing if path A starts with the suffixed path B. Unfortunately there are two potential flaws: (1) path A needs to be suffixed with / as well or the directory itself won't match (semantics perhaps, but in a lot of scenarios returning True is correct); (2) you need to run os.path.abspath() on both paths first or you will wrongly return False for some relative paths where you should return True. Let's solve this once and for all by writing a function that takes care of these and put it in oe.path. (From OE-Core rev: dd3d4b0367272a5826a9a51afa26f426dd003e5d) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe/path.py: copyhardlinktree: don't overwrite existing symlinksIoan-Adrian Ratiu2017-08-231-1/+1
| | | | | | | | | | | | | | | | | | | Before GNU tar 1.24, only the archive creation command had the '-h' argument to preserve and follow symlinks. After >= 1.24 via commit 14efeb9f956e38d7be (tar: --dereference consistency) the capability to preserve symlinks was also added to the archive extraction command. -h is default at archive creation but is not default at extraction, meaning that it will replace symlinks with directories even if the original filesystem directory tree and archive contains them. Add -h to the copyhardlinktree extraction step so the build can support symlinks in variables like ${DEPLOY_DIR_IPK/RPM/DEB}. (From OE-Core rev: c1b73fe26540c38086e259be6eb313e0826dceed) Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe/path.py: fix for "Argument list too long"Robert Yang2017-03-171-4/+6
| | | | | | | | | | | | | | | | | | | | | | | | Issue: LIN9-1648 Fixed when len(TMPDIR) = 410: $ bitbake core-image-sato-sdk [snip] Subprocess output: /bin/sh: /bin/cp: Argument list too long ERROR: core-image-sato-sdk-1.0-r0 do_rootfs: Function failed: do_rootfs [snip] This is because "copyhardlinktree(src, dst)" does "cp -afl src/* dst", while src/* is expanded to "src/file1 src/file2, src/file3..." which causes the "Argument list too long", use ./* as src and change cwd in subprocess.check_output() to fix the problem. (From OE-Core rev: a3dc93eb25fba32109edd1db6e8766074fb52e4b) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/path: Add replace_absolute_symlinks functionRichard Purdie2017-02-091-0/+21
| | | | | | | | | | | We need a function to walk a directory and replace absolute symlinks with relative ones. Add such a function to the path module of lib/oe. It does this relative to the directory being walked for files. (From OE-Core rev: 15376e5ff35367c1b40941d10e7b19302058a53e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/path: add warning comment about oe.path.remove() with wildcarded ↵Paul Eggleton2017-01-091-1/+8
| | | | | | | | | | | | | | filenames Add a warning in the doc comment for oe.path.remove() about using that function on paths that may contain wildcards in the actual file/directory names. (From OE-Core rev: 18cc0965741102bccc62dfb32ed7753cdacbadc7) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta: remove True option to getVar callsJoshua Lock2016-12-161-1/+1
| | | | | | | | | | | | | getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/path: remove duplicate importJoshua Lock2016-11-151-1/+0
| | | | | | | | | | | There's no need to import glob inside copyhardlinktree() as it's already imported for the entire path module. (From OE-Core rev: 42dc4695da136a15bebb7525b1da5c2722b10a28) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe.path: fix copyhardlinktree()Joshua Lock2016-09-061-3/+6
| | | | | | | | | | | | | | | | | The change to preserve extended attributes in copytree() and copyhardlinktree() (e591d69103a40ec4f76d1132a6039d9cb1555103) resulted in an incorrect cp invocation in copyhardlinktree() when the source directory contained hidden files. This was because the passed src was modified in place but some code paths expected it to remain unmodified from the passed value. Resolve the issue by constructing a new source string, rather than modifying the passed in string. (From OE-Core rev: 2b9fdd8448c2c29418d1c3fca9fe1789466f09b4) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe.path: preserve xattr in copytree() and copyhardlinktree()Joshua Lock2016-09-031-3/+8
| | | | | | | | | | | | | | | | Pass appropriate options to tar invocations in copytree() and copyhardlinktree() to ensure that any extended attributes on the files are preserved during the copy. We have to drop the use cpio in "Copy-pass" mode in copyhardlinktree() because cpio doesn't support extended attributes on files. Instead we revert back to using cp with different patterns depending on whether or not the directory contains dot files. (From OE-Core rev: e591d69103a40ec4f76d1132a6039d9cb1555103) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/path: Fix tar invocation with --no-recursionClemens Lang2016-07-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | tar's --no-recursion flag only applies to files mentioned after the flag, which made it a no-op in this invocation of tar, because it was at the end of the command line. This is simple to verify with GNU tar 1.29: | $ mkdir foo | $ mkdir foo/dir | $ touch foo/dir/file | $ tar -cf - foo --no-recursion | tar t | foo/ | foo/dir/ | foo/dir/file | $ tar -cf - --no-recursion foo | tar t | foo/ Modify the code so that it actually does what the comment says by moving the flag in front of the --files-from argument. (From OE-Core rev: d45f5e71fef5ffbd4408f69c5c179dc71a3eb452) Signed-off-by: Clemens Lang <clemens.lang@bmw-carit.de> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/path: remove oe.path.check_outputRoss Burton2016-07-081-45/+4
| | | | | | | | | | This was a copy-and-paste of subprocess.check_output() from when we supported Python <2.7, so simply delete it and use subprocess.check_output() instead. (From OE-Core rev: b1f2d9ed8d4dc89c9e669f43f546463ccc2a76b9) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe: drop custom implementation of oe.path.relativeRoss Burton2014-03-071-17/+1
| | | | | | | | | | | | | | As we now require Python 2.7 and os.path.relpath() was added in 2.6 we can now drop the reimplementation in oe.path. oe.path.relative is simple now a wrapper that changes the order of the arguments and it's use discouraged. (From OE-Core rev: 1a03cd16401d2926bba902ffc5df30911b5c9394) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Saul Wold <sgw@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/path: Fixup handling of .* files in top level paths for ↵Richard Purdie2013-11-121-3/+1
| | | | | | | | | | | | copyhardlinktree() Files named .* in the top level of directories handled by this function were getting lost after the directory copying command was fixed. Rather than complicate the function further, use cpio instead. (From OE-Core rev: 59612905e9bb865762667aa7b3cb06e53c3a4071) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/path: Fix performance issue got copyhardlinktree()Richard Purdie2013-11-121-1/+1
| | | | | | | | | | | | | | With the directory copy was added to avoid race issues, it wasn't noticed that tar was recursing the directories and copying files too. This is completely crazy when we hardlink those files in the next command. Resolve the issue by telling tar not to recurse. This gives a significant performance boost to various parts of the system (do_package for linux-yocto 256s -> 178s for example). (From OE-Core rev: 8b90ed084d59b4e07aa547255d327b25cfb2ee2b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/path: Fix copytree functions for tar 1.27Richard Purdie2013-10-141-2/+2
| | | | | | | | | | | | | tar version 1.27 returns: tar: --same-order option cannot be used with -c with the commandlines we have been using. We can remove the -s option (which is --same-order) to remove the error. (From OE-Core rev: 69c26e795c117aabfaf313abbfd10e70ede633d9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/lib: Update to use print as a function callRichard Purdie2013-05-091-1/+1
| | | | | | | | | In python 3 print is a function call. In some cases bb.note is a more appropriate call to make. (From OE-Core rev: 754874e68d1037c76fc4cc7d35d3fa12946d00f8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* clases/lib: Use modern exception syntaxRichard Purdie2013-05-091-3/+3
| | | | | | | | | Update older code to use modern exception handling syntax which is the form accepted by python 3. (From OE-Core rev: b010501cd089e649a68f683be0cf4d0aac90fbe3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* path.py: Deal with race issueRichard Purdie2013-05-031-7/+11
| | | | | | | | | | | | | | | | | The change to use copyhardlinktree in some of the sstate code instead of copytree exposed a race condition. This is due to cp failing if it finds a directory doesn't exist yet some other process creates it while cp was trying to create it itself. tar doesn't error in this case. To fix this we need to create the directory structure with tar, then use cp to hardlink the files. Messy but probably worth doing. I also took the opportunity to remove src_bak since the code is neater without it. (From OE-Core rev: 2f954a9a6932f1e6c564e7e7aacaac628a75eed7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate.bbclass: make hard links for staging filesRobert Yang2013-05-021-2/+7
| | | | | | | | | | | | | | | | | | | Make hard links for staging files instead of copy to save the disk space (3G will be saved for a core-image-sato build), and it doesn't affect much on the build time. The following directories are affected: 1) The sysroot 2) The DEPLOY_DIR 3) The pkgdata [YOCTO #4372] (From OE-Core rev: 5853e0f482b22258c909268fe71673a29e31989b) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Saul Wold <sgw@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/path.py: support missing directory components in realpath()Enrico Scholz2013-02-111-12/+14
| | | | | | | | | | Some use cases in OE operate on symlinks which dangling path components. Assume that these are directories instead of raising ENOENT. (From OE-Core rev: a96e2c84f24c15b77ee1fbc1f998b8b4796b8664) Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib: implemented oe.path.realpath()Enrico Scholz2013-02-111-0/+87
| | | | | | | | | | | | | | | Various parts of the buildsystem have to work with symlinks. Resolving them is not trivial because they are always relative to a sysroot directory. Patch adds a function which returns the destination of a symlink by assuming a given path as the / toplevel directory. A testsuite was added too. (From OE-Core rev: 76e0bd7f8e3a3bd052a6e329f88e2d8099e899c4) Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/path.py: Add copyhardlinktree() function and use for performance ↵Richard Purdie2013-02-061-0/+8
| | | | | | | | | | | | optimisation Add a function which copys a tree as a set of hardlinks to the original files, then use this in sstate to reduce some of the overhead of sstate package creation since the file isn't actually copied. (From OE-Core rev: 8e373e69acac853213a62afb8bbdf0adc0c5045a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Remove a number of unneeded import os/bb callsRichard Purdie2012-07-191-2/+0
| | | | | | | | | | The bb and os modules are always imported so having these extra import calls are a waste of space/execution time. They also set a bad example for people copy and pasting code so clean them up. (From OE-Core rev: 7d674820958be3a7051ea619effe1a6061d9cbe2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* path.py: add make_relative_symlink methodScott Garman2012-03-131-0/+27
| | | | | | | | | | This method allows you to convert an absolute symlink into a relative one. (From OE-Core rev: 71062c1e0fb45a4b4e58ea5d217706aa2b402d88) Signed-off-by: Scott Garman <scott.a.garman@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta: Convert getVar/getVarFlag(xxx, 1) -> (xxx, True)Richard Purdie2012-03-051-1/+1
| | | | | | | | | | | | Using "1" with getVar is bad coding style and "True" is preferred. This patch is a sed over the meta directory of the form: sed \ -e 's:\(\.getVar([^,()]*, \)1 *):\1True):g' \ -e 's:\(\.getVarFlag([^,()]*, [^,()]*, \)1 *):\1True):g' \ -i `grep -ril getVar *` Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe.path: sync up with current OEChris Larson2011-03-231-7/+14
| | | | | | | (From OE-Core rev: 1958b303f98b8db5bab00344823bbb8e086b8dba) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/path.py: Use bb.utils.mkdirhier instead of bb.mkdirhierKhem Raj2011-03-021-1/+1
| | | | | | | (From OE-Core rev: 5a22a8c06743b0a8a3d949288b99d53bd4b7ceb3) Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/path.py: Add expection class to handle the output argumentRichard Purdie2011-02-211-1/+8
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/lib/oe/path: Use check_output for subprocess so we can see error info. ↵Richard Purdie2011-02-181-10/+38
| | | | | | Import code to be python 2.6 compatible. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib.oe.path: Update copytree function to call shell commands since its twice ↵Richard Purdie2011-02-091-32/+12
| | | | | | | | as fast As an added bonus, hardlinks between files in the tree will be preserved too. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/path: Use bb.utils.copyfile as shutils can't cope with copying ↵Richard Purdie2010-08-221-1/+1
| | | | | | unreadable files Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* lib/oe/path.py: Add copytree function that worksRichard Purdie2010-08-121-0/+41
| | | | Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* lib/oe: support wildcards in path.removeJoshua Lock2010-08-121-8/+9
| | | | Signed-off-by: Joshua Lock <josh@linux.intel.com>
* lib/oe: sync with OE.devJoshua Lock2010-08-041-0/+22
| | | | | | | Most notable change is the move to creating symlinks to patches in the metadata tree rather than copying them. Signed-off-by: Joshua Lock <josh@linux.intel.com>
* lib/oe: Import oe lib from OE.devJoshua Lock2010-05-061-0/+44
This library moves the common Python methods into modules of an 'oe' Python package. Signed-off-by: Joshua Lock <josh@linux.intel.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-13 06:53:04 +0000