linux/poky.git/scripts, branch 2.5_M2

postinst_intercepts: add a delay_to_first_boot hook

2018-01-30T11:50:12+00:00

This allows postinsts scripts to explicitly request a deferral to first boot (by calling 'postinst_intercept delay_to_first_boot') instead of 'exit 1' which should be used only to indicate actual script failures. (From OE-Core rev: 853ac4a061e47c70b61e255c3cf6109d2058d2cc) Signed-off-by: Alexander Kanavin Signed-off-by: Richard Purdie

distrodata: add a utility script to compare list of recipes

2018-01-29T08:49:50+00:00

distrocompare.sh is added to compare the added list of recipes between two releases. The output of the script will share the information of the licenses required and other distributions that are using the package. If a single input is provided, it will compare the current branch with the provided branch/commit-ish package list. To run : distrocompare.sh E.g. distrocompare.sh morty 92aa0e7 E.g. distrocompare.sh morty pyro E.g. distrocompare.sh morty output : The script will produce a file ending with new_recipe_list.txt preceeded by the branch name from input (From OE-Core rev: 32b363c2ba91fde4f10e5fe2c898b2fc2702aa85) Signed-off-by: Tan Shen Joon Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

oe-pkgdata-util: Include colon when searching for variables

2018-01-29T08:49:49+00:00

When searching for variables, include colon to ensure the script doesn't find a variable that starts with the same name. (From OE-Core rev: 3d2c87c4f4115b01534ab198c27682c7e4c5f31f) Signed-off-by: Amanda Brindle Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

oe-pkgdata-util: Add support for RPROVIDES

2018-01-22T10:39:09+00:00

In lookup_recipe, package_info, and list_pkg_files, check if the package name exists in runtime-rprovides. If so, and the provider package has a different name than the inputted package, print a note that says the specified package is in another package's RPROVIDES. If the provider package has the same name as the inputted package, continue as before. Fixes [YOCTO 11943] (From OE-Core rev: f78478f0d0379ea02727c81ad2455207c70d140b) Signed-off-by: Amanda Brindle Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

oe-pkgdata-util: Refactor functions for consistency

2018-01-22T10:39:09+00:00

Refactor functions lookup_recipe and package_info to be consistent with list_pkg_files. Print the appropriate information as soon as it's found, rather than storing it in a mappings variable and wait to print after searching all packages. (From OE-Core rev: 64d3ce83d5c48d479709b4c1355e23b3768493fb) Signed-off-by: Amanda Brindle Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

python3: Restructure python3 packaging and replace it with autopackaging

2018-01-20T22:31:56+00:00

See previous commit (python2 version) for more info, since mostly everything applies here as well. Old manifest file had several issues: - Its unorganized and hard to read and understand it for an average human being. - When a new package needs to be added, the user actually has to modify the script that creates the manifest, then call the script to create a new manifest, and then submit a patch for both the script and the manifest, so its a little convoluted. - Git complains every single time a patch is submitted to the manifest, since it violates some of its guidelines. - It changes or may change with every release of python, its impossible to know if the required files for a certain package have changed (it could have more or less dependencies), the only way of doing so would be to install and test them all one by one on separate individual images, and even then we wouldnt know if they require less dependencies, we would just know if an extra dependency is required since it would complain, lets face it, this isnt feasible. - The same thing happens for new packages, if someone wants to add a new package, its dependencies need to be checked manually one by one. Features/Fixes: - A new manifest format is used (JSON), easy to read and understand. This file is parsed by the python recipe and python packages read from here are passed directly to bitbake during parsing time. - It provides an automatic manifest creation task (explained on previous commit), which automagically checks for every package dependencies and adds them to the new manifest, hence we will have on each package exactly what that package needs to be run, providing finer granularity. - Dependencies are also checked automagically for new packages (explained on previous commit). This patch has the same features as the python2 version but it differs in the following ways: - Python3 handles precompiled bytecode files (*.pyc) differently. for this reason and since we are cross compiling, wildcards couldnt be avoided on python3 (See PEP #3147 [1]). Both the manifest and the manifest creation script handle this differently, the manifest for python3 has an extra field for cached files, which is how it lets the user install the cached files or not via : INCLUDE_PYCS = "1" on their local.conf. - Shared libraries nomenclature also changed on python3, so again, we use wildcards to deal with this issue ( See PEP #3149 [2]): - Fixes python3 manifest, python3-core should be base and everything should depend on it, hence several packages were deleted: python3-enum, re, gdbm, subprocess, signal, readline. - When building python3-native it adds as symlink to it called nativepython3, which is then isued by the create_manifest task. - Fixes [YOCTO #11513] while were at it. References: [1] https://www.python.org/dev/peps/pep-3147/ [2] https://www.python.org/dev/peps/pep-3149/ (From OE-Core rev: 54ac820b8a639950ccb534dcd9d6eaf8b2b736e0) Signed-off-by: Alejandro Hernandez Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

python: Restructure python packaging and replace it with autopackaging

2018-01-20T22:31:56+00:00

The reason we have a manifest file for python is that our goal is to keep python-core as small as posible and add other python packages only when the user needs them, hence why we split upstream python into several packages. Although our manifest file has several issues: - Its unorganized and hard to read and understand it for an average human being. - When a new package needs to be added, the user actually has to modify the script that creates the manifest, then call the script to create a new manifest, and then submit a patch for both the script and the manifest, so its a little convoluted. - Git complains every single time a patch is submitted to the manifest, since it violates some of its guidelines. - It changes or may change with every release of python, its impossible to know if the required files for a certain package have changed (it could have more or less dependencies), the only way of doing so would be to install and test them all one by one on separate individual images, and even then we wouldnt know if they require less dependencies, we would just know if an extra dependency is required since it would complain, lets face it, this isnt feasible. - The same thing happens for new packages, if someone wants to add a new package, its dependencies need to be checked manually one by one. This patch fixes those issues, while adding some additional features. Features/Fixes: - A new manifest format is used (JSON), easy to read and understand. This file is parsed by the python recipe and python packages read from here are passed directly to bitbake during parsing time. - It provides an automatic manifest creation task (explained below), which automagically checks for every package dependencies and adds them to the new manifest, hence we will have on each package exactly what that package needs to be run, providing finer granularity. - Dependencies are also checked automagically for new packages (explained below). - Fixes the manifest in the following ways: * python-core should be base and all packages should depend on it, fixes lang, string, codecs, etc. * Fixes packages with repeated files (e.g. bssdb and db, or netclient and mime, and many others). - Sitecustomize was fixed since encoding was deprecated. - The JSON manifest file invalidates bitbake's cache, so if it changes the python package will be rebuilt. - It creates a solution for users that want precompiled bytecode files (*.pyc) INCLUDE_PYCS = "1" can be set by the user on their local.conf to include such files, some argument they get faster boot time, even when the files would be created on their first run?, but they also sometimes give a magic number error and take up space, so we leave it to the user to decide if they want them or not. - Fixes python-core dependencies, e.g. When python is run on an image, it TRIES to import everything it needs, but it doesnt necessarily fails when it doesnt find something, so even if we didnt know, we had errors like (trimmed on purpose): # trying /usr/lib/python2.7/_locale.so # trying /usr/lib/python2.7/lib-dynload/_locale.so # trying /usr/lib/python2.7/_sysconfigdata.so while it didnt complain about _locale it should have imported it, after creating a new manifest with the automated script we get: # trying /usr/lib/python2.7/lib-dynload/_locale.so dlopen("/usr/lib/python2.7/lib-dynload/_locale.so", 2); import _locale # dynamically loaded from /usr/lib/python2.7/lib-dynload/_locale.so How to use (after a new release of python, or maybe before every OE release): - A new task called create_manifest was added to the python package, which may be invoked via: $ bitbake python -c create_manifest This task runs a script on native python on our HOST system, and since the python and python-native packages come from the same source, we can use it to know the dependencies of each module as if we were doing it on an image, this script is called create_manifest.py and in a very simplistic way it does the following: 1. Reads the JSON manifest file and creates a dictionary data structure with all of our python packages, their FILES, RDEPENDS and SUMMARY. 2. Loops through all of them and runs every module listed on them asynchronously, determining every dependency that they have. 3. These module dependencies are then handled, to be able to know which packages contain those files and which should RDEPEND on one another. 4. The data structure that comes out of this, is then used to create a new manifest file which is automatically copied onto the user's python directory replacing the old one. Create_manifest script features: - Handles modules which dont exist anymore (new release for example). - Handles modules that are builtin. - Deals with modules which were not compiled (e.g. bsddb or ossaudiodev) - Deals with packages which include folders. - Deals with packages which include FILES with a wildcard. - The manifest can be constructed on a multilib environment as well. - This method works for both python modules and shared libraries used by python. How to add a new package: - If a user wants to add a new package all that has to be done is modify the python2-manifest.json file, and add the required file(s) to the FILES list, the script should handle all the rest. Real example: We want to add a web browser package, including the file webbrowser.py which at the moment is on python-misc. "webbrowser": { "files": ["${libdir}/python2.7/lib-dynload/webbrowser.py"], "rdepends": [], "summary": "Python Web Browser support"} Run bitbake python -c create_manifest and the resulting manifest should be completed after a few seconds, showing something like: "webbrowser": { "files": ["${libdir}/python2.7/webbrowser.py"], "rdepends": ["core","fcntl","io","pickle","shell","subprocess"], "summary": "Python Web Browser support"} Known errors/issues: - Some special packages are handled differently: core, misc, modules,dev, staticdev. All these should be handled manually, because they either include binaries, static libraries, include files, etc. (something that we cant import). Specifically static libraries are not not supported by this method and have to be handled by the user. - The change should be transparent to the user, other than the fact that now we CANT build python-foo (it was pretty dumb anyway, since what building python-foo actually did was building the whole python package anyway), but doing IMAGE_INSTALL_append = " python-foo" would create an image with the requested package with no issues. [YOCTO #11510] [YOCTO #11694] [YOCTO #11695] (From OE-Core rev: 6959e2e4dba5bbfa6ffd49c44e738cc1c38bc280) Signed-off-by: Alejandro Hernandez Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

scripts/oe-depends-dot: add it to handle dot files

2018-01-19T12:37:13+00:00

Add it to handle recipe-depends.dot and task-depends.dot. E.g.: * Print why rpm is built $ oe-depends-dot -k rpm --why/-w recipe-depends.dot Because: core-image-sato libdnf libsolv dnf * Print bzip2-native's depends $ oe-depends-dot -k bzip2-native --depends/-d recipe-depends.dot Depends: automake-native gnu-config-native libtool-native quilt-native autoconf-native * Remove duplicated dependencies to reduce the size of the dot files. For example, A->B, B->C, A->C, then A->C can be removed. The dot files are too big, we nearly couldn't use 'dot -T' to generate pictcures for target recipes, remove the duplicated dependencies makes is it possible. $ bitbake core-image-sato -g $ oe-depends-dot -r recipe-depends.dot Saving reduced dot file to recipe-depends-reduced.dot $ du -sh recipe-depends*.dot 608K recipe-depends.dot 32K recipe-depends-reduced.dot It has been recuded from 608K to 32K, now we can generate a picture, otherwise, it is too big: $ dot -Tpng recipe-depends-reduced.dot -O It also can handle task-depends.dot. (From OE-Core rev: 7dc7860691304d63e7ad728d2180474906fe0a5c) Signed-off-by: Robert Yang Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

contrib/yocto-bsp-kernel-update.sh: remove this script

2018-01-18T13:05:56+00:00

This script was unused since yocto-bsp tool had been removed. (From meta-yocto rev: 594ced59dc80c11d573ae4a16e1aff910a8d9300) Signed-off-by: Yi Zhao Signed-off-by: Richard Purdie

buildhistory-diff: add support for colourising the output

2018-01-18T12:50:37+00:00

Colour value removes in red and additions in green, making it easier to scan the output for relevant changes. This adds a --colour option to specify whether colouring should be on, off, or detected. The default is detected, and depends on whether stdout is a TTY (same behaviour as git). (From OE-Core rev: 4208f1546c92f069e432d1865269ce539db8cea7) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie