summaryrefslogtreecommitdiffstats
path: root/meta/recipes-devtools/python
Commit message (Collapse)AuthorAgeFilesLines
* python: backport patch to fix CVE-2018-14647Chen Qi2018-11-162-0/+99
| | | | | | | | | | | | | | | Backport patch to fix the following CVE. CVE: CVE-2018-14647 (From OE-Core rev: 68e51756f67499081c3c53cff6c5c1efdf4b60f0) (From OE-Core rev: c566c8d6525a263a48035d4de5249780ab08e521) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: backport patch to fix CVE-2018-1000802Chen Qi2018-11-162-0/+70
| | | | | | | | | | | | | | | Backport a patch to fix the following CVE. CVE: CVE-2018-1000802 (From OE-Core rev: c0343f1035af98cb451eea0de94c16fe89ffdf48) (From OE-Core rev: 64d0cfb0f2291434f3ceacff99015f6a35942868) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: don't use runtime checks to identify float endianismRoss Burton2018-11-162-0/+217
| | | | | | | | | | | | | | | | | | | | | Python uses AC_RUN_IFELSE to determine the byte order for floats and doubles, and falls back onto "I don't know" if it can't run code. This results in crippled floating point numbers in Python, and the regression tests fail. Instead of running code, take a macro from autoconf-archive which compiles C with a special double in which has an ASCII representation, and then greps the binary to identify the format. This is essentially a backport of the Python 3 patch in oe-core 1781b87. (From OE-Core rev: 94cea72a23a374eb616d5642977b45172537beac) (From OE-Core rev: ceae3eb0d8a0ee69182cf4f4cfa5a6a3814df1f8) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: clean up ptestRoss Burton2018-11-163-60/+3
| | | | | | | | | | | | | | | | | | As the manifest handling is done differently now, just inherit ptest with the other inherits. test_shutil needs unzip so add to RDEPENDS. Instead of using a patched Makefile, call test.regrtest directly. (From OE-Core rev: 84f34ad223b1e3f36cab2ac12246eb90efc919bc) (From OE-Core rev: c4647674da480c5925178cd821ce2d485c7467b7) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: update to version 2.7.15Derek Straka2018-11-166-495/+4
| | | | | | | | | | | | | | | | | | | Update to the latest stable version License-Update: Copyright year updated to include 2018 Remove the alignment patch that is included upstream (From OE-Core rev: 855020053906478cea164ed254c08bedce48479d) (From OE-Core rev: ab2dd15f72a94cce528276e6e3e38c56677e7ba4) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> [Bug fix update only, drop patches included in update] Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: CVE-2018-1061Sinan Kaya2018-10-182-0/+166
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * CVE-2018-1060 Prevent low-grade poplib REDOS: The regex to test a mail server's timestamp is susceptible to catastrophic backtracking on long evil responses from the server. Happily, the maximum length of malicious inputs is 2K thanks to a limit introduced in the fix for CVE-2013-1752. * CVE-2018-1061 Prevent difflib REDOS The default regex for IS_LINE_JUNK is susceptible to catastrophic backtracking. This is a potential DOS vector. Replace it with an equivalent non-vulnerable regex. Affects < 3.5.6rc1 CVE: CVE-2018-1060 CVE: CVE-2018-1061 Ref: https://access.redhat.com/security/cve/cve-2018-1060 Ref: https://access.redhat.com/security/cve/cve-2018-1061 (From OE-Core rev: 1461bcc72e6649920ecf4226e006e5667c48a21c) Signed-off-by: Sinan Kaya <okaya@kernel.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: CVE-2018-1000030Jagadeesh Krishnanjanappa2018-08-293-1/+447
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * CVE-2018-1000030-1 [2.7] bpo-31530: Stop crashes when iterating over a file on multiple threads * CVE-2018-1000030-2 Multiple threads iterating over a file can corrupt the file's internal readahead buffer resulting in crashes. To fix this, cache buffer state thread-locally for the duration of a file_iternext call and only update the file's internal state after reading completes. No attempt is made to define or provide "reasonable" semantics for iterating over a file on multiple threads. (Non-crashing) races are still present. Duplicated, corrupt, and missing data will happen. This was originally fixed by 6401e56, which raised an exception from seek() and next() when concurrent operations were detected. Alas, this simpler solution breaks legitimate use cases such as capturing the standard streams when multiple threads are logging. Affects python <= 2.7.14 (From OE-Core rev: 4b6c84e0f950f839bfb8c40f197197f838d8b733) Signed-off-by: Jagadeesh Krishnanjanappa <jkrishnanjanappa@mvista.com> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python2: Fix build with gcc8Ross Burton2018-08-043-0/+45
| | | | | | | | | | | | | | (From OE-Core rev: 910f68c9c8dc26e12d28ef29e956af63d100f121) (From OE-Core rev: 04c2d53ef48a09747d0577d9ec1ffa548d247615) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Martin Hundebøll <martin@geanix.com> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* nativesdk-python*: suppress user site dirsMartin Kelly2018-07-023-3/+3
| | | | | | | | | | | | | | | | | | | Currently, $HOME/.local is being added into sys.path in the Python SDK causing subtle host contamination. Suppress this by exporting PYTHONNOUSERSITE = "1" as documented in PEP 370. This issue occurred in the past for python*-native and was fixed similarly in OE-core commit 8fe9fb4d5a61dcbcb3fc5b9ee0234cc135af873f ("python*native.bbclass: suppress user site dirs"). (From OE-Core rev: 0dc36439cb9fe1cea50bed59da6302f78372a30b) (From OE-Core rev: 376827d359a3769ee6477eac6e6b349a2050a867) Signed-off-by: Martin Kelly <mkelly@xevo.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3-native: correctly invoke regen-importlib make targetJoe Slater2018-06-152-2/+28
| | | | | | | | | | | | | | | | | Redefiine regen-all in Makefile to invoke regen-importlib after building other regen- targets. Change the recipe to not build it before regen-all. This avoids trying to build it multiple times, which can occasionally fail. (From OE-Core rev: 72d62c9af07bf34bb8fbb3958742eb592985acc2) (From OE-Core rev: 5b9af58be9194233a05a10c3e5b5efd053cc28d2) Signed-off-by: Joe Slater <joe.slater@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* uninative: Add allow-shlib-undefined to BUILD_LDFLAGS and drop other workaroundsRichard Purdie2018-04-181-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We have a problem when for example, a glibc 2.27 based system builds some library like libpopt-native and puts it into sstate then it is reused on a pre glibc-2.27 system to build something which depends on popt like rpm-native. This results in an error like: recipe-sysroot-native/usr/lib/libpopt.so: undefined reference to `glob@GLIBC_2.27' In the past we've had this problem with new symbols like getrandom and getentropy, here its with a more complex symbol where there is an old version and a newer version. We've looked into various options, basically we cannot link against our uninative libc/ld.so since we don't have the right headers or compiler link libraries. The compiler doesn't allow you to switch in a new set either, even if we did want to ship them. Shipping a complete compiler, dev headers and libs also isn't an option. On the other hand if we follow the ld man page, it does say: """ The reasons for allowing undefined symbol references in shared libraries specified at link time are that: - A shared library specified at link time may not be the same as the one that is available at load time, so the symbol might actually be resolvable at load time. """ which is exactly this case. By the time the binary runs, it will use our uninative loader and libc and the symbol will be available. Therefore we basically have a choice, we get weird intermittent bugs, we drop uninative entirely, or we pass this option. If we pass the option, we can drop the other workarounds too. (From OE-Core rev: 75a62ede393bf6b4972390ef5290d50add19341a) (From OE-Core rev: d18bf7fa8e80d6cfaf3fdbe1ab06eec84b954432) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: Add recommended modules to nativesdk installTom Hochstein2018-04-131-0/+1
| | | | | | | | | | | | | | | The python3 installation in the SDK did not include the minimum set of modules to be functional, particularly in the case where Python is brought in through dependencies. Rather than requiring the user to explicitly add the modules, it's better to pull in the modules through RRECOMMENDS. Note that the Python 2 recipe already does this. (From OE-Core rev: 5a88e59e488248b7ec53b072f934052b96c78a51) Signed-off-by: Tom Hochstein <tom.hochstein@nxp.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Revert "python3: fix create_manifest to handle pycache folders"Richard Purdie2018-04-072-15/+10
| | | | | | | | | Alejandro asked this be reverted as the patch causes more problems than it solves. This reverts commit 5d288d286e0adb221649d896c132a607ecddc490. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3-pygobject: remove duplication in inheritMaxin B. John2018-04-051-1/+1
| | | | | | | | | | Since gnomebase class already inherits autotools and pkgconfig, there is no need to repeat it here. (From OE-Core rev: 2eb7ba6cb68e171c880bcb1fc614f1ae6b300e4f) Signed-off-by: Maxin B. John <maxin.john@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: Improve logging capabilities for do_create_manifestAlejandro Enedino Hernandez Samaniego2018-04-031-7/+27
| | | | | | | | | | | | | Adds a couple of prints to get a nicer log, and creates a small summary or report after checking every module, it makes it more feasible for adoption, easier to debug why a module ended at a certain package and see how the manifest was created. (From OE-Core rev: 4c2af72f51a7bf187615fc0b3a229d25c3e191e9) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: Fix do_create_manifest for python3-sqlite3Alejandro Enedino Hernandez Samaniego2018-03-312-21/+37
| | | | | | | | | | | | | | | | Some of the sqlite3 files ended up in python3-misc incorrectly, this is caused becuse we couldnt add the whole ${libdir}/python3/sqlite3 folder on the package because we also have another sqlite3-tests package that needs to include another folder from that directory. This patch not only fixes the do_create_manifest script to handle this situation, but also patches the manifest (created using the script) which also fixes a hiddn runtime dependency that we wouldn't have seen. (From OE-Core rev: 3324cb31670f33ffe193e550e3b3da8380b3c8c9) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: fix create_manifest to handle pycache foldersAlejandro Enedino Hernandez Samaniego2018-03-312-10/+15
| | | | | | | | | | | | | | | We have a couple of python modules that contain folders themselves, for that reason they also contain a __pycache__ folder inside those directories, since we include the whole folder in the manifest, the pycache directories end up with the files and not the cache files. This patch catches that and adds the directories to the correct structure. (From OE-Core rev: df9401e7e69ce162e257e827d67eb217666e532d) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: consolidate sqlite3 filesRoss Burton2018-03-301-2/+5
| | | | | | | | | | Some of the sqlite3 module was in python3-misc by accident, move the files into python3-sqlite3 where they belong. (From OE-Core rev: f06a8d20560b8e93cf875ee58eddda0a976e7b14) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3-pip: update to 9.0.2Derek Straka2018-03-201-2/+2
| | | | | | | | | | Update to the latest stable release (From OE-Core rev: 998ac44d8be496c852a41fa112f997362db7da4f) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3-pygobject: update to 3.28.1Derek Straka2018-03-201-3/+2
| | | | | | | | | | | | Update to the latest stable release Tested in qemux86-64 running core-image-minimal (From OE-Core rev: aa33df15dca690b6556794b604cb99bd13dbce72) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python*-setuptools: update to 39.0.0Derek Straka2018-03-203-2/+2
| | | | | | | | | | | | Update the python{3}-setuptools to the latest stable version Tested on the qemu with core-image-minimal (From OE-Core rev: 8422880acf65802dbaa08238ae9e63670ed49ff3) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3-native: update to version 3.5.5 to fix security issuesDerek Straka2018-03-201-3/+3
| | | | | | | | | | | | | | License-Update: checksum change is due to bump in copyright year Resolves CVE-2017-1000158 and other potential security issues See https://docs.python.org/3.5/whatsnew/changelog.html#python-3-5-5-final (From OE-Core rev: e19df4b40d76cdae65a26fb08efc17542e0e86b9) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3-native: fix build races in 'make regen-all'Alexander Kanavin2018-03-201-0/+2
| | | | | | | | | | | | Python 2.x and target Python 3.x do not have this issue. [YOCTO #12596] (From OE-Core rev: 5bfba3d09cfabb3a1895dcf9921801df318d804c) Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: add readline to PACKAGECONFIGAnuj Mittal2018-03-201-1/+4
| | | | | | | | | | | | | Provide a way to make readline dependency optional in case someone might want to use BSD alternative editline instead. Using editline would need some changes though (python issue13501). (From OE-Core rev: 9b12a3f031373ad0696409e4f933b1a585ea7f1f) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: Fix purelib install and runtime pathsJason Wessel2018-03-151-8/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | oe-core commit: 45afadf0b6 fixed the pip problem with purelib for python2, even though the the patch stated it was for python3. This patch addresses the purelib problem for python3. If you install the package python3-pip you will have a pip3 binary where you can see the problem on the device easily where the modules install into the incorrect area and are not able to be referenced by python3 at all. Example error: pip3 install imutils pip3 list |grep imutils || echo ERROR no imutils ERROR no imutils python3 -c 'import imutils' Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named 'imutils' (From OE-Core rev: 54e0b3bfc132613902418be148a900b10f6d9e38) Signed-off-by: Jason Wessel <jason.wessel@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: update to version 3.5.5 to fix security issuesDerek Straka2018-03-151-3/+3
| | | | | | | | | | | | | License-Update: checksum change is due to bump in copyright year Resolves CVE-2017-1000158 and other potential security issues See https://docs.python.org/3.5/whatsnew/changelog.html#python-3-5-5-final (From OE-Core rev: 4a27d50e4e8db87d005aca9d976fe8e674952777) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python*-setuptools: update to 38.5.2Derek Straka2018-03-153-2/+2
| | | | | | | | | | | Update the python{3}-setuptools to the latest stable version Tested on the qemu with core-image-minimal (From OE-Core rev: 106239a250488508f5c3593d9c8c3d4f70ff0ba3) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: refresh patchesRoss Burton2018-03-116-168/+166
| | | | | | | | | | | | | | | | | | | | | The patch tool will apply patches by default with "fuzz", which is where if the hunk context isn't present but what is there is close enough, it will force the patch in. Whilst this is useful when there's just whitespace changes, when applied to source it is possible for a patch applied with fuzz to produce broken code which still compiles (see #10450). This is obviously bad. We'd like to eventually have do_patch() rejecting any fuzz on these grounds. For that to be realistic the existing patches with fuzz need to be rebased and reviewed. (From OE-Core rev: 8a5c1328c4ea63443a92813c54bd2229c9959ff9) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: refresh patchesRoss Burton2018-03-097-81/+57
| | | | | | | | | | | | | | | | | | | | | The patch tool will apply patches by default with "fuzz", which is where if the hunk context isn't present but what is there is close enough, it will force the patch in. Whilst this is useful when there's just whitespace changes, when applied to source it is possible for a patch applied with fuzz to produce broken code which still compiles (see #10450). This is obviously bad. We'd like to eventually have do_patch() rejecting any fuzz on these grounds. For that to be realistic the existing patches with fuzz need to be rebased and reviewed. (From OE-Core rev: 0b25fcee333e6207a8596d26adfa65fec85c26df) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: Fix install purelib to make pip3-python work properlyJason Wessel2018-03-081-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | The oe-core version of python3 patches the purelib use directory to the system libdir so as to make it work with multilibs properly inside the patch fix_for_using_different_libdir.patch with: - 'purelib': '{base}/lib/python{py_version_short}/site-packages', + 'purelib': '{base}/'+sys.lib+'/python{py_version_short}/site-packages', The problem is that this broke the pip3-python package because the install directory is out of sync when using a multilib version of python. When ever a module is installed with pip3 install that is a purelib it will get installed to a location that python3 will never reference and cause random failures. This patch fixes the purelib install directory to match the purelib use directory for externally managed python modules when using multilibs. (From OE-Core rev: 45afadf0b652922f9e60c5a778acd3612da83306) Signed-off-by: Jason Wessel <jason.wessel@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3-dbus: update to version 1.2.6Derek Straka2018-02-241-3/+3
| | | | | | | | | | | License-Update: Checksum updated for added clarification to licenses in other directories and verified the license terms remain the same (From OE-Core rev: a6e8e789413c8f953fca2ad0c2591dea0993ccd1) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python*-setuptools: update to 38.5.1Derek Straka2018-02-243-2/+2
| | | | | | | | | | | | Update the python{3}-setuptools to the latest stable version Tested on the qemu with core-image-minimal (From OE-Core rev: 1753025597a9053922155bb98511a33cab6057a7) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3-pygobject: remove unused patchesMaxin B. John2018-02-162-74/+0
| | | | | | | | | | | Remove the following patches that are no-longer used in python3-pygobject: 1. 0001-configure.ac-Don-t-use-gnome-common-macros.patch 2. 0001-configure.ac-add-sysroot-path-to-GI_DATADIR-don-t-se.patch (From OE-Core rev: ff229db47c520d77ca50a10c802ef9343b58efb9) Signed-off-by: Maxin B. John <maxin.john@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python-core/python-dev: improve reproducibilityHongxu Jia2018-02-161-1/+6
| | | | | | | | | | | | | - Remove remaining build host references from packaged files. (sync with python3) - Use relative path to recompile _sysconfigdata.py (From OE-Core rev: 40a899544c9c7255aff92fea52fae4b32485fd75) Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: Fix python3-logging dependenciesTomasz Meresiński2018-02-161-1/+2
| | | | | | | | | | python3-logging depends on python3-netserver (logging/config.py:42) (From OE-Core rev: ea5f51cf2b885ba32a3e5fe6049141e45606ecf3) Signed-off-by: Tomasz Meresiński<tomasz.meresinski@comarch.pl> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3-core: improve reproducibilityHongxu Jia2018-02-161-1/+12
| | | | | | | | | | | | | Since modify _sysconfigdata.py to improve reproducibility, its pyo files should be recompiled. Remove remaining build host references of python3.5m-config (From OE-Core rev: 49403dde0b69ee624c2dce7c60cf3a3962e83c54) Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3-pycairo: update to 1.15.6Derek Straka2018-02-061-2/+2
| | | | | | | | | | | | Update to the latest stable release Tested in qemux86-64 running core-image-minimal (From OE-Core rev: fa4db2f80f6149e402f977e8de022493e6c0e1d2) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: update target and native recipes to 3.5.4Derek Straka2018-02-066-804/+52
| | | | | | | | | | | | | | | | | | | | | | Use the latest 3.5 version until the 3.6 migration is complete Removed the following upstreamed patches: - python3/Fix-29519-weakref-spewing-exceptions-during-interp-f.patch - python3/upstream-random-fixes.patch Rebased the following pathes: - python3/0001-cross-compile-support.patch Regenerated the manifest based on the latest release version Updated the license checksum for the latest version that updated the copyright dates (From OE-Core rev: eb80d0391d7d4e83a61ed8850d936b102be3fa02) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python2-manifest: update the manifest file for python 2.7.14Derek Straka2018-02-061-2/+6
| | | | | | | | | | The manifest file was not regenerated when the package was updated. (From OE-Core rev: c42a1cf42b88c0dc02baf807ee2aa3dab058c4cd) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python-native: add dependency for gdbm and db native packagesDerek Straka2018-02-061-1/+1
| | | | | | | | | | | | These two packages are required to ensure the manifest files contain all of the generated packages. Without this, the db and gdbm packages will not contain the .so files as they are skipped during the compilation steps (From OE-Core rev: 912c06615269f42230db2d93d70db2b340ed270a) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: Upgrade both python and python-native to 2.7.14Derek Straka2018-01-297-61/+28
| | | | | | | | | | | | | | | | | | | | | Rebased: - python/01-use-proper-tools-for-cross-build.patch - python/fix-makefile-for-ptest.patch - python/parallel-makeinst-create-bindir.patch Removed Upstreamed Patch: - python/Don-t-use-getentropy-on-Linux.patch Updated license checksum for changes in the copyright date. The license terms remain unchanged Added an extra do_compile item to create the native pgen that no longer gets compiled by default (From OE-Core rev: 9f2de4f9cf1eb6de75dc789bd0549f45c7a68c55) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3-manifest: add the _pydecimal files to the numbers packageDerek Straka2018-01-291-3/+5
| | | | | | | | | | | | The _pydecimal files are required to use the numbers package and downstream packages are currently required to add a RDEPENDS on python3-misc to avoid an import error (From OE-Core rev: 4ca2d607a6b6fe8686bf89177287f2b9bae01245) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: fix parse dependenciesRoss Burton2018-01-292-44/+39
| | | | | | | | | | | | | | | | | | | | | | | Adding a file-checksums flag for the manifest to do_split_packages doesn't achieve anything as do_split_packages isn't a task. Changing this to tha task do_package shows that the path is wrong, but we also know that as the manifest is in SRC_URI any changes to it would result in a rebuild anyway, so this line can be deleted. However there is a problem of the recipe not being reparsed when it needs to be, if the JSON has changed. The main bitbake process can hash the recipe and use stale data from the cache as it hasn't considered the manifest file changing. This results in non-determinism warnings when the worker parses the recipe again and comes to a different hash (as the manifest has changed, so the packaging changed). Solve this by calling bb.parse.mark_dependency() to declare the dependency on the manifest. (From OE-Core rev: a321b28c8dafc9775f465ce7c0f6bcbe8ccc2945) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python*-setuptools: update to 38.4.0Derek Straka2018-01-233-2/+2
| | | | | | | | | | | | Update the python{3}-setuptools to the latest stable version Tested on the qemu with core-image-minimal (From OE-Core rev: 32a30b3edfdb76b554775817f98851f851316fe0) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: fix RDEPENDS on several recipes, due to non-existent packagesAlejandro Hernandez2018-01-208-15/+3
| | | | | | | | | | | The packaging has been altered slightly so ensure the dependencies are all still valid. (From OE-Core rev: 3328211afdef8ffb00dd4dff1143959d5412b075) Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: Fix native compilation of gdbm module and manifestAlejandro Hernandez2018-01-202-3/+14
| | | | | | | | | | | | | | | | | | | | | The gdbm module wasnt being built on python3-native showing the following error during compilation: Failed to build these modules: _gdbm This patch adds the required dependency to fix the compilation problem. This issue on python3-native caused the manifest creation script to be unaware of the gdbm library, so this patch also fixes the create_manifest task for target python, and the manifest file to reflect the changes on target python as well. (From OE-Core rev: c47b54aeae5daabb458d6f7118a16257021c1822) Signed-off-by: Alejandro Hernandez <alejandr@xilinx.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python*-manifest.json: add dependencies, runpyTim Orling2018-01-202-5/+49
| | | | | | | | | | | | | | | | | | | | | * runpy allows running modules/scripts with 'python -m foo' - create explicit python2 and 3 packages rather than the misc catchall * python3-setuptools and html.parser RDEPENDS on _markupbase - add to python3-core rather than misc catchall * pip3 RDEPENDS on plistlib, http.client - already packaged in python2, add to python3 - add http/ to -netclient * "pip3 install" RDEPENDS on encodingds.idna - encodings.idna packaged in -core, but missing: - stringprep (move from -codecs to -core) - unicodedata (move from -codecs to -core) (From OE-Core rev: 65a85c7db3de8d16ff91a5208a59cc2202d34e5b) Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: Restructure python3 packaging and replace it with autopackagingAlejandro Hernandez2018-01-207-1148/+1675
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | See previous commit (python2 version) for more info, since mostly everything applies here as well. Old manifest file had several issues: - Its unorganized and hard to read and understand it for an average human being. - When a new package needs to be added, the user actually has to modify the script that creates the manifest, then call the script to create a new manifest, and then submit a patch for both the script and the manifest, so its a little convoluted. - Git complains every single time a patch is submitted to the manifest, since it violates some of its guidelines. - It changes or may change with every release of python, its impossible to know if the required files for a certain package have changed (it could have more or less dependencies), the only way of doing so would be to install and test them all one by one on separate individual images, and even then we wouldnt know if they require less dependencies, we would just know if an extra dependency is required since it would complain, lets face it, this isnt feasible. - The same thing happens for new packages, if someone wants to add a new package, its dependencies need to be checked manually one by one. Features/Fixes: - A new manifest format is used (JSON), easy to read and understand. This file is parsed by the python recipe and python packages read from here are passed directly to bitbake during parsing time. - It provides an automatic manifest creation task (explained on previous commit), which automagically checks for every package dependencies and adds them to the new manifest, hence we will have on each package exactly what that package needs to be run, providing finer granularity. - Dependencies are also checked automagically for new packages (explained on previous commit). This patch has the same features as the python2 version but it differs in the following ways: - Python3 handles precompiled bytecode files (*.pyc) differently. for this reason and since we are cross compiling, wildcards couldnt be avoided on python3 (See PEP #3147 [1]). Both the manifest and the manifest creation script handle this differently, the manifest for python3 has an extra field for cached files, which is how it lets the user install the cached files or not via : INCLUDE_PYCS = "1" on their local.conf. - Shared libraries nomenclature also changed on python3, so again, we use wildcards to deal with this issue ( See PEP #3149 [2]): - Fixes python3 manifest, python3-core should be base and everything should depend on it, hence several packages were deleted: python3-enum, re, gdbm, subprocess, signal, readline. - When building python3-native it adds as symlink to it called nativepython3, which is then isued by the create_manifest task. - Fixes [YOCTO #11513] while were at it. References: [1] https://www.python.org/dev/peps/pep-3147/ [2] https://www.python.org/dev/peps/pep-3149/ (From OE-Core rev: 54ac820b8a639950ccb534dcd9d6eaf8b2b736e0) Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python2 create_manifest.py: fix trailing whitespace in jsonTim Orling2018-01-201-1/+1
| | | | | | | | | | | | | | | | The json.dumps function adds trailing whitespace when using indent, because the default separator is not ','. The workaround [1] is to set the separators to be ',' and ': ', e.g. separators=(',', ': ') [1] https://hg.python.org/cpython/rev/78bad589f205 (From OE-Core rev: e4cc287e41af2e52240a20d4bf2b6cc0f1f85314) Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: Restructure python packaging and replace it with autopackagingAlejandro Hernandez2018-01-208-862/+1528
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The reason we have a manifest file for python is that our goal is to keep python-core as small as posible and add other python packages only when the user needs them, hence why we split upstream python into several packages. Although our manifest file has several issues: - Its unorganized and hard to read and understand it for an average human being. - When a new package needs to be added, the user actually has to modify the script that creates the manifest, then call the script to create a new manifest, and then submit a patch for both the script and the manifest, so its a little convoluted. - Git complains every single time a patch is submitted to the manifest, since it violates some of its guidelines. - It changes or may change with every release of python, its impossible to know if the required files for a certain package have changed (it could have more or less dependencies), the only way of doing so would be to install and test them all one by one on separate individual images, and even then we wouldnt know if they require less dependencies, we would just know if an extra dependency is required since it would complain, lets face it, this isnt feasible. - The same thing happens for new packages, if someone wants to add a new package, its dependencies need to be checked manually one by one. This patch fixes those issues, while adding some additional features. Features/Fixes: - A new manifest format is used (JSON), easy to read and understand. This file is parsed by the python recipe and python packages read from here are passed directly to bitbake during parsing time. - It provides an automatic manifest creation task (explained below), which automagically checks for every package dependencies and adds them to the new manifest, hence we will have on each package exactly what that package needs to be run, providing finer granularity. - Dependencies are also checked automagically for new packages (explained below). - Fixes the manifest in the following ways: * python-core should be base and all packages should depend on it, fixes lang, string, codecs, etc. * Fixes packages with repeated files (e.g. bssdb and db, or netclient and mime, and many others). - Sitecustomize was fixed since encoding was deprecated. - The JSON manifest file invalidates bitbake's cache, so if it changes the python package will be rebuilt. - It creates a solution for users that want precompiled bytecode files (*.pyc) INCLUDE_PYCS = "1" can be set by the user on their local.conf to include such files, some argument they get faster boot time, even when the files would be created on their first run?, but they also sometimes give a magic number error and take up space, so we leave it to the user to decide if they want them or not. - Fixes python-core dependencies, e.g. When python is run on an image, it TRIES to import everything it needs, but it doesnt necessarily fails when it doesnt find something, so even if we didnt know, we had errors like (trimmed on purpose): # trying /usr/lib/python2.7/_locale.so # trying /usr/lib/python2.7/lib-dynload/_locale.so # trying /usr/lib/python2.7/_sysconfigdata.so while it didnt complain about _locale it should have imported it, after creating a new manifest with the automated script we get: # trying /usr/lib/python2.7/lib-dynload/_locale.so dlopen("/usr/lib/python2.7/lib-dynload/_locale.so", 2); import _locale # dynamically loaded from /usr/lib/python2.7/lib-dynload/_locale.so How to use (after a new release of python, or maybe before every OE release): - A new task called create_manifest was added to the python package, which may be invoked via: $ bitbake python -c create_manifest This task runs a script on native python on our HOST system, and since the python and python-native packages come from the same source, we can use it to know the dependencies of each module as if we were doing it on an image, this script is called create_manifest.py and in a very simplistic way it does the following: 1. Reads the JSON manifest file and creates a dictionary data structure with all of our python packages, their FILES, RDEPENDS and SUMMARY. 2. Loops through all of them and runs every module listed on them asynchronously, determining every dependency that they have. 3. These module dependencies are then handled, to be able to know which packages contain those files and which should RDEPEND on one another. 4. The data structure that comes out of this, is then used to create a new manifest file which is automatically copied onto the user's python directory replacing the old one. Create_manifest script features: - Handles modules which dont exist anymore (new release for example). - Handles modules that are builtin. - Deals with modules which were not compiled (e.g. bsddb or ossaudiodev) - Deals with packages which include folders. - Deals with packages which include FILES with a wildcard. - The manifest can be constructed on a multilib environment as well. - This method works for both python modules and shared libraries used by python. How to add a new package: - If a user wants to add a new package all that has to be done is modify the python2-manifest.json file, and add the required file(s) to the FILES list, the script should handle all the rest. Real example: We want to add a web browser package, including the file webbrowser.py which at the moment is on python-misc. "webbrowser": { "files": ["${libdir}/python2.7/lib-dynload/webbrowser.py"], "rdepends": [], "summary": "Python Web Browser support"} Run bitbake python -c create_manifest and the resulting manifest should be completed after a few seconds, showing something like: "webbrowser": { "files": ["${libdir}/python2.7/webbrowser.py"], "rdepends": ["core","fcntl","io","pickle","shell","subprocess"], "summary": "Python Web Browser support"} Known errors/issues: - Some special packages are handled differently: core, misc, modules,dev, staticdev. All these should be handled manually, because they either include binaries, static libraries, include files, etc. (something that we cant import). Specifically static libraries are not not supported by this method and have to be handled by the user. - The change should be transparent to the user, other than the fact that now we CANT build python-foo (it was pretty dumb anyway, since what building python-foo actually did was building the whole python package anyway), but doing IMAGE_INSTALL_append = " python-foo" would create an image with the requested package with no issues. [YOCTO #11510] [YOCTO #11694] [YOCTO #11695] (From OE-Core rev: 6959e2e4dba5bbfa6ffd49c44e738cc1c38bc280) Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-19 13:23:25 +0000