summaryrefslogtreecommitdiffstats
path: root/meta/recipes-devtools/python/python
Commit message (Collapse)AuthorAgeFilesLines
* python2: add ntpathArmin Kuster2020-02-211-0/+1
| | | | | | | | | | | | | python3 has this but python is missing this. [Yocto #13740] (From OE-Core rev: 3e69681857b88be44797d1d302b9e44a46a7634d) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit af41a2238beec0c34c1c1e5f25eed55f2a214643) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: update to 2.7.17Alexander Kanavin2020-01-1610-879/+33
| | | | | | | | | | | | | | | | | | | | | | | Drop backports, rebase a couple of patches. This is the second last release of py 2.x; upstream support ends on 1 January 2020, there will be one final 2.x afterwards. Note that the only thing that still needs python 2.x in oe-core is u-boot; when the next u-boot update arrives, we should find out where the py3 migration is for that component before merging the update. (From OE-Core rev: 184b60eb905bb75ecc7a0c29a175e624d8555fac) (From OE-Core rev: 7009d823a0799ce7132bd77329b273a476718c8c) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> [Minor fixup for warrior context] Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: fix CVE-2018-20852Anuj Mittal2020-01-161-0/+123
| | | | | | | | | (From OE-Core rev: 98cc3bfc6656b8648da591dcb64de8472e6c97e0) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: fix CVE-2019-16935Chen Qi2020-01-161-0/+101
| | | | | | | | | | | | | (From OE-Core rev: 1a7593bcdaf8a8cf15259aee8a0e2686247f2987) (From OE-Core rev: 27fea8ea1da28bb3163b5d503e6d16948c50f2ae) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: Fix CVE-2019-10160Changqing Li2019-10-301-0/+81
| | | | | | | | | | | | (From OE-Core rev: 23d48f2bea2d358bd8d7d4efd07792bc1f3666bd) Signed-off-by: Changqing Li <changqing.li@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> (cherry picked from commit b4240b585d7fcac2fdbf33a8e72d48cb732eb696) Signed-off-by: Armin Kuster <akuster808@gmail.com> (cherry picked from commit 10d87a3085665a959a5fda64ae3895cb27ddf343) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: CVE-2019-16056Chen Qi2019-10-301-0/+90
| | | | | | | | | | | (From OE-Core rev: 49ff6c7ef1d366007c49083f4e5faaf5a9eb086f) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> (cherry picked from commit 27be9cf71a6fe906a23e81b56f1cc18a6fc9ef97) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: fix CVE-2019-9740Anuj Mittal2019-09-301-0/+215
| | | | | | | | | | | (From OE-Core rev: 8eddac3305b7b428565103cde88cba444e3f0dd0) (From OE-Core rev: feb8ba6821da5c5ccb7b3f9d71eb9f8859006d23) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: add a fix for CVE-2019-9948 and CVE-2019-9636Martin Jansa2019-06-074-0/+249
| | | | | | | | | (From OE-Core rev: 6c8994a6ae8860c4b27d8ec990344c6b099cc685) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: update to 2.7.16Alexander Kanavin2019-06-077-446/+0
| | | | | | | | | | | | | Drop backported patches License-update: copyright years (From OE-Core rev: 0fec1d486a3ef663dc4bbab76e5ec0ed646f23e4) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* openssh/util-linux/python*: Ensure ptest output is unbufferedRichard Purdie2019-04-091-1/+1
| | | | | | | | | | | | | We need to run sed with the -u option to ensure the output is unbuffered else ptest-runner may timeout thinkig things were idle. Busybox doesn't have the -u option so we need to RDEPEND on sed (which is a good thing to do if we use it anyway). Alex Kanavin should get credit for discovering the problem. (From OE-Core rev: d3ffbebf43c23faa43af81c9ecf6fcaef36d675b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: improve ptestRoss Burton2019-04-021-1/+1
| | | | | | | | | | | | | Add missing runtime dependencies: the test suite needs proper chmod from coreutils, and the Europe/Minsk timezone. Also change run-ptest to use -W (run verbosely on failure) instead of -v (always verbose) to hopefully reduce the noise in the ptest reports. (From OE-Core rev: 7574f6850797fb7556d2f8077860b7c1d26ae8ec) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: make the python 3.x version of 2to3 utility the default oneAlexander Kanavin2019-02-081-1/+1
| | | | | | | (From OE-Core rev: 98ba7ba8c5bc091f5cc590299f1a636c9b8dc1f5) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python2-manifest: Add missing xmlrpclib.pyRichard Purdie2019-02-021-1/+2
| | | | | | | | | | | | | | | The manifest creation bug that was masking this file was fixed, rerun and add the missing file to fix: File "/usr/lib64/python2.7/SimpleXMLRPCServer.py", line 102, in <module> import xmlrpclib ImportError: No module named xmlrpclib [YOCTO #12814] (From OE-Core rev: e203b71b06df1e7d73fa300d45c428008a190d00) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python/python3: use cc_basename to replace CC for checking compilerChangqing Li2019-01-071-0/+114
| | | | | | | | | | | | | | | | | | | | | | | | When working path contains "clang"/"gcc"/"icc", it might be part of $CC because of the "--sysroot" parameter. That could cause judgement error about clang/gcc/icc compilers. eg: if build under /yocto/builds/xicc/, bitbake python, $CC will contains xicc, will make $CC match *icc, but actuall xicc just folder name. When "*icc" is matched, below errors are reported when compiling python/python3: x86_64-wrs-linux-gcc: error: strict: No such file or directory x86_64-wrs-linux-gcc: error: unrecognized command line option '-fp-model' Here use cc_basename to replace CC for checking compiler to avoid such kind of issue. (From OE-Core rev: 96383efa2726ed1bf7893332d726112a8552fc24) Signed-off-by: Li Zhou <li.zhou@windriver.com> Signed-off-by: Changqing Li <changqing.li@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* create_manifest2: Dont match filenames which contain the directory name for ↵Alejandro Enedino Hernandez Samaniego2018-12-131-11/+17
| | | | | | | | | | | | | | | | | | | | | | new manifest When creating a new python2 manifest, there is a corner case on which the filepath for a certain dependency that was found, could contain the path of an existing folder, e.g. ${libdir}/python2.7/xmlrpclib.py module path contains ${libdir}/python2.7/xml, this causes an issue where the dependency doesnt get eventually added on FILES for that module. This patch checks if the dependency that was found is a directory, if it is, it checks if it matches one of the existing directories on the manifest, if it is not, then it checks if the dependency's path (without the filename) matches one of the directories. Also some misc indentation fixes. (From OE-Core rev: a1c1253b44eb2000de55b7fa3836e5cdaa28a508) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: Adds instructions to the manifest fileAlejandro Enedino Hernandez Samaniego2018-12-012-4/+107
| | | | | | | | | | | | | | | | | | | | | While there is a bit of documentation regarding building a new manifest file for python, it seems that users usually only read the manifest file. The manifest file is in JSON format which doesn't allow comments, hence why instructions were initially put elsewhere. This patch hacks the call to open the JSON manifest file by using a marker to trick it into reading only part of the file as the manifest itself, and keep the other part as comments, which contain instructions for the user to run the create_manifest task after an upgrade or when adding a new package. (From OE-Core rev: 5641a24a70b54544012c04c6a082514d9a5aa49a) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: add tk supportMingli Yu2018-11-231-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add support to enable tk via PACKGECONFIG. before the patch: # python Python 2.7.15 (default, Nov 8 2018, 04:53:50) [GCC 8.2.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import Tkinter Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/lib-tk/Tkinter.py", line 39, in <module> import _tkinter # If this fails your Python may not be configured for Tk ImportError: No module named _tkinter >>> After the patch, if enable tk in PACKGECONFIG, then # python Python 2.7.15 (default, Oct 25 2018, 08:12:45) [GCC 8.2.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import Tkinter >>> (From OE-Core rev: 26b2702475697c123937d237e00c34eb2216b5ea) Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: add missing CVE tag to patchesRoss Burton2018-10-292-6/+2
| | | | | | | (From OE-Core rev: 67f9e9045ab91a9df15876ad73e44ff98f11bf59) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: backport patch to fix CVE-2018-14647Chen Qi2018-10-201-0/+98
| | | | | | | | | | | Backport patch to fix the following CVE. CVE: CVE-2018-14647 (From OE-Core rev: 68e51756f67499081c3c53cff6c5c1efdf4b60f0) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: backport patch to fix CVE-2018-1000802Chen Qi2018-10-201-0/+69
| | | | | | | | | | | Backport a patch to fix the following CVE. CVE: CVE-2018-1000802 (From OE-Core rev: c0343f1035af98cb451eea0de94c16fe89ffdf48) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: don't use runtime checks to identify float endianismRoss Burton2018-10-141-0/+216
| | | | | | | | | | | | | | | | | Python uses AC_RUN_IFELSE to determine the byte order for floats and doubles, and falls back onto "I don't know" if it can't run code. This results in crippled floating point numbers in Python, and the regression tests fail. Instead of running code, take a macro from autoconf-archive which compiles C with a special double in which has an ASCII representation, and then greps the binary to identify the format. This is essentially a backport of the Python 3 patch in oe-core 1781b87. (From OE-Core rev: 94cea72a23a374eb616d5642977b45172537beac) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: clean up ptestRoss Burton2018-10-142-37/+1
| | | | | | | | | | | | | | As the manifest handling is done differently now, just inherit ptest with the other inherits. test_shutil needs unzip so add to RDEPENDS. Instead of using a patched Makefile, call test.regrtest directly. (From OE-Core rev: 84f34ad223b1e3f36cab2ac12246eb90efc919bc) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: fix patch headersRoss Burton2018-10-081-3/+1
| | | | | | | (From OE-Core rev: 5aba4b7540a71908a5e1086f444b2f205ba5829d) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: fix failing ssl testsAnuj Mittal2018-10-085-0/+285
| | | | | | | | | | | | | Backport two and pick some other in-review patches from Ubuntu to fix ssl test failures because of OpenSSL 1.1.x changes. Fixes [YOCTO #12788] (From OE-Core rev: 4f4bbb936231dd30c3745ef573993f1062937ffd) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: move sqlite module into python-sqliteRoss Burton2018-10-011-11/+13
| | | | | | | | | | | | | | The module was accidentally included in python-misc. Also re-run create_manifest to update the dependencies of python-sqlite3, causing some whitespace changes. [ YOCTO #12933 ] (From OE-Core rev: 9c2b1a0bfc8783052b5dda344cc334b9c13736f5) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: mark the tests modules as special when updating the manifestRoss Burton2018-10-011-1/+1
| | | | | | | | | | We manually maintain the tests package's content and dependencies, so mark is as special (matching create_manifest3.py). (From OE-Core rev: b4bd4e7e26bb79223129abb9fb6e22e9045e3707) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: consolidate testsAndrew Geissler2018-09-211-19/+14
| | | | | | | | | | | | | | | | | | Currently the bulk of the tests in python-tests, some more in python-sqlite3-tests, and others in their parent module (such as python-ctypes). This is pointless space usage if we're not planning on running the tests, so consolidate all the tests into python-tests. This is a backport of the same changes done by Ross Burton for python3 Changes since v1: - Rebase (From OE-Core rev: 0e94737e7124f689c3697d4227bfcd228cc04295) Signed-off-by: Andrew Geissler <geissonator@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: don't sort the manifest in create_manifestAndrew Geissler2018-09-212-836/+840
| | | | | | | | | | | | | | | | | | | | | | | | | | Instead of sorting the entire manifest when it is updated, use OrderedDict to preserve the order of fields.This means that packages can be ordered in the manifest to allow non-trivial FILES assignments (such as a package that picks up pieces of other packages) The manifest has been regenerated with the new stable ordering, and distutils-staticdev moved above distutils so the packaging rules work as expected. This is a backport of the same changes done by Ross Burton for python3 Changes since v1: - Moved distutils-staticdev above distutils so packaging rules work as expected. Changes since v2: - Rebase (From OE-Core rev: 3c62c42ebde9dd4acdc74c56160d6ce8639b497c) Signed-off-by: Andrew Geissler <geissonator@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: update to version 2.7.15Derek Straka2018-07-051-43/+0
| | | | | | | | | | | | | | Update to the latest stable version License-Update: Copyright year updated to include 2018 Remove the alignment patch that is included upstream (From OE-Core rev: 855020053906478cea164ed254c08bedce48479d) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python2: Fix build with gcc8Ross Burton2018-05-221-0/+43
| | | | | | | | | (From OE-Core rev: 910f68c9c8dc26e12d28ef29e956af63d100f121) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python 2.7: fix multilib patch to accept multiarch style pathsKoen Kooi2018-05-041-1/+1
| | | | | | | | | | Using 'basename' to strip the prefix fails when using multiarch style paths. (From OE-Core rev: c61c416a6504f7e8885df3c94c839d1031920a1c) Signed-off-by: Koen Kooi <koen.kooi@linaro.org> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: refresh patchesRoss Burton2018-03-113-61/+62
| | | | | | | | | | | | | | | | | | | | | The patch tool will apply patches by default with "fuzz", which is where if the hunk context isn't present but what is there is close enough, it will force the patch in. Whilst this is useful when there's just whitespace changes, when applied to source it is possible for a patch applied with fuzz to produce broken code which still compiles (see #10450). This is obviously bad. We'd like to eventually have do_patch() rejecting any fuzz on these grounds. For that to be realistic the existing patches with fuzz need to be rebased and reviewed. (From OE-Core rev: 8a5c1328c4ea63443a92813c54bd2229c9959ff9) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: refresh patchesRoss Burton2018-03-093-31/+21
| | | | | | | | | | | | | | | | | | | | | The patch tool will apply patches by default with "fuzz", which is where if the hunk context isn't present but what is there is close enough, it will force the patch in. Whilst this is useful when there's just whitespace changes, when applied to source it is possible for a patch applied with fuzz to produce broken code which still compiles (see #10450). This is obviously bad. We'd like to eventually have do_patch() rejecting any fuzz on these grounds. For that to be realistic the existing patches with fuzz need to be rebased and reviewed. (From OE-Core rev: 0b25fcee333e6207a8596d26adfa65fec85c26df) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: Fix install purelib to make pip3-python work properlyJason Wessel2018-03-081-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | The oe-core version of python3 patches the purelib use directory to the system libdir so as to make it work with multilibs properly inside the patch fix_for_using_different_libdir.patch with: - 'purelib': '{base}/lib/python{py_version_short}/site-packages', + 'purelib': '{base}/'+sys.lib+'/python{py_version_short}/site-packages', The problem is that this broke the pip3-python package because the install directory is out of sync when using a multilib version of python. When ever a module is installed with pip3 install that is a purelib it will get installed to a location that python3 will never reference and cause random failures. This patch fixes the purelib install directory to match the purelib use directory for externally managed python modules when using multilibs. (From OE-Core rev: 45afadf0b652922f9e60c5a778acd3612da83306) Signed-off-by: Jason Wessel <jason.wessel@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python2-manifest: update the manifest file for python 2.7.14Derek Straka2018-02-061-2/+6
| | | | | | | | | | The manifest file was not regenerated when the package was updated. (From OE-Core rev: c42a1cf42b88c0dc02baf807ee2aa3dab058c4cd) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: Upgrade both python and python-native to 2.7.14Derek Straka2018-01-294-57/+19
| | | | | | | | | | | | | | | | | | | | | Rebased: - python/01-use-proper-tools-for-cross-build.patch - python/fix-makefile-for-ptest.patch - python/parallel-makeinst-create-bindir.patch Removed Upstreamed Patch: - python/Don-t-use-getentropy-on-Linux.patch Updated license checksum for changes in the copyright date. The license terms remain unchanged Added an extra do_compile item to create the native pgen that no longer gets compiled by default (From OE-Core rev: 9f2de4f9cf1eb6de75dc789bd0549f45c7a68c55) Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python*-manifest.json: add dependencies, runpyTim Orling2018-01-201-1/+12
| | | | | | | | | | | | | | | | | | | | | * runpy allows running modules/scripts with 'python -m foo' - create explicit python2 and 3 packages rather than the misc catchall * python3-setuptools and html.parser RDEPENDS on _markupbase - add to python3-core rather than misc catchall * pip3 RDEPENDS on plistlib, http.client - already packaged in python2, add to python3 - add http/ to -netclient * "pip3 install" RDEPENDS on encodingds.idna - encodings.idna packaged in -core, but missing: - stringprep (move from -codecs to -core) - unicodedata (move from -codecs to -core) (From OE-Core rev: 65a85c7db3de8d16ff91a5208a59cc2202d34e5b) Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python2 create_manifest.py: fix trailing whitespace in jsonTim Orling2018-01-201-1/+1
| | | | | | | | | | | | | | | | The json.dumps function adds trailing whitespace when using indent, because the default separator is not ','. The workaround [1] is to set the separators to be ',' and ': ', e.g. separators=(',', ': ') [1] https://hg.python.org/cpython/rev/78bad589f205 (From OE-Core rev: e4cc287e41af2e52240a20d4bf2b6cc0f1f85314) Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: Restructure python packaging and replace it with autopackagingAlejandro Hernandez2018-01-204-8/+1421
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The reason we have a manifest file for python is that our goal is to keep python-core as small as posible and add other python packages only when the user needs them, hence why we split upstream python into several packages. Although our manifest file has several issues: - Its unorganized and hard to read and understand it for an average human being. - When a new package needs to be added, the user actually has to modify the script that creates the manifest, then call the script to create a new manifest, and then submit a patch for both the script and the manifest, so its a little convoluted. - Git complains every single time a patch is submitted to the manifest, since it violates some of its guidelines. - It changes or may change with every release of python, its impossible to know if the required files for a certain package have changed (it could have more or less dependencies), the only way of doing so would be to install and test them all one by one on separate individual images, and even then we wouldnt know if they require less dependencies, we would just know if an extra dependency is required since it would complain, lets face it, this isnt feasible. - The same thing happens for new packages, if someone wants to add a new package, its dependencies need to be checked manually one by one. This patch fixes those issues, while adding some additional features. Features/Fixes: - A new manifest format is used (JSON), easy to read and understand. This file is parsed by the python recipe and python packages read from here are passed directly to bitbake during parsing time. - It provides an automatic manifest creation task (explained below), which automagically checks for every package dependencies and adds them to the new manifest, hence we will have on each package exactly what that package needs to be run, providing finer granularity. - Dependencies are also checked automagically for new packages (explained below). - Fixes the manifest in the following ways: * python-core should be base and all packages should depend on it, fixes lang, string, codecs, etc. * Fixes packages with repeated files (e.g. bssdb and db, or netclient and mime, and many others). - Sitecustomize was fixed since encoding was deprecated. - The JSON manifest file invalidates bitbake's cache, so if it changes the python package will be rebuilt. - It creates a solution for users that want precompiled bytecode files (*.pyc) INCLUDE_PYCS = "1" can be set by the user on their local.conf to include such files, some argument they get faster boot time, even when the files would be created on their first run?, but they also sometimes give a magic number error and take up space, so we leave it to the user to decide if they want them or not. - Fixes python-core dependencies, e.g. When python is run on an image, it TRIES to import everything it needs, but it doesnt necessarily fails when it doesnt find something, so even if we didnt know, we had errors like (trimmed on purpose): # trying /usr/lib/python2.7/_locale.so # trying /usr/lib/python2.7/lib-dynload/_locale.so # trying /usr/lib/python2.7/_sysconfigdata.so while it didnt complain about _locale it should have imported it, after creating a new manifest with the automated script we get: # trying /usr/lib/python2.7/lib-dynload/_locale.so dlopen("/usr/lib/python2.7/lib-dynload/_locale.so", 2); import _locale # dynamically loaded from /usr/lib/python2.7/lib-dynload/_locale.so How to use (after a new release of python, or maybe before every OE release): - A new task called create_manifest was added to the python package, which may be invoked via: $ bitbake python -c create_manifest This task runs a script on native python on our HOST system, and since the python and python-native packages come from the same source, we can use it to know the dependencies of each module as if we were doing it on an image, this script is called create_manifest.py and in a very simplistic way it does the following: 1. Reads the JSON manifest file and creates a dictionary data structure with all of our python packages, their FILES, RDEPENDS and SUMMARY. 2. Loops through all of them and runs every module listed on them asynchronously, determining every dependency that they have. 3. These module dependencies are then handled, to be able to know which packages contain those files and which should RDEPEND on one another. 4. The data structure that comes out of this, is then used to create a new manifest file which is automatically copied onto the user's python directory replacing the old one. Create_manifest script features: - Handles modules which dont exist anymore (new release for example). - Handles modules that are builtin. - Deals with modules which were not compiled (e.g. bsddb or ossaudiodev) - Deals with packages which include folders. - Deals with packages which include FILES with a wildcard. - The manifest can be constructed on a multilib environment as well. - This method works for both python modules and shared libraries used by python. How to add a new package: - If a user wants to add a new package all that has to be done is modify the python2-manifest.json file, and add the required file(s) to the FILES list, the script should handle all the rest. Real example: We want to add a web browser package, including the file webbrowser.py which at the moment is on python-misc. "webbrowser": { "files": ["${libdir}/python2.7/lib-dynload/webbrowser.py"], "rdepends": [], "summary": "Python Web Browser support"} Run bitbake python -c create_manifest and the resulting manifest should be completed after a few seconds, showing something like: "webbrowser": { "files": ["${libdir}/python2.7/webbrowser.py"], "rdepends": ["core","fcntl","io","pickle","shell","subprocess"], "summary": "Python Web Browser support"} Known errors/issues: - Some special packages are handled differently: core, misc, modules,dev, staticdev. All these should be handled manually, because they either include binaries, static libraries, include files, etc. (something that we cant import). Specifically static libraries are not not supported by this method and have to be handled by the user. - The change should be transparent to the user, other than the fact that now we CANT build python-foo (it was pretty dumb anyway, since what building python-foo actually did was building the whole python package anyway), but doing IMAGE_INSTALL_append = " python-foo" would create an image with the requested package with no issues. [YOCTO #11510] [YOCTO #11694] [YOCTO #11695] (From OE-Core rev: 6959e2e4dba5bbfa6ffd49c44e738cc1c38bc280) Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python2.7: improve reproducibilityJuro Bystricky2017-08-231-0/+34
| | | | | | | | | | | | | | | | The compiled .pyc files contain time stamp corresponding to the compile time. This prevents binary reproducibility. This patch allows to achieve binary reproducibility by overriding the build time stamp by the value exported via SOURCE_DATE_EPOCH. Patch by Bernhard M. Wiedemann, backported from https://github.com/python/cpython/pull/296 [YOCTO#11241] (From OE-Core rev: 2a044f1e4f5c63e11e631b31f741c7aabfa6f601) Signed-off-by: Juro Bystricky <juro.bystricky@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python2/python3: fix multiprocessing.BoundedSemaphore not work on ↵Hongxu Jia2017-08-131-0/+82
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | qemux86/qemuarm In upstream, the following commit: https://github.com/python/cpython/commit/e711cafab13efc9c1fe6c5cd75826401445eb585 ... commit e711cafab13efc9c1fe6c5cd75826401445eb585 Author: Benjamin Peterson <benjamin@python.org> Date: Wed Jun 11 16:44:04 2008 +0000 Merged revisions 64104,64117 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ... (see diff in setup.py) It assigned libraries for multiprocessing module according the host_platform, but not pass it to Extension. In glibc, the following commit caused two definition of sem_getvalue are different. https://sourceware.org/git/?p=glibc.git;a=commit;h=042e1521c794a945edc43b5bfa7e69ad70420524 (see diff in nptl/sem_getvalue.c for detail) `__new_sem_getvalue' is the latest sem_getvalue@@GLIBC_2.1 and `__old_sem_getvalue' is to compat the old version sem_getvalue@GLIBC_2.0. If not explicitly link to library pthread (-lpthread), it will load glibc's sem_getvalue randomly at runtime. Such as build python on linux x86_64 host and run the python on linux x86_32 target. If not link library pthread, it caused multiprocessing bounded semaphore could not work correctly. ... >>> import multiprocessing >>> pool_sema = multiprocessing.BoundedSemaphore(value=1) >>> pool_sema.acquire() True >>> pool_sema.release() Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: semaphore or lock released too many times ... And the semaphore issue also caused multiprocessing.Queue().put() hung. (From OE-Core rev: ca1542cdf6b6437a2f3dcdb33ac5216bf841c04a) Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta: Add/fix missing Upstream-Status to patchesRichard Purdie2017-06-272-1/+2
| | | | | | | | | This adds or fixes the Upstream-Status for all remaining patches missing it in OE-Core. (From OE-Core rev: 563cab8e823c3fde8ae4785ceaf4d68a5d3e25df) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: Remove unused patchesJussi Kukkonen2017-03-162-71/+0
| | | | | | | | | | CVE-2016-5636.patch and avoid_parallel_make_races_on_pgen.patch were removed from SRC_URI as handled upstream in adf4266524d0d. (From OE-Core rev: 53c0d1f18d4a11b0130e54466c91320acf4f5b18) Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: Upgrade both python and python-native to 2.7.13Alejandro Hernandez2017-03-013-238/+76
| | | | | | | | | | | | | | | | Rebased: - python-native/multilib.patch - python/multilib.patch - python/01-use-proper-tools-for-cross-build.patch Upstream: - CVE-2016-1000110 (From OE-Core rev: 2eaadc5464e3340359b626026d80afb6bc01d3f1) Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: Don't use getentropy on LinuxAndreas Oberritter2017-02-151-0/+41
| | | | | | | | | | | | Backport a patch from 2.7 branch to fix a regression with glibc 2.24 causing "OSError: [Errno 38] Function not implemented" when calling urandom() with older kernels. (From OE-Core rev: 3f2be1c857a44030478ce25b4a722667b73de446) Signed-off-by: Andreas Oberritter <obi@opendreambox.org> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: fix CVE-2016-1000110Mingli Yu2016-09-281-0/+162
| | | | | | | | | | | | | Backport patch to fix CVE-2016-1000110 from python upstream: for python2.7 https://hg.python.org/cpython/rev/ba915d561667/ for python3 https://hg.python.org/cpython/rev/a0ac52ed8f79 (From OE-Core rev: 1dd22b9d35983f35c481a1fcf67425aa0fd07a5b) Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python: upgrade from 2.7.11 to 2.7.12Alejandro Hernandez2016-08-102-77/+77
| | | | | | | | | | | | | | | | | | LICENSE did not change, only dates were changed Rebases: - multilib.patch - 01-use-proper-tools-for-cross-build.patch Upstream: - avoid_parallel_make_races_on_pgen.patch - CVE-2016-5636.patch (From OE-Core rev: 2e64fdc99a0e00bd0a4b4bf09a128e56fd8e9f8b) Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python2: Security fix CVE-2016-5636Armin Kuster2016-07-201-0/+44
| | | | | | | | | | | Affects python2 < 2.7.11 Base score (4.4) Medium (From OE-Core rev: 4d1f651047a045955b436357753c7e094468b4ed) Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta: update patch metadataRoss Burton2016-07-085-8/+8
| | | | | | | | | Enforce the correct tag names across all of oe-core for consistency. (From OE-Core rev: 606a43dc38a00cc243f933722db657aea4129f8e) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python-native, python3-native: remove the use of exported HOST_SYS and ↵Alexander Kanavin2016-06-011-0/+51
| | | | | | | | | | | | | | BUILD_SYS variables The code that utilized them was superseded by the code (in the same patch!) that is utilizing STAGING_LIBDIR/STAGING_INCDIR, and wasn't correct in the first place as HOST_SYS is not necessarily the same as the sysroot directory name. (From OE-Core rev: 8834e81a38c24a066bb4fefa93da61011d0db244) Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>