diff options
author | Alejandro Hernandez <alejandro.hernandez@linux.intel.com> | 2017-08-04 14:06:14 -0700 |
---|---|---|
committer | Richard Purdie <richard.purdie@linuxfoundation.org> | 2018-01-20 22:31:56 +0000 |
commit | b6777878ff03c3e956386020a19d11c875c835ae (patch) | |
tree | 753d1f85db4fb803204178b121ed544f792edd1b /meta/recipes-devtools/python/python3_3.5.3.bb | |
parent | 86e002572d53cf2e60196269928a411375d24294 (diff) | |
download | poky-b6777878ff03c3e956386020a19d11c875c835ae.tar.gz |
python3: Restructure python3 packaging and replace it with autopackaging
See previous commit (python2 version) for more info, since mostly
everything applies here as well.
Old manifest file had several issues:
- Its unorganized and hard to read and understand it for an average
human being.
- When a new package needs to be added, the user actually has to modify
the script that creates the manifest, then call the script to create
a new manifest, and then submit a patch for both the script and the
manifest, so its a little convoluted.
- Git complains every single time a patch is submitted to the manifest,
since it violates some of its guidelines.
- It changes or may change with every release of python, its impossible
to know if the required files for a certain package have changed
(it could have more or less dependencies), the only way of doing so
would be to install and test them all one by one on separate individual
images, and even then we wouldnt know if they require less dependencies,
we would just know if an extra dependency is required since it would
complain, lets face it, this isnt feasible.
- The same thing happens for new packages, if someone wants to add a new
package, its dependencies need to be checked manually one by one.
Features/Fixes:
- A new manifest format is used (JSON), easy to read and understand.
This file is parsed by the python recipe and python packages
read from here are passed directly to bitbake during parsing time.
- It provides an automatic manifest creation task (explained on previous
commit), which automagically checks for every package dependencies and
adds them to the new manifest, hence we will have on each package
exactly what that package needs to be run, providing finer granularity.
- Dependencies are also checked automagically for new packages
(explained on previous commit).
This patch has the same features as the python2 version but it differs
in the following ways:
- Python3 handles precompiled bytecode files (*.pyc) differently.
for this reason and since we are cross compiling, wildcards couldnt be
avoided on python3 (See PEP #3147 [1]).
Both the manifest and the manifest creation script handle this
differently, the manifest for python3 has an extra field for cached
files, which is how it lets the user install the cached files or not
via : INCLUDE_PYCS = "1" on their local.conf.
- Shared libraries nomenclature also changed on python3, so again, we
use wildcards to deal with this issue ( See PEP #3149 [2]):
- Fixes python3 manifest, python3-core should be base and everything
should depend on it, hence several packages were deleted:
python3-enum, re, gdbm, subprocess, signal, readline.
- When building python3-native it adds as symlink to it called
nativepython3, which is then isued by the create_manifest task.
- Fixes [YOCTO #11513] while were at it.
References:
[1] https://www.python.org/dev/peps/pep-3147/
[2] https://www.python.org/dev/peps/pep-3149/
(From OE-Core rev: 54ac820b8a639950ccb534dcd9d6eaf8b2b736e0)
Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'meta/recipes-devtools/python/python3_3.5.3.bb')
-rw-r--r-- | meta/recipes-devtools/python/python3_3.5.3.bb | 91 |
1 files changed, 87 insertions, 4 deletions
diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb index 7f54ea4ecd..970ba63445 100644 --- a/meta/recipes-devtools/python/python3_3.5.3.bb +++ b/meta/recipes-devtools/python/python3_3.5.3.bb | |||
@@ -1,6 +1,7 @@ | |||
1 | require recipes-devtools/python/python.inc | 1 | require recipes-devtools/python/python.inc |
2 | 2 | ||
3 | DEPENDS = "python3-native libffi bzip2 gdbm openssl readline sqlite3 zlib virtual/libintl xz" | 3 | DEPENDS = "python3-native libffi bzip2 gdbm openssl readline sqlite3 zlib virtual/libintl xz" |
4 | |||
4 | PR = "${INC_PR}.0" | 5 | PR = "${INC_PR}.0" |
5 | PYTHON_MAJMIN = "3.5" | 6 | PYTHON_MAJMIN = "3.5" |
6 | PYTHON_BINABI= "${PYTHON_MAJMIN}m" | 7 | PYTHON_BINABI= "${PYTHON_MAJMIN}m" |
@@ -196,11 +197,8 @@ py_package_preprocess () { | |||
196 | ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata.py | 197 | ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata.py |
197 | } | 198 | } |
198 | 199 | ||
199 | require python-${PYTHON_MAJMIN}-manifest.inc | ||
200 | |||
201 | # manual dependency additions | 200 | # manual dependency additions |
202 | RPROVIDES_${PN}-modules = "${PN}" | 201 | RPROVIDES_${PN}-modules = "${PN}" |
203 | RRECOMMENDS_${PN}-core = "${PN}-readline" | ||
204 | RRECOMMENDS_${PN}-crypt = "openssl" | 202 | RRECOMMENDS_${PN}-crypt = "openssl" |
205 | RRECOMMENDS_${PN}-crypt_class-nativesdk = "nativesdk-openssl" | 203 | RRECOMMENDS_${PN}-crypt_class-nativesdk = "nativesdk-openssl" |
206 | 204 | ||
@@ -219,7 +217,7 @@ INSANE_SKIP_${PN}-dev += "dev-elf" | |||
219 | 217 | ||
220 | # catch all the rest (unsorted) | 218 | # catch all the rest (unsorted) |
221 | PACKAGES += "${PN}-misc" | 219 | PACKAGES += "${PN}-misc" |
222 | RDEPENDS_${PN}-misc += "${PN}-core ${PN}-email ${PN}-codecs ${PN}-textutils ${PN}-argparse" | 220 | RDEPENDS_${PN}-misc += "${PN}-core ${PN}-email ${PN}-codecs" |
223 | RDEPENDS_${PN}-modules += "${PN}-misc" | 221 | RDEPENDS_${PN}-modules += "${PN}-misc" |
224 | FILES_${PN}-misc = "${libdir}/python${PYTHON_MAJMIN}" | 222 | FILES_${PN}-misc = "${libdir}/python${PYTHON_MAJMIN}" |
225 | 223 | ||
@@ -228,3 +226,88 @@ PACKAGES += "${PN}-man" | |||
228 | FILES_${PN}-man = "${datadir}/man" | 226 | FILES_${PN}-man = "${datadir}/man" |
229 | 227 | ||
230 | BBCLASSEXTEND = "nativesdk" | 228 | BBCLASSEXTEND = "nativesdk" |
229 | |||
230 | RPROVIDES_${PN} += "${PN}-modules" | ||
231 | |||
232 | # We want bytecode precompiled .py files (.pyc's) by default | ||
233 | # but the user may set it on their own conf | ||
234 | |||
235 | INCLUDE_PYCS ?= "1" | ||
236 | |||
237 | python(){ | ||
238 | |||
239 | pythondir = d.getVar('THISDIR',True) | ||
240 | |||
241 | # Read JSON manifest | ||
242 | import json | ||
243 | with open(pythondir+'/python3/python3-manifest.json') as manifest_file: | ||
244 | python_manifest=json.load(manifest_file) | ||
245 | |||
246 | include_pycs = d.getVar('INCLUDE_PYCS') | ||
247 | |||
248 | packages = d.getVar('PACKAGES').split() | ||
249 | pn = d.getVar('PN') | ||
250 | |||
251 | newpackages=[] | ||
252 | for key in python_manifest: | ||
253 | pypackage= pn + '-' + key | ||
254 | |||
255 | if pypackage not in packages: | ||
256 | # We need to prepend, otherwise python-misc gets everything | ||
257 | # so we use a new variable | ||
258 | newpackages.append(pypackage) | ||
259 | |||
260 | # "Build" python's manifest FILES, RDEPENDS and SUMMARY | ||
261 | d.setVar('FILES_' + pypackage, '') | ||
262 | for value in python_manifest[key]['files']: | ||
263 | d.appendVar('FILES_' + pypackage, ' ' + value) | ||
264 | |||
265 | # Add cached files | ||
266 | if include_pycs == '1': | ||
267 | for value in python_manifest[key]['cached']: | ||
268 | d.appendVar('FILES_' + pypackage, ' ' + value) | ||
269 | |||
270 | d.setVar('RDEPENDS_' + pypackage, '') | ||
271 | for value in python_manifest[key]['rdepends']: | ||
272 | # Make it work with or without $PN | ||
273 | if '${PN}' in value: | ||
274 | value=value.split('-')[1] | ||
275 | d.appendVar('RDEPENDS_' + pypackage, ' ' + pn + '-' + value) | ||
276 | d.setVar('SUMMARY_' + pypackage, python_manifest[key]['summary']) | ||
277 | |||
278 | # We need to ensure staticdev packages match for files first so we sort in reverse | ||
279 | newpackages.sort(reverse=True) | ||
280 | # Prepending so to avoid python-misc getting everything | ||
281 | packages = newpackages + packages | ||
282 | d.setVar('PACKAGES', ' '.join(packages)) | ||
283 | d.setVar('ALLOW_EMPTY_${PN}-modules', '1') | ||
284 | } | ||
285 | do_split_packages[file-checksums] += "${THISDIR}/python/python3-manifest.json:True" | ||
286 | |||
287 | |||
288 | |||
289 | # Files needed to create a new manifest | ||
290 | SRC_URI += "file://create_manifest3.py file://get_module_deps3.py file://python3-manifest.json" | ||
291 | |||
292 | do_create_manifest() { | ||
293 | |||
294 | # This task should be run with every new release of Python. | ||
295 | # We must ensure that PACKAGECONFIG enables everything when creating | ||
296 | # a new manifest, this is to base our new manifest on a complete | ||
297 | # native python build, containing all dependencies, otherwise the task | ||
298 | # wont be able to find the required files. | ||
299 | # e.g. BerkeleyDB is an optional build dependency so it may or may not | ||
300 | # be present, we must ensure it is. | ||
301 | |||
302 | cd ${WORKDIR} | ||
303 | # This needs to be executed by python-native and NOT by HOST's python | ||
304 | nativepython3 create_manifest3.py | ||
305 | cp python3-manifest.json.new ${THISDIR}/python3/python3-manifest.json | ||
306 | } | ||
307 | |||
308 | # bitbake python -c create_manifest | ||
309 | addtask do_create_manifest | ||
310 | |||
311 | # Make sure we have native python ready when we create a new manifest | ||
312 | do_create_manifest[depends] += "python3:do_prepare_recipe_sysroot" | ||
313 | do_create_manifest[depends] += "python3:do_patch" | ||