| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We have been seeing obscure failures in devtool, particularly on newer
autobuilder workers where it appears the cache is assumed to be valid
when it shouldn't be.
We're using the 'seconds' granulation mtime field which is not really
a good way of telling if a file has changed. We can switch to the "ns"
version which is better however also add in inode number and size as
precautions. We already have all this data and tuples are fast so there
isn't really any cost to do so.
This hopefully fixes [YOCTO #15318].
(Bitbake rev: d9e5d313c79500e3c70ab9c3239b6b2180194f67)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With the flush in serverlog() removed and a memory resident bitbake with a
60s timeout, the following could fail in strange ways:
rm bitbake-cookerdaemon.log
bitbake-layers add-layer ../meta-virtualization/
bitbake-layers add-layer ../meta-openembedded/meta-oe/
bitbake -m
specifically that it might error adding meta-oe with an error related to meta-virt.
This clearly shows that whilst bblayers.conf was modified, bitbake was not
recognising that. This would fit with the random autobuilder issues seen when
the serverlog flush() call was removed.
The issue appears to be that you have no way to "sync()" the inotify events with
the command stream coming over the socket. There is no way to know if there are
changes in the IO queue which bitbake needs to wait for before proceeding with
the next command.
I did experiment with os.sync() and fsync on the inotify fd, however nothing
addressed the issue. Since it is extremely important we have accurate cache data,
the only realistic thing to do is to switch to stat() calls and check mtime.
For bitbake commands, this is straightforward since we can revalidate the cache
upon new connections/commands. For tinfoil this is problematic and we need to
introduce and explict command "revalidateCaches" that the code can use to force
bitbake to re-check it's cache validity. I've exposed this through tinfoil with
a new "modified_files" function.
So, this patch:
a) drops inotify support within bitbake's cooker/server and switch to using mtime
b) requires a new function call in tinfoil when metadata has been modified
(Bitbake rev: da3ec3801bdb80180b3f1ac24edb27a698415ff7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
BB_GLOBAL_PYMODULES
For many years OE-Core has injected it's own python modules into the
python namespace using an immediate expansion of a variable in base.bbclass.
It also added all entries from BBPATH to sys.path.
We really need this to become a first class citizen of the langauge, this new
addpylib directive allows that. Usage is of the form:
addpylib <directory> <namespace>
The namespace is imported and if there is an attribute BBIMPORT, that
list of names is iterated and imported too.
This mirrors what OE-Core has done for a long time with one difference in
implmentation, sys.path is only appended to. This means later imported
namespaces can't overwrite an earlier one and can't overwrite the main
python module space. In practice we've never done that and it isn't
something we should encourage or support.
The new directive is only applied for .conf files and not other filetypes
as it only makes sense in that context. It is also only allowed in the
"base configuration" context of cookerdata since adding it at the recipe
level wouldn't work properly due to the way it changes the global namespace.
At the same time, move the list of modules to place in the global namespace
into a BB_GLOBAL_PYMODULES variable. It is intended that only the core layer
should touch this and it is meant to be a very small list, usually os and sys.
BB_GLOBAL_PYMODULES is expected to be set before the first addpylib directive.
Layers adding a lib directory will now need to use this directive as BBPATH
is not going to be added automatically by OE-Core in future. The directives are
immediate operations so it does make modules available sooner than the current
OE-Core approach.
The new code appends to sys.path rather than prepends as core did, as
overwriting python standard library modules would be a bad idea and naturally
encouraging people to collaborate around our own core modules is desireable.
(Bitbake rev: afb8478d3853f6edf3669b93588314627d617d6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
We're still seeing issues with unclosed asyncio event loops. At the
init site, make sure any existing one is closed first to try and avoid
this.
(Bitbake rev: 78dee3c03c75a27531fcff26f9298fce2519bdde)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.
Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.
[RP: Small fix to ensure bb.debug calls bbdebug()]
(Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.
(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.
The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.
The COPYING file is replaced with LICENSE.X files which contain the full
license texts.
(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
The resolve_file() calls mark_dependency(), so the one which calls
resolve_file() doesn't need call mark_dependency() again.
(Bitbake rev: 4682571107323a39b42cd9ec8ee67419e7f15acc)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
| |
Add a proper function for clearing the mtime cache. Clean up the inotify
event overflow case to err on the side of caution and clear any potentially
now out of sync caches.
(Bitbake rev: ec60459fe2ba16966544eebff43b061abb7ed3ba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
| |
Trying to expand a variable which isn't a string doesn't make sense.
(Bitbake rev: 62367cca1f1793eb9827406bcdd5980fdeb80a60)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
If a file no longer exists, drop it from the cache silently instead of
generating a traceback. This was visible in some cases when a recipe was
deleted when bitbake was resident in memory.
(Bitbake rev: fe105b9042bdac4afd9f38fcf92bfdc2c04ec23f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We've had a long-standing bug where a legitimate error reading a file (IOError
or OSError) is always suppressed as though it was a 'file not found' case. As
a concrete example, if you do a `chmod 000 conf/local.conf`, it'll silently
not parse local.conf, rather than erroring to let the user know about the
problem.
Fix this by handling the ENOENT case specifically.
(Bitbake rev: e691312a3add222b04e7b2f52f8df6abcb9068bf)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.
This patch was mostly made using the command:
sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`
(Bitbake rev: 659ef95c9b8aced3c4ded81c48bcc0fbde4d429f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Memory resident bitbake has one current flaw, changes in the base configuration
are not noticed by bitbake. The parsing cache is also refreshed on each invocation
of bitbake (although the mtime cache is not cleared so its pointless).
This change adds in pyinotify support and adds two different watchers, one
for the base configuration and one for the parsed recipes.
Changes in the latter will trigger a reparse (and an update of the mtime cache).
The former will trigger a complete reload of the configuration.
Note that this code will also correctly handle creation of new configuration files
since the __depends and __base_depends variables already track these for cache
correctness purposes.
We could be a little more clever about parsing cache invalidation, right now we just
invalidate the whole thing and recheck. For now, its better than what we have and doesn't
seem to perform that badly though.
For education and QA purposes I can document a workflow that illustrates this:
$ source oe-init-build-env-memres
$ time bitbake bash
[base configuration is loaded, recipes are parsed, bash builds]
$ time bitbake bash
[command returns quickly since all caches are valid]
$ touch ../meta/classes/gettext.bbclass
$ time bitbake bash
[reparse is triggered, time is longer than above]
$ echo 'FOO = "1"' >> conf/local.conf
$ time bitbake bash
[reparse is triggered, but with a base configuration reload too]
As far as changes go, I like this one a lot, it makes memory resident bitbake
truly usable and may be the tweak we need to make it the default.
The new pyinotify dependency is covered in the previous commit.
(Bitbake rev: 0557d03c170fba8d7efe82be1b9641d0eb229213)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
In the depths of time we were rather confused about naming. bb files
are recipes, the event to skip parsing them should be SkipRecipe,
not SkipPackage. This changes bitbake to use the better name but
leaves the other around for now. We can therefore start removing
references to it from the metadata.
(Bitbake rev: 98d9e6e0f514a7cb7da1d99bf4bd5602b89426d6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The file inheritance logging has been here since the dawn of time. It
duplicates output many times over and logs to debug level 2. When running
with the debug option, its understandable the user may want to see the
paths of files included in the build.
These changes remove pointless/duplicate output and print the
include paths clearly at debug level one in a form which
users should be able to more easily parse.
(Bitbake rev: 633a56987df639295fe9128418da5634050063e5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
It doesn't really make sense to set PN from .conf files, for example.
More concretely, this avoids the config hash changing unnecessarily
within Hob due to PN effectively changing (since bblayers.conf is
parsed first and then .hob.conf).
(Bitbake rev: 22e03ef7ac9bb6b7245250347ae9c10c19f1d74e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently bitbake only adds files to its dependency list if they exist.
If you add 'include foo.inc' to your recipe and the file doesn't exist,
then later you add the file, the cache will not be invalidated.
This leads to another bug which is that if files don't exist and then
you add them and they should be found first due to BBPATH, again the
cache won't invalidate.
This patch adds in tracking of files we check for the existence of so
that if they are added later, the cache correctly invalidates. This
necessitated a new version of bb.utils.which which returns a list of
files tested for.
The patch also adds in checks for duplicate file includes and for now
prints a warning about this. That will likely become a fatal error at
some point since its never usually desired to include a file twice.
The same issue is also fixed for class inheritance. Now when a class
is added which would be found in the usual search path, it will cause
the cache to be invalidated.
Unfortunately this is old code in bitbake and the patch isn't the
neatest since we have to work within that framework.
[YOCTO #5611]
[YOCTO #4425]
(Bitbake rev: 78d285871e4b8c54ccc4602d571e85f922e37ccd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* resolve_file was behaving different when relative and absolute
paths were passed to it
* include relative-path/non-existent-file.inc
works correctly resolve_file throws IOError, BBHandler.py:handle()
doesn't catch it, ConfHandler.py:include() catches IOError and shows:
DEBUG: CONF file 'relative-path/non-existent-file.inc' not found
* include /absolute-path/non-existent-file.inc
was failing, because resolve_file just returns fn,
BBHandler.py:handle() calls bb.parse.mark_dependency(d, abs_fn)
which throws:
OSError: [Errno 2] No such file or directory: '/absolute-path/non-existent-file.inc'
and parsing fails.
Ad isfile() test for absolute fn and throw IOError to make
resolve_file behavior consistent for both paths.
* I know we had some issues with -b relative-path-to-recipe.bb and
absolute path, so consider this patch only as RFC and documentation of
this problem
* Catch OSError too in ConfHandler.py:include() e.g. in case the file exists, but user
cannot read it or something like that.
(Bitbake rev: b0bbd89a4f0b98fa1ab28b8e0526cd9ddb76fa57)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
bitbake -e
This code adds inclusion history to bitbake -e output, showing
which files were included, in what order. This doesn't completely
resolve timing questions, because it doesn't show you which lines
of a file were processed before or after a given include, but it
does let you figure out what the path was by which a particular
file ended up in your build at all.
How it works: data_smart acquires a .history member, which is an
IncludeHistory; this represents the inclusion of a file and all its
inclusions, recursively. It provides methods for including files,
for finishing inclusion (done as an __exit__), and for
dumping the whole tree.
The parser is modified to run includes inside a with() to push
and pop the include filename.
RP Modifications:
a) Split Include and Variable tracking
b) Replace deepcopy usage with dedicated copy function
c) Simplify some variable and usage
(Bitbake rev: b2dda721262da8abb7dc32d019e18fbc32ed8860)
Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
The data returned by get_file_depends() may me used in contexts like
checksums where order is important. The current usage of sets means
that some of the checksums can change in circumstances they should not.
This patch changes to use lists, thereby removing the problem.
(Bitbake rev: a44285fc4109236ab89f7aad0a1fc9220eec19b6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
Added new variable BBINCLUDED indicating the file dependency
information. It exposes the internal variable '__base_depends' and
'__depends'.
(Bitbake rev: af524a656fce32d01687481b86c31bef00eb9fc3)
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
|
|
|
|
| |
Ensure that a file and line number are reported for ParseError where
possible. This helps particularly in the case of inherit and require
which previously did not report either of these upon failure.
(Bitbake rev: f588ba69622a2df35417ced184e56c79ac1b40d5)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
| |
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
|
|
|
|
|
| |
(Bitbake rev: bcabe2dfb587042e139890329ff52d9bb9201cf4)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
| |
bitbake uptream
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
|
| |
(Bitbake rev: 2bc893dc407bf608cac7e12673161b75b712da83)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We use a custom Logger subclass for our loggers
This logger provides:
- 'debug' method which accepts a debug level
- 'plain' method which bypasses log formatting
- 'verbose' method which is more detail than info, but less than debug
(Bitbake rev: 3b2c1fe5ca56daebb24073a9dd45723d3efd2a8d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
| |
reducing memory consumption
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
| |
Includes functionality to find out what changes between two different singature data dumps.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
|
|
|
|
| |
to make updating depends easier/more intuitive/eventually faster
(Bitbake rev: f7c69462b8ba726861898817cc5b13174c78e35a)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
|
| |
(Bitbake rev: 1cfcbee014478eb129ed382c13622317412b036e)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
|
| |
(Bitbake rev: d39ab776e7ceaefc8361150151cf0892dcb70d9c)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
|
| |
(Bitbake rev: b4983cc7bfe5300c1d2904f34fb03d4fb2f41d90)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
|
| |
(Bitbake rev: 03a6c26f8da226f442c3cab557e4733f7cd6eeac)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
|
| |
(Bitbake rev: 3cd06ed487ee0617892f154cff461379a323a1fb)
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
|
|
| |
Do not attempt to open the file in the resolve_file method
(a lot like bb.which... maybe bb.which can be used). This way
we don't need to open/close a file which we have already parsed.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
|
|
|
| |
Evaluate the statements after having parsed one file. This is
referred to as "entwirren" and we can remove the direct evaluation
and postpone a bit, in the future we can use a cached copy instead
of parsing the original.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
|
| |
(Bitbake rev: fda0707d772e0964a0185d4ec4d016522f6972f3)
Signed-off-by: Chris Larson <clarson@mvista.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
|
| |
(Bitbake rev: 0a2bb3dd790e3e40867195f14f4e174f98f8a47c)
Signed-off-by: Chris Larson <clarson@mvista.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
|
|
|
| |
When parsing we will collect a number of statements
that can be evaluated...The plan is to be evaluate
things twice (old+new) and then compare the result,
it should be the same.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
| |
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
|
|
|
|
| |
shell reparse failures
git-svn-id: https://svn.o-hand.com/repos/poky/trunk@5196 311d38ba-8fff-0310-9ca6-ca027cbcb966
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* File licence headers were sanitised causing most of the diff.
* cooker.py was created from bin/bitbake.
* cvs fetcher port option was added
* The -f force option was fixed to work correctly
* Multiple entries in rrecrdeps are now handled correctly
(allows adding do_deploy to image depends)
git-svn-id: https://svn.o-hand.com/repos/poky/trunk@1129 311d38ba-8fff-0310-9ca6-ca027cbcb966
|
|
|
|
| |
git-svn-id: https://svn.o-hand.com/repos/poky/trunk@863 311d38ba-8fff-0310-9ca6-ca027cbcb966
|
|
|
|
|
|
| |
caching speedups
git-svn-id: https://svn.o-hand.com/repos/poky/trunk@371 311d38ba-8fff-0310-9ca6-ca027cbcb966
|
|
|
|
| |
git-svn-id: https://svn.o-hand.com/repos/poky/trunk@262 311d38ba-8fff-0310-9ca6-ca027cbcb966
|
|
git-svn-id: https://svn.o-hand.com/repos/poky@2 311d38ba-8fff-0310-9ca6-ca027cbcb966
|