diff options
author | Richard Purdie <richard.purdie@linuxfoundation.org> | 2021-11-11 11:20:12 +0000 |
---|---|---|
committer | Richard Purdie <richard.purdie@linuxfoundation.org> | 2021-11-11 17:27:20 +0000 |
commit | 711d173d6e0559cb988596095ab45d22a9257dec (patch) | |
tree | 777cd92424eeac645a26a2c979a17d715572f709 /bitbake/lib | |
parent | c277bbaa8bc65645a030e1ff9ecf811d0ca8c20c (diff) | |
download | poky-711d173d6e0559cb988596095ab45d22a9257dec.tar.gz |
bitbake: fetch: Add README on fetcher design constraints
There have been requests to better document the contraints of fetcher
design and operation. This README attempts to start that.
(Bitbake rev: d9cda7835816ecd5a60f0575f6ce832ec9c6aced)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'bitbake/lib')
-rw-r--r-- | bitbake/lib/bb/fetch2/README | 57 |
1 files changed, 57 insertions, 0 deletions
diff --git a/bitbake/lib/bb/fetch2/README b/bitbake/lib/bb/fetch2/README new file mode 100644 index 0000000000..67b787ef47 --- /dev/null +++ b/bitbake/lib/bb/fetch2/README | |||
@@ -0,0 +1,57 @@ | |||
1 | There are expectations of users of the fetcher code. This file attempts to document | ||
2 | some of the constraints that are present. Some are obvious, some are less so. It is | ||
3 | documented in the context of how OE uses it but the API calls are generic. | ||
4 | |||
5 | a) network access for sources is only expected to happen in the do_fetch task. | ||
6 | This is not enforced or tested but is required so that we can: | ||
7 | |||
8 | i) audit the sources used (i.e. for license/manifest reasons) | ||
9 | ii) support offline builds with a suitable cache | ||
10 | iii) allow work to continue even with downtime upstream | ||
11 | iv) allow for changes upstream in incompatible ways | ||
12 | v) allow rebuilding of the software in X years time | ||
13 | |||
14 | b) network access is not expected in do_unpack task. | ||
15 | |||
16 | c) you can take DL_DIR and use it as a mirror for offline builds. | ||
17 | |||
18 | d) access to the network is only made when explicitly configured in recipes | ||
19 | (e.g. use of AUTOREV, or use of git tags which change revision). | ||
20 | |||
21 | e) fetcher output is deterministic (i.e. if you fetch configuration XXX now it | ||
22 | will match in future exactly in a clean build with a new DL_DIR). | ||
23 | One specific pain point example are git tags. They can be replaced and change | ||
24 | so the git fetcher has to resolve them with the network. We use git revisions | ||
25 | where possible to avoid this and ensure determinism. | ||
26 | |||
27 | f) network access is expected to work with the standard linux proxy variables | ||
28 | so that access behind firewalls works (the fetcher sets these in the | ||
29 | environment but only in the do_fetch tasks). | ||
30 | |||
31 | g) access during parsing has to be minimal, a "git ls-remote" for an AUTOREV | ||
32 | git recipe might be ok but you can't expect to checkout a git tree. | ||
33 | |||
34 | h) we need to provide revision information during parsing such that a version | ||
35 | for the recipe can be constructed. | ||
36 | |||
37 | i) versions are expected to be able to increase in a way which sorts allowing | ||
38 | package feeds to operate (see PR server required for git revisions to sort). | ||
39 | |||
40 | j) API to query for possible version upgrades of a url is highly desireable to | ||
41 | allow our automated upgrage code to function (it is implied this does always | ||
42 | have network access). | ||
43 | |||
44 | k) Where fixes or changes to behaviour in the fetcher are made, we ask that | ||
45 | test cases are added (run with "bitbake-selftest bb.tests.fetch"). We do | ||
46 | have fairly extensive test coverage of the fetcher as it is the only way | ||
47 | to track all of its corner cases, it still doesn't give entire coverage | ||
48 | though sadly. | ||
49 | |||
50 | l) If using tools during parse time, they will have to be in ASSUME_PROVIDED | ||
51 | in OE's context as we can't build git-native, then parse a recipe and use | ||
52 | git ls-remote. | ||
53 | |||
54 | Not all fetchers support all features, autorev is optional and doesn't make | ||
55 | sense for some. Upgrade detection means different things in different contexts | ||
56 | too. | ||
57 | |||