path: root/meta/classes/externalsrc.bbclass
diff options
authorMarkus Lehtonen <>2016-03-22 16:01:09 (GMT)
committerRichard Purdie <>2016-03-28 14:55:50 (GMT)
commitbbfc2100515fc4b972fd012ce22c20863212c0c4 (patch)
tree4143994db573ec39267ba7c5fb77c6a844bffe37 /meta/classes/externalsrc.bbclass
parent9ee403b31067895695f0143b716187d7262c560c (diff)
externalsrc: do not use do_configure[nostamp] for git srctrees
Be a bit more intelligent than mindlessly re-compiling every time. Instead of always using 'nostamp' flag for do_compile run a python function to get a list of files to add as 'file-checksums' flag. The intention is to only re-run do_compile if something in the source tree content changes. This python function, srctree_hash_files(), works differently, depending if the source tree is a git repository clone or not. If the source tree is a git repository, the function runs 'git add .' and 'git write tree' to get a hash of the working tree and writes this hash into a file under the .git directory. This file containing the hash is then returned as the file for the task to depend on. Hash is changed if any changes are made in the source tree causing the task to be re-run. A trick is used to parse the recipe every time so that the hash file gets updated. If the source tree is not a git repository behaviour remains the same. In this case srctree_hash_files() currently sets the 'nostamp' flag for do_compile causing it to be re-run every time. This method of tracking changes source tree changes to determine if re-build is needed does not work perofectly, though. Many packages are built under ${S} which effectively changes the source tree causing some unwanted re-compilations. However, if do_compile of the recipe does not produce new/different artefacts on every run (as commonly is and should be the case) the re-compilation loop stops. Thus, you should usually see only one re-compilation (if any) after which the source tree is "stabilized" and no more re-compilations happen. [YOCTO #8853] (From OE-Core rev: a26becdf981b35d7ef8524f9e65c25a74b842f1d) Signed-off-by: Markus Lehtonen <> Signed-off-by: Richard Purdie <>
Diffstat (limited to 'meta/classes/externalsrc.bbclass')
1 files changed, 29 insertions, 2 deletions
diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index d3cede2..f983359 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -85,8 +85,10 @@ python () {
85 d.prependVarFlag('do_compile', 'prefuncs', "externalsrc_compile_prefunc ") 85 d.prependVarFlag('do_compile', 'prefuncs', "externalsrc_compile_prefunc ")
86 d.prependVarFlag('do_configure', 'prefuncs', "externalsrc_configure_prefunc ") 86 d.prependVarFlag('do_configure', 'prefuncs', "externalsrc_configure_prefunc ")
87 87
88 # Ensure compilation happens every time 88 # Force the recipe to be always re-parsed so that the file_checksums
89 d.setVarFlag('do_compile', 'nostamp', '1') 89 # function is run every time
90 d.setVar('BB_DONT_CACHE', '1')
91 d.setVarFlag('do_compile', 'file-checksums', '${@srctree_hash_files(d)}')
90 92
91 # We don't want the workdir to go away 93 # We don't want the workdir to go away
92 d.appendVar('RM_WORK_EXCLUDE', ' ' + d.getVar('PN', True)) 94 d.appendVar('RM_WORK_EXCLUDE', ' ' + d.getVar('PN', True))
@@ -124,3 +126,28 @@ python externalsrc_compile_prefunc() {
124 # Make it obvious that this is happening, since forgetting about it could lead to much confusion 126 # Make it obvious that this is happening, since forgetting about it could lead to much confusion
125 bb.plain('NOTE: %s: compiling from external source tree %s' % (d.getVar('PN', True), d.getVar('EXTERNALSRC', True))) 127 bb.plain('NOTE: %s: compiling from external source tree %s' % (d.getVar('PN', True), d.getVar('EXTERNALSRC', True)))
126} 128}
130def srctree_hash_files(d):
131 import shutil
132 import subprocess
134 s_dir = d.getVar('EXTERNALSRC', True)
135 git_dir = os.path.join(s_dir, '.git')
136 oe_index_file = os.path.join(git_dir, 'oe-devtool-index')
137 oe_hash_file = os.path.join(git_dir, 'oe-devtool-tree-sha1')
139 ret = " "
140 if os.path.exists(git_dir):
141 # Clone index
142 shutil.copy2(os.path.join(git_dir, 'index'), oe_index_file)
143 # Update our custom index
144 env = os.environ.copy()
145 env['GIT_INDEX_FILE'] = oe_index_file
146 subprocess.check_output(['git', 'add', '.'], cwd=s_dir, env=env)
147 sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env)
148 with open(oe_hash_file, 'w') as fobj:
149 fobj.write(sha1)
150 ret = oe_hash_file + ':True'
151 else:
152 d.setVarFlag('do_compile', 'nostamp', '1')
153 return ret