tools/git-repo.git, branch v2.13

forall: greatly speed up processing overhead

2021-03-01T15:58:06+00:00

With the recent commit 0501b29e7ae072e0b10ea9ddd913ec6d5975f690 ("status: Use multiprocessing for `repo status -j` instead of threading"), the limitation with project serialization no longer applies. It turns out that ad-hoc logic is expensive. In the CrOS checkout (~1000 projects w/8 jobs by default), it adds about ~7sec overhead to all invocations. With a fast nop run: time repo forall -j8 -c true This goes from ~11sec to ~4sec -- more than 50% speedup. Change-Id: Ie6bcccd21eef20440692751b7ebd36c890d5bbcc Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298724 Reviewed-by: Michael Mortensen Tested-by: Mike Frysinger

forall: simplify arg passing to worker children

2021-03-01T15:58:06+00:00

The ProjectArgs function can be inlined which simplifies it quite a bit. We shouldn't need the custom exception handling here either. This also makes the next commit easier to review. Change-Id: If3be04f58c302c36a0f20b99de0f67e78beac141 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298723 Reviewed-by: Michael Mortensen Tested-by: Mike Frysinger

forall: move nested func out to the class

2021-03-01T15:57:32+00:00

This is in preparation for simplifying the jobs support. The nested function is referenced in the options object which can't be pickled, so pull it out into a static method instead. Change-Id: I01d3c4eaabcb8b8775ddf22312a6e142c84cb77d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298722 Reviewed-by: Michael Mortensen Tested-by: Mike Frysinger

download: handle shared projects a bit better

2021-03-01T15:57:17+00:00

If a manifest checksout a project multiple times, repo download isn't able to accurately pick the right project. We were just picking the first result which could be a bit random for the user. If we hit that situation, check if the cwd is one of the projects, and if it isn't, we emit an error and tell the user it's an ambiguous request. Bug: https://crbug.com/gerrit/13070 Change-Id: Id1059b81330229126b48c7312569b37504808383 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298702 Reviewed-by: Michael Mortensen Tested-by: Mike Frysinger

project: skip clone bundles when we've already initialized the object dir

2021-03-01T15:57:12+00:00

The clone bundle logic assumes there is a one-to-one mapping between the projects/ and project-objects/ trees. When using shared projects (where we checkout different branches from the same project), this would lead us to fetching the same clone bundle multiple times. Automatically skip the clone bundle logic if the project-objects/ dir already exists. Bug: https://crbug.com/gerrit/10993 Change-Id: I82c6fa1faf8605fd56c104fcea2a43dd4eecbce4 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298682 Reviewed-by: Michael Mortensen Tested-by: Mike Frysinger

git_refs: fix crash with binary . files in .git/refs/

2021-02-28T16:07:24+00:00

On macOS, the Finder app likes to poop .DS_Store files in every path that the user browses. If the user pokes around the .git/ tree, it could generate a .DS_Store file in there too. When repo goes to read all the local refs, it tries to decode this binary file as UTF-8 and subsequently crashes. Since paths that begin with . are not valid refs, ignore them like we already do with paths that end in .lock. Also bump the check up to ignore dirs that match since that follows the git rules: they apply to any component in its path, not just the final path (name). We don't implement the full valid ref algorithm that git employs as it's a bit complicated, and we only really need to focus on what will practically show up locally. Bug: https://crbug.com/gerrit/14162 Change-Id: I6519f990e33cc58a72fcb00c0f983ad3285ace3d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298662 Tested-by: Mike Frysinger Reviewed-by: Michael Mortensen

git_command: pass GIT_DIR on Windows with /

2021-02-28T16:07:20+00:00

When using Git under Windows, it seems that Git doesn't always parse GIT_DIR correctly when it uses the Windows \ form, but does when it uses / only. For example, when using worktrees: $ GIT_DIR='C:\Users\vapier\Desktop\repo\breakpad\tools\test\.git' git worktree list fatal: not a git repository: ..\..\.repo\worktrees\linux-syscall-support.git\worktrees\test $ GIT_DIR='C:/Users/vapier/Desktop/repo/breakpad/tools/test/.git' git worktree list C:/Users/vapier/Desktop/repo/breakpad/.repo/worktrees/linux-syscall-support.git fd00dbbd0c06 (detached HEAD) ..\..\..\..\..\src\src\third_party\lss\.git fd00dbbd0c06 (detached HEAD) ..\..\..\..\..\tools\test\.git fd00dbbd0c06 (detached HEAD) Change-Id: I666c03ae845ecb55d7f9800731ea6987d3e7f401 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298622 Reviewed-by: Michael Mortensen Tested-by: Mike Frysinger

manifest: validate project name & path and include name attributes

2021-02-28T16:07:12+00:00

These attribute values are used to construct local filesystem paths, so apply the existing filesystem checks to them. Bug: https://crbug.com/gerrit/14156 Change-Id: Ibcceecd60fa74f0eb97cd9ed1a9792e139534ed4 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298443 Reviewed-by: Michael Mortensen Tested-by: Mike Frysinger

manifest: refactor the filesystem checking logic for more reuse

2021-02-28T16:07:12+00:00

This function is currently written with copyfile & linkfile in mind. Generalize the logic & function arguments slightly so we can reuse in more places that make sense. This changes the validation logic slightly too in that we no longer allow "." for the dest attribute with copyfile & linkfile, nor for the src attribute with copyfile. We already rejected those later on when checking against the active filesystem, but now we reject them a little sooner when parsing. The empty path check isn't a new requirement exactly -- repo used to crash on it, so it was effectively blocked, but now we diagnosis it. Bug: https://crbug.com/gerrit/14156 Change-Id: I0fdb42a3da60ed149ff1997c5dd4b85da70eec3d Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/298442 Reviewed-by: Michael Mortensen Tested-by: Mike Frysinger

checkout: add --jobs support

2021-02-27T19:56:24+00:00

Use multiprocessing to run in parallel. When operating on multiple projects, this can speed things up. Across 1000 repos, it goes from ~9sec to ~5sec with the default -j8. Change-Id: Ida6dd565db78ff7bac0ecb25d2805e8a1bf78048 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297982 Reviewed-by: Chris Mcdonald Tested-by: Mike Frysinger