tools/git-repo.git/subcmds, branch v2.50

sync: Delete symlinks relative to client topdir

2024-12-16T18:23:40+00:00

If repo sync is invoked outside the repo root, and the latest manifest removes symlinks, repo incorrectly tries to remove symlink - it starts from `cwd` instead of the repo root. Bug: b/113935847 Bug: 40010423 Change-Id: Ia50ea70a376e38c94389880f020c80da3c3f453c Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/445901 Tested-by: Josip Sokcevic Reviewed-by: Gavin Mak

branches: Escape percent signs in branch names

2024-12-03T19:02:20+00:00

If a branch name contains a percent sign, it will be interpreted as a placeholder and color.py will fail to format it. To avoid this, escape the percent signs prior to calling Coloring method. Bug: b/379090488 Change-Id: Id019c776bbf8cbed5c101f2773606f1d32c9e057 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/443801 Reviewed-by: Scott Lee Tested-by: Josip Sokcevic Commit-Queue: Josip Sokcevic

sync: Do not fail to sync a manifest with no projects

2024-11-26T10:16:03+00:00

Since commit 454fdaf1191c87e5c770ab865a911e10e600e178 (v2.48), syncing a manifest without any projects would result in: Repo command failed: RepoUnhandledExceptionError Number of processes must be at least 1 Bug: 377546300 Change-Id: Iaa2f6a3ac64542ad65a19c0eef449f53c09cae67 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/443442 Reviewed-by: Erik Elmeke Reviewed-by: Josip Sokcevic Commit-Queue: Peter Kjellerstedt Tested-by: Peter Kjellerstedt

sync: fix connection error on macOS

2024-11-06T16:33:17+00:00

With a large number of sync workers, the sync process may fail on macOS due to connection errors. The root cause is that multiple workers may attempt to connect to the multiprocessing manager server at the same time when handling the first job. This can lead to connection failures if there are too many pending connections, exceeding the socket listening backlog. Bug: 377538810 Change-Id: I1924d318d076ca3be61d75daa37bfa8d7dc23ed7 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/441541 Tested-by: Josip Sokcevic Commit-Queue: Josip Sokcevic Reviewed-by: Josip Sokcevic

upload: Return correct tuple values in _ProcessResults

2024-10-31T21:18:53+00:00

Incorrect tuple values were returned with http://go/grev/440221 - instead of returning (Project, ReviewableBranch), _ProcessResults was returning (int, ReviewableBranch). R=jojwang@google.com Bug: 376731172 Change-Id: I75205f42fd23f5ee6bd8d0c15b18066189b42bd9 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/441121 Reviewed-by: Sam Saccone Commit-Queue: Josip Sokcevic Tested-by: Josip Sokcevic

forall: Fix returning results early

2024-10-30T16:11:04+00:00

rc should be returned only after all results are processed. R=jojwang@google.com Bug: b/376454189 Change-Id: I8200b9954240dd3e8e9f2ab82494779a3cb38627 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440901 Tested-by: Josip Sokcevic Commit-Queue: Josip Sokcevic Reviewed-by: Joanna Wang

Use full name of the revision when checking dest-branch

2024-10-28T23:47:08+00:00

The manifest usually doesn't sepecify the revision with the full name (e.g. refs/heads/REV). However, when checking if the name of the merge branch, full name is used on the merge branch. The CL use full name of revision when comparing it with the merge branch. Bug: b/370919047 Test: repo upload on a project with `dest-branch` set Change-Id: Ib6fa2f7246beb5bae0a26a70048a7ac03b6c5a2f Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438401 Reviewed-by: Josip Sokcevic Tested-by: Joe Hsu Commit-Queue: Josip Sokcevic

Add REPO_SKIP_SELF_UPDATE check in sync

2024-10-28T17:46:25+00:00

The command _PostRepoFetch will try to self update during repo sync. That is beneficial but adds version uncertainty, fail potential and slow downs in non-interactive scenarios. Conditionally skip the update if env variable REPO_SKIP_SELF_UPDATE is defined. A call to selfupdate works as before, meaning even with the variable set, it will run the update. Change-Id: Iab0ef55dc3d3db3cbf1ba1f506c57fbb58a504c3 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439967 Tested-by: Fredrik de Groot Commit-Queue: Josip Sokcevic Reviewed-by: Josip Sokcevic

subcmds: reduce multiprocessing serialization overhead

2024-10-23T23:34:34+00:00

Follow the same approach as 39ffd9977e to reduce serialization overhead. Below benchmarks are tested with 2.7k projects on my workstation (warm cache). git tracing is disabled for benchmark. (seconds) | v2.48 | v2.48 | this CL | this CL | | -j32 | | -j32 ----------------------------------------------------------- with clean tree state: branches (none) | 5.6 | 5.9 | 1.0 | 0.9 status (clean) | 21.3 | 9.4 | 19.4 | 4.7 diff (none) | 7.6 | 7.2 | 5.7 | 2.2 prune (none) | 5.7 | 6.1 | 1.3 | 1.2 abandon (none) | 19.4 | 18.6 | 0.9 | 0.8 upload (none) | 19.7 | 18.7 | 0.9 | 0.8 forall -c true | 7.5 | 7.6 | 0.6 | 0.6 forall -c "git log -1" | 11.3 | 11.1 | 0.6 | 0.6 with branches: start BRANCH --all | 21.9 | 20.3 | 13.6 | 2.6 checkout BRANCH | 29.1 | 27.8 | 1.1 | 1.0 branches (2) | 28.0 | 28.6 | 1.5 | 1.3 abandon BRANCH | 29.2 | 27.5 | 9.7 | 2.2 Bug: b/371638995 Change-Id: I53989a3d1e43063587b3f52f852b1c2c56b49412 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440221 Reviewed-by: Josip Sokcevic Tested-by: Kuang-che Wu Commit-Queue: Kuang-che Wu

sync: reduce multiprocessing serialization overhead

2024-10-23T02:58:45+00:00

Background: - Manifest object is large (for projects like Android) in terms of serialization cost and size (more than 1mb). - Lots of Project objects usually share only a few manifest objects. Before this CL, Project objects were passed to workers via function parameters. Function parameters are pickled separately (in chunk). In other words, manifests are serialized again and again. The major serialization overhead of repo sync was O(manifest_size * projects / chunksize) This CL uses following tricks to reduce serialization overhead. - All projects are pickled in one invocation. Because Project objects share manifests, pickle library remembers which objects are already seen and avoid the serialization cost. - Pass the Project objects to workers at worker intialization time. And pass project index as function parameters instead. The number of workers is much smaller than the number of projects. - Worker init state are shared on Linux (fork based). So it requires zero serialization for Project objects. On Linux (fork based), the serialization overhead is O(projects) --- one int per project On Windows (spawn based), the serialization overhead is O(manifest_size * min(workers, projects)) Moreover, use chunksize=1 to avoid the chance that some workers are idle while other workers still have more than one job in their chunk queue. Using 2.7k projects as the baseline, originally "repo sync" no-op sync takes 31s for fetch and 25s for checkout on my Linux workstation. With this CL, it takes 12s for fetch and 1s for checkout. Bug: b/371638995 Change-Id: Ifa22072ea54eacb4a5c525c050d84de371e87caa Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439921 Tested-by: Kuang-che Wu Reviewed-by: Josip Sokcevic Commit-Queue: Kuang-che Wu