tools/git-repo.git/subcmds/sync.py, branch v2.49.3

sync: fix connection error on macOS

2024-11-06T16:33:17+00:00

With a large number of sync workers, the sync process may fail on macOS due to connection errors. The root cause is that multiple workers may attempt to connect to the multiprocessing manager server at the same time when handling the first job. This can lead to connection failures if there are too many pending connections, exceeding the socket listening backlog. Bug: 377538810 Change-Id: I1924d318d076ca3be61d75daa37bfa8d7dc23ed7 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/441541 Tested-by: Josip Sokcevic Commit-Queue: Josip Sokcevic Reviewed-by: Josip Sokcevic

Add REPO_SKIP_SELF_UPDATE check in sync

2024-10-28T17:46:25+00:00

The command _PostRepoFetch will try to self update during repo sync. That is beneficial but adds version uncertainty, fail potential and slow downs in non-interactive scenarios. Conditionally skip the update if env variable REPO_SKIP_SELF_UPDATE is defined. A call to selfupdate works as before, meaning even with the variable set, it will run the update. Change-Id: Iab0ef55dc3d3db3cbf1ba1f506c57fbb58a504c3 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439967 Tested-by: Fredrik de Groot Commit-Queue: Josip Sokcevic Reviewed-by: Josip Sokcevic

sync: reduce multiprocessing serialization overhead

2024-10-23T02:58:45+00:00

Background: - Manifest object is large (for projects like Android) in terms of serialization cost and size (more than 1mb). - Lots of Project objects usually share only a few manifest objects. Before this CL, Project objects were passed to workers via function parameters. Function parameters are pickled separately (in chunk). In other words, manifests are serialized again and again. The major serialization overhead of repo sync was O(manifest_size * projects / chunksize) This CL uses following tricks to reduce serialization overhead. - All projects are pickled in one invocation. Because Project objects share manifests, pickle library remembers which objects are already seen and avoid the serialization cost. - Pass the Project objects to workers at worker intialization time. And pass project index as function parameters instead. The number of workers is much smaller than the number of projects. - Worker init state are shared on Linux (fork based). So it requires zero serialization for Project objects. On Linux (fork based), the serialization overhead is O(projects) --- one int per project On Windows (spawn based), the serialization overhead is O(manifest_size * min(workers, projects)) Moreover, use chunksize=1 to avoid the chance that some workers are idle while other workers still have more than one job in their chunk queue. Using 2.7k projects as the baseline, originally "repo sync" no-op sync takes 31s for fetch and 25s for checkout on my Linux workstation. With this CL, it takes 12s for fetch and 1s for checkout. Bug: b/371638995 Change-Id: Ifa22072ea54eacb4a5c525c050d84de371e87caa Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439921 Tested-by: Kuang-che Wu Reviewed-by: Josip Sokcevic Commit-Queue: Kuang-che Wu

sync: Always use WORKER_BATCH_SIZE

2024-10-07T18:44:19+00:00

With 551285fa35ccd0836513e9cf64ee8d3372e5e3f4, the comment about number of workers no longer stands - dict is shared among multiprocesses and real time information is available. Using 2.7k projects as the baseline, using chunk size of 4 takes close to 5 minutes. A chunk size of 32 takes this down to 40s - a reduction of rougly 8 times which matches the increase. R=gavinmak@google.com Bug: b/371638995 Change-Id: Ida5fd8f7abc44b3b82c02aa0f7f7ae01dff5eb07 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438523 Commit-Queue: Josip Sokcevic Tested-by: Josip Sokcevic Reviewed-by: Gavin Mak

sync: include TARGET_RELEASE when constructing smart sync target.

2024-09-12T16:15:50+00:00

When using the smart sync option, we try to construct the target that was "lunched" from the TARGET_PRODUCT and TARGET_BUILD_VARIANT envvars. However, an android target is now made of three parts, {TARGET_PRODUCT}-{TARGET_RELEASE}-{TARGET_BUILD_VARIANT}. I am leaving the option of creating a target if a TARGET_RELEASE is not specified in case there are other consumers who depend on that option. BUG=b:358101714 TEST=./run_tests TEST=smart sync on android repo and manually inspecting smart_sync_override.xml Change-Id: I556137e33558783a86a0631f29756910b4a93d92 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/436977 Tested-by: Yiwei Zhang Reviewed-by: Yiwei Zhang Commit-Queue: Yiwei Zhang

Add a `--rebase` option to sync command

2024-08-30T09:08:29+00:00

Previously repo would abort a sync if there were published changes not merged upstream. The --rebase option allows the modification of published commits. This is a copy of http://go/grev/369694 with the merge conflicts resolved. Bug: 40014610 Change-Id: Idac8199400346327b530abea33f1ed794e5bb4c2 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/435838 Reviewed-by: Josip Sokcevic Tested-by: Jeroen Dhollander Commit-Queue: Jeroen Dhollander

sync: Fix sorting for nested projects

2024-03-08T17:58:24+00:00

The current logic to create checkout layers doesn't work in all cases. For example, let's assume there are three projects: "foo", "foo/bar" and "foo-bar". Sorting lexicographical order is incorrect as foo-bar would be placed between foo and foo/bar, breaking layering logic. Instead, we split filepaths based using path delimiter (always /) and then use lexicographical sort. BUG=b:325119758 TEST=./run_tests, manual sync on chromiumos repository Change-Id: I76924c3cc6ba2bb860d7a3e48406a6bba8f58c10 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/412338 Tested-by: Josip Sokcevic Commit-Queue: Josip Sokcevic Reviewed-by: George Engelbrecht

sync: introduce --force-checkout

2024-03-07T17:21:51+00:00

In some cases (e.g. in a CI system), it's desirable to be able to instruct repo to force checkout. This flag passes --force flag to `git checkout` operations. Bug: b/327624021 Change-Id: I579edda546fb8147c4e1a267e2605fcf6e597421 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/411518 Commit-Queue: Josip Sokcevic Reviewed-by: Gavin Mak Reviewed-by: George Engelbrecht Tested-by: Josip Sokcevic

sync: Introduce git checkout levels

2024-02-27T17:28:33+00:00

If a repo manifest is updated so that project B is placed within a project A, and if project A had content in new B's location in the old checkout, then repo sync could break depending on checkout order, since B can't be checked out before A. This change introduces checkout levels which enforces right sequence of checkouts while still allowing for parallel checkout. In an example above, A will always be checked out first before B. BUG=b:325119758 TEST=./run_tests, manual sync on ChromeOS repository Change-Id: Ib3b5e4d2639ca56620a1e4c6bf76d7b1ab805250 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/410421 Tested-by: Josip Sokcevic Reviewed-by: Greg Edelston Commit-Queue: Josip Sokcevic Reviewed-by: Gavin Mak

subcmds: sync: Remove deprecated _AUTO_GC

2024-02-20T19:55:15+00:00

Opportunistic cleanup. It looks like this deprecated feature was slated for deletion nearly a year ago. Bug: None Test: ./run_tests Change-Id: I0bd0c0e6acbd1eaee1c0b4945c79038257d22f44 Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/410198 Reviewed-by: Yiwei Zhang Commit-Queue: Greg Edelston Tested-by: Greg Edelston