1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
|
# repo Smart Syncing
Repo normally fetches & syncs manifests from the same URL specified during
`repo init`, and that often fetches the latest revisions of all projects in
the manifest. This flow works well for tracking and developing with the
latest code, but often it's desirable to sync to other points. For example,
to get a local build matching a specific release or build to reproduce bugs
reported by other people.
Repo's sync subcommand has support for fetching manifests from a server over
an XML-RPC connection. The local configuration and network API are defined by
repo, but individual projects have to host their own server for the client to
communicate with.
This process is called "smart syncing" -- instead of blindly fetching the latest
revision of all projects and getting an unknown state to develop against, the
client passes a request to the server and is given a matching manifest that
typically specifies specific commits for every project to fetch a known source
state.
[TOC]
## Manifest Configuration
The manifest specifies the server to communicate with via the
the [`<manifest-server>` element](manifest-format.md#Element-manifest_server)
element. This is how the client knows what service to talk to.
```xml
<manifest-server url="https://example.com/your/manifest/server/url"
helper="repo-remote-helper-name" />
```
If the URL starts with `persistent-`, then the
[`git-remote-persistent-https` helper](https://github.com/git/git/blob/HEAD/contrib/persistent-https/README)
is used to communicate with the server.
### Pluggable Remote Helpers
For custom proxying or authentication, Repo supports pluggable remote helpers.
You can declare a helper binary via the optional `helper` attribute on the
`<manifest-server>` element.
If the `helper` attribute is present in `<manifest-server>`:
1. Repo searches your system `PATH` for the specified helper binary (e.g.,
`repo-remote-sso`). If the helper cannot be found in `PATH`, Repo will raise
a `SmartSyncError` and abort the sync.
2. The helper is executed with the manifest server URL as its first argument:
```bash
<helper-binary-name> <url>
```
3. The helper must output a single-line JSON object on stdout and exit:
* On success, return `status: "ok"` and a loopback proxy URL (including
scheme, e.g., `http://127.0.0.1:999`):
```json
{"status": "ok", "message": "http://127.0.0.1:999"}
```
* On failure, return `status: "error"` and a detailed error message:
```json
{"status": "error", "message": "unauthorized"}
```
4. Repo routes the XML-RPC request through the returned proxy address.
**Timeout Constraint**: The remote helper must complete and exit within 10
seconds. If it hangs or exceeds this limit, Repo will force-terminate (`kill`)
the process and abort the synchronization.
## Credentials
Credentials may be specified directly in typical `username:password`
[URI syntax](https://en.wikipedia.org/wiki/URI#Syntax) in the
`<manifest-server>` element directly in the manifest.
If they are not specified, `repo sync` has `--manifest-server-username=USERNAME`
and `--manifest-server-password=PASSWORD` options.
If those are not used, then repo will look up the host in your
[`~/.netrc`](https://docs.python.org/3/library/netrc.html) database.
When making the connection, cookies matching the host are automatically loaded
from the cookiejar specified in
[Git's `http.cookiefile` setting](https://git-scm.com/docs/git-config#Documentation/git-config.txt-httpcookieFile).
## Manifest Server
Unfortunately, there are no public reference implementations. Google has an
internal one for Android, but it is written using Google's internal systems,
so wouldn't be that helpful as a reference.
That said, the XML-RPC API is pretty simple, so any standard XML-RPC server
example would do. Google's internal server uses Python's
[xmlrpc.server.SimpleXMLRPCDispatcher](https://docs.python.org/3/library/xmlrpc.server.html).
## Network API
The manifest server should implement the following RPC methods.
### GetApprovedManifestV2
> `GetApprovedManifestV2(request: dict) -> dict`
Like [GetApprovedManifest](#GetApprovedManifest), but takes a single struct of
named options so new options can be added without changing the method signature.
Unset options are omitted from the struct. Servers should ignore options they do
not understand.
Request options:
* `branch` (`str`, required): Same as in `GetApprovedManifest`.
* `target` (`str`, optional): Same as in `GetApprovedManifest`.
* `best_effort` (`bool`, optional, default `False`): The client can proceed
without a manifest, so the server may decline the request (e.g. during a
staged rollout or to shed load). Repo sets this for Smart Syncs enabled by
the manifest's `sync-smartsync` default, and falls back to a regular ToT
sync if the request is declined. Servers must not decline requests without
this option for these reasons.
Response fields:
* `success` (`bool`): Whether a manifest was returned.
* `manifest_xml` (`str`): The manifest. Required if `success` is true.
* `error_message` (`str`): Why the request failed or was declined. Repo shows
it to the user like `GetApprovedManifest` errors.
If the server does not implement this method, repo calls `GetApprovedManifest`
instead and drops the options it cannot express.
### GetApprovedManifest
> `GetApprovedManifest(branch: str, target: Optional[str]) -> str`
The meaning of `branch` and `target` is not strictly defined. The server may
interpret them however it wants. The recommended interpretation is that the
`branch` matches the manifest branch, and `target` is an identifier for your
project that matches something users would build.
See the client section below for how repo typically generates these values.
The server will return a manifest or an error. If it's an error, repo will
show the output directly to the user to provide a limited feedback channel.
If the user's request is ambiguous and could match multiple manifests, the
server has to decide whether to pick one automatically (and silently such that
the user won't know there were multiple matches), or return an error and force
the user to be more specific.
### GetManifest
> `GetManifest(tag: str) -> str`
The meaning of `tag` is not strictly defined. Projects are encouraged to use
a system where the tag matches a unique source state.
See the client section below for how repo typically generates these values.
The server will return a manifest or an error. If it's an error, repo will
show the output directly to the user to provide a limited feedback channel.
If the user's request is ambiguous and could match multiple manifests, the
server has to decide whether to pick one automatically (and silently such that
the user won't know there were multiple matches), or return an error and force
the user to be more specific.
## Client Options
Once repo has successfully downloaded the manifest from the server, it saves a
copy into `.repo/manifests/smart_sync_override.xml` so users can examine it.
The next time `repo sync` is run, this file is automatically replaced or removed
based on the current set of options.
With `-l`/`--local-only` and no explicit `-s`/`--smart-sync` or
`-t`/`--smart-tag`, repo does not contact the manifest server. Instead, it keeps
using the existing `smart_sync_override.xml`, if there is one, whether it came
from an explicit `-s`/`--smart-sync` or from the manifest's `sync-smartsync`
default. This way, `repo sync -n` followed by `repo sync -l` ends up on the same
revisions as the equivalent single `repo sync` (e.g. `repo sync -s -n` +
`repo sync -l` matches `repo sync -s`). Pass `--no-smart-sync` to discard it and
use the default manifest instead.
`--no-manifest-update` only skips updating the manifest checkout. It does not
affect Smart Sync: the default Smart Sync from `sync-smartsync` (see below)
still runs, and an existing `smart_sync_override.xml` is replaced or removed as
in a regular sync.
### sync-smartsync (Manifest Attribute)
The manifest can enable Smart Sync by default for all `repo sync` invocations
by setting `sync-smartsync="true"` on the `<default>` element:
```xml
<default sync-smartsync="true" ... />
```
When enabled in the manifest, `repo sync` will automatically perform a Smart
Sync unless explicitly overridden on the command line. Since the user did not
ask for a Smart Sync, any failure to obtain a manifest from the manifest server
(e.g. the server is unreachable, or it returns a manifest that cannot be parsed)
is silently ignored and `repo sync` falls back to a regular ToT sync; pass
`--verbose` to see the reason. Repo marks these requests as `best_effort` (see
[GetApprovedManifestV2](#GetApprovedManifestV2)), so the server may also decline
them, e.g. during a staged rollout. This is only a fallback for the
manifest-driven default: an explicit `-s`/`--smart-sync` or `-t`/`--smart-tag`
still fails the sync, as before. Options that specify an explicit target or
manifest source (such as `-t`/`--smart-tag`, `-m`/`--manifest-name` or
`--superproject-revision`) will also disable the default Smart Sync behavior.
Users can override this manifest default on their workstation with the
`sync.smartsync` user setting; see
[Per-user default](#Per_user-default-sync_smartsync) below.
### --smart-sync / -s
Explicitly enables Smart Sync. Repo will call
`GetApprovedManifestV2({branch[, target]})`, or
`GetApprovedManifest(branch[, target])` if the server does not implement it.
The `branch` is determined by the current manifest branch as specified by
`--manifest-branch=BRANCH` when running `repo init`. If `${SYNC_BRANCH}` is
defined and non-empty, its value is sent instead. This allows selecting a
branch other than the one the checkout tracks, e.g. a server-side (CI) branch
name that builds from the same manifest branch.
The `target` is defined by environment variables in the order below. If none
of them match, then `target` is omitted. These variables were decided as they
match the settings Android build environments automatically setup.
1. `${SYNC_TARGET}`: If defined, the value is used directly.
2. `${TARGET_PRODUCT}-${TARGET_RELEASE}-${TARGET_BUILD_VARIANT}`: If these
variables are all defined, then they are merged with `-` and used.
3. `${TARGET_PRODUCT}-${TARGET_BUILD_VARIANT}`: If these variables are all
defined, then they are merged with `-` and used.
### --no-smart-sync
Explicitly disables Smart Sync, overriding any `sync-smartsync="true"` setting
declared in the manifest `<default>` element.
### Per-user default (sync.smartsync)
A user can decide once, for all their checkouts, whether `repo sync` should
Smart Sync by default by setting the boolean `sync.smartsync` key in their
[per-user repo config](./repoconfig.md#smartsync).
Command line flags (`-s`, `--no-smart-sync`, `-t`) override the user setting,
which in turn overrides the manifest's `sync-smartsync` default. A Smart Sync
enabled this way is treated as an explicit request, so it is subject to the
same restrictions as `-s` and fails the sync if no manifest can be obtained
from the server instead of silently falling back to ToT.
### --smart-tag=TAG / -t
Repo will call `GetManifest(TAG)`.
|
