diff options
author | Vu Tran <vu.tran@windriver.com> | 2014-07-10 13:48:28 -0400 |
---|---|---|
committer | Bruce Ashfield <bruce.ashfield@windriver.com> | 2014-07-17 00:12:13 -0400 |
commit | a06e28ced8f622dbe6d6fbd9bea20b215d8ac938 (patch) | |
tree | 680e3e7e262748a976b22648ed075e4217a4718f /meta-openstack/Documentation | |
parent | 7550843c5be8063ce6cc147bf3efa19d10c92660 (diff) | |
download | meta-cloud-services-a06e28ced8f622dbe6d6fbd9bea20b215d8ac938.tar.gz |
rally add readme doc
Signed-off-by: Vu Tran <vu.tran@windriver.com>
Diffstat (limited to 'meta-openstack/Documentation')
-rw-r--r-- | meta-openstack/Documentation/README.rally | 335 |
1 files changed, 335 insertions, 0 deletions
diff --git a/meta-openstack/Documentation/README.rally b/meta-openstack/Documentation/README.rally new file mode 100644 index 0000000..352ef25 --- /dev/null +++ b/meta-openstack/Documentation/README.rally | |||
@@ -0,0 +1,335 @@ | |||
1 | Summary | ||
2 | ======= | ||
3 | |||
4 | This document is not intended to provide detail of how Rally in general | ||
5 | works, but rather it highlights how Rally is integrated/configured into | ||
6 | meta-cloud-services and also describes how Rally is tested to ensure that | ||
7 | Rally Verification and Benchmarking components are working correctly. | ||
8 | |||
9 | |||
10 | Rally Overview | ||
11 | ============== | ||
12 | |||
13 | Rally - OpenStack benchmarking at a scale - is intended to provide the | ||
14 | community with a benchmarking tool that is capable of performing "specific", | ||
15 | "complicated" and "reproducible" test cases on "real deployment" scenarios. | ||
16 | |||
17 | Rally has 4 main components: | ||
18 | |||
19 | * Server Providers - provides virtual servers | ||
20 | * Deploy Engines - deploys OpenStack cloud on servers that are presented by Server Providers | ||
21 | * Verification - runs tempest | ||
22 | * Benchmarking - allows to write parameterized benchmark scenarios & run them against the cloud | ||
23 | |||
24 | |||
25 | Rally Deployment | ||
26 | ================ | ||
27 | |||
28 | Rally is configured to use existing deployment (by using deployment configuration file | ||
29 | /etc/rally/deployments/existing.json) and therefore Rally components "Server Providers" | ||
30 | and "Deploy Engines" are not used heavily. | ||
31 | |||
32 | |||
33 | Rally Verification | ||
34 | ================== | ||
35 | |||
36 | By default, Rally verification performs the following steps: | ||
37 | |||
38 | * git clone tempest source from upstream | ||
39 | * setup virtualenv for this tempest | ||
40 | * setup testr environment with virtualenv created above | ||
41 | * create tempest.conf for this tempest | ||
42 | * use testr and subunit.run module to run tempest | ||
43 | |||
44 | However, meta-cloud-services already includes tempest which is also configured/modified | ||
45 | to have low failure/error testcases, therefore it's desired to use this tempest | ||
46 | (without using virtualenv) instead of letting Rally to download tempest and running it | ||
47 | on virtualenv. | ||
48 | |||
49 | The option "existing_tempest_config" in /etc/rally/rally.conf can be used to configure | ||
50 | Rally to either use the existing tempest or to download from upstream. | ||
51 | |||
52 | If the option "existing_tempest_config" is not set then Rally follows the default path. | ||
53 | If "existing_tempest_config" is set to absolute path of tempest config folder (which | ||
54 | contains tempest "tools" and .testr.conf, e.g. /etc/tempest) then Rally uses this | ||
55 | existing tempest. By default, "existing_tempest_config" is set to "/etc/tempest/". | ||
56 | |||
57 | |||
58 | Build Configuration Options | ||
59 | =========================== | ||
60 | |||
61 | To have Rally and tempest included in final built image, include layer | ||
62 | meta-openstack-controller-test-config into Controller build and | ||
63 | layer meta-openstack-compute-test-config into Compute build. | ||
64 | |||
65 | |||
66 | Test Steps | ||
67 | ========== | ||
68 | |||
69 | Please note: the following commands/steps are carried on Controller | ||
70 | node, unless otherwise explicitly indicated. | ||
71 | |||
72 | # Start Controller and Compute node | ||
73 | $ ps aux | grep rally | ||
74 | root 13954 4.2 0.0 130856 27500 ? Ss 17:05 0:00 python /usr/bin/rally-api --log-dir=/var/log/rally | ||
75 | |||
76 | $ curl http://localhost:8877 | ||
77 | {"versions": [{"status": "CURRENT", "media_types": [{"base": "application/json", "type": "application/vnd.openstack.rally.v1+json"}], "id": "v1", "links": [{"href": "http://localhost:8877/v1", "rel": "self"}], "updated_at": "2014-01-07T00:00:00Z"}], "description": "Rally is a Benchmark-as-a-Service project for OpenStack.", "name": "OpenStack Rally API"} | ||
78 | |||
79 | $ rally deployment create --filename=/etc/rally/deployments/existing.json --name=wr-ops | ||
80 | +--------------------------------------+----------------------------+--------+------------------+--------+ | ||
81 | | uuid | created_at | name | status | active | | ||
82 | +--------------------------------------+----------------------------+--------+------------------+--------+ | ||
83 | | 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | 2014-07-07 17:09:11.160741 | wr-ops | deploy->finished | | | ||
84 | +--------------------------------------+----------------------------+--------+------------------+--------+ | ||
85 | |||
86 | $ rally deployment list | ||
87 | +--------------------------------------+----------------------------+--------+------------------+--------+ | ||
88 | | uuid | created_at | name | status | active | | ||
89 | +--------------------------------------+----------------------------+--------+------------------+--------+ | ||
90 | | 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | 2014-07-07 17:09:11.160741 | wr-ops | deploy->finished | * | | ||
91 | +--------------------------------------+----------------------------+--------+------------------+--------+ | ||
92 | |||
93 | $ rally use deployment --uuid=5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | ||
94 | Using deployment: 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | ||
95 | ~/.rally/openrc was updated | ||
96 | |||
97 | $ rally deployment check | ||
98 | keystone endpoints are valid and following services are available: | ||
99 | +------------+----------------+-----------+ | ||
100 | | services | type | status | | ||
101 | +------------+----------------+-----------+ | ||
102 | | barbican | keystore | Available | | ||
103 | | ceilometer | metering | Available | | ||
104 | | cinder | volume | Available | | ||
105 | | ec2 | ec2 | Available | | ||
106 | | glance | image | Available | | ||
107 | | heat | orchestration | Available | | ||
108 | | heat-cfn | cloudformation | Available | | ||
109 | | keystone | identity | Available | | ||
110 | | neutron | network | Available | | ||
111 | | nova | compute | Available | | ||
112 | | swift | object-store | Available | | ||
113 | +------------+----------------+-----------+ | ||
114 | |||
115 | $ rally show flavors | ||
116 | +----+-----------+-------+----------+-----------+-----------+ | ||
117 | | ID | Name | vCPUs | RAM (MB) | Swap (MB) | Disk (GB) | | ||
118 | +----+-----------+-------+----------+-----------+-----------+ | ||
119 | | 1 | m1.tiny | 1 | 512 | | 1 | | ||
120 | | 2 | m1.small | 1 | 2048 | | 20 | | ||
121 | | 3 | m1.medium | 2 | 4096 | | 40 | | ||
122 | | 4 | m1.large | 4 | 8192 | | 80 | | ||
123 | | 5 | m1.xlarge | 8 | 16384 | | 160 | | ||
124 | +----+-----------+-------+----------+-----------+-----------+ | ||
125 | |||
126 | $ rally show images | ||
127 | +--------------------------------------+--------------+----------+ | ||
128 | | UUID | Name | Size (B) | | ||
129 | +--------------------------------------+--------------+----------+ | ||
130 | | 3ec2f841-b179-47d7-908a-edb766b65c5e | myfirstimage | 13167616 | | ||
131 | +--------------------------------------+--------------+----------+ | ||
132 | |||
133 | $ rally show keypairs | ||
134 | +------+-------------+ | ||
135 | | Name | Fingerprint | | ||
136 | +------+-------------+ | ||
137 | +------+-------------+ | ||
138 | |||
139 | $ rally show networks | ||
140 | +--------------------------------------+--------------------------------------------+------+ | ||
141 | | ID | Label | CIDR | | ||
142 | +--------------------------------------+--------------------------------------------+------+ | ||
143 | | 7a9cf643-d9ab-404e-8155-65134bd937fb | TEMPEST_NET | None | | ||
144 | +--------------------------------------+--------------------------------------------+------+ | ||
145 | |||
146 | $ rally verify list | ||
147 | There are no results from verifier. To run a verifier, use: | ||
148 | rally verify start | ||
149 | |||
150 | $ rally verify start | ||
151 | Configuring existing Tempest for deployment 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | ||
152 | Verifier has been successfully configured to use existing Tempest | ||
153 | tempest.api.network.test_extensions.ExtensionsTestXML | ||
154 | test_list_show_extensions[gate,smoke] FAIL | ||
155 | setUpClass (tempest.api.network.test_networks | ||
156 | BulkNetworkOpsJSON) FAIL | ||
157 | tempest.api.orchestration.stacks.test_stacks.StacksTestJSON | ||
158 | test_stack_crud_no_resources[gate,smoke] OK 6.56 | ||
159 | test_stack_list_responds[gate,smoke] OK 0.02 | ||
160 | setUpClass (tempest.api.network.test_load_balancer | ||
161 | LoadBalancerXML) FAIL | ||
162 | tempest.api.compute.test_quotas.QuotasTestJSON | ||
163 | test_get_default_quotas[gate,smoke] OK 1.10 | ||
164 | test_get_quotas[gate,smoke] OK 0.02 | ||
165 | tempest.api.identity.admin.v3.test_domains.DomainsTestJSON | ||
166 | test_create_update_delete_domain[gate,smoke] OK 5.67 | ||
167 | |||
168 | ... | ||
169 | |||
170 | (Start Rally verification by running all on tempest testcases which are marked as smoke) | ||
171 | |||
172 | $ rally verify list | ||
173 | +--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ | ||
174 | | UUID | Deployment UUID | Set name | Tests | Failures | Created at | Status | | ||
175 | +--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ | ||
176 | | a5575bd2-d132-4522-b625-eed324396dda | 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | smoke | 196 | 58 | 2014-07-07 17:14:35.926483 | finished | | ||
177 | +--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ | ||
178 | |||
179 | $ rally verify show a5575bd2-d132-4522-b625-eed324396dda | ||
180 | Total results of verification: | ||
181 | |||
182 | +--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ | ||
183 | | UUID | Deployment UUID | Set name | Tests | Failures | Created at | Status | | ||
184 | +--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ | ||
185 | | a5575bd2-d132-4522-b625-eed324396dda | 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | smoke | 196 | 58 | 2014-07-07 17:14:35.926483 | finished | | ||
186 | +--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ | ||
187 | |||
188 | Tests: | ||
189 | |||
190 | +--------------------------------------------------------------------------------------------------------------------------------------------+------------+--------+ | ||
191 | | name | time | status | | ||
192 | +--------------------------------------------------------------------------------------------------------------------------------------------+------------+--------+ | ||
193 | | process-returncode | 0.0 | FAIL | | ||
194 | | setUpClass (tempest.api.compute.servers.test_create_server.ServersTestJSON) | 0.0 | FAIL | | ||
195 | | setUpClass (tempest.api.compute.servers.test_create_server.ServersTestManualDisk) | 0.0 | FAIL | | ||
196 | | setUpClass (tempest.api.compute.servers.test_create_server.ServersTestXML) | 0.0 | FAIL | | ||
197 | | setUpClass (tempest.api.network.test_floating_ips.FloatingIPTestJSON) | 0.0 | FAIL | | ||
198 | | setUpClass (tempest.api.network.test_floating_ips.FloatingIPTestXML) | 0.0 | FAIL | | ||
199 | | setUpClass (tempest.api.network.test_load_balancer.LoadBalancerJSON) | 0.0 | FAIL | | ||
200 | |||
201 | ... | ||
202 | |||
203 | $ rally verify results a5575bd2-d132-4522-b625-eed324396dda | ||
204 | (lot of details is printed out. Just make sure that there is no error when running this command) | ||
205 | |||
206 | $ rally verify detailed a5575bd2-d132-4522-b625-eed324396dda | ||
207 | (the out put is the same to the output of command "rally verify show" plus all the error/warning | ||
208 | messages from running tempest) | ||
209 | |||
210 | $ rally verify start --set identity | ||
211 | root@controller:~# rally verify start --set identity | ||
212 | tempest.api.identity.admin.test_services.ServicesTestJSON | ||
213 | test_create_get_delete_service[gate,smoke] OK 1.06 | ||
214 | test_list_services[gate,smoke] OK 0.82 | ||
215 | tempest.api.identity.admin.v3.test_services.ServicesTestJSON | ||
216 | test_update_service[gate] OK 3.65 | ||
217 | tempest.api.identity.admin.test_tenant_negative.TenantsNegativeTestJSON | ||
218 | test_create_tenant_by_unauthorized_user[gate,negative] OK 1.71 | ||
219 | test_create_tenant_request_without_token[gate,negative] OK 0.73 | ||
220 | test_create_tenant_with_empty_name[gate,negative] OK 0.89 | ||
221 | |||
222 | ... | ||
223 | |||
224 | (Start Rally verification by running all tempest testcases in tempest/api/identiy) | ||
225 | |||
226 | $ rally verify list | ||
227 | +--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ | ||
228 | | UUID | Deployment UUID | Set name | Tests | Failures | Created at | Status | | ||
229 | +--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ | ||
230 | | a5575bd2-d132-4522-b625-eed324396dda | 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | smoke | 196 | 58 | 2014-07-07 17:14:35.926483 | finished | | ||
231 | | b05bbc4c-eab4-4296-9020-d8305aa6457b | 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | identity | 216 | 0 | 2014-07-07 17:59:35.358702 | finished | | ||
232 | +--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+ | ||
233 | |||
234 | $ rally -v task start /etc/rally/tasks/example.json | ||
235 | ================================================================================ | ||
236 | Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 is started | ||
237 | -------------------------------------------------------------------------------- | ||
238 | 2014-07-07 18:06:30.608 9641 INFO rally.orchestrator.api [-] Benchmark Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 on Deployment 5cdc0ae5-03bd-4bba-bb7b-c43d1e295bdb | ||
239 | 2014-07-07 18:06:30.609 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Starting: Check cloud. | ||
240 | 2014-07-07 18:06:30.766 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Completed: Check cloud. | ||
241 | 2014-07-07 18:06:30.766 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Starting: Task validation. | ||
242 | 2014-07-07 18:06:30.785 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Starting: Task validation of scenarios names. | ||
243 | 2014-07-07 18:06:30.788 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Completed: Task validation of scenarios names. | ||
244 | 2014-07-07 18:06:30.788 9641 INFO rally.benchmark.engine [-] Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | Starting: Task validation of syntax. | ||
245 | |||
246 | ... | ||
247 | |||
248 | ================================================================================ | ||
249 | Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 is finished. | ||
250 | -------------------------------------------------------------------------------- | ||
251 | |||
252 | test scenario Authenticate.keystone | ||
253 | args position 0 | ||
254 | args values: | ||
255 | {u'context': {u'users': {u'concurrent': 30, | ||
256 | u'tenants': 3, | ||
257 | u'users_per_tenant': 50}}, | ||
258 | u'runner': {u'concurrency': 5, u'times': 100, u'type': u'constant'}} | ||
259 | +-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+ | ||
260 | | action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count | | ||
261 | +-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+ | ||
262 | | authenticate.keystone | 0.449 | 0.695 | 1.375 | 1.131 | 1.223 | 100.0% | 100 | | ||
263 | | total | 0.449 | 0.695 | 1.375 | 1.131 | 1.223 | 100.0% | 100 | | ||
264 | +-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+ | ||
265 | |||
266 | (This test step runs a benchmarking Rally task defined by /etc/rally/tasks/example.json) | ||
267 | |||
268 | $ rally task list | ||
269 | +--------------------------------------+----------------------------+----------+--------+-----+ | ||
270 | | uuid | created_at | status | failed | tag | | ||
271 | +--------------------------------------+----------------------------+----------+--------+-----+ | ||
272 | | 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | 2014-07-07 18:06:30.584805 | finished | False | | | ||
273 | +--------------------------------------+----------------------------+----------+--------+-----+ | ||
274 | |||
275 | $ rally task detailed 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 | ||
276 | ================================================================================ | ||
277 | Task 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 is finished. | ||
278 | -------------------------------------------------------------------------------- | ||
279 | /etc/rally/tasks/example.json | ||
280 | test scenario Authenticate.keystone | ||
281 | args position 0 | ||
282 | args values: | ||
283 | {u'context': {u'users': {u'concurrent': 30, | ||
284 | u'tenants': 3, | ||
285 | u'users_per_tenant': 50}}, | ||
286 | u'runner': {u'concurrency': 5, u'times': 100, u'type': u'constant'}} | ||
287 | +-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+ | ||
288 | | action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count | | ||
289 | +-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+ | ||
290 | | authenticate.keystone | 0.449 | 0.695 | 1.375 | 1.131 | 1.223 | 100.0% | 100 | | ||
291 | | total | 0.449 | 0.695 | 1.375 | 1.131 | 1.223 | 100.0% | 100 | | ||
292 | +-----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+ | ||
293 | |||
294 | $ rally task plot2html 8bf6d40c-4ffb-4e3e-87c9-f078bfbc0b77 --out output.html | ||
295 | $ ls -l output.html | ||
296 | -rw-r--r-- 1 root root 23028 Jul 7 18:12 output.html | ||
297 | |||
298 | # scp output.html into a host machine and use Browser on host machine to open | ||
299 | this ouput.html file. A nice graphical report for this benchmark is displayed | ||
300 | on Browser. Make sure the tale in Section "Table for task results" matches with | ||
301 | table printed out by "rally task detailed" above. | ||
302 | |||
303 | |||
304 | Rally Built-In Unit Tests | ||
305 | ========================= | ||
306 | |||
307 | This section describes how to run Rally built-in unit | ||
308 | tests which are located at: | ||
309 | |||
310 | /usr/lib64/python2.7/site-packages/rally/tests | ||
311 | /usr/lib64/python2.7/site-packages/rally/tests_ci | ||
312 | |||
313 | To run Rally built-in unit test with nosetests: | ||
314 | |||
315 | $ cd /usr/lib64/python2.7/site-packages/rally/tests | ||
316 | $ nosetests -v | ||
317 | |||
318 | ---------------------------------------------------------------------- | ||
319 | Ran 647 tests in 7.598s | ||
320 | |||
321 | FAILED (errors=3, failures=2) | ||
322 | |||
323 | $ /usr/lib64/python2.7/site-packages/rally/tests_ci | ||
324 | $ nosetests -v | ||
325 | |||
326 | ---------------------------------------------------------------------- | ||
327 | Ran 1 test in 2.620s | ||
328 | |||
329 | OK | ||
330 | |||
331 | |||
332 | References | ||
333 | ========== | ||
334 | |||
335 | https://wiki.openstack.org/wiki/Rally | ||