# CRI-O Ptest Guide The CRI-O ptest suite is a comprehensive and complex testing framework. This document provides key information and tips for its usage. ## 1. Vendor Directory By default, the `vendor` directory is not installed for ptest. However, the `ctr_seccomp.bats` test relies on a JSON file located at: `vendor/github.com/containers/common/pkg/seccomp/seccomp.json`. As a result, the ctr_seccomp.bats test will fail unless this file is manually added. ### Steps to add seccomp.json: - Manually create/copy the required JSON file. - Set the `CONTAINER_SECCOMP_PROFILE` environment variable to point to the file's location. ## 2. Runtime Dependencies (RDEPENDS) The ptest suite requires several dependencies. As of the initial implementation, the runtime dependencies are defined as follows: ```bash RDEPENDS:${PN}-ptest += " \ bash \ bats \ cni \ crictl \ coreutils \ dbus-daemon-proxy \ iproute2 \ util-linux-unshare \ jq \ slirp4netns \ parallel \ podman \ " ``` ### Explanation of Dependencies: - **bash / bats**: The ptest suite is written using BATS and requires support from Bash. - **cni / crictl / podman**: Tools for testing container creation, runtime, and networking, directly invoked by the tests. - **jq**: Used in test scripts to create or manipulate JSON files. - **iproute2 / slirp4netns**: Networking utilities required for validating network-related functionalities. - **coreutils / dbus-daemon-proxy / util-linux-unshare**: Additional utilities supporting various test cases. - **paralle**: bats using "parallel" to execute test in serial. ## 3. Testing Log (Baseline Reference) A summary of the test results is provided below for baseline reference: - **PASS**: 317 tests - **FAIL**: 33 tests - **SKIP**: 32 tests #### Full Log Example: Below is an excerpt from a typical ptest log: ```bash root@intel-x86-64:~# ptest-runner cri-o -t 1000 START: ptest-runner 2024-11-23T14:50 BEGIN: /usr/lib64/cri-o/ptest SKIP: 1 apparmor tests (in sequence) # skip apparmor not enabled PASS: 2 no CDI errors, create ctr without CDI devices PASS: 3 no CDI errors, create ctr with CDI devices PASS: 4 no CDI errors, create ctr with annotated CDI devices PASS: 5 no CDI errors, create ctr with duplicate annotated CDI devices PASS: 6 no CDI errors, fail to create ctr with unresolvable CDI devices PASS: 7 no CDI errors, fail to create ctr with unresolvable annotated CDI devices PASS: 8 CDI registry refresh PASS: 9 CDI registry refresh, annotated CDI devices PASS: 10 reload CRI-O CDI parameters PASS: 11 reload CRI-O CDI parameters, with annotated CDI devices PASS: 12 CDI with errors, create ctr without CDI devices PASS: 13 CDI with errors, create ctr with (unaffected) CDI devices PASS: 14 CDI with errors, create ctr with (unaffected) annotated CDI devices PASS: 15 pids limit PASS: 16 conmon pod cgroup PASS: 17 conmon custom cgroup PASS: 18 conmon custom cgroup with no infra container SKIP: 19 conmonrs custom cgroup with no infra container # skip not supported for conmon SKIP: 20 ctr with swap should be configured # skip swap not enabled SKIP: 21 ctr with swap should fail when swap is lower # skip swap not enabled PASS: 22 ctr swap only configured if enabled SKIP: 23 ctr with swap should succeed when swap is unlimited # skip swap not enabled PASS: 24 cgroupv2 unified support SKIP: 25 cpu-quota.crio.io can disable quota # skip node must be configured with cgroupv1 for this test SKIP: 26 checkpoint and restore one container into a new pod (drop infra:true) # skip CRIU check failed SKIP: 27 checkpoint and restore one container into a new pod (drop infra:false) # skip CRIU check failed SKIP: 28 checkpoint and restore one container into a new pod using --export to OCI image # skip CRIU check failed SKIP: 29 checkpoint and restore one container into a new pod using --export to OCI image using repoDigest # skip CRIU check failed SKIP: 30 checkpoint and restore one container into a new pod with a new name # skip CRIU check failed PASS: 31 crio commands PASS: 32 invalid ulimits PASS: 33 invalid devices PASS: 34 invalid metrics port PASS: 35 invalid log max PASS: 36 log max boundary testing PASS: 37 default config should be empty PASS: 38 config dir should succeed PASS: 39 config dir should fail with invalid option PASS: 40 config dir should fail with invalid evented_pleg option PASS: 41 choose different default runtime should succeed PASS: 42 runc not existing when default_runtime changed should succeed PASS: 43 retain default runtime should succeed PASS: 44 monitor fields should be translated PASS: 45 handle nil workloads PASS: 46 config dir should fail with invalid disable_hostport_mapping option SKIP: 47 conmonrs is used # skip not using conmonrs SKIP: 48 test cpu load balancing # skip not yet supported on cgroup2 SKIP: 49 test cpu load balance disabled on manual stop # skip not yet supported on cgroup2 SKIP: 50 test cpu load balance disabled on container exit # skip not yet supported on cgroup2 PASS: 51 container memory metrics SKIP: 52 container memory cgroupv1-specific metrics # skip PASS: 53 storage directory check should find no issues PASS: 54 storage directory check should find errors PASS: 55 storage directory check should repair errors PASS: 56 storage directory check should wipe everything on repair errors PASS: 57 remove containers and images when remove both PASS: 58 remove containers when remove temporary PASS: 59 clear neither when remove persist PASS: 60 don't clear podman containers PASS: 61 clear everything when shutdown file not found PASS: 62 clear podman containers when shutdown file not found PASS: 63 fail to clear podman containers when shutdown file not found but container still running PASS: 64 don't clear containers on a forced restart of crio PASS: 65 don't clear containers if clean shutdown supported file not present PASS: 66 internal_wipe remove containers and images when remove both PASS: 67 internal_wipe remove containers when remove temporary and node reboots PASS: 68 internal_wipe remove containers when remove temporary PASS: 69 internal_wipe clear both when remove persist PASS: 70 internal_wipe don't clear podman containers PASS: 71 internal_wipe don't clear containers on a forced restart of crio PASS: 72 internal_wipe eventually cleans network on forced restart of crio if network is slow to come up PASS: 73 clean up image if corrupted on server restore PASS: 74 recover from badly corrupted storage directory SKIP: 75 run the critest suite # skip critest because RUN_CRITEST is not set PASS: 76 ctr not found correct error message PASS: 77 ctr termination reason Completed PASS: 78 ctr termination reason Error PASS: 79 ulimits PASS: 80 ctr remove PASS: 81 ctr lifecycle PASS: 82 ctr pod lifecycle with evented pleg enabled FAIL: 83 ctr logging PASS: 84 ctr log cleaned up if container create failed PASS: 85 ctr journald logging PASS: 86 ctr logging [tty=true] FAIL: 87 ctr log max FAIL: 88 ctr log max with default value FAIL: 89 ctr log max with minimum value FAIL: 90 ctr partial line logging PASS: 91 ctrs status for a pod PASS: 92 ctr list filtering PASS: 93 ctr list label filtering PASS: 94 ctr metadata in list & status PASS: 95 ctr execsync conflicting with conmon flags parsing PASS: 96 ctr execsync PASS: 97 ctr execsync should not overwrite initial spec args PASS: 98 ctr execsync should succeed if container has a terminal PASS: 99 ctr execsync should cap output PASS: 100 ctr exec{,sync} should be cancelled when container is stopped PASS: 101 ctr device add PASS: 102 privileged ctr device add PASS: 103 privileged ctr add duplicate device as host PASS: 104 ctr hostname env PASS: 105 ctr execsync failure PASS: 106 ctr execsync exit code PASS: 107 ctr execsync std{out,err} PASS: 108 ctr stop idempotent PASS: 109 ctr caps drop PASS: 110 ctr with default list of capabilities from crio.conf PASS: 111 ctr with list of capabilities given by user in crio.conf PASS: 112 ctr with add_inheritable_capabilities has inheritable capabilities PASS: 113 ctr /etc/resolv.conf rw/ro mode PASS: 114 ctr create with non-existent command PASS: 115 ctr create with non-existent command [tty] PASS: 116 ctr update resources PASS: 117 ctr correctly setup working directory PASS: 118 ctr execsync conflicting with conmon env PASS: 119 ctr resources PASS: 120 ctr with non-root user has no effective capabilities PASS: 121 ctr has gid in supplemental groups PASS: 122 ctr has gid in supplemental groups with Merge policy PASS: 123 ctr has only specified gid in supplemental groups with Strict policy PASS: 124 ctr with low memory configured should not be created PASS: 125 privileged ctr -- check for rw mounts PASS: 126 annotations passed through PASS: 127 ctr with default_env set in configuration PASS: 128 ctr with absent mount that should be rejected PASS: 129 ctr that mounts container storage as shared should keep shared PASS: 130 ctr that mounts container storage as private should not be private PASS: 131 ctr that mounts container storage as read-only option but not recursively SKIP: 132 ctr that mounts container storage as recursively read-only # skip requires crictl version "1.30" or newer SKIP: 133 ctr that fails to mount container storage as recursively read-only without readonly option # skip requires crictl version "1.30" or newer SKIP: 134 ctr that fails to mount container storage as recursively read-only without private propagation # skip requires crictl version "1.30" or newer PASS: 135 ctr has containerenv PASS: 136 ctr stop timeouts should decrease PASS: 137 ctr with node level pid namespace should not leak children PASS: 138 ctr HOME env newline invalid PASS: 139 ctr log linking PASS: 140 ctr stop loop kill retry attempts PASS: 141 ctr multiple stop calls PASS: 142 pause/unpause ctr with right ctr id PASS: 143 pause ctr with invalid ctr id PASS: 144 pause ctr with already paused ctr PASS: 145 unpause ctr with right ctr id with running ctr PASS: 146 unpause ctr with invalid ctr id PASS: 147 remove paused ctr FAIL: 148 ctr seccomp profiles unconfined FAIL: 149 ctr seccomp profiles runtime/default FAIL: 150 ctr seccomp profiles wrong profile name FAIL: 151 ctr seccomp profiles localhost profile name FAIL: 152 ctr seccomp overrides unconfined profile with runtime/default when overridden FAIL: 153 ctr seccomp profiles runtime/default block unshare SKIP: 154 ctr_userns run container # skip userns testing not enabled PASS: 155 bind secrets mounts to container PASS: 156 default mounts correctly sorted with other mounts PASS: 157 additional devices support PASS: 158 additional devices permissions PASS: 159 annotation devices support PASS: 160 annotation should not be processed if not allowed PASS: 161 annotation should override configured additional_devices PASS: 162 annotation should not be processed if not allowed in allowed_devices PASS: 163 annotation should configure multiple devices PASS: 164 annotation should fail if one device is invalid PASS: 165 test infra ctr dropped PASS: 166 test infra ctr not dropped PASS: 167 test infra ctr dropped status PASS: 168 pod test hooks PASS: 169 run container in pod with image ID PASS: 170 container status when created by image ID PASS: 171 container status when created by image tagged reference PASS: 172 container status when created by image canonical reference PASS: 173 container status when created by image list canonical reference PASS: 174 image pull and list PASS: 175 image pull and list using imagestore SKIP: 176 image pull with signature # skip registry has some issues PASS: 177 image pull and list by tag and ID PASS: 178 image pull and list by digest and ID PASS: 179 image pull and list by manifest list digest PASS: 180 image pull and list by manifest list tag PASS: 181 image pull and list by manifest list and individual digest PASS: 182 image pull and list by individual and manifest list digest PASS: 183 image list with filter PASS: 184 image list/remove PASS: 185 image status/remove SKIP: 186 run container in pod with crun-wasm enabled # skip crun-wasm not installed or runtime type is VM PASS: 187 check if image is pinned appropriately PASS: 188 run container in pod with timezone configured PASS: 189 run container in pod with local timezone PASS: 190 run container with memory_limit_in_bytes -1 PASS: 191 run container with memory_limit_in_bytes 12.5MiB PASS: 192 run container with container_min_memory 17.5MiB PASS: 193 run container with container_min_memory 5.5MiB PASS: 194 run container with empty container_min_memory PASS: 195 image remove with multiple names, by name PASS: 196 image remove with multiple names, by ID PASS: 197 image volume ignore PASS: 198 image volume bind PASS: 199 image volume user mkdir PASS: 200 image fs info with default settings should return matching container_filesystem and image_filesystem PASS: 201 image fs info with imagestore set should return different filesystems PASS: 202 test infra ctr cpuset PASS: 203 info inspect PASS: 204 ctr inspect PASS: 205 pod inspect when dropping infra PASS: 206 ctr inspect not found PASS: 207 inspect image should succeed contain all necessary information SKIP: 208 irqbalance tests (in sequence) # skip irqbalance not found. SKIP: 209 container run with kata should have containerd-shim-kata-v2 process running # skip Not PASS: 210 metrics with default host and port FAIL: 211 metrics with custom host using localhost and random port FAIL: 212 secure metrics with random port FAIL: 213 secure metrics with random port and missing cert/key PASS: 214 pid namespace mode pod test PASS: 215 pid namespace mode target test PASS: 216 KUBENSMNT mount namespace PASS: 217 ensure correct hostname PASS: 218 ensure correct hostname for hostnetwork:true PASS: 219 Check for valid pod netns CIDR PASS: 220 Ensure correct CNI plugin namespace/name/container-id arguments SKIP: 221 Connect to pod hostport from the host # skip node configured with cgroupv2 flakes this test sometimes PASS: 222 Clean up network if pod sandbox fails PASS: 223 Clean up network if pod sandbox fails after plugin success PASS: 224 Clean up network if pod sandbox gets killed PASS: 225 Ping pod from the host / another pod PASS: 226 run NRI PluginRegistration test PASS: 227 run NRI PluginSynchronization test PASS: 228 run NRI PodEvents test PASS: 229 run NRI ContainerEvents test PASS: 230 run NRI MountInjection test PASS: 231 run NRI EnvironmentInjection test PASS: 232 run NRI AnnotationInjection test PASS: 233 run NRI DeviceInjection test PASS: 234 run NRI CpusetAdjustment test PASS: 235 run NRI MemsetAdjustment test PASS: 236 run NRI CpusetAdjustmentUpdate test PASS: 237 run NRI MemsetAdjustmentUpdate test SKIP: 238 OCI image volume mount lifecycle # skip requires crictl version "1.31" or newer PASS: 239 pod release name on remove PASS: 240 pod remove PASS: 241 pod stop ignores not found sandboxes PASS: 242 pod list filtering PASS: 243 pod metadata in list & status PASS: 244 pass pod sysctls to runtime PASS: 245 pass pod sysctls to runtime when in userns SKIP: 246 disable crypto.fips_enabled when FIPS_DISABLE is set # skip The directory /proc/sys/crypto does not exist on this host. PASS: 247 fail to pass pod sysctls to runtime if invalid spaces PASS: 248 fail to pass pod sysctl to runtime if invalid value PASS: 249 skip pod sysctls to runtime if host PASS: 250 pod stop idempotent PASS: 251 pod remove idempotent PASS: 252 pod stop idempotent with ctrs already stopped PASS: 253 restart crio and still get pod status PASS: 254 invalid systemd cgroup_parent fail PASS: 255 systemd cgroup_parent correctly set PASS: 256 kubernetes pod terminationGracePeriod passthru PASS: 257 pod pause image matches configured image in crio.conf PASS: 258 pod stop cleans up all namespaces PASS: 259 pod with the correct etc folder ownership PASS: 260 verify RunAsGroup in container PASS: 261 single cni plugin with pod annotations capability enabled PASS: 262 single cni plugin with pod annotations capability disabled PASS: 263 pod annotations capability for chained cni plugins PASS: 264 accept unsigned image with default policy PASS: 265 deny unsigned image with restrictive policy PASS: 266 accept signed image with default policy FAIL: 267 accept signed image with restrictive policy PASS: 268 accept unsigned image with not existing namespace policy PASS: 269 accept unsigned image with higher priority namespace policy PASS: 270 deny unsigned image with higher priority namespace policy FAIL: 271 accept signed image with higher priority namespace policy PASS: 272 pprof PASS: 273 pprof over unix socket PASS: 274 reload config should succeed PASS: 275 reload config should succeed with 'log_level' PASS: 276 reload config should fail with 'log_level' if invalid PASS: 277 reload config should fail with if config is malformed PASS: 278 reload config should succeed with 'pause_image' PASS: 279 reload config should succeed with 'pause_command' PASS: 280 reload config should succeed with 'pause_image_auth_file' PASS: 281 reload config should fail with non existing 'pause_image_auth_file' PASS: 282 reload config should succeed with 'log_filter' PASS: 283 reload config should fail with invalid 'log_filter' PASS: 284 reload config should succeed with 'decryption_keys_path' PASS: 285 reload config should succeed with 'seccomp_profile' FAIL: 286 reload config should not fail with invalid 'seccomp_profile' SKIP: 287 reload config should succeed with 'apparmor_profile' # skip apparmor not enabled SKIP: 288 reload config should fail with invalid 'apparmor_profile' # skip apparmor not enabled PASS: 289 reload config should add new runtime PASS: 290 reload config should update 'pinned_images' PASS: 291 reload config should update 'pinned_images' and only 'pause_image' is pinned PASS: 292 reload config should update 'pause_image' and it becomes 'pinned_images' PASS: 293 reload config should remove pinned images when an empty list is provided PASS: 294 reload system registries should succeed PASS: 295 reload system registries should succeed with new registry PASS: 296 reload system registries should fail on invalid syntax in file PASS: 297 system registries should succeed with new registry without reload PASS: 298 system registries should fail on invalid syntax in file without reload PASS: 299 system handles burst of configuration changes without excessive reloads PASS: 300 system handles duplicate events for the same file PASS: 301 crio restore PASS: 302 crio restore with pod stopped PASS: 303 crio restore with bad state and pod stopped PASS: 304 crio restore with bad state and ctr stopped PASS: 305 crio restore with bad state and ctr removed PASS: 306 crio restore with bad state and pod removed PASS: 307 crio restore with bad state PASS: 308 crio restore with missing config.json PASS: 309 crio restore first not managing then managing PASS: 310 crio restore first managing then not managing PASS: 311 crio restore changing managing dir PASS: 312 crio restore upon entering KUBENSMNT PASS: 313 crio restore upon exiting KUBENSMNT PASS: 314 crio restore volumes for containers PASS: 315 crictl runtimeversion PASS: 316 if fs.may_detach_mounts is set FAIL: 317 seccomp notifier with runtime/default FAIL: 318 seccomp notifier with runtime/default but not stop FAIL: 319 seccomp notifier with custom profile FAIL: 320 seccomp notifier should not work if annotation is not allowed FAIL: 321 seccomp OCI artifact with image annotation without suffix FAIL: 322 seccomp OCI artifact with image annotation for pod FAIL: 323 seccomp OCI artifact with image annotation for container PASS: 324 seccomp OCI artifact with image annotation but not allowed annotation on runtime config FAIL: 325 seccomp OCI artifact with image annotation and profile set to unconfined PASS: 326 seccomp OCI artifact with image annotation but set runtime default profile with higher priority FAIL: 327 seccomp OCI artifact with image annotation but set localhost profile with higher priority FAIL: 328 seccomp OCI artifact with pod annotation FAIL: 329 seccomp OCI artifact with container annotation PASS: 330 seccomp OCI artifact with bogus annotation PASS: 331 seccomp OCI artifact with missing artifact PASS: 332 selinux label level=s0 is sufficient SKIP: 333 selinux skips relabeling if TrySkipVolumeSELinuxLabel annotation is present # skip not enforcing SKIP: 334 selinux skips relabeling for super privileged container # skip not enforcing PASS: 335 ctr check shared /dev/shm PASS: 336 check /dev/shm is changed PASS: 337 check /dev/shm fails with incorrect values PASS: 338 stats PASS: 339 container stats PASS: 340 pod stats PASS: 341 status not should fail if no subcommand is provided PASS: 342 status should succeed to retrieve the config PASS: 343 status should fail to retrieve the config with invalid socket PASS: 344 status should succeed to retrieve the info PASS: 345 status should fail to retrieve the info with invalid socket PASS: 346 succeed to retrieve the container info PASS: 347 should fail to retrieve the container info without ID PASS: 348 should fail to retrieve the container with invalid socket PASS: 349 should not clean up pod after timeout FAIL: 350 emit metric when sandbox is re-requested PASS: 351 should not clean up container after timeout PASS: 352 should clean up pod after timeout if request changes PASS: 353 should clean up container after timeout if request changes PASS: 354 should clean up pod after timeout if not re-requested PASS: 355 should not wait for actual duplicate pod request PASS: 356 should clean up container after timeout if not re-requested FAIL: 357 emit metric when container is re-requested PASS: 358 should not be able to operate on a timed out pod PASS: 359 should not be able to operate on a timed out container PASS: 360 should not wait for actual duplicate container request PASS: 361 check umask is changed FAIL: 362 userns annotation auto should succeed PASS: 363 userns annotation auto with keep-id and map-to-root should fail FAIL: 364 userns annotation auto should map host run_as_user FAIL: 365 version PASS: 366 version -j PASS: 367 test workload gets configured to defaults PASS: 368 test workload can override defaults PASS: 369 test workload should not be set if not defaulted or specified PASS: 370 test workload should not be set if annotation not specified PASS: 371 test workload pod gets configured to defaults PASS: 372 test workload can override pod defaults PASS: 373 test workload pod should not be set if not defaulted or specified PASS: 374 test workload pod should not be set if annotation not specified PASS: 375 test workload pod should override infra_ctr_cpuset option PASS: 376 test workload allowed annotation should not work if not configured PASS: 377 test workload allowed annotation appended with runtime PASS: 378 test workload allowed annotation works for pod PASS: 379 test resource cleanup on bad annotation contents PASS: 380 test workload pod should not be set if annotation not specified even if prefix PASS: 381 test special runtime annotations not allowed PASS: 382 test special runtime annotations allowed real 9m12.847s user 42m18.946s sys 8m15.064s DURATION: 553 END: /usr/lib64/cri-o/ptest 2024-11-23T14:59 STOP: ptest-runner TOTAL: 1 FAIL: 0 root@intel-x86-64:~# ```