summaryrefslogtreecommitdiffstats
path: root/recipes-containers/k3s/README.md
diff options
context:
space:
mode:
Diffstat (limited to 'recipes-containers/k3s/README.md')
-rw-r--r--recipes-containers/k3s/README.md356
1 files changed, 123 insertions, 233 deletions
diff --git a/recipes-containers/k3s/README.md b/recipes-containers/k3s/README.md
index d167bcdb..13517db3 100644
--- a/recipes-containers/k3s/README.md
+++ b/recipes-containers/k3s/README.md
@@ -2,267 +2,157 @@
2 2
3Rancher's [k3s](https://k3s.io/), available under 3Rancher's [k3s](https://k3s.io/), available under
4[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides 4[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides
5lightweight Kubernetes suitable for small/edge devices. There are use cases 5lightweight Kubernetes suitable for small/edge devices.
6where the
7[installation procedures provided by Rancher](https://rancher.com/docs/k3s/latest/en/installation/)
8are not ideal but a bitbake-built version is what is needed. And only a few
9mods to the [k3s source code](https://github.com/rancher/k3s) is needed to
10accomplish that.
11 6
12## CNI 7## Build
13
14By default, K3s will run with flannel as the CNI, using VXLAN as the default
15backend. It is both possible to change the flannel backend and to change from
16flannel to another CNI.
17 8
18Please see <https://rancher.com/docs/k3s/latest/en/installation/network-options/> 9Add to `local.conf`:
19for further k3s networking details.
20 10
21## Configure and run a k3s agent 11```bash
12require conf/distro/include/meta-virt-host.conf
13require conf/distro/include/container-host-k3s.conf
14require conf/distro/include/meta-virt-dev.conf
15MACHINE = "qemux86-64"
16```
22 17
23The convenience script `k3s-agent` can be used to set up a k3s agent (service): 18Build:
24 19
25```shell 20```bash
26k3s-agent -t <token> -s https://<master>:6443 21bitbake container-image-host
27``` 22```
28(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the
29k3s master.)
30 23
31Example: 24## Single-Node Quick Start
32```shell
33k3s-agent -t /var/lib/rancher/k3s/server/node-token -s https://localhost:6443
34```
35 25
36If you are running an all in one node (both the server and agent) for testing 26```bash
37purposes, do not run the above script. It will perform cleanup and break flannel 27runqemu qemux86-64 container-image-host ext4 nographic kvm slirp qemuparams="-m 4096"
38networking on your host. 28```
39 29
40Instead, run the following (note the space between 'k3s' and 'agent'): 30After boot, k3s server starts automatically:
41 31
42```shell 32```bash
43k3s agent -t /var/lib/rancher/k3s/server/token --server http://localhost:6443/ 33export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
34kubectl get nodes # should show Ready
35kubectl get pods -A # system pods (coredns, metrics-server, etc.)
36kubectl run test --image=busybox --restart=Never -- sleep 300
37kubectl get pods # test pod Running
44``` 38```
45 39
46## Notes: 40## Multi-Node Cluster (QEMU Socket Networking)
41
42Uses QEMU socket networking to connect two VMs on a shared L2 segment.
43No root, no TAP, no bridge required.
47 44
48Memory: 45**Terminal 1 — Server:**
49 46
50 if running under qemu, the default of 256M of memory is not enough, k3s will 47```bash
51 OOM and exit. 48./scripts/run-k3s-multinode.sh server
49```
52 50
53 Boot with qemuparams="-m 2048" to boot with 2G of memory (or choose the 51After boot, get the join token:
54 appropriate amount for your configuration)
55 52
56Disk: 53```bash
54k3s-get-token
55```
57 56
58 if using qemu and core-image* you'll need to add extra space in your disks 57**Terminal 2 — Agent:**
59 to ensure containers can start. The following in your image recipe, or
60 local.conf would add 2G of extra space to the rootfs:
61 58
62```shell 59```bash
63IMAGE_ROOTFS_EXTRA_SPACE = "2097152" 60./scripts/run-k3s-multinode.sh agent --token <TOKEN>
64``` 61```
65 62
66## Example qemux86-64 boot line: 63**Verify on server (~30s after agent boot):**
67 64
68```shell 65```bash
69runqemu qemux86-64 nographic kvm slirp qemuparams="-m 2048" 66export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
67kubectl get nodes # 2 nodes Ready
70``` 68```
71 69
72k3s logs can be seen via: 70### How It Works
71
72- Both VMs boot the same `container-image-host` image (k3s profile)
73- The agent VM gets `k3s.role=agent` on the kernel cmdline
74- `k3s-role-setup.service` reads the cmdline and:
75 - Configures the cluster network interface (eth1) via systemd-networkd
76 - Masks the k3s server service
77 - Starts the k3s agent with the provided token
78- The `10-k3s-cluster.network` file (installed via `virt_networking` bbclass)
79 claims eth1 and disables DHCP, preventing networkd from interfering
80
81### Kernel Cmdline Parameters
82
83| Parameter | Description | Default |
84|-----------|-------------|---------|
85| `k3s.role=` | `server` or `agent` | `server` |
86| `k3s.server=` | Server IP (agent mode) | — |
87| `k3s.token=` | Join token (agent mode) | — |
88| `k3s.node-name=` | Override node name | hostname |
89| `k3s.node-ip=` | Static IP on cluster interface | — |
90| `k3s.iface=` | Cluster network interface | `eth1` |
91
92## CNI
93
94K3s uses flannel as the default CNI with VXLAN backend. The flannel
95CNI config is installed to `/etc/cni/net.d/cni-flannel.conflist`.
96CNI plugin binaries are in `/opt/cni/bin/`.
73 97
98See <https://docs.k3s.io/networking> for further k3s networking details.
74 99
75```shell 100## Traefik Ingress
76% journalctl -u k3s 101
102Traefik is enabled by default via PACKAGECONFIG. To disable:
103
104```bash
105PACKAGECONFIG:remove:pn-k3s = "traefik"
77``` 106```
78 107
79or 108## Packages
109
110| Package | Contents |
111|---------|----------|
112| `k3s` | Base binary, kubectl symlink, helpers |
113| `k3s-server` | k3s.service (systemd, auto-enabled) |
114| `k3s-agent` | k3s-agent.service (systemd, disabled by default) |
115| `k3s-cni` | Flannel CNI config |
116| `k3s-net-conf` | Cluster interface networkd config |
117
118## Useful Commands
80 119
81```shell 120```bash
82% journalctl -xe 121# Get join token (server only)
122k3s-get-token
123
124# Check k3s status
125systemctl status k3s
126journalctl -u k3s --no-pager -n 30
127
128# Kubernetes commands
129export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
130kubectl get nodes
131kubectl get pods -A
132kubectl run test --image=busybox --restart=Never -- sleep 300
133kubectl delete pod test
83``` 134```
84 135
85## Example output from qemux86-64 running k3s server: 136## Automated Testing
86 137
87```shell 138```bash
88root@qemux86-64:~# kubectl get nodes 139# Single-node tests
89NAME STATUS ROLES AGE VERSION 140pytest tests/test_k3s_runtime.py -v -k "not multinode" --machine qemux86-64
90qemux86-64 Ready master 46s v1.18.9-k3s1 141
91 142# Multi-node tests
92root@qemux86-64:~# kubectl get pods -n kube-system 143pytest tests/test_k3s_runtime.py -v -k "multinode" --machine qemux86-64
93NAME READY STATUS RESTARTS AGE
94local-path-provisioner-6d59f47c7-h7lxk 1/1 Running 0 2m32s
95metrics-server-7566d596c8-mwntr 1/1 Running 0 2m32s
96helm-install-traefik-229v7 0/1 Completed 0 2m32s
97coredns-7944c66d8d-9rfj7 1/1 Running 0 2m32s
98svclb-traefik-pb5j4 2/2 Running 0 89s
99traefik-758cd5fc85-lxpr8 1/1 Running 0 89s
100
101root@qemux86-64:~# kubectl describe pods -n kube-system
102
103root@qemux86-64:~# ip a s
1041: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
105 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
106 inet 127.0.0.1/8 scope host lo
107 valid_lft forever preferred_lft forever
108 inet6 ::1/128 scope host
109 valid_lft forever preferred_lft forever
1102: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
111 link/ether 52:54:00:12:35:02 brd ff:ff:ff:ff:ff:ff
112 inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
113 valid_lft forever preferred_lft forever
114 inet6 fec0::5054:ff:fe12:3502/64 scope site dynamic mngtmpaddr
115 valid_lft 86239sec preferred_lft 14239sec
116 inet6 fe80::5054:ff:fe12:3502/64 scope link
117 valid_lft forever preferred_lft forever
1183: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
119 link/sit 0.0.0.0 brd 0.0.0.0
1204: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
121 link/ether e2:aa:04:89:e6:0a brd ff:ff:ff:ff:ff:ff
122 inet 10.42.0.0/32 brd 10.42.0.0 scope global flannel.1
123 valid_lft forever preferred_lft forever
124 inet6 fe80::e0aa:4ff:fe89:e60a/64 scope link
125 valid_lft forever preferred_lft forever
1265: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
127 link/ether 02:42:be:3e:25:e7 brd ff:ff:ff:ff:ff:ff
128 inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
129 valid_lft forever preferred_lft forever
1306: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
131 link/ether 82:8e:b4:f8:06:e7 brd ff:ff:ff:ff:ff:ff
132 inet 10.42.0.1/24 brd 10.42.0.255 scope global cni0
133 valid_lft forever preferred_lft forever
134 inet6 fe80::808e:b4ff:fef8:6e7/64 scope link
135 valid_lft forever preferred_lft forever
1367: veth82ac482e@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
137 link/ether ea:9d:14:c1:00:70 brd ff:ff:ff:ff:ff:ff link-netns cni-c52e6e09-f6e0-a47b-aea3-d6c47d3e2d01
138 inet6 fe80::e89d:14ff:fec1:70/64 scope link
139 valid_lft forever preferred_lft forever
1408: vethb94745ed@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
141 link/ether 1e:7f:7e:d3:ca:e8 brd ff:ff:ff:ff:ff:ff link-netns cni-86958efe-2462-016f-292d-81dbccc16a83
142 inet6 fe80::8046:3cff:fe23:ced1/64 scope link
143 valid_lft forever preferred_lft forever
1449: veth81ffb276@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
145 link/ether 2a:1d:48:54:76:50 brd ff:ff:ff:ff:ff:ff link-netns cni-5d77238e-6452-4fa3-40d2-91d48386080b
146 inet6 fe80::acf4:7fff:fe11:b6f2/64 scope link
147 valid_lft forever preferred_lft forever
14810: vethce261f6a@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
149 link/ether 72:a3:90:4a:c5:12 brd ff:ff:ff:ff:ff:ff link-netns cni-55675948-77f2-a952-31ce-615f2bdb0093
150 inet6 fe80::4d5:1bff:fe5d:db3a/64 scope link
151 valid_lft forever preferred_lft forever
15211: vethee199cf4@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
153 link/ether e6:90:a4:a3:bc:a1 brd ff:ff:ff:ff:ff:ff link-netns cni-4aeccd16-2976-8a78-b2c4-e028da3bb1ea
154 inet6 fe80::c85a:8bff:fe0b:aea0/64 scope link
155 valid_lft forever preferred_lft forever
156
157
158root@qemux86-64:~# kubectl describe nodes
159
160Name: qemux86-64
161Roles: master
162Labels: beta.kubernetes.io/arch=amd64
163 beta.kubernetes.io/instance-type=k3s
164 beta.kubernetes.io/os=linux
165 k3s.io/hostname=qemux86-64
166 k3s.io/internal-ip=10.0.2.15
167 kubernetes.io/arch=amd64
168 kubernetes.io/hostname=qemux86-64
169 kubernetes.io/os=linux
170 node-role.kubernetes.io/master=true
171 node.kubernetes.io/instance-type=k3s
172Annotations: flannel.alpha.coreos.com/backend-data: {"VtepMAC":"2e:52:6a:1b:76:d4"}
173 flannel.alpha.coreos.com/backend-type: vxlan
174 flannel.alpha.coreos.com/kube-subnet-manager: true
175 flannel.alpha.coreos.com/public-ip: 10.0.2.15
176 k3s.io/node-args: ["server"]
177 k3s.io/node-config-hash: MLFMUCBMRVINLJJKSG32TOUFWB4CN55GMSNY25AZPESQXZCYRN2A====
178 k3s.io/node-env: {}
179 node.alpha.kubernetes.io/ttl: 0
180 volumes.kubernetes.io/controller-managed-attach-detach: true
181CreationTimestamp: Tue, 10 Nov 2020 14:01:28 +0000
182Taints: <none>
183Unschedulable: false
184Lease:
185 HolderIdentity: qemux86-64
186 AcquireTime: <unset>
187 RenewTime: Tue, 10 Nov 2020 14:56:27 +0000
188Conditions:
189 Type Status LastHeartbeatTime LastTransitionTime Reason Message
190 ---- ------ ----------------- ------------------ ------ -------
191 NetworkUnavailable False Tue, 10 Nov 2020 14:43:46 +0000 Tue, 10 Nov 2020 14:43:46 +0000 FlannelIsUp Flannel is running on this node
192 MemoryPressure False Tue, 10 Nov 2020 14:51:48 +0000 Tue, 10 Nov 2020 14:45:46 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
193 DiskPressure False Tue, 10 Nov 2020 14:51:48 +0000 Tue, 10 Nov 2020 14:45:46 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
194 PIDPressure False Tue, 10 Nov 2020 14:51:48 +0000 Tue, 10 Nov 2020 14:45:46 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
195 Ready True Tue, 10 Nov 2020 14:51:48 +0000 Tue, 10 Nov 2020 14:45:46 +0000 KubeletReady kubelet is posting ready status
196Addresses:
197 InternalIP: 10.0.2.15
198 Hostname: qemux86-64
199Capacity:
200 cpu: 1
201 ephemeral-storage: 39748144Ki
202 memory: 2040164Ki
203 pods: 110
204Allocatable:
205 cpu: 1
206 ephemeral-storage: 38666994453
207 memory: 2040164Ki
208 pods: 110
209System Info:
210 Machine ID: 6a4abfacbf83457e9a0cbb5777457c5d
211 System UUID: 6a4abfacbf83457e9a0cbb5777457c5d
212 Boot ID: f5ddf6c8-1abf-4aef-9e29-106488e3c337
213 Kernel Version: 5.8.13-yocto-standard
214 OS Image: Poky (Yocto Project Reference Distro) 3.2+snapshot-20201105 (master)
215 Operating System: linux
216 Architecture: amd64
217 Container Runtime Version: containerd://1.4.1-4-ge44e8ebea.m
218 Kubelet Version: v1.18.9-k3s1
219 Kube-Proxy Version: v1.18.9-k3s1
220PodCIDR: 10.42.0.0/24
221PodCIDRs: 10.42.0.0/24
222ProviderID: k3s://qemux86-64
223Non-terminated Pods: (5 in total)
224 Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
225 --------- ---- ------------ ---------- --------------- ------------- ---
226 kube-system svclb-traefik-jpmnd 0 (0%) 0 (0%) 0 (0%) 0 (0%) 54m
227 kube-system metrics-server-7566d596c8-wh29d 0 (0%) 0 (0%) 0 (0%) 0 (0%) 56m
228 kube-system local-path-provisioner-6d59f47c7-npn4d 0 (0%) 0 (0%) 0 (0%) 0 (0%) 56m
229 kube-system coredns-7944c66d8d-md8hr 100m (10%) 0 (0%) 70Mi (3%) 170Mi (8%) 56m
230 kube-system traefik-758cd5fc85-phjr2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 54m
231Allocated resources:
232 (Total limits may be over 100 percent, i.e., overcommitted.)
233 Resource Requests Limits
234 -------- -------- ------
235 cpu 100m (10%) 0 (0%)
236 memory 70Mi (3%) 170Mi (8%)
237 ephemeral-storage 0 (0%) 0 (0%)
238Events:
239 Type Reason Age From Message
240 ---- ------ ---- ---- -------
241 Normal Starting 56m kube-proxy Starting kube-proxy.
242 Normal Starting 55m kubelet Starting kubelet.
243 Warning InvalidDiskCapacity 55m kubelet invalid capacity 0 on image filesystem
244 Normal NodeHasSufficientPID 55m (x2 over 55m) kubelet Node qemux86-64 status is now: NodeHasSufficientPID
245 Normal NodeHasSufficientMemory 55m (x2 over 55m) kubelet Node qemux86-64 status is now: NodeHasSufficientMemory
246 Normal NodeHasNoDiskPressure 55m (x2 over 55m) kubelet Node qemux86-64 status is now: NodeHasNoDiskPressure
247 Normal NodeAllocatableEnforced 55m kubelet Updated Node Allocatable limit across pods
248 Normal NodeReady 54m kubelet Node qemux86-64 status is now: NodeReady
249 Normal Starting 52m kube-proxy Starting kube-proxy.
250 Normal NodeReady 50m kubelet Node qemux86-64 status is now: NodeReady
251 Normal NodeAllocatableEnforced 50m kubelet Updated Node Allocatable limit across pods
252 Warning Rebooted 50m kubelet Node qemux86-64 has been rebooted, boot id: a4e4d2d8-ddb4-49b8-b0a9-e81d12707113
253 Normal NodeHasSufficientMemory 50m (x2 over 50m) kubelet Node qemux86-64 status is now: NodeHasSufficientMemory
254 Normal Starting 50m kubelet Starting kubelet.
255 Normal NodeHasSufficientPID 50m (x2 over 50m) kubelet Node qemux86-64 status is now: NodeHasSufficientPID
256 Normal NodeHasNoDiskPressure 50m (x2 over 50m) kubelet Node qemux86-64 status is now: NodeHasNoDiskPressure
257 Normal NodeNotReady 17m kubelet Node qemux86-64 status is now: NodeNotReady
258 Warning InvalidDiskCapacity 15m (x2 over 50m) kubelet invalid capacity 0 on image filesystem
259 Normal Starting 12m kube-proxy Starting kube-proxy.
260 Normal Starting 10m kubelet Starting kubelet.
261 Warning InvalidDiskCapacity 10m kubelet invalid capacity 0 on image filesystem
262 Normal NodeAllocatableEnforced 10m kubelet Updated Node Allocatable limit across pods
263 Warning Rebooted 10m kubelet Node qemux86-64 has been rebooted, boot id: f5ddf6c8-1abf-4aef-9e29-106488e3c337
264 Normal NodeHasSufficientMemory 10m (x2 over 10m) kubelet Node qemux86-64 status is now: NodeHasSufficientMemory
265 Normal NodeHasNoDiskPressure 10m (x2 over 10m) kubelet Node qemux86-64 status is now: NodeHasNoDiskPressure
266 Normal NodeHasSufficientPID 10m (x2 over 10m) kubelet Node qemux86-64 status is now: NodeHasSufficientPID
267 Normal NodeReady 10m kubelet Node qemux86-64 status is now: NodeReady
268``` 144```
145
146## Notes
147
148**Memory:** K3s needs at least 2GB. Boot with `-m 4096` for comfortable
149operation with system pods + workloads.
150
151**Disk:** The default ext4 rootfs has enough space for k3s. If using
152core-image-minimal, add `IMAGE_ROOTFS_EXTRA_SPACE = "2097152"`.
153
154**KVM:** Strongly recommended. Without KVM, k3s startup takes
155significantly longer under TCG emulation.
156
157**k3s kubectl:** The embedded `k3s kubectl` subcommand is not available
158in this build. Use `kubectl` directly with `KUBECONFIG=/etc/rancher/k3s/k3s.yaml`.
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2026-09-03 03:55:04 +0000