Adding k3s recipe
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 1 line · Erik Jansson
Signed-off-by Erik Jansson <[email protected]>
Bruce, what is the current status of the k3s activities? (I know you were busy with some other things last month.)
BR,
/Joakim
It is close. I had to abandon one approach, since it grew more complex than if I just let a chunk of functionality be duplicated.
I'm at this again this week, and if I can't sort it out, I'll merge the core recipes and start factoring in tree (I'm just worried about people assuming what appears in tree is the final version).
Cheers,
Bruce
Quoted 15 lines · Joakim Roubert
Quoted 1 line · Erik Jansson
Signed-off-by Erik Jansson <[email protected]>Bruce, what is the current status of the k3s activities? (I know you were busy with some other things last month.)
BR,
/Joakim
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 4 lines · Bruce Ashfield
It is close. I had to abandon one approach, since it grew more complex than if I just let a chunk of functionality be duplicated.
I'm at this again this week
Excellent news!
Quoted 3 lines
and if I can't sort it out, I'll merge the core recipes and start factoring in tree (I'm just worried about people assuming what appears in tree is the final version).
But with those words you have just informed eveybody on the list about it, so that should make things clearer for the community!
BR,
/Joakim
Signature · 7 lines
Joakim Roubert Senior Engineer Axis Communications AB Emdalavägen 14, SE-223 69 Lund, Sweden Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48 Fax: +46 46 13 61 30, www.axis.com
Quoted 7 lines · Joakim Roubert
Quoted 4 lines · Bruce Ashfield
It is close. I had to abandon one approach, since it grew more complex than if I just let a chunk of functionality be duplicated.
I'm at this again this week
Quoted 8 lines
Excellent news!
Quoted 3 lines
and if I can't sort it out, I'll merge the core recipes and start factoring in tree (I'm just worried about people assuming what appears in tree is the final version).
But with those words you have just informed eveybody on the list about it, so that should make things clearer for the community!
That I did!
So I've decided to do this in a couple phases, since I keep making the bbclass more complex that it needs to be.
I've gone back through the patches, and have some questions/comments. Would you be willing to tweak things basd on those comments ? I realize it's been quite the delay from submission to this, so I can work through issues if you don't have the cycles.
Bruce
Quoted 16 lines
BR,
/Joakim
--
Joakim Roubert
Senior EngineerAxis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 3 lines · Bruce Ashfield
So I've decided to do this in a couple phases, since I keep making the bbclass more complex that it needs to be.
Sounds like a good plan! For what we know, concepts as "Big Bang" only was successful once in the history of Universe, and it was not within the field of software development. ;-)
Quoted 4 lines
I've gone back through the patches, and have some questions/comments. Would you be willing to tweak things basd on those comments ? I realize it's been quite the delay from submission to this, so I can work through issues if you don't have the cycles.
Yes, I am happy to help in whatever way I can. Erik is a colleague of mine but currently on parental leave, so I will try to fill his shoes here now. But I have plenty of openembedded k3s build experience with these recipes too, so let's do this!
BR,
/Joakim
Signature · 7 lines
Joakim Roubert Senior Engineer Axis Communications AB Emdalavägen 14, SE-223 69 Lund, Sweden Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48 Fax: +46 46 13 61 30, www.axis.com
Quoted 15 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield
So I've decided to do this in a couple phases, since I keep making the bbclass more complex that it needs to be.
Sounds like a good plan! For what we know, concepts as "Big Bang" only was successful once in the history of Universe, and it was not within the field of software development. ;-)
Quoted 4 lines
I've gone back through the patches, and have some questions/comments. Would you be willing to tweak things basd on those comments ? I realize it's been quite the delay from submission to this, so I can work through issues if you don't have the cycles.
Quoted 4 lines
Yes, I am happy to help in whatever way I can. Erik is a colleague of mine but currently on parental leave, so I will try to fill his shoes here now. But I have plenty of openembedded k3s build experience with these recipes too, so let's do this!
By phases, I meant some smaller changes to the series while I work a bit more through the bbclass and DISTRO feature flag efforts.
I can say immediately, that we have a problem with the dependencies, and we need to sort those out.
There are at least two utilities that are shared with the main kubernetes and not in our existing layer dependencies. We shouldn't expand our dependencies more, so we have to look at importing a variant of the recipes to meta-virt, or seeing about getting the recipes into a more common location. In particular, I'm talking about ipset and and upx, and I'm still looking for more.
As for the service files, the k3s.service is different from the one from the latest on the k3s 1.18 branch, and there's a patch that modifies it. Although tempting to not use /usr/local/bin, for maintenance, we'd be better off either just installing where things are expected. Was something broken if /usr/local/bin was used ?
The other .service files, we have no idea where they come from by looking at the series, and as such they need some sort of comment (by looking at the repo, you can tell they are variants of the k3s.service file, but that needs to be clarified).
The patches themselves to the code base (not just the overall git commit) should have Signed-off-by: and Upstream-status: flags, so they can be maintained as I bump through releases.
The scripts being created by the recipe, need license headers and spdx tags. Boring, but it is something we want to make sure is in new files. Also, at a glance, we need to make sure the dependencies of the scripts are captured in the DEPENDS of the recipe (i.e. I see 'ip' being called, make sure that is captured).
As for the networking, what we have in the submission is specific to a setup, that's the parts that I'm really trying to unify. We need to document what is expected in the CNI configuration, and then control it via a distro/image flag such that it won't clobber/conflict with other CNI users. I can take care of that distro part, but the documentation would be ideal in a README that comes along with the recipe.
As I said, I'm still working through some of the factoring out and de-duplication, but there's some things in there that are best done on your end, versus mine.
Bruce
Quoted 13 lines
BR,
/Joakim
--
Joakim Roubert
Senior EngineerAxis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 7 lines · Bruce Ashfield
There are at least two utilities that are shared with the main kubernetes and not in our existing layer dependencies. We shouldn't expand our dependencies more, so we have to look at importing a variant of the recipes to meta-virt, or seeing about getting the recipes into a more common location. In particular, I'm talking about ipset and and upx, and I'm still looking for more.
Ah, yes.
Quoted 5 lines
As for the service files, the k3s.service is different from the one from the latest on the k3s 1.18 branch, and there's a patch that modifies it. Although tempting to not use
/usr/local/bin, for maintenance, we'd be better off either just installing where things are expected. Was something broken if/usr/local/binwas used ?
The target systems I work on prohibits installations to /usr/local, so from my perspective something would in fact be broken if /usr/local/bin was to be used. Nevertheless, configuring the recipe to have /usr/{$VARIABLE}/bin and VARIABLE set to "local" as default would perhaps be an option for everybody to eat the cake and still have it?
Quoted 4 lines
The other .service files, we have no idea where they come from by looking at the series, and as such they need some sort of comment (by looking at the repo, you can tell they are variants of the k3s.service file, but that needs to be clarified).
I will investigate.
Quoted 3 lines
The patches themselves to the code base (not just the overall git commit) should have Signed-off-by: and Upstream-status: flags, so they can be maintained as I bump through releases.
I will see to that.
Quoted 3 lines
The scripts being created by the recipe, need license headers and spdx tags. Boring, but it is something we want to make sure is in new files.
Absolutely!
Quoted 3 lines
Also, at a glance, we need to make sure the dependencies of the scripts are captured in the DEPENDS of the recipe (i.e. I see 'ip' being called, make sure that is captured).
Ah, yes, I will go through that.
Quoted 7 lines
As for the networking, what we have in the submission is specific to a setup, that's the parts that I'm really trying to unify. We need to document what is expected in the CNI configuration, and then control it via a distro/image flag such that it won't clobber/conflict with other CNI users. I can take care of that distro part, but the documentation would be ideal in a README that comes along with the recipe.
That also sounds like a good plan. I am not relly sure what text to write in that README file, though. But I guess that is something that can be iteratively created.
Quoted 3 lines
As I said, I'm still working through some of the factoring out and de-duplication, but there's some things in there that are best done on your end, versus mine.
Excellent!
BR,
/Joakim
Quoted 18 lines · Joakim Roubert
Quoted 7 lines · Bruce Ashfield
There are at least two utilities that are shared with the main kubernetes and not in our existing layer dependencies. We shouldn't expand our dependencies more, so we have to look at importing a variant of the recipes to meta-virt, or seeing about getting the recipes into a more common location. In particular, I'm talking about ipset and and upx, and I'm still looking for more.
Ah, yes.
Quoted 5 lines
As for the service files, the k3s.service is different from the one from the latest on the k3s 1.18 branch, and there's a patch that modifies it. Although tempting to not use
/usr/local/bin, for maintenance, we'd be better off either just installing where things are expected. Was something broken if/usr/local/binwas used ?
Quoted 5 lines
The target systems I work on prohibits installations to
/usr/local, so from my perspective something would in fact be broken if/usr/local/binwas to be used. Nevertheless, configuring the recipe to have/usr/{$VARIABLE}/binand VARIABLE set to "local" as default would perhaps be an option for everybody to eat the cake and still have it?
Absolutely. I'm always in favour of making these configurable, so we can let users customize as they see fit.
Alternatively, rather than substituting just the 'local', the path up to 'bin' could be controlled. And by that, I mean a variable local to the recipe, like you have above.
That allows the use of $bindir by default, and also covers the case automatically where someone has changed $bindir to /usr/local/bin.
The local variable allows yet another override if the global $bindir hasn't been changed, and you just want to modify it for this recipe (which is actually what I think will be most common).
Quoted 12 lines
Quoted 5 lines
The other .service files, we have no idea where they come from by looking at the series, and as such they need some sort of comment (by looking at the repo, you can tell they are variants of the k3s.service file, but that needs to be clarified).
I will investigate.
Quoted 3 lines
The patches themselves to the code base (not just the overall git commit) should have Signed-off-by: and Upstream-status: flags, so they can be maintained as I bump through releases.
Quoted 7 lines
I will see to that.
Quoted 3 lines
The scripts being created by the recipe, need license headers and spdx tags. Boring, but it is something we want to make sure is in new files.
Absolutely!
I just had to go through the License / SPDX header addition for hundreds of files. So I know what a pain this can be, so thanks!
Quoted 15 lines
Quoted 4 lines
Also, at a glance, we need to make sure the dependencies of the scripts are captured in the DEPENDS of the recipe (i.e. I see 'ip' being called, make sure that is captured).
Ah, yes, I will go through that.
Quoted 7 lines
As for the networking, what we have in the submission is specific to a setup, that's the parts that I'm really trying to unify. We need to document what is expected in the CNI configuration, and then control it via a distro/image flag such that it won't clobber/conflict with other CNI users. I can take care of that distro part, but the documentation would be ideal in a README that comes along with the recipe.
Quoted 3 lines
That also sounds like a good plan. I am not relly sure what text to write in that README file, though. But I guess that is something that can be iteratively created.
Absolutely. Just a stream of thoughts is fine for now, with a quick overview, and perhaps a guideline on how to build a single node system ? With a few notes about what is configured out of the box and what we expect the end user to continue to configure / customize ?
That would be more than enough as a start.
Bruce
Quoted 10 lines
Quoted 4 lines
As I said, I'm still working through some of the factoring out and de-duplication, but there's some things in there that are best done on your end, versus mine.
Excellent!
BR,
/Joakim
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 3 lines · Bruce Ashfield
Absolutely. I'm always in favour of making these configurable, so we can let users customize as they see fit.
Wonderful.
I finalized a quite (IMO) nice solution during the day, where ${exec_prefix} would be the fixed starting point and /bin the ending dito, and what is in between configurable. But I like your suggestion with even more flexibility better and will update according to that.
Quoted 2 lines
I just had to go through the License / SPDX header addition for hundreds of files. So I know what a pain this can be, so thanks!
No worries, I find those things very important too.
Quoted 4 lines
Absolutely. Just a stream of thoughts is fine for now, with a quick overview, and perhaps a guideline on how to build a single node system ? With a few notes about what is configured out of the box and what we expect the end user to continue to configure / customize ?
Ok!
I am running out of time here today, so even though I have ticked off several of the tasks on the list, it will be next week where I get back with an update!
BR,
/Joakim
A message that is not in the archive (sent off-list or not yet received): [email protected]
--- recipes-containers/k3s/README.md | 26 +++++ ...01-Finding-host-local-in-usr-libexec.patch | 27 +++++ .../k3s/k3s/cni-containerd-net.conf | 24 +++++ recipes-containers/k3s/k3s/k3s-agent | 100 ++++++++++++++++++ recipes-containers/k3s/k3s/k3s-agent.service | 26 +++++ recipes-containers/k3s/k3s/k3s-clean | 25 +++++ recipes-containers/k3s/k3s/k3s.service | 27 +++++ recipes-containers/k3s/k3s_git.bb | 75 +++++++++++++ 8 files changed, 330 insertions(+) create mode 100644 recipes-containers/k3s/README.md create mode 100644 recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch create mode 100644 recipes-containers/k3s/k3s/cni-containerd-net.conf create mode 100755 recipes-containers/k3s/k3s/k3s-agent create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service create mode 100755 recipes-containers/k3s/k3s/k3s-clean create mode 100644 recipes-containers/k3s/k3s/k3s.service create mode 100644 recipes-containers/k3s/k3s_git.bb
recipes-containers/k3s/README.md new file +26 -0
@@ -0,0 +1,26 @@ +# k3s: Lightweight Kubernetes + +Rancher's [k3s](https://k3s.io/), available under +[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides +lightweight Kubernetes suitable for small/edge devices. There are use cases +where the +[installation procedures provided by Rancher](https://rancher.com/docs/k3s/latest/en/installation/) +are not ideal but a bitbake-built version is what is needed. And only a few +mods to the [k3s source code](https://github.com/rancher/k3s) is needed to +accomplish that. + +## CNI +By default, K3s will run with flannel as the CNI, using VXLAN as the default +backend. It is both possible to change the flannel backend and to change from +flannel to another CNI. + +Please see https://rancher.com/docs/k3s/latest/en/installation/network-options/ +for further k3s networking details. + +## Configure and run a k3s agent +The convenience script `k3s-agent` can be used to set up a k3s agent (service): + + k3s-agent -t <token> -s https://<master>:6443 + +(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the +k3s master.)
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch new file +27 -0
@@ -0,0 +1,27 @@ +From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001 +From: Erik Jansson <[email protected]> +Date: Wed, 16 Oct 2019 15:07:48 +0200 +Subject: [PATCH] Finding host-local in /usr/libexec + +Upstream-status: Inappropriate [embedded specific] +Signed-off-by: <[email protected]> +--- + pkg/agent/config/config.go | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go +index b4296f360a..6af9dab895 100644 +--- a/pkg/agent/config/config.go ++++ b/pkg/agent/config/config.go +@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) { + return nil, err + } + +- hostLocal, err := exec.LookPath("host-local") ++ hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local") + if err != nil { + return nil, errors.Wrapf(err, "failed to find host-local") + } +-- +2.11.0 +
recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24 -0
@@ -0,0 +1,24 @@ +{ + "cniVersion": "0.4.0", + "name": "containerd-net", + "plugins": [ + { + "type": "bridge", + "bridge": "cni0", + "isGateway": true, + "ipMasq": true, + "promiscMode": true, + "ipam": { + "type": "host-local", + "subnet": "10.88.0.0/16", + "routes": [ + { "dst": "0.0.0.0/0" } + ] + } + }, + { + "type": "portmap", + "capabilities": {"portMappings": true} + } + ] +}
recipes-containers/k3s/k3s/k3s-agent new file +100 -0
100 lines not on this page · raw message
recipes-containers/k3s/k3s/k3s-agent.service new file +26 -0
@@ -0,0 +1,26 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes Agent +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=control-group +Delegate=yes +LimitNOFILE=infinity +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s agent +ExecStopPost=/usr/local/bin/k3s-clean +
recipes-containers/k3s/k3s/k3s-clean new file +25 -0
@@ -0,0 +1,25 @@ +#!/bin/sh -eu +# SPDX-License-Identifier: Apache-2.0 +do_unmount() { + [ $# -eq 2 ] || return + local mounts= + while read ignore mount ignore; do + case $mount in + $1/*|$2/*) + mounts="$mount $mounts" + ;; + esac + done </proc/self/mounts + [ -z "$mounts" ] || umount $mounts +} + +do_unmount /run/k3s /var/lib/rancher/k3s + +ip link show | grep 'master cni0' | while read ignore iface ignore; do + iface=${iface%%@*} + [ -z "$iface" ] || ip link delete $iface +done + +ip link delete cni0 +ip link delete flannel.1 +rm -rf /var/lib/cni/
recipes-containers/k3s/k3s/k3s.service new file +27 -0
@@ -0,0 +1,27 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=process +Delegate=yes +# Having non-zero Limit*s causes performance problems due to accounting overhead +# in the kernel. We recommend using cgroups to do container-local accounting. +LimitNOFILE=1048576 +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s server +
recipes-containers/k3s/k3s_git.bb new file +75 -0
75 lines not on this page · raw message
Signature · 1 line
2.20.1
Thanks for the updated series, see some comments inline.
Quoted 55 lines · Joakim Roubert
Signed-off-by Joakim Roubert <[email protected]>--- recipes-containers/k3s/README.md | 26 +++++ ...01-Finding-host-local-in-usr-libexec.patch | 27 +++++ .../k3s/k3s/cni-containerd-net.conf | 24 +++++ recipes-containers/k3s/k3s/k3s-agent | 100 ++++++++++++++++++ recipes-containers/k3s/k3s/k3s-agent.service | 26 +++++ recipes-containers/k3s/k3s/k3s-clean | 25 +++++ recipes-containers/k3s/k3s/k3s.service | 27 +++++ recipes-containers/k3s/k3s_git.bb | 75 +++++++++++++ 8 files changed, 330 insertions(+)create mode 100644
recipes-containers/k3s/README.md
create mode 100644recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch
create mode 100644recipes-containers/k3s/k3s/cni-containerd-net.confcreate mode 100755 recipes-containers/k3s/k3s/k3s-agent create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service create mode 100755 recipes-containers/k3s/k3s/k3s-clean create mode 100644 recipes-containers/k3s/k3s/k3s.service create mode 100644 recipes-containers/k3s/k3s_git.bbrecipes-containers/k3s/README.md new file +7
-0@@ -0,0 +1,26 @@ +# k3s: Lightweight Kubernetes + +Rancher's [k3s](https://k3s.io/), available under +[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides +lightweight Kubernetes suitable for small/edge devices. There are use cases +where the +[installation procedures provided byRancher](https://rancher.com/docs/k3s/latest/en/installation/)
recipes-containers/k3s/README.md new file +6
-0+are not ideal but a bitbake-built version is what is needed. And only a few +mods to the [k3s source code](https://github.com/rancher/k3s) is needed to +accomplish that. + +## CNI +By default, K3s will run with flannel as the CNI, using VXLAN as thedefault
+backend. It is both possible to change the flannel backend and to change fromrecipes-containers/k3s/README.md new file +3
-0+flannel to another CNI. + +Please seehttps://rancher.com/docs/k3s/latest/en/installation/network-options/
recipes-containers/k3s/README.md new file +4
-0+for further k3s networking details. + +## Configure and run a k3s agent +The convenience script `k3s-agent` can be used to set up a k3s agent(service):
recipes-containers/k3s/README.md line 26
Quoted 5 lines
recipes-containers/k3s/README.md new file +5
-0+ + k3s-agent -t <token> -s https://<master>:6443 + +(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the +k3s master.)
Thanks for the README. It's a good start and pretty much all we need for now. I'll be doing some build and runtime testing and can do some tweaks as required.
Quoted 74 lines
diff --git
a/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patchb/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patchnew file mode 100644
index 0000000..8205d73
---/dev/null
+++b/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patchrecipes-containers/k3s/README.md new file +27
-0@@ -0,0 +1,27 @@ +From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001 +From: Erik Jansson <[email protected]> +Date: Wed, 16 Oct 2019 15:07:48 +0200 +Subject: [PATCH] Finding host-local in /usr/libexec + +Upstream-status: Inappropriate [embedded specific] +Signed-off-by: <[email protected]> +--- + pkg/agent/config/config.go | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go +index b4296f360a..6af9dab895 100644 +--- a/pkg/agent/config/config.go ++++ b/pkg/agent/config/config.go +@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) { + return nil, err + } + +- hostLocal, err := exec.LookPath("host-local") ++ hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local") + if err != nil { + return nil, errors.Wrapf(err, "failed to find host-local") + } +-- +2.11.0 +recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24
-0@@ -0,0 +1,24 @@ +{ + "cniVersion": "0.4.0", + "name": "containerd-net", + "plugins": [ + { + "type": "bridge", + "bridge": "cni0", + "isGateway": true, + "ipMasq": true, + "promiscMode": true, + "ipam": { + "type": "host-local", + "subnet": "10.88.0.0/16", + "routes": [ + { "dst": "0.0.0.0/0" } + ] + } + }, + { + "type": "portmap", + "capabilities": {"portMappings": true} + } + ] +}recipes-containers/k3s/k3s/k3s-agent new file +0
-0
recipes-containers/k3s/k3s/k3s-agent line 1
Quoted 1 line
recipes-containers/k3s/k3s/k3s-agent new file +1
-0@@ -0,0 +1,100 @@ +#!/bin/sh -eu
For these scripts, we have the license, but not a copyright. Which should be ok, but are the scripts completely written by you (or someone at your company?), if so, it is a good idea to put a copyright header on the scripts, so we can know the origin.
recipes-containers/k3s/k3s/k3s-agent line 2
Quoted 1 line
recipes-containers/k3s/k3s/k3s-agent new file +1
-0+# SPDX-License-Identifier: Apache-2.0
The SPDX headers look good.
Quoted 109 lines
recipes-containers/k3s/k3s/k3s-agent new file +9
-0+ +ENV_CONF=/etc/systemd/system/k3s-agent.service.d/10-env.conf + +usage() { + echo " +USAGE: + ${0##*/} [OPTIONS] +OPTIONS: + --token value, -t value Token to use for authentication[\$K3S_TOKEN]
+ --token-file value Token file to use for
authentication [\$K3S_TOKEN_FILE]recipes-containers/k3s/k3s/k3s-agent new file +3
-0+ --server value, -s value Server to connect to [\$K3S_URL] + --node-name value Node name [\$K3S_NODE_NAME] + --resolv-conf value Kubelet resolv.conf file[\$K3S_RESOLV_CONF]
+ --cluster-secret value Shared secret used to bootstrap
a cluster [\$K3S_CLUSTER_SECRET]recipes-containers/k3s/k3s/k3s-agent new file +84
-0+ -h print this +" +} + +[ $# -gt 0 ] || { + usage + exit +} + +case $1 in + -*) + ;; + *) + usage + exit 1 + ;; +esac + +rm -f $ENV_CONF +mkdir -p ${ENV_CONF%/*} +echo [Service] > $ENV_CONF + +while getopts "t:s:-:h" opt; do + case $opt in + h) + usage + exit + ;; + t) + VAR_NAME=K3S_TOKEN + ;; + s) + VAR_NAME=K3S_URL + ;; + -) + [ $# -ge $OPTIND ] || { + usage + exit 1 + } + opt=$OPTARG + eval OPTARG='$'$OPTIND + OPTIND=$(($OPTIND + 1)) + case $opt in + token) + VAR_NAME=K3S_TOKEN + ;; + token-file) + VAR_NAME=K3S_TOKEN_FILE + ;; + server) + VAR_NAME=K3S_URL + ;; + node-name) + VAR_NAME=K3S_NODE_NAME + ;; + resolv-conf) + VAR_NAME=K3S_RESOLV_CONF + ;; + cluster-secret) + VAR_NAME=K3S_CLUSTER_SECRET + ;; + help) + usage + exit + ;; + *) + usage + exit 1 + ;; + esac + ;; + *) + usage + exit 1 + ;; + esac + echo Environment=$VAR_NAME=$OPTARG >> $ENV_CONF +done + +chmod 0644 $ENV_CONF +rm -rf /var/lib/rancher/k3s/agent +systemctl daemon-reload +systemctl restart k3s-agent +systemctl enable k3s-agent.servicerecipes-containers/k3s/k3s/k3s-agent.service new file +0
-0
recipes-containers/k3s/k3s/k3s-agent.service line 1
Quoted 1 line
recipes-containers/k3s/k3s/k3s-agent.service new file +1
-0@@ -0,0 +1,26 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function
Perfect. This is what I was looking for.
Quoted 158 lines
recipes-containers/k3s/k3s/k3s-agent.service new file +25
-0+[Unit] +Description=Lightweight Kubernetes Agent +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=control-group +Delegate=yes +LimitNOFILE=infinity +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s agent +ExecStopPost=/usr/local/bin/k3s-clean +recipes-containers/k3s/k3s/k3s-clean new file +25
-0@@ -0,0 +1,25 @@ +#!/bin/sh -eu +# SPDX-License-Identifier: Apache-2.0 +do_unmount() { + [ $# -eq 2 ] || return + local mounts= + while read ignore mount ignore; do + case $mount in + $1/*|$2/*) + mounts="$mount $mounts" + ;; + esac + done </proc/self/mounts + [ -z "$mounts" ] || umount $mounts +} + +do_unmount /run/k3s /var/lib/rancher/k3s + +ip link show | grep 'master cni0' | while read ignore iface ignore; do + iface=${iface%%@*} + [ -z "$iface" ] || ip link delete $iface +done + +ip link delete cni0 +ip link delete flannel.1 +rm -rf /var/lib/cni/recipes-containers/k3s/k3s/k3s.service new file +15
-0@@ -0,0 +1,27 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=process +Delegate=yes +# Having non-zero Limit*s causes performance problems due to accountingoverhead
+# in the kernel. We recommend using cgroups to do container-local accounting.recipes-containers/k3s/k3s/k3s.service new file +11
-0+LimitNOFILE=1048576 +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s server +recipes-containers/k3s/k3s_git.bb new file +2
-0@@ -0,0 +1,75 @@ +SUMMARY = "Production-Grade Container Scheduling and Management" +DESCRIPTION = "Lightweight Kubernetes, intended to be a fully compliantKubernetes."
recipes-containers/k3s/k3s_git.bb new file +3
-0+HOMEPAGE = "https://k3s.io/" +LICENSE = "Apache-2.0" +LIC_FILES_CHKSUM ="file://${S}/src/import/LICENSE;md5=2ee41112a44fe7014dce33e26468ba93"
recipes-containers/k3s/k3s_git.bb new file +9
-0+PV = "v1.18.9+k3s1-dirty" + +SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.18;name=k3s \ + file://k3s.service \ + file://k3s-agent.service \ + file://k3s-agent \ + file://k3s-clean \ + file://cni-containerd-net.conf \ +file://0001-Finding-host-local-in-usr-libexec.patch;patchdir=src/import \
recipes-containers/k3s/k3s_git.bb new file +13
-0+ " +SRC_URI[k3s.md5sum] = "363d3a08dc0b72ba6e6577964f6e94a5" +SRCREV_k3s = "630bebf94b9dce6b8cd3d402644ed023b3af8f90" + +inherit go +inherit goarch +inherit systemd + +PACKAGECONFIG = "" +PACKAGECONFIG[upx] = ",,upx-native" +GO_IMPORT = "import" +GO_BUILD_LDFLAGS = "-X github.com/rancher/k3s/pkg/version.Version=${PV} \ + -Xgithub.com/rancher/k3s/pkg/version.GitCommit=${@d.getVar('SRCREV_k3s', d, 1)[:8]} \
recipes-containers/k3s/k3s_git.bb new file +6
-0+ -w -s \ + " +BIN_PREFIX ?= "${exec_prefix}/local" + +do_compile() { + exportGOPATH="${S}/src/import/.gopath:${S}/src/import/vendor:${STAGING_DIR_TARGET}/${prefix}/local/go"
recipes-containers/k3s/k3s_git.bb new file +12
-1+ export CGO_ENABLED="1" + export GOFLAGS="-mod=vendor" + cd ${S}/src/import + ${GO} build -tags providerless -ldflags "${GO_BUILD_LDFLAGS}" -o ./dist/artifacts/k3s ./cmd/server/main.go + # Use UPX if it is enabled (and thus exists) to compress binary + if command -v upx > /dev/null 2>&1; then + upx -9 ./dist/artifacts/k3s + fi +} +do_install() { + install -d "${D}${BIN_PREFIX}/bin" + install -m 755 "${S}/src/import/dist/artifacts/k3s""${D}${BIN_PREFIX}/bin"
recipes-containers/k3s/k3s_git.bb new file +5
-0+ ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/crictl" + ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/ctr" + ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/kubectl" + install -m 755 "${WORKDIR}/k3s-clean" "${D}${BIN_PREFIX}/bin" + install -D -m 0644 "${WORKDIR}/cni-containerd-net.conf"
Quoted 2 lines
"
${D}/${sysconfdir}/cni/net.d/10-containerd-net.conf"
+ if
I'm going to abstract the networking configuration into a kubernetes-networking package, so we can share it amongst the various recipes, and have a way to control whether or not this configuration is installed. That allows an easy way to
both share and override the networking.
Quoted 28 lines
${@bb.utils.contains('DISTRO_FEATURES','systemd','true','false',d)}; then
+ install -D -m 0644 "${WORKDIR}/k3s.service"
"${D}${systemd_system_unitdir}/k3s.service"
+ install -D -m 0644 "${WORKDIR}/k3s-agent.service"
"${D}${systemd_system_unitdir}/k3s-agent.service"
+ sed -i
"s#\(Exec\)\(.*\)=\(.*\)\(k3s\)#\1\2=${BIN_PREFIX}/bin/\4#g" "${D}${systemd_system_unitdir}/k3s.service" "${D}${systemd_system_unitdir}/k3s-agent.service"
+ install -m 755 "${WORKDIR}/k3s-agent"
"${D}${BIN_PREFIX}/bin"recipes-containers/k3s/k3s_git.bb new file +6
-0+ fi +} + +PACKAGES =+ "${PN}-server ${PN}-agent" + +SYSTEMD_PACKAGES ="${@bb.utils.contains('DISTRO_FEATURES','systemd','${PN}-server ${PN}-agent','',d)}"
+SYSTEMD_SERVICE_${PN}-server = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s.service','',d)}" +SYSTEMD_SERVICE_${PN}-agent = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s-agent.service','',d)}"recipes-containers/k3s/k3s_git.bb new file +5
-0+SYSTEMD_AUTO_ENABLE_${PN}-agent = "disable" + +FILES_${PN}-agent = "${BIN_PREFIX}/bin/k3s-agent" + +RDEPENDS_${PN} = "cni conntrack-tools coreutils findutils iproute2
Quoted 1 line
ipset virtual/containerd"
I'll also take care of getting ipset in a place where we don't have to add extra layer
dependencies.
Bruce
Quoted 13 lines
recipes-containers/k3s/k3s_git.bb new file +7
-1+RDEPENDS_${PN}-server = "${PN}" +RDEPENDS_${PN}-agent = "${PN}" + +RCONFLICTS_${PN} = "kubectl" + +INHIBIT_PACKAGE_STRIP = "1" +INSANE_SKIP_${PN} += "ldflags already-stripped" --2.20.1
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 1 line · Bruce Ashfield
Thanks for the updated series, see some comments inline.
Awesome, I am happy that you liked it!
Quoted 2 lines
Thanks for the README. It's a good start and pretty much all we need for now.
I realized I had forgotten to run markdownlint for it, so a slightly updated version is included in the upcoming patch update.
Quoted 2 lines
I'll be doing some build and runtime testing and can do some tweaks as required.
Perfect!
Quoted 3 lines
For these scripts, we have the license, but not a copyright. Which should be ok, but are the scripts completely written by you (or someone at your company?), if so, it is a good idea to put a
copyright header on the scripts, so we can know the origin.
I have added that now; I am quite sure that Erik is the author of those. By the look of the coding style, it sure seems like it. (I have sent a text to him just to verify and if I am wrong in my assumption here, I will update accordingly.)
Quoted 4 lines
I'm going to abstract the networking configuration into a kubernetes-networking package, so we can share it amongst the various recipes, and have a way to control whether or not this configuration
is installed. That allows an easy way to both share and override the
networking.
I like that!
Quoted 2 lines
I'll also take care of getting ipset in a place where we don't have to add extra layer dependencies.
Perfect, thank you!
Following this reply, I will send the latest updates in a separate e-mail.
BR,
/Joakim
A message that is not in the archive (sent off-list or not yet received): [email protected]
--- recipes-containers/k3s/README.md | 30 +++++ ...01-Finding-host-local-in-usr-libexec.patch | 27 +++++ .../k3s/k3s/cni-containerd-net.conf | 24 ++++ recipes-containers/k3s/k3s/k3s-agent | 103 ++++++++++++++++++ recipes-containers/k3s/k3s/k3s-agent.service | 26 +++++ recipes-containers/k3s/k3s/k3s-clean | 29 +++++ recipes-containers/k3s/k3s/k3s.service | 27 +++++ recipes-containers/k3s/k3s_git.bb | 75 +++++++++++++ 8 files changed, 341 insertions(+) create mode 100644 recipes-containers/k3s/README.md create mode 100644 recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch create mode 100644 recipes-containers/k3s/k3s/cni-containerd-net.conf create mode 100755 recipes-containers/k3s/k3s/k3s-agent create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service create mode 100755 recipes-containers/k3s/k3s/k3s-clean create mode 100644 recipes-containers/k3s/k3s/k3s.service create mode 100644 recipes-containers/k3s/k3s_git.bb
recipes-containers/k3s/README.md new file +30 -0
@@ -0,0 +1,30 @@ +# k3s: Lightweight Kubernetes + +Rancher's [k3s](https://k3s.io/), available under +[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides +lightweight Kubernetes suitable for small/edge devices. There are use cases +where the +[installation procedures provided by Rancher](https://rancher.com/docs/k3s/latest/en/installation/) +are not ideal but a bitbake-built version is what is needed. And only a few +mods to the [k3s source code](https://github.com/rancher/k3s) is needed to +accomplish that. + +## CNI + +By default, K3s will run with flannel as the CNI, using VXLAN as the default +backend. It is both possible to change the flannel backend and to change from +flannel to another CNI. + +Please see <https://rancher.com/docs/k3s/latest/en/installation/network-options/> +for further k3s networking details. + +## Configure and run a k3s agent + +The convenience script `k3s-agent` can be used to set up a k3s agent (service): + +```shell +k3s-agent -t <token> -s https://<master>:6443 +``` + +(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the +k3s master.)
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch new file +27 -0
@@ -0,0 +1,27 @@ +From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001 +From: Erik Jansson <[email protected]> +Date: Wed, 16 Oct 2019 15:07:48 +0200 +Subject: [PATCH] Finding host-local in /usr/libexec + +Upstream-status: Inappropriate [embedded specific] +Signed-off-by: <[email protected]> +--- + pkg/agent/config/config.go | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go +index b4296f360a..6af9dab895 100644 +--- a/pkg/agent/config/config.go ++++ b/pkg/agent/config/config.go +@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) { + return nil, err + } + +- hostLocal, err := exec.LookPath("host-local") ++ hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local") + if err != nil { + return nil, errors.Wrapf(err, "failed to find host-local") + } +-- +2.11.0 +
recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24 -0
@@ -0,0 +1,24 @@ +{ + "cniVersion": "0.4.0", + "name": "containerd-net", + "plugins": [ + { + "type": "bridge", + "bridge": "cni0", + "isGateway": true, + "ipMasq": true, + "promiscMode": true, + "ipam": { + "type": "host-local", + "subnet": "10.88.0.0/16", + "routes": [ + { "dst": "0.0.0.0/0" } + ] + } + }, + { + "type": "portmap", + "capabilities": {"portMappings": true} + } + ] +}
recipes-containers/k3s/k3s/k3s-agent new file +103 -0
103 lines not on this page · raw message
recipes-containers/k3s/k3s/k3s-agent.service new file +26 -0
@@ -0,0 +1,26 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes Agent +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=control-group +Delegate=yes +LimitNOFILE=infinity +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s agent +ExecStopPost=/usr/local/bin/k3s-clean +
recipes-containers/k3s/k3s/k3s-clean new file +29 -0
@@ -0,0 +1,29 @@ +#!/bin/sh -eu +# +# Copyright (C) 2020 Axis Communications AB +# +# SPDX-License-Identifier: Apache-2.0 + +do_unmount() { + [ $# -eq 2 ] || return + local mounts= + while read ignore mount ignore; do + case $mount in + $1/*|$2/*) + mounts="$mount $mounts" + ;; + esac + done </proc/self/mounts + [ -z "$mounts" ] || umount $mounts +} + +do_unmount /run/k3s /var/lib/rancher/k3s + +ip link show | grep 'master cni0' | while read ignore iface ignore; do + iface=${iface%%@*} + [ -z "$iface" ] || ip link delete $iface +done + +ip link delete cni0 +ip link delete flannel.1 +rm -rf /var/lib/cni/
recipes-containers/k3s/k3s/k3s.service new file +27 -0
@@ -0,0 +1,27 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=process +Delegate=yes +# Having non-zero Limit*s causes performance problems due to accounting overhead +# in the kernel. We recommend using cgroups to do container-local accounting. +LimitNOFILE=1048576 +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s server +
recipes-containers/k3s/k3s_git.bb new file +75 -0
75 lines not on this page · raw message
Signature · 1 line
2.20.1
Quoted 5 lines · Joakim Roubert
I have added that now; I am quite sure that Erik is the author of those. By the look of the coding style, it sure seems like it. (I have sent a text to him just to verify and if I am wrong in my assumption here, I will update accordingly.)
Update accordingly: the last couple of lines in k3s-clean are from install.sh's create_killall.sh function. I have now added a comment that clearly states that. Hopefully now everything is in order so that we do not claim someone else's cred.
The patchset with this update follows this reply in a separate e-mail.
BR,
/Joakim
A message that is not in the archive (sent off-list or not yet received): [email protected]
--- recipes-containers/k3s/README.md | 30 +++++ ...01-Finding-host-local-in-usr-libexec.patch | 27 +++++ .../k3s/k3s/cni-containerd-net.conf | 24 ++++ recipes-containers/k3s/k3s/k3s-agent | 103 ++++++++++++++++++ recipes-containers/k3s/k3s/k3s-agent.service | 26 +++++ recipes-containers/k3s/k3s/k3s-clean | 30 +++++ recipes-containers/k3s/k3s/k3s.service | 27 +++++ recipes-containers/k3s/k3s_git.bb | 75 +++++++++++++ 8 files changed, 342 insertions(+) create mode 100644 recipes-containers/k3s/README.md create mode 100644 recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch create mode 100644 recipes-containers/k3s/k3s/cni-containerd-net.conf create mode 100755 recipes-containers/k3s/k3s/k3s-agent create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service create mode 100755 recipes-containers/k3s/k3s/k3s-clean create mode 100644 recipes-containers/k3s/k3s/k3s.service create mode 100644 recipes-containers/k3s/k3s_git.bb
recipes-containers/k3s/README.md new file +30 -0
@@ -0,0 +1,30 @@ +# k3s: Lightweight Kubernetes + +Rancher's [k3s](https://k3s.io/), available under +[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides +lightweight Kubernetes suitable for small/edge devices. There are use cases +where the +[installation procedures provided by Rancher](https://rancher.com/docs/k3s/latest/en/installation/) +are not ideal but a bitbake-built version is what is needed. And only a few +mods to the [k3s source code](https://github.com/rancher/k3s) is needed to +accomplish that. + +## CNI + +By default, K3s will run with flannel as the CNI, using VXLAN as the default +backend. It is both possible to change the flannel backend and to change from +flannel to another CNI. + +Please see <https://rancher.com/docs/k3s/latest/en/installation/network-options/> +for further k3s networking details. + +## Configure and run a k3s agent + +The convenience script `k3s-agent` can be used to set up a k3s agent (service): + +```shell +k3s-agent -t <token> -s https://<master>:6443 +``` + +(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the +k3s master.)
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch new file +27 -0
@@ -0,0 +1,27 @@ +From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001 +From: Erik Jansson <[email protected]> +Date: Wed, 16 Oct 2019 15:07:48 +0200 +Subject: [PATCH] Finding host-local in /usr/libexec + +Upstream-status: Inappropriate [embedded specific] +Signed-off-by: <[email protected]> +--- + pkg/agent/config/config.go | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go +index b4296f360a..6af9dab895 100644 +--- a/pkg/agent/config/config.go ++++ b/pkg/agent/config/config.go +@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) { + return nil, err + } + +- hostLocal, err := exec.LookPath("host-local") ++ hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local") + if err != nil { + return nil, errors.Wrapf(err, "failed to find host-local") + } +-- +2.11.0 +
recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24 -0
@@ -0,0 +1,24 @@ +{ + "cniVersion": "0.4.0", + "name": "containerd-net", + "plugins": [ + { + "type": "bridge", + "bridge": "cni0", + "isGateway": true, + "ipMasq": true, + "promiscMode": true, + "ipam": { + "type": "host-local", + "subnet": "10.88.0.0/16", + "routes": [ + { "dst": "0.0.0.0/0" } + ] + } + }, + { + "type": "portmap", + "capabilities": {"portMappings": true} + } + ] +}
recipes-containers/k3s/k3s/k3s-agent new file +103 -0
103 lines not on this page · raw message
recipes-containers/k3s/k3s/k3s-agent.service new file +26 -0
@@ -0,0 +1,26 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes Agent +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=control-group +Delegate=yes +LimitNOFILE=infinity +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s agent +ExecStopPost=/usr/local/bin/k3s-clean +
recipes-containers/k3s/k3s/k3s-clean new file +30 -0
@@ -0,0 +1,30 @@ +#!/bin/sh -eu +# +# Copyright (C) 2020 Axis Communications AB +# +# SPDX-License-Identifier: Apache-2.0 + +do_unmount() { + [ $# -eq 2 ] || return + local mounts= + while read ignore mount ignore; do + case $mount in + $1/*|$2/*) + mounts="$mount $mounts" + ;; + esac + done </proc/self/mounts + [ -z "$mounts" ] || umount $mounts +} + +do_unmount /run/k3s /var/lib/rancher/k3s + +# The lines below come from install.sh's create_killall() function: +ip link show 2>/dev/null | grep 'master cni0' | while read ignore iface ignore; do + iface=${iface%%@*} + [ -z "$iface" ] || ip link delete $iface +done + +ip link delete cni0 +ip link delete flannel.1 +rm -rf /var/lib/cni/
recipes-containers/k3s/k3s/k3s.service new file +27 -0
@@ -0,0 +1,27 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=process +Delegate=yes +# Having non-zero Limit*s causes performance problems due to accounting overhead +# in the kernel. We recommend using cgroups to do container-local accounting. +LimitNOFILE=1048576 +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s server +
recipes-containers/k3s/k3s_git.bb new file +75 -0
75 lines not on this page · raw message
Signature · 1 line
2.20.1
Something has happened to this version of the patch in transit.
When I save and apply it using my normal workflow, the patch is corrupted.
Can you re-send it with a "v4" in the subject ? That will hopefully let it escape gmail's conversation filter.
I'll try and hack the patch by hand, but would like to see a v4 for comparison.
I'm currently importing some recipes for temporary support in meta-virt and creating a networking recipe while I wait for v4.
Cheers,
Bruce
Quoted 461 lines · Joakim Roubert
Change-Id Id1c52727593bc5ea8d0cd2de192faa44304d7a45Signed-off-by Joakim Roubert <[email protected]>--- recipes-containers/k3s/README.md | 30 +++++ ...01-Finding-host-local-in-usr-libexec.patch | 27 +++++ .../k3s/k3s/cni-containerd-net.conf | 24 ++++ recipes-containers/k3s/k3s/k3s-agent | 103 ++++++++++++++++++ recipes-containers/k3s/k3s/k3s-agent.service | 26 +++++ recipes-containers/k3s/k3s/k3s-clean | 30 +++++ recipes-containers/k3s/k3s/k3s.service | 27 +++++ recipes-containers/k3s/k3s_git.bb | 75 +++++++++++++ 8 files changed, 342 insertions(+)create mode 100644
recipes-containers/k3s/README.md
create mode 100644recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch
create mode 100644recipes-containers/k3s/k3s/cni-containerd-net.confcreate mode 100755 recipes-containers/k3s/k3s/k3s-agent create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service create mode 100755 recipes-containers/k3s/k3s/k3s-clean create mode 100644 recipes-containers/k3s/k3s/k3s.service create mode 100644 recipes-containers/k3s/k3s_git.bbrecipes-containers/k3s/README.md new file +7
-0@@ -0,0 +1,30 @@ +# k3s: Lightweight Kubernetes + +Rancher's [k3s](https://k3s.io/), available under +[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides +lightweight Kubernetes suitable for small/edge devices. There are use cases +where the +[installation procedures provided byRancher](https://rancher.com/docs/k3s/latest/en/installation/)
recipes-containers/k3s/README.md new file +7
-0+are not ideal but a bitbake-built version is what is needed. And only a few +mods to the [k3s source code](https://github.com/rancher/k3s) is needed to +accomplish that. + +## CNI + +By default, K3s will run with flannel as the CNI, using VXLAN as thedefault
+backend. It is both possible to change the flannel backend and to change fromrecipes-containers/k3s/README.md new file +3
-0+flannel to another CNI. + +Please see<https://rancher.com/docs/k3s/latest/en/installation/network-options/>
recipes-containers/k3s/README.md new file +5
-0+for further k3s networking details. + +## Configure and run a k3s agent + +The convenience script `k3s-agent` can be used to set up a k3s agent(service):
recipes-containers/k3s/README.md new file +7
-0+ +```shell +k3s-agent -t <token> -s https://<master>:6443 +``` + +(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the +k3s master.)diff --git
a/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patchb/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patchnew file mode 100644
index 0000000..8205d73
---/dev/null
+++b/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patchrecipes-containers/k3s/README.md new file +27
-0@@ -0,0 +1,27 @@ +From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001 +From: Erik Jansson <[email protected]> +Date: Wed, 16 Oct 2019 15:07:48 +0200 +Subject: [PATCH] Finding host-local in /usr/libexec + +Upstream-status: Inappropriate [embedded specific] +Signed-off-by: <[email protected]> +--- + pkg/agent/config/config.go | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go +index b4296f360a..6af9dab895 100644 +--- a/pkg/agent/config/config.go ++++ b/pkg/agent/config/config.go +@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) { + return nil, err + } + +- hostLocal, err := exec.LookPath("host-local") ++ hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local") + if err != nil { + return nil, errors.Wrapf(err, "failed to find host-local") + } +-- +2.11.0 +recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24
-0@@ -0,0 +1,24 @@ +{ + "cniVersion": "0.4.0", + "name": "containerd-net", + "plugins": [ + { + "type": "bridge", + "bridge": "cni0", + "isGateway": true, + "ipMasq": true, + "promiscMode": true, + "ipam": { + "type": "host-local", + "subnet": "10.88.0.0/16", + "routes": [ + { "dst": "0.0.0.0/0" } + ] + } + }, + { + "type": "portmap", + "capabilities": {"portMappings": true} + } + ] +}recipes-containers/k3s/k3s/k3s-agent new file +14
-0@@ -0,0 +1,103 @@ +#!/bin/sh -eu +# +# Copyright (C) 2020 Axis Communications AB +# +# SPDX-License-Identifier: Apache-2.0 + +ENV_CONF=/etc/systemd/system/k3s-agent.service.d/10-env.conf + +usage() { + echo " +USAGE: + ${0##*/} [OPTIONS] +OPTIONS: + --token value, -t value Token to use for authentication[\$K3S_TOKEN]
+ --token-file value Token file to use for
authentication [\$K3S_TOKEN_FILE]recipes-containers/k3s/k3s/k3s-agent new file +3
-0+ --server value, -s value Server to connect to [\$K3S_URL] + --node-name value Node name [\$K3S_NODE_NAME] + --resolv-conf value Kubelet resolv.conf file[\$K3S_RESOLV_CONF]
+ --cluster-secret value Shared secret used to bootstrap
a cluster [\$K3S_CLUSTER_SECRET]recipes-containers/k3s/k3s/k3s-agent new file +84
-0+ -h print this +" +} + +[ $# -gt 0 ] || { + usage + exit +} + +case $1 in + -*) + ;; + *) + usage + exit 1 + ;; +esac + +rm -f $ENV_CONF +mkdir -p ${ENV_CONF%/*} +echo [Service] > $ENV_CONF + +while getopts "t:s:-:h" opt; do + case $opt in + h) + usage + exit + ;; + t) + VAR_NAME=K3S_TOKEN + ;; + s) + VAR_NAME=K3S_URL + ;; + -) + [ $# -ge $OPTIND ] || { + usage + exit 1 + } + opt=$OPTARG + eval OPTARG='$'$OPTIND + OPTIND=$(($OPTIND + 1)) + case $opt in + token) + VAR_NAME=K3S_TOKEN + ;; + token-file) + VAR_NAME=K3S_TOKEN_FILE + ;; + server) + VAR_NAME=K3S_URL + ;; + node-name) + VAR_NAME=K3S_NODE_NAME + ;; + resolv-conf) + VAR_NAME=K3S_RESOLV_CONF + ;; + cluster-secret) + VAR_NAME=K3S_CLUSTER_SECRET + ;; + help) + usage + exit + ;; + *) + usage + exit 1 + ;; + esac + ;; + *) + usage + exit 1 + ;; + esac + echo Environment=$VAR_NAME=$OPTARG >> $ENV_CONF +done + +chmod 0644 $ENV_CONF +rm -rf /var/lib/rancher/k3s/agent +systemctl daemon-reload +systemctl restart k3s-agent +systemctl enable k3s-agent.servicerecipes-containers/k3s/k3s/k3s-agent.service new file +26
-0@@ -0,0 +1,26 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes Agent +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=control-group +Delegate=yes +LimitNOFILE=infinity +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s agent +ExecStopPost=/usr/local/bin/k3s-clean +recipes-containers/k3s/k3s/k3s-clean new file +23
-0@@ -0,0 +1,30 @@ +#!/bin/sh -eu +# +# Copyright (C) 2020 Axis Communications AB +# +# SPDX-License-Identifier: Apache-2.0 + +do_unmount() { + [ $# -eq 2 ] || return + local mounts= + while read ignore mount ignore; do + case $mount in + $1/*|$2/*) + mounts="$mount $mounts" + ;; + esac + done </proc/self/mounts + [ -z "$mounts" ] || umount $mounts +} + +do_unmount /run/k3s /var/lib/rancher/k3s + +# The lines below come from install.sh's create_killall() function: +ip link show 2>/dev/null | grep 'master cni0' | while read ignore ifaceignore; do
recipes-containers/k3s/k3s/k3s-clean new file +7
-0+ iface=${iface%%@*} + [ -z "$iface" ] || ip link delete $iface +done + +ip link delete cni0 +ip link delete flannel.1 +rm -rf /var/lib/cni/recipes-containers/k3s/k3s/k3s.service new file +15
-0@@ -0,0 +1,27 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=process +Delegate=yes +# Having non-zero Limit*s causes performance problems due to accountingoverhead
+# in the kernel. We recommend using cgroups to do container-local accounting.recipes-containers/k3s/k3s/k3s.service new file +11
-0+LimitNOFILE=1048576 +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s server +recipes-containers/k3s/k3s_git.bb new file +2
-0@@ -0,0 +1,75 @@ +SUMMARY = "Production-Grade Container Scheduling and Management" +DESCRIPTION = "Lightweight Kubernetes, intended to be a fully compliantKubernetes."
recipes-containers/k3s/k3s_git.bb new file +3
-0+HOMEPAGE = "https://k3s.io/" +LICENSE = "Apache-2.0" +LIC_FILES_CHKSUM ="file://${S}/src/import/LICENSE;md5=2ee41112a44fe7014dce33e26468ba93"
recipes-containers/k3s/k3s_git.bb new file +9
-0+PV = "v1.18.9+k3s1-dirty" + +SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.18;name=k3s \ + file://k3s.service \ + file://k3s-agent.service \ + file://k3s-agent \ + file://k3s-clean \ + file://cni-containerd-net.conf \ +file://0001-Finding-host-local-in-usr-libexec.patch;patchdir=src/import \
recipes-containers/k3s/k3s_git.bb new file +13
-0+ " +SRC_URI[k3s.md5sum] = "363d3a08dc0b72ba6e6577964f6e94a5" +SRCREV_k3s = "630bebf94b9dce6b8cd3d402644ed023b3af8f90" + +inherit go +inherit goarch +inherit systemd + +PACKAGECONFIG = "" +PACKAGECONFIG[upx] = ",,upx-native" +GO_IMPORT = "import" +GO_BUILD_LDFLAGS = "-X github.com/rancher/k3s/pkg/version.Version=${PV} \ + -Xgithub.com/rancher/k3s/pkg/version.GitCommit=${@d.getVar('SRCREV_k3s', d, 1)[:8]} \
recipes-containers/k3s/k3s_git.bb new file +6
-0+ -w -s \ + " +BIN_PREFIX ?= "${exec_prefix}/local" + +do_compile() { + exportGOPATH="${S}/src/import/.gopath:${S}/src/import/vendor:${STAGING_DIR_TARGET}/${prefix}/local/go"
recipes-containers/k3s/k3s_git.bb new file +12
-1+ export CGO_ENABLED="1" + export GOFLAGS="-mod=vendor" + cd ${S}/src/import + ${GO} build -tags providerless -ldflags "${GO_BUILD_LDFLAGS}" -o ./dist/artifacts/k3s ./cmd/server/main.go + # Use UPX if it is enabled (and thus exists) to compress binary + if command -v upx > /dev/null 2>&1; then + upx -9 ./dist/artifacts/k3s + fi +} +do_install() { + install -d "${D}${BIN_PREFIX}/bin" + install -m 755 "${S}/src/import/dist/artifacts/k3s""${D}${BIN_PREFIX}/bin"
recipes-containers/k3s/k3s_git.bb new file +5
-0+ ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/crictl" + ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/ctr" + ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/kubectl" + install -m 755 "${WORKDIR}/k3s-clean" "${D}${BIN_PREFIX}/bin" + install -D -m 0644 "${WORKDIR}/cni-containerd-net.conf""
${D}/${sysconfdir}/cni/net.d/10-containerd-net.conf"
+ if
${@bb.utils.contains('DISTRO_FEATURES','systemd','true','false',d)}; then
+ install -D -m 0644 "${WORKDIR}/k3s.service"
"${D}${systemd_system_unitdir}/k3s.service"
+ install -D -m 0644 "${WORKDIR}/k3s-agent.service"
"${D}${systemd_system_unitdir}/k3s-agent.service"
+ sed -i
"s#\(Exec\)\(.*\)=\(.*\)\(k3s\)#\1\2=${BIN_PREFIX}/bin/\4#g" "${D}${systemd_system_unitdir}/k3s.service" "${D}${systemd_system_unitdir}/k3s-agent.service"
+ install -m 755 "${WORKDIR}/k3s-agent"
"${D}${BIN_PREFIX}/bin"recipes-containers/k3s/k3s_git.bb new file +6
-0+ fi +} + +PACKAGES =+ "${PN}-server ${PN}-agent" + +SYSTEMD_PACKAGES ="${@bb.utils.contains('DISTRO_FEATURES','systemd','${PN}-server ${PN}-agent','',d)}"
+SYSTEMD_SERVICE_${PN}-server = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s.service','',d)}" +SYSTEMD_SERVICE_${PN}-agent = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s-agent.service','',d)}"recipes-containers/k3s/k3s_git.bb new file +5
-0+SYSTEMD_AUTO_ENABLE_${PN}-agent = "disable" + +FILES_${PN}-agent = "${BIN_PREFIX}/bin/k3s-agent" + +RDEPENDS_${PN} = "cni conntrack-tools coreutils findutils iproute2ipset virtual/containerd"
recipes-containers/k3s/k3s_git.bb new file +7
-1+RDEPENDS_${PN}-server = "${PN}" +RDEPENDS_${PN}-agent = "${PN}" + +RCONFLICTS_${PN} = "kubectl" + +INHIBIT_PACKAGE_STRIP = "1" +INSANE_SKIP_${PN} += "ldflags already-stripped" --2.20.1
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 3 lines · Bruce Ashfield
Can you re-send it with a "v4" in the subject ? That will hopefully let it escape gmail's conversation filter.
Sure! No problem.
BR,
/Joakim
PATCH v4 1/1 Adding k3s recipe
In reply to Bruce Ashfield--- recipes-containers/k3s/README.md | 30 +++++ ...01-Finding-host-local-in-usr-libexec.patch | 27 +++++ .../k3s/k3s/cni-containerd-net.conf | 24 ++++ recipes-containers/k3s/k3s/k3s-agent | 103 ++++++++++++++++++ recipes-containers/k3s/k3s/k3s-agent.service | 26 +++++ recipes-containers/k3s/k3s/k3s-clean | 30 +++++ recipes-containers/k3s/k3s/k3s.service | 27 +++++ recipes-containers/k3s/k3s_git.bb | 75 +++++++++++++ 8 files changed, 342 insertions(+) create mode 100644 recipes-containers/k3s/README.md create mode 100644 recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch create mode 100644 recipes-containers/k3s/k3s/cni-containerd-net.conf create mode 100755 recipes-containers/k3s/k3s/k3s-agent create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service create mode 100755 recipes-containers/k3s/k3s/k3s-clean create mode 100644 recipes-containers/k3s/k3s/k3s.service create mode 100644 recipes-containers/k3s/k3s_git.bb
recipes-containers/k3s/README.md new file +30 -0
@@ -0,0 +1,30 @@ +# k3s: Lightweight Kubernetes + +Rancher's [k3s](https://k3s.io/), available under +[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides +lightweight Kubernetes suitable for small/edge devices. There are use cases +where the +[installation procedures provided by Rancher](https://rancher.com/docs/k3s/latest/en/installation/) +are not ideal but a bitbake-built version is what is needed. And only a few +mods to the [k3s source code](https://github.com/rancher/k3s) is needed to +accomplish that. + +## CNI + +By default, K3s will run with flannel as the CNI, using VXLAN as the default +backend. It is both possible to change the flannel backend and to change from +flannel to another CNI. + +Please see <https://rancher.com/docs/k3s/latest/en/installation/network-options/> +for further k3s networking details. + +## Configure and run a k3s agent + +The convenience script `k3s-agent` can be used to set up a k3s agent (service): + +```shell +k3s-agent -t <token> -s https://<master>:6443 +``` + +(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the +k3s master.)
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch new file +27 -0
@@ -0,0 +1,27 @@ +From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001 +From: Erik Jansson <[email protected]> +Date: Wed, 16 Oct 2019 15:07:48 +0200 +Subject: [PATCH] Finding host-local in /usr/libexec + +Upstream-status: Inappropriate [embedded specific] +Signed-off-by: <[email protected]> +--- + pkg/agent/config/config.go | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go +index b4296f360a..6af9dab895 100644 +--- a/pkg/agent/config/config.go ++++ b/pkg/agent/config/config.go +@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) { + return nil, err + } + +- hostLocal, err := exec.LookPath("host-local") ++ hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local") + if err != nil { + return nil, errors.Wrapf(err, "failed to find host-local") + } +-- +2.11.0 +
recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24 -0
@@ -0,0 +1,24 @@ +{ + "cniVersion": "0.4.0", + "name": "containerd-net", + "plugins": [ + { + "type": "bridge", + "bridge": "cni0", + "isGateway": true, + "ipMasq": true, + "promiscMode": true, + "ipam": { + "type": "host-local", + "subnet": "10.88.0.0/16", + "routes": [ + { "dst": "0.0.0.0/0" } + ] + } + }, + { + "type": "portmap", + "capabilities": {"portMappings": true} + } + ] +}
recipes-containers/k3s/k3s/k3s-agent new file +103 -0
103 lines not on this page · raw message
recipes-containers/k3s/k3s/k3s-agent.service new file +26 -0
@@ -0,0 +1,26 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes Agent +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=control-group +Delegate=yes +LimitNOFILE=infinity +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s agent +ExecStopPost=/usr/local/bin/k3s-clean +
recipes-containers/k3s/k3s/k3s-clean new file +30 -0
@@ -0,0 +1,30 @@ +#!/bin/sh -eu +# +# Copyright (C) 2020 Axis Communications AB +# +# SPDX-License-Identifier: Apache-2.0 + +do_unmount() { + [ $# -eq 2 ] || return + local mounts= + while read ignore mount ignore; do + case $mount in + $1/*|$2/*) + mounts="$mount $mounts" + ;; + esac + done </proc/self/mounts + [ -z "$mounts" ] || umount $mounts +} + +do_unmount /run/k3s /var/lib/rancher/k3s + +# The lines below come from install.sh's create_killall() function: +ip link show 2>/dev/null | grep 'master cni0' | while read ignore iface ignore; do + iface=${iface%%@*} + [ -z "$iface" ] || ip link delete $iface +done + +ip link delete cni0 +ip link delete flannel.1 +rm -rf /var/lib/cni/
recipes-containers/k3s/k3s/k3s.service new file +27 -0
@@ -0,0 +1,27 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=process +Delegate=yes +# Having non-zero Limit*s causes performance problems due to accounting overhead +# in the kernel. We recommend using cgroups to do container-local accounting. +LimitNOFILE=1048576 +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s server +
recipes-containers/k3s/k3s_git.bb new file +75 -0
75 lines not on this page · raw message
Signature · 1 line
2.20.1
I just merged 3 patches from the mailing list, using my standard workflow (save mbox, apply, buid, push).
But with your patch, saved the same way. I get:
------------
build [/home/bruc...ualization]> git am -s k3.mbox
Applying: Adding k3s recipe
.git/rebase-apply/patch:35: trailing whitespace. [installation procedures provided by
error: corrupt patch at line 36
Patch failed at 0001 Adding k3s recipe
hint: Use 'git am --show-current-patch' to see the failed patch When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort".
----------
I've tried hacking the patch by hand to apply, but it is just making things worse. Is there another way
you can send the patch ?
Bruce
Quoted 461 lines · Joakim Roubert
Change-Id Id1c52727593bc5ea8d0cd2de192faa44304d7a45Signed-off-by Joakim Roubert <[email protected]>--- recipes-containers/k3s/README.md | 30 +++++ ...01-Finding-host-local-in-usr-libexec.patch | 27 +++++ .../k3s/k3s/cni-containerd-net.conf | 24 ++++ recipes-containers/k3s/k3s/k3s-agent | 103 ++++++++++++++++++ recipes-containers/k3s/k3s/k3s-agent.service | 26 +++++ recipes-containers/k3s/k3s/k3s-clean | 30 +++++ recipes-containers/k3s/k3s/k3s.service | 27 +++++ recipes-containers/k3s/k3s_git.bb | 75 +++++++++++++ 8 files changed, 342 insertions(+)create mode 100644
recipes-containers/k3s/README.md
create mode 100644recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch
create mode 100644recipes-containers/k3s/k3s/cni-containerd-net.confcreate mode 100755 recipes-containers/k3s/k3s/k3s-agent create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service create mode 100755 recipes-containers/k3s/k3s/k3s-clean create mode 100644 recipes-containers/k3s/k3s/k3s.service create mode 100644 recipes-containers/k3s/k3s_git.bbrecipes-containers/k3s/README.md new file +7
-0@@ -0,0 +1,30 @@ +# k3s: Lightweight Kubernetes + +Rancher's [k3s](https://k3s.io/), available under +[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides +lightweight Kubernetes suitable for small/edge devices. There are use cases +where the +[installation procedures provided byRancher](https://rancher.com/docs/k3s/latest/en/installation/)
recipes-containers/k3s/README.md new file +7
-0+are not ideal but a bitbake-built version is what is needed. And only a few +mods to the [k3s source code](https://github.com/rancher/k3s) is needed to +accomplish that. + +## CNI + +By default, K3s will run with flannel as the CNI, using VXLAN as thedefault
+backend. It is both possible to change the flannel backend and to change fromrecipes-containers/k3s/README.md new file +3
-0+flannel to another CNI. + +Please see<https://rancher.com/docs/k3s/latest/en/installation/network-options/>
recipes-containers/k3s/README.md new file +5
-0+for further k3s networking details. + +## Configure and run a k3s agent + +The convenience script `k3s-agent` can be used to set up a k3s agent(service):
recipes-containers/k3s/README.md new file +7
-0+ +```shell +k3s-agent -t <token> -s https://<master>:6443 +``` + +(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the +k3s master.)diff --git
a/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patchb/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patchnew file mode 100644
index 0000000..8205d73
---/dev/null
+++b/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patchrecipes-containers/k3s/README.md new file +27
-0@@ -0,0 +1,27 @@ +From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001 +From: Erik Jansson <[email protected]> +Date: Wed, 16 Oct 2019 15:07:48 +0200 +Subject: [PATCH] Finding host-local in /usr/libexec + +Upstream-status: Inappropriate [embedded specific] +Signed-off-by: <[email protected]> +--- + pkg/agent/config/config.go | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go +index b4296f360a..6af9dab895 100644 +--- a/pkg/agent/config/config.go ++++ b/pkg/agent/config/config.go +@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) { + return nil, err + } + +- hostLocal, err := exec.LookPath("host-local") ++ hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local") + if err != nil { + return nil, errors.Wrapf(err, "failed to find host-local") + } +-- +2.11.0 +recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24
-0@@ -0,0 +1,24 @@ +{ + "cniVersion": "0.4.0", + "name": "containerd-net", + "plugins": [ + { + "type": "bridge", + "bridge": "cni0", + "isGateway": true, + "ipMasq": true, + "promiscMode": true, + "ipam": { + "type": "host-local", + "subnet": "10.88.0.0/16", + "routes": [ + { "dst": "0.0.0.0/0" } + ] + } + }, + { + "type": "portmap", + "capabilities": {"portMappings": true} + } + ] +}recipes-containers/k3s/k3s/k3s-agent new file +14
-0@@ -0,0 +1,103 @@ +#!/bin/sh -eu +# +# Copyright (C) 2020 Axis Communications AB +# +# SPDX-License-Identifier: Apache-2.0 + +ENV_CONF=/etc/systemd/system/k3s-agent.service.d/10-env.conf + +usage() { + echo " +USAGE: + ${0##*/} [OPTIONS] +OPTIONS: + --token value, -t value Token to use for authentication[\$K3S_TOKEN]
+ --token-file value Token file to use for
authentication [\$K3S_TOKEN_FILE]recipes-containers/k3s/k3s/k3s-agent new file +3
-0+ --server value, -s value Server to connect to [\$K3S_URL] + --node-name value Node name [\$K3S_NODE_NAME] + --resolv-conf value Kubelet resolv.conf file[\$K3S_RESOLV_CONF]
+ --cluster-secret value Shared secret used to bootstrap
a cluster [\$K3S_CLUSTER_SECRET]recipes-containers/k3s/k3s/k3s-agent new file +84
-0+ -h print this +" +} + +[ $# -gt 0 ] || { + usage + exit +} + +case $1 in + -*) + ;; + *) + usage + exit 1 + ;; +esac + +rm -f $ENV_CONF +mkdir -p ${ENV_CONF%/*} +echo [Service] > $ENV_CONF + +while getopts "t:s:-:h" opt; do + case $opt in + h) + usage + exit + ;; + t) + VAR_NAME=K3S_TOKEN + ;; + s) + VAR_NAME=K3S_URL + ;; + -) + [ $# -ge $OPTIND ] || { + usage + exit 1 + } + opt=$OPTARG + eval OPTARG='$'$OPTIND + OPTIND=$(($OPTIND + 1)) + case $opt in + token) + VAR_NAME=K3S_TOKEN + ;; + token-file) + VAR_NAME=K3S_TOKEN_FILE + ;; + server) + VAR_NAME=K3S_URL + ;; + node-name) + VAR_NAME=K3S_NODE_NAME + ;; + resolv-conf) + VAR_NAME=K3S_RESOLV_CONF + ;; + cluster-secret) + VAR_NAME=K3S_CLUSTER_SECRET + ;; + help) + usage + exit + ;; + *) + usage + exit 1 + ;; + esac + ;; + *) + usage + exit 1 + ;; + esac + echo Environment=$VAR_NAME=$OPTARG >> $ENV_CONF +done + +chmod 0644 $ENV_CONF +rm -rf /var/lib/rancher/k3s/agent +systemctl daemon-reload +systemctl restart k3s-agent +systemctl enable k3s-agent.servicerecipes-containers/k3s/k3s/k3s-agent.service new file +26
-0@@ -0,0 +1,26 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes Agent +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=control-group +Delegate=yes +LimitNOFILE=infinity +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s agent +ExecStopPost=/usr/local/bin/k3s-clean +recipes-containers/k3s/k3s/k3s-clean new file +23
-0@@ -0,0 +1,30 @@ +#!/bin/sh -eu +# +# Copyright (C) 2020 Axis Communications AB +# +# SPDX-License-Identifier: Apache-2.0 + +do_unmount() { + [ $# -eq 2 ] || return + local mounts= + while read ignore mount ignore; do + case $mount in + $1/*|$2/*) + mounts="$mount $mounts" + ;; + esac + done </proc/self/mounts + [ -z "$mounts" ] || umount $mounts +} + +do_unmount /run/k3s /var/lib/rancher/k3s + +# The lines below come from install.sh's create_killall() function: +ip link show 2>/dev/null | grep 'master cni0' | while read ignore ifaceignore; do
recipes-containers/k3s/k3s/k3s-clean new file +7
-0+ iface=${iface%%@*} + [ -z "$iface" ] || ip link delete $iface +done + +ip link delete cni0 +ip link delete flannel.1 +rm -rf /var/lib/cni/recipes-containers/k3s/k3s/k3s.service new file +15
-0@@ -0,0 +1,27 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=process +Delegate=yes +# Having non-zero Limit*s causes performance problems due to accountingoverhead
+# in the kernel. We recommend using cgroups to do container-local accounting.recipes-containers/k3s/k3s/k3s.service new file +11
-0+LimitNOFILE=1048576 +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s server +recipes-containers/k3s/k3s_git.bb new file +2
-0@@ -0,0 +1,75 @@ +SUMMARY = "Production-Grade Container Scheduling and Management" +DESCRIPTION = "Lightweight Kubernetes, intended to be a fully compliantKubernetes."
recipes-containers/k3s/k3s_git.bb new file +3
-0+HOMEPAGE = "https://k3s.io/" +LICENSE = "Apache-2.0" +LIC_FILES_CHKSUM ="file://${S}/src/import/LICENSE;md5=2ee41112a44fe7014dce33e26468ba93"
recipes-containers/k3s/k3s_git.bb new file +9
-0+PV = "v1.18.9+k3s1-dirty" + +SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.18;name=k3s \ + file://k3s.service \ + file://k3s-agent.service \ + file://k3s-agent \ + file://k3s-clean \ + file://cni-containerd-net.conf \ +file://0001-Finding-host-local-in-usr-libexec.patch;patchdir=src/import \
recipes-containers/k3s/k3s_git.bb new file +13
-0+ " +SRC_URI[k3s.md5sum] = "363d3a08dc0b72ba6e6577964f6e94a5" +SRCREV_k3s = "630bebf94b9dce6b8cd3d402644ed023b3af8f90" + +inherit go +inherit goarch +inherit systemd + +PACKAGECONFIG = "" +PACKAGECONFIG[upx] = ",,upx-native" +GO_IMPORT = "import" +GO_BUILD_LDFLAGS = "-X github.com/rancher/k3s/pkg/version.Version=${PV} \ + -Xgithub.com/rancher/k3s/pkg/version.GitCommit=${@d.getVar('SRCREV_k3s', d, 1)[:8]} \
recipes-containers/k3s/k3s_git.bb new file +6
-0+ -w -s \ + " +BIN_PREFIX ?= "${exec_prefix}/local" + +do_compile() { + exportGOPATH="${S}/src/import/.gopath:${S}/src/import/vendor:${STAGING_DIR_TARGET}/${prefix}/local/go"
recipes-containers/k3s/k3s_git.bb new file +12
-1+ export CGO_ENABLED="1" + export GOFLAGS="-mod=vendor" + cd ${S}/src/import + ${GO} build -tags providerless -ldflags "${GO_BUILD_LDFLAGS}" -o ./dist/artifacts/k3s ./cmd/server/main.go + # Use UPX if it is enabled (and thus exists) to compress binary + if command -v upx > /dev/null 2>&1; then + upx -9 ./dist/artifacts/k3s + fi +} +do_install() { + install -d "${D}${BIN_PREFIX}/bin" + install -m 755 "${S}/src/import/dist/artifacts/k3s""${D}${BIN_PREFIX}/bin"
recipes-containers/k3s/k3s_git.bb new file +5
-0+ ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/crictl" + ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/ctr" + ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/kubectl" + install -m 755 "${WORKDIR}/k3s-clean" "${D}${BIN_PREFIX}/bin" + install -D -m 0644 "${WORKDIR}/cni-containerd-net.conf""
${D}/${sysconfdir}/cni/net.d/10-containerd-net.conf"
+ if
${@bb.utils.contains('DISTRO_FEATURES','systemd','true','false',d)}; then
+ install -D -m 0644 "${WORKDIR}/k3s.service"
"${D}${systemd_system_unitdir}/k3s.service"
+ install -D -m 0644 "${WORKDIR}/k3s-agent.service"
"${D}${systemd_system_unitdir}/k3s-agent.service"
+ sed -i
"s#\(Exec\)\(.*\)=\(.*\)\(k3s\)#\1\2=${BIN_PREFIX}/bin/\4#g" "${D}${systemd_system_unitdir}/k3s.service" "${D}${systemd_system_unitdir}/k3s-agent.service"
+ install -m 755 "${WORKDIR}/k3s-agent"
"${D}${BIN_PREFIX}/bin"recipes-containers/k3s/k3s_git.bb new file +6
-0+ fi +} + +PACKAGES =+ "${PN}-server ${PN}-agent" + +SYSTEMD_PACKAGES ="${@bb.utils.contains('DISTRO_FEATURES','systemd','${PN}-server ${PN}-agent','',d)}"
+SYSTEMD_SERVICE_${PN}-server = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s.service','',d)}" +SYSTEMD_SERVICE_${PN}-agent = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s-agent.service','',d)}"recipes-containers/k3s/k3s_git.bb new file +5
-0+SYSTEMD_AUTO_ENABLE_${PN}-agent = "disable" + +FILES_${PN}-agent = "${BIN_PREFIX}/bin/k3s-agent" + +RDEPENDS_${PN} = "cni conntrack-tools coreutils findutils iproute2ipset virtual/containerd"
recipes-containers/k3s/k3s_git.bb new file +7
-1+RDEPENDS_${PN}-server = "${PN}" +RDEPENDS_${PN}-agent = "${PN}" + +RCONFLICTS_${PN} = "kubectl" + +INHIBIT_PACKAGE_STRIP = "1" +INSANE_SKIP_${PN} += "ldflags already-stripped" --2.20.1
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
PATCH v5 1/1 Adding k3s recipe
--- recipes-containers/k3s/README.md | 30 +++++ ...01-Finding-host-local-in-usr-libexec.patch | 27 +++++ .../k3s/k3s/cni-containerd-net.conf | 24 ++++ recipes-containers/k3s/k3s/k3s-agent | 103 ++++++++++++++++++ recipes-containers/k3s/k3s/k3s-agent.service | 26 +++++ recipes-containers/k3s/k3s/k3s-clean | 30 +++++ recipes-containers/k3s/k3s/k3s.service | 27 +++++ recipes-containers/k3s/k3s_git.bb | 75 +++++++++++++ 8 files changed, 342 insertions(+) create mode 100644 recipes-containers/k3s/README.md create mode 100644 recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch create mode 100644 recipes-containers/k3s/k3s/cni-containerd-net.conf create mode 100755 recipes-containers/k3s/k3s/k3s-agent create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service create mode 100755 recipes-containers/k3s/k3s/k3s-clean create mode 100644 recipes-containers/k3s/k3s/k3s.service create mode 100644 recipes-containers/k3s/k3s_git.bb
recipes-containers/k3s/README.md new file +30 -0
@@ -0,0 +1,30 @@ +# k3s: Lightweight Kubernetes + +Rancher's [k3s](https://k3s.io/), available under +[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides +lightweight Kubernetes suitable for small/edge devices. There are use cases +where the +[installation procedures provided by Rancher](https://rancher.com/docs/k3s/latest/en/installation/) +are not ideal but a bitbake-built version is what is needed. And only a few +mods to the [k3s source code](https://github.com/rancher/k3s) is needed to +accomplish that. + +## CNI + +By default, K3s will run with flannel as the CNI, using VXLAN as the default +backend. It is both possible to change the flannel backend and to change from +flannel to another CNI. + +Please see <https://rancher.com/docs/k3s/latest/en/installation/network-options/> +for further k3s networking details. + +## Configure and run a k3s agent + +The convenience script `k3s-agent` can be used to set up a k3s agent (service): + +```shell +k3s-agent -t <token> -s https://<master>:6443 +``` + +(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the +k3s master.)
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch new file +27 -0
@@ -0,0 +1,27 @@ +From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001 +From: Erik Jansson <[email protected]> +Date: Wed, 16 Oct 2019 15:07:48 +0200 +Subject: [PATCH] Finding host-local in /usr/libexec + +Upstream-status: Inappropriate [embedded specific] +Signed-off-by: <[email protected]> +--- + pkg/agent/config/config.go | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go +index b4296f360a..6af9dab895 100644 +--- a/pkg/agent/config/config.go ++++ b/pkg/agent/config/config.go +@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) { + return nil, err + } + +- hostLocal, err := exec.LookPath("host-local") ++ hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local") + if err != nil { + return nil, errors.Wrapf(err, "failed to find host-local") + } +-- +2.11.0 +
recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24 -0
@@ -0,0 +1,24 @@ +{ + "cniVersion": "0.4.0", + "name": "containerd-net", + "plugins": [ + { + "type": "bridge", + "bridge": "cni0", + "isGateway": true, + "ipMasq": true, + "promiscMode": true, + "ipam": { + "type": "host-local", + "subnet": "10.88.0.0/16", + "routes": [ + { "dst": "0.0.0.0/0" } + ] + } + }, + { + "type": "portmap", + "capabilities": {"portMappings": true} + } + ] +}
recipes-containers/k3s/k3s/k3s-agent new file +103 -0
103 lines not on this page · raw message
recipes-containers/k3s/k3s/k3s-agent.service new file +26 -0
@@ -0,0 +1,26 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes Agent +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=control-group +Delegate=yes +LimitNOFILE=infinity +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s agent +ExecStopPost=/usr/local/bin/k3s-clean +
recipes-containers/k3s/k3s/k3s-clean new file +30 -0
@@ -0,0 +1,30 @@ +#!/bin/sh -eu +# +# Copyright (C) 2020 Axis Communications AB +# +# SPDX-License-Identifier: Apache-2.0 + +do_unmount() { + [ $# -eq 2 ] || return + local mounts= + while read ignore mount ignore; do + case $mount in + $1/*|$2/*) + mounts="$mount $mounts" + ;; + esac + done </proc/self/mounts + [ -z "$mounts" ] || umount $mounts +} + +do_unmount /run/k3s /var/lib/rancher/k3s + +# The lines below come from install.sh's create_killall() function: +ip link show 2>/dev/null | grep 'master cni0' | while read ignore iface ignore; do + iface=${iface%%@*} + [ -z "$iface" ] || ip link delete $iface +done + +ip link delete cni0 +ip link delete flannel.1 +rm -rf /var/lib/cni/
recipes-containers/k3s/k3s/k3s.service new file +27 -0
@@ -0,0 +1,27 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=process +Delegate=yes +# Having non-zero Limit*s causes performance problems due to accounting overhead +# in the kernel. We recommend using cgroups to do container-local accounting. +LimitNOFILE=1048576 +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s server +
recipes-containers/k3s/k3s_git.bb new file +75 -0
75 lines not on this page · raw message
Signature · 1 line
2.20.1
Ha!!!!
This applies.
I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.
Bruce
In message: [meta-virtualization][PATCH v5] Adding k3s recipe
Quoted 418 lines · on 20/10/2020 Joakim Roubert wrote:
Change-Id Id1c52727593bc5ea8d0cd2de192faa44304d7a45Signed-off-by Joakim Roubert <[email protected]>--- recipes-containers/k3s/README.md | 30 +++++ ...01-Finding-host-local-in-usr-libexec.patch | 27 +++++ .../k3s/k3s/cni-containerd-net.conf | 24 ++++ recipes-containers/k3s/k3s/k3s-agent | 103 ++++++++++++++++++ recipes-containers/k3s/k3s/k3s-agent.service | 26 +++++ recipes-containers/k3s/k3s/k3s-clean | 30 +++++ recipes-containers/k3s/k3s/k3s.service | 27 +++++ recipes-containers/k3s/k3s_git.bb | 75 +++++++++++++ 8 files changed, 342 insertions(+) create mode 100644 recipes-containers/k3s/README.md create mode 100644 recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch create mode 100644 recipes-containers/k3s/k3s/cni-containerd-net.conf create mode 100755 recipes-containers/k3s/k3s/k3s-agent create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service create mode 100755 recipes-containers/k3s/k3s/k3s-clean create mode 100644 recipes-containers/k3s/k3s/k3s.service create mode 100644 recipes-containers/k3s/k3s_git.bbrecipes-containers/k3s/README.md new file +30
-0@@ -0,0 +1,30 @@ +# k3s: Lightweight Kubernetes + +Rancher's [k3s](https://k3s.io/), available under +[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides +lightweight Kubernetes suitable for small/edge devices. There are use cases +where the +[installation procedures provided by Rancher](https://rancher.com/docs/k3s/latest/en/installation/) +are not ideal but a bitbake-built version is what is needed. And only a few +mods to the [k3s source code](https://github.com/rancher/k3s) is needed to +accomplish that. + +## CNI + +By default, K3s will run with flannel as the CNI, using VXLAN as the default +backend. It is both possible to change the flannel backend and to change from +flannel to another CNI. + +Please see <https://rancher.com/docs/k3s/latest/en/installation/network-options/> +for further k3s networking details. + +## Configure and run a k3s agent + +The convenience script `k3s-agent` can be used to set up a k3s agent (service): + +```shell +k3s-agent -t <token> -s https://<master>:6443 +``` + +(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the +k3s master.)recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch new file +27
-0@@ -0,0 +1,27 @@ +From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001 +From: Erik Jansson <[email protected]> +Date: Wed, 16 Oct 2019 15:07:48 +0200 +Subject: [PATCH] Finding host-local in /usr/libexec + +Upstream-status: Inappropriate [embedded specific] +Signed-off-by: <[email protected]> +--- + pkg/agent/config/config.go | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go +index b4296f360a..6af9dab895 100644 +--- a/pkg/agent/config/config.go ++++ b/pkg/agent/config/config.go +@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) { + return nil, err + } + +- hostLocal, err := exec.LookPath("host-local") ++ hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local") + if err != nil { + return nil, errors.Wrapf(err, "failed to find host-local") + } +-- +2.11.0 +recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24
-0@@ -0,0 +1,24 @@ +{ + "cniVersion": "0.4.0", + "name": "containerd-net", + "plugins": [ + { + "type": "bridge", + "bridge": "cni0", + "isGateway": true, + "ipMasq": true, + "promiscMode": true, + "ipam": { + "type": "host-local", + "subnet": "10.88.0.0/16", + "routes": [ + { "dst": "0.0.0.0/0" } + ] + } + }, + { + "type": "portmap", + "capabilities": {"portMappings": true} + } + ] +}recipes-containers/k3s/k3s/k3s-agent new file +103
-0@@ -0,0 +1,103 @@ +#!/bin/sh -eu +# +# Copyright (C) 2020 Axis Communications AB +# +# SPDX-License-Identifier: Apache-2.0 + +ENV_CONF=/etc/systemd/system/k3s-agent.service.d/10-env.conf + +usage() { + echo " +USAGE: + ${0##*/} [OPTIONS] +OPTIONS: + --token value, -t value Token to use for authentication [\$K3S_TOKEN] + --token-file value Token file to use for authentication [\$K3S_TOKEN_FILE] + --server value, -s value Server to connect to [\$K3S_URL] + --node-name value Node name [\$K3S_NODE_NAME] + --resolv-conf value Kubelet resolv.conf file [\$K3S_RESOLV_CONF] + --cluster-secret value Shared secret used to bootstrap a cluster [\$K3S_CLUSTER_SECRET] + -h print this +" +} + +[ $# -gt 0 ] || { + usage + exit +} + +case $1 in + -*) + ;; + *) + usage + exit 1 + ;; +esac + +rm -f $ENV_CONF +mkdir -p ${ENV_CONF%/*} +echo [Service] > $ENV_CONF + +while getopts "t:s:-:h" opt; do + case $opt in + h) + usage + exit + ;; + t) + VAR_NAME=K3S_TOKEN + ;; + s) + VAR_NAME=K3S_URL + ;; + -) + [ $# -ge $OPTIND ] || { + usage + exit 1 + } + opt=$OPTARG + eval OPTARG='$'$OPTIND + OPTIND=$(($OPTIND + 1)) + case $opt in + token) + VAR_NAME=K3S_TOKEN + ;; + token-file) + VAR_NAME=K3S_TOKEN_FILE + ;; + server) + VAR_NAME=K3S_URL + ;; + node-name) + VAR_NAME=K3S_NODE_NAME + ;; + resolv-conf) + VAR_NAME=K3S_RESOLV_CONF + ;; + cluster-secret) + VAR_NAME=K3S_CLUSTER_SECRET + ;; + help) + usage + exit + ;; + *) + usage + exit 1 + ;; + esac + ;; + *) + usage + exit 1 + ;; + esac + echo Environment=$VAR_NAME=$OPTARG >> $ENV_CONF +done + +chmod 0644 $ENV_CONF +rm -rf /var/lib/rancher/k3s/agent +systemctl daemon-reload +systemctl restart k3s-agent +systemctl enable k3s-agent.servicerecipes-containers/k3s/k3s/k3s-agent.service new file +26
-0@@ -0,0 +1,26 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes Agent +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=control-group +Delegate=yes +LimitNOFILE=infinity +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s agent +ExecStopPost=/usr/local/bin/k3s-clean +recipes-containers/k3s/k3s/k3s-clean new file +30
-0@@ -0,0 +1,30 @@ +#!/bin/sh -eu +# +# Copyright (C) 2020 Axis Communications AB +# +# SPDX-License-Identifier: Apache-2.0 + +do_unmount() { + [ $# -eq 2 ] || return + local mounts= + while read ignore mount ignore; do + case $mount in + $1/*|$2/*) + mounts="$mount $mounts" + ;; + esac + done </proc/self/mounts + [ -z "$mounts" ] || umount $mounts +} + +do_unmount /run/k3s /var/lib/rancher/k3s + +# The lines below come from install.sh's create_killall() function: +ip link show 2>/dev/null | grep 'master cni0' | while read ignore iface ignore; do + iface=${iface%%@*} + [ -z "$iface" ] || ip link delete $iface +done + +ip link delete cni0 +ip link delete flannel.1 +rm -rf /var/lib/cni/recipes-containers/k3s/k3s/k3s.service new file +27
-0@@ -0,0 +1,27 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=process +Delegate=yes +# Having non-zero Limit*s causes performance problems due to accounting overhead +# in the kernel. We recommend using cgroups to do container-local accounting. +LimitNOFILE=1048576 +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s server +recipes-containers/k3s/k3s_git.bb new file +75
-0@@ -0,0 +1,75 @@ +SUMMARY = "Production-Grade Container Scheduling and Management" +DESCRIPTION = "Lightweight Kubernetes, intended to be a fully compliant Kubernetes." +HOMEPAGE = "https://k3s.io/" +LICENSE = "Apache-2.0" +LIC_FILES_CHKSUM = "file://${S}/src/import/LICENSE;md5=2ee41112a44fe7014dce33e26468ba93" +PV = "v1.18.9+k3s1-dirty" + +SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.18;name=k3s \ + file://k3s.service \ + file://k3s-agent.service \ + file://k3s-agent \ + file://k3s-clean \ + file://cni-containerd-net.conf \ + file://0001-Finding-host-local-in-usr-libexec.patch;patchdir=src/import \ + " +SRC_URI[k3s.md5sum] = "363d3a08dc0b72ba6e6577964f6e94a5" +SRCREV_k3s = "630bebf94b9dce6b8cd3d402644ed023b3af8f90" + +inherit go +inherit goarch +inherit systemd + +PACKAGECONFIG = "" +PACKAGECONFIG[upx] = ",,upx-native" +GO_IMPORT = "import" +GO_BUILD_LDFLAGS = "-X github.com/rancher/k3s/pkg/version.Version=${PV} \ + -X github.com/rancher/k3s/pkg/version.GitCommit=${@d.getVar('SRCREV_k3s', d, 1)[:8]} \ + -w -s \ + " +BIN_PREFIX ?= "${exec_prefix}/local" + +do_compile() { + export GOPATH="${S}/src/import/.gopath:${S}/src/import/vendor:${STAGING_DIR_TARGET}/${prefix}/local/go" + export CGO_ENABLED="1" + export GOFLAGS="-mod=vendor" + cd ${S}/src/import + ${GO} build -tags providerless -ldflags "${GO_BUILD_LDFLAGS}" -o ./dist/artifacts/k3s ./cmd/server/main.go + # Use UPX if it is enabled (and thus exists) to compress binary + if command -v upx > /dev/null 2>&1; then + upx -9 ./dist/artifacts/k3s + fi +} +do_install() { + install -d "${D}${BIN_PREFIX}/bin" + install -m 755 "${S}/src/import/dist/artifacts/k3s" "${D}${BIN_PREFIX}/bin" + ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/crictl" + ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/ctr" + ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/kubectl" + install -m 755 "${WORKDIR}/k3s-clean" "${D}${BIN_PREFIX}/bin" + install -D -m 0644 "${WORKDIR}/cni-containerd-net.conf" "${D}/${sysconfdir}/cni/net.d/10-containerd-net.conf" + if ${@bb.utils.contains('DISTRO_FEATURES','systemd','true','false',d)}; then + install -D -m 0644 "${WORKDIR}/k3s.service" "${D}${systemd_system_unitdir}/k3s.service" + install -D -m 0644 "${WORKDIR}/k3s-agent.service" "${D}${systemd_system_unitdir}/k3s-agent.service" + sed -i "s#\(Exec\)\(.*\)=\(.*\)\(k3s\)#\1\2=${BIN_PREFIX}/bin/\4#g" "${D}${systemd_system_unitdir}/k3s.service" "${D}${systemd_system_unitdir}/k3s-agent.service" + install -m 755 "${WORKDIR}/k3s-agent" "${D}${BIN_PREFIX}/bin" + fi +} + +PACKAGES =+ "${PN}-server ${PN}-agent" + +SYSTEMD_PACKAGES = "${@bb.utils.contains('DISTRO_FEATURES','systemd','${PN}-server ${PN}-agent','',d)}" +SYSTEMD_SERVICE_${PN}-server = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s.service','',d)}" +SYSTEMD_SERVICE_${PN}-agent = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s-agent.service','',d)}" +SYSTEMD_AUTO_ENABLE_${PN}-agent = "disable" + +FILES_${PN}-agent = "${BIN_PREFIX}/bin/k3s-agent" + +RDEPENDS_${PN} = "cni conntrack-tools coreutils findutils iproute2 ipset virtual/containerd" +RDEPENDS_${PN}-server = "${PN}" +RDEPENDS_${PN}-agent = "${PN}" + +RCONFLICTS_${PN} = "kubectl" + +INHIBIT_PACKAGE_STRIP = "1" +INSANE_SKIP_${PN} += "ldflags already-stripped"Signature · 1 line
2.20.1
Quoted 3 lines · Bruce Ashfield
Ha!!!!
This applies.
Wonderful, thank you! I guess this is what is called "five times lucky"...
Quoted 3 lines
I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.
Excellent!
BR,
/Joakim
Signature · 7 lines
Joakim Roubert Senior Engineer Axis Communications AB Emdalavägen 14, SE-223 69 Lund, Sweden Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48 Fax: +46 46 13 61 30, www.axis.com
Quoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield
Ha!!!!
This applies.
Quoted 7 lines
Wonderful, thank you! I guess this is what is called "five times lucky"...
Quoted 3 lines
I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.
Excellent!
I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/log/?h=k3s-wip
That includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.
I did have a couple of questions on the k3s packaging itself, I was getting the following
error:
ERROR: k3s-v1.18.9+k3s1-dirty-r0 do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3s
So I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?
Bruce
Quoted 8 lines
BR,
/Joakim
--
Joakim Roubert
Senior Engineer
Quoted 5 lines
Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
Quoted 11 lines · Bruce Ashfield
ERROR: k3s-v1.18.9+k3s1-dirty-r0
do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3sSo I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?
Yes, it is! In my build, all of those are found in the k3s RPM, although the FILES directive for them is missing. So yes, your fix here is the right one, IMO.
BR,
/Joakim
Quoted 10 lines · Joakim Roubert
Quoted 8 lines · Bruce Ashfield
ERROR: k3s-v1.18.9+k3s1-dirty-r0
do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:/usr/local/bin/k3s-clean /usr/local/bin/crictl /usr/local/bin/kubectl /usr/local/bin/k3s
Quoted 7 lines
Quoted 3 lines
So I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?
Yes, it is! In my build, all of those are found in the k3s RPM, although the FILES directive for them is missing. So yes, your fix here is the right one, IMO.
I now have another 6 or 7 WIP patches on top of this to try and get a single node "cluster" working with k3s. I'll clean them up and get them into the k3s WIP branch shortly.
I'm failing on the networking component though.
In your working references, which iptables do you have installed ? (legacy ? nftables?)
I'm failing to get flannel to start, with a series of errors like this:
-----------
I1106 21:19:00.985656 10641 eviction_manager.go:351] eviction
manager: able to reduce ephemeral-storage pressure without evicting pods.
E1106 21:19:10.636899 10641 proxier.go:841] Failed to ensure that
filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error checking
rule: exit status 2: iptables v1.8.6 (legacy): Couldn't load match `comment':No such y
Try `iptables -h' or 'iptables --help' for more information.
I1106 21:19:10.641647 10641 proxier.go:825] Sync failed; retrying in 30s
------------
Which means as soon as I try and run an agent command, the server dies and I can't even do a basic test.
Bruce
Quoted 4 lines
BR,
/Joakim
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 4 lines · Bruce Ashfield
I now have another 6 or 7 WIP patches on top of this to try and get a single node "cluster" working with k3s. I'll clean them up and get them into the k3s WIP branch shortly.
Awesome!
Quoted 2 lines
In your working references, which iptables do you have installed ? (legacy ? nftables?)
# iptables --version iptables v1.8.4 (legacy)
Quoted 13 lines
I'm failing to get flannel to start, with a series of errors like this:
-----------
I1106 21:19:00.985656 10641 eviction_manager.go:351] eviction
manager: able to reduce ephemeral-storage pressure without evicting pods.
E1106 21:19:10.636899 10641 proxier.go:841] Failed to ensure that filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error checking
rule: exit status 2: iptables v1.8.6 (legacy): Couldn't load match `comment':No such y
Try `iptables -h' or 'iptables --help' for more information. I1106 21:19:10.641647 10641 proxier.go:825] Sync failed; retrying in 30s
------------
This is a bit strange, as it seems you are running in legacy mode too, although a somewhat newer version than I have, and the only thing I know is that Rancher recommends 1.6.1 or newer (which it is).
https://rancher.com/docs/k3s/latest/en/known-issues/
Might there be something missing in the kernel?
I have these RPMs installed in my image:
iptables
iptables-module-ip6t-ah
iptables-module-ip6t-dnat
iptables-module-ip6t-dnpt
iptables-module-ip6t-dst
iptables-module-ip6t-eui64
iptables-module-ip6t-frag
iptables-module-ip6t-hbh
iptables-module-ip6t-hl
iptables-module-ip6t-icmp6
iptables-module-ip6t-ipv6header
iptables-module-ip6t-log
iptables-module-ip6t-masquerade
iptables-module-ip6t-mh
iptables-module-ip6t-netmap
iptables-module-ip6t-redirect
iptables-module-ip6t-reject
iptables-module-ip6t-rt
iptables-module-ip6t-snat
iptables-module-ip6t-snpt
iptables-module-ip6t-srh
iptables-module-ipt-ah
iptables-module-ipt-clusterip
iptables-module-ipt-dnat
iptables-module-ipt-ecn
iptables-module-ipt-icmp
iptables-module-ipt-log
iptables-module-ipt-masquerade
iptables-module-ipt-netmap
iptables-module-ipt-realm
iptables-module-ipt-redirect
iptables-module-ipt-reject
iptables-module-ipt-snat
iptables-module-ipt-ttl
iptables-module-ipt-ulog
iptables-modules
iptables-module-xt-addrtype
iptables-module-xt-audit
iptables-module-xt-bpf
iptables-module-xt-cgroup
iptables-module-xt-checksum
iptables-module-xt-classify
iptables-module-xt-cluster
iptables-module-xt-comment
iptables-module-xt-connbytes
iptables-module-xt-connlimit
iptables-module-xt-connmark
iptables-module-xt-connsecmark
iptables-module-xt-conntrack
iptables-module-xt-cpu
iptables-module-xt-ct
iptables-module-xt-dccp
iptables-module-xt-devgroup
iptables-module-xt-dscp
iptables-module-xt-ecn
iptables-module-xt-esp
iptables-module-xt-hashlimit
iptables-module-xt-helper
iptables-module-xt-hmark
iptables-module-xt-idletimer
iptables-module-xt-ipcomp
iptables-module-xt-iprange
iptables-module-xt-ipvs
iptables-module-xt-led
iptables-module-xt-length
iptables-module-xt-limit
iptables-module-xt-mac
iptables-module-xt-mark
iptables-module-xt-multiport
iptables-module-xt-nfacct
iptables-module-xt-nflog
iptables-module-xt-nfqueue
iptables-module-xt-osf
iptables-module-xt-owner
iptables-module-xt-physdev
iptables-module-xt-pkttype
iptables-module-xt-policy
iptables-module-xt-quota
iptables-module-xt-rateest
iptables-module-xt-recent
iptables-module-xt-rpfilter
iptables-module-xt-sctp
iptables-module-xt-secmark
iptables-module-xt-set
iptables-module-xt-socket
iptables-module-xt-standard
iptables-module-xt-statistic
iptables-module-xt-string
iptables-module-xt-synproxy
iptables-module-xt-tcp
iptables-module-xt-tcpmss
iptables-module-xt-tcpoptstrip
iptables-module-xt-tee
iptables-module-xt-time
iptables-module-xt-tos
iptables-module-xt-tproxy
iptables-module-xt-trace
iptables-module-xt-u32
iptables-module-xt-udp
and in my kernel config, I have (apart from what is needed for running containers with containerd):
CONFIG_NETFILTER_NETLINK=mCONFIG_NETFILTER_XT_MATCH_OWNER=mCONFIG_NET_UDP_TUNNEL=mCONFIG_NF_DUP_NETDEV=mCONFIG_NF_LOG_BRIDGE=mCONFIG_NF_TABLES_ARP=yCONFIG_NF_TABLES_BRIDGE=yCONFIG_NF_TABLES_INET=yCONFIG_NF_TABLES_IPV4=yCONFIG_NF_TABLES_IPV6=yCONFIG_NF_TABLES=mCONFIG_NF_TABLES_NETDEV=yCONFIG_NFT_BRIDGE_REJECT=mCONFIG_NFT_CHAIN_NAT_IPV4=mCONFIG_NFT_CHAIN_ROUTE_IPV4=mCONFIG_NFT_CHAIN_ROUTE_IPV6=mCONFIG_NFT_COMPAT=mCONFIG_NFT_COUNTER=mCONFIG_NFT_CT=mCONFIG_NFT_DUP_IPV4=mCONFIG_NFT_DUP_IPV6=mCONFIG_NFT_DUP_NETDEV=m
# CONFIG_NFT_EXTHDR is not setCONFIG_NFT_FIB_INET=mCONFIG_NFT_FIB_IPV4=mCONFIG_NFT_FIB_IPV6=mCONFIG_NFT_FIB_NETDEV=mCONFIG_NFT_FWD_NETDEV=mCONFIG_NFT_HASH=mCONFIG_NFT_LIMIT=mCONFIG_NFT_LOG=mCONFIG_NFT_MASQ_IPV4=mCONFIG_NFT_MASQ=m
# CONFIG_NFT_META is not setCONFIG_NFT_NAT=mCONFIG_NFT_NUMGEN=m
# CONFIG_NFT_OBJREF is not setCONFIG_NFT_QUEUE=mCONFIG_NFT_QUOTA=mCONFIG_NFT_REDIR_IPV4=mCONFIG_NFT_REDIR=mCONFIG_NFT_REJECT=m
# CONFIG_NFT_RT is not set
# CONFIG_NFT_SET_BITMAP is not set
# CONFIG_NFT_SET_HASH is not set
# CONFIG_NFT_SET_RBTREE is not setCONFIG_OVERLAY_FS=mCONFIG_STP=m
BR,
/Joakim
Hi Bruce,
For the iptables issue, I tested iptables.
As iptables comment module belonging to iptables extension, I checked my kernel config and set the parameter: CONFIG_NETFILTER_XT_MATCH_COMMENT=m.
iptables -V
iptables v1.8.5 (legacy)
I used this iptables command to check
iptables -A INPUT -p tcp --dport 22 -m comment --comment "SSH" -j ACCEPT
It works fine from my side.
Best Regards,
Lance
Quoted 199 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Joakim Roubert via lists.yoctoproject.org
Sent: Monday, November 9, 2020 3:49 PM
To: Bruce Ashfield <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 4 lines · Bruce Ashfield
I now have another 6 or 7 WIP patches on top of this to try and get a single node "cluster" working with k3s. I'll clean them up and get them into the k3s WIP branch shortly.
Awesome!
Quoted 2 lines
In your working references, which iptables do you have installed ? (legacy ? nftables?)
# iptables --version iptables v1.8.4 (legacy)Quoted 14 lines
I'm failing to get flannel to start, with a series of errors like this:
-----------
I1106 21:19:00.985656 10641 eviction_manager.go:351] eviction
manager: able to reduce ephemeral-storage pressure without evicting pods.
E1106 21:19:10.636899 10641 proxier.go:841] Failed to ensure that
filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error checking
rule: exit status 2: iptables v1.8.6 (legacy): Couldn't load matchcomment':No such y Tryiptables -h' or 'iptables --help' for more information.
I1106 21:19:10.641647 10641 proxier.go:825] Sync failed; retrying in
30s
------------This is a bit strange, as it seems you are running in legacy mode too, although a somewhat newer version than I have, and the only thing I know is that Rancher recommends 1.6.1 or newer (which it is).
https://rancher.com/docs/k3s/latest/en/known-issues/
Might there be something missing in the kernel?
I have these RPMs installed in my image:
iptables
iptables-module-ip6t-ah
iptables-module-ip6t-dnat
iptables-module-ip6t-dnpt
iptables-module-ip6t-dst
iptables-module-ip6t-eui64
iptables-module-ip6t-frag
iptables-module-ip6t-hbh
iptables-module-ip6t-hl
iptables-module-ip6t-icmp6
iptables-module-ip6t-ipv6header
iptables-module-ip6t-log
iptables-module-ip6t-masquerade
iptables-module-ip6t-mh
iptables-module-ip6t-netmap
iptables-module-ip6t-redirect
iptables-module-ip6t-reject
iptables-module-ip6t-rt
iptables-module-ip6t-snat
iptables-module-ip6t-snpt
iptables-module-ip6t-srh
iptables-module-ipt-ah
iptables-module-ipt-clusterip
iptables-module-ipt-dnat
iptables-module-ipt-ecn
iptables-module-ipt-icmp
iptables-module-ipt-log
iptables-module-ipt-masquerade
iptables-module-ipt-netmap
iptables-module-ipt-realm
iptables-module-ipt-redirect
iptables-module-ipt-reject
iptables-module-ipt-snat
iptables-module-ipt-ttl
iptables-module-ipt-ulog
iptables-modules
iptables-module-xt-addrtype
iptables-module-xt-audit
iptables-module-xt-bpf
iptables-module-xt-cgroup
iptables-module-xt-checksum
iptables-module-xt-classify
iptables-module-xt-cluster
iptables-module-xt-comment
iptables-module-xt-connbytes
iptables-module-xt-connlimit
iptables-module-xt-connmark
iptables-module-xt-connsecmark
iptables-module-xt-conntrack
iptables-module-xt-cpu
iptables-module-xt-ct
iptables-module-xt-dccp
iptables-module-xt-devgroup
iptables-module-xt-dscp
iptables-module-xt-ecn
iptables-module-xt-esp
iptables-module-xt-hashlimit
iptables-module-xt-helper
iptables-module-xt-hmark
iptables-module-xt-idletimer
iptables-module-xt-ipcomp
iptables-module-xt-iprange
iptables-module-xt-ipvs
iptables-module-xt-led
iptables-module-xt-length
iptables-module-xt-limit
iptables-module-xt-mac
iptables-module-xt-mark
iptables-module-xt-multiport
iptables-module-xt-nfacct
iptables-module-xt-nflog
iptables-module-xt-nfqueue
iptables-module-xt-osf
iptables-module-xt-owner
iptables-module-xt-physdev
iptables-module-xt-pkttype
iptables-module-xt-policy
iptables-module-xt-quota
iptables-module-xt-rateest
iptables-module-xt-recent
iptables-module-xt-rpfilter
iptables-module-xt-sctp
iptables-module-xt-secmark
iptables-module-xt-set
iptables-module-xt-socket
iptables-module-xt-standard
iptables-module-xt-statistic
iptables-module-xt-string
iptables-module-xt-synproxy
iptables-module-xt-tcp
iptables-module-xt-tcpmss
iptables-module-xt-tcpoptstrip
iptables-module-xt-tee
iptables-module-xt-time
iptables-module-xt-tos
iptables-module-xt-tproxy
iptables-module-xt-trace
iptables-module-xt-u32
iptables-module-xt-udpand in my kernel config, I have (apart from what is needed for running containers with containerd):
CONFIG_NETFILTER_NETLINK=mCONFIG_NETFILTER_XT_MATCH_OWNER=mCONFIG_NET_UDP_TUNNEL=mCONFIG_NF_DUP_NETDEV=mCONFIG_NF_LOG_BRIDGE=mCONFIG_NF_TABLES_ARP=yCONFIG_NF_TABLES_BRIDGE=yCONFIG_NF_TABLES_INET=yCONFIG_NF_TABLES_IPV4=yCONFIG_NF_TABLES_IPV6=yCONFIG_NF_TABLES=mCONFIG_NF_TABLES_NETDEV=yCONFIG_NFT_BRIDGE_REJECT=mCONFIG_NFT_CHAIN_NAT_IPV4=mCONFIG_NFT_CHAIN_ROUTE_IPV4=mCONFIG_NFT_CHAIN_ROUTE_IPV6=mCONFIG_NFT_COMPAT=mCONFIG_NFT_COUNTER=mCONFIG_NFT_CT=mCONFIG_NFT_DUP_IPV4=mCONFIG_NFT_DUP_IPV6=mCONFIG_NFT_DUP_NETDEV=m
#CONFIG_NFT_EXTHDRis not setCONFIG_NFT_FIB_INET=mCONFIG_NFT_FIB_IPV4=mCONFIG_NFT_FIB_IPV6=mCONFIG_NFT_FIB_NETDEV=mCONFIG_NFT_FWD_NETDEV=mCONFIG_NFT_HASH=mCONFIG_NFT_LIMIT=mCONFIG_NFT_LOG=mCONFIG_NFT_MASQ_IPV4=mCONFIG_NFT_MASQ=m
#CONFIG_NFT_METAis not setCONFIG_NFT_NAT=mCONFIG_NFT_NUMGEN=m
#CONFIG_NFT_OBJREFis not setCONFIG_NFT_QUEUE=mCONFIG_NFT_QUOTA=mCONFIG_NFT_REDIR_IPV4=mCONFIG_NFT_REDIR=mCONFIG_NFT_REJECT=m
#CONFIG_NFT_RTis not set
#CONFIG_NFT_SET_BITMAPis not set
#CONFIG_NFT_SET_HASHis not set
#CONFIG_NFT_SET_RBTREEis not setCONFIG_OVERLAY_FS=mCONFIG_STP=m
Quoted 3 lines
BR,
/Joakim
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 7 lines · Lance Yang
Hi Bruce,
For the iptables issue, I tested iptables.
As iptables comment module belonging to iptables extension, I checked my kernel config and set the parameter:
CONFIG_NETFILTER_XT_MATCH_COMMENT=m.
Quoted 8 lines
iptables -V
iptables v1.8.5 (legacy)I used this iptables command to check
iptables -A INPUT -p tcp --dport 22 -m comment --comment "SSH" -j ACCEPT
It works fine from my side.
Yah, that's what I assumed it was as well, but yet, when I added it in .. I didn't see a change.
That being said, this is helpful, so I started a clean build to see if I had picked up something stale that was masking my fix.
Bruce
Quoted 207 lines
Best Regards,
LanceQuoted 199 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Joakim Roubert via lists.yoctoproject.org
Sent: Monday, November 9, 2020 3:49 PM
To: Bruce Ashfield <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 4 lines · Bruce Ashfield
I now have another 6 or 7 WIP patches on top of this to try and get a single node "cluster" working with k3s. I'll clean them up and get them into the k3s WIP branch shortly.
Awesome!
Quoted 2 lines
In your working references, which iptables do you have installed ? (legacy ? nftables?)
# iptables --version iptables v1.8.4 (legacy)Quoted 14 lines
I'm failing to get flannel to start, with a series of errors like this:
-----------
I1106 21:19:00.985656 10641 eviction_manager.go:351] eviction
manager: able to reduce ephemeral-storage pressure without evicting pods.
E1106 21:19:10.636899 10641 proxier.go:841] Failed to ensure that
filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error checking
rule: exit status 2: iptables v1.8.6 (legacy): Couldn't load matchcomment':No such y Tryiptables -h' or 'iptables --help' for more information.
I1106 21:19:10.641647 10641 proxier.go:825] Sync failed; retrying in
30s
------------This is a bit strange, as it seems you are running in legacy mode too, although a somewhat newer version than I have, and the only thing I know is that Rancher recommends 1.6.1 or newer (which it is).
https://rancher.com/docs/k3s/latest/en/known-issues/
Might there be something missing in the kernel?
I have these RPMs installed in my image:
iptables
iptables-module-ip6t-ah
iptables-module-ip6t-dnat
iptables-module-ip6t-dnpt
iptables-module-ip6t-dst
iptables-module-ip6t-eui64
iptables-module-ip6t-frag
iptables-module-ip6t-hbh
iptables-module-ip6t-hl
iptables-module-ip6t-icmp6
iptables-module-ip6t-ipv6header
iptables-module-ip6t-log
iptables-module-ip6t-masquerade
iptables-module-ip6t-mh
iptables-module-ip6t-netmap
iptables-module-ip6t-redirect
iptables-module-ip6t-reject
iptables-module-ip6t-rt
iptables-module-ip6t-snat
iptables-module-ip6t-snpt
iptables-module-ip6t-srh
iptables-module-ipt-ah
iptables-module-ipt-clusterip
iptables-module-ipt-dnat
iptables-module-ipt-ecn
iptables-module-ipt-icmp
iptables-module-ipt-log
iptables-module-ipt-masquerade
iptables-module-ipt-netmap
iptables-module-ipt-realm
iptables-module-ipt-redirect
iptables-module-ipt-reject
iptables-module-ipt-snat
iptables-module-ipt-ttl
iptables-module-ipt-ulog
iptables-modules
iptables-module-xt-addrtype
iptables-module-xt-audit
iptables-module-xt-bpf
iptables-module-xt-cgroup
iptables-module-xt-checksum
iptables-module-xt-classify
iptables-module-xt-cluster
iptables-module-xt-comment
iptables-module-xt-connbytes
iptables-module-xt-connlimit
iptables-module-xt-connmark
iptables-module-xt-connsecmark
iptables-module-xt-conntrack
iptables-module-xt-cpu
iptables-module-xt-ct
iptables-module-xt-dccp
iptables-module-xt-devgroup
iptables-module-xt-dscp
iptables-module-xt-ecn
iptables-module-xt-esp
iptables-module-xt-hashlimit
iptables-module-xt-helper
iptables-module-xt-hmark
iptables-module-xt-idletimer
iptables-module-xt-ipcomp
iptables-module-xt-iprange
iptables-module-xt-ipvs
iptables-module-xt-led
iptables-module-xt-length
iptables-module-xt-limit
iptables-module-xt-mac
iptables-module-xt-mark
iptables-module-xt-multiport
iptables-module-xt-nfacct
iptables-module-xt-nflog
iptables-module-xt-nfqueue
iptables-module-xt-osf
iptables-module-xt-owner
iptables-module-xt-physdev
iptables-module-xt-pkttype
iptables-module-xt-policy
iptables-module-xt-quota
iptables-module-xt-rateest
iptables-module-xt-recent
iptables-module-xt-rpfilter
iptables-module-xt-sctp
iptables-module-xt-secmark
iptables-module-xt-set
iptables-module-xt-socket
iptables-module-xt-standard
iptables-module-xt-statistic
iptables-module-xt-string
iptables-module-xt-synproxy
iptables-module-xt-tcp
iptables-module-xt-tcpmss
iptables-module-xt-tcpoptstrip
iptables-module-xt-tee
iptables-module-xt-time
iptables-module-xt-tos
iptables-module-xt-tproxy
iptables-module-xt-trace
iptables-module-xt-u32
iptables-module-xt-udpand in my kernel config, I have (apart from what is needed for running containers with containerd):
CONFIG_NETFILTER_NETLINK=mCONFIG_NETFILTER_XT_MATCH_OWNER=mCONFIG_NET_UDP_TUNNEL=mCONFIG_NF_DUP_NETDEV=mCONFIG_NF_LOG_BRIDGE=mCONFIG_NF_TABLES_ARP=yCONFIG_NF_TABLES_BRIDGE=yCONFIG_NF_TABLES_INET=yCONFIG_NF_TABLES_IPV4=yCONFIG_NF_TABLES_IPV6=yCONFIG_NF_TABLES=mCONFIG_NF_TABLES_NETDEV=yCONFIG_NFT_BRIDGE_REJECT=mCONFIG_NFT_CHAIN_NAT_IPV4=mCONFIG_NFT_CHAIN_ROUTE_IPV4=mCONFIG_NFT_CHAIN_ROUTE_IPV6=mCONFIG_NFT_COMPAT=mCONFIG_NFT_COUNTER=mCONFIG_NFT_CT=mCONFIG_NFT_DUP_IPV4=mCONFIG_NFT_DUP_IPV6=mCONFIG_NFT_DUP_NETDEV=m
#CONFIG_NFT_EXTHDRis not setCONFIG_NFT_FIB_INET=mCONFIG_NFT_FIB_IPV4=mCONFIG_NFT_FIB_IPV6=mCONFIG_NFT_FIB_NETDEV=mCONFIG_NFT_FWD_NETDEV=mCONFIG_NFT_HASH=mCONFIG_NFT_LIMIT=mCONFIG_NFT_LOG=mCONFIG_NFT_MASQ_IPV4=mCONFIG_NFT_MASQ=m
#CONFIG_NFT_METAis not setCONFIG_NFT_NAT=mCONFIG_NFT_NUMGEN=m
#CONFIG_NFT_OBJREFis not setCONFIG_NFT_QUEUE=mCONFIG_NFT_QUOTA=mCONFIG_NFT_REDIR_IPV4=mCONFIG_NFT_REDIR=mCONFIG_NFT_REJECT=m
#CONFIG_NFT_RTis not set
#CONFIG_NFT_SET_BITMAPis not set
#CONFIG_NFT_SET_HASHis not set
#CONFIG_NFT_SET_RBTREEis not setCONFIG_OVERLAY_FS=mCONFIG_STP=mQuoted 3 lines
BR,
/Joakim
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Hi Bruce,
Thanks for your reply. Because I installed "kernel-modules" in my local.conf:
IMAGE_INSTALL_append = " ... \ (skip something unrelated)
iptables kernel-modules k3s"
So I am not able to figure out which modules are missing. But hopefully, my investigation can help a little bit.
I referred to the file iptables_1.8.5.bb, I found something like:
RDEPENDS_${PN} = "${PN}-module-xt-standard"
RRECOMMENDS_${PN} = " \
${PN}-modules \
kernel-module-x-tables \
kernel-module-ip-tables \
kernel-module-iptable-filter \
kernel-module-iptable-nat \
kernel-module-nf-defrag-ipv4 \
kernel-module-nf-conntrack \
kernel-module-nf-conntrack-ipv4 \
kernel-module-nf-nat \
kernel-module-ipt-masquerade \
${@bb.utils.contains('PACKAGECONFIG', 'ipv6', '\
kernel-module-ip6table-filter \
kernel-module-ip6-tables \
', '', d)} \I am not sure whether we need install kernel-module-x-tables manually or something like that. Maybe it is not a bad way to try install "kernel-modules" directly to confirm if this issue is related to some kernel-modules. And then figure out which specific netfilter/iptables/xtables modules we are missing. What do you think?
Best Regards,
Lance Yang
Quoted 248 lines
-----Original Message-----
From: Bruce Ashfield <[email protected]>
Sent: Monday, November 9, 2020 9:46 PM
To: Lance Yang <[email protected]>
Cc: [email protected]; [email protected]; nd <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 6 lines · Lance Yang
Hi Bruce,
For the iptables issue, I tested iptables.
As iptables comment module belonging to iptables extension, I checked my kernel config and
set the parameter:
CONFIG_NETFILTER_XT_MATCH_COMMENT=m.Quoted 10 lines
iptables -V
iptables v1.8.5 (legacy)I used this iptables command to check
iptables -A INPUT -p tcp --dport 22 -m comment --comment "SSH" -j ACCEPT
It works fine from my side.
Yah, that's what I assumed it was as well, but yet, when I added it in .. I didn't see a change.
That being said, this is helpful, so I started a clean build to see if I had picked up something stale that was masking my fix.
Bruce
Quoted 205 lines
Best Regards,
LanceQuoted 201 lines
-----Original Message-----
From: [email protected] <[email protected]>
On Behalf Of Joakim Roubert via lists.yoctoproject.org
Sent: Monday, November 9, 2020 3:49 PM
To: Bruce Ashfield <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 4 lines · Bruce Ashfield
I now have another 6 or 7 WIP patches on top of this to try and get a single node "cluster" working with k3s. I'll clean them up and get them into the k3s WIP branch shortly.
Awesome!
Quoted 2 lines
In your working references, which iptables do you have installed ? (legacy ? nftables?)
# iptables --version iptables v1.8.4 (legacy)Quoted 14 lines
I'm failing to get flannel to start, with a series of errors like this:
-----------
I1106 21:19:00.985656 10641 eviction_manager.go:351] eviction
manager: able to reduce ephemeral-storage pressure without evicting pods.
E1106 21:19:10.636899 10641 proxier.go:841] Failed to ensure that
filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error checking
rule: exit status 2: iptables v1.8.6 (legacy): Couldn't load matchcomment':No such y Tryiptables -h' or 'iptables --help' for more information.
I1106 21:19:10.641647 10641 proxier.go:825] Sync failed; retrying in
30s
------------This is a bit strange, as it seems you are running in legacy mode too, although a somewhat newer version than I have, and the only thing I know is that Rancher recommends 1.6.1 or newer (which it is).
https://rancher.com/docs/k3s/latest/en/known-issues/
Might there be something missing in the kernel?
I have these RPMs installed in my image:
iptables
iptables-module-ip6t-ah
iptables-module-ip6t-dnat
iptables-module-ip6t-dnpt
iptables-module-ip6t-dst
iptables-module-ip6t-eui64
iptables-module-ip6t-frag
iptables-module-ip6t-hbh
iptables-module-ip6t-hl
iptables-module-ip6t-icmp6
iptables-module-ip6t-ipv6header
iptables-module-ip6t-log
iptables-module-ip6t-masquerade
iptables-module-ip6t-mh
iptables-module-ip6t-netmap
iptables-module-ip6t-redirect
iptables-module-ip6t-reject
iptables-module-ip6t-rt
iptables-module-ip6t-snat
iptables-module-ip6t-snpt
iptables-module-ip6t-srh
iptables-module-ipt-ah
iptables-module-ipt-clusterip
iptables-module-ipt-dnat
iptables-module-ipt-ecn
iptables-module-ipt-icmp
iptables-module-ipt-log
iptables-module-ipt-masquerade
iptables-module-ipt-netmap
iptables-module-ipt-realm
iptables-module-ipt-redirect
iptables-module-ipt-reject
iptables-module-ipt-snat
iptables-module-ipt-ttl
iptables-module-ipt-ulog
iptables-modules
iptables-module-xt-addrtype
iptables-module-xt-audit
iptables-module-xt-bpf
iptables-module-xt-cgroup
iptables-module-xt-checksum
iptables-module-xt-classify
iptables-module-xt-cluster
iptables-module-xt-comment
iptables-module-xt-connbytes
iptables-module-xt-connlimit
iptables-module-xt-connmark
iptables-module-xt-connsecmark
iptables-module-xt-conntrack
iptables-module-xt-cpu
iptables-module-xt-ct
iptables-module-xt-dccp
iptables-module-xt-devgroup
iptables-module-xt-dscp
iptables-module-xt-ecn
iptables-module-xt-esp
iptables-module-xt-hashlimit
iptables-module-xt-helper
iptables-module-xt-hmark
iptables-module-xt-idletimer
iptables-module-xt-ipcomp
iptables-module-xt-iprange
iptables-module-xt-ipvs
iptables-module-xt-led
iptables-module-xt-length
iptables-module-xt-limit
iptables-module-xt-mac
iptables-module-xt-mark
iptables-module-xt-multiport
iptables-module-xt-nfacct
iptables-module-xt-nflog
iptables-module-xt-nfqueue
iptables-module-xt-osf
iptables-module-xt-owner
iptables-module-xt-physdev
iptables-module-xt-pkttype
iptables-module-xt-policy
iptables-module-xt-quota
iptables-module-xt-rateest
iptables-module-xt-recent
iptables-module-xt-rpfilter
iptables-module-xt-sctp
iptables-module-xt-secmark
iptables-module-xt-set
iptables-module-xt-socket
iptables-module-xt-standard
iptables-module-xt-statistic
iptables-module-xt-string
iptables-module-xt-synproxy
iptables-module-xt-tcp
iptables-module-xt-tcpmss
iptables-module-xt-tcpoptstrip
iptables-module-xt-tee
iptables-module-xt-time
iptables-module-xt-tos
iptables-module-xt-tproxy
iptables-module-xt-trace
iptables-module-xt-u32
iptables-module-xt-udpand in my kernel config, I have (apart from what is needed for running containers with
containerd):
CONFIG_NETFILTER_NETLINK=mCONFIG_NETFILTER_XT_MATCH_OWNER=mCONFIG_NET_UDP_TUNNEL=mCONFIG_NF_DUP_NETDEV=mCONFIG_NF_LOG_BRIDGE=mCONFIG_NF_TABLES_ARP=yCONFIG_NF_TABLES_BRIDGE=yCONFIG_NF_TABLES_INET=yCONFIG_NF_TABLES_IPV4=yCONFIG_NF_TABLES_IPV6=yCONFIG_NF_TABLES=mCONFIG_NF_TABLES_NETDEV=yCONFIG_NFT_BRIDGE_REJECT=mCONFIG_NFT_CHAIN_NAT_IPV4=mCONFIG_NFT_CHAIN_ROUTE_IPV4=mCONFIG_NFT_CHAIN_ROUTE_IPV6=mCONFIG_NFT_COMPAT=mCONFIG_NFT_COUNTER=mCONFIG_NFT_CT=mCONFIG_NFT_DUP_IPV4=mCONFIG_NFT_DUP_IPV6=mCONFIG_NFT_DUP_NETDEV=m
#CONFIG_NFT_EXTHDRis not setCONFIG_NFT_FIB_INET=mCONFIG_NFT_FIB_IPV4=mCONFIG_NFT_FIB_IPV6=mCONFIG_NFT_FIB_NETDEV=mCONFIG_NFT_FWD_NETDEV=mCONFIG_NFT_HASH=mCONFIG_NFT_LIMIT=mCONFIG_NFT_LOG=mCONFIG_NFT_MASQ_IPV4=mCONFIG_NFT_MASQ=m
#CONFIG_NFT_METAis not setCONFIG_NFT_NAT=mCONFIG_NFT_NUMGEN=m
#CONFIG_NFT_OBJREFis not setCONFIG_NFT_QUEUE=mCONFIG_NFT_QUOTA=mCONFIG_NFT_REDIR_IPV4=mCONFIG_NFT_REDIR=mCONFIG_NFT_REJECT=m
#CONFIG_NFT_RTis not set
#CONFIG_NFT_SET_BITMAPis not set
#CONFIG_NFT_SET_HASHis not set
#CONFIG_NFT_SET_RBTREEis not setCONFIG_OVERLAY_FS=mCONFIG_STP=mQuoted 4 lines
Quoted 3 lines
BR,
/Joakim
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may
also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 3 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 182 lines · Joakim Roubert
Quoted 4 lines · Bruce Ashfield
I now have another 6 or 7 WIP patches on top of this to try and get a single node "cluster" working with k3s. I'll clean them up and get them into the k3s WIP branch shortly.
Awesome!
Quoted 2 lines
In your working references, which iptables do you have installed ? (legacy ? nftables?)
# iptables --version iptables v1.8.4 (legacy)Quoted 13 lines
I'm failing to get flannel to start, with a series of errors like this:
-----------
I1106 21:19:00.985656 10641 eviction_manager.go:351] eviction
manager: able to reduce ephemeral-storage pressure without evicting pods.
E1106 21:19:10.636899 10641 proxier.go:841] Failed to ensure that
filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error checking
rule: exit status 2: iptables v1.8.6 (legacy): Couldn't load match `comment':No such y
Try `iptables -h' or 'iptables --help' for more information.
I1106 21:19:10.641647 10641 proxier.go:825] Sync failed; retrying in 30s
------------This is a bit strange, as it seems you are running in legacy mode too, although a somewhat newer version than I have, and the only thing I know is that Rancher recommends 1.6.1 or newer (which it is).
https://rancher.com/docs/k3s/latest/en/known-issues/
Might there be something missing in the kernel?
I have these RPMs installed in my image:
iptables
iptables-module-ip6t-ah
iptables-module-ip6t-dnat
iptables-module-ip6t-dnpt
iptables-module-ip6t-dst
iptables-module-ip6t-eui64
iptables-module-ip6t-frag
iptables-module-ip6t-hbh
iptables-module-ip6t-hl
iptables-module-ip6t-icmp6
iptables-module-ip6t-ipv6header
iptables-module-ip6t-log
iptables-module-ip6t-masquerade
iptables-module-ip6t-mh
iptables-module-ip6t-netmap
iptables-module-ip6t-redirect
iptables-module-ip6t-reject
iptables-module-ip6t-rt
iptables-module-ip6t-snat
iptables-module-ip6t-snpt
iptables-module-ip6t-srh
iptables-module-ipt-ah
iptables-module-ipt-clusterip
iptables-module-ipt-dnat
iptables-module-ipt-ecn
iptables-module-ipt-icmp
iptables-module-ipt-log
iptables-module-ipt-masquerade
iptables-module-ipt-netmap
iptables-module-ipt-realm
iptables-module-ipt-redirect
iptables-module-ipt-reject
iptables-module-ipt-snat
iptables-module-ipt-ttl
iptables-module-ipt-ulog
iptables-modules
iptables-module-xt-addrtype
iptables-module-xt-audit
iptables-module-xt-bpf
iptables-module-xt-cgroup
iptables-module-xt-checksum
iptables-module-xt-classify
iptables-module-xt-cluster
iptables-module-xt-comment
iptables-module-xt-connbytes
iptables-module-xt-connlimit
iptables-module-xt-connmark
iptables-module-xt-connsecmark
iptables-module-xt-conntrack
iptables-module-xt-cpu
iptables-module-xt-ct
iptables-module-xt-dccp
iptables-module-xt-devgroup
iptables-module-xt-dscp
iptables-module-xt-ecn
iptables-module-xt-esp
iptables-module-xt-hashlimit
iptables-module-xt-helper
iptables-module-xt-hmark
iptables-module-xt-idletimer
iptables-module-xt-ipcomp
iptables-module-xt-iprange
iptables-module-xt-ipvs
iptables-module-xt-led
iptables-module-xt-length
iptables-module-xt-limit
iptables-module-xt-mac
iptables-module-xt-mark
iptables-module-xt-multiport
iptables-module-xt-nfacct
iptables-module-xt-nflog
iptables-module-xt-nfqueue
iptables-module-xt-osf
iptables-module-xt-owner
iptables-module-xt-physdev
iptables-module-xt-pkttype
iptables-module-xt-policy
iptables-module-xt-quota
iptables-module-xt-rateest
iptables-module-xt-recent
iptables-module-xt-rpfilter
iptables-module-xt-sctp
iptables-module-xt-secmark
iptables-module-xt-set
iptables-module-xt-socket
iptables-module-xt-standard
iptables-module-xt-statistic
iptables-module-xt-string
iptables-module-xt-synproxy
iptables-module-xt-tcp
iptables-module-xt-tcpmss
iptables-module-xt-tcpoptstrip
iptables-module-xt-tee
iptables-module-xt-time
iptables-module-xt-tos
iptables-module-xt-tproxy
iptables-module-xt-trace
iptables-module-xt-u32
iptables-module-xt-udpand in my kernel config, I have (apart from what is needed for running containers with containerd):
CONFIG_NETFILTER_NETLINK=mCONFIG_NETFILTER_XT_MATCH_OWNER=mCONFIG_NET_UDP_TUNNEL=mCONFIG_NF_DUP_NETDEV=mCONFIG_NF_LOG_BRIDGE=mCONFIG_NF_TABLES_ARP=yCONFIG_NF_TABLES_BRIDGE=yCONFIG_NF_TABLES_INET=yCONFIG_NF_TABLES_IPV4=yCONFIG_NF_TABLES_IPV6=yCONFIG_NF_TABLES=mCONFIG_NF_TABLES_NETDEV=yCONFIG_NFT_BRIDGE_REJECT=mCONFIG_NFT_CHAIN_NAT_IPV4=mCONFIG_NFT_CHAIN_ROUTE_IPV4=mCONFIG_NFT_CHAIN_ROUTE_IPV6=mCONFIG_NFT_COMPAT=mCONFIG_NFT_COUNTER=mCONFIG_NFT_CT=mCONFIG_NFT_DUP_IPV4=mCONFIG_NFT_DUP_IPV6=mCONFIG_NFT_DUP_NETDEV=m
#CONFIG_NFT_EXTHDRis not setCONFIG_NFT_FIB_INET=mCONFIG_NFT_FIB_IPV4=mCONFIG_NFT_FIB_IPV6=mCONFIG_NFT_FIB_NETDEV=mCONFIG_NFT_FWD_NETDEV=mCONFIG_NFT_HASH=mCONFIG_NFT_LIMIT=mCONFIG_NFT_LOG=mCONFIG_NFT_MASQ_IPV4=mCONFIG_NFT_MASQ=m
#CONFIG_NFT_METAis not setCONFIG_NFT_NAT=mCONFIG_NFT_NUMGEN=m
#CONFIG_NFT_OBJREFis not setCONFIG_NFT_QUEUE=mCONFIG_NFT_QUOTA=mCONFIG_NFT_REDIR_IPV4=m
Quoted 8 lines
CONFIG_NFT_REDIR=mCONFIG_NFT_REJECT=m
#CONFIG_NFT_RTis not set
#CONFIG_NFT_SET_BITMAPis not set
#CONFIG_NFT_SET_HASHis not set
#CONFIG_NFT_SET_RBTREEis not setCONFIG_OVERLAY_FS=mCONFIG_STP=m
Cool, the above helps.
I have some new fragments, etc, developed for both k8s and k3s, so I'll double check that I didn't fat finger something.
Bruce
Quoted 4 lines
BR,
/Joakim
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Hi Bruce and Joakim,
Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.
My Image: Linux qemuarm64 by yocto.
The master node can be ready after I started the k3s server. However, the pods in kube-system (which are essential components for k3s) cannot turn to ready state on qemuarm64.
After the master node itself turned to ready state, I check the pods with kubectl:
kubectl get nodes
NAME STATUS ROLES AGE VERSION qemuarm64 Ready master 11m v1.18.9-k3s1 root@qemuarm64:~# ls root@qemuarm64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-xxvbl 0/1 ContainerCreating 0 12m coredns-7944c66d8d-tlrm9 0/1 ContainerCreating 0 12m metrics-server-7566d596c8-svkff 0/1 ContainerCreating 0 12m helm-install-traefik-s8p5g 0/1 ContainerCreating 0 12m
Then I describe the pods with:
Events:
Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 16m default-scheduler Successfully assigned kube-system/coredns-7944c66d8d-tlrm9 to qemuarm64 Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to attach or mount volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token-b7nlh config-volume]: timed out waiting for the condition Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach or mount volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume coredns-token-b7nlh]: timed out waiting for the condition Warning FailedMount 11s (x16 over 16m) kubelet MountVolume.SetUp failed for volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable file not found in $PATH
I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and mount binary on my qemuarm64 image:
root@qemuarm64:~# echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin root@qemuarm64:~# which mount /bin/mount
When I type mount command, it worked fine:
/dev/root on / type ext4 (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
...
... (skipped the verbose output)
I would like to know whether you have met this "mount" issue ever?
Best Regards,
Lance
Quoted 52 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield
Ha!!!!
This applies.
Quoted 7 lines
Wonderful, thank you! I guess this is what is called "five times lucky"...
Quoted 3 lines
I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.
Excellent!
I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/log/?h=k3s-wipThat includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.
I did have a couple of questions on the k3s packaging itself, I was getting the following error:
ERROR: k3s-v1.18.9+k3s1-dirty-r0
do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3sSo I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?
Bruce
Quoted 8 lines
BR,
/Joakim
--
Joakim Roubert
Senior Engineer
Quoted 8 lines
Quoted 5 lines
Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 4 lines · Lance Yang
Hi Bruce and Joakim,
Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.
The branch will be more functional shortly, I have quite a few changes to factor things for
k8s and generally more usable :D
modified: classes/cni_networking.bbclass
modified: conf/layer.conf
modified: recipes-containers/containerd/containerd-docker_git.bb
modified:
recipes-containers/containerd/containerd-opencontainers_git.bb
modified: recipes-containers/k3s/README.md
modified: recipes-containers/k3s/k3s_git.bb
modified: recipes-kernel/linux/linux-yocto/kubernetes.cfg
modified: recipes-networking/cni/cni_git.bb container-deploy.txt
recipes-core/packagegroups/
Quoted 5 lines
My Image: Linux qemuarm64 by yocto.
The master node can be ready after I started the k3s server. However, the pods in kube-system (which are essential components for k3s) cannot turn to ready state on qemuarm64.
That's interesting, since in my configuration, the master never comes ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 NotReady master 15h v1.18.9-k3s1
I've sorted out more of the dependencies, and have packagegroups to make them easier
now.
Hopefully, I can figure out what is now missing and keeping my master from moving into
ready today.
Bruce
Quoted 107 lines
After the master node itself turned to ready state, I check the pods with kubectl:
kubectl get nodes
NAME STATUS ROLES AGE VERSION qemuarm64 Ready master 11m v1.18.9-k3s1 root@qemuarm64:~# ls root@qemuarm64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-xxvbl 0/1 ContainerCreating 0 12m coredns-7944c66d8d-tlrm9 0/1 ContainerCreating 0 12m metrics-server-7566d596c8-svkff 0/1 ContainerCreating 0 12m helm-install-traefik-s8p5g 0/1 ContainerCreating 0 12mThen I describe the pods with:
Events:
Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 16m default-scheduler Successfully assigned kube-system/coredns-7944c66d8d-tlrm9 to qemuarm64 Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to attach or mount volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token-b7nlh config-volume]: timed out waiting for the condition Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach or mount volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume coredns-token-b7nlh]: timed out waiting for the condition Warning FailedMount 11s (x16 over 16m) kubelet MountVolume.SetUp failed for volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable file not found in $PATHI found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and mount binary on my qemuarm64 image:
root@qemuarm64:~# echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin root@qemuarm64:~# which mount /bin/mountWhen I type mount command, it worked fine:
/dev/rooton / type ext4 (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
debugfs on/sys/kernel/debugtype debugfs (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
...
... (skipped the verbose output)I would like to know whether you have met this "mount" issue ever?
Best Regards,
LanceQuoted 52 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield
Ha!!!!
This applies.
Quoted 7 lines
Wonderful, thank you! I guess this is what is called "five times lucky"...
Quoted 3 lines
I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.
Excellent!
I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/log/?h=k3s-wipThat includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.
I did have a couple of questions on the k3s packaging itself, I was getting the following error:
ERROR: k3s-v1.18.9+k3s1-dirty-r0
do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3sSo I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?
Bruce
Quoted 8 lines
BR,
/Joakim
--
Joakim Roubert
Senior EngineerQuoted 8 lines
Quoted 5 lines
Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIIMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 26 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 4 lines · Lance Yang
Hi Bruce and Joakim,
Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.
The branch will be more functional shortly, I have quite a few changes to factor things for
k8s and generally more usable :Dmodified: classes/cni_networking.bbclass modified: conf/layer.conf modified: recipes-containers/containerd/containerd-docker_git.bb modified: recipes-containers/containerd/containerd-opencontainers_git.bb modified: recipes-containers/k3s/README.md modified: recipes-containers/k3s/k3s_git.bb modified: recipes-kernel/linux/linux-yocto/kubernetes.cfg modified: recipes-networking/cni/cni_git.bbcontainer-deploy.txt
recipes-core/packagegroups/Quoted 3 lines
My Image: Linux qemuarm64 by yocto.
Quoted 9 lines
Quoted 2 lines
The master node can be ready after I started the k3s server. However, the pods in kube-system (which are essential components for k3s) cannot turn to ready state on qemuarm64.
That's interesting, since in my configuration, the master never comes ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 NotReady master 15h v1.18.9-k3s1
Hah.
I finally got the node to show up as ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 Ready master 112s v1.18.9-k3s1
I'm attempting to build an all-in-one node, and that is likely causing me some issues.
I'm revisiting those potential conflicts now.
But if anyone else does have an all in one working and has some tips, feel free to share :D
Bruce
Quoted 127 lines
I've sorted out more of the dependencies, and have packagegroups to make them easier
now.Hopefully, I can figure out what is now missing and keeping my master from moving into
ready today.Bruce
Quoted 103 lines
After the master node itself turned to ready state, I check the pods with kubectl:
kubectl get nodes
NAME STATUS ROLES AGE VERSION qemuarm64 Ready master 11m v1.18.9-k3s1 root@qemuarm64:~# ls root@qemuarm64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-xxvbl 0/1 ContainerCreating 0 12m coredns-7944c66d8d-tlrm9 0/1 ContainerCreating 0 12m metrics-server-7566d596c8-svkff 0/1 ContainerCreating 0 12m helm-install-traefik-s8p5g 0/1 ContainerCreating 0 12mThen I describe the pods with:
Events:
Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 16m default-scheduler Successfully assigned kube-system/coredns-7944c66d8d-tlrm9 to qemuarm64 Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to attach or mount volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token-b7nlh config-volume]: timed out waiting for the condition Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach or mount volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume coredns-token-b7nlh]: timed out waiting for the condition Warning FailedMount 11s (x16 over 16m) kubelet MountVolume.SetUp failed for volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable file not found in $PATHI found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and mount binary on my qemuarm64 image:
root@qemuarm64:~# echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin root@qemuarm64:~# which mount /bin/mountWhen I type mount command, it worked fine:
/dev/rooton / type ext4 (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
debugfs on/sys/kernel/debugtype debugfs (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
...
... (skipped the verbose output)I would like to know whether you have met this "mount" issue ever?
Best Regards,
LanceQuoted 57 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield
Ha!!!!
This applies.
Quoted 7 lines
Wonderful, thank you! I guess this is what is called "five times lucky"...
Quoted 3 lines
I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.
Excellent!
I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/log/?h=k3s-wipThat includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.
I did have a couple of questions on the k3s packaging itself, I was getting the following error:
ERROR: k3s-v1.18.9+k3s1-dirty-r0
do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3sSo I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?
Bruce
Quoted 13 lines
BR,
/Joakim
--
Joakim Roubert
Senior EngineerAxis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.comQuoted 4 lines
Quoted 3 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIIMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Hi Bruce,
I pulled your latest k3s-wip branch. In your recipes-networking, the ipset:
cat ipset_6.38.bb
# Copyright (C) 2017 Aaron Brice <[email protected]> # Released under the MIT license (see COPYING.MIT for the terms)
...
... (skip verbose output)
DEPENDS = "libtool libmnl"
RDEPENDS_${PN} = "kernel-module-ip-set"
...
I use bitbake to install k3s and ipset but I got errors below:
Problem 1: conflicting requests
- nothing provides kernel-module-ip-set needed by ipset-6.38-r0
I think RDEPENDS_${PN} is a slightly over-strict condition in some situations. For example, in the following situation:
In my build/conf/local.conf, I added "kernel-modules" in IMAGE_INSTALL_append section. kernel-modules-ip-set has already included in kernel-modules when I check in qemuarm64 image with "lsmod|grep ip_set". But if RDEPENDS is used, bitbake won't continue to compile the package unless you add "kernel-module-ip-set".
So I suggested to use " RRECOMMENDS_${PN}".
Best Regards,
Lance
Quoted 204 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Tuesday, November 10, 2020 9:17 PM
To: Bruce Ashfield <[email protected]>
Cc: Lance Yang <[email protected]>; Joakim Roubert <[email protected]>; meta- [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 19 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 4 lines · Lance Yang
Hi Bruce and Joakim,
Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.
The branch will be more functional shortly, I have quite a few changes to factor things for k8s and generally more usable :D
modified: classes/cni_networking.bbclass modified: conf/layer.conf modified: recipes-containers/containerd/containerd-docker_git.bb modified: recipes-containers/containerd/containerd-opencontainers_git.bb modified: recipes-containers/k3s/README.md modified: recipes-containers/k3s/k3s_git.bb modified: recipes-kernel/linux/linux-yocto/kubernetes.cfg modified: recipes-networking/cni/cni_git.bbQuoted 7 lines
container-deploy.txt
recipes-core/packagegroups/Quoted 4 lines
My Image: Linux qemuarm64 by yocto.
The master node can be ready after I started the k3s server. However, the pods in kube-
system (which are essential components for k3s) cannot turn to ready state on qemuarm64.
Quoted 8 lines
Quoted 1 line
That's interesting, since in my configuration, the master never comes ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 NotReady master 15h v1.18.9-k3s1Hah.
I finally got the node to show up as ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 Ready master 112s v1.18.9-k3s1I'm attempting to build an all-in-one node, and that is likely causing me some issues.
I'm revisiting those potential conflicts now.
But if anyone else does have an all in one working and has some tips, feel free to share :D
Bruce
Quoted 27 lines
I've sorted out more of the dependencies, and have packagegroups to make them easier now.
Hopefully, I can figure out what is now missing and keeping my master from moving into ready today.
Bruce
Quoted 19 lines
After the master node itself turned to ready state, I check the pods with kubectl:
kubectl get nodes
NAME STATUS ROLES AGE VERSION qemuarm64 Ready master 11m v1.18.9-k3s1 root@qemuarm64:~# ls root@qemuarm64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-xxvbl 0/1 ContainerCreating 0 12m coredns-7944c66d8d-tlrm9 0/1 ContainerCreating 0 12m metrics-server-7566d596c8-svkff 0/1 ContainerCreating 0 12m helm-install-traefik-s8p5g 0/1 ContainerCreating 0 12mThen I describe the pods with:
Events:
Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 16m default-scheduler Successfully assigned kube-system/coredns-7944c66d8d-tlrm9 to qemuarm64
Quoted 1 line
Quoted 1 line
Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to attach or mount
volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token- b7nlh config-volume]: timed out waiting for the condition
Quoted 1 line
Quoted 1 line
Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach or mount
volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume coredns-token-b7nlh]: timed out waiting for the condition
Quoted 1 line
Quoted 1 line
Warning FailedMount 11s (x16 over 16m) kubelet MountVolume.SetUp failed for
volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable file not found in $PATH
Quoted 2 lines
Quoted 2 lines
I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and
mount binary on my qemuarm64 image:
Quoted 81 lines
Quoted 81 lines
root@qemuarm64:~# echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin root@qemuarm64:~# which mount /bin/mountWhen I type mount command, it worked fine:
/dev/rooton / type ext4 (rw,relatime) devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) debugfs on/sys/kernel/debugtype debugfs (rw,relatime) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
...
... (skipped the verbose output)I would like to know whether you have met this "mount" issue ever?
Best Regards,
LanceQuoted 60 lines
-----Original Message-----
From: [email protected] <[email protected]>
On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield
Ha!!!!
This applies.
Quoted 7 lines
Wonderful, thank you! I guess this is what is called "five times lucky"...
Quoted 3 lines
I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.
Excellent!
I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/log
/?h=k3s-wipThat includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.
I did have a couple of questions on the k3s packaging itself, I was getting the following
error:ERROR: k3s-v1.18.9+k3s1-dirty-r0
do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3sSo I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?
Bruce
Quoted 13 lines
BR,
/Joakim
--
Joakim Roubert
Senior EngineerAxis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.comQuoted 5 lines
Quoted 5 lines
Quoted 4 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIIMPORTANT NOTICE: The contents of this email and any attachments are confidential and
may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 10 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
Quoted 3 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 18 lines · Lance Yang
Hi Bruce,
I pulled your latest k3s-wip branch. In your recipes-networking, the ipset:
cat ipset_6.38.bb
# Copyright (C) 2017 Aaron Brice <[email protected]> # Released under the MIT license (see COPYING.MIT for the terms)...
... (skip verbose output)DEPENDS = "libtool libmnl"
RDEPENDS_${PN} = "kernel-module-ip-set"
...I use bitbake to install k3s and ipset but I got errors below:
Quoted 8 lines
Problem 1: conflicting requests
- nothing provides kernel-module-ip-set needed by ipset-6.38-r0I think RDEPENDS_${PN} is a slightly over-strict condition in some situations. For example, in the following situation:
In my
build/conf/local.conf, I added "kernel-modules" inIMAGE_INSTALL_appendsection. kernel-modules-ip-set has already included in kernel-modules when I check in qemuarm64 image with "lsmod|grepip_set". But if RDEPENDS is used, bitbake won't continue to compile the package unless you add "kernel-module-ip-set".So I suggested to use " RRECOMMENDS_${PN}".
That's already the case in the forked recipe (rdepends), and there's significant issues with getting the right kernel configuration .. so I'm going to leave it as a rdepends, since if you have that not as a module, it means you aren't using the linux-yocto reference configs (or the reference configs are broken), and that there may be other issues in your configuration.
Bruce
Quoted 211 lines
Best Regards,
LanceQuoted 204 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Tuesday, November 10, 2020 9:17 PM
To: Bruce Ashfield <[email protected]>
Cc: Lance Yang <[email protected]>; Joakim Roubert <[email protected]>; meta- [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 19 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 4 lines · Lance Yang
Hi Bruce and Joakim,
Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.
The branch will be more functional shortly, I have quite a few changes to factor things for k8s and generally more usable :D
modified: classes/cni_networking.bbclass modified: conf/layer.conf modified: recipes-containers/containerd/containerd-docker_git.bb modified: recipes-containers/containerd/containerd-opencontainers_git.bb modified: recipes-containers/k3s/README.md modified: recipes-containers/k3s/k3s_git.bb modified: recipes-kernel/linux/linux-yocto/kubernetes.cfg modified: recipes-networking/cni/cni_git.bbQuoted 7 lines
container-deploy.txt
recipes-core/packagegroups/Quoted 4 lines
My Image: Linux qemuarm64 by yocto.
The master node can be ready after I started the k3s server. However, the pods in kube-
system (which are essential components for k3s) cannot turn to ready state on qemuarm64.
Quoted 8 lines
Quoted 1 line
That's interesting, since in my configuration, the master never comes ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 NotReady master 15h v1.18.9-k3s1Hah.
I finally got the node to show up as ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 Ready master 112s v1.18.9-k3s1I'm attempting to build an all-in-one node, and that is likely causing me some issues.
I'm revisiting those potential conflicts now.
But if anyone else does have an all in one working and has some tips, feel free to share :D
Bruce
Quoted 27 lines
I've sorted out more of the dependencies, and have packagegroups to make them easier now.
Hopefully, I can figure out what is now missing and keeping my master from moving into ready today.
Bruce
Quoted 19 lines
After the master node itself turned to ready state, I check the pods with kubectl:
kubectl get nodes
NAME STATUS ROLES AGE VERSION qemuarm64 Ready master 11m v1.18.9-k3s1 root@qemuarm64:~# ls root@qemuarm64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-xxvbl 0/1 ContainerCreating 0 12m coredns-7944c66d8d-tlrm9 0/1 ContainerCreating 0 12m metrics-server-7566d596c8-svkff 0/1 ContainerCreating 0 12m helm-install-traefik-s8p5g 0/1 ContainerCreating 0 12mThen I describe the pods with:
Events:
Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 16m default-scheduler Successfully assigned kube-system/coredns-7944c66d8d-tlrm9 to qemuarm64
Quoted 1 line
Quoted 1 line
Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to attach or mount
volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token- b7nlh config-volume]: timed out waiting for the condition
Quoted 1 line
Quoted 1 line
Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach or mount
volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume coredns-token-b7nlh]: timed out waiting for the condition
Quoted 1 line
Quoted 1 line
Warning FailedMount 11s (x16 over 16m) kubelet MountVolume.SetUp failed for
volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable file not found in $PATH
Quoted 2 lines
Quoted 2 lines
I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and
mount binary on my qemuarm64 image:
Quoted 81 lines
Quoted 81 lines
root@qemuarm64:~# echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin root@qemuarm64:~# which mount /bin/mountWhen I type mount command, it worked fine:
/dev/rooton / type ext4 (rw,relatime) devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) debugfs on/sys/kernel/debugtype debugfs (rw,relatime) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
...
... (skipped the verbose output)I would like to know whether you have met this "mount" issue ever?
Best Regards,
LanceQuoted 60 lines
-----Original Message-----
From: [email protected] <[email protected]>
On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield
Ha!!!!
This applies.
Quoted 7 lines
Wonderful, thank you! I guess this is what is called "five times lucky"...
Quoted 3 lines
I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.
Excellent!
I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/log
/?h=k3s-wipThat includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.
I did have a couple of questions on the k3s packaging itself, I was getting the following
error:ERROR: k3s-v1.18.9+k3s1-dirty-r0
do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3sSo I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?
Bruce
Quoted 13 lines
BR,
/Joakim
--
Joakim Roubert
Senior EngineerAxis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.comQuoted 5 lines
Quoted 5 lines
Quoted 4 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIIMPORTANT NOTICE: The contents of this email and any attachments are confidential and
may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 10 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIQuoted 3 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIIMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
PATCH 1/1 k3s: Update README.md
Fix suggestions from markdownlint, i.e.
docker run -it --rm -v $(pwd):/x -w /x \
peterdavehello/markdownlint:0.22.0 markdownlint .
and some formatting done.
--- recipes-containers/k3s/README.md | 63 +++++++++++++++++--------------- 1 file changed, 34 insertions(+), 29 deletions(-)
recipes-containers/k3s/README.md +34 -29
@@ -29,35 +29,37 @@ k3s-agent -t <token> -s https://<master>:6443 (Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the k3s master.) -Example: +### Example + ```shell k3s-agent -t /var/lib/rancher/k3s/server/node-token -s https://localhost:6443 ``` -## Notes: +## Notes -if running under qemu, the default of 256M of memory is not enough, k3s will -OOM and exit. +**If running under qemu**, the default of 256 MB of memory is not enough, k3s +will OOM and exit. -Boot with qemuparams="-m 2048" to boot with 2G of memory (or choose the +Boot with `qemuparams="-m 2048"` to boot with 2 GB of memory (or choose the appropriate amount for your configuration) -Disk: if using qemu and core-image* you'll need to add extra space in your disks -to ensure containers can start. The following in your image recipe, or local.conf -would add 2G of extra space to the rootfs: +Disk: if using qemu and core-image* you will need to add extra space in your +disks to ensure containers can start. The following in your image recipe, +or `local.conf` would add 2 GB of extra space to the rootfs: ```shell IMAGE_ROOTFS_EXTRA_SPACE = "2097152" ``` -## Example output from qemux86-64 running k3s server: +## Example output from qemux86-64 running k3s server ```shell root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 Ready master 46s v1.18.9-k3s1 +``` - +```shell root@qemux86-64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-h7lxk 1/1 Running 0 2m32s
@@ -66,7 +68,9 @@ helm-install-traefik-229v7 0/1 Completed 0 2m32s coredns-7944c66d8d-9rfj7 1/1 Running 0 2m32s svclb-traefik-pb5j4 2/2 Running 0 89s traefik-758cd5fc85-lxpr8 1/1 Running 0 89s +``` +```shell root@qemux86-64:~# kubectl describe pods -n kube-system root@qemux86-64:~# ip a s
@@ -74,25 +78,25 @@ root@qemux86-64:~# ip a s link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever - inet6 ::1/128 scope host + inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:12:35:02 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0 valid_lft forever preferred_lft forever - inet6 fec0::5054:ff:fe12:3502/64 scope site dynamic mngtmpaddr + inet6 fec0::5054:ff:fe12:3502/64 scope site dynamic mngtmpaddrg valid_lft 86239sec preferred_lft 14239sec - inet6 fe80::5054:ff:fe12:3502/64 scope link + inet6 fe80::5054:ff:fe12:3502/64 scope linkg valid_lft forever preferred_lft forever 3: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/sit 0.0.0.0 brd 0.0.0.0 -4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default +4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group defaultg link/ether e2:aa:04:89:e6:0a brd ff:ff:ff:ff:ff:ff inet 10.42.0.0/32 brd 10.42.0.0 scope global flannel.1 valid_lft forever preferred_lft forever - inet6 fe80::e0aa:4ff:fe89:e60a/64 scope link + inet6 fe80::e0aa:4ff:fe89:e60a/64 scope linkg valid_lft forever preferred_lft forever -5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default +5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group defaultg link/ether 02:42:be:3e:25:e7 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever
@@ -100,30 +104,31 @@ root@qemux86-64:~# ip a s link/ether 82:8e:b4:f8:06:e7 brd ff:ff:ff:ff:ff:ff inet 10.42.0.1/24 brd 10.42.0.255 scope global cni0 valid_lft forever preferred_lft forever - inet6 fe80::808e:b4ff:fef8:6e7/64 scope link + inet6 fe80::808e:b4ff:fef8:6e7/64 scope linkg valid_lft forever preferred_lft forever -7: veth82ac482e@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default +7: veth82ac482e@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group defaultg link/ether ea:9d:14:c1:00:70 brd ff:ff:ff:ff:ff:ff link-netns cni-c52e6e09-f6e0-a47b-aea3-d6c47d3e2d01 - inet6 fe80::e89d:14ff:fec1:70/64 scope link + inet6 fe80::e89d:14ff:fec1:70/64 scope linkg valid_lft forever preferred_lft forever -8: vethb94745ed@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default +8: vethb94745ed@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group defaultg link/ether 1e:7f:7e:d3:ca:e8 brd ff:ff:ff:ff:ff:ff link-netns cni-86958efe-2462-016f-292d-81dbccc16a83 - inet6 fe80::8046:3cff:fe23:ced1/64 scope link + inet6 fe80::8046:3cff:fe23:ced1/64 scope linkg valid_lft forever preferred_lft forever -9: veth81ffb276@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default +9: veth81ffb276@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group defaultg link/ether 2a:1d:48:54:76:50 brd ff:ff:ff:ff:ff:ff link-netns cni-5d77238e-6452-4fa3-40d2-91d48386080b - inet6 fe80::acf4:7fff:fe11:b6f2/64 scope link + inet6 fe80::acf4:7fff:fe11:b6f2/64 scope linkg valid_lft forever preferred_lft forever -10: vethce261f6a@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default +10: vethce261f6a@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group defaultg link/ether 72:a3:90:4a:c5:12 brd ff:ff:ff:ff:ff:ff link-netns cni-55675948-77f2-a952-31ce-615f2bdb0093 - inet6 fe80::4d5:1bff:fe5d:db3a/64 scope link + inet6 fe80::4d5:1bff:fe5d:db3a/64 scope linkg valid_lft forever preferred_lft forever -11: vethee199cf4@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default +11: vethee199cf4@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group defaultg link/ether e6:90:a4:a3:bc:a1 brd ff:ff:ff:ff:ff:ff link-netns cni-4aeccd16-2976-8a78-b2c4-e028da3bb1ea - inet6 fe80::c85a:8bff:fe0b:aea0/64 scope link + inet6 fe80::c85a:8bff:fe0b:aea0/64 scope linkg valid_lft forever preferred_lft forever +``` - +```shell root@qemux86-64:~# kubectl describe nodes Name: qemux86-64
@@ -235,4 +240,4 @@ Events: Normal NodeHasSufficientPID 10m (x2 over 10m) kubelet Node qemux86-64 status is now: NodeHasSufficientPID Normal NodeReady 10m kubelet Node qemux86-64 status is now: NodeReady -```shell +```
Signature · 1 line
2.20.1
PATCH 1/1 k3s: Bump to v1.19.3+k3s3
In reply to Bruce AshfieldThe current stable release installed when running the regular k3s install script is now v1.19.3+k3s3. Using that version in our recipe also mitigates the fixes for
https://github.com/rancher/kine/issues/61
--- recipes-containers/k3s/k3s_git.bb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
recipes-containers/k3s/k3s_git.bb +3 -3
@@ -4,7 +4,7 @@ HOMEPAGE = "https://k3s.io/" LICENSE = "Apache-2.0" LIC_FILES_CHKSUM = "file://${S}/src/import/LICENSE;md5=2ee41112a44fe7014dce33e26468ba93" -SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.18;name=k3s \ +SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.19;name=k3s \ file://k3s.service \ file://k3s-agent.service \ file://k3s-agent \
@@ -13,9 +13,9 @@ SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.18;name=k3s \ file://0001-Finding-host-local-in-usr-libexec.patch;patchdir=src/import \ " SRC_URI[k3s.md5sum] = "363d3a08dc0b72ba6e6577964f6e94a5" -SRCREV_k3s = "630bebf94b9dce6b8cd3d402644ed023b3af8f90" +SRCREV_k3s = "0e4fbfefe1dd8734756dfa4f9ab4fc89665cece4" -PV = "v1.18.9+git${SRCPV}" +PV = "v1.19.3+git${SRCPV}" CNI_NETWORKING_FILES ?= "${WORKDIR}/cni-containerd-net.conf"
Signature · 1 line
2.20.1
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 4 lines · Joakim Roubert
The current stable release installed when running the regular k3s install script is now v1.19.3+k3s3. Using that version in our recipe also mitigates the fixes for
I already have this queued in my WIP branch.
But without being able to assemble a single node working "cluster", this is still stuck in my staging branch.
Bruce
Quoted 47 lines
https://github.com/rancher/kine/issues/61
Change-Id I4c53d777d4d837628a6c7f8a9234da9d71eb45aeSigned-off-by Joakim Roubert <[email protected]>--- recipes-containers/k3s/k3s_git.bb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)recipes-containers/k3s/ renamed from recipes-containers/k3s/k3s_git.bb +0
-0k3s_git.bb
index ea5ed75..28ce66b 100644recipes-containers/k3s/k3s_git.bb +0
-0@@ -4,7 +4,7 @@ HOMEPAGE = "https://k3s.io/" LICENSE = "Apache-2.0" LIC_FILES_CHKSUM ="file://${S}/src/import/LICENSE;md5=2ee41112a44fe7014dce33e26468ba93"
-SRC_URI = "
git://github.com/rancher/k3s.git;branch=release-1.18;name=k3s\
+SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.19;name=k3s\ file://k3s.service \ file://k3s-agent.service \ file://k3s-agent \recipes-containers/k3s/k3s_git.bb +2
-2@@ -13,9 +13,9 @@ SRC_URI = "git:// github.com/rancher/k3s.git;branch=release-1.18;name=k3s \ file://0001-Finding-host-local-in-usr-libexec.patch;patchdir=src/import \ " SRC_URI[k3s.md5sum] = "363d3a08dc0b72ba6e6577964f6e94a5" -SRCREV_k3s = "630bebf94b9dce6b8cd3d402644ed023b3af8f90" +SRCREV_k3s = "0e4fbfefe1dd8734756dfa4f9ab4fc89665cece4" -PV = "v1.18.9+git${SRCPV}" +PV = "v1.19.3+git${SRCPV}"
CNI_NETWORKING_FILES?= "${WORKDIR}/cni-containerd-net.conf"--
2.20.1
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 2 lines · Bruce Ashfield
I already have this queued in my WIP branch.
Ah, good.
Quoted 1 line
But without being able to assemble a single node working "cluster", this is still stuck in my staging branch.
I am not sure what this single node cluster is. Is it a k3s master running on a single machine, where you can deploy things? What are the current issues that you face there?
BR,
/Joakim
Signature · 7 lines
Joakim Roubert Senior Engineer Axis Communications AB Emdalavägen 14, SE-223 69 Lund, Sweden Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48 Fax: +46 46 13 61 30, www.axis.com
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 4 lines · Joakim Roubert
Quoted 2 lines · Bruce Ashfield
I already have this queued in my WIP branch.
Quoted 8 lines
Ah, good.
Quoted 2 lines
But without being able to assemble a single node working "cluster", this is still stuck in my staging branch.
I am not sure what this single node cluster is. Is it a k3s master running on a single machine, where you can deploy things? What are the
Yes. That's the typical description we've been using of single node "cluster" (I put cluster in quotes, because technically .. it's not a cluster :D
Quoted 2 lines
current issues that you face there?
The issues that we were describing at length in the other longer (more muddled thread). It was very easy to get lost in that thread, so I'll start something new here.
The agent is unable to register with the master due to one of several errors, either certifications or some sort of authentication issue. I've worked through the flannel and other config issues, so they are taken care of at this point.
I'm rebuilding on a rebased master now, and will follow up with better logs when I relaunch my test.
Bruce
Quoted 17 lines
BR,
/Joakim
--
Joakim Roubert
Senior EngineerAxis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 3 lines · Bruce Ashfield
The agent is unable to register with the master due to one of several
errors
But if you have an agent, that would be 2 nodes, right? One master and one worker node?
Quoted 2 lines
I'm rebuilding on a rebased master now, and will follow up with better logs when I relaunch my test.
Excellent!
BR,
/Joakim
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 8 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield
The agent is unable to register with the master due to one of several
errorsBut if you have an agent, that would be 2 nodes, right? One master and one worker node?
In kubernetes terms, perhaps. But physical boxes .. no. This needs to be testable under a single instance (qemu in my case). I can use docker / container tricks from within the instance if required (the guides are mixed on that), but it needs to be a single box as I have no way to easily coordinate deployment on multiple boxes for my CI testing.
Bruce
Quoted 13 lines
Quoted 3 lines
I'm rebuilding on a rebased master now, and will follow up with better logs when I relaunch my test.
Excellent!
BR,
/Joakim
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 10 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 5 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield
The agent is unable to register with the master due to one of several
errors
Quoted 10 lines
Quoted 3 lines
But if you have an agent, that would be 2 nodes, right? One master and one worker node?
In kubernetes terms, perhaps. But physical boxes .. no. This needs to be testable under a single instance (qemu in my case). I can use docker / container tricks from within the instance if required (the guides are mixed on that), but it needs to be a single box as I have no way to easily coordinate deployment on multiple boxes for my CI testing.
I should also say that this is how I've been testing k8s (but that also needs work again), so it has been up and running in the past.
Bruce
Quoted 27 lines
Bruce
Quoted 13 lines
Quoted 3 lines
I'm rebuilding on a rebased master now, and will follow up with better logs when I relaunch my test.
Excellent!
BR,
/Joakim
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 7 lines · Bruce Ashfield
But if you have an agent, that would be 2 nodes, right? One master and
one worker node?In kubernetes terms, perhaps. But physical boxes .. no. This needs to be testable under a single instance (qemu in my case).
I see. Well, I have never tried that, but would in that case use one qemu instance for the master and a separate qemu instance for each worker node. (I should try to run both master and a worker on one machine and see what happens on my setup.)
BR,
/Joakim
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 6 lines · Joakim Roubert
Quoted 5 lines · Bruce Ashfield
But if you have an agent, that would be 2 nodes, right? One master and
one worker node?
Quoted 8 lines
Quoted 2 lines
In kubernetes terms, perhaps. But physical boxes .. no. This needs to be testable under a single instance (qemu in my case).
I see. Well, I have never tried that, but would in that case use one qemu instance for the master and a separate qemu instance for each worker node. (I should try to run both master and a worker on one machine and see what happens on my setup.)
I'm experimenting with two qemu sessions as well (I had to do this with meta-openstack), but then host networking becomes an issue and if that's also the box you are doing builds on (like I am), it can cause issues.
Either way, I'm fairly close now and hopefully will have this sorted out soon.
My build is still running, but I'll post specific logs later today.
Bruce
Quoted 9 lines
BR,
/Joakim
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Here's problem #1, using qemux86-64 and the latest on my k3s-wip branch, while the master shows itself as ready, the pods are crashing. I need to sort that out before I can get into the single node config.
root@qemux86-64:~# uname -a
Linux qemux86-64 5.8.13-yocto-standard #1 SMP PREEMPT Tue Oct 6 12:23:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@qemux86-64:~# k3s --version
k3s version v1.19.3+gitAUTOINC+970fbc66d3 (970fbc66)
root@qemux86-64:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
qemux86-64 Ready master 17m v1.19.3-k3s1
And flannel is up:
5: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether be:5e:55:3e:52:e6 brd ff:ff:ff:ff:ff:ff
inet 10.42.0.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::bc5e:55ff:fe3e:52e6/64 scope link
valid_lft forever preferred_lft forever6: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether e2:38:35:11:31:6e brd ff:ff:ff:ff:ff:ff
inet 10.42.0.1/24 brd 10.42.0.255 scope global cni0
valid_lft forever preferred_lft forever
inet6 fe80::e038:35ff:fe11:316e/64 scope link
valid_lft forever preferred_lft forever1081: veth89c8ca0d@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether b6:76:4f:77:e7:a6 brd ff:ff:ff:ff:ff:ff link-netns
cni-255cf837-ca3c-f08d-730f-bbb359cb6ae1
inet6 fe80::8cf:bff:febb:85d7/64 scope link
valid_lft forever preferred_lft forever
1082: veth001513a2@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether 52:5c:64:41:c6:e1 brd ff:ff:ff:ff:ff:ff link-netns
cni-71490108-5f0a-a266-de1f-4bed565dcf88
inet6 fe80::605b:23ff:fe50:c22f/64 scope link
valid_lft forever preferred_lft forever
1083: veth4ae47008@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether be:0e:af:76:4d:66 brd ff:ff:ff:ff:ff:ff link-netns
cni-bb1a1e05-6fa1-e9dd-3de8-b3a4be47197a
inet6 fe80::8467:31ff:fec3:6b20/64 scope link
valid_lft forever preferred_lft forever
1084: veth07cd05c1@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
link/ether 5e:75:7b:bc:ee:1c brd ff:ff:ff:ff:ff:ff link-netns
cni-07e997d6-7918-0f38-b1e5-1b08e9ac0581
inet6 fe80::28e0:ceff:fe42:364d/64 scope link tentative
valid_lft forever preferred_lft forever
root@qemux86-64:~# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS
AGE
local-path-provisioner-7ff9579c6-bd947 0/1 CrashLoopBackOff 8
22m
coredns-66c464876b-76bdl 0/1 CrashLoopBackOff 8
22m
helm-install-traefik-mvt5k 0/1 CrashLoopBackOff 8
22m
metrics-server-7b4f8b595-ck5kz 0/1 CrashLoopBackOff 8
22m
root@qemux86-64:~#
Those failures are a bit different than I used to see before going to 1.19
For whatever reason, my logs are empty, so I'm trying to see if my k3s host image is missing something.
Bruce
Quoted 52 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 6 lines · Joakim Roubert
Quoted 5 lines · Bruce Ashfield
But if you have an agent, that would be 2 nodes, right? One master and
one worker node?Quoted 8 lines
Quoted 2 lines
In kubernetes terms, perhaps. But physical boxes .. no. This needs to be testable under a single instance (qemu in my case).
I see. Well, I have never tried that, but would in that case use one qemu instance for the master and a separate qemu instance for each worker node. (I should try to run both master and a worker on one machine and see what happens on my setup.)
I'm experimenting with two qemu sessions as well (I had to do this with meta-openstack), but then host networking becomes an issue and if that's also the box you are doing builds on (like I am), it can cause issues.
Either way, I'm fairly close now and hopefully will have this sorted out soon.
My build is still running, but I'll post specific logs later today.
Bruce
Quoted 9 lines
BR,
/Joakim
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 8 lines · Bruce Ashfield
root@qemux86-64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-7ff9579c6-bd947 0/1 CrashLoopBackOff 822m coredns-66c464876b-76bdl 0/1 CrashLoopBackOff 822m helm-install-traefik-mvt5k 0/1 CrashLoopBackOff 822m metrics-server-7b4f8b595-ck5kz 0/1 CrashLoopBackOff 822m root@qemux86-64:~#
From my own experience with CrashLoopBackOff situations is that there may be serveral different things causing that. Can you trace down what is causing the CrashLoopBackOff here?
BR,
/Joakim
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 10 lines · Joakim Roubert
Quoted 8 lines · Bruce Ashfield
root@qemux86-64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-7ff9579c6-bd947 0/1 CrashLoopBackOff 822m coredns-66c464876b-76bdl 0/1 CrashLoopBackOff 822m helm-install-traefik-mvt5k 0/1 CrashLoopBackOff 822m metrics-server-7b4f8b595-ck5kz 0/1 CrashLoopBackOff 822m root@qemux86-64:~#
Quoted 4 lines
From my own experience with CrashLoopBackOff situations is that there
may be serveral different things causing that. Can you trace down what is causing the CrashLoopBackOff here?
Not really. The build seems to have some sort of misconfiguration and no logs are created (and hence can't be accessed via kubectl).
Running the server on the command line just gets us reams of messages and a lot of false leads (too many nameservers, iptables --random-fully not supported, etc ).
I've tried a few variants today, but can't get any really simple logs out. I'll do more tomorrow.
I've been able to confirm that my packagegroup/image type runs docker, containerd, and runc launched containers just fine, so this is just some quirk/idiocy of k*s that is causing the problem.
Bruce
Quoted 8 lines
BR,
/Joakim
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Hi Bruce and Joakim,
I recently saw a new branch master-next has been created in meta-virtualization. It mainly related to k3s and I saw k3s had been bumped to v1.19. I would like to know if the k3s patch will be merged into master branch soon. I remembered you mentioned k3s v1.18 has some agent registration issue and some had crash loop issues.
What is the current status of the k3s patch? Is there any further issues?
Best Regards,
Lance
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Thursday, November 19, 2020 4:39 AM
To: Joakim Roubert <[email protected]>
Cc: meta-virtualization <[email protected]>
Subject: Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
On Wed, Nov 18, 2020 at 1:27 PM Joakim Roubert <[email protected]<mailto:[email protected]>> wrote:
Quoted 8 lines · Bruce Ashfield
root@qemux86-64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-7ff9579c6-bd947 0/1 CrashLoopBackOff 822m coredns-66c464876b-76bdl 0/1 CrashLoopBackOff 822m helm-install-traefik-mvt5k 0/1 CrashLoopBackOff 822m metrics-server-7b4f8b595-ck5kz 0/1 CrashLoopBackOff 822m root@qemux86-64:~#
From my own experience with CrashLoopBackOff situations is that there
may be serveral different things causing that. Can you trace down what is causing the CrashLoopBackOff here?
Not really. The build seems to have some sort of misconfiguration and no logs are created (and hence can't be accessed via kubectl).
Running the server on the command line just gets us reams of messages and a lot of false leads (too many nameservers, iptables --random-fully not supported, etc ).
I've tried a few variants today, but can't get any really simple logs out. I'll do more tomorrow.
I've been able to confirm that my packagegroup/image type runs docker, containerd, and runc launched containers just fine, so this is just some quirk/idiocy of k*s that is causing the problem.
Bruce
BR,
/Joakim
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 10 lines · Lance Yang
Hi Bruce and Joakim,
I recently saw a new branch master-next has been created in meta-virtualization. It mainly related to k3s and I saw k3s had been bumped to v1.19. I would like to know if the k3s patch will be merged into master branch soon. I remembered you mentioned k3s v1.18 has some agent registration issue and some had crash loop issues.
What is the current status of the k3s patch? Is there any further issues?
It is just in master-next, to show that some cleanups have been done, and the supporting component version bumps that I've done, but no, it still doesn't work under qemux86-64, so it can't actually merge to master.
The node becomes ready, but all the core containers go into a crashbackoff loop, kubectl logs doesn't work, and we still can't register a client with it (likely because of symptom #1).
I'm also chasing down an iptables issue (that I had previously fixed), but should have that sorted shortly.
Bruce
Quoted 72 lines
Best Regards,
Lance
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Thursday, November 19, 2020 4:39 AM
To: Joakim Roubert <[email protected]>
Cc: meta-virtualization <[email protected]>
Subject: Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3On Wed, Nov 18, 2020 at 1:27 PM Joakim Roubert <[email protected]> wrote:
Quoted 8 lines · Bruce Ashfield
root@qemux86-64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-7ff9579c6-bd947 0/1 CrashLoopBackOff 822m coredns-66c464876b-76bdl 0/1 CrashLoopBackOff 822m helm-install-traefik-mvt5k 0/1 CrashLoopBackOff 822m metrics-server-7b4f8b595-ck5kz 0/1 CrashLoopBackOff 822m root@qemux86-64:~#From my own experience with CrashLoopBackOff situations is that there
may be serveral different things causing that. Can you trace down what is causing the CrashLoopBackOff here?Not really. The build seems to have some sort of misconfiguration and no logs are created (and hence can't be accessed via kubectl).
Running the server on the command line just gets us reams of messages and a lot of false leads (too many nameservers, iptables --random-fully not supported, etc ).
I've tried a few variants today, but can't get any really simple logs out. I'll do more tomorrow.
I've been able to confirm that my packagegroup/image type runs docker, containerd, and runc launched containers just fine, so this is just some quirk/idiocy of k*s that is causing the problem.
Bruce
BR,
/Joakim
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIIMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Hi Bruce,
Thanks for your quick reply.
Yesterday, I pulled meta-virtualization master-next branch to my local server and trying to build k3s with qemux86-64 image. In master-next branch, I found you had created DISTRO_FEATURE check for k3s and some package groups for k3s. E.g. packagegroup-k3s-node, packagegroup-k3s-host.
I configured my local.conf to install k3s with following configurations:
DISTRO_FEATURES_append = " k3s virtualization"IMAGE_INSTALL_append = " packagegroup-k3s-node packagegroup-k3s-host cgroup-lite kernel-modules"
However, when I use:
bitbake core-image-minimal
to compile a minimal image to test k3s. I encountered the following error message:
gcc: error: unrecognized command line option ‘-fmacro-prefix-map=/path/to/build/tmp/work/core2-64-poky-linux/libvirt/6.3.0-r0=/usr/src/debug/libvirt/6.3.0-r0’
| error: command '/home/lance/repos/build/tmp/hosttools/gcc' failed with exit code 1 | WARNING: exit code 1 from a shell command. | ERROR: Execution of '/path/to/build/tmp/work/core2-64-poky-linux/libvirt/6.3.0-r0/temp/run.do_compile.162048' failed with exit code 1:
It seems the error was related to libvirt. I two three questions here. If you need more log details, please let me know.
1. Do I mis-config anything so that the error above appeared? Could you please share your configuration when you compile the image with k3s?
2. Do we need libvirt when we install k3s? From my own experience, libvirt is for virtualization. When I run k3s on bare-metal, I don't think I install a libvirt component. And before, I ran a k3s successfully without libvirt on aarch64 Yocto Linux.
Once I successfully built the qemux86-64 image, I am willing to spare some effort to reproduce the bugs you mentioned in the previous email. Would you mind sharing your config to me?
Best Regards,
Lance
Quoted 128 lines
-----Original Message-----
From: Bruce Ashfield <[email protected]>
Sent: Friday, December 11, 2020 9:44 PM
To: Lance Yang <[email protected]>
Cc: Joakim Roubert <[email protected]>; meta-virtualization <meta- [email protected]>; nd <[email protected]>; Michael Zhao <[email protected]>
Subject: Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3Quoted 6 lines · Lance Yang
Hi Bruce and Joakim,
I recently saw a new branch master-next has been created in meta-
virtualization. It mainly related to k3s and I saw k3s had been bumped to v1.19. I would like to know if the k3s patch will be merged into master branch soon. I remembered you mentioned k3s v1.18 has some agent registration issue and some had crash loop issues.
Quoted 4 lines
What is the current status of the k3s patch? Is there any further issues?
It is just in master-next, to show that some cleanups have been done, and the supporting component version bumps that I've done, but no, it still doesn't work under qemux86-64, so it can't actually merge to master.
The node becomes ready, but all the core containers go into a crashbackoff loop, kubectl logs doesn't work, and we still can't register a client with it (likely because of symptom #1).
I'm also chasing down an iptables issue (that I had previously fixed), but should have that sorted shortly.
Bruce
Quoted 22 lines
Best Regards,
Lance
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Thursday, November 19, 2020 4:39 AM
To: Joakim Roubert <[email protected]>
Cc: meta-virtualization <[email protected]>
Subject: Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3On Wed, Nov 18, 2020 at 1:27 PM Joakim Roubert
Quoted 4 lines · <[email protected]> wrote:
Quoted 2 lines · Bruce Ashfield
root@qemux86-64:~# kubectl get pods -n kube-system NAME READYSTATUS
Quoted 7 lines
Quoted 6 lines
RESTARTS AGE
local-path-provisioner-7ff9579c6-bd947 0/1 CrashLoopBackOff 822m coredns-66c464876b-76bdl 0/1 CrashLoopBackOff 822m helm-install-traefik-mvt5k 0/1 CrashLoopBackOff 822m metrics-server-7b4f8b595-ck5kz 0/1 CrashLoopBackOff 822m root@qemux86-64:~#Quoted 7 lines
From my own experience with CrashLoopBackOff situations is that there
may be serveral different things causing that. Can you trace down what is causing the CrashLoopBackOff here?Not really. The build seems to have some sort of misconfiguration and no
logs are created (and hence can't be accessed via kubectl).
Quoted 4 lines
Running the server on the command line just gets us reams of messages
and a lot of false leads (too many nameservers, iptables --random-fully not supported, etc ).
Quoted 4 lines
I've tried a few variants today, but can't get any really simple logs out. I'll do
more tomorrow.
Quoted 4 lines
I've been able to confirm that my packagegroup/image type runs docker,
containerd, and runc launched containers just fine, so this is just some quirk/idiocy of k*s that is causing the problem.
Quoted 16 lines
Bruce
BR,
/Joakim
Quoted 7 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIIMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 4 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 18 lines · Lance Yang
Hi Bruce,
Thanks for your quick reply.
Yesterday, I pulled meta-virtualization master-next branch to my local server and trying to build k3s with qemux86-64 image. In master-next branch, I found you had created
DISTRO_FEATUREcheck for k3s and some package groups for k3s. E.g. packagegroup-k3s-node, packagegroup-k3s-host.I configured my local.conf to install k3s with following configurations:
DISTRO_FEATURES_append= " k3s virtualization"IMAGE_INSTALL_append= " packagegroup-k3s-node packagegroup-k3s-host cgroup-lite kernel-modules"However, when I use:
bitbake core-image-minimal
to compile a minimal image to test k3s. I encountered the following error message:
Quoted 8 lines
gcc: error: unrecognized command line option ‘-fmacro-prefix-map=/path/to/build/tmp/work/core2-64-poky-linux/libvirt/6.3.0-r0=/usr/src/debug/libvirt/6.3.0-r0’
| error: command '/home/lance/repos/build/tmp/hosttools/gcc' failed with exit code 1 | WARNING: exit code 1 from a shell command. | ERROR: Execution of '/path/to/build/tmp/work/core2-64-poky-linux/libvirt/6.3.0-r0/temp/run.do_compile.162048' failed with exit code 1:It seems the error was related to libvirt. I two three questions here. If you need more log details, please let me know.
1. Do I mis-config anything so that the error above appeared? Could you please share your configuration when you compile the image with k3s?
The packagegroups are still a work in progress, and I actually have some changes to them that aren't on that branch yet. In particular docker has been removed as a dependency and libvirt has been shuffled to an optional packagegroup.
That being said, I just built libvirt on two different machines on an up to date master branch for all layers, and didn't get that same error. So
You don't need the packagegroups for testing, but once I have a working configuration, it will be split and captured into package lists for re-use across many of the different configurations.
So I'd suggest dropping it and ignoring that failure for now (since) I can't reproduce it.
Quoted 4 lines
2. Do we need libvirt when we install k3s? From my own experience, libvirt is for virtualization. When I run k3s on bare-metal, I don't think I install a libvirt component. And before, I ran a k3s successfully without libvirt on aarch64 Yocto Linux.
Once I successfully built the qemux86-64 image, I am willing to spare some effort to reproduce the bugs you mentioned in the previous email. Would you mind sharing your config to me?
I've actually tracked this down to something in our build of the k3s binary. I can bring up a node by substituting just the k3s all-in-one executable. There have been issues with the go compiler reported in master, so I'm doing a full rebuild now, in case the linker is causing the problem. I've looked all through the build instructions from the k3s repository, and nothing obvious is different between the two (except possibly the compiler) .. I'm building the exact same git hash of the one that works.
For now, I'd suggest waiting a couple more days, since I may have it solved shortly.
Bruce
Quoted 131 lines
Best Regards,
LanceQuoted 128 lines
-----Original Message-----
From: Bruce Ashfield <[email protected]>
Sent: Friday, December 11, 2020 9:44 PM
To: Lance Yang <[email protected]>
Cc: Joakim Roubert <[email protected]>; meta-virtualization <meta- [email protected]>; nd <[email protected]>; Michael Zhao <[email protected]>
Subject: Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3Quoted 6 lines · Lance Yang
Hi Bruce and Joakim,
I recently saw a new branch master-next has been created in meta-
virtualization. It mainly related to k3s and I saw k3s had been bumped to v1.19. I would like to know if the k3s patch will be merged into master branch soon. I remembered you mentioned k3s v1.18 has some agent registration issue and some had crash loop issues.
Quoted 4 lines
What is the current status of the k3s patch? Is there any further issues?
It is just in master-next, to show that some cleanups have been done, and the supporting component version bumps that I've done, but no, it still doesn't work under qemux86-64, so it can't actually merge to master.
The node becomes ready, but all the core containers go into a crashbackoff loop, kubectl logs doesn't work, and we still can't register a client with it (likely because of symptom #1).
I'm also chasing down an iptables issue (that I had previously fixed), but should have that sorted shortly.
Bruce
Quoted 22 lines
Best Regards,
Lance
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Thursday, November 19, 2020 4:39 AM
To: Joakim Roubert <[email protected]>
Cc: meta-virtualization <[email protected]>
Subject: Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3On Wed, Nov 18, 2020 at 1:27 PM Joakim Roubert
Quoted 4 lines · <[email protected]> wrote:
Quoted 2 lines · Bruce Ashfield
root@qemux86-64:~# kubectl get pods -n kube-system NAME READYSTATUS
Quoted 7 lines
Quoted 6 lines
RESTARTS AGE
local-path-provisioner-7ff9579c6-bd947 0/1 CrashLoopBackOff 822m coredns-66c464876b-76bdl 0/1 CrashLoopBackOff 822m helm-install-traefik-mvt5k 0/1 CrashLoopBackOff 822m metrics-server-7b4f8b595-ck5kz 0/1 CrashLoopBackOff 822m root@qemux86-64:~#Quoted 7 lines
From my own experience with CrashLoopBackOff situations is that there
may be serveral different things causing that. Can you trace down what is causing the CrashLoopBackOff here?Not really. The build seems to have some sort of misconfiguration and no
logs are created (and hence can't be accessed via kubectl).
Quoted 4 lines
Running the server on the command line just gets us reams of messages
and a lot of false leads (too many nameservers, iptables --random-fully not supported, etc ).
Quoted 4 lines
I've tried a few variants today, but can't get any really simple logs out. I'll do
more tomorrow.
Quoted 4 lines
I've been able to confirm that my packagegroup/image type runs docker,
containerd, and runc launched containers just fine, so this is just some quirk/idiocy of k*s that is causing the problem.
Quoted 16 lines
Bruce
BR,
/Joakim
Quoted 7 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIIMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 5 lines
Quoted 4 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIIMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 3 lines · Bruce Ashfield
For now, I'd suggest waiting a couple more days, since I may have it solved shortly.
If there continues to be strange problems, perhaps the Rancher guys would be interested in helping out? They are both skilled and friendly, and given Rancher's focused agenda on being _the_ Kubernetes provider for edge with k3s, I guess having k3s in meta-virtualization would be attractive to them.
BR,
/Joakim
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 11 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield
For now, I'd suggest waiting a couple more days, since I may have it solved shortly.
If there continues to be strange problems, perhaps the Rancher guys would be interested in helping out? They are both skilled and friendly, and given Rancher's focused agenda on being _the_ Kubernetes provider for edge with k3s, I guess having k3s in meta-virtualization would be attractive to them.
I have reached out to them, but had to put down my debug on this for a few days as I prepared 5.10 as the new reference kernel for oe-core.
I'm almost through that, and will cycle back to this shortly.
The bottom line is that only the rancher build k3s binary works with our constructed rootfs (in my tests).
If I use our built binary, or one that I build outside of yocto (natively, on my build server), we end up in crashloops on all the core containers. The binaries are built from the identical hash (for yocto) and using the official build container images / process on my workstation .. same result. So that rules out something in the kernel (and its config), the go version or something in our rootfs being missing. But as soon as you fetch the rancher binary, restart the services .. the containers come up just fine.
And that unfortunately means that it is difficult to get help from the project, since they don't have the right understanding of the runtime environment (but I'm trying!).
I'm diving deeper to try and sort it out.
Bruce
Quoted 5 lines
BR,
/Joakim
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 1 line · Bruce Ashfield
So that rules out something in the kernel (and its config), the go version or something in our rootfs being missing.
That surely narrows things down.
Quoted 2 lines
But as soon as you fetch the rancher binary, restart the services .. the containers come up just fine.
Is this the same thing that happened for the k3s v1.18.9 build too?
Quoted 1 line
I'm diving deeper to try and sort it out.
👍
BR,
/Joakim
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 5 lines · Joakim Roubert
Quoted 2 lines · Bruce Ashfield
So that rules out something in the kernel (and its config), the go version or something in our rootfs being missing.
Quoted 7 lines
That surely narrows things down.
Quoted 2 lines
But as soon as you fetch the rancher binary, restart the services .. the containers come up just fine.
Is this the same thing that happened for the k3s v1.18.9 build too?
Yep. It is consistent with all versions that I've built, including the pre-release master branch as well. I was hoping for some "uprev magic" to fix it, but that wasn't the case :)
I'm back from the holidays now, and will be back into this shortly.
Bruce
Quoted 7 lines
Quoted 1 line
I'm diving deeper to try and sort it out.
BR,
/Joakim
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 12 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 10 lines · Joakim Roubert
Quoted 2 lines · Bruce Ashfield
So that rules out something in the kernel (and its config), the go version or something in our rootfs being missing.
That surely narrows things down.
Quoted 2 lines
But as soon as you fetch the rancher binary, restart the services .. the containers come up just fine.
Quoted 9 lines
Quoted 2 lines
Is this the same thing that happened for the k3s v1.18.9 build too?
Yep. It is consistent with all versions that I've built, including the pre-release master branch as well. I was hoping for some "uprev magic" to fix it, but that wasn't the case :)
I'm back from the holidays now, and will be back into this shortly.
I do have one more thing I'd like to try (I didn't get much debug done yesterday), do you happen to have a k3s binary that works in your environment ?
I'd like to toss it into my image and see if it behaves differently from my builds and the rancher builds of k3s.
Bruce
Quoted 19 lines
Bruce
Quoted 7 lines
Quoted 1 line
I'm diving deeper to try and sort it out.
BR,
/Joakim
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3
Quoted 2 lines · Joakim Roubert
The current stable release installed when running the regular k3s install script is now v1.19.3+k3s3.
I launched a multi-arch cluster with this v1.19.3+k3s3, deployed Linkerd 2.9.0 on it and then meshed a small test application sending traffic back and forth between a client and a server, and it all seems to work quite nicely.
BR,
/Joakim
Signature · 7 lines
Joakim Roubert Senior Engineer Axis Communications AB Emdalavägen 14, SE-223 69 Lund, Sweden Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48 Fax: +46 46 13 61 30, www.axis.com
Quoted 3 lines · Lance Yang
Problem 1: conflicting requests
- nothing provides kernel-module-ip-set needed by ipset-6.38-r0
Do you have
CONFIG_IP_SET=m
in your kernel config?
BR,
/Joakim
Hi Joakim,
Please see the comments inline.
Quoted 7 lines
-----Original Message-----
From: Joakim Roubert <[email protected]>
Sent: Thursday, November 12, 2020 9:43 PM
To: Lance Yang <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe
Quoted 8 lines
Quoted 3 lines · Lance Yang
Problem 1: conflicting requests
- nothing provides kernel-module-ip-set needed by ipset-6.38-r0Do you have
CONFIG_IP_SET=m
[ Uh! That works! I thought I set it but when I double checked the menuconfig, I found I forgot to set it.
As far as I know, ipset is a tool to simplify iptables configurations. (e.g. iptables -t nat -A PREROUTING -m set --match-set podnet dst -j xxx) which decreases the number of iptables rules.
Because of my limited knowledge of k3s/k8s, I have a questions about ipset.
Is it a must installed component for k3s/k8s?
]
Quoted 6 lines
in your kernel config?
BR,
/Joakim
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 20 lines · Lance Yang
Hi Joakim,
Please see the comments inline.
Quoted 7 lines
-----Original Message-----
From: Joakim Roubert <[email protected]>
Sent: Thursday, November 12, 2020 9:43 PM
To: Lance Yang <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 8 lines
Quoted 3 lines · Lance Yang
Problem 1: conflicting requests
- nothing provides kernel-module-ip-set needed by ipset-6.38-r0Do you have
CONFIG_IP_SET=m
Quoted 7 lines
[ Uh! That works! I thought I set it but when I double checked the menuconfig, I found I forgot to set it.
As far as I know, ipset is a tool to simplify iptables configurations. (e.g. iptables -t nat -A PREROUTING -m set --match-set podnet dst -j xxx) which decreases the number of iptables rules.
Because of my limited knowledge of k3s/k8s, I have a questions about ipset.
Is it a must installed component for k3s/k8s?
Check my reviews on the mailing list, you'll see where it is discussed.
Bruce
Quoted 8 lines
]
Quoted 6 lines
in your kernel config?
BR,
/Joakim
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 3 lines · Bruce Ashfield
But if anyone else does have an all in one working and has some tips, feel free to share :D
When it comes to mount issues as were mentioned here earlier, I have seen those when I have has /var/lib/rancher/k3s mounted on overlayfs file systems (which does not work when the containers are unpacking their layers). Mounting another file system (e.g. ext4 or someting) on /var/lib/rancher/k3s has mitigated such errors.
BR,
/Joakim
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 40 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 26 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 4 lines · Lance Yang
Hi Bruce and Joakim,
Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.
The branch will be more functional shortly, I have quite a few changes to factor things for
k8s and generally more usable :Dmodified: classes/cni_networking.bbclass modified: conf/layer.conf modified: recipes-containers/containerd/containerd-docker_git.bb modified: recipes-containers/containerd/containerd-opencontainers_git.bb modified: recipes-containers/k3s/README.md modified: recipes-containers/k3s/k3s_git.bb modified: recipes-kernel/linux/linux-yocto/kubernetes.cfg modified: recipes-networking/cni/cni_git.bbcontainer-deploy.txt
recipes-core/packagegroups/Quoted 3 lines
My Image: Linux qemuarm64 by yocto.
Quoted 9 lines
Quoted 2 lines
The master node can be ready after I started the k3s server. However, the pods in kube-system (which are essential components for k3s) cannot turn to ready state on qemuarm64.
That's interesting, since in my configuration, the master never comes ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 NotReady master 15h v1.18.9-k3s1
Quoted 8 lines
Hah.
I finally got the node to show up as ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 Ready master 112s v1.18.9-k3s1
Lance,
What image type were you building ? I'm pulling in dependencies to packagegroups and the recipes themselves.
I'm not seeing the mount issue on my master/server node:
root@qemux86-64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-h7lxk 1/1 Running 0 3m32s metrics-server-7566d596c8-mwntr 1/1 Running 0 3m32s helm-install-traefik-229v7 0/1 Completed 0 3m32s coredns-7944c66d8d-9rfj7 1/1 Running 0 3m32s svclb-traefik-pb5j4 2/2 Running 0 2m29s traefik-758cd5fc85-lxpr8 1/1 Running 0 2m29s
I'm going back to all-in-one node debugging, but can look into the mount issue more later.
Bruce
Quoted 146 lines
I'm attempting to build an all-in-one node, and that is likely causing me some issues.
I'm revisiting those potential conflicts now.
But if anyone else does have an all in one working and has some tips, feel free to share :D
Bruce
Quoted 116 lines
I've sorted out more of the dependencies, and have packagegroups to make them easier
now.Hopefully, I can figure out what is now missing and keeping my master from moving into
ready today.Bruce
Quoted 106 lines
After the master node itself turned to ready state, I check the pods with kubectl:
kubectl get nodes
NAME STATUS ROLES AGE VERSION qemuarm64 Ready master 11m v1.18.9-k3s1 root@qemuarm64:~# ls root@qemuarm64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-xxvbl 0/1 ContainerCreating 0 12m coredns-7944c66d8d-tlrm9 0/1 ContainerCreating 0 12m metrics-server-7566d596c8-svkff 0/1 ContainerCreating 0 12m helm-install-traefik-s8p5g 0/1 ContainerCreating 0 12mThen I describe the pods with:
Events:
Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 16m default-scheduler Successfully assigned kube-system/coredns-7944c66d8d-tlrm9 to qemuarm64 Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to attach or mount volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token-b7nlh config-volume]: timed out waiting for the condition Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach or mount volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume coredns-token-b7nlh]: timed out waiting for the condition Warning FailedMount 11s (x16 over 16m) kubelet MountVolume.SetUp failed for volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable file not found in $PATHI found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and mount binary on my qemuarm64 image:
root@qemuarm64:~# echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin root@qemuarm64:~# which mount /bin/mountWhen I type mount command, it worked fine:
/dev/rooton / type ext4 (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
debugfs on/sys/kernel/debugtype debugfs (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
...
... (skipped the verbose output)I would like to know whether you have met this "mount" issue ever?
Best Regards,
LanceQuoted 60 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield
Ha!!!!
This applies.
Quoted 7 lines
Wonderful, thank you! I guess this is what is called "five times lucky"...
Quoted 3 lines
I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.
Excellent!
I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/log/?h=k3s-wipThat includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.
I did have a couple of questions on the k3s packaging itself, I was getting the following error:
ERROR: k3s-v1.18.9+k3s1-dirty-r0
do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3sSo I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?
Bruce
Quoted 8 lines
BR,
/Joakim
--
Joakim Roubert
Senior EngineerQuoted 5 lines
Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIQuoted 11 lines
Quoted 1 line
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Hi Bruce,
Thanks for your reply. It took me a long time to address the $PATH issue.
I found the cause after I cleaned my environment. it is because I set my own $PATH before k3s startup which did not include "/bin" and some components in k3s will use "os.Getenv("PATH") to search the related tool.
Since some OS may not have systemd and the k3s-clean script seems not to delete all interfaces created by cni and umount all mount points related to k3s. I wrote a script for this situation and I will send it through a separate email based on your previous scripts.
Best Regards,
Lance
Quoted 241 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Tuesday, November 10, 2020 9:35 PM
To: Bruce Ashfield <[email protected]>
Cc: Lance Yang <[email protected]>; Joakim Roubert <[email protected]>; meta- [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 27 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 23 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 4 lines · Lance Yang
Hi Bruce and Joakim,
Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.
The branch will be more functional shortly, I have quite a few changes to factor things for k8s and generally more usable :D
modified: classes/cni_networking.bbclass modified: conf/layer.conf modified: recipes-containers/containerd/containerd-docker_git.bb modified: recipes-containers/containerd/containerd-opencontainers_git.bb modified: recipes-containers/k3s/README.md modified: recipes-containers/k3s/k3s_git.bb modified: recipes-kernel/linux/linux-yocto/kubernetes.cfg modified: recipes-networking/cni/cni_git.bbcontainer-deploy.txt
recipes-core/packagegroups/Quoted 1 line
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
My Image: Linux qemuarm64 by yocto.
The master node can be ready after I started the k3s server. However, the pods in kube-
system (which are essential components for k3s) cannot turn to ready state on qemuarm64.
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
That's interesting, since in my configuration, the master never comes ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 NotReady master 15h v1.18.9-k3s1Quoted 8 lines
Hah.
I finally got the node to show up as ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 Ready master 112s v1.18.9-k3s1Lance,
What image type were you building ? I'm pulling in dependencies to packagegroups and the recipes themselves.
I'm not seeing the mount issue on my master/server node:
root@qemux86-64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-h7lxk 1/1 Running 0 3m32s metrics-server-7566d596c8-mwntr 1/1 Running 0 3m32s helm-install-traefik-229v7 0/1 Completed 0 3m32s coredns-7944c66d8d-9rfj7 1/1 Running 0 3m32s svclb-traefik-pb5j4 2/2 Running 0 2m29s traefik-758cd5fc85-lxpr8 1/1 Running 0 2m29sI'm going back to all-in-one node debugging, but can look into the mount issue more later.
Bruce
Quoted 37 lines
I'm attempting to build an all-in-one node, and that is likely causing me some issues.
I'm revisiting those potential conflicts now.
But if anyone else does have an all in one working and has some tips, feel free to share :D
Bruce
Quoted 27 lines
I've sorted out more of the dependencies, and have packagegroups to make them easier now.
Hopefully, I can figure out what is now missing and keeping my master from moving into ready today.
Bruce
Quoted 19 lines
After the master node itself turned to ready state, I check the pods with kubectl:
kubectl get nodes
NAME STATUS ROLES AGE VERSION qemuarm64 Ready master 11m v1.18.9-k3s1 root@qemuarm64:~# ls root@qemuarm64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-xxvbl 0/1 ContainerCreating 0 12m coredns-7944c66d8d-tlrm9 0/1 ContainerCreating 0 12m metrics-server-7566d596c8-svkff 0/1 ContainerCreating 0 12m helm-install-traefik-s8p5g 0/1 ContainerCreating 0 12mThen I describe the pods with:
Events:
Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 16m default-scheduler Successfully assigned kube-system/coredns-7944c66d8d-tlrm9 to qemuarm64
Quoted 1 line
Quoted 1 line
Quoted 1 line
Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to attach or mount
volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token- b7nlh config-volume]: timed out waiting for the condition
Quoted 1 line
Quoted 1 line
Quoted 1 line
Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach or mount
volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume coredns-token-b7nlh]: timed out waiting for the condition
Quoted 1 line
Quoted 1 line
Quoted 1 line
Warning FailedMount 11s (x16 over 16m) kubelet MountVolume.SetUp failed
for volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable file not found in $PATH
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and
mount binary on my qemuarm64 image:
Quoted 81 lines
Quoted 81 lines
Quoted 81 lines
root@qemuarm64:~# echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin root@qemuarm64:~# which mount /bin/mountWhen I type mount command, it worked fine:
/dev/rooton / type ext4 (rw,relatime) devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) debugfs on/sys/kernel/debugtype debugfs (rw,relatime) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
...
... (skipped the verbose output)I would like to know whether you have met this "mount" issue ever?
Best Regards,
LanceQuoted 60 lines
-----Original Message-----
From: [email protected] <[email protected]>
On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield
Ha!!!!
This applies.
Quoted 7 lines
Wonderful, thank you! I guess this is what is called "five times lucky"...
Quoted 3 lines
I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.
Excellent!
I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/l
og/?h=k3s-wipThat includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.
I did have a couple of questions on the k3s packaging itself, I was getting the following
error:ERROR: k3s-v1.18.9+k3s1-dirty-r0
do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3sSo I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?
Bruce
Quoted 13 lines
BR,
/Joakim
--
Joakim Roubert
Senior EngineerAxis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.comQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIIMPORTANT NOTICE: The contents of this email and any attachments are confidential and
may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 12 lines
Quoted 10 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIQuoted 7 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
Quoted 3 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 9 lines · Lance Yang
Hi Bruce,
Thanks for your reply. It took me a long time to address the $PATH issue.
I found the cause after I cleaned my environment. it is because I set my own $PATH before k3s startup which did not include "/bin" and some components in k3s will use "
os.Getenv("PATH")to search the related tool.Since some OS may not have systemd and the k3s-clean script seems not to delete all interfaces created by cni and umount all mount points related to k3s. I wrote a script for this situation and I will send it through a separate email based on your previous scripts.
Sounds good.
You'll notice that I pushed an update to the k3s-wip branch, things are a bit closer to working there now.
I am interested to hear if flannel crashes if you (or anyone else) install (and use) the k3s-agent on the same node as the k3s-server.
As soon as you run any k3s-agent command (with the proper token and local host as the server), the node moves into NotReady with what looks like a CNI error.
I fetched and ran the binaries directly from rancher, and they showed the same behaviour as the meta-virt ones, so it isn't something fundamental with the integration.
Bruce
Quoted 247 lines
Best Regards,
LanceQuoted 241 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Tuesday, November 10, 2020 9:35 PM
To: Bruce Ashfield <[email protected]>
Cc: Lance Yang <[email protected]>; Joakim Roubert <[email protected]>; meta- [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 27 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 23 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 4 lines · Lance Yang
Hi Bruce and Joakim,
Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.
The branch will be more functional shortly, I have quite a few changes to factor things for k8s and generally more usable :D
modified: classes/cni_networking.bbclass modified: conf/layer.conf modified: recipes-containers/containerd/containerd-docker_git.bb modified: recipes-containers/containerd/containerd-opencontainers_git.bb modified: recipes-containers/k3s/README.md modified: recipes-containers/k3s/k3s_git.bb modified: recipes-kernel/linux/linux-yocto/kubernetes.cfg modified: recipes-networking/cni/cni_git.bbcontainer-deploy.txt
recipes-core/packagegroups/Quoted 1 line
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
My Image: Linux qemuarm64 by yocto.
The master node can be ready after I started the k3s server. However, the pods in kube-
system (which are essential components for k3s) cannot turn to ready state on qemuarm64.
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
That's interesting, since in my configuration, the master never comes ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 NotReady master 15h v1.18.9-k3s1Quoted 8 lines
Hah.
I finally got the node to show up as ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 Ready master 112s v1.18.9-k3s1Lance,
What image type were you building ? I'm pulling in dependencies to packagegroups and the recipes themselves.
I'm not seeing the mount issue on my master/server node:
root@qemux86-64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-h7lxk 1/1 Running 0 3m32s metrics-server-7566d596c8-mwntr 1/1 Running 0 3m32s helm-install-traefik-229v7 0/1 Completed 0 3m32s coredns-7944c66d8d-9rfj7 1/1 Running 0 3m32s svclb-traefik-pb5j4 2/2 Running 0 2m29s traefik-758cd5fc85-lxpr8 1/1 Running 0 2m29sI'm going back to all-in-one node debugging, but can look into the mount issue more later.
Bruce
Quoted 37 lines
I'm attempting to build an all-in-one node, and that is likely causing me some issues.
I'm revisiting those potential conflicts now.
But if anyone else does have an all in one working and has some tips, feel free to share :D
Bruce
Quoted 27 lines
I've sorted out more of the dependencies, and have packagegroups to make them easier now.
Hopefully, I can figure out what is now missing and keeping my master from moving into ready today.
Bruce
Quoted 19 lines
After the master node itself turned to ready state, I check the pods with kubectl:
kubectl get nodes
NAME STATUS ROLES AGE VERSION qemuarm64 Ready master 11m v1.18.9-k3s1 root@qemuarm64:~# ls root@qemuarm64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-xxvbl 0/1 ContainerCreating 0 12m coredns-7944c66d8d-tlrm9 0/1 ContainerCreating 0 12m metrics-server-7566d596c8-svkff 0/1 ContainerCreating 0 12m helm-install-traefik-s8p5g 0/1 ContainerCreating 0 12mThen I describe the pods with:
Events:
Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 16m default-scheduler Successfully assigned kube-system/coredns-7944c66d8d-tlrm9 to qemuarm64
Quoted 1 line
Quoted 1 line
Quoted 1 line
Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to attach or mount
volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token- b7nlh config-volume]: timed out waiting for the condition
Quoted 1 line
Quoted 1 line
Quoted 1 line
Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach or mount
volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume coredns-token-b7nlh]: timed out waiting for the condition
Quoted 1 line
Quoted 1 line
Quoted 1 line
Warning FailedMount 11s (x16 over 16m) kubelet MountVolume.SetUp failed
for volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable file not found in $PATH
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and
mount binary on my qemuarm64 image:
Quoted 81 lines
Quoted 81 lines
Quoted 81 lines
root@qemuarm64:~# echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin root@qemuarm64:~# which mount /bin/mountWhen I type mount command, it worked fine:
/dev/rooton / type ext4 (rw,relatime) devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) debugfs on/sys/kernel/debugtype debugfs (rw,relatime) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
...
... (skipped the verbose output)I would like to know whether you have met this "mount" issue ever?
Best Regards,
LanceQuoted 60 lines
-----Original Message-----
From: [email protected] <[email protected]>
On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield
Ha!!!!
This applies.
Quoted 7 lines
Wonderful, thank you! I guess this is what is called "five times lucky"...
Quoted 3 lines
I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.
Excellent!
I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/l
og/?h=k3s-wipThat includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.
I did have a couple of questions on the k3s packaging itself, I was getting the following
error:ERROR: k3s-v1.18.9+k3s1-dirty-r0
do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3sSo I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?
Bruce
Quoted 13 lines
BR,
/Joakim
--
Joakim Roubert
Senior EngineerAxis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.comQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIIMPORTANT NOTICE: The contents of this email and any attachments are confidential and
may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 12 lines
Quoted 10 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIQuoted 7 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIQuoted 3 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIIMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Hi Bruce,
On my side, the flannel did not crash.
I started k3s server without agent and confirmed the node is ready. Then I copied the node_token and then ran k3s agent with that token and specified the server url. Then I can see node registered successfully.
The pods traefik and coredns will be not ready after I started k3s agent in a short while:
kube-system traefik-758cd5fc85-8tqhf 0/1 Running 0 24m kube-system coredns-7944c66d8d-584jt 0/1 Running 0 21m
but they soon were back to ready state:
kube-system coredns-7944c66d8d-584jt 1/1 Running 0 21m kube-system traefik-758cd5fc85-8tqhf 1/1 Running 0 24m
I did not see any errors related flannel crash. Hope this helped a little bit.
Best Regards,
Lance
Quoted 304 lines
-----Original Message-----
From: Bruce Ashfield <[email protected]>
Sent: Wednesday, November 11, 2020 9:40 PM
To: Lance Yang <[email protected]>
Cc: Joakim Roubert <[email protected]>; [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 6 lines · Lance Yang
Hi Bruce,
Thanks for your reply. It took me a long time to address the $PATH issue.
I found the cause after I cleaned my environment. it is because I set my own $PATH before k3s
startup which did not include "/bin" and some components in k3s will use "
os.Getenv("PATH")to search the related tool.Quoted 2 lines
Since some OS may not have systemd and the k3s-clean script seems not to delete all
interfaces created by cni and umount all mount points related to k3s. I wrote a script for this situation and I will send it through a separate email based on your previous scripts.
Quoted 1 line
Sounds good.
You'll notice that I pushed an update to the k3s-wip branch, things are a bit closer to working there now.
I am interested to hear if flannel crashes if you (or anyone else) install (and use) the k3s-agent on the same node as the k3s-server.
As soon as you run any k3s-agent command (with the proper token and local host as the server), the node moves into NotReady with what looks like a CNI error.
I fetched and ran the binaries directly from rancher, and they showed the same behaviour as the meta-virt ones, so it isn't something fundamental with the integration.
Bruce
Quoted 124 lines
Best Regards,
LanceQuoted 122 lines
-----Original Message-----
From: [email protected] <[email protected]>
On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Tuesday, November 10, 2020 9:35 PM
To: Bruce Ashfield <[email protected]>
Cc: Lance Yang <[email protected]>; Joakim Roubert <[email protected]>; meta- [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 27 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 23 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 4 lines · Lance Yang
Hi Bruce and Joakim,
Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.
The branch will be more functional shortly, I have quite a few changes to factor things for k8s and generally more usable :D
modified: classes/cni_networking.bbclass modified: conf/layer.conf modified: recipes-containers/containerd/containerd-docker_git.bb modified: recipes-containers/containerd/containerd-opencontainers_git.bb modified: recipes-containers/k3s/README.md modified: recipes-containers/k3s/k3s_git.bb modified: recipes-kernel/linux/linux-yocto/kubernetes.cfg modified: recipes-networking/cni/cni_git.bbcontainer-deploy.txt
recipes-core/packagegroups/Quoted 1 line
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
My Image: Linux qemuarm64 by yocto.
The master node can be ready after I started the k3s server. However, the pods in kube-
system (which are essential components for k3s) cannot turn to ready state on qemuarm64.
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
That's interesting, since in my configuration, the master never comes ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 NotReady master 15h v1.18.9-k3s1Quoted 8 lines
Hah.
I finally got the node to show up as ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 Ready master 112s v1.18.9-k3s1Lance,
What image type were you building ? I'm pulling in dependencies to packagegroups and the recipes themselves.
I'm not seeing the mount issue on my master/server node:
root@qemux86-64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-h7lxk 1/1 Running 0 3m32s metrics-server-7566d596c8-mwntr 1/1 Running 0 3m32s helm-install-traefik-229v7 0/1 Completed 0 3m32s coredns-7944c66d8d-9rfj7 1/1 Running 0 3m32s svclb-traefik-pb5j4 2/2 Running 0 2m29s traefik-758cd5fc85-lxpr8 1/1 Running 0 2m29sI'm going back to all-in-one node debugging, but can look into the mount issue more later.
Bruce
Quoted 37 lines
I'm attempting to build an all-in-one node, and that is likely causing me some issues.
I'm revisiting those potential conflicts now.
But if anyone else does have an all in one working and has some tips, feel free to share :D
Bruce
Quoted 27 lines
I've sorted out more of the dependencies, and have packagegroups to make them easier now.
Hopefully, I can figure out what is now missing and keeping my master from moving into ready today.
Bruce
Quoted 19 lines
After the master node itself turned to ready state, I check the pods with kubectl:
kubectl get nodes
NAME STATUS ROLES AGE VERSION qemuarm64 Ready master 11m v1.18.9-k3s1 root@qemuarm64:~# ls root@qemuarm64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-xxvbl 0/1 ContainerCreating 0 12m coredns-7944c66d8d-tlrm9 0/1 ContainerCreating 0 12m metrics-server-7566d596c8-svkff 0/1 ContainerCreating 0 12m helm-install-traefik-s8p5g 0/1 ContainerCreating 0 12mThen I describe the pods with:
Events:
Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 16m default-scheduler Successfully assigned kube-system/coredns-7944c66d8d-tlrm9 to qemuarm64
Quoted 1 line
Quoted 1 line
Quoted 1 line
Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to attach or
mount
Quoted 8 lines
Quoted 8 lines
volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token- b7nlh config-volume]: timed out waiting for the condition
Quoted 1 line
Quoted 1 line
Quoted 1 line
Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach or mount
volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume
coredns-token-b7nlh]: timed out waiting for the conditionQuoted 1 line
Quoted 1 line
Quoted 1 line
Warning FailedMount 11s (x16 over 16m) kubelet MountVolume.SetUp
failed
Quoted 93 lines
Quoted 93 lines
for volume "coredns-token-b7nlh" : mount failed: exec: "mount":
executable file not found in $PATHQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and
mount binary on my qemuarm64 image:
Quoted 87 lines
Quoted 87 lines
Quoted 87 lines
root@qemuarm64:~# echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin root@qemuarm64:~# which mount /bin/mountWhen I type mount command, it worked fine:
/dev/rooton / type ext4 (rw,relatime) devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) debugfs on/sys/kernel/debugtype debugfs (rw,relatime) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) ...
... (skipped the verbose output)I would like to know whether you have met this "mount" issue ever?
Best Regards,
LanceQuoted 66 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield
Ha!!!!
This applies.
Quoted 7 lines
Wonderful, thank you! I guess this is what is called "five times lucky"...
Quoted 3 lines
I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.
Excellent!
I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualizati
on/l
og/?h=k3s-wipThat includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.
I did have a couple of questions on the k3s packaging itself, I was getting the following
error:ERROR: k3s-v1.18.9+k3s1-dirty-r0
do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3sSo I added them to the FILES of the k3s package itself (so
both k3s-server and k3s-agent will get them), is that the split you were looking for ?Bruce
Quoted 8 lines
BR,
/Joakim
--
Joakim Roubert
Senior EngineerQuoted 5 lines
Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIQuoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and
may also be privileged. If you are not the intended recipient,
please notify the sender immediately and do not disclose the
contents to any other person, use it for any purpose, or store or copy the information in anymedium. Thank you.
Quoted 21 lines
Quoted 21 lines
Quoted 19 lines
Quoted 10 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIQuoted 5 lines
Quoted 4 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIIMPORTANT NOTICE: The contents of this email and any attachments are confidential and may
also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 3 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 6 lines · Lance Yang
Hi Bruce,
On my side, the flannel did not crash.
I started k3s server without agent and confirmed the node is ready. Then I copied the
node_tokenand then ran k3s agent with that token and specified the server url. Then I can see node registered successfully.
When you say 'copied', do you mean copied to a different machine ?
Quoted 11 lines
The pods traefik and coredns will be not ready after I started k3s agent in a short while:
kube-system traefik-758cd5fc85-8tqhf 0/1 Running 0 24m kube-system coredns-7944c66d8d-584jt 0/1 Running 0 21mbut they soon were back to ready state:
kube-system coredns-7944c66d8d-584jt 1/1 Running 0 21m kube-system traefik-758cd5fc85-8tqhf 1/1 Running 0 24mI did not see any errors related flannel crash. Hope this helped a little bit.
unfortunately .. no.
I need a single node configuration to work, before I can actually merge the changes, since without it, I have no way to ensure that it continues working.
Cheers,
Bruce
Quoted 310 lines
Best Regards,
LanceQuoted 304 lines
-----Original Message-----
From: Bruce Ashfield <[email protected]>
Sent: Wednesday, November 11, 2020 9:40 PM
To: Lance Yang <[email protected]>
Cc: Joakim Roubert <[email protected]>; [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 6 lines · Lance Yang
Hi Bruce,
Thanks for your reply. It took me a long time to address the $PATH issue.
I found the cause after I cleaned my environment. it is because I set my own $PATH before k3s
startup which did not include "/bin" and some components in k3s will use "
os.Getenv("PATH")to search the related tool.Quoted 2 lines
Since some OS may not have systemd and the k3s-clean script seems not to delete all
interfaces created by cni and umount all mount points related to k3s. I wrote a script for this situation and I will send it through a separate email based on your previous scripts.
Quoted 1 line
Sounds good.
You'll notice that I pushed an update to the k3s-wip branch, things are a bit closer to working there now.
I am interested to hear if flannel crashes if you (or anyone else) install (and use) the k3s-agent on the same node as the k3s-server.
As soon as you run any k3s-agent command (with the proper token and local host as the server), the node moves into NotReady with what looks like a CNI error.
I fetched and ran the binaries directly from rancher, and they showed the same behaviour as the meta-virt ones, so it isn't something fundamental with the integration.
Bruce
Quoted 124 lines
Best Regards,
LanceQuoted 122 lines
-----Original Message-----
From: [email protected] <[email protected]>
On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Tuesday, November 10, 2020 9:35 PM
To: Bruce Ashfield <[email protected]>
Cc: Lance Yang <[email protected]>; Joakim Roubert <[email protected]>; meta- [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 27 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 23 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 4 lines · Lance Yang
Hi Bruce and Joakim,
Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.
The branch will be more functional shortly, I have quite a few changes to factor things for k8s and generally more usable :D
modified: classes/cni_networking.bbclass modified: conf/layer.conf modified: recipes-containers/containerd/containerd-docker_git.bb modified: recipes-containers/containerd/containerd-opencontainers_git.bb modified: recipes-containers/k3s/README.md modified: recipes-containers/k3s/k3s_git.bb modified: recipes-kernel/linux/linux-yocto/kubernetes.cfg modified: recipes-networking/cni/cni_git.bbcontainer-deploy.txt
recipes-core/packagegroups/Quoted 1 line
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
My Image: Linux qemuarm64 by yocto.
The master node can be ready after I started the k3s server. However, the pods in kube-
system (which are essential components for k3s) cannot turn to ready state on qemuarm64.
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
That's interesting, since in my configuration, the master never comes ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 NotReady master 15h v1.18.9-k3s1Quoted 8 lines
Hah.
I finally got the node to show up as ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 Ready master 112s v1.18.9-k3s1Lance,
What image type were you building ? I'm pulling in dependencies to packagegroups and the recipes themselves.
I'm not seeing the mount issue on my master/server node:
root@qemux86-64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-h7lxk 1/1 Running 0 3m32s metrics-server-7566d596c8-mwntr 1/1 Running 0 3m32s helm-install-traefik-229v7 0/1 Completed 0 3m32s coredns-7944c66d8d-9rfj7 1/1 Running 0 3m32s svclb-traefik-pb5j4 2/2 Running 0 2m29s traefik-758cd5fc85-lxpr8 1/1 Running 0 2m29sI'm going back to all-in-one node debugging, but can look into the mount issue more later.
Bruce
Quoted 37 lines
I'm attempting to build an all-in-one node, and that is likely causing me some issues.
I'm revisiting those potential conflicts now.
But if anyone else does have an all in one working and has some tips, feel free to share :D
Bruce
Quoted 27 lines
I've sorted out more of the dependencies, and have packagegroups to make them easier now.
Hopefully, I can figure out what is now missing and keeping my master from moving into ready today.
Bruce
Quoted 19 lines
After the master node itself turned to ready state, I check the pods with kubectl:
kubectl get nodes
NAME STATUS ROLES AGE VERSION qemuarm64 Ready master 11m v1.18.9-k3s1 root@qemuarm64:~# ls root@qemuarm64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-xxvbl 0/1 ContainerCreating 0 12m coredns-7944c66d8d-tlrm9 0/1 ContainerCreating 0 12m metrics-server-7566d596c8-svkff 0/1 ContainerCreating 0 12m helm-install-traefik-s8p5g 0/1 ContainerCreating 0 12mThen I describe the pods with:
Events:
Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 16m default-scheduler Successfully assigned kube-system/coredns-7944c66d8d-tlrm9 to qemuarm64
Quoted 1 line
Quoted 1 line
Quoted 1 line
Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to attach or
mount
Quoted 8 lines
Quoted 8 lines
volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token- b7nlh config-volume]: timed out waiting for the condition
Quoted 1 line
Quoted 1 line
Quoted 1 line
Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach or mount
volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume
coredns-token-b7nlh]: timed out waiting for the conditionQuoted 1 line
Quoted 1 line
Quoted 1 line
Warning FailedMount 11s (x16 over 16m) kubelet MountVolume.SetUp
failed
Quoted 93 lines
Quoted 93 lines
for volume "coredns-token-b7nlh" : mount failed: exec: "mount":
executable file not found in $PATHQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and
mount binary on my qemuarm64 image:
Quoted 87 lines
Quoted 87 lines
Quoted 87 lines
root@qemuarm64:~# echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin root@qemuarm64:~# which mount /bin/mountWhen I type mount command, it worked fine:
/dev/rooton / type ext4 (rw,relatime) devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) debugfs on/sys/kernel/debugtype debugfs (rw,relatime) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) ...
... (skipped the verbose output)I would like to know whether you have met this "mount" issue ever?
Best Regards,
LanceQuoted 66 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield
Ha!!!!
This applies.
Quoted 7 lines
Wonderful, thank you! I guess this is what is called "five times lucky"...
Quoted 3 lines
I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.
Excellent!
I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualizati
on/l
og/?h=k3s-wipThat includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.
I did have a couple of questions on the k3s packaging itself, I was getting the following
error:ERROR: k3s-v1.18.9+k3s1-dirty-r0
do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3sSo I added them to the FILES of the k3s package itself (so
both k3s-server and k3s-agent will get them), is that the split you were looking for ?Bruce
Quoted 8 lines
BR,
/Joakim
--
Joakim Roubert
Senior EngineerQuoted 5 lines
Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIQuoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and
may also be privileged. If you are not the intended recipient,
please notify the sender immediately and do not disclose the
contents to any other person, use it for any purpose, or store or copy the information in anymedium. Thank you.
Quoted 21 lines
Quoted 21 lines
Quoted 19 lines
Quoted 10 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIQuoted 5 lines
Quoted 4 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIIMPORTANT NOTICE: The contents of this email and any attachments are confidential and may
also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 3 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIIMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Hi Bruce,
No, it is not. I just use the command:
k3s agent --token (copied token) --server https://localhost:6443
Best Regards,
Lance
Quoted 383 lines
-----Original Message-----
From: Bruce Ashfield <[email protected]>
Sent: Thursday, November 12, 2020 9:40 PM
To: Lance Yang <[email protected]>
Cc: Joakim Roubert <[email protected]>; [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>; nd <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 6 lines · Lance Yang
Hi Bruce,
On my side, the flannel did not crash.
I started k3s server without agent and confirmed the node is ready. Then I copied the
node_tokenand then ran k3s agent with that token and specified the server url. Then I can see node registered successfully.When you say 'copied', do you mean copied to a different machine ?
Quoted 11 lines
The pods traefik and coredns will be not ready after I started k3s agent in a short while:
kube-system traefik-758cd5fc85-8tqhf 0/1 Running 0 24m kube-system coredns-7944c66d8d-584jt 0/1 Running 0 21mbut they soon were back to ready state:
kube-system coredns-7944c66d8d-584jt 1/1 Running 0 21m kube-system traefik-758cd5fc85-8tqhf 1/1 Running 0 24mI did not see any errors related flannel crash. Hope this helped a little bit.
unfortunately .. no.
I need a single node configuration to work, before I can actually merge the changes, since without it, I have no way to ensure that it continues working.
Cheers,
Bruce
Quoted 26 lines
Best Regards,
LanceQuoted 24 lines
-----Original Message-----
From: Bruce Ashfield <[email protected]>
Sent: Wednesday, November 11, 2020 9:40 PM
To: Lance Yang <[email protected]>
Cc: Joakim Roubert <[email protected]>; [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 7 lines · Lance Yang
Hi Bruce,
Thanks for your reply. It took me a long time to address the $PATH issue.
I found the cause after I cleaned my environment. it is because I set my own $PATH before k3s
startup which did not include "/bin" and some components in k3s will use "
os.Getenv("PATH")to search the related tool.Quoted 3 lines
Since some OS may not have systemd and the k3s-clean script seems not to delete all
interfaces created by cni and umount all mount points related to
k3s. I wrote a script for this situation and I will send it through a separate emailbased on your previous scripts.
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Sounds good.
Quoted 8 lines
Quoted 8 lines
You'll notice that I pushed an update to the k3s-wip branch, things are a bit closer to working there now.
I am interested to hear if flannel crashes if you (or anyone else) install (and use) the k3s-agent on the same node as the k3s-server.
As soon as you run any k3s-agent command (with the proper token and local host as the server), the node moves into NotReady with what looks like a CNI
error.
Quoted 3 lines
Quoted 3 lines
I fetched and ran the binaries directly from rancher, and they
showed the same behaviour as the meta-virt ones, so it isn't something fundamentalwith the integration.
Quoted 24 lines
Quoted 24 lines
Bruce
Quoted 21 lines
Best Regards,
LanceQuoted 19 lines
-----Original Message-----
From: [email protected] <[email protected]>
On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Tuesday, November 10, 2020 9:35 PM
To: Bruce Ashfield <[email protected]>
Cc: Lance Yang <[email protected]>; Joakim Roubert <[email protected]>; meta- [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeQuoted 5 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 1 line · Bruce Ashfield via lists.yoctoproject.org
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines · Lance Yang
Quoted 4 lines
Hi Bruce and Joakim,
Thanks for sharing this branch: k3s-wip. I have tested against my yocto
build.
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
The branch will be more functional shortly, I have quite a few changes to factor things for k8s and generally more usable :D
modified: classes/cni_networking.bbclass modified: conf/layer.conf modified: recipes-containers/containerd/containerd-docker_git.bb modified: recipes-containers/containerd/containerd-opencontainers_git.bb modified: recipes-containers/k3s/README.md modified: recipes-containers/k3s/k3s_git.bb modified: recipes-kernel/linux/linux-yocto/kubernetes.cfg modified: recipes-networking/cni/cni_git.bbcontainer-deploy.txt
recipes-core/packagegroups/Quoted 1 line
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
My Image: Linux qemuarm64 by yocto.
The master node can be ready after I started the k3s server. However, the pods in kube-
system (which are essential components for k3s) cannot turn to ready state on
qemuarm64.
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
That's interesting, since in my configuration, the master never comes ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 NotReady master 15h v1.18.9-k3s1Quoted 8 lines
Hah.
I finally got the node to show up as ready:
root@qemux86-64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemux86-64 Ready master 112s v1.18.9-k3s1Lance,
What image type were you building ? I'm pulling in dependencies to packagegroups and the recipes themselves.
I'm not seeing the mount issue on my master/server node:
root@qemux86-64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-h7lxk 1/1 Running 0 3m32s metrics-server-7566d596c8-mwntr 1/1 Running 0 3m32s helm-install-traefik-229v7 0/1 Completed 0 3m32s coredns-7944c66d8d-9rfj7 1/1 Running 0 3m32s svclb-traefik-pb5j4 2/2 Running 0 2m29s traefik-758cd5fc85-lxpr8 1/1 Running 0 2m29sQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I'm going back to all-in-one node debugging, but can look into the mount issue
more later.
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Bruce
Quoted 13 lines
I'm attempting to build an all-in-one node, and that is likely causing me some issues.
I'm revisiting those potential conflicts now.
But if anyone else does have an all in one working and has some tips, feel free to share :D
Bruce
Quoted 3 lines
I've sorted out more of the dependencies, and have packagegroups to make them easier now.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Hopefully, I can figure out what is now missing and keeping my master from moving into ready today.
Bruce
Quoted 1 line
After the master node itself turned to ready state, I check the pods with
kubectl:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
kubectl get nodes
NAME STATUS ROLES AGE VERSION qemuarm64 Ready master 11m v1.18.9-k3s1 root@qemuarm64:~# ls root@qemuarm64:~# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE local-path-provisioner-6d59f47c7-xxvbl 0/1 ContainerCreating 012m
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
coredns-7944c66d8d-tlrm9 0/1 ContainerCreating 0 12m metrics-server-7566d596c8-svkff 0/1 ContainerCreating 012m
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
helm-install-traefik-s8p5g 0/1 ContainerCreating 0 12m
Then I describe the pods with:
Events:
Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 16m default-scheduler Successfullyassigned kube-
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
system/coredns-7944c66d8d-tlrm9 to qemuarm64
Quoted 1 line
Quoted 1 line
Quoted 1 line
Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to
attach or
Quoted 5 lines
Quoted 5 lines
mount
Quoted 4 lines
Quoted 4 lines
volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token- b7nlh config-volume]: timed out waiting for the condition
Quoted 1 line
Quoted 1 line
Quoted 1 line
Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach
or mount
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume
coredns-token-b7nlh]: timed out waiting for the conditionQuoted 1 line
Quoted 1 line
Quoted 1 line
Warning FailedMount 11s (x16 over 16m) kubelet
MountVolume.SetUp
Quoted 40 lines
Quoted 40 lines
failed
Quoted 39 lines
Quoted 39 lines
for volume "coredns-token-b7nlh" : mount failed: exec: "mount":
executable file not found in $PATHQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and
mount binary on my qemuarm64 image:
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
root@qemuarm64:~# echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sb in root@qemuarm64:~# which mount /bin/mountWhen I type mount command, it worked fine:
/dev/rooton / type ext4 (rw,relatime) devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs
(rw,relatime) debugfs on/sys/kernel/debugtype debugfs (rw,relatime) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) ...
... (skipped the verbose output)I would like to know whether you have met this "mount" issue ever?
Best Regards,
LanceQuoted 11 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipeOn Wed, Oct 21, 2020 at 2:00 AM Joakim Roubert
Quoted 34 lines · <[email protected]> wrote:
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 6 lines · Bruce Ashfield
Quoted 3 lines
Ha!!!!
This applies.
Quoted 7 lines
Wonderful, thank you! I guess this is what is called "five times lucky"...
Quoted 3 lines
I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.
Excellent!
I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtuali
zati
on/l
og/?h=k3s-wipThat includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.
I did have a couple of questions on the k3s packaging itself, I was getting the following
error:ERROR: k3s-v1.18.9+k3s1-dirty-r0
do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3sQuoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
So I added them to the FILES of the k3s package itself
(so both k3s-server and k3s-agent will get them), is that the split you werelooking for ?
Quoted 52 lines
Quoted 52 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Bruce
Quoted 8 lines
BR,
/Joakim
--
Joakim Roubert
Senior EngineerQuoted 5 lines
Axis Communications AB Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIQuoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and
may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any
medium. Thank you.
Quoted 25 lines
Quoted 25 lines
Quoted 12 lines
Quoted 10 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIQuoted 7 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIQuoted 5 lines
Quoted 5 lines
Quoted 2 lines
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may
also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any
medium. Thank you.
Quoted 8 lines
Quoted 7 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek IIIMPORTANT NOTICE: The contents of this email and any attachments are confidential
and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 3 lines
--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Quoted 2 lines · Lance Yang
I started k3s server without agent and confirmed the node is ready. Then I copied the
node_tokenand then ran k3s agent with that token and specified the server url.
Did you do this on the same device?
I always run the k3s master on one device and the k3s agents on other machines (and thought that was how it is supposed to work).
BR,
/Joakim
Quoted 11 lines · Joakim Roubert
Quoted 4 lines · Lance Yang
I started k3s server without agent and confirmed the node is ready. Then I copied the
node_tokenand then ran k3s agent with that token and specified the server url.Did you do this on the same device?
I always run the k3s master on one device and the k3s agents on other machines (and thought that was how it is supposed to work).
FYI: after literally 5 months of on and off (most off) debugging and integration, I have finally gotten to the bottom of this, and now can bring up my all in one node out of the box!
I can now pull k3s into lots of little bits and am confident that I can support it, given my recent findings and debug.
I'm cleaning up changes, and will push a fully updated k3s to meta-virt master in the coming week.
Cheers,
Bruce
Quoted 7 lines
BR,
/Joakim
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 5 lines · Bruce Ashfield
Quoted 4 lines · Joakim Roubert
Quoted 3 lines · Lance Yang
I started k3s server without agent and confirmed the node is ready. Then I copied the
node_tokenand then ran k3s agent with that token and specified the server url.
Quoted 8 lines
Quoted 4 lines
Did you do this on the same device?
I always run the k3s master on one device and the k3s agents on other machines (and thought that was how it is supposed to work).FYI: after literally 5 months of on and off (most off) debugging and integration, I have finally gotten to the bottom of this, and now can bring up my all in one node out of the box!
bravo kube-edge next, or starlingx simplex aio ---- meta-stx
which combines k8s and openstack into a single platform
which i thought would be the ultimate goal for OverC :)
just thinking to myself outloud...
Quoted 22 lines
I can now pull k3s into lots of little bits and am confident that I can support it, given my recent findings and debug.
I'm cleaning up changes, and will push a fully updated k3s to meta-virt master in the coming week.
Cheers,
Bruce
Quoted 7 lines
BR,
/Joakim
Quoted 7 lines · Bruce Ashfield
FYI: after literally 5 months of on and off (most off) debugging and integration, I have finally gotten to the bottom of this, and now can bring up my all in one node out of the box!
I can now pull k3s into lots of little bits and am confident that I can support it, given my recent findings and debug.
This is fantastic news, Bruce!
Quoted 2 lines
I'm cleaning up changes, and will push a fully updated k3s to meta-virt master in the coming week.
This officially makes you the king.
BR,
/Joakim
FYI: This version of the patch hasn't been forgotten.
I'm just having some issues with containerd and alternate runtimes + the networking consolidation + preparing slides for ELC-e (x3!!) .. so progress has been slower than I wanted.
That being said, I expect to have it in place before branching the release.
Cheers,
Bruce
Quoted 448 lines · Joakim Roubert
Signed-off-by Joakim Roubert <[email protected]>--- recipes-containers/k3s/README.md | 26 +++++ ...01-Finding-host-local-in-usr-libexec.patch | 27 +++++ .../k3s/k3s/cni-containerd-net.conf | 24 +++++ recipes-containers/k3s/k3s/k3s-agent | 100 ++++++++++++++++++ recipes-containers/k3s/k3s/k3s-agent.service | 26 +++++ recipes-containers/k3s/k3s/k3s-clean | 25 +++++ recipes-containers/k3s/k3s/k3s.service | 27 +++++ recipes-containers/k3s/k3s_git.bb | 75 +++++++++++++ 8 files changed, 330 insertions(+)create mode 100644
recipes-containers/k3s/README.md
create mode 100644recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch
create mode 100644recipes-containers/k3s/k3s/cni-containerd-net.confcreate mode 100755 recipes-containers/k3s/k3s/k3s-agent create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service create mode 100755 recipes-containers/k3s/k3s/k3s-clean create mode 100644 recipes-containers/k3s/k3s/k3s.service create mode 100644 recipes-containers/k3s/k3s_git.bbrecipes-containers/k3s/README.md new file +7
-0@@ -0,0 +1,26 @@ +# k3s: Lightweight Kubernetes + +Rancher's [k3s](https://k3s.io/), available under +[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides +lightweight Kubernetes suitable for small/edge devices. There are use cases +where the +[installation procedures provided byRancher](https://rancher.com/docs/k3s/latest/en/installation/)
recipes-containers/k3s/README.md new file +6
-0+are not ideal but a bitbake-built version is what is needed. And only a few +mods to the [k3s source code](https://github.com/rancher/k3s) is needed to +accomplish that. + +## CNI +By default, K3s will run with flannel as the CNI, using VXLAN as thedefault
+backend. It is both possible to change the flannel backend and to change fromrecipes-containers/k3s/README.md new file +3
-0+flannel to another CNI. + +Please seehttps://rancher.com/docs/k3s/latest/en/installation/network-options/
recipes-containers/k3s/README.md new file +4
-0+for further k3s networking details. + +## Configure and run a k3s agent +The convenience script `k3s-agent` can be used to set up a k3s agent(service):
recipes-containers/k3s/README.md new file +5
-0+ + k3s-agent -t <token> -s https://<master>:6443 + +(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the +k3s master.)diff --git
a/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patchb/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patchnew file mode 100644
index 0000000..8205d73
---/dev/null
+++b/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patchrecipes-containers/k3s/README.md new file +27
-0@@ -0,0 +1,27 @@ +From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001 +From: Erik Jansson <[email protected]> +Date: Wed, 16 Oct 2019 15:07:48 +0200 +Subject: [PATCH] Finding host-local in /usr/libexec + +Upstream-status: Inappropriate [embedded specific] +Signed-off-by: <[email protected]> +--- + pkg/agent/config/config.go | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go +index b4296f360a..6af9dab895 100644 +--- a/pkg/agent/config/config.go ++++ b/pkg/agent/config/config.go +@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) { + return nil, err + } + +- hostLocal, err := exec.LookPath("host-local") ++ hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local") + if err != nil { + return nil, errors.Wrapf(err, "failed to find host-local") + } +-- +2.11.0 +recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24
-0@@ -0,0 +1,24 @@ +{ + "cniVersion": "0.4.0", + "name": "containerd-net", + "plugins": [ + { + "type": "bridge", + "bridge": "cni0", + "isGateway": true, + "ipMasq": true, + "promiscMode": true, + "ipam": { + "type": "host-local", + "subnet": "10.88.0.0/16", + "routes": [ + { "dst": "0.0.0.0/0" } + ] + } + }, + { + "type": "portmap", + "capabilities": {"portMappings": true} + } + ] +}recipes-containers/k3s/k3s/k3s-agent new file +11
-0@@ -0,0 +1,100 @@ +#!/bin/sh -eu +# SPDX-License-Identifier: Apache-2.0 + +ENV_CONF=/etc/systemd/system/k3s-agent.service.d/10-env.conf + +usage() { + echo " +USAGE: + ${0##*/} [OPTIONS] +OPTIONS: + --token value, -t value Token to use for authentication[\$K3S_TOKEN]
+ --token-file value Token file to use for
authentication [\$K3S_TOKEN_FILE]recipes-containers/k3s/k3s/k3s-agent new file +3
-0+ --server value, -s value Server to connect to [\$K3S_URL] + --node-name value Node name [\$K3S_NODE_NAME] + --resolv-conf value Kubelet resolv.conf file[\$K3S_RESOLV_CONF]
+ --cluster-secret value Shared secret used to bootstrap
a cluster [\$K3S_CLUSTER_SECRET]recipes-containers/k3s/k3s/k3s-agent new file +84
-0+ -h print this +" +} + +[ $# -gt 0 ] || { + usage + exit +} + +case $1 in + -*) + ;; + *) + usage + exit 1 + ;; +esac + +rm -f $ENV_CONF +mkdir -p ${ENV_CONF%/*} +echo [Service] > $ENV_CONF + +while getopts "t:s:-:h" opt; do + case $opt in + h) + usage + exit + ;; + t) + VAR_NAME=K3S_TOKEN + ;; + s) + VAR_NAME=K3S_URL + ;; + -) + [ $# -ge $OPTIND ] || { + usage + exit 1 + } + opt=$OPTARG + eval OPTARG='$'$OPTIND + OPTIND=$(($OPTIND + 1)) + case $opt in + token) + VAR_NAME=K3S_TOKEN + ;; + token-file) + VAR_NAME=K3S_TOKEN_FILE + ;; + server) + VAR_NAME=K3S_URL + ;; + node-name) + VAR_NAME=K3S_NODE_NAME + ;; + resolv-conf) + VAR_NAME=K3S_RESOLV_CONF + ;; + cluster-secret) + VAR_NAME=K3S_CLUSTER_SECRET + ;; + help) + usage + exit + ;; + *) + usage + exit 1 + ;; + esac + ;; + *) + usage + exit 1 + ;; + esac + echo Environment=$VAR_NAME=$OPTARG >> $ENV_CONF +done + +chmod 0644 $ENV_CONF +rm -rf /var/lib/rancher/k3s/agent +systemctl daemon-reload +systemctl restart k3s-agent +systemctl enable k3s-agent.servicerecipes-containers/k3s/k3s/k3s-agent.service new file +26
-0@@ -0,0 +1,26 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes Agent +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=control-group +Delegate=yes +LimitNOFILE=infinity +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s agent +ExecStopPost=/usr/local/bin/k3s-clean +recipes-containers/k3s/k3s/k3s-clean new file +25
-0@@ -0,0 +1,25 @@ +#!/bin/sh -eu +# SPDX-License-Identifier: Apache-2.0 +do_unmount() { + [ $# -eq 2 ] || return + local mounts= + while read ignore mount ignore; do + case $mount in + $1/*|$2/*) + mounts="$mount $mounts" + ;; + esac + done </proc/self/mounts + [ -z "$mounts" ] || umount $mounts +} + +do_unmount /run/k3s /var/lib/rancher/k3s + +ip link show | grep 'master cni0' | while read ignore iface ignore; do + iface=${iface%%@*} + [ -z "$iface" ] || ip link delete $iface +done + +ip link delete cni0 +ip link delete flannel.1 +rm -rf /var/lib/cni/recipes-containers/k3s/k3s/k3s.service new file +15
-0@@ -0,0 +1,27 @@ +# Derived from the k3s install.sh's create_systemd_service_file() function +[Unit] +Description=Lightweight Kubernetes +Documentation=https://k3s.io +Requires=containerd.service +After=containerd.service + +[Install] +WantedBy=multi-user.target + +[Service] +Type=notify +KillMode=process +Delegate=yes +# Having non-zero Limit*s causes performance problems due to accountingoverhead
+# in the kernel. We recommend using cgroups to do container-local accounting.recipes-containers/k3s/k3s/k3s.service new file +11
-0+LimitNOFILE=1048576 +LimitNPROC=infinity +LimitCORE=infinity +TasksMax=infinity +TimeoutStartSec=0 +Restart=always +RestartSec=5s +ExecStartPre=-/sbin/modprobe br_netfilter +ExecStartPre=-/sbin/modprobe overlay +ExecStart=/usr/local/bin/k3s server +recipes-containers/k3s/k3s_git.bb new file +2
-0@@ -0,0 +1,75 @@ +SUMMARY = "Production-Grade Container Scheduling and Management" +DESCRIPTION = "Lightweight Kubernetes, intended to be a fully compliantKubernetes."
recipes-containers/k3s/k3s_git.bb new file +3
-0+HOMEPAGE = "https://k3s.io/" +LICENSE = "Apache-2.0" +LIC_FILES_CHKSUM ="file://${S}/src/import/LICENSE;md5=2ee41112a44fe7014dce33e26468ba93"
recipes-containers/k3s/k3s_git.bb new file +9
-0+PV = "v1.18.9+k3s1-dirty" + +SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.18;name=k3s \ + file://k3s.service \ + file://k3s-agent.service \ + file://k3s-agent \ + file://k3s-clean \ + file://cni-containerd-net.conf \ +file://0001-Finding-host-local-in-usr-libexec.patch;patchdir=src/import \
recipes-containers/k3s/k3s_git.bb new file +13
-0+ " +SRC_URI[k3s.md5sum] = "363d3a08dc0b72ba6e6577964f6e94a5" +SRCREV_k3s = "630bebf94b9dce6b8cd3d402644ed023b3af8f90" + +inherit go +inherit goarch +inherit systemd + +PACKAGECONFIG = "" +PACKAGECONFIG[upx] = ",,upx-native" +GO_IMPORT = "import" +GO_BUILD_LDFLAGS = "-X github.com/rancher/k3s/pkg/version.Version=${PV} \ + -Xgithub.com/rancher/k3s/pkg/version.GitCommit=${@d.getVar('SRCREV_k3s', d, 1)[:8]} \
recipes-containers/k3s/k3s_git.bb new file +6
-0+ -w -s \ + " +BIN_PREFIX ?= "${exec_prefix}/local" + +do_compile() { + exportGOPATH="${S}/src/import/.gopath:${S}/src/import/vendor:${STAGING_DIR_TARGET}/${prefix}/local/go"
recipes-containers/k3s/k3s_git.bb new file +12
-1+ export CGO_ENABLED="1" + export GOFLAGS="-mod=vendor" + cd ${S}/src/import + ${GO} build -tags providerless -ldflags "${GO_BUILD_LDFLAGS}" -o ./dist/artifacts/k3s ./cmd/server/main.go + # Use UPX if it is enabled (and thus exists) to compress binary + if command -v upx > /dev/null 2>&1; then + upx -9 ./dist/artifacts/k3s + fi +} +do_install() { + install -d "${D}${BIN_PREFIX}/bin" + install -m 755 "${S}/src/import/dist/artifacts/k3s""${D}${BIN_PREFIX}/bin"
recipes-containers/k3s/k3s_git.bb new file +5
-0+ ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/crictl" + ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/ctr" + ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/kubectl" + install -m 755 "${WORKDIR}/k3s-clean" "${D}${BIN_PREFIX}/bin" + install -D -m 0644 "${WORKDIR}/cni-containerd-net.conf""
${D}/${sysconfdir}/cni/net.d/10-containerd-net.conf"
+ if
${@bb.utils.contains('DISTRO_FEATURES','systemd','true','false',d)}; then
+ install -D -m 0644 "${WORKDIR}/k3s.service"
"${D}${systemd_system_unitdir}/k3s.service"
+ install -D -m 0644 "${WORKDIR}/k3s-agent.service"
"${D}${systemd_system_unitdir}/k3s-agent.service"
+ sed -i
"s#\(Exec\)\(.*\)=\(.*\)\(k3s\)#\1\2=${BIN_PREFIX}/bin/\4#g" "${D}${systemd_system_unitdir}/k3s.service" "${D}${systemd_system_unitdir}/k3s-agent.service"
+ install -m 755 "${WORKDIR}/k3s-agent"
"${D}${BIN_PREFIX}/bin"recipes-containers/k3s/k3s_git.bb new file +6
-0+ fi +} + +PACKAGES =+ "${PN}-server ${PN}-agent" + +SYSTEMD_PACKAGES ="${@bb.utils.contains('DISTRO_FEATURES','systemd','${PN}-server ${PN}-agent','',d)}"
+SYSTEMD_SERVICE_${PN}-server = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s.service','',d)}" +SYSTEMD_SERVICE_${PN}-agent = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s-agent.service','',d)}"recipes-containers/k3s/k3s_git.bb new file +5
-0+SYSTEMD_AUTO_ENABLE_${PN}-agent = "disable" + +FILES_${PN}-agent = "${BIN_PREFIX}/bin/k3s-agent" + +RDEPENDS_${PN} = "cni conntrack-tools coreutils findutils iproute2ipset virtual/containerd"
recipes-containers/k3s/k3s_git.bb new file +7
-1+RDEPENDS_${PN}-server = "${PN}" +RDEPENDS_${PN}-agent = "${PN}" + +RCONFLICTS_${PN} = "kubectl" + +INHIBIT_PACKAGE_STRIP = "1" +INSANE_SKIP_${PN} += "ldflags already-stripped" --2.20.1
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II