Skip to contentSkip to conversation

Adding k3s recipe

PATCH Superseded Joakim Roubert 4 participants 73 messages Sep 21, 2020 – Mar 15, 2021 10 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw

It is close. I had to abandon one approach, since it grew more complex than if I just let a chunk of functionality be duplicated.

I'm at this again this week, and if I can't sort it out, I'll merge the core recipes and start factoring in tree (I'm just worried about people assuming what appears in tree is the final version).

Cheers,

Bruce

Quoted 15 lines · Joakim Roubert
Quoted 1 line · Erik Jansson

Bruce, what is the current status of the k3s activities? (I know you were busy with some other things last month.)

BR,

/Joakim

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#3 lore raw
Quoted 4 lines · Bruce Ashfield

It is close. I had to abandon one approach, since it grew more complex than if I just let a chunk of functionality be duplicated.

I'm at this again this week

Excellent news!

Quoted 3 lines

and if I can't sort it out, I'll merge the core recipes and start factoring in tree (I'm just worried about people assuming what appears in tree is the final version).

But with those words you have just informed eveybody on the list about it, so that should make things clearer for the community!

BR,

/Joakim

Signature · 7 lines
Joakim Roubert
Senior Engineer

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com
#4 lore raw
Quoted 7 lines · Joakim Roubert
Quoted 4 lines · Bruce Ashfield

It is close. I had to abandon one approach, since it grew more complex than if I just let a chunk of functionality be duplicated.

I'm at this again this week

Quoted 8 lines

Excellent news!

Quoted 3 lines

and if I can't sort it out, I'll merge the core recipes and start factoring in tree (I'm just worried about people assuming what appears in tree is the final version).

But with those words you have just informed eveybody on the list about it, so that should make things clearer for the community!

That I did!

So I've decided to do this in a couple phases, since I keep making the bbclass more complex that it needs to be.

I've gone back through the patches, and have some questions/comments. Would you be willing to tweak things basd on those comments ? I realize it's been quite the delay from submission to this, so I can work through issues if you don't have the cycles.

Bruce

Quoted 16 lines

BR,

/Joakim
--
Joakim Roubert
Senior Engineer

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#5 lore raw
Quoted 3 lines · Bruce Ashfield

So I've decided to do this in a couple phases, since I keep making the bbclass more complex that it needs to be.

Sounds like a good plan! For what we know, concepts as "Big Bang" only was successful once in the history of Universe, and it was not within the field of software development. ;-)

Quoted 4 lines

I've gone back through the patches, and have some questions/comments. Would you be willing to tweak things basd on those comments ? I realize it's been quite the delay from submission to this, so I can work through issues if you don't have the cycles.

Yes, I am happy to help in whatever way I can. Erik is a colleague of mine but currently on parental leave, so I will try to fill his shoes here now. But I have plenty of openembedded k3s build experience with these recipes too, so let's do this!

BR,

/Joakim

Signature · 7 lines
Joakim Roubert
Senior Engineer

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com
#6 lore raw
Quoted 15 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield

So I've decided to do this in a couple phases, since I keep making the bbclass more complex that it needs to be.

Sounds like a good plan! For what we know, concepts as "Big Bang" only was successful once in the history of Universe, and it was not within the field of software development. ;-)

Quoted 4 lines

I've gone back through the patches, and have some questions/comments. Would you be willing to tweak things basd on those comments ? I realize it's been quite the delay from submission to this, so I can work through issues if you don't have the cycles.

Quoted 4 lines

Yes, I am happy to help in whatever way I can. Erik is a colleague of mine but currently on parental leave, so I will try to fill his shoes here now. But I have plenty of openembedded k3s build experience with these recipes too, so let's do this!

By phases, I meant some smaller changes to the series while I work a bit more through the bbclass and DISTRO feature flag efforts.

I can say immediately, that we have a problem with the dependencies, and we need to sort those out.

There are at least two utilities that are shared with the main kubernetes and not in our existing layer dependencies. We shouldn't expand our dependencies more, so we have to look at importing a variant of the recipes to meta-virt, or seeing about getting the recipes into a more common location. In particular, I'm talking about ipset and and upx, and I'm still looking for more.

As for the service files, the k3s.service is different from the one from the latest on the k3s 1.18 branch, and there's a patch that modifies it. Although tempting to not use /usr/local/bin, for maintenance, we'd be better off either just installing where things are expected. Was something broken if /usr/local/bin was used ?

The other .service files, we have no idea where they come from by looking at the series, and as such they need some sort of comment (by looking at the repo, you can tell they are variants of the k3s.service file, but that needs to be clarified).

The patches themselves to the code base (not just the overall git commit) should have Signed-off-by: and Upstream-status: flags, so they can be maintained as I bump through releases.

The scripts being created by the recipe, need license headers and spdx tags. Boring, but it is something we want to make sure is in new files. Also, at a glance, we need to make sure the dependencies of the scripts are captured in the DEPENDS of the recipe (i.e. I see 'ip' being called, make sure that is captured).

As for the networking, what we have in the submission is specific to a setup, that's the parts that I'm really trying to unify. We need to document what is expected in the CNI configuration, and then control it via a distro/image flag such that it won't clobber/conflict with other CNI users. I can take care of that distro part, but the documentation would be ideal in a README that comes along with the recipe.

As I said, I'm still working through some of the factoring out and de-duplication, but there's some things in there that are best done on your end, versus mine.

Bruce

Quoted 13 lines

BR,

/Joakim
--
Joakim Roubert
Senior Engineer

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#7 lore raw
Quoted 7 lines · Bruce Ashfield

There are at least two utilities that are shared with the main kubernetes and not in our existing layer dependencies. We shouldn't expand our dependencies more, so we have to look at importing a variant of the recipes to meta-virt, or seeing about getting the recipes into a more common location. In particular, I'm talking about ipset and and upx, and I'm still looking for more.

Ah, yes.

Quoted 5 lines

As for the service files, the k3s.service is different from the one from the latest on the k3s 1.18 branch, and there's a patch that modifies it. Although tempting to not use /usr/local/bin, for maintenance, we'd be better off either just installing where things are expected. Was something broken if /usr/local/bin was used ?

The target systems I work on prohibits installations to /usr/local, so from my perspective something would in fact be broken if /usr/local/bin was to be used. Nevertheless, configuring the recipe to have /usr/{$VARIABLE}/bin and VARIABLE set to "local" as default would perhaps be an option for everybody to eat the cake and still have it?

Quoted 4 lines

The other .service files, we have no idea where they come from by looking at the series, and as such they need some sort of comment (by looking at the repo, you can tell they are variants of the k3s.service file, but that needs to be clarified).

I will investigate.

Quoted 3 lines

The patches themselves to the code base (not just the overall git commit) should have Signed-off-by: and Upstream-status: flags, so they can be maintained as I bump through releases.

I will see to that.

Quoted 3 lines

The scripts being created by the recipe, need license headers and spdx tags. Boring, but it is something we want to make sure is in new files.

Absolutely!

Quoted 3 lines

Also, at a glance, we need to make sure the dependencies of the scripts are captured in the DEPENDS of the recipe (i.e. I see 'ip' being called, make sure that is captured).

Ah, yes, I will go through that.

Quoted 7 lines

As for the networking, what we have in the submission is specific to a setup, that's the parts that I'm really trying to unify. We need to document what is expected in the CNI configuration, and then control it via a distro/image flag such that it won't clobber/conflict with other CNI users. I can take care of that distro part, but the documentation would be ideal in a README that comes along with the recipe.

That also sounds like a good plan. I am not relly sure what text to write in that README file, though. But I guess that is something that can be iteratively created.

Quoted 3 lines

As I said, I'm still working through some of the factoring out and de-duplication, but there's some things in there that are best done on your end, versus mine.

Excellent!

BR,

/Joakim

#8 lore raw
Quoted 18 lines · Joakim Roubert
Quoted 7 lines · Bruce Ashfield

There are at least two utilities that are shared with the main kubernetes and not in our existing layer dependencies. We shouldn't expand our dependencies more, so we have to look at importing a variant of the recipes to meta-virt, or seeing about getting the recipes into a more common location. In particular, I'm talking about ipset and and upx, and I'm still looking for more.

Ah, yes.

Quoted 5 lines

As for the service files, the k3s.service is different from the one from the latest on the k3s 1.18 branch, and there's a patch that modifies it. Although tempting to not use /usr/local/bin, for maintenance, we'd be better off either just installing where things are expected. Was something broken if /usr/local/bin was used ?

Quoted 5 lines

The target systems I work on prohibits installations to /usr/local, so from my perspective something would in fact be broken if /usr/local/bin was to be used. Nevertheless, configuring the recipe to have /usr/{$VARIABLE}/bin and VARIABLE set to "local" as default would perhaps be an option for everybody to eat the cake and still have it?

Absolutely. I'm always in favour of making these configurable, so we can let users customize as they see fit.

Alternatively, rather than substituting just the 'local', the path up to 'bin' could be controlled. And by that, I mean a variable local to the recipe, like you have above.

That allows the use of $bindir by default, and also covers the case automatically where someone has changed $bindir to /usr/local/bin.

The local variable allows yet another override if the global $bindir hasn't been changed, and you just want to modify it for this recipe (which is actually what I think will be most common).

Quoted 12 lines
Quoted 5 lines

The other .service files, we have no idea where they come from by looking at the series, and as such they need some sort of comment (by looking at the repo, you can tell they are variants of the k3s.service file, but that needs to be clarified).

I will investigate.

Quoted 3 lines

The patches themselves to the code base (not just the overall git commit) should have Signed-off-by: and Upstream-status: flags, so they can be maintained as I bump through releases.

Quoted 7 lines

I will see to that.

Quoted 3 lines

The scripts being created by the recipe, need license headers and spdx tags. Boring, but it is something we want to make sure is in new files.

Absolutely!

I just had to go through the License / SPDX header addition for hundreds of files. So I know what a pain this can be, so thanks!

Quoted 15 lines
Quoted 4 lines

Also, at a glance, we need to make sure the dependencies of the scripts are captured in the DEPENDS of the recipe (i.e. I see 'ip' being called, make sure that is captured).

Ah, yes, I will go through that.

Quoted 7 lines

As for the networking, what we have in the submission is specific to a setup, that's the parts that I'm really trying to unify. We need to document what is expected in the CNI configuration, and then control it via a distro/image flag such that it won't clobber/conflict with other CNI users. I can take care of that distro part, but the documentation would be ideal in a README that comes along with the recipe.

Quoted 3 lines

That also sounds like a good plan. I am not relly sure what text to write in that README file, though. But I guess that is something that can be iteratively created.

Absolutely. Just a stream of thoughts is fine for now, with a quick overview, and perhaps a guideline on how to build a single node system ? With a few notes about what is configured out of the box and what we expect the end user to continue to configure / customize ?

That would be more than enough as a start.

Bruce

Quoted 10 lines
Quoted 4 lines

As I said, I'm still working through some of the factoring out and de-duplication, but there's some things in there that are best done on your end, versus mine.

Excellent!

BR,

/Joakim

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#9 lore raw
Quoted 3 lines · Bruce Ashfield

Absolutely. I'm always in favour of making these configurable, so we can let users customize as they see fit.

Wonderful.

I finalized a quite (IMO) nice solution during the day, where ${exec_prefix} would be the fixed starting point and /bin the ending dito, and what is in between configurable. But I like your suggestion with even more flexibility better and will update according to that.

Quoted 2 lines

I just had to go through the License / SPDX header addition for hundreds of files. So I know what a pain this can be, so thanks!

No worries, I find those things very important too.

Quoted 4 lines

Absolutely. Just a stream of thoughts is fine for now, with a quick overview, and perhaps a guideline on how to build a single node system ? With a few notes about what is configured out of the box and what we expect the end user to continue to configure / customize ?

Ok!

I am running out of time here today, so even though I have ticked off several of the tasks on the list, it will be next week where I get back with an update!

BR,

/Joakim

A message that is not in the archive (sent off-list or not yet received): [email protected]

#11 lore raw
---
 recipes-containers/k3s/README.md              |  26 +++++
 ...01-Finding-host-local-in-usr-libexec.patch |  27 +++++
 .../k3s/k3s/cni-containerd-net.conf           |  24 +++++
 recipes-containers/k3s/k3s/k3s-agent          | 100 ++++++++++++++++++
 recipes-containers/k3s/k3s/k3s-agent.service  |  26 +++++
 recipes-containers/k3s/k3s/k3s-clean          |  25 +++++
 recipes-containers/k3s/k3s/k3s.service        |  27 +++++
 recipes-containers/k3s/k3s_git.bb             |  75 +++++++++++++
 8 files changed, 330 insertions(+)
 create mode 100644 recipes-containers/k3s/README.md
 create mode 100644 
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch
 create mode 100644 recipes-containers/k3s/k3s/cni-containerd-net.conf
 create mode 100755 recipes-containers/k3s/k3s/k3s-agent
 create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service
 create mode 100755 recipes-containers/k3s/k3s/k3s-clean
 create mode 100644 recipes-containers/k3s/k3s/k3s.service
 create mode 100644 recipes-containers/k3s/k3s_git.bb
recipes-containers/k3s/README.md new file +26 -0
new file mode 100644
index 0000000..8a0a994
@@ -0,0 +1,26 @@
+# k3s: Lightweight Kubernetes
+
+Rancher's [k3s](https://k3s.io/), available under
+[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides
+lightweight Kubernetes suitable for small/edge devices. There are use cases
+where the
+[installation procedures provided by Rancher](https://rancher.com/docs/k3s/latest/en/installation/)
+are not ideal but a bitbake-built version is what is needed. And only a few
+mods to the [k3s source code](https://github.com/rancher/k3s) is needed to
+accomplish that.
+
+## CNI
+By default, K3s will run with flannel as the CNI, using VXLAN as the default
+backend. It is both possible to change the flannel backend and to change from
+flannel to another CNI.
+
+Please see https://rancher.com/docs/k3s/latest/en/installation/network-options/
+for further k3s networking details.
+
+## Configure and run a k3s agent
+The convenience script `k3s-agent` can be used to set up a k3s agent (service):
+
+    k3s-agent -t <token> -s https://<master>:6443
+
+(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the
+k3s master.)
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch new file +27 -0
new file mode 100644
index 0000000..8205d73
@@ -0,0 +1,27 @@
+From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001
+From: Erik Jansson <[email protected]>
+Date: Wed, 16 Oct 2019 15:07:48 +0200
+Subject: [PATCH] Finding host-local in /usr/libexec
+
+Upstream-status: Inappropriate [embedded specific]
+Signed-off-by: <[email protected]>
+---
+ pkg/agent/config/config.go | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go
+index b4296f360a..6af9dab895 100644
+--- a/pkg/agent/config/config.go
++++ b/pkg/agent/config/config.go
+@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) {
+                return nil, err
+        }
+
+-      hostLocal, err := exec.LookPath("host-local")
++      hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local")
+        if err != nil {
+                return nil, errors.Wrapf(err, "failed to find host-local")
+        }
+--
+2.11.0
+
recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24 -0
new file mode 100644
index 0000000..ca434d6
@@ -0,0 +1,24 @@
+{
+  "cniVersion": "0.4.0",
+  "name": "containerd-net",
+  "plugins": [
+    {
+      "type": "bridge",
+      "bridge": "cni0",
+      "isGateway": true,
+      "ipMasq": true,
+      "promiscMode": true,
+      "ipam": {
+        "type": "host-local",
+        "subnet": "10.88.0.0/16",
+        "routes": [
+          { "dst": "0.0.0.0/0" }
+        ]
+      }
+    },
+    {
+      "type": "portmap",
+      "capabilities": {"portMappings": true}
+    }
+  ]
+}
recipes-containers/k3s/k3s/k3s-agent new file +100 -0

100 lines not on this page · raw message

recipes-containers/k3s/k3s/k3s-agent.service new file +26 -0
new file mode 100644
index 0000000..9f9016d
@@ -0,0 +1,26 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes Agent
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=control-group
+Delegate=yes
+LimitNOFILE=infinity
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s agent
+ExecStopPost=/usr/local/bin/k3s-clean
+
recipes-containers/k3s/k3s/k3s-clean new file +25 -0
new file mode 100755
index 0000000..8eff829
@@ -0,0 +1,25 @@
+#!/bin/sh -eu
+# SPDX-License-Identifier: Apache-2.0
+do_unmount() {
+       [ $# -eq 2 ] || return
+       local mounts=
+       while read ignore mount ignore; do
+               case $mount in
+                       $1/*|$2/*)
+                               mounts="$mount $mounts"
+                               ;;
+               esac
+       done </proc/self/mounts
+       [ -z "$mounts" ] || umount $mounts
+}
+
+do_unmount /run/k3s /var/lib/rancher/k3s
+
+ip link show | grep 'master cni0' | while read ignore iface ignore; do
+    iface=${iface%%@*}
+    [ -z "$iface" ] || ip link delete $iface
+done
+
+ip link delete cni0
+ip link delete flannel.1
+rm -rf /var/lib/cni/
recipes-containers/k3s/k3s/k3s.service new file +27 -0
new file mode 100644
index 0000000..34c7a80
@@ -0,0 +1,27 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=process
+Delegate=yes
+# Having non-zero Limit*s causes performance problems due to accounting overhead
+# in the kernel. We recommend using cgroups to do container-local accounting.
+LimitNOFILE=1048576
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s server
+
recipes-containers/k3s/k3s_git.bb new file +75 -0

75 lines not on this page · raw message

Signature · 1 line
2.20.1
#12 lore raw

Thanks for the updated series, see some comments inline.

Quoted 55 lines · Joakim Roubert
---
  recipes-containers/k3s/README.md              |  26 +++++
  ...01-Finding-host-local-in-usr-libexec.patch |  27 +++++
  .../k3s/k3s/cni-containerd-net.conf           |  24 +++++
  recipes-containers/k3s/k3s/k3s-agent          | 100 ++++++++++++++++++
  recipes-containers/k3s/k3s/k3s-agent.service  |  26 +++++
  recipes-containers/k3s/k3s/k3s-clean          |  25 +++++
  recipes-containers/k3s/k3s/k3s.service        |  27 +++++
  recipes-containers/k3s/k3s_git.bb             |  75 +++++++++++++
  8 files changed, 330 insertions(+)

create mode 100644 recipes-containers/k3s/README.md
create mode 100644
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch
create mode 100644 recipes-containers/k3s/k3s/cni-containerd-net.conf create mode 100755 recipes-containers/k3s/k3s/k3s-agent create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service create mode 100755 recipes-containers/k3s/k3s/k3s-clean create mode 100644 recipes-containers/k3s/k3s/k3s.service create mode 100644 recipes-containers/k3s/k3s_git.bb

recipes-containers/k3s/README.md new file +7 -0
new file mode 100644
index 0000000..8a0a994
@@ -0,0 +1,26 @@
+# k3s: Lightweight Kubernetes
+
+Rancher's [k3s](https://k3s.io/), available under
+[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides
+lightweight Kubernetes suitable for small/edge devices. There are use cases
+where the
+[installation procedures provided by

Rancher](https://rancher.com/docs/k3s/latest/en/installation/)

recipes-containers/k3s/README.md new file +6 -0
+are not ideal but a bitbake-built version is what is needed. And only a few
+mods to the [k3s source code](https://github.com/rancher/k3s) is needed to
+accomplish that.
+
+## CNI
+By default, K3s will run with flannel as the CNI, using VXLAN as the

default
+backend. It is both possible to change the flannel backend and to change from

recipes-containers/k3s/README.md new file +3 -0
+flannel to another CNI.
+
+Please see

https://rancher.com/docs/k3s/latest/en/installation/network-options/

recipes-containers/k3s/README.md new file +4 -0
+for further k3s networking details.
+
+## Configure and run a k3s agent
+The convenience script `k3s-agent` can be used to set up a k3s agent

(service):

recipes-containers/k3s/README.md line 26

Quoted 5 lines
recipes-containers/k3s/README.md new file +5 -0
+
+    k3s-agent -t <token> -s https://<master>:6443
+
+(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the
+k3s master.)

Thanks for the README. It's a good start and pretty much all we need for now. I'll be doing some build and runtime testing and can do some tweaks as required.

Quoted 74 lines

diff --git a/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch b/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch new file mode 100644
index 0000000..8205d73
--- /dev/null
+++
b/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch

recipes-containers/k3s/README.md new file +27 -0
@@ -0,0 +1,27 @@
+From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001
+From: Erik Jansson <[email protected]>
+Date: Wed, 16 Oct 2019 15:07:48 +0200
+Subject: [PATCH] Finding host-local in /usr/libexec
+
+Upstream-status: Inappropriate [embedded specific]
+Signed-off-by: <[email protected]>
+---
+ pkg/agent/config/config.go | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go
+index b4296f360a..6af9dab895 100644
+--- a/pkg/agent/config/config.go
++++ b/pkg/agent/config/config.go
+@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) {
+                return nil, err
+        }
+
+-      hostLocal, err := exec.LookPath("host-local")
++      hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local")
+        if err != nil {
+                return nil, errors.Wrapf(err, "failed to find host-local")
+        }
+--
+2.11.0
+
recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24 -0
new file mode 100644
index 0000000..ca434d6
@@ -0,0 +1,24 @@
+{
+  "cniVersion": "0.4.0",
+  "name": "containerd-net",
+  "plugins": [
+    {
+      "type": "bridge",
+      "bridge": "cni0",
+      "isGateway": true,
+      "ipMasq": true,
+      "promiscMode": true,
+      "ipam": {
+        "type": "host-local",
+        "subnet": "10.88.0.0/16",
+        "routes": [
+          { "dst": "0.0.0.0/0" }
+        ]
+      }
+    },
+    {
+      "type": "portmap",
+      "capabilities": {"portMappings": true}
+    }
+  ]
+}
recipes-containers/k3s/k3s/k3s-agent new file +0 -0
new file mode 100755
index 0000000..1bb4c78

recipes-containers/k3s/k3s/k3s-agent line 1

Quoted 1 line
recipes-containers/k3s/k3s/k3s-agent new file +1 -0
@@ -0,0 +1,100 @@
+#!/bin/sh -eu

For these scripts, we have the license, but not a copyright. Which should be ok, but are the scripts completely written by you (or someone at your company?), if so, it is a good idea to put a copyright header on the scripts, so we can know the origin.

recipes-containers/k3s/k3s/k3s-agent line 2

Quoted 1 line
recipes-containers/k3s/k3s/k3s-agent new file +1 -0
+# SPDX-License-Identifier: Apache-2.0

The SPDX headers look good.

Quoted 109 lines
recipes-containers/k3s/k3s/k3s-agent new file +9 -0
+
+ENV_CONF=/etc/systemd/system/k3s-agent.service.d/10-env.conf
+
+usage() {
+       echo "
+USAGE:
+    ${0##*/} [OPTIONS]
+OPTIONS:
+    --token value, -t value             Token to use for authentication

[\$K3S_TOKEN]
+ --token-file value Token file to use for
authentication [\$K3S_TOKEN_FILE]

recipes-containers/k3s/k3s/k3s-agent new file +3 -0
+    --server value, -s value            Server to connect to [\$K3S_URL]
+    --node-name value                   Node name [\$K3S_NODE_NAME]
+    --resolv-conf value                 Kubelet resolv.conf file

[\$K3S_RESOLV_CONF]
+ --cluster-secret value Shared secret used to bootstrap
a cluster [\$K3S_CLUSTER_SECRET]

recipes-containers/k3s/k3s/k3s-agent new file +84 -0
+    -h                                  print this
+"
+}
+
+[ $# -gt 0 ] || {
+       usage
+       exit
+}
+
+case $1 in
+       -*)
+               ;;
+       *)
+               usage
+               exit 1
+               ;;
+esac
+
+rm -f $ENV_CONF
+mkdir -p ${ENV_CONF%/*}
+echo [Service] > $ENV_CONF
+
+while getopts "t:s:-:h" opt; do
+       case $opt in
+               h)
+                       usage
+                       exit
+                       ;;
+               t)
+                       VAR_NAME=K3S_TOKEN
+                       ;;
+               s)
+                       VAR_NAME=K3S_URL
+                       ;;
+               -)
+                       [ $# -ge $OPTIND ] || {
+                               usage
+                               exit 1
+                       }
+                       opt=$OPTARG
+                       eval OPTARG='$'$OPTIND
+                       OPTIND=$(($OPTIND + 1))
+                       case $opt in
+                               token)
+                                       VAR_NAME=K3S_TOKEN
+                                       ;;
+                               token-file)
+                                       VAR_NAME=K3S_TOKEN_FILE
+                                       ;;
+                               server)
+                                       VAR_NAME=K3S_URL
+                                       ;;
+                               node-name)
+                                       VAR_NAME=K3S_NODE_NAME
+                                       ;;
+                               resolv-conf)
+                                       VAR_NAME=K3S_RESOLV_CONF
+                                       ;;
+                               cluster-secret)
+                                       VAR_NAME=K3S_CLUSTER_SECRET
+                                       ;;
+                               help)
+                                       usage
+                                       exit
+                                       ;;
+                               *)
+                                       usage
+                                       exit 1
+                                       ;;
+                       esac
+                       ;;
+               *)
+                       usage
+                       exit 1
+                       ;;
+       esac
+    echo Environment=$VAR_NAME=$OPTARG >> $ENV_CONF
+done
+
+chmod 0644 $ENV_CONF
+rm -rf /var/lib/rancher/k3s/agent
+systemctl daemon-reload
+systemctl restart k3s-agent
+systemctl enable k3s-agent.service
recipes-containers/k3s/k3s/k3s-agent.service new file +0 -0
new file mode 100644
index 0000000..9f9016d

recipes-containers/k3s/k3s/k3s-agent.service line 1

Quoted 1 line
recipes-containers/k3s/k3s/k3s-agent.service new file +1 -0
@@ -0,0 +1,26 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function

Perfect. This is what I was looking for.

Quoted 158 lines
recipes-containers/k3s/k3s/k3s-agent.service new file +25 -0
+[Unit]
+Description=Lightweight Kubernetes Agent
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=control-group
+Delegate=yes
+LimitNOFILE=infinity
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s agent
+ExecStopPost=/usr/local/bin/k3s-clean
+
recipes-containers/k3s/k3s/k3s-clean new file +25 -0
new file mode 100755
index 0000000..8eff829
@@ -0,0 +1,25 @@
+#!/bin/sh -eu
+# SPDX-License-Identifier: Apache-2.0
+do_unmount() {
+       [ $# -eq 2 ] || return
+       local mounts=
+       while read ignore mount ignore; do
+               case $mount in
+                       $1/*|$2/*)
+                               mounts="$mount $mounts"
+                               ;;
+               esac
+       done </proc/self/mounts
+       [ -z "$mounts" ] || umount $mounts
+}
+
+do_unmount /run/k3s /var/lib/rancher/k3s
+
+ip link show | grep 'master cni0' | while read ignore iface ignore; do
+    iface=${iface%%@*}
+    [ -z "$iface" ] || ip link delete $iface
+done
+
+ip link delete cni0
+ip link delete flannel.1
+rm -rf /var/lib/cni/
recipes-containers/k3s/k3s/k3s.service new file +15 -0
new file mode 100644
index 0000000..34c7a80
@@ -0,0 +1,27 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=process
+Delegate=yes
+# Having non-zero Limit*s causes performance problems due to accounting

overhead
+# in the kernel. We recommend using cgroups to do container-local accounting.

recipes-containers/k3s/k3s/k3s.service new file +11 -0
+LimitNOFILE=1048576
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s server
+
recipes-containers/k3s/k3s_git.bb new file +2 -0
new file mode 100644
index 0000000..cfc2c64
@@ -0,0 +1,75 @@
+SUMMARY = "Production-Grade Container Scheduling and Management"
+DESCRIPTION = "Lightweight Kubernetes, intended to be a fully compliant

Kubernetes."

recipes-containers/k3s/k3s_git.bb new file +3 -0
+HOMEPAGE = "https://k3s.io/"
+LICENSE = "Apache-2.0"
+LIC_FILES_CHKSUM =

"file://${S}/src/import/LICENSE;md5=2ee41112a44fe7014dce33e26468ba93"

recipes-containers/k3s/k3s_git.bb new file +9 -0
+PV = "v1.18.9+k3s1-dirty"
+
+SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.18;name=k3s \
+           file://k3s.service \
+           file://k3s-agent.service \
+           file://k3s-agent \
+           file://k3s-clean \
+           file://cni-containerd-net.conf \
+

file://0001-Finding-host-local-in-usr-libexec.patch;patchdir=src/import \

recipes-containers/k3s/k3s_git.bb new file +13 -0
+          "
+SRC_URI[k3s.md5sum] = "363d3a08dc0b72ba6e6577964f6e94a5"
+SRCREV_k3s = "630bebf94b9dce6b8cd3d402644ed023b3af8f90"
+
+inherit go
+inherit goarch
+inherit systemd
+
+PACKAGECONFIG = ""
+PACKAGECONFIG[upx] = ",,upx-native"
+GO_IMPORT = "import"
+GO_BUILD_LDFLAGS = "-X github.com/rancher/k3s/pkg/version.Version=${PV} \
+                    -X

github.com/rancher/k3s/pkg/version.GitCommit=${@d.getVar('SRCREV_k3s', d, 1)[:8]} \

recipes-containers/k3s/k3s_git.bb new file +6 -0
+                    -w -s \
+                   "
+BIN_PREFIX ?= "${exec_prefix}/local"
+
+do_compile() {
+        export

GOPATH="${S}/src/import/.gopath:${S}/src/import/vendor:${STAGING_DIR_TARGET}/${prefix}/local/go"

recipes-containers/k3s/k3s_git.bb new file +12 -1
+        export CGO_ENABLED="1"
+        export GOFLAGS="-mod=vendor"
+        cd ${S}/src/import
+        ${GO} build -tags providerless -ldflags "${GO_BUILD_LDFLAGS}"
-o ./dist/artifacts/k3s ./cmd/server/main.go
+        # Use UPX if it is enabled (and thus exists) to compress binary
+        if command -v upx > /dev/null 2>&1; then
+                upx -9 ./dist/artifacts/k3s
+        fi
+}
+do_install() {
+        install -d "${D}${BIN_PREFIX}/bin"
+        install -m 755 "${S}/src/import/dist/artifacts/k3s"

"${D}${BIN_PREFIX}/bin"

recipes-containers/k3s/k3s_git.bb new file +5 -0
+        ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/crictl"
+        ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/ctr"
+        ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/kubectl"
+        install -m 755 "${WORKDIR}/k3s-clean" "${D}${BIN_PREFIX}/bin"
+        install -D -m 0644 "${WORKDIR}/cni-containerd-net.conf"
Quoted 2 lines

"${D}/${sysconfdir}/cni/net.d/10-containerd-net.conf"
+ if

I'm going to abstract the networking configuration into a kubernetes-networking package, so we can share it amongst the various recipes, and have a way to control whether or not this configuration is installed. That allows an easy way to
both share and override the networking.

Quoted 28 lines

${@bb.utils.contains('DISTRO_FEATURES','systemd','true','false',d)}; then
+ install -D -m 0644 "${WORKDIR}/k3s.service"
"${D}${systemd_system_unitdir}/k3s.service"
+ install -D -m 0644 "${WORKDIR}/k3s-agent.service"
"${D}${systemd_system_unitdir}/k3s-agent.service"
+ sed -i
"s#\(Exec\)\(.*\)=\(.*\)\(k3s\)#\1\2=${BIN_PREFIX}/bin/\4#g" "${D}${systemd_system_unitdir}/k3s.service" "${D}${systemd_system_unitdir}/k3s-agent.service"
+ install -m 755 "${WORKDIR}/k3s-agent"
"${D}${BIN_PREFIX}/bin"

recipes-containers/k3s/k3s_git.bb new file +6 -0
+        fi
+}
+
+PACKAGES =+ "${PN}-server ${PN}-agent"
+
+SYSTEMD_PACKAGES =

"${@bb.utils.contains('DISTRO_FEATURES','systemd','${PN}-server ${PN}-agent','',d)}"
+SYSTEMD_SERVICE_${PN}-server = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s.service','',d)}" +SYSTEMD_SERVICE_${PN}-agent = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s-agent.service','',d)}"

recipes-containers/k3s/k3s_git.bb new file +5 -0
+SYSTEMD_AUTO_ENABLE_${PN}-agent = "disable"
+
+FILES_${PN}-agent = "${BIN_PREFIX}/bin/k3s-agent"
+
+RDEPENDS_${PN} = "cni conntrack-tools coreutils findutils iproute2
Quoted 1 line

ipset virtual/containerd"

I'll also take care of getting ipset in a place where we don't have to add extra layer
dependencies.

Bruce

Quoted 13 lines
recipes-containers/k3s/k3s_git.bb new file +7 -1
+RDEPENDS_${PN}-server = "${PN}"
+RDEPENDS_${PN}-agent = "${PN}"
+
+RCONFLICTS_${PN} = "kubectl"
+
+INHIBIT_PACKAGE_STRIP = "1"
+INSANE_SKIP_${PN} += "ldflags already-stripped"
--

2.20.1

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#13 lore raw
Quoted 1 line · Bruce Ashfield

Thanks for the updated series, see some comments inline.

Awesome, I am happy that you liked it!

Quoted 2 lines

Thanks for the README. It's a good start and pretty much all we need for now.

I realized I had forgotten to run markdownlint for it, so a slightly updated version is included in the upcoming patch update.

Quoted 2 lines

I'll be doing some build and runtime testing and can do some tweaks as required.

Perfect!

Quoted 3 lines

For these scripts, we have the license, but not a copyright. Which should be ok, but are the scripts completely written by you (or someone at your company?), if so, it is a good idea to put a
copyright header on the scripts, so we can know the origin.

I have added that now; I am quite sure that Erik is the author of those. By the look of the coding style, it sure seems like it. (I have sent a text to him just to verify and if I am wrong in my assumption here, I will update accordingly.)

Quoted 4 lines

I'm going to abstract the networking configuration into a kubernetes-networking package, so we can share it amongst the various recipes, and have a way to control whether or not this configuration
is installed. That allows an easy way to both share and override the
networking.

I like that!

Quoted 2 lines

I'll also take care of getting ipset in a place where we don't have to add extra layer dependencies.

Perfect, thank you!

Following this reply, I will send the latest updates in a separate e-mail.

BR,

/Joakim

A message that is not in the archive (sent off-list or not yet received): [email protected]

#15 lore raw
Change-Id Id1c52727593bc5ea8d0cd2de192faa44304d7a45
---
 recipes-containers/k3s/README.md              |  30 +++++
 ...01-Finding-host-local-in-usr-libexec.patch |  27 +++++
 .../k3s/k3s/cni-containerd-net.conf           |  24 ++++
 recipes-containers/k3s/k3s/k3s-agent          | 103 ++++++++++++++++++
 recipes-containers/k3s/k3s/k3s-agent.service  |  26 +++++
 recipes-containers/k3s/k3s/k3s-clean          |  29 +++++
 recipes-containers/k3s/k3s/k3s.service        |  27 +++++
 recipes-containers/k3s/k3s_git.bb             |  75 +++++++++++++
 8 files changed, 341 insertions(+)
 create mode 100644 recipes-containers/k3s/README.md
 create mode 100644 
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch
 create mode 100644 recipes-containers/k3s/k3s/cni-containerd-net.conf
 create mode 100755 recipes-containers/k3s/k3s/k3s-agent
 create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service
 create mode 100755 recipes-containers/k3s/k3s/k3s-clean
 create mode 100644 recipes-containers/k3s/k3s/k3s.service
 create mode 100644 recipes-containers/k3s/k3s_git.bb
recipes-containers/k3s/README.md new file +30 -0
new file mode 100644
index 0000000..3fe5ccd
@@ -0,0 +1,30 @@
+# k3s: Lightweight Kubernetes
+
+Rancher's [k3s](https://k3s.io/), available under
+[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides
+lightweight Kubernetes suitable for small/edge devices. There are use cases
+where the
+[installation procedures provided by Rancher](https://rancher.com/docs/k3s/latest/en/installation/)
+are not ideal but a bitbake-built version is what is needed. And only a few
+mods to the [k3s source code](https://github.com/rancher/k3s) is needed to
+accomplish that.
+
+## CNI
+
+By default, K3s will run with flannel as the CNI, using VXLAN as the default
+backend. It is both possible to change the flannel backend and to change from
+flannel to another CNI.
+
+Please see <https://rancher.com/docs/k3s/latest/en/installation/network-options/>
+for further k3s networking details.
+
+## Configure and run a k3s agent
+
+The convenience script `k3s-agent` can be used to set up a k3s agent (service):
+
+```shell
+k3s-agent -t <token> -s https://<master>:6443
+```
+
+(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the
+k3s master.)
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch new file +27 -0
new file mode 100644
index 0000000..8205d73
@@ -0,0 +1,27 @@
+From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001
+From: Erik Jansson <[email protected]>
+Date: Wed, 16 Oct 2019 15:07:48 +0200
+Subject: [PATCH] Finding host-local in /usr/libexec
+
+Upstream-status: Inappropriate [embedded specific]
+Signed-off-by: <[email protected]>
+---
+ pkg/agent/config/config.go | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go
+index b4296f360a..6af9dab895 100644
+--- a/pkg/agent/config/config.go
++++ b/pkg/agent/config/config.go
+@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) {
+                return nil, err
+        }
+
+-      hostLocal, err := exec.LookPath("host-local")
++      hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local")
+        if err != nil {
+                return nil, errors.Wrapf(err, "failed to find host-local")
+        }
+--
+2.11.0
+
recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24 -0
new file mode 100644
index 0000000..ca434d6
@@ -0,0 +1,24 @@
+{
+  "cniVersion": "0.4.0",
+  "name": "containerd-net",
+  "plugins": [
+    {
+      "type": "bridge",
+      "bridge": "cni0",
+      "isGateway": true,
+      "ipMasq": true,
+      "promiscMode": true,
+      "ipam": {
+        "type": "host-local",
+        "subnet": "10.88.0.0/16",
+        "routes": [
+          { "dst": "0.0.0.0/0" }
+        ]
+      }
+    },
+    {
+      "type": "portmap",
+      "capabilities": {"portMappings": true}
+    }
+  ]
+}
recipes-containers/k3s/k3s/k3s-agent new file +103 -0

103 lines not on this page · raw message

recipes-containers/k3s/k3s/k3s-agent.service new file +26 -0
new file mode 100644
index 0000000..9f9016d
@@ -0,0 +1,26 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes Agent
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=control-group
+Delegate=yes
+LimitNOFILE=infinity
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s agent
+ExecStopPost=/usr/local/bin/k3s-clean
+
recipes-containers/k3s/k3s/k3s-clean new file +29 -0
new file mode 100755
index 0000000..30b74a7
@@ -0,0 +1,29 @@
+#!/bin/sh -eu
+#
+# Copyright (C) 2020 Axis Communications AB
+#
+# SPDX-License-Identifier: Apache-2.0
+
+do_unmount() {
+       [ $# -eq 2 ] || return
+       local mounts=
+       while read ignore mount ignore; do
+               case $mount in
+                       $1/*|$2/*)
+                               mounts="$mount $mounts"
+                               ;;
+               esac
+       done </proc/self/mounts
+       [ -z "$mounts" ] || umount $mounts
+}
+
+do_unmount /run/k3s /var/lib/rancher/k3s
+
+ip link show | grep 'master cni0' | while read ignore iface ignore; do
+    iface=${iface%%@*}
+    [ -z "$iface" ] || ip link delete $iface
+done
+
+ip link delete cni0
+ip link delete flannel.1
+rm -rf /var/lib/cni/
recipes-containers/k3s/k3s/k3s.service new file +27 -0
new file mode 100644
index 0000000..34c7a80
@@ -0,0 +1,27 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=process
+Delegate=yes
+# Having non-zero Limit*s causes performance problems due to accounting overhead
+# in the kernel. We recommend using cgroups to do container-local accounting.
+LimitNOFILE=1048576
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s server
+
recipes-containers/k3s/k3s_git.bb new file +75 -0

75 lines not on this page · raw message

Signature · 1 line
2.20.1
#16 lore raw
In reply to a message not in the archive
Quoted 5 lines · Joakim Roubert

I have added that now; I am quite sure that Erik is the author of those. By the look of the coding style, it sure seems like it. (I have sent a text to him just to verify and if I am wrong in my assumption here, I will update accordingly.)

Update accordingly: the last couple of lines in k3s-clean are from install.sh's create_killall.sh function. I have now added a comment that clearly states that. Hopefully now everything is in order so that we do not claim someone else's cred.

The patchset with this update follows this reply in a separate e-mail.

BR,

/Joakim

A message that is not in the archive (sent off-list or not yet received): [email protected]

#18 lore raw
Change-Id Id1c52727593bc5ea8d0cd2de192faa44304d7a45
---
 recipes-containers/k3s/README.md              |  30 +++++
 ...01-Finding-host-local-in-usr-libexec.patch |  27 +++++
 .../k3s/k3s/cni-containerd-net.conf           |  24 ++++
 recipes-containers/k3s/k3s/k3s-agent          | 103 ++++++++++++++++++
 recipes-containers/k3s/k3s/k3s-agent.service  |  26 +++++
 recipes-containers/k3s/k3s/k3s-clean          |  30 +++++
 recipes-containers/k3s/k3s/k3s.service        |  27 +++++
 recipes-containers/k3s/k3s_git.bb             |  75 +++++++++++++
 8 files changed, 342 insertions(+)
 create mode 100644 recipes-containers/k3s/README.md
 create mode 100644 
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch
 create mode 100644 recipes-containers/k3s/k3s/cni-containerd-net.conf
 create mode 100755 recipes-containers/k3s/k3s/k3s-agent
 create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service
 create mode 100755 recipes-containers/k3s/k3s/k3s-clean
 create mode 100644 recipes-containers/k3s/k3s/k3s.service
 create mode 100644 recipes-containers/k3s/k3s_git.bb
recipes-containers/k3s/README.md new file +30 -0
new file mode 100644
index 0000000..3fe5ccd
@@ -0,0 +1,30 @@
+# k3s: Lightweight Kubernetes
+
+Rancher's [k3s](https://k3s.io/), available under
+[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides
+lightweight Kubernetes suitable for small/edge devices. There are use cases
+where the
+[installation procedures provided by Rancher](https://rancher.com/docs/k3s/latest/en/installation/)
+are not ideal but a bitbake-built version is what is needed. And only a few
+mods to the [k3s source code](https://github.com/rancher/k3s) is needed to
+accomplish that.
+
+## CNI
+
+By default, K3s will run with flannel as the CNI, using VXLAN as the default
+backend. It is both possible to change the flannel backend and to change from
+flannel to another CNI.
+
+Please see <https://rancher.com/docs/k3s/latest/en/installation/network-options/>
+for further k3s networking details.
+
+## Configure and run a k3s agent
+
+The convenience script `k3s-agent` can be used to set up a k3s agent (service):
+
+```shell
+k3s-agent -t <token> -s https://<master>:6443
+```
+
+(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the
+k3s master.)
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch new file +27 -0
new file mode 100644
index 0000000..8205d73
@@ -0,0 +1,27 @@
+From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001
+From: Erik Jansson <[email protected]>
+Date: Wed, 16 Oct 2019 15:07:48 +0200
+Subject: [PATCH] Finding host-local in /usr/libexec
+
+Upstream-status: Inappropriate [embedded specific]
+Signed-off-by: <[email protected]>
+---
+ pkg/agent/config/config.go | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go
+index b4296f360a..6af9dab895 100644
+--- a/pkg/agent/config/config.go
++++ b/pkg/agent/config/config.go
+@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) {
+ 		return nil, err
+ 	}
+
+-	hostLocal, err := exec.LookPath("host-local")
++	hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local")
+ 	if err != nil {
+ 		return nil, errors.Wrapf(err, "failed to find host-local")
+ 	}
+--
+2.11.0
+
recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24 -0
new file mode 100644
index 0000000..ca434d6
@@ -0,0 +1,24 @@
+{
+  "cniVersion": "0.4.0",
+  "name": "containerd-net",
+  "plugins": [
+    {
+      "type": "bridge",
+      "bridge": "cni0",
+      "isGateway": true,
+      "ipMasq": true,
+      "promiscMode": true,
+      "ipam": {
+        "type": "host-local",
+        "subnet": "10.88.0.0/16",
+        "routes": [
+          { "dst": "0.0.0.0/0" }
+        ]
+      }
+    },
+    {
+      "type": "portmap",
+      "capabilities": {"portMappings": true}
+    }
+  ]
+}
recipes-containers/k3s/k3s/k3s-agent new file +103 -0

103 lines not on this page · raw message

recipes-containers/k3s/k3s/k3s-agent.service new file +26 -0
new file mode 100644
index 0000000..9f9016d
@@ -0,0 +1,26 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes Agent
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=control-group
+Delegate=yes
+LimitNOFILE=infinity
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s agent
+ExecStopPost=/usr/local/bin/k3s-clean
+
recipes-containers/k3s/k3s/k3s-clean new file +30 -0
new file mode 100755
index 0000000..8eca918
@@ -0,0 +1,30 @@
+#!/bin/sh -eu
+#
+# Copyright (C) 2020 Axis Communications AB
+#
+# SPDX-License-Identifier: Apache-2.0
+
+do_unmount() {
+	[ $# -eq 2 ] || return
+	local mounts=
+	while read ignore mount ignore; do
+		case $mount in
+			$1/*|$2/*)
+				mounts="$mount $mounts"
+				;;
+		esac
+	done </proc/self/mounts
+	[ -z "$mounts" ] || umount $mounts
+}
+
+do_unmount /run/k3s /var/lib/rancher/k3s
+
+# The lines below come from install.sh's create_killall() function:
+ip link show 2>/dev/null | grep 'master cni0' | while read ignore iface ignore; do
+    iface=${iface%%@*}
+    [ -z "$iface" ] || ip link delete $iface
+done
+
+ip link delete cni0
+ip link delete flannel.1
+rm -rf /var/lib/cni/
recipes-containers/k3s/k3s/k3s.service new file +27 -0
new file mode 100644
index 0000000..34c7a80
@@ -0,0 +1,27 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=process
+Delegate=yes
+# Having non-zero Limit*s causes performance problems due to accounting overhead
+# in the kernel. We recommend using cgroups to do container-local accounting.
+LimitNOFILE=1048576
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s server
+
recipes-containers/k3s/k3s_git.bb new file +75 -0

75 lines not on this page · raw message

Signature · 1 line
2.20.1
#19 lore raw

Something has happened to this version of the patch in transit.

When I save and apply it using my normal workflow, the patch is corrupted.

Can you re-send it with a "v4" in the subject ? That will hopefully let it escape gmail's conversation filter.

I'll try and hack the patch by hand, but would like to see a v4 for comparison.

I'm currently importing some recipes for temporary support in meta-virt and creating a networking recipe while I wait for v4.

Cheers,

Bruce

Quoted 461 lines · Joakim Roubert
Change-Id Id1c52727593bc5ea8d0cd2de192faa44304d7a45
---
  recipes-containers/k3s/README.md              |  30 +++++
  ...01-Finding-host-local-in-usr-libexec.patch |  27 +++++
  .../k3s/k3s/cni-containerd-net.conf           |  24 ++++
  recipes-containers/k3s/k3s/k3s-agent          | 103 ++++++++++++++++++
  recipes-containers/k3s/k3s/k3s-agent.service  |  26 +++++
  recipes-containers/k3s/k3s/k3s-clean          |  30 +++++
  recipes-containers/k3s/k3s/k3s.service        |  27 +++++
  recipes-containers/k3s/k3s_git.bb             |  75 +++++++++++++
  8 files changed, 342 insertions(+)

create mode 100644 recipes-containers/k3s/README.md
create mode 100644
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch
create mode 100644 recipes-containers/k3s/k3s/cni-containerd-net.conf create mode 100755 recipes-containers/k3s/k3s/k3s-agent create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service create mode 100755 recipes-containers/k3s/k3s/k3s-clean create mode 100644 recipes-containers/k3s/k3s/k3s.service create mode 100644 recipes-containers/k3s/k3s_git.bb

recipes-containers/k3s/README.md new file +7 -0
new file mode 100644
index 0000000..3fe5ccd
@@ -0,0 +1,30 @@
+# k3s: Lightweight Kubernetes
+
+Rancher's [k3s](https://k3s.io/), available under
+[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides
+lightweight Kubernetes suitable for small/edge devices. There are use cases
+where the
+[installation procedures provided by

Rancher](https://rancher.com/docs/k3s/latest/en/installation/)

recipes-containers/k3s/README.md new file +7 -0
+are not ideal but a bitbake-built version is what is needed. And only a few
+mods to the [k3s source code](https://github.com/rancher/k3s) is needed to
+accomplish that.
+
+## CNI
+
+By default, K3s will run with flannel as the CNI, using VXLAN as the

default
+backend. It is both possible to change the flannel backend and to change from

recipes-containers/k3s/README.md new file +3 -0
+flannel to another CNI.
+
+Please see

<https://rancher.com/docs/k3s/latest/en/installation/network-options/>

recipes-containers/k3s/README.md new file +5 -0
+for further k3s networking details.
+
+## Configure and run a k3s agent
+
+The convenience script `k3s-agent` can be used to set up a k3s agent

(service):

recipes-containers/k3s/README.md new file +7 -0
+
+```shell
+k3s-agent -t <token> -s https://<master>:6443
+```
+
+(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the
+k3s master.)

diff --git
a/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch b/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch new file mode 100644
index 0000000..8205d73
--- /dev/null
+++
b/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch

recipes-containers/k3s/README.md new file +27 -0
@@ -0,0 +1,27 @@
+From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001
+From: Erik Jansson <[email protected]>
+Date: Wed, 16 Oct 2019 15:07:48 +0200
+Subject: [PATCH] Finding host-local in /usr/libexec
+
+Upstream-status: Inappropriate [embedded specific]
+Signed-off-by: <[email protected]>
+---
+ pkg/agent/config/config.go | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go
+index b4296f360a..6af9dab895 100644
+--- a/pkg/agent/config/config.go
++++ b/pkg/agent/config/config.go
+@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) {
+               return nil, err
+       }
+
+-      hostLocal, err := exec.LookPath("host-local")
++      hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local")
+       if err != nil {
+               return nil, errors.Wrapf(err, "failed to find host-local")
+       }
+--
+2.11.0
+
recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24 -0
new file mode 100644
index 0000000..ca434d6
@@ -0,0 +1,24 @@
+{
+  "cniVersion": "0.4.0",
+  "name": "containerd-net",
+  "plugins": [
+    {
+      "type": "bridge",
+      "bridge": "cni0",
+      "isGateway": true,
+      "ipMasq": true,
+      "promiscMode": true,
+      "ipam": {
+        "type": "host-local",
+        "subnet": "10.88.0.0/16",
+        "routes": [
+          { "dst": "0.0.0.0/0" }
+        ]
+      }
+    },
+    {
+      "type": "portmap",
+      "capabilities": {"portMappings": true}
+    }
+  ]
+}
recipes-containers/k3s/k3s/k3s-agent new file +14 -0
new file mode 100755
index 0000000..b6c6cb6
@@ -0,0 +1,103 @@
+#!/bin/sh -eu
+#
+# Copyright (C) 2020 Axis Communications AB
+#
+# SPDX-License-Identifier: Apache-2.0
+
+ENV_CONF=/etc/systemd/system/k3s-agent.service.d/10-env.conf
+
+usage() {
+       echo "
+USAGE:
+    ${0##*/} [OPTIONS]
+OPTIONS:
+    --token value, -t value             Token to use for authentication

[\$K3S_TOKEN]
+ --token-file value Token file to use for
authentication [\$K3S_TOKEN_FILE]

recipes-containers/k3s/k3s/k3s-agent new file +3 -0
+    --server value, -s value            Server to connect to [\$K3S_URL]
+    --node-name value                   Node name [\$K3S_NODE_NAME]
+    --resolv-conf value                 Kubelet resolv.conf file

[\$K3S_RESOLV_CONF]
+ --cluster-secret value Shared secret used to bootstrap
a cluster [\$K3S_CLUSTER_SECRET]

recipes-containers/k3s/k3s/k3s-agent new file +84 -0
+    -h                                  print this
+"
+}
+
+[ $# -gt 0 ] || {
+       usage
+       exit
+}
+
+case $1 in
+       -*)
+               ;;
+       *)
+               usage
+               exit 1
+               ;;
+esac
+
+rm -f $ENV_CONF
+mkdir -p ${ENV_CONF%/*}
+echo [Service] > $ENV_CONF
+
+while getopts "t:s:-:h" opt; do
+       case $opt in
+               h)
+                       usage
+                       exit
+                       ;;
+               t)
+                       VAR_NAME=K3S_TOKEN
+                       ;;
+               s)
+                       VAR_NAME=K3S_URL
+                       ;;
+               -)
+                       [ $# -ge $OPTIND ] || {
+                               usage
+                               exit 1
+                       }
+                       opt=$OPTARG
+                       eval OPTARG='$'$OPTIND
+                       OPTIND=$(($OPTIND + 1))
+                       case $opt in
+                               token)
+                                       VAR_NAME=K3S_TOKEN
+                                       ;;
+                               token-file)
+                                       VAR_NAME=K3S_TOKEN_FILE
+                                       ;;
+                               server)
+                                       VAR_NAME=K3S_URL
+                                       ;;
+                               node-name)
+                                       VAR_NAME=K3S_NODE_NAME
+                                       ;;
+                               resolv-conf)
+                                       VAR_NAME=K3S_RESOLV_CONF
+                                       ;;
+                               cluster-secret)
+                                       VAR_NAME=K3S_CLUSTER_SECRET
+                                       ;;
+                               help)
+                                       usage
+                                       exit
+                                       ;;
+                               *)
+                                       usage
+                                       exit 1
+                                       ;;
+                       esac
+                       ;;
+               *)
+                       usage
+                       exit 1
+                       ;;
+       esac
+    echo Environment=$VAR_NAME=$OPTARG >> $ENV_CONF
+done
+
+chmod 0644 $ENV_CONF
+rm -rf /var/lib/rancher/k3s/agent
+systemctl daemon-reload
+systemctl restart k3s-agent
+systemctl enable k3s-agent.service
recipes-containers/k3s/k3s/k3s-agent.service new file +26 -0
new file mode 100644
index 0000000..9f9016d
@@ -0,0 +1,26 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes Agent
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=control-group
+Delegate=yes
+LimitNOFILE=infinity
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s agent
+ExecStopPost=/usr/local/bin/k3s-clean
+
recipes-containers/k3s/k3s/k3s-clean new file +23 -0
new file mode 100755
index 0000000..8eca918
@@ -0,0 +1,30 @@
+#!/bin/sh -eu
+#
+# Copyright (C) 2020 Axis Communications AB
+#
+# SPDX-License-Identifier: Apache-2.0
+
+do_unmount() {
+       [ $# -eq 2 ] || return
+       local mounts=
+       while read ignore mount ignore; do
+               case $mount in
+                       $1/*|$2/*)
+                               mounts="$mount $mounts"
+                               ;;
+               esac
+       done </proc/self/mounts
+       [ -z "$mounts" ] || umount $mounts
+}
+
+do_unmount /run/k3s /var/lib/rancher/k3s
+
+# The lines below come from install.sh's create_killall() function:
+ip link show 2>/dev/null | grep 'master cni0' | while read ignore iface

ignore; do

recipes-containers/k3s/k3s/k3s-clean new file +7 -0
+    iface=${iface%%@*}
+    [ -z "$iface" ] || ip link delete $iface
+done
+
+ip link delete cni0
+ip link delete flannel.1
+rm -rf /var/lib/cni/
recipes-containers/k3s/k3s/k3s.service new file +15 -0
new file mode 100644
index 0000000..34c7a80
@@ -0,0 +1,27 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=process
+Delegate=yes
+# Having non-zero Limit*s causes performance problems due to accounting

overhead
+# in the kernel. We recommend using cgroups to do container-local accounting.

recipes-containers/k3s/k3s/k3s.service new file +11 -0
+LimitNOFILE=1048576
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s server
+
recipes-containers/k3s/k3s_git.bb new file +2 -0
new file mode 100644
index 0000000..cfc2c64
@@ -0,0 +1,75 @@
+SUMMARY = "Production-Grade Container Scheduling and Management"
+DESCRIPTION = "Lightweight Kubernetes, intended to be a fully compliant

Kubernetes."

recipes-containers/k3s/k3s_git.bb new file +3 -0
+HOMEPAGE = "https://k3s.io/"
+LICENSE = "Apache-2.0"
+LIC_FILES_CHKSUM =

"file://${S}/src/import/LICENSE;md5=2ee41112a44fe7014dce33e26468ba93"

recipes-containers/k3s/k3s_git.bb new file +9 -0
+PV = "v1.18.9+k3s1-dirty"
+
+SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.18;name=k3s \
+           file://k3s.service \
+           file://k3s-agent.service \
+           file://k3s-agent \
+           file://k3s-clean \
+           file://cni-containerd-net.conf \
+

file://0001-Finding-host-local-in-usr-libexec.patch;patchdir=src/import \

recipes-containers/k3s/k3s_git.bb new file +13 -0
+          "
+SRC_URI[k3s.md5sum] = "363d3a08dc0b72ba6e6577964f6e94a5"
+SRCREV_k3s = "630bebf94b9dce6b8cd3d402644ed023b3af8f90"
+
+inherit go
+inherit goarch
+inherit systemd
+
+PACKAGECONFIG = ""
+PACKAGECONFIG[upx] = ",,upx-native"
+GO_IMPORT = "import"
+GO_BUILD_LDFLAGS = "-X github.com/rancher/k3s/pkg/version.Version=${PV} \
+                    -X

github.com/rancher/k3s/pkg/version.GitCommit=${@d.getVar('SRCREV_k3s', d, 1)[:8]} \

recipes-containers/k3s/k3s_git.bb new file +6 -0
+                    -w -s \
+                   "
+BIN_PREFIX ?= "${exec_prefix}/local"
+
+do_compile() {
+        export

GOPATH="${S}/src/import/.gopath:${S}/src/import/vendor:${STAGING_DIR_TARGET}/${prefix}/local/go"

recipes-containers/k3s/k3s_git.bb new file +12 -1
+        export CGO_ENABLED="1"
+        export GOFLAGS="-mod=vendor"
+        cd ${S}/src/import
+        ${GO} build -tags providerless -ldflags "${GO_BUILD_LDFLAGS}"
-o ./dist/artifacts/k3s ./cmd/server/main.go
+        # Use UPX if it is enabled (and thus exists) to compress binary
+        if command -v upx > /dev/null 2>&1; then
+                upx -9 ./dist/artifacts/k3s
+        fi
+}
+do_install() {
+        install -d "${D}${BIN_PREFIX}/bin"
+        install -m 755 "${S}/src/import/dist/artifacts/k3s"

"${D}${BIN_PREFIX}/bin"

recipes-containers/k3s/k3s_git.bb new file +5 -0
+        ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/crictl"
+        ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/ctr"
+        ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/kubectl"
+        install -m 755 "${WORKDIR}/k3s-clean" "${D}${BIN_PREFIX}/bin"
+        install -D -m 0644 "${WORKDIR}/cni-containerd-net.conf"

"${D}/${sysconfdir}/cni/net.d/10-containerd-net.conf"
+ if
${@bb.utils.contains('DISTRO_FEATURES','systemd','true','false',d)}; then
+ install -D -m 0644 "${WORKDIR}/k3s.service"
"${D}${systemd_system_unitdir}/k3s.service"
+ install -D -m 0644 "${WORKDIR}/k3s-agent.service"
"${D}${systemd_system_unitdir}/k3s-agent.service"
+ sed -i
"s#\(Exec\)\(.*\)=\(.*\)\(k3s\)#\1\2=${BIN_PREFIX}/bin/\4#g" "${D}${systemd_system_unitdir}/k3s.service" "${D}${systemd_system_unitdir}/k3s-agent.service"
+ install -m 755 "${WORKDIR}/k3s-agent"
"${D}${BIN_PREFIX}/bin"

recipes-containers/k3s/k3s_git.bb new file +6 -0
+        fi
+}
+
+PACKAGES =+ "${PN}-server ${PN}-agent"
+
+SYSTEMD_PACKAGES =

"${@bb.utils.contains('DISTRO_FEATURES','systemd','${PN}-server ${PN}-agent','',d)}"
+SYSTEMD_SERVICE_${PN}-server = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s.service','',d)}" +SYSTEMD_SERVICE_${PN}-agent = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s-agent.service','',d)}"

recipes-containers/k3s/k3s_git.bb new file +5 -0
+SYSTEMD_AUTO_ENABLE_${PN}-agent = "disable"
+
+FILES_${PN}-agent = "${BIN_PREFIX}/bin/k3s-agent"
+
+RDEPENDS_${PN} = "cni conntrack-tools coreutils findutils iproute2

ipset virtual/containerd"

recipes-containers/k3s/k3s_git.bb new file +7 -1
+RDEPENDS_${PN}-server = "${PN}"
+RDEPENDS_${PN}-agent = "${PN}"
+
+RCONFLICTS_${PN} = "kubectl"
+
+INHIBIT_PACKAGE_STRIP = "1"
+INSANE_SKIP_${PN} += "ldflags already-stripped"
--

2.20.1

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#20 lore raw
Quoted 3 lines · Bruce Ashfield

Can you re-send it with a "v4" in the subject ? That will hopefully let it escape gmail's conversation filter.

Sure! No problem.

BR,

/Joakim

#21 lore raw

PATCH v4 1/1 Adding k3s recipe

In reply to Bruce Ashfield
Change-Id Id1c52727593bc5ea8d0cd2de192faa44304d7a45
---
 recipes-containers/k3s/README.md              |  30 +++++
 ...01-Finding-host-local-in-usr-libexec.patch |  27 +++++
 .../k3s/k3s/cni-containerd-net.conf           |  24 ++++
 recipes-containers/k3s/k3s/k3s-agent          | 103 ++++++++++++++++++
 recipes-containers/k3s/k3s/k3s-agent.service  |  26 +++++
 recipes-containers/k3s/k3s/k3s-clean          |  30 +++++
 recipes-containers/k3s/k3s/k3s.service        |  27 +++++
 recipes-containers/k3s/k3s_git.bb             |  75 +++++++++++++
 8 files changed, 342 insertions(+)
 create mode 100644 recipes-containers/k3s/README.md
 create mode 100644 
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch
 create mode 100644 recipes-containers/k3s/k3s/cni-containerd-net.conf
 create mode 100755 recipes-containers/k3s/k3s/k3s-agent
 create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service
 create mode 100755 recipes-containers/k3s/k3s/k3s-clean
 create mode 100644 recipes-containers/k3s/k3s/k3s.service
 create mode 100644 recipes-containers/k3s/k3s_git.bb
recipes-containers/k3s/README.md new file +30 -0
new file mode 100644
index 0000000..3fe5ccd
@@ -0,0 +1,30 @@
+# k3s: Lightweight Kubernetes
+
+Rancher's [k3s](https://k3s.io/), available under
+[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides
+lightweight Kubernetes suitable for small/edge devices. There are use cases
+where the
+[installation procedures provided by Rancher](https://rancher.com/docs/k3s/latest/en/installation/)
+are not ideal but a bitbake-built version is what is needed. And only a few
+mods to the [k3s source code](https://github.com/rancher/k3s) is needed to
+accomplish that.
+
+## CNI
+
+By default, K3s will run with flannel as the CNI, using VXLAN as the default
+backend. It is both possible to change the flannel backend and to change from
+flannel to another CNI.
+
+Please see <https://rancher.com/docs/k3s/latest/en/installation/network-options/>
+for further k3s networking details.
+
+## Configure and run a k3s agent
+
+The convenience script `k3s-agent` can be used to set up a k3s agent (service):
+
+```shell
+k3s-agent -t <token> -s https://<master>:6443
+```
+
+(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the
+k3s master.)
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch new file +27 -0
new file mode 100644
index 0000000..8205d73
@@ -0,0 +1,27 @@
+From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001
+From: Erik Jansson <[email protected]>
+Date: Wed, 16 Oct 2019 15:07:48 +0200
+Subject: [PATCH] Finding host-local in /usr/libexec
+
+Upstream-status: Inappropriate [embedded specific]
+Signed-off-by: <[email protected]>
+---
+ pkg/agent/config/config.go | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go
+index b4296f360a..6af9dab895 100644
+--- a/pkg/agent/config/config.go
++++ b/pkg/agent/config/config.go
+@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) {
+ 		return nil, err
+ 	}
+
+-	hostLocal, err := exec.LookPath("host-local")
++	hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local")
+ 	if err != nil {
+ 		return nil, errors.Wrapf(err, "failed to find host-local")
+ 	}
+--
+2.11.0
+
recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24 -0
new file mode 100644
index 0000000..ca434d6
@@ -0,0 +1,24 @@
+{
+  "cniVersion": "0.4.0",
+  "name": "containerd-net",
+  "plugins": [
+    {
+      "type": "bridge",
+      "bridge": "cni0",
+      "isGateway": true,
+      "ipMasq": true,
+      "promiscMode": true,
+      "ipam": {
+        "type": "host-local",
+        "subnet": "10.88.0.0/16",
+        "routes": [
+          { "dst": "0.0.0.0/0" }
+        ]
+      }
+    },
+    {
+      "type": "portmap",
+      "capabilities": {"portMappings": true}
+    }
+  ]
+}
recipes-containers/k3s/k3s/k3s-agent new file +103 -0

103 lines not on this page · raw message

recipes-containers/k3s/k3s/k3s-agent.service new file +26 -0
new file mode 100644
index 0000000..9f9016d
@@ -0,0 +1,26 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes Agent
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=control-group
+Delegate=yes
+LimitNOFILE=infinity
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s agent
+ExecStopPost=/usr/local/bin/k3s-clean
+
recipes-containers/k3s/k3s/k3s-clean new file +30 -0
new file mode 100755
index 0000000..8eca918
@@ -0,0 +1,30 @@
+#!/bin/sh -eu
+#
+# Copyright (C) 2020 Axis Communications AB
+#
+# SPDX-License-Identifier: Apache-2.0
+
+do_unmount() {
+	[ $# -eq 2 ] || return
+	local mounts=
+	while read ignore mount ignore; do
+		case $mount in
+			$1/*|$2/*)
+				mounts="$mount $mounts"
+				;;
+		esac
+	done </proc/self/mounts
+	[ -z "$mounts" ] || umount $mounts
+}
+
+do_unmount /run/k3s /var/lib/rancher/k3s
+
+# The lines below come from install.sh's create_killall() function:
+ip link show 2>/dev/null | grep 'master cni0' | while read ignore iface ignore; do
+    iface=${iface%%@*}
+    [ -z "$iface" ] || ip link delete $iface
+done
+
+ip link delete cni0
+ip link delete flannel.1
+rm -rf /var/lib/cni/
recipes-containers/k3s/k3s/k3s.service new file +27 -0
new file mode 100644
index 0000000..34c7a80
@@ -0,0 +1,27 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=process
+Delegate=yes
+# Having non-zero Limit*s causes performance problems due to accounting overhead
+# in the kernel. We recommend using cgroups to do container-local accounting.
+LimitNOFILE=1048576
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s server
+
recipes-containers/k3s/k3s_git.bb new file +75 -0

75 lines not on this page · raw message

Signature · 1 line
2.20.1
#22 lore raw

I just merged 3 patches from the mailing list, using my standard workflow (save mbox, apply, buid, push).

But with your patch, saved the same way. I get:

------------

build [/home/bruc...ualization]> git am -s k3.mbox
Applying: Adding k3s recipe
.git/rebase-apply/patch:35: trailing whitespace. [installation procedures provided by
error: corrupt patch at line 36
Patch failed at 0001 Adding k3s recipe
hint: Use 'git am --show-current-patch' to see the failed patch When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort".

----------

I've tried hacking the patch by hand to apply, but it is just making things worse. Is there another way
you can send the patch ?

Bruce

Quoted 461 lines · Joakim Roubert
Change-Id Id1c52727593bc5ea8d0cd2de192faa44304d7a45
---
  recipes-containers/k3s/README.md              |  30 +++++
  ...01-Finding-host-local-in-usr-libexec.patch |  27 +++++
  .../k3s/k3s/cni-containerd-net.conf           |  24 ++++
  recipes-containers/k3s/k3s/k3s-agent          | 103 ++++++++++++++++++
  recipes-containers/k3s/k3s/k3s-agent.service  |  26 +++++
  recipes-containers/k3s/k3s/k3s-clean          |  30 +++++
  recipes-containers/k3s/k3s/k3s.service        |  27 +++++
  recipes-containers/k3s/k3s_git.bb             |  75 +++++++++++++
  8 files changed, 342 insertions(+)

create mode 100644 recipes-containers/k3s/README.md
create mode 100644
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch
create mode 100644 recipes-containers/k3s/k3s/cni-containerd-net.conf create mode 100755 recipes-containers/k3s/k3s/k3s-agent create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service create mode 100755 recipes-containers/k3s/k3s/k3s-clean create mode 100644 recipes-containers/k3s/k3s/k3s.service create mode 100644 recipes-containers/k3s/k3s_git.bb

recipes-containers/k3s/README.md new file +7 -0
new file mode 100644
index 0000000..3fe5ccd
@@ -0,0 +1,30 @@
+# k3s: Lightweight Kubernetes
+
+Rancher's [k3s](https://k3s.io/), available under
+[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides
+lightweight Kubernetes suitable for small/edge devices. There are use cases
+where the
+[installation procedures provided by

Rancher](https://rancher.com/docs/k3s/latest/en/installation/)

recipes-containers/k3s/README.md new file +7 -0
+are not ideal but a bitbake-built version is what is needed. And only a few
+mods to the [k3s source code](https://github.com/rancher/k3s) is needed to
+accomplish that.
+
+## CNI
+
+By default, K3s will run with flannel as the CNI, using VXLAN as the

default
+backend. It is both possible to change the flannel backend and to change from

recipes-containers/k3s/README.md new file +3 -0
+flannel to another CNI.
+
+Please see

<https://rancher.com/docs/k3s/latest/en/installation/network-options/>

recipes-containers/k3s/README.md new file +5 -0
+for further k3s networking details.
+
+## Configure and run a k3s agent
+
+The convenience script `k3s-agent` can be used to set up a k3s agent

(service):

recipes-containers/k3s/README.md new file +7 -0
+
+```shell
+k3s-agent -t <token> -s https://<master>:6443
+```
+
+(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the
+k3s master.)

diff --git
a/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch b/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch new file mode 100644
index 0000000..8205d73
--- /dev/null
+++
b/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch

recipes-containers/k3s/README.md new file +27 -0
@@ -0,0 +1,27 @@
+From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001
+From: Erik Jansson <[email protected]>
+Date: Wed, 16 Oct 2019 15:07:48 +0200
+Subject: [PATCH] Finding host-local in /usr/libexec
+
+Upstream-status: Inappropriate [embedded specific]
+Signed-off-by: <[email protected]>
+---
+ pkg/agent/config/config.go | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go
+index b4296f360a..6af9dab895 100644
+--- a/pkg/agent/config/config.go
++++ b/pkg/agent/config/config.go
+@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) {
+               return nil, err
+       }
+
+-      hostLocal, err := exec.LookPath("host-local")
++      hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local")
+       if err != nil {
+               return nil, errors.Wrapf(err, "failed to find host-local")
+       }
+--
+2.11.0
+
recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24 -0
new file mode 100644
index 0000000..ca434d6
@@ -0,0 +1,24 @@
+{
+  "cniVersion": "0.4.0",
+  "name": "containerd-net",
+  "plugins": [
+    {
+      "type": "bridge",
+      "bridge": "cni0",
+      "isGateway": true,
+      "ipMasq": true,
+      "promiscMode": true,
+      "ipam": {
+        "type": "host-local",
+        "subnet": "10.88.0.0/16",
+        "routes": [
+          { "dst": "0.0.0.0/0" }
+        ]
+      }
+    },
+    {
+      "type": "portmap",
+      "capabilities": {"portMappings": true}
+    }
+  ]
+}
recipes-containers/k3s/k3s/k3s-agent new file +14 -0
new file mode 100755
index 0000000..b6c6cb6
@@ -0,0 +1,103 @@
+#!/bin/sh -eu
+#
+# Copyright (C) 2020 Axis Communications AB
+#
+# SPDX-License-Identifier: Apache-2.0
+
+ENV_CONF=/etc/systemd/system/k3s-agent.service.d/10-env.conf
+
+usage() {
+       echo "
+USAGE:
+    ${0##*/} [OPTIONS]
+OPTIONS:
+    --token value, -t value             Token to use for authentication

[\$K3S_TOKEN]
+ --token-file value Token file to use for
authentication [\$K3S_TOKEN_FILE]

recipes-containers/k3s/k3s/k3s-agent new file +3 -0
+    --server value, -s value            Server to connect to [\$K3S_URL]
+    --node-name value                   Node name [\$K3S_NODE_NAME]
+    --resolv-conf value                 Kubelet resolv.conf file

[\$K3S_RESOLV_CONF]
+ --cluster-secret value Shared secret used to bootstrap
a cluster [\$K3S_CLUSTER_SECRET]

recipes-containers/k3s/k3s/k3s-agent new file +84 -0
+    -h                                  print this
+"
+}
+
+[ $# -gt 0 ] || {
+       usage
+       exit
+}
+
+case $1 in
+       -*)
+               ;;
+       *)
+               usage
+               exit 1
+               ;;
+esac
+
+rm -f $ENV_CONF
+mkdir -p ${ENV_CONF%/*}
+echo [Service] > $ENV_CONF
+
+while getopts "t:s:-:h" opt; do
+       case $opt in
+               h)
+                       usage
+                       exit
+                       ;;
+               t)
+                       VAR_NAME=K3S_TOKEN
+                       ;;
+               s)
+                       VAR_NAME=K3S_URL
+                       ;;
+               -)
+                       [ $# -ge $OPTIND ] || {
+                               usage
+                               exit 1
+                       }
+                       opt=$OPTARG
+                       eval OPTARG='$'$OPTIND
+                       OPTIND=$(($OPTIND + 1))
+                       case $opt in
+                               token)
+                                       VAR_NAME=K3S_TOKEN
+                                       ;;
+                               token-file)
+                                       VAR_NAME=K3S_TOKEN_FILE
+                                       ;;
+                               server)
+                                       VAR_NAME=K3S_URL
+                                       ;;
+                               node-name)
+                                       VAR_NAME=K3S_NODE_NAME
+                                       ;;
+                               resolv-conf)
+                                       VAR_NAME=K3S_RESOLV_CONF
+                                       ;;
+                               cluster-secret)
+                                       VAR_NAME=K3S_CLUSTER_SECRET
+                                       ;;
+                               help)
+                                       usage
+                                       exit
+                                       ;;
+                               *)
+                                       usage
+                                       exit 1
+                                       ;;
+                       esac
+                       ;;
+               *)
+                       usage
+                       exit 1
+                       ;;
+       esac
+    echo Environment=$VAR_NAME=$OPTARG >> $ENV_CONF
+done
+
+chmod 0644 $ENV_CONF
+rm -rf /var/lib/rancher/k3s/agent
+systemctl daemon-reload
+systemctl restart k3s-agent
+systemctl enable k3s-agent.service
recipes-containers/k3s/k3s/k3s-agent.service new file +26 -0
new file mode 100644
index 0000000..9f9016d
@@ -0,0 +1,26 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes Agent
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=control-group
+Delegate=yes
+LimitNOFILE=infinity
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s agent
+ExecStopPost=/usr/local/bin/k3s-clean
+
recipes-containers/k3s/k3s/k3s-clean new file +23 -0
new file mode 100755
index 0000000..8eca918
@@ -0,0 +1,30 @@
+#!/bin/sh -eu
+#
+# Copyright (C) 2020 Axis Communications AB
+#
+# SPDX-License-Identifier: Apache-2.0
+
+do_unmount() {
+       [ $# -eq 2 ] || return
+       local mounts=
+       while read ignore mount ignore; do
+               case $mount in
+                       $1/*|$2/*)
+                               mounts="$mount $mounts"
+                               ;;
+               esac
+       done </proc/self/mounts
+       [ -z "$mounts" ] || umount $mounts
+}
+
+do_unmount /run/k3s /var/lib/rancher/k3s
+
+# The lines below come from install.sh's create_killall() function:
+ip link show 2>/dev/null | grep 'master cni0' | while read ignore iface

ignore; do

recipes-containers/k3s/k3s/k3s-clean new file +7 -0
+    iface=${iface%%@*}
+    [ -z "$iface" ] || ip link delete $iface
+done
+
+ip link delete cni0
+ip link delete flannel.1
+rm -rf /var/lib/cni/
recipes-containers/k3s/k3s/k3s.service new file +15 -0
new file mode 100644
index 0000000..34c7a80
@@ -0,0 +1,27 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=process
+Delegate=yes
+# Having non-zero Limit*s causes performance problems due to accounting

overhead
+# in the kernel. We recommend using cgroups to do container-local accounting.

recipes-containers/k3s/k3s/k3s.service new file +11 -0
+LimitNOFILE=1048576
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s server
+
recipes-containers/k3s/k3s_git.bb new file +2 -0
new file mode 100644
index 0000000..cfc2c64
@@ -0,0 +1,75 @@
+SUMMARY = "Production-Grade Container Scheduling and Management"
+DESCRIPTION = "Lightweight Kubernetes, intended to be a fully compliant

Kubernetes."

recipes-containers/k3s/k3s_git.bb new file +3 -0
+HOMEPAGE = "https://k3s.io/"
+LICENSE = "Apache-2.0"
+LIC_FILES_CHKSUM =

"file://${S}/src/import/LICENSE;md5=2ee41112a44fe7014dce33e26468ba93"

recipes-containers/k3s/k3s_git.bb new file +9 -0
+PV = "v1.18.9+k3s1-dirty"
+
+SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.18;name=k3s \
+           file://k3s.service \
+           file://k3s-agent.service \
+           file://k3s-agent \
+           file://k3s-clean \
+           file://cni-containerd-net.conf \
+

file://0001-Finding-host-local-in-usr-libexec.patch;patchdir=src/import \

recipes-containers/k3s/k3s_git.bb new file +13 -0
+          "
+SRC_URI[k3s.md5sum] = "363d3a08dc0b72ba6e6577964f6e94a5"
+SRCREV_k3s = "630bebf94b9dce6b8cd3d402644ed023b3af8f90"
+
+inherit go
+inherit goarch
+inherit systemd
+
+PACKAGECONFIG = ""
+PACKAGECONFIG[upx] = ",,upx-native"
+GO_IMPORT = "import"
+GO_BUILD_LDFLAGS = "-X github.com/rancher/k3s/pkg/version.Version=${PV} \
+                    -X

github.com/rancher/k3s/pkg/version.GitCommit=${@d.getVar('SRCREV_k3s', d, 1)[:8]} \

recipes-containers/k3s/k3s_git.bb new file +6 -0
+                    -w -s \
+                   "
+BIN_PREFIX ?= "${exec_prefix}/local"
+
+do_compile() {
+        export

GOPATH="${S}/src/import/.gopath:${S}/src/import/vendor:${STAGING_DIR_TARGET}/${prefix}/local/go"

recipes-containers/k3s/k3s_git.bb new file +12 -1
+        export CGO_ENABLED="1"
+        export GOFLAGS="-mod=vendor"
+        cd ${S}/src/import
+        ${GO} build -tags providerless -ldflags "${GO_BUILD_LDFLAGS}"
-o ./dist/artifacts/k3s ./cmd/server/main.go
+        # Use UPX if it is enabled (and thus exists) to compress binary
+        if command -v upx > /dev/null 2>&1; then
+                upx -9 ./dist/artifacts/k3s
+        fi
+}
+do_install() {
+        install -d "${D}${BIN_PREFIX}/bin"
+        install -m 755 "${S}/src/import/dist/artifacts/k3s"

"${D}${BIN_PREFIX}/bin"

recipes-containers/k3s/k3s_git.bb new file +5 -0
+        ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/crictl"
+        ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/ctr"
+        ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/kubectl"
+        install -m 755 "${WORKDIR}/k3s-clean" "${D}${BIN_PREFIX}/bin"
+        install -D -m 0644 "${WORKDIR}/cni-containerd-net.conf"

"${D}/${sysconfdir}/cni/net.d/10-containerd-net.conf"
+ if
${@bb.utils.contains('DISTRO_FEATURES','systemd','true','false',d)}; then
+ install -D -m 0644 "${WORKDIR}/k3s.service"
"${D}${systemd_system_unitdir}/k3s.service"
+ install -D -m 0644 "${WORKDIR}/k3s-agent.service"
"${D}${systemd_system_unitdir}/k3s-agent.service"
+ sed -i
"s#\(Exec\)\(.*\)=\(.*\)\(k3s\)#\1\2=${BIN_PREFIX}/bin/\4#g" "${D}${systemd_system_unitdir}/k3s.service" "${D}${systemd_system_unitdir}/k3s-agent.service"
+ install -m 755 "${WORKDIR}/k3s-agent"
"${D}${BIN_PREFIX}/bin"

recipes-containers/k3s/k3s_git.bb new file +6 -0
+        fi
+}
+
+PACKAGES =+ "${PN}-server ${PN}-agent"
+
+SYSTEMD_PACKAGES =

"${@bb.utils.contains('DISTRO_FEATURES','systemd','${PN}-server ${PN}-agent','',d)}"
+SYSTEMD_SERVICE_${PN}-server = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s.service','',d)}" +SYSTEMD_SERVICE_${PN}-agent = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s-agent.service','',d)}"

recipes-containers/k3s/k3s_git.bb new file +5 -0
+SYSTEMD_AUTO_ENABLE_${PN}-agent = "disable"
+
+FILES_${PN}-agent = "${BIN_PREFIX}/bin/k3s-agent"
+
+RDEPENDS_${PN} = "cni conntrack-tools coreutils findutils iproute2

ipset virtual/containerd"

recipes-containers/k3s/k3s_git.bb new file +7 -1
+RDEPENDS_${PN}-server = "${PN}"
+RDEPENDS_${PN}-agent = "${PN}"
+
+RCONFLICTS_${PN} = "kubectl"
+
+INHIBIT_PACKAGE_STRIP = "1"
+INSANE_SKIP_${PN} += "ldflags already-stripped"
--

2.20.1

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#23 lore raw

PATCH v5 1/1 Adding k3s recipe

Change-Id Id1c52727593bc5ea8d0cd2de192faa44304d7a45
---
 recipes-containers/k3s/README.md              |  30 +++++
 ...01-Finding-host-local-in-usr-libexec.patch |  27 +++++
 .../k3s/k3s/cni-containerd-net.conf           |  24 ++++
 recipes-containers/k3s/k3s/k3s-agent          | 103 ++++++++++++++++++
 recipes-containers/k3s/k3s/k3s-agent.service  |  26 +++++
 recipes-containers/k3s/k3s/k3s-clean          |  30 +++++
 recipes-containers/k3s/k3s/k3s.service        |  27 +++++
 recipes-containers/k3s/k3s_git.bb             |  75 +++++++++++++
 8 files changed, 342 insertions(+)
 create mode 100644 recipes-containers/k3s/README.md
 create mode 100644 recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch
 create mode 100644 recipes-containers/k3s/k3s/cni-containerd-net.conf
 create mode 100755 recipes-containers/k3s/k3s/k3s-agent
 create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service
 create mode 100755 recipes-containers/k3s/k3s/k3s-clean
 create mode 100644 recipes-containers/k3s/k3s/k3s.service
 create mode 100644 recipes-containers/k3s/k3s_git.bb
recipes-containers/k3s/README.md new file +30 -0
new file mode 100644
index 0000000..3fe5ccd
@@ -0,0 +1,30 @@
+# k3s: Lightweight Kubernetes
+
+Rancher's [k3s](https://k3s.io/), available under
+[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides
+lightweight Kubernetes suitable for small/edge devices. There are use cases
+where the
+[installation procedures provided by Rancher](https://rancher.com/docs/k3s/latest/en/installation/)
+are not ideal but a bitbake-built version is what is needed. And only a few
+mods to the [k3s source code](https://github.com/rancher/k3s) is needed to
+accomplish that.
+
+## CNI
+
+By default, K3s will run with flannel as the CNI, using VXLAN as the default
+backend. It is both possible to change the flannel backend and to change from
+flannel to another CNI.
+
+Please see <https://rancher.com/docs/k3s/latest/en/installation/network-options/>
+for further k3s networking details.
+
+## Configure and run a k3s agent
+
+The convenience script `k3s-agent` can be used to set up a k3s agent (service):
+
+```shell
+k3s-agent -t <token> -s https://<master>:6443
+```
+
+(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the
+k3s master.)
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch new file +27 -0
new file mode 100644
index 0000000..8205d73
@@ -0,0 +1,27 @@
+From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001
+From: Erik Jansson <[email protected]>
+Date: Wed, 16 Oct 2019 15:07:48 +0200
+Subject: [PATCH] Finding host-local in /usr/libexec
+
+Upstream-status: Inappropriate [embedded specific]
+Signed-off-by: <[email protected]>
+---
+ pkg/agent/config/config.go | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go
+index b4296f360a..6af9dab895 100644
+--- a/pkg/agent/config/config.go
++++ b/pkg/agent/config/config.go
+@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) {
+ 		return nil, err
+ 	}
+ 
+-	hostLocal, err := exec.LookPath("host-local")
++	hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local")
+ 	if err != nil {
+ 		return nil, errors.Wrapf(err, "failed to find host-local")
+ 	}
+-- 
+2.11.0
+
recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24 -0
new file mode 100644
index 0000000..ca434d6
@@ -0,0 +1,24 @@
+{
+  "cniVersion": "0.4.0",
+  "name": "containerd-net",
+  "plugins": [
+    {
+      "type": "bridge",
+      "bridge": "cni0",
+      "isGateway": true,
+      "ipMasq": true,
+      "promiscMode": true,
+      "ipam": {
+        "type": "host-local",
+        "subnet": "10.88.0.0/16",
+        "routes": [
+          { "dst": "0.0.0.0/0" }
+        ]
+      }
+    },
+    {
+      "type": "portmap",
+      "capabilities": {"portMappings": true}
+    }
+  ]
+}
recipes-containers/k3s/k3s/k3s-agent new file +103 -0

103 lines not on this page · raw message

recipes-containers/k3s/k3s/k3s-agent.service new file +26 -0
new file mode 100644
index 0000000..9f9016d
@@ -0,0 +1,26 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes Agent
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=control-group
+Delegate=yes
+LimitNOFILE=infinity
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s agent
+ExecStopPost=/usr/local/bin/k3s-clean
+
recipes-containers/k3s/k3s/k3s-clean new file +30 -0
new file mode 100755
index 0000000..8eca918
@@ -0,0 +1,30 @@
+#!/bin/sh -eu
+#
+# Copyright (C) 2020 Axis Communications AB
+#
+# SPDX-License-Identifier: Apache-2.0
+
+do_unmount() {
+	[ $# -eq 2 ] || return
+	local mounts=
+	while read ignore mount ignore; do
+		case $mount in
+			$1/*|$2/*)
+				mounts="$mount $mounts"
+				;;
+		esac
+	done </proc/self/mounts
+	[ -z "$mounts" ] || umount $mounts
+}
+
+do_unmount /run/k3s /var/lib/rancher/k3s
+
+# The lines below come from install.sh's create_killall() function:
+ip link show 2>/dev/null | grep 'master cni0' | while read ignore iface ignore; do
+    iface=${iface%%@*}
+    [ -z "$iface" ] || ip link delete $iface
+done
+
+ip link delete cni0
+ip link delete flannel.1
+rm -rf /var/lib/cni/
recipes-containers/k3s/k3s/k3s.service new file +27 -0
new file mode 100644
index 0000000..34c7a80
@@ -0,0 +1,27 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=process
+Delegate=yes
+# Having non-zero Limit*s causes performance problems due to accounting overhead
+# in the kernel. We recommend using cgroups to do container-local accounting.
+LimitNOFILE=1048576
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s server
+
recipes-containers/k3s/k3s_git.bb new file +75 -0

75 lines not on this page · raw message

Signature · 1 line
2.20.1
#24 lore raw

Ha!!!!

This applies.

I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.

Bruce

In message: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 418 lines · on 20/10/2020 Joakim Roubert wrote:
Change-Id Id1c52727593bc5ea8d0cd2de192faa44304d7a45
---
 recipes-containers/k3s/README.md              |  30 +++++
 ...01-Finding-host-local-in-usr-libexec.patch |  27 +++++
 .../k3s/k3s/cni-containerd-net.conf           |  24 ++++
 recipes-containers/k3s/k3s/k3s-agent          | 103 ++++++++++++++++++
 recipes-containers/k3s/k3s/k3s-agent.service  |  26 +++++
 recipes-containers/k3s/k3s/k3s-clean          |  30 +++++
 recipes-containers/k3s/k3s/k3s.service        |  27 +++++
 recipes-containers/k3s/k3s_git.bb             |  75 +++++++++++++
 8 files changed, 342 insertions(+)
 create mode 100644 recipes-containers/k3s/README.md
 create mode 100644 recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch
 create mode 100644 recipes-containers/k3s/k3s/cni-containerd-net.conf
 create mode 100755 recipes-containers/k3s/k3s/k3s-agent
 create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service
 create mode 100755 recipes-containers/k3s/k3s/k3s-clean
 create mode 100644 recipes-containers/k3s/k3s/k3s.service
 create mode 100644 recipes-containers/k3s/k3s_git.bb
recipes-containers/k3s/README.md new file +30 -0
new file mode 100644
index 0000000..3fe5ccd
@@ -0,0 +1,30 @@
+# k3s: Lightweight Kubernetes
+
+Rancher's [k3s](https://k3s.io/), available under
+[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides
+lightweight Kubernetes suitable for small/edge devices. There are use cases
+where the
+[installation procedures provided by Rancher](https://rancher.com/docs/k3s/latest/en/installation/)
+are not ideal but a bitbake-built version is what is needed. And only a few
+mods to the [k3s source code](https://github.com/rancher/k3s) is needed to
+accomplish that.
+
+## CNI
+
+By default, K3s will run with flannel as the CNI, using VXLAN as the default
+backend. It is both possible to change the flannel backend and to change from
+flannel to another CNI.
+
+Please see <https://rancher.com/docs/k3s/latest/en/installation/network-options/>
+for further k3s networking details.
+
+## Configure and run a k3s agent
+
+The convenience script `k3s-agent` can be used to set up a k3s agent (service):
+
+```shell
+k3s-agent -t <token> -s https://<master>:6443
+```
+
+(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the
+k3s master.)
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch new file +27 -0
new file mode 100644
index 0000000..8205d73
@@ -0,0 +1,27 @@
+From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001
+From: Erik Jansson <[email protected]>
+Date: Wed, 16 Oct 2019 15:07:48 +0200
+Subject: [PATCH] Finding host-local in /usr/libexec
+
+Upstream-status: Inappropriate [embedded specific]
+Signed-off-by: <[email protected]>
+---
+ pkg/agent/config/config.go | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go
+index b4296f360a..6af9dab895 100644
+--- a/pkg/agent/config/config.go
++++ b/pkg/agent/config/config.go
+@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) {
+ 		return nil, err
+ 	}
+ 
+-	hostLocal, err := exec.LookPath("host-local")
++	hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local")
+ 	if err != nil {
+ 		return nil, errors.Wrapf(err, "failed to find host-local")
+ 	}
+-- 
+2.11.0
+
recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24 -0
new file mode 100644
index 0000000..ca434d6
@@ -0,0 +1,24 @@
+{
+  "cniVersion": "0.4.0",
+  "name": "containerd-net",
+  "plugins": [
+    {
+      "type": "bridge",
+      "bridge": "cni0",
+      "isGateway": true,
+      "ipMasq": true,
+      "promiscMode": true,
+      "ipam": {
+        "type": "host-local",
+        "subnet": "10.88.0.0/16",
+        "routes": [
+          { "dst": "0.0.0.0/0" }
+        ]
+      }
+    },
+    {
+      "type": "portmap",
+      "capabilities": {"portMappings": true}
+    }
+  ]
+}
recipes-containers/k3s/k3s/k3s-agent new file +103 -0
new file mode 100755
index 0000000..b6c6cb6
@@ -0,0 +1,103 @@
+#!/bin/sh -eu
+#
+# Copyright (C) 2020 Axis Communications AB
+#
+# SPDX-License-Identifier: Apache-2.0
+
+ENV_CONF=/etc/systemd/system/k3s-agent.service.d/10-env.conf
+
+usage() {
+	echo "
+USAGE:
+    ${0##*/} [OPTIONS]
+OPTIONS:
+    --token value, -t value             Token to use for authentication [\$K3S_TOKEN]
+    --token-file value                  Token file to use for authentication [\$K3S_TOKEN_FILE]
+    --server value, -s value            Server to connect to [\$K3S_URL]
+    --node-name value                   Node name [\$K3S_NODE_NAME]
+    --resolv-conf value                 Kubelet resolv.conf file [\$K3S_RESOLV_CONF]
+    --cluster-secret value              Shared secret used to bootstrap a cluster [\$K3S_CLUSTER_SECRET]
+    -h                                  print this
+"
+}
+
+[ $# -gt 0 ] || {
+	usage
+	exit
+}
+
+case $1 in
+	-*)
+		;;
+	*)
+		usage
+		exit 1
+		;;
+esac
+
+rm -f $ENV_CONF
+mkdir -p ${ENV_CONF%/*}
+echo [Service] > $ENV_CONF
+
+while getopts "t:s:-:h" opt; do
+	case $opt in
+		h)
+			usage
+			exit
+			;;
+		t)
+			VAR_NAME=K3S_TOKEN
+			;;
+		s)
+			VAR_NAME=K3S_URL
+			;;
+		-)
+			[ $# -ge $OPTIND ] || {
+				usage
+				exit 1
+			}
+			opt=$OPTARG
+			eval OPTARG='$'$OPTIND
+			OPTIND=$(($OPTIND + 1))
+			case $opt in
+				token)
+					VAR_NAME=K3S_TOKEN
+					;;
+				token-file)
+					VAR_NAME=K3S_TOKEN_FILE
+					;;
+				server)
+					VAR_NAME=K3S_URL
+					;;
+				node-name)
+					VAR_NAME=K3S_NODE_NAME
+					;;
+				resolv-conf)
+					VAR_NAME=K3S_RESOLV_CONF
+					;;
+				cluster-secret)
+					VAR_NAME=K3S_CLUSTER_SECRET
+					;;
+				help)
+					usage
+					exit
+					;;
+				*)
+					usage
+					exit 1
+					;;
+			esac
+			;;
+		*)
+			usage
+			exit 1
+			;;
+	esac
+    echo Environment=$VAR_NAME=$OPTARG >> $ENV_CONF
+done
+
+chmod 0644 $ENV_CONF
+rm -rf /var/lib/rancher/k3s/agent
+systemctl daemon-reload
+systemctl restart k3s-agent
+systemctl enable k3s-agent.service
recipes-containers/k3s/k3s/k3s-agent.service new file +26 -0
new file mode 100644
index 0000000..9f9016d
@@ -0,0 +1,26 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes Agent
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=control-group
+Delegate=yes
+LimitNOFILE=infinity
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s agent
+ExecStopPost=/usr/local/bin/k3s-clean
+
recipes-containers/k3s/k3s/k3s-clean new file +30 -0
new file mode 100755
index 0000000..8eca918
@@ -0,0 +1,30 @@
+#!/bin/sh -eu
+#
+# Copyright (C) 2020 Axis Communications AB
+#
+# SPDX-License-Identifier: Apache-2.0
+
+do_unmount() {
+	[ $# -eq 2 ] || return
+	local mounts=
+	while read ignore mount ignore; do
+		case $mount in
+			$1/*|$2/*)
+				mounts="$mount $mounts"
+				;;
+		esac
+	done </proc/self/mounts
+	[ -z "$mounts" ] || umount $mounts
+}
+
+do_unmount /run/k3s /var/lib/rancher/k3s
+
+# The lines below come from install.sh's create_killall() function:
+ip link show 2>/dev/null | grep 'master cni0' | while read ignore iface ignore; do
+    iface=${iface%%@*}
+    [ -z "$iface" ] || ip link delete $iface
+done
+
+ip link delete cni0
+ip link delete flannel.1
+rm -rf /var/lib/cni/
recipes-containers/k3s/k3s/k3s.service new file +27 -0
new file mode 100644
index 0000000..34c7a80
@@ -0,0 +1,27 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=process
+Delegate=yes
+# Having non-zero Limit*s causes performance problems due to accounting overhead
+# in the kernel. We recommend using cgroups to do container-local accounting.
+LimitNOFILE=1048576
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s server
+
recipes-containers/k3s/k3s_git.bb new file +75 -0
new file mode 100644
index 0000000..cfc2c64
@@ -0,0 +1,75 @@
+SUMMARY = "Production-Grade Container Scheduling and Management"
+DESCRIPTION = "Lightweight Kubernetes, intended to be a fully compliant Kubernetes."
+HOMEPAGE = "https://k3s.io/"
+LICENSE = "Apache-2.0"
+LIC_FILES_CHKSUM = "file://${S}/src/import/LICENSE;md5=2ee41112a44fe7014dce33e26468ba93"
+PV = "v1.18.9+k3s1-dirty"
+
+SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.18;name=k3s \
+           file://k3s.service \
+           file://k3s-agent.service \
+           file://k3s-agent \
+           file://k3s-clean \
+           file://cni-containerd-net.conf \
+           file://0001-Finding-host-local-in-usr-libexec.patch;patchdir=src/import \
+          "
+SRC_URI[k3s.md5sum] = "363d3a08dc0b72ba6e6577964f6e94a5"
+SRCREV_k3s = "630bebf94b9dce6b8cd3d402644ed023b3af8f90"
+
+inherit go
+inherit goarch
+inherit systemd
+
+PACKAGECONFIG = ""
+PACKAGECONFIG[upx] = ",,upx-native"
+GO_IMPORT = "import"
+GO_BUILD_LDFLAGS = "-X github.com/rancher/k3s/pkg/version.Version=${PV} \
+                    -X github.com/rancher/k3s/pkg/version.GitCommit=${@d.getVar('SRCREV_k3s', d, 1)[:8]} \
+                    -w -s \
+                   "
+BIN_PREFIX ?= "${exec_prefix}/local"
+
+do_compile() {
+        export GOPATH="${S}/src/import/.gopath:${S}/src/import/vendor:${STAGING_DIR_TARGET}/${prefix}/local/go"
+        export CGO_ENABLED="1"
+        export GOFLAGS="-mod=vendor"
+        cd ${S}/src/import
+        ${GO} build -tags providerless -ldflags "${GO_BUILD_LDFLAGS}" -o ./dist/artifacts/k3s ./cmd/server/main.go
+        # Use UPX if it is enabled (and thus exists) to compress binary
+        if command -v upx > /dev/null 2>&1; then
+                upx -9 ./dist/artifacts/k3s
+        fi
+}
+do_install() {
+        install -d "${D}${BIN_PREFIX}/bin"
+        install -m 755 "${S}/src/import/dist/artifacts/k3s" "${D}${BIN_PREFIX}/bin"
+        ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/crictl"
+        ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/ctr"
+        ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/kubectl"
+        install -m 755 "${WORKDIR}/k3s-clean" "${D}${BIN_PREFIX}/bin"
+        install -D -m 0644 "${WORKDIR}/cni-containerd-net.conf" "${D}/${sysconfdir}/cni/net.d/10-containerd-net.conf"
+        if ${@bb.utils.contains('DISTRO_FEATURES','systemd','true','false',d)}; then
+                install -D -m 0644 "${WORKDIR}/k3s.service" "${D}${systemd_system_unitdir}/k3s.service"
+                install -D -m 0644 "${WORKDIR}/k3s-agent.service" "${D}${systemd_system_unitdir}/k3s-agent.service"
+                sed -i "s#\(Exec\)\(.*\)=\(.*\)\(k3s\)#\1\2=${BIN_PREFIX}/bin/\4#g" "${D}${systemd_system_unitdir}/k3s.service" "${D}${systemd_system_unitdir}/k3s-agent.service"
+                install -m 755 "${WORKDIR}/k3s-agent" "${D}${BIN_PREFIX}/bin"
+        fi
+}
+
+PACKAGES =+ "${PN}-server ${PN}-agent"
+
+SYSTEMD_PACKAGES = "${@bb.utils.contains('DISTRO_FEATURES','systemd','${PN}-server ${PN}-agent','',d)}"
+SYSTEMD_SERVICE_${PN}-server = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s.service','',d)}"
+SYSTEMD_SERVICE_${PN}-agent = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s-agent.service','',d)}"
+SYSTEMD_AUTO_ENABLE_${PN}-agent = "disable"
+
+FILES_${PN}-agent = "${BIN_PREFIX}/bin/k3s-agent"
+
+RDEPENDS_${PN} = "cni conntrack-tools coreutils findutils iproute2 ipset virtual/containerd"
+RDEPENDS_${PN}-server = "${PN}"
+RDEPENDS_${PN}-agent = "${PN}"
+
+RCONFLICTS_${PN} = "kubectl"
+
+INHIBIT_PACKAGE_STRIP = "1"
+INSANE_SKIP_${PN} += "ldflags already-stripped"
Signature · 1 line
2.20.1
#25 lore raw
Quoted 3 lines · Bruce Ashfield

Ha!!!!

This applies.

Wonderful, thank you! I guess this is what is called "five times lucky"...

Quoted 3 lines

I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.

Excellent!

BR,

/Joakim

Signature · 7 lines
Joakim Roubert
Senior Engineer

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com
#26 lore raw
Quoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield

Ha!!!!

This applies.

Quoted 7 lines

Wonderful, thank you! I guess this is what is called "five times lucky"...

Quoted 3 lines

I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.

Excellent!

I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/log/?h=k3s-wip

That includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.

I did have a couple of questions on the k3s packaging itself, I was getting the following
error:

ERROR: k3s-v1.18.9+k3s1-dirty-r0 do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3s

So I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?

Bruce

Quoted 8 lines

BR,

/Joakim
--
Joakim Roubert
Senior Engineer

Quoted 5 lines

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

#27 lore raw
Quoted 11 lines · Bruce Ashfield

ERROR: k3s-v1.18.9+k3s1-dirty-r0 do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
  /usr/local/bin/k3s-clean
  /usr/local/bin/crictl
  /usr/local/bin/kubectl
  /usr/local/bin/k3s

So I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?

Yes, it is! In my build, all of those are found in the k3s RPM, although the FILES directive for them is missing. So yes, your fix here is the right one, IMO.

BR,

/Joakim

#28 lore raw
Quoted 10 lines · Joakim Roubert
Quoted 8 lines · Bruce Ashfield

ERROR: k3s-v1.18.9+k3s1-dirty-r0 do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:

   /usr/local/bin/k3s-clean
   /usr/local/bin/crictl
   /usr/local/bin/kubectl
   /usr/local/bin/k3s
Quoted 7 lines
Quoted 3 lines

So I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?

Yes, it is! In my build, all of those are found in the k3s RPM, although the FILES directive for them is missing. So yes, your fix here is the right one, IMO.

I now have another 6 or 7 WIP patches on top of this to try and get a single node "cluster" working with k3s. I'll clean them up and get them into the k3s WIP branch shortly.

I'm failing on the networking component though.

In your working references, which iptables do you have installed ? (legacy ? nftables?)

I'm failing to get flannel to start, with a series of errors like this:

-----------
I1106 21:19:00.985656 10641 eviction_manager.go:351] eviction
manager: able to reduce ephemeral-storage pressure without evicting pods.
E1106 21:19:10.636899 10641 proxier.go:841] Failed to ensure that
filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error checking
rule: exit status 2: iptables v1.8.6 (legacy): Couldn't load match `comment':No such y
Try `iptables -h' or 'iptables --help' for more information.
I1106 21:19:10.641647 10641 proxier.go:825] Sync failed; retrying in 30s
------------

Which means as soon as I try and run an agent command, the server dies and I can't even do a basic test.

Bruce

Quoted 4 lines

BR,

/Joakim

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#29 lore raw
Quoted 4 lines · Bruce Ashfield

I now have another 6 or 7 WIP patches on top of this to try and get a single node "cluster" working with k3s. I'll clean them up and get them into the k3s WIP branch shortly.

Awesome!

Quoted 2 lines

In your working references, which iptables do you have installed ? (legacy ? nftables?)

# iptables --version
iptables v1.8.4 (legacy)
Quoted 13 lines

I'm failing to get flannel to start, with a series of errors like this:

-----------
I1106 21:19:00.985656   10641 eviction_manager.go:351] eviction
manager: able to reduce ephemeral-storage pressure without evicting pods.
E1106 21:19:10.636899   10641 proxier.go:841] Failed to ensure that filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error checking
rule: exit status 2: iptables v1.8.6 (legacy): Couldn't load match `comment':No such y
Try `iptables -h' or 'iptables --help' for more information. I1106 21:19:10.641647   10641 proxier.go:825] Sync failed; retrying in 30s
------------

This is a bit strange, as it seems you are running in legacy mode too, although a somewhat newer version than I have, and the only thing I know is that Rancher recommends 1.6.1 or newer (which it is).

https://rancher.com/docs/k3s/latest/en/known-issues/

Might there be something missing in the kernel?

I have these RPMs installed in my image:

iptables
iptables-module-ip6t-ah
iptables-module-ip6t-dnat
iptables-module-ip6t-dnpt
iptables-module-ip6t-dst
iptables-module-ip6t-eui64
iptables-module-ip6t-frag
iptables-module-ip6t-hbh
iptables-module-ip6t-hl
iptables-module-ip6t-icmp6
iptables-module-ip6t-ipv6header
iptables-module-ip6t-log
iptables-module-ip6t-masquerade
iptables-module-ip6t-mh
iptables-module-ip6t-netmap
iptables-module-ip6t-redirect
iptables-module-ip6t-reject
iptables-module-ip6t-rt
iptables-module-ip6t-snat
iptables-module-ip6t-snpt
iptables-module-ip6t-srh
iptables-module-ipt-ah
iptables-module-ipt-clusterip
iptables-module-ipt-dnat
iptables-module-ipt-ecn
iptables-module-ipt-icmp
iptables-module-ipt-log
iptables-module-ipt-masquerade
iptables-module-ipt-netmap
iptables-module-ipt-realm
iptables-module-ipt-redirect
iptables-module-ipt-reject
iptables-module-ipt-snat
iptables-module-ipt-ttl
iptables-module-ipt-ulog
iptables-modules
iptables-module-xt-addrtype
iptables-module-xt-audit
iptables-module-xt-bpf
iptables-module-xt-cgroup
iptables-module-xt-checksum
iptables-module-xt-classify
iptables-module-xt-cluster
iptables-module-xt-comment
iptables-module-xt-connbytes
iptables-module-xt-connlimit
iptables-module-xt-connmark
iptables-module-xt-connsecmark
iptables-module-xt-conntrack
iptables-module-xt-cpu
iptables-module-xt-ct
iptables-module-xt-dccp
iptables-module-xt-devgroup
iptables-module-xt-dscp
iptables-module-xt-ecn
iptables-module-xt-esp
iptables-module-xt-hashlimit
iptables-module-xt-helper
iptables-module-xt-hmark
iptables-module-xt-idletimer
iptables-module-xt-ipcomp
iptables-module-xt-iprange
iptables-module-xt-ipvs
iptables-module-xt-led
iptables-module-xt-length
iptables-module-xt-limit
iptables-module-xt-mac
iptables-module-xt-mark
iptables-module-xt-multiport
iptables-module-xt-nfacct
iptables-module-xt-nflog
iptables-module-xt-nfqueue
iptables-module-xt-osf
iptables-module-xt-owner
iptables-module-xt-physdev
iptables-module-xt-pkttype
iptables-module-xt-policy
iptables-module-xt-quota
iptables-module-xt-rateest
iptables-module-xt-recent
iptables-module-xt-rpfilter
iptables-module-xt-sctp
iptables-module-xt-secmark
iptables-module-xt-set
iptables-module-xt-socket
iptables-module-xt-standard
iptables-module-xt-statistic
iptables-module-xt-string
iptables-module-xt-synproxy
iptables-module-xt-tcp
iptables-module-xt-tcpmss
iptables-module-xt-tcpoptstrip
iptables-module-xt-tee
iptables-module-xt-time
iptables-module-xt-tos
iptables-module-xt-tproxy
iptables-module-xt-trace
iptables-module-xt-u32
iptables-module-xt-udp

and in my kernel config, I have (apart from what is needed for running containers with containerd):

CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NET_UDP_TUNNEL=m
CONFIG_NF_DUP_NETDEV=m
CONFIG_NF_LOG_BRIDGE=m
CONFIG_NF_TABLES_ARP=y
CONFIG_NF_TABLES_BRIDGE=y
CONFIG_NF_TABLES_INET=y
CONFIG_NF_TABLES_IPV4=y
CONFIG_NF_TABLES_IPV6=y
CONFIG_NF_TABLES=m
CONFIG_NF_TABLES_NETDEV=y
CONFIG_NFT_BRIDGE_REJECT=m
CONFIG_NFT_CHAIN_NAT_IPV4=m
CONFIG_NFT_CHAIN_ROUTE_IPV4=m
CONFIG_NFT_CHAIN_ROUTE_IPV6=m
CONFIG_NFT_COMPAT=m
CONFIG_NFT_COUNTER=m
CONFIG_NFT_CT=m
CONFIG_NFT_DUP_IPV4=m
CONFIG_NFT_DUP_IPV6=m
CONFIG_NFT_DUP_NETDEV=m
# CONFIG_NFT_EXTHDR is not set
CONFIG_NFT_FIB_INET=m
CONFIG_NFT_FIB_IPV4=m
CONFIG_NFT_FIB_IPV6=m
CONFIG_NFT_FIB_NETDEV=m
CONFIG_NFT_FWD_NETDEV=m
CONFIG_NFT_HASH=m
CONFIG_NFT_LIMIT=m
CONFIG_NFT_LOG=m
CONFIG_NFT_MASQ_IPV4=m
CONFIG_NFT_MASQ=m
# CONFIG_NFT_META is not set
CONFIG_NFT_NAT=m
CONFIG_NFT_NUMGEN=m
# CONFIG_NFT_OBJREF is not set
CONFIG_NFT_QUEUE=m
CONFIG_NFT_QUOTA=m
CONFIG_NFT_REDIR_IPV4=m
CONFIG_NFT_REDIR=m
CONFIG_NFT_REJECT=m
# CONFIG_NFT_RT is not set
# CONFIG_NFT_SET_BITMAP is not set
# CONFIG_NFT_SET_HASH is not set
# CONFIG_NFT_SET_RBTREE is not set
CONFIG_OVERLAY_FS=m
CONFIG_STP=m

BR,

/Joakim

#30 lore raw

Hi Bruce,

For the iptables issue, I tested iptables.

As iptables comment module belonging to iptables extension, I checked my kernel config and set the parameter: CONFIG_NETFILTER_XT_MATCH_COMMENT=m.

iptables -V
iptables v1.8.5 (legacy)

I used this iptables command to check

iptables -A INPUT -p tcp --dport 22 -m comment --comment "SSH" -j ACCEPT

It works fine from my side.

Best Regards,
Lance

Quoted 199 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Joakim Roubert via lists.yoctoproject.org
Sent: Monday, November 9, 2020 3:49 PM
To: Bruce Ashfield <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 4 lines · Bruce Ashfield

I now have another 6 or 7 WIP patches on top of this to try and get a single node "cluster" working with k3s. I'll clean them up and get them into the k3s WIP branch shortly.

Awesome!

Quoted 2 lines

In your working references, which iptables do you have installed ? (legacy ? nftables?)

# iptables --version
iptables v1.8.4 (legacy)
Quoted 14 lines

I'm failing to get flannel to start, with a series of errors like this:

-----------
I1106 21:19:00.985656 10641 eviction_manager.go:351] eviction
manager: able to reduce ephemeral-storage pressure without evicting pods.
E1106 21:19:10.636899 10641 proxier.go:841] Failed to ensure that
filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error checking
rule: exit status 2: iptables v1.8.6 (legacy): Couldn't load match comment':No such y Try iptables -h' or 'iptables --help' for more information.
I1106 21:19:10.641647 10641 proxier.go:825] Sync failed; retrying in
30s
------------

This is a bit strange, as it seems you are running in legacy mode too, although a somewhat newer version than I have, and the only thing I know is that Rancher recommends 1.6.1 or newer (which it is).

https://rancher.com/docs/k3s/latest/en/known-issues/

Might there be something missing in the kernel?

I have these RPMs installed in my image:

iptables
iptables-module-ip6t-ah
iptables-module-ip6t-dnat
iptables-module-ip6t-dnpt
iptables-module-ip6t-dst
iptables-module-ip6t-eui64
iptables-module-ip6t-frag
iptables-module-ip6t-hbh
iptables-module-ip6t-hl
iptables-module-ip6t-icmp6
iptables-module-ip6t-ipv6header
iptables-module-ip6t-log
iptables-module-ip6t-masquerade
iptables-module-ip6t-mh
iptables-module-ip6t-netmap
iptables-module-ip6t-redirect
iptables-module-ip6t-reject
iptables-module-ip6t-rt
iptables-module-ip6t-snat
iptables-module-ip6t-snpt
iptables-module-ip6t-srh
iptables-module-ipt-ah
iptables-module-ipt-clusterip
iptables-module-ipt-dnat
iptables-module-ipt-ecn
iptables-module-ipt-icmp
iptables-module-ipt-log
iptables-module-ipt-masquerade
iptables-module-ipt-netmap
iptables-module-ipt-realm
iptables-module-ipt-redirect
iptables-module-ipt-reject
iptables-module-ipt-snat
iptables-module-ipt-ttl
iptables-module-ipt-ulog
iptables-modules
iptables-module-xt-addrtype
iptables-module-xt-audit
iptables-module-xt-bpf
iptables-module-xt-cgroup
iptables-module-xt-checksum
iptables-module-xt-classify
iptables-module-xt-cluster
iptables-module-xt-comment
iptables-module-xt-connbytes
iptables-module-xt-connlimit
iptables-module-xt-connmark
iptables-module-xt-connsecmark
iptables-module-xt-conntrack
iptables-module-xt-cpu
iptables-module-xt-ct
iptables-module-xt-dccp
iptables-module-xt-devgroup
iptables-module-xt-dscp
iptables-module-xt-ecn
iptables-module-xt-esp
iptables-module-xt-hashlimit
iptables-module-xt-helper
iptables-module-xt-hmark
iptables-module-xt-idletimer
iptables-module-xt-ipcomp
iptables-module-xt-iprange
iptables-module-xt-ipvs
iptables-module-xt-led
iptables-module-xt-length
iptables-module-xt-limit
iptables-module-xt-mac
iptables-module-xt-mark
iptables-module-xt-multiport
iptables-module-xt-nfacct
iptables-module-xt-nflog
iptables-module-xt-nfqueue
iptables-module-xt-osf
iptables-module-xt-owner
iptables-module-xt-physdev
iptables-module-xt-pkttype
iptables-module-xt-policy
iptables-module-xt-quota
iptables-module-xt-rateest
iptables-module-xt-recent
iptables-module-xt-rpfilter
iptables-module-xt-sctp
iptables-module-xt-secmark
iptables-module-xt-set
iptables-module-xt-socket
iptables-module-xt-standard
iptables-module-xt-statistic
iptables-module-xt-string
iptables-module-xt-synproxy
iptables-module-xt-tcp
iptables-module-xt-tcpmss
iptables-module-xt-tcpoptstrip
iptables-module-xt-tee
iptables-module-xt-time
iptables-module-xt-tos
iptables-module-xt-tproxy
iptables-module-xt-trace
iptables-module-xt-u32
iptables-module-xt-udp

and in my kernel config, I have (apart from what is needed for running containers with containerd):

CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NET_UDP_TUNNEL=m
CONFIG_NF_DUP_NETDEV=m
CONFIG_NF_LOG_BRIDGE=m
CONFIG_NF_TABLES_ARP=y
CONFIG_NF_TABLES_BRIDGE=y
CONFIG_NF_TABLES_INET=y
CONFIG_NF_TABLES_IPV4=y
CONFIG_NF_TABLES_IPV6=y
CONFIG_NF_TABLES=m
CONFIG_NF_TABLES_NETDEV=y
CONFIG_NFT_BRIDGE_REJECT=m
CONFIG_NFT_CHAIN_NAT_IPV4=m
CONFIG_NFT_CHAIN_ROUTE_IPV4=m
CONFIG_NFT_CHAIN_ROUTE_IPV6=m
CONFIG_NFT_COMPAT=m
CONFIG_NFT_COUNTER=m
CONFIG_NFT_CT=m
CONFIG_NFT_DUP_IPV4=m
CONFIG_NFT_DUP_IPV6=m
CONFIG_NFT_DUP_NETDEV=m
# CONFIG_NFT_EXTHDR is not set
CONFIG_NFT_FIB_INET=m
CONFIG_NFT_FIB_IPV4=m
CONFIG_NFT_FIB_IPV6=m
CONFIG_NFT_FIB_NETDEV=m
CONFIG_NFT_FWD_NETDEV=m
CONFIG_NFT_HASH=m
CONFIG_NFT_LIMIT=m
CONFIG_NFT_LOG=m
CONFIG_NFT_MASQ_IPV4=m
CONFIG_NFT_MASQ=m
# CONFIG_NFT_META is not set
CONFIG_NFT_NAT=m
CONFIG_NFT_NUMGEN=m
# CONFIG_NFT_OBJREF is not set
CONFIG_NFT_QUEUE=m
CONFIG_NFT_QUOTA=m
CONFIG_NFT_REDIR_IPV4=m
CONFIG_NFT_REDIR=m
CONFIG_NFT_REJECT=m
# CONFIG_NFT_RT is not set
# CONFIG_NFT_SET_BITMAP is not set
# CONFIG_NFT_SET_HASH is not set
# CONFIG_NFT_SET_RBTREE is not set
CONFIG_OVERLAY_FS=m
CONFIG_STP=m

Quoted 3 lines

BR,

/Joakim

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#31 lore raw
Quoted 7 lines · Lance Yang

Hi Bruce,

For the iptables issue, I tested iptables.

As iptables comment module belonging to iptables extension, I checked my kernel config and set the parameter: CONFIG_NETFILTER_XT_MATCH_COMMENT=m.

Quoted 8 lines

iptables -V
iptables v1.8.5 (legacy)

I used this iptables command to check

iptables -A INPUT -p tcp --dport 22 -m comment --comment "SSH" -j ACCEPT

It works fine from my side.

Yah, that's what I assumed it was as well, but yet, when I added it in .. I didn't see a change.

That being said, this is helpful, so I started a clean build to see if I had picked up something stale that was masking my fix.

Bruce

Quoted 207 lines

Best Regards,
Lance

Quoted 199 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Joakim Roubert via lists.yoctoproject.org
Sent: Monday, November 9, 2020 3:49 PM
To: Bruce Ashfield <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 4 lines · Bruce Ashfield

I now have another 6 or 7 WIP patches on top of this to try and get a single node "cluster" working with k3s. I'll clean them up and get them into the k3s WIP branch shortly.

Awesome!

Quoted 2 lines

In your working references, which iptables do you have installed ? (legacy ? nftables?)

# iptables --version
iptables v1.8.4 (legacy)
Quoted 14 lines

I'm failing to get flannel to start, with a series of errors like this:

-----------
I1106 21:19:00.985656 10641 eviction_manager.go:351] eviction
manager: able to reduce ephemeral-storage pressure without evicting pods.
E1106 21:19:10.636899 10641 proxier.go:841] Failed to ensure that
filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error checking
rule: exit status 2: iptables v1.8.6 (legacy): Couldn't load match comment':No such y Try iptables -h' or 'iptables --help' for more information.
I1106 21:19:10.641647 10641 proxier.go:825] Sync failed; retrying in
30s
------------

This is a bit strange, as it seems you are running in legacy mode too, although a somewhat newer version than I have, and the only thing I know is that Rancher recommends 1.6.1 or newer (which it is).

https://rancher.com/docs/k3s/latest/en/known-issues/

Might there be something missing in the kernel?

I have these RPMs installed in my image:

iptables
iptables-module-ip6t-ah
iptables-module-ip6t-dnat
iptables-module-ip6t-dnpt
iptables-module-ip6t-dst
iptables-module-ip6t-eui64
iptables-module-ip6t-frag
iptables-module-ip6t-hbh
iptables-module-ip6t-hl
iptables-module-ip6t-icmp6
iptables-module-ip6t-ipv6header
iptables-module-ip6t-log
iptables-module-ip6t-masquerade
iptables-module-ip6t-mh
iptables-module-ip6t-netmap
iptables-module-ip6t-redirect
iptables-module-ip6t-reject
iptables-module-ip6t-rt
iptables-module-ip6t-snat
iptables-module-ip6t-snpt
iptables-module-ip6t-srh
iptables-module-ipt-ah
iptables-module-ipt-clusterip
iptables-module-ipt-dnat
iptables-module-ipt-ecn
iptables-module-ipt-icmp
iptables-module-ipt-log
iptables-module-ipt-masquerade
iptables-module-ipt-netmap
iptables-module-ipt-realm
iptables-module-ipt-redirect
iptables-module-ipt-reject
iptables-module-ipt-snat
iptables-module-ipt-ttl
iptables-module-ipt-ulog
iptables-modules
iptables-module-xt-addrtype
iptables-module-xt-audit
iptables-module-xt-bpf
iptables-module-xt-cgroup
iptables-module-xt-checksum
iptables-module-xt-classify
iptables-module-xt-cluster
iptables-module-xt-comment
iptables-module-xt-connbytes
iptables-module-xt-connlimit
iptables-module-xt-connmark
iptables-module-xt-connsecmark
iptables-module-xt-conntrack
iptables-module-xt-cpu
iptables-module-xt-ct
iptables-module-xt-dccp
iptables-module-xt-devgroup
iptables-module-xt-dscp
iptables-module-xt-ecn
iptables-module-xt-esp
iptables-module-xt-hashlimit
iptables-module-xt-helper
iptables-module-xt-hmark
iptables-module-xt-idletimer
iptables-module-xt-ipcomp
iptables-module-xt-iprange
iptables-module-xt-ipvs
iptables-module-xt-led
iptables-module-xt-length
iptables-module-xt-limit
iptables-module-xt-mac
iptables-module-xt-mark
iptables-module-xt-multiport
iptables-module-xt-nfacct
iptables-module-xt-nflog
iptables-module-xt-nfqueue
iptables-module-xt-osf
iptables-module-xt-owner
iptables-module-xt-physdev
iptables-module-xt-pkttype
iptables-module-xt-policy
iptables-module-xt-quota
iptables-module-xt-rateest
iptables-module-xt-recent
iptables-module-xt-rpfilter
iptables-module-xt-sctp
iptables-module-xt-secmark
iptables-module-xt-set
iptables-module-xt-socket
iptables-module-xt-standard
iptables-module-xt-statistic
iptables-module-xt-string
iptables-module-xt-synproxy
iptables-module-xt-tcp
iptables-module-xt-tcpmss
iptables-module-xt-tcpoptstrip
iptables-module-xt-tee
iptables-module-xt-time
iptables-module-xt-tos
iptables-module-xt-tproxy
iptables-module-xt-trace
iptables-module-xt-u32
iptables-module-xt-udp

and in my kernel config, I have (apart from what is needed for running containers with containerd):

CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NET_UDP_TUNNEL=m
CONFIG_NF_DUP_NETDEV=m
CONFIG_NF_LOG_BRIDGE=m
CONFIG_NF_TABLES_ARP=y
CONFIG_NF_TABLES_BRIDGE=y
CONFIG_NF_TABLES_INET=y
CONFIG_NF_TABLES_IPV4=y
CONFIG_NF_TABLES_IPV6=y
CONFIG_NF_TABLES=m
CONFIG_NF_TABLES_NETDEV=y
CONFIG_NFT_BRIDGE_REJECT=m
CONFIG_NFT_CHAIN_NAT_IPV4=m
CONFIG_NFT_CHAIN_ROUTE_IPV4=m
CONFIG_NFT_CHAIN_ROUTE_IPV6=m
CONFIG_NFT_COMPAT=m
CONFIG_NFT_COUNTER=m
CONFIG_NFT_CT=m
CONFIG_NFT_DUP_IPV4=m
CONFIG_NFT_DUP_IPV6=m
CONFIG_NFT_DUP_NETDEV=m
# CONFIG_NFT_EXTHDR is not set
CONFIG_NFT_FIB_INET=m
CONFIG_NFT_FIB_IPV4=m
CONFIG_NFT_FIB_IPV6=m
CONFIG_NFT_FIB_NETDEV=m
CONFIG_NFT_FWD_NETDEV=m
CONFIG_NFT_HASH=m
CONFIG_NFT_LIMIT=m
CONFIG_NFT_LOG=m
CONFIG_NFT_MASQ_IPV4=m
CONFIG_NFT_MASQ=m
# CONFIG_NFT_META is not set
CONFIG_NFT_NAT=m
CONFIG_NFT_NUMGEN=m
# CONFIG_NFT_OBJREF is not set
CONFIG_NFT_QUEUE=m
CONFIG_NFT_QUOTA=m
CONFIG_NFT_REDIR_IPV4=m
CONFIG_NFT_REDIR=m
CONFIG_NFT_REJECT=m
# CONFIG_NFT_RT is not set
# CONFIG_NFT_SET_BITMAP is not set
# CONFIG_NFT_SET_HASH is not set
# CONFIG_NFT_SET_RBTREE is not set
CONFIG_OVERLAY_FS=m
CONFIG_STP=m

Quoted 3 lines

BR,

/Joakim

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#32 lore raw

Hi Bruce,

Thanks for your reply. Because I installed "kernel-modules" in my local.conf:

IMAGE_INSTALL_append = " ... \ (skip something unrelated)
iptables kernel-modules k3s"

So I am not able to figure out which modules are missing. But hopefully, my investigation can help a little bit.

I referred to the file iptables_1.8.5.bb, I found something like:

RDEPENDS_${PN} = "${PN}-module-xt-standard"

RRECOMMENDS_${PN} = " \
    ${PN}-modules \
    kernel-module-x-tables \
    kernel-module-ip-tables \
    kernel-module-iptable-filter \
    kernel-module-iptable-nat \
    kernel-module-nf-defrag-ipv4 \
    kernel-module-nf-conntrack \
    kernel-module-nf-conntrack-ipv4 \
    kernel-module-nf-nat \
    kernel-module-ipt-masquerade \
    ${@bb.utils.contains('PACKAGECONFIG', 'ipv6', '\
        kernel-module-ip6table-filter \
        kernel-module-ip6-tables \
    ', '', d)} \

I am not sure whether we need install kernel-module-x-tables manually or something like that. Maybe it is not a bad way to try install "kernel-modules" directly to confirm if this issue is related to some kernel-modules. And then figure out which specific netfilter/iptables/xtables modules we are missing. What do you think?

Best Regards,
Lance Yang

Quoted 248 lines

-----Original Message-----
From: Bruce Ashfield <[email protected]>
Sent: Monday, November 9, 2020 9:46 PM
To: Lance Yang <[email protected]>
Cc: [email protected]; [email protected]; nd <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 6 lines · Lance Yang

Hi Bruce,

For the iptables issue, I tested iptables.

As iptables comment module belonging to iptables extension, I checked my kernel config and

set the parameter: CONFIG_NETFILTER_XT_MATCH_COMMENT=m.

Quoted 10 lines

iptables -V
iptables v1.8.5 (legacy)

I used this iptables command to check

iptables -A INPUT -p tcp --dport 22 -m comment --comment "SSH" -j ACCEPT

It works fine from my side.

Yah, that's what I assumed it was as well, but yet, when I added it in .. I didn't see a change.

That being said, this is helpful, so I started a clean build to see if I had picked up something stale that was masking my fix.

Bruce

Quoted 205 lines

Best Regards,
Lance

Quoted 201 lines

-----Original Message-----
From: [email protected] <[email protected]>
On Behalf Of Joakim Roubert via lists.yoctoproject.org
Sent: Monday, November 9, 2020 3:49 PM
To: Bruce Ashfield <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 4 lines · Bruce Ashfield

I now have another 6 or 7 WIP patches on top of this to try and get a single node "cluster" working with k3s. I'll clean them up and get them into the k3s WIP branch shortly.

Awesome!

Quoted 2 lines

In your working references, which iptables do you have installed ? (legacy ? nftables?)

# iptables --version
iptables v1.8.4 (legacy)
Quoted 14 lines

I'm failing to get flannel to start, with a series of errors like this:

-----------
I1106 21:19:00.985656 10641 eviction_manager.go:351] eviction
manager: able to reduce ephemeral-storage pressure without evicting pods.
E1106 21:19:10.636899 10641 proxier.go:841] Failed to ensure that
filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error checking
rule: exit status 2: iptables v1.8.6 (legacy): Couldn't load match comment':No such y Try iptables -h' or 'iptables --help' for more information.
I1106 21:19:10.641647 10641 proxier.go:825] Sync failed; retrying in
30s
------------

This is a bit strange, as it seems you are running in legacy mode too, although a somewhat newer version than I have, and the only thing I know is that Rancher recommends 1.6.1 or newer (which it is).

https://rancher.com/docs/k3s/latest/en/known-issues/

Might there be something missing in the kernel?

I have these RPMs installed in my image:

iptables
iptables-module-ip6t-ah
iptables-module-ip6t-dnat
iptables-module-ip6t-dnpt
iptables-module-ip6t-dst
iptables-module-ip6t-eui64
iptables-module-ip6t-frag
iptables-module-ip6t-hbh
iptables-module-ip6t-hl
iptables-module-ip6t-icmp6
iptables-module-ip6t-ipv6header
iptables-module-ip6t-log
iptables-module-ip6t-masquerade
iptables-module-ip6t-mh
iptables-module-ip6t-netmap
iptables-module-ip6t-redirect
iptables-module-ip6t-reject
iptables-module-ip6t-rt
iptables-module-ip6t-snat
iptables-module-ip6t-snpt
iptables-module-ip6t-srh
iptables-module-ipt-ah
iptables-module-ipt-clusterip
iptables-module-ipt-dnat
iptables-module-ipt-ecn
iptables-module-ipt-icmp
iptables-module-ipt-log
iptables-module-ipt-masquerade
iptables-module-ipt-netmap
iptables-module-ipt-realm
iptables-module-ipt-redirect
iptables-module-ipt-reject
iptables-module-ipt-snat
iptables-module-ipt-ttl
iptables-module-ipt-ulog
iptables-modules
iptables-module-xt-addrtype
iptables-module-xt-audit
iptables-module-xt-bpf
iptables-module-xt-cgroup
iptables-module-xt-checksum
iptables-module-xt-classify
iptables-module-xt-cluster
iptables-module-xt-comment
iptables-module-xt-connbytes
iptables-module-xt-connlimit
iptables-module-xt-connmark
iptables-module-xt-connsecmark
iptables-module-xt-conntrack
iptables-module-xt-cpu
iptables-module-xt-ct
iptables-module-xt-dccp
iptables-module-xt-devgroup
iptables-module-xt-dscp
iptables-module-xt-ecn
iptables-module-xt-esp
iptables-module-xt-hashlimit
iptables-module-xt-helper
iptables-module-xt-hmark
iptables-module-xt-idletimer
iptables-module-xt-ipcomp
iptables-module-xt-iprange
iptables-module-xt-ipvs
iptables-module-xt-led
iptables-module-xt-length
iptables-module-xt-limit
iptables-module-xt-mac
iptables-module-xt-mark
iptables-module-xt-multiport
iptables-module-xt-nfacct
iptables-module-xt-nflog
iptables-module-xt-nfqueue
iptables-module-xt-osf
iptables-module-xt-owner
iptables-module-xt-physdev
iptables-module-xt-pkttype
iptables-module-xt-policy
iptables-module-xt-quota
iptables-module-xt-rateest
iptables-module-xt-recent
iptables-module-xt-rpfilter
iptables-module-xt-sctp
iptables-module-xt-secmark
iptables-module-xt-set
iptables-module-xt-socket
iptables-module-xt-standard
iptables-module-xt-statistic
iptables-module-xt-string
iptables-module-xt-synproxy
iptables-module-xt-tcp
iptables-module-xt-tcpmss
iptables-module-xt-tcpoptstrip
iptables-module-xt-tee
iptables-module-xt-time
iptables-module-xt-tos
iptables-module-xt-tproxy
iptables-module-xt-trace
iptables-module-xt-u32
iptables-module-xt-udp

and in my kernel config, I have (apart from what is needed for running containers with
containerd):

CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NET_UDP_TUNNEL=m
CONFIG_NF_DUP_NETDEV=m
CONFIG_NF_LOG_BRIDGE=m
CONFIG_NF_TABLES_ARP=y
CONFIG_NF_TABLES_BRIDGE=y
CONFIG_NF_TABLES_INET=y
CONFIG_NF_TABLES_IPV4=y
CONFIG_NF_TABLES_IPV6=y
CONFIG_NF_TABLES=m
CONFIG_NF_TABLES_NETDEV=y
CONFIG_NFT_BRIDGE_REJECT=m
CONFIG_NFT_CHAIN_NAT_IPV4=m
CONFIG_NFT_CHAIN_ROUTE_IPV4=m
CONFIG_NFT_CHAIN_ROUTE_IPV6=m
CONFIG_NFT_COMPAT=m
CONFIG_NFT_COUNTER=m
CONFIG_NFT_CT=m
CONFIG_NFT_DUP_IPV4=m
CONFIG_NFT_DUP_IPV6=m
CONFIG_NFT_DUP_NETDEV=m
# CONFIG_NFT_EXTHDR is not set
CONFIG_NFT_FIB_INET=m
CONFIG_NFT_FIB_IPV4=m
CONFIG_NFT_FIB_IPV6=m
CONFIG_NFT_FIB_NETDEV=m
CONFIG_NFT_FWD_NETDEV=m
CONFIG_NFT_HASH=m
CONFIG_NFT_LIMIT=m
CONFIG_NFT_LOG=m
CONFIG_NFT_MASQ_IPV4=m
CONFIG_NFT_MASQ=m
# CONFIG_NFT_META is not set
CONFIG_NFT_NAT=m
CONFIG_NFT_NUMGEN=m
# CONFIG_NFT_OBJREF is not set
CONFIG_NFT_QUEUE=m
CONFIG_NFT_QUOTA=m
CONFIG_NFT_REDIR_IPV4=m
CONFIG_NFT_REDIR=m
CONFIG_NFT_REJECT=m
# CONFIG_NFT_RT is not set
# CONFIG_NFT_SET_BITMAP is not set
# CONFIG_NFT_SET_HASH is not set
# CONFIG_NFT_SET_RBTREE is not set
CONFIG_OVERLAY_FS=m
CONFIG_STP=m

Quoted 4 lines
Quoted 3 lines

BR,

/Joakim

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may

also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Quoted 3 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#33 lore raw
In reply to Joakim Roubert
Quoted 182 lines · Joakim Roubert
Quoted 4 lines · Bruce Ashfield

I now have another 6 or 7 WIP patches on top of this to try and get a single node "cluster" working with k3s. I'll clean them up and get them into the k3s WIP branch shortly.

Awesome!

Quoted 2 lines

In your working references, which iptables do you have installed ? (legacy ? nftables?)

# iptables --version
iptables v1.8.4 (legacy)
Quoted 13 lines

I'm failing to get flannel to start, with a series of errors like this:

-----------
I1106 21:19:00.985656 10641 eviction_manager.go:351] eviction
manager: able to reduce ephemeral-storage pressure without evicting pods.
E1106 21:19:10.636899 10641 proxier.go:841] Failed to ensure that
filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error checking
rule: exit status 2: iptables v1.8.6 (legacy): Couldn't load match `comment':No such y
Try `iptables -h' or 'iptables --help' for more information.
I1106 21:19:10.641647 10641 proxier.go:825] Sync failed; retrying in 30s
------------

This is a bit strange, as it seems you are running in legacy mode too, although a somewhat newer version than I have, and the only thing I know is that Rancher recommends 1.6.1 or newer (which it is).

https://rancher.com/docs/k3s/latest/en/known-issues/

Might there be something missing in the kernel?

I have these RPMs installed in my image:

iptables
iptables-module-ip6t-ah
iptables-module-ip6t-dnat
iptables-module-ip6t-dnpt
iptables-module-ip6t-dst
iptables-module-ip6t-eui64
iptables-module-ip6t-frag
iptables-module-ip6t-hbh
iptables-module-ip6t-hl
iptables-module-ip6t-icmp6
iptables-module-ip6t-ipv6header
iptables-module-ip6t-log
iptables-module-ip6t-masquerade
iptables-module-ip6t-mh
iptables-module-ip6t-netmap
iptables-module-ip6t-redirect
iptables-module-ip6t-reject
iptables-module-ip6t-rt
iptables-module-ip6t-snat
iptables-module-ip6t-snpt
iptables-module-ip6t-srh
iptables-module-ipt-ah
iptables-module-ipt-clusterip
iptables-module-ipt-dnat
iptables-module-ipt-ecn
iptables-module-ipt-icmp
iptables-module-ipt-log
iptables-module-ipt-masquerade
iptables-module-ipt-netmap
iptables-module-ipt-realm
iptables-module-ipt-redirect
iptables-module-ipt-reject
iptables-module-ipt-snat
iptables-module-ipt-ttl
iptables-module-ipt-ulog
iptables-modules
iptables-module-xt-addrtype
iptables-module-xt-audit
iptables-module-xt-bpf
iptables-module-xt-cgroup
iptables-module-xt-checksum
iptables-module-xt-classify
iptables-module-xt-cluster
iptables-module-xt-comment
iptables-module-xt-connbytes
iptables-module-xt-connlimit
iptables-module-xt-connmark
iptables-module-xt-connsecmark
iptables-module-xt-conntrack
iptables-module-xt-cpu
iptables-module-xt-ct
iptables-module-xt-dccp
iptables-module-xt-devgroup
iptables-module-xt-dscp
iptables-module-xt-ecn
iptables-module-xt-esp
iptables-module-xt-hashlimit
iptables-module-xt-helper
iptables-module-xt-hmark
iptables-module-xt-idletimer
iptables-module-xt-ipcomp
iptables-module-xt-iprange
iptables-module-xt-ipvs
iptables-module-xt-led
iptables-module-xt-length
iptables-module-xt-limit
iptables-module-xt-mac
iptables-module-xt-mark
iptables-module-xt-multiport
iptables-module-xt-nfacct
iptables-module-xt-nflog
iptables-module-xt-nfqueue
iptables-module-xt-osf
iptables-module-xt-owner
iptables-module-xt-physdev
iptables-module-xt-pkttype
iptables-module-xt-policy
iptables-module-xt-quota
iptables-module-xt-rateest
iptables-module-xt-recent
iptables-module-xt-rpfilter
iptables-module-xt-sctp
iptables-module-xt-secmark
iptables-module-xt-set
iptables-module-xt-socket
iptables-module-xt-standard
iptables-module-xt-statistic
iptables-module-xt-string
iptables-module-xt-synproxy
iptables-module-xt-tcp
iptables-module-xt-tcpmss
iptables-module-xt-tcpoptstrip
iptables-module-xt-tee
iptables-module-xt-time
iptables-module-xt-tos
iptables-module-xt-tproxy
iptables-module-xt-trace
iptables-module-xt-u32
iptables-module-xt-udp

and in my kernel config, I have (apart from what is needed for running containers with containerd):

CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NET_UDP_TUNNEL=m
CONFIG_NF_DUP_NETDEV=m
CONFIG_NF_LOG_BRIDGE=m
CONFIG_NF_TABLES_ARP=y
CONFIG_NF_TABLES_BRIDGE=y
CONFIG_NF_TABLES_INET=y
CONFIG_NF_TABLES_IPV4=y
CONFIG_NF_TABLES_IPV6=y
CONFIG_NF_TABLES=m
CONFIG_NF_TABLES_NETDEV=y
CONFIG_NFT_BRIDGE_REJECT=m
CONFIG_NFT_CHAIN_NAT_IPV4=m
CONFIG_NFT_CHAIN_ROUTE_IPV4=m
CONFIG_NFT_CHAIN_ROUTE_IPV6=m
CONFIG_NFT_COMPAT=m
CONFIG_NFT_COUNTER=m
CONFIG_NFT_CT=m
CONFIG_NFT_DUP_IPV4=m
CONFIG_NFT_DUP_IPV6=m
CONFIG_NFT_DUP_NETDEV=m
# CONFIG_NFT_EXTHDR is not set
CONFIG_NFT_FIB_INET=m
CONFIG_NFT_FIB_IPV4=m
CONFIG_NFT_FIB_IPV6=m
CONFIG_NFT_FIB_NETDEV=m
CONFIG_NFT_FWD_NETDEV=m
CONFIG_NFT_HASH=m
CONFIG_NFT_LIMIT=m
CONFIG_NFT_LOG=m
CONFIG_NFT_MASQ_IPV4=m
CONFIG_NFT_MASQ=m
# CONFIG_NFT_META is not set
CONFIG_NFT_NAT=m
CONFIG_NFT_NUMGEN=m
# CONFIG_NFT_OBJREF is not set
CONFIG_NFT_QUEUE=m
CONFIG_NFT_QUOTA=m
CONFIG_NFT_REDIR_IPV4=m

Quoted 8 lines

CONFIG_NFT_REDIR=m
CONFIG_NFT_REJECT=m
# CONFIG_NFT_RT is not set
# CONFIG_NFT_SET_BITMAP is not set
# CONFIG_NFT_SET_HASH is not set
# CONFIG_NFT_SET_RBTREE is not set
CONFIG_OVERLAY_FS=m
CONFIG_STP=m

Cool, the above helps.

I have some new fragments, etc, developed for both k8s and k3s, so I'll double check that I didn't fat finger something.

Bruce

Quoted 4 lines

BR,

/Joakim

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#34 lore raw
In reply to Bruce Ashfield

Hi Bruce and Joakim,

Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.

My Image: Linux qemuarm64 by yocto.

The master node can be ready after I started the k3s server. However, the pods in kube-system (which are essential components for k3s) cannot turn to ready state on qemuarm64.

After the master node itself turned to ready state, I check the pods with kubectl:

kubectl get nodes

NAME        STATUS   ROLES    AGE   VERSION
qemuarm64   Ready    master   11m   v1.18.9-k3s1
root@qemuarm64:~# ls
root@qemuarm64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS              RESTARTS   AGE
local-path-provisioner-6d59f47c7-xxvbl   0/1     ContainerCreating   0          12m
coredns-7944c66d8d-tlrm9                 0/1     ContainerCreating   0          12m
metrics-server-7566d596c8-svkff          0/1     ContainerCreating   0          12m
helm-install-traefik-s8p5g               0/1     ContainerCreating   0          12m

Then I describe the pods with:

Events:

  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    16m                  default-scheduler  Successfully assigned kube-system/coredns-7944c66d8d-tlrm9 to qemuarm64
  Warning  FailedMount  5m23s (x3 over 14m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token-b7nlh config-volume]: timed out waiting for the condition
  Warning  FailedMount  50s (x4 over 12m)    kubelet            Unable to attach or mount volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume coredns-token-b7nlh]: timed out waiting for the condition
  Warning  FailedMount  11s (x16 over 16m)   kubelet            MountVolume.SetUp failed for volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable file not found in $PATH

I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and mount binary on my qemuarm64 image:

root@qemuarm64:~# echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
root@qemuarm64:~# which mount
/bin/mount

When I type mount command, it worked fine:

/dev/root on / type ext4 (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
...
... (skipped the verbose output)

I would like to know whether you have met this "mount" issue ever?

Best Regards,
Lance

Quoted 52 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield

Ha!!!!

This applies.

Quoted 7 lines

Wonderful, thank you! I guess this is what is called "five times lucky"...

Quoted 3 lines

I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.

Excellent!

I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/log/?h=k3s-wip

That includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.

I did have a couple of questions on the k3s packaging itself, I was getting the following error:

ERROR: k3s-v1.18.9+k3s1-dirty-r0 do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3s

So I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?

Bruce

Quoted 8 lines

BR,

/Joakim
--
Joakim Roubert
Senior Engineer

Quoted 8 lines
Quoted 5 lines

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#35 lore raw
Quoted 4 lines · Lance Yang

Hi Bruce and Joakim,

Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.

The branch will be more functional shortly, I have quite a few changes to factor things for
k8s and generally more usable :D

        modified:   classes/cni_networking.bbclass
        modified:   conf/layer.conf
        modified:   recipes-containers/containerd/containerd-docker_git.bb
        modified:
recipes-containers/containerd/containerd-opencontainers_git.bb
        modified:   recipes-containers/k3s/README.md
        modified:   recipes-containers/k3s/k3s_git.bb
        modified:   recipes-kernel/linux/linux-yocto/kubernetes.cfg
        modified:   recipes-networking/cni/cni_git.bb

container-deploy.txt
recipes-core/packagegroups/

Quoted 5 lines

My Image: Linux qemuarm64 by yocto.

The master node can be ready after I started the k3s server. However, the pods in kube-system (which are essential components for k3s) cannot turn to ready state on qemuarm64.

That's interesting, since in my configuration, the master never comes ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
qemux86-64   NotReady   master   15h   v1.18.9-k3s1

I've sorted out more of the dependencies, and have packagegroups to make them easier
now.

Hopefully, I can figure out what is now missing and keeping my master from moving into
ready today.

Bruce

Quoted 107 lines

After the master node itself turned to ready state, I check the pods with kubectl:

kubectl get nodes

NAME        STATUS   ROLES    AGE   VERSION
qemuarm64   Ready    master   11m   v1.18.9-k3s1
root@qemuarm64:~# ls
root@qemuarm64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS              RESTARTS   AGE
local-path-provisioner-6d59f47c7-xxvbl   0/1     ContainerCreating   0          12m
coredns-7944c66d8d-tlrm9                 0/1     ContainerCreating   0          12m
metrics-server-7566d596c8-svkff          0/1     ContainerCreating   0          12m
helm-install-traefik-s8p5g               0/1     ContainerCreating   0          12m

Then I describe the pods with:

Events:

  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    16m                  default-scheduler  Successfully assigned kube-system/coredns-7944c66d8d-tlrm9 to qemuarm64
  Warning  FailedMount  5m23s (x3 over 14m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token-b7nlh config-volume]: timed out waiting for the condition
  Warning  FailedMount  50s (x4 over 12m)    kubelet            Unable to attach or mount volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume coredns-token-b7nlh]: timed out waiting for the condition
  Warning  FailedMount  11s (x16 over 16m)   kubelet            MountVolume.SetUp failed for volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable file not found in $PATH

I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and mount binary on my qemuarm64 image:

root@qemuarm64:~# echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
root@qemuarm64:~# which mount
/bin/mount

When I type mount command, it worked fine:

/dev/root on / type ext4 (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
...
... (skipped the verbose output)

I would like to know whether you have met this "mount" issue ever?

Best Regards,
Lance

Quoted 52 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield

Ha!!!!

This applies.

Quoted 7 lines

Wonderful, thank you! I guess this is what is called "five times lucky"...

Quoted 3 lines

I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.

Excellent!

I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/log/?h=k3s-wip

That includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.

I did have a couple of questions on the k3s packaging itself, I was getting the following error:

ERROR: k3s-v1.18.9+k3s1-dirty-r0 do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3s

So I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?

Bruce

Quoted 8 lines

BR,

/Joakim
--
Joakim Roubert
Senior Engineer

Quoted 8 lines
Quoted 5 lines

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

A message that is not in the archive (sent off-list or not yet received): [email protected]

#37 lore raw
Quoted 26 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 4 lines · Lance Yang

Hi Bruce and Joakim,

Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.

The branch will be more functional shortly, I have quite a few changes to factor things for
k8s and generally more usable :D

        modified:   classes/cni_networking.bbclass
        modified:   conf/layer.conf
        modified:   recipes-containers/containerd/containerd-docker_git.bb
        modified:
recipes-containers/containerd/containerd-opencontainers_git.bb
        modified:   recipes-containers/k3s/README.md
        modified:   recipes-containers/k3s/k3s_git.bb
        modified:   recipes-kernel/linux/linux-yocto/kubernetes.cfg
        modified:   recipes-networking/cni/cni_git.bb

container-deploy.txt
recipes-core/packagegroups/

Quoted 3 lines

My Image: Linux qemuarm64 by yocto.

Quoted 9 lines
Quoted 2 lines

The master node can be ready after I started the k3s server. However, the pods in kube-system (which are essential components for k3s) cannot turn to ready state on qemuarm64.

That's interesting, since in my configuration, the master never comes ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
qemux86-64   NotReady   master   15h   v1.18.9-k3s1

Hah.

I finally got the node to show up as ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS   ROLES    AGE    VERSION
qemux86-64   Ready    master   112s   v1.18.9-k3s1

I'm attempting to build an all-in-one node, and that is likely causing me some issues.

I'm revisiting those potential conflicts now.

But if anyone else does have an all in one working and has some tips, feel free to share :D

Bruce

Quoted 127 lines

I've sorted out more of the dependencies, and have packagegroups to make them easier
now.

Hopefully, I can figure out what is now missing and keeping my master from moving into
ready today.

Bruce

Quoted 103 lines

After the master node itself turned to ready state, I check the pods with kubectl:

kubectl get nodes

NAME        STATUS   ROLES    AGE   VERSION
qemuarm64   Ready    master   11m   v1.18.9-k3s1
root@qemuarm64:~# ls
root@qemuarm64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS              RESTARTS   AGE
local-path-provisioner-6d59f47c7-xxvbl   0/1     ContainerCreating   0          12m
coredns-7944c66d8d-tlrm9                 0/1     ContainerCreating   0          12m
metrics-server-7566d596c8-svkff          0/1     ContainerCreating   0          12m
helm-install-traefik-s8p5g               0/1     ContainerCreating   0          12m

Then I describe the pods with:

Events:

  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    16m                  default-scheduler  Successfully assigned kube-system/coredns-7944c66d8d-tlrm9 to qemuarm64
  Warning  FailedMount  5m23s (x3 over 14m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token-b7nlh config-volume]: timed out waiting for the condition
  Warning  FailedMount  50s (x4 over 12m)    kubelet            Unable to attach or mount volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume coredns-token-b7nlh]: timed out waiting for the condition
  Warning  FailedMount  11s (x16 over 16m)   kubelet            MountVolume.SetUp failed for volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable file not found in $PATH

I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and mount binary on my qemuarm64 image:

root@qemuarm64:~# echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
root@qemuarm64:~# which mount
/bin/mount

When I type mount command, it worked fine:

/dev/root on / type ext4 (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
...
... (skipped the verbose output)

I would like to know whether you have met this "mount" issue ever?

Best Regards,
Lance

Quoted 57 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield

Ha!!!!

This applies.

Quoted 7 lines

Wonderful, thank you! I guess this is what is called "five times lucky"...

Quoted 3 lines

I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.

Excellent!

I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/log/?h=k3s-wip

That includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.

I did have a couple of questions on the k3s packaging itself, I was getting the following error:

ERROR: k3s-v1.18.9+k3s1-dirty-r0 do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3s

So I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?

Bruce

Quoted 13 lines

BR,

/Joakim
--
Joakim Roubert
Senior Engineer

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com

Quoted 4 lines
Quoted 3 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#38 lore raw

Hi Bruce,

I pulled your latest k3s-wip branch. In your recipes-networking, the ipset:

cat ipset_6.38.bb
# Copyright (C) 2017 Aaron Brice <[email protected]> # Released under the MIT license (see COPYING.MIT for the terms)

...
... (skip verbose output)

DEPENDS = "libtool libmnl"
RDEPENDS_${PN} = "kernel-module-ip-set"
...

I use bitbake to install k3s and ipset but I got errors below:

Problem 1: conflicting requests
- nothing provides kernel-module-ip-set needed by ipset-6.38-r0

I think RDEPENDS_${PN} is a slightly over-strict condition in some situations. For example, in the following situation:

In my build/conf/local.conf, I added "kernel-modules" in IMAGE_INSTALL_append section. kernel-modules-ip-set has already included in kernel-modules when I check in qemuarm64 image with "lsmod|grep ip_set". But if RDEPENDS is used, bitbake won't continue to compile the package unless you add "kernel-module-ip-set".

So I suggested to use " RRECOMMENDS_${PN}".

Best Regards,
Lance

Quoted 204 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Tuesday, November 10, 2020 9:17 PM
To: Bruce Ashfield <[email protected]>
Cc: Lance Yang <[email protected]>; Joakim Roubert <[email protected]>; meta- [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 19 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 4 lines · Lance Yang

Hi Bruce and Joakim,

Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.

The branch will be more functional shortly, I have quite a few changes to factor things for k8s and generally more usable :D

        modified:   classes/cni_networking.bbclass
        modified:   conf/layer.conf
        modified:   recipes-containers/containerd/containerd-docker_git.bb
        modified:
recipes-containers/containerd/containerd-opencontainers_git.bb
        modified:   recipes-containers/k3s/README.md
        modified:   recipes-containers/k3s/k3s_git.bb
        modified:   recipes-kernel/linux/linux-yocto/kubernetes.cfg
        modified:   recipes-networking/cni/cni_git.bb
Quoted 7 lines

container-deploy.txt
recipes-core/packagegroups/

Quoted 4 lines

My Image: Linux qemuarm64 by yocto.

The master node can be ready after I started the k3s server. However, the pods in kube-

system (which are essential components for k3s) cannot turn to ready state on qemuarm64.

Quoted 8 lines
Quoted 1 line

That's interesting, since in my configuration, the master never comes ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
qemux86-64   NotReady   master   15h   v1.18.9-k3s1

Hah.

I finally got the node to show up as ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS   ROLES    AGE    VERSION
qemux86-64   Ready    master   112s   v1.18.9-k3s1

I'm attempting to build an all-in-one node, and that is likely causing me some issues.

I'm revisiting those potential conflicts now.

But if anyone else does have an all in one working and has some tips, feel free to share :D

Bruce

Quoted 27 lines

I've sorted out more of the dependencies, and have packagegroups to make them easier now.

Hopefully, I can figure out what is now missing and keeping my master from moving into ready today.

Bruce

Quoted 19 lines

After the master node itself turned to ready state, I check the pods with kubectl:

kubectl get nodes

NAME        STATUS   ROLES    AGE   VERSION
qemuarm64   Ready    master   11m   v1.18.9-k3s1
root@qemuarm64:~# ls
root@qemuarm64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS              RESTARTS   AGE
local-path-provisioner-6d59f47c7-xxvbl   0/1     ContainerCreating   0          12m
coredns-7944c66d8d-tlrm9                 0/1     ContainerCreating   0          12m
metrics-server-7566d596c8-svkff          0/1     ContainerCreating   0          12m
helm-install-traefik-s8p5g               0/1     ContainerCreating   0          12m

Then I describe the pods with:

Events:

  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    16m                  default-scheduler  Successfully assigned kube-

system/coredns-7944c66d8d-tlrm9 to qemuarm64

Quoted 1 line
Quoted 1 line

Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to attach or mount

volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token- b7nlh config-volume]: timed out waiting for the condition

Quoted 1 line
Quoted 1 line

Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach or mount

volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume coredns-token-b7nlh]: timed out waiting for the condition

Quoted 1 line
Quoted 1 line

Warning FailedMount 11s (x16 over 16m) kubelet MountVolume.SetUp failed for

volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable file not found in $PATH

Quoted 2 lines
Quoted 2 lines

I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and

mount binary on my qemuarm64 image:

Quoted 81 lines
Quoted 81 lines
root@qemuarm64:~# echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
root@qemuarm64:~# which mount
/bin/mount

When I type mount command, it worked fine:

/dev/root on / type ext4 (rw,relatime) devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
...
... (skipped the verbose output)

I would like to know whether you have met this "mount" issue ever?

Best Regards,
Lance

Quoted 60 lines

-----Original Message-----
From: [email protected] <[email protected]>
On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield

Ha!!!!

This applies.

Quoted 7 lines

Wonderful, thank you! I guess this is what is called "five times lucky"...

Quoted 3 lines

I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.

Excellent!

I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/log
/?h=k3s-wip

That includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.

I did have a couple of questions on the k3s packaging itself, I was getting the following
error:

ERROR: k3s-v1.18.9+k3s1-dirty-r0 do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3s

So I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?

Bruce

Quoted 13 lines

BR,

/Joakim
--
Joakim Roubert
Senior Engineer

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and

may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Quoted 10 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Quoted 3 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#39 lore raw
Quoted 18 lines · Lance Yang

Hi Bruce,

I pulled your latest k3s-wip branch. In your recipes-networking, the ipset:

cat ipset_6.38.bb
# Copyright (C) 2017 Aaron Brice <[email protected]> # Released under the MIT license (see COPYING.MIT for the terms)

...
... (skip verbose output)

DEPENDS = "libtool libmnl"
RDEPENDS_${PN} = "kernel-module-ip-set"
...

I use bitbake to install k3s and ipset but I got errors below:

Quoted 8 lines

Problem 1: conflicting requests
- nothing provides kernel-module-ip-set needed by ipset-6.38-r0

I think RDEPENDS_${PN} is a slightly over-strict condition in some situations. For example, in the following situation:

In my build/conf/local.conf, I added "kernel-modules" in IMAGE_INSTALL_append section. kernel-modules-ip-set has already included in kernel-modules when I check in qemuarm64 image with "lsmod|grep ip_set". But if RDEPENDS is used, bitbake won't continue to compile the package unless you add "kernel-module-ip-set".

So I suggested to use " RRECOMMENDS_${PN}".

That's already the case in the forked recipe (rdepends), and there's significant issues with getting the right kernel configuration .. so I'm going to leave it as a rdepends, since if you have that not as a module, it means you aren't using the linux-yocto reference configs (or the reference configs are broken), and that there may be other issues in your configuration.

Bruce

Quoted 211 lines

Best Regards,
Lance

Quoted 204 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Tuesday, November 10, 2020 9:17 PM
To: Bruce Ashfield <[email protected]>
Cc: Lance Yang <[email protected]>; Joakim Roubert <[email protected]>; meta- [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 19 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 4 lines · Lance Yang

Hi Bruce and Joakim,

Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.

The branch will be more functional shortly, I have quite a few changes to factor things for k8s and generally more usable :D

        modified:   classes/cni_networking.bbclass
        modified:   conf/layer.conf
        modified:   recipes-containers/containerd/containerd-docker_git.bb
        modified:
recipes-containers/containerd/containerd-opencontainers_git.bb
        modified:   recipes-containers/k3s/README.md
        modified:   recipes-containers/k3s/k3s_git.bb
        modified:   recipes-kernel/linux/linux-yocto/kubernetes.cfg
        modified:   recipes-networking/cni/cni_git.bb
Quoted 7 lines

container-deploy.txt
recipes-core/packagegroups/

Quoted 4 lines

My Image: Linux qemuarm64 by yocto.

The master node can be ready after I started the k3s server. However, the pods in kube-

system (which are essential components for k3s) cannot turn to ready state on qemuarm64.

Quoted 8 lines
Quoted 1 line

That's interesting, since in my configuration, the master never comes ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
qemux86-64   NotReady   master   15h   v1.18.9-k3s1

Hah.

I finally got the node to show up as ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS   ROLES    AGE    VERSION
qemux86-64   Ready    master   112s   v1.18.9-k3s1

I'm attempting to build an all-in-one node, and that is likely causing me some issues.

I'm revisiting those potential conflicts now.

But if anyone else does have an all in one working and has some tips, feel free to share :D

Bruce

Quoted 27 lines

I've sorted out more of the dependencies, and have packagegroups to make them easier now.

Hopefully, I can figure out what is now missing and keeping my master from moving into ready today.

Bruce

Quoted 19 lines

After the master node itself turned to ready state, I check the pods with kubectl:

kubectl get nodes

NAME        STATUS   ROLES    AGE   VERSION
qemuarm64   Ready    master   11m   v1.18.9-k3s1
root@qemuarm64:~# ls
root@qemuarm64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS              RESTARTS   AGE
local-path-provisioner-6d59f47c7-xxvbl   0/1     ContainerCreating   0          12m
coredns-7944c66d8d-tlrm9                 0/1     ContainerCreating   0          12m
metrics-server-7566d596c8-svkff          0/1     ContainerCreating   0          12m
helm-install-traefik-s8p5g               0/1     ContainerCreating   0          12m

Then I describe the pods with:

Events:

  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    16m                  default-scheduler  Successfully assigned kube-

system/coredns-7944c66d8d-tlrm9 to qemuarm64

Quoted 1 line
Quoted 1 line

Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to attach or mount

volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token- b7nlh config-volume]: timed out waiting for the condition

Quoted 1 line
Quoted 1 line

Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach or mount

volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume coredns-token-b7nlh]: timed out waiting for the condition

Quoted 1 line
Quoted 1 line

Warning FailedMount 11s (x16 over 16m) kubelet MountVolume.SetUp failed for

volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable file not found in $PATH

Quoted 2 lines
Quoted 2 lines

I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and

mount binary on my qemuarm64 image:

Quoted 81 lines
Quoted 81 lines
root@qemuarm64:~# echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
root@qemuarm64:~# which mount
/bin/mount

When I type mount command, it worked fine:

/dev/root on / type ext4 (rw,relatime) devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
...
... (skipped the verbose output)

I would like to know whether you have met this "mount" issue ever?

Best Regards,
Lance

Quoted 60 lines

-----Original Message-----
From: [email protected] <[email protected]>
On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield

Ha!!!!

This applies.

Quoted 7 lines

Wonderful, thank you! I guess this is what is called "five times lucky"...

Quoted 3 lines

I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.

Excellent!

I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/log
/?h=k3s-wip

That includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.

I did have a couple of questions on the k3s packaging itself, I was getting the following
error:

ERROR: k3s-v1.18.9+k3s1-dirty-r0 do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3s

So I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?

Bruce

Quoted 13 lines

BR,

/Joakim
--
Joakim Roubert
Senior Engineer

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and

may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Quoted 10 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Quoted 3 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#40 lore raw

PATCH 1/1 k3s: Update README.md

Fix suggestions from markdownlint, i.e.

docker run -it --rm -v $(pwd):/x -w /x \
peterdavehello/markdownlint:0.22.0 markdownlint .

and some formatting done.

Change-Id Ic8f9ccd40607258d934a9510113ae44344253f65
---
 recipes-containers/k3s/README.md | 63 +++++++++++++++++---------------
 1 file changed, 34 insertions(+), 29 deletions(-)
recipes-containers/k3s/README.md +34 -29
index e4cb3e3..70f92a6 100644
@@ -29,35 +29,37 @@ k3s-agent -t <token> -s https://<master>:6443
 (Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the
 k3s master.)
 
-Example:
+### Example
+
 ```shell
 k3s-agent -t /var/lib/rancher/k3s/server/node-token -s https://localhost:6443
 ```
 
-## Notes:
+## Notes
 
-if running under qemu, the default of 256M of memory is not enough, k3s will
-OOM and exit.
+**If running under qemu**, the default of 256 MB of memory is not enough, k3s
+will OOM and exit.
 
-Boot with qemuparams="-m 2048" to boot with 2G of memory (or choose the
+Boot with `qemuparams="-m 2048"` to boot with 2 GB of memory (or choose the
 appropriate amount for your configuration)
 
-Disk: if using qemu and core-image* you'll need to add extra space in your disks
-to ensure containers can start. The following in your image recipe, or local.conf
-would add 2G of extra space to the rootfs:
+Disk: if using qemu and core-image* you will need to add extra space in your
+disks to ensure containers can start. The following in your image recipe,
+or `local.conf` would add 2 GB of extra space to the rootfs:
 
 ```shell
 IMAGE_ROOTFS_EXTRA_SPACE = "2097152"
 ```
 
-## Example output from qemux86-64 running k3s server:
+## Example output from qemux86-64 running k3s server
 
 ```shell
 root@qemux86-64:~# kubectl get nodes
 NAME         STATUS   ROLES    AGE   VERSION
 qemux86-64   Ready    master   46s   v1.18.9-k3s1
+```
 
-
+```shell
 root@qemux86-64:~# kubectl get pods -n kube-system
 NAME                                     READY   STATUS      RESTARTS   AGE
 local-path-provisioner-6d59f47c7-h7lxk   1/1     Running     0          2m32s
@@ -66,7 +68,9 @@ helm-install-traefik-229v7               0/1     Completed   0          2m32s
 coredns-7944c66d8d-9rfj7                 1/1     Running     0          2m32s
 svclb-traefik-pb5j4                      2/2     Running     0          89s
 traefik-758cd5fc85-lxpr8                 1/1     Running     0          89s
+```
 
+```shell
 root@qemux86-64:~# kubectl describe pods -n kube-system
 
 root@qemux86-64:~# ip a s
@@ -74,25 +78,25 @@ root@qemux86-64:~# ip a s
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
     inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
-    inet6 ::1/128 scope host 
+    inet6 ::1/128 scope host
        valid_lft forever preferred_lft forever
 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
     link/ether 52:54:00:12:35:02 brd ff:ff:ff:ff:ff:ff
     inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
        valid_lft forever preferred_lft forever
-    inet6 fec0::5054:ff:fe12:3502/64 scope site dynamic mngtmpaddr 
+    inet6 fec0::5054:ff:fe12:3502/64 scope site dynamic mngtmpaddrg
        valid_lft 86239sec preferred_lft 14239sec
-    inet6 fe80::5054:ff:fe12:3502/64 scope link 
+    inet6 fe80::5054:ff:fe12:3502/64 scope linkg
        valid_lft forever preferred_lft forever
 3: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
     link/sit 0.0.0.0 brd 0.0.0.0
-4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
+4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group defaultg
     link/ether e2:aa:04:89:e6:0a brd ff:ff:ff:ff:ff:ff
     inet 10.42.0.0/32 brd 10.42.0.0 scope global flannel.1
        valid_lft forever preferred_lft forever
-    inet6 fe80::e0aa:4ff:fe89:e60a/64 scope link 
+    inet6 fe80::e0aa:4ff:fe89:e60a/64 scope linkg
        valid_lft forever preferred_lft forever
-5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
+5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group defaultg
     link/ether 02:42:be:3e:25:e7 brd ff:ff:ff:ff:ff:ff
     inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
        valid_lft forever preferred_lft forever
@@ -100,30 +104,31 @@ root@qemux86-64:~# ip a s
     link/ether 82:8e:b4:f8:06:e7 brd ff:ff:ff:ff:ff:ff
     inet 10.42.0.1/24 brd 10.42.0.255 scope global cni0
        valid_lft forever preferred_lft forever
-    inet6 fe80::808e:b4ff:fef8:6e7/64 scope link 
+    inet6 fe80::808e:b4ff:fef8:6e7/64 scope linkg
        valid_lft forever preferred_lft forever
-7: veth82ac482e@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
+7: veth82ac482e@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group defaultg
     link/ether ea:9d:14:c1:00:70 brd ff:ff:ff:ff:ff:ff link-netns cni-c52e6e09-f6e0-a47b-aea3-d6c47d3e2d01
-    inet6 fe80::e89d:14ff:fec1:70/64 scope link 
+    inet6 fe80::e89d:14ff:fec1:70/64 scope linkg
        valid_lft forever preferred_lft forever
-8: vethb94745ed@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
+8: vethb94745ed@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group defaultg
     link/ether 1e:7f:7e:d3:ca:e8 brd ff:ff:ff:ff:ff:ff link-netns cni-86958efe-2462-016f-292d-81dbccc16a83
-    inet6 fe80::8046:3cff:fe23:ced1/64 scope link 
+    inet6 fe80::8046:3cff:fe23:ced1/64 scope linkg
        valid_lft forever preferred_lft forever
-9: veth81ffb276@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
+9: veth81ffb276@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group defaultg
     link/ether 2a:1d:48:54:76:50 brd ff:ff:ff:ff:ff:ff link-netns cni-5d77238e-6452-4fa3-40d2-91d48386080b
-    inet6 fe80::acf4:7fff:fe11:b6f2/64 scope link 
+    inet6 fe80::acf4:7fff:fe11:b6f2/64 scope linkg
        valid_lft forever preferred_lft forever
-10: vethce261f6a@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
+10: vethce261f6a@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group defaultg
     link/ether 72:a3:90:4a:c5:12 brd ff:ff:ff:ff:ff:ff link-netns cni-55675948-77f2-a952-31ce-615f2bdb0093
-    inet6 fe80::4d5:1bff:fe5d:db3a/64 scope link 
+    inet6 fe80::4d5:1bff:fe5d:db3a/64 scope linkg
        valid_lft forever preferred_lft forever
-11: vethee199cf4@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default 
+11: vethee199cf4@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group defaultg
     link/ether e6:90:a4:a3:bc:a1 brd ff:ff:ff:ff:ff:ff link-netns cni-4aeccd16-2976-8a78-b2c4-e028da3bb1ea
-    inet6 fe80::c85a:8bff:fe0b:aea0/64 scope link 
+    inet6 fe80::c85a:8bff:fe0b:aea0/64 scope linkg
        valid_lft forever preferred_lft forever
+```
 
-
+```shell
 root@qemux86-64:~# kubectl describe nodes
 
 Name:               qemux86-64
@@ -235,4 +240,4 @@ Events:
   Normal   NodeHasSufficientPID     10m (x2 over 10m)  kubelet     Node qemux86-64 status is now: NodeHasSufficientPID
   Normal   NodeReady                10m                kubelet     Node qemux86-64 status is now: NodeReady
 
-```shell
+```
Signature · 1 line
2.20.1
#41 lore raw

PATCH 1/1 k3s: Bump to v1.19.3+k3s3

In reply to Bruce Ashfield

The current stable release installed when running the regular k3s install script is now v1.19.3+k3s3. Using that version in our recipe also mitigates the fixes for

https://github.com/rancher/kine/issues/61

Change-Id I4c53d777d4d837628a6c7f8a9234da9d71eb45ae
---
 recipes-containers/k3s/k3s_git.bb | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
recipes-containers/k3s/k3s_git.bb +3 -3
index ea5ed75..28ce66b 100644
@@ -4,7 +4,7 @@ HOMEPAGE = "https://k3s.io/"
 LICENSE = "Apache-2.0"
 LIC_FILES_CHKSUM = "file://${S}/src/import/LICENSE;md5=2ee41112a44fe7014dce33e26468ba93"
 
-SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.18;name=k3s \
+SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.19;name=k3s \
            file://k3s.service \
            file://k3s-agent.service \
            file://k3s-agent \
@@ -13,9 +13,9 @@ SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.18;name=k3s \
            file://0001-Finding-host-local-in-usr-libexec.patch;patchdir=src/import \
           "
 SRC_URI[k3s.md5sum] = "363d3a08dc0b72ba6e6577964f6e94a5"
-SRCREV_k3s = "630bebf94b9dce6b8cd3d402644ed023b3af8f90"
+SRCREV_k3s = "0e4fbfefe1dd8734756dfa4f9ab4fc89665cece4"
 
-PV = "v1.18.9+git${SRCPV}"
+PV = "v1.19.3+git${SRCPV}"
 
 CNI_NETWORKING_FILES ?= "${WORKDIR}/cni-containerd-net.conf"
 
Signature · 1 line
2.20.1
#42 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 4 lines · Joakim Roubert

The current stable release installed when running the regular k3s install script is now v1.19.3+k3s3. Using that version in our recipe also mitigates the fixes for

I already have this queued in my WIP branch.

But without being able to assemble a single node working "cluster", this is still stuck in my staging branch.

Bruce

Quoted 47 lines

https://github.com/rancher/kine/issues/61

Change-Id I4c53d777d4d837628a6c7f8a9234da9d71eb45ae
---
 recipes-containers/k3s/k3s_git.bb | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
recipes-containers/k3s/ renamed from recipes-containers/k3s/k3s_git.bb +0 -0

k3s_git.bb
index ea5ed75..28ce66b 100644

recipes-containers/k3s/k3s_git.bb +0 -0
@@ -4,7 +4,7 @@ HOMEPAGE = "https://k3s.io/"
 LICENSE = "Apache-2.0"
 LIC_FILES_CHKSUM =

"file://${S}/src/import/LICENSE;md5=2ee41112a44fe7014dce33e26468ba93"

-SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.18;name=k3s \
+SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.19;name=k3s

\
            file://k3s.service \
            file://k3s-agent.service \
            file://k3s-agent \
recipes-containers/k3s/k3s_git.bb +2 -2
@@ -13,9 +13,9 @@ SRC_URI = "git://
 github.com/rancher/k3s.git;branch=release-1.18;name=k3s \
 
 file://0001-Finding-host-local-in-usr-libexec.patch;patchdir=src/import \
           "
 SRC_URI[k3s.md5sum] = "363d3a08dc0b72ba6e6577964f6e94a5"
-SRCREV_k3s = "630bebf94b9dce6b8cd3d402644ed023b3af8f90"
+SRCREV_k3s = "0e4fbfefe1dd8734756dfa4f9ab4fc89665cece4"
 
-PV = "v1.18.9+git${SRCPV}"
+PV = "v1.19.3+git${SRCPV}"
 

CNI_NETWORKING_FILES ?= "${WORKDIR}/cni-containerd-net.conf"

--
2.20.1

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
#43 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 2 lines · Bruce Ashfield

I already have this queued in my WIP branch.

Ah, good.

Quoted 1 line

But without being able to assemble a single node working "cluster", this is still stuck in my staging branch.

I am not sure what this single node cluster is. Is it a k3s master running on a single machine, where you can deploy things? What are the current issues that you face there?

BR,

/Joakim

Signature · 7 lines
Joakim Roubert
Senior Engineer

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com
#44 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 4 lines · Joakim Roubert
Quoted 2 lines · Bruce Ashfield

I already have this queued in my WIP branch.

Quoted 8 lines

Ah, good.

Quoted 2 lines

But without being able to assemble a single node working "cluster", this is still stuck in my staging branch.

I am not sure what this single node cluster is. Is it a k3s master running on a single machine, where you can deploy things? What are the

Yes. That's the typical description we've been using of single node "cluster" (I put cluster in quotes, because technically .. it's not a cluster :D

Quoted 2 lines

current issues that you face there?

The issues that we were describing at length in the other longer (more muddled thread). It was very easy to get lost in that thread, so I'll start something new here.

The agent is unable to register with the master due to one of several errors, either certifications or some sort of authentication issue. I've worked through the flannel and other config issues, so they are taken care of at this point.

I'm rebuilding on a rebased master now, and will follow up with better logs when I relaunch my test.

Bruce

Quoted 17 lines

BR,

/Joakim
--
Joakim Roubert
Senior Engineer

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
#45 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 3 lines · Bruce Ashfield

The agent is unable to register with the master due to one of several
errors

But if you have an agent, that would be 2 nodes, right? One master and one worker node?

Quoted 2 lines

I'm rebuilding on a rebased master now, and will follow up with better logs when I relaunch my test.

Excellent!

BR,

/Joakim

#46 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 8 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield

The agent is unable to register with the master due to one of several
errors

But if you have an agent, that would be 2 nodes, right? One master and one worker node?

In kubernetes terms, perhaps. But physical boxes .. no. This needs to be testable under a single instance (qemu in my case). I can use docker / container tricks from within the instance if required (the guides are mixed on that), but it needs to be a single box as I have no way to easily coordinate deployment on multiple boxes for my CI testing.

Bruce

Quoted 13 lines
Quoted 3 lines

I'm rebuilding on a rebased master now, and will follow up with better logs when I relaunch my test.

Excellent!

BR,

/Joakim

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II

A message that is not in the archive (sent off-list or not yet received): [email protected]

#48 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 10 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 5 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield

The agent is unable to register with the master due to one of several
errors

Quoted 10 lines
Quoted 3 lines

But if you have an agent, that would be 2 nodes, right? One master and one worker node?

In kubernetes terms, perhaps. But physical boxes .. no. This needs to be testable under a single instance (qemu in my case). I can use docker / container tricks from within the instance if required (the guides are mixed on that), but it needs to be a single box as I have no way to easily coordinate deployment on multiple boxes for my CI testing.

I should also say that this is how I've been testing k8s (but that also needs work again), so it has been up and running in the past.

Bruce

Quoted 27 lines

Bruce

Quoted 13 lines
Quoted 3 lines

I'm rebuilding on a rebased master now, and will follow up with better logs when I relaunch my test.

Excellent!

BR,

/Joakim

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
#49 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 7 lines · Bruce Ashfield

But if you have an agent, that would be 2 nodes, right? One master and
one worker node?

In kubernetes terms, perhaps. But physical boxes .. no. This needs to be testable under a single instance (qemu in my case).

I see. Well, I have never tried that, but would in that case use one qemu instance for the master and a separate qemu instance for each worker node. (I should try to run both master and a worker on one machine and see what happens on my setup.)

BR,

/Joakim

#50 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 6 lines · Joakim Roubert
Quoted 5 lines · Bruce Ashfield

But if you have an agent, that would be 2 nodes, right? One master and
one worker node?

Quoted 8 lines
Quoted 2 lines

In kubernetes terms, perhaps. But physical boxes .. no. This needs to be testable under a single instance (qemu in my case).

I see. Well, I have never tried that, but would in that case use one qemu instance for the master and a separate qemu instance for each worker node. (I should try to run both master and a worker on one machine and see what happens on my setup.)

I'm experimenting with two qemu sessions as well (I had to do this with meta-openstack), but then host networking becomes an issue and if that's also the box you are doing builds on (like I am), it can cause issues.

Either way, I'm fairly close now and hopefully will have this sorted out soon.

My build is still running, but I'll post specific logs later today.

Bruce

Quoted 9 lines

BR,

/Joakim

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II

A message that is not in the archive (sent off-list or not yet received): [email protected]

#52 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Here's problem #1, using qemux86-64 and the latest on my k3s-wip branch, while the master shows itself as ready, the pods are crashing. I need to sort that out before I can get into the single node config.

root@qemux86-64:~# uname -a

Linux qemux86-64 5.8.13-yocto-standard #1 SMP PREEMPT Tue Oct 6 12:23:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

root@qemux86-64:~# k3s --version

k3s version v1.19.3+gitAUTOINC+970fbc66d3 (970fbc66)

root@qemux86-64:~# kubectl get nodes

NAME STATUS ROLES AGE VERSION

qemux86-64 Ready master 17m v1.19.3-k3s1

And flannel is up:

5: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default

    link/ether be:5e:55:3e:52:e6 brd ff:ff:ff:ff:ff:ff

    inet 10.42.0.0/32 scope global flannel.1

       valid_lft forever preferred_lft forever

    inet6 fe80::bc5e:55ff:fe3e:52e6/64 scope link

       valid_lft forever preferred_lft forever

6: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000

    link/ether e2:38:35:11:31:6e brd ff:ff:ff:ff:ff:ff

    inet 10.42.0.1/24 brd 10.42.0.255 scope global cni0

       valid_lft forever preferred_lft forever

    inet6 fe80::e038:35ff:fe11:316e/64 scope link

       valid_lft forever preferred_lft forever

1081: veth89c8ca0d@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default

    link/ether b6:76:4f:77:e7:a6 brd ff:ff:ff:ff:ff:ff link-netns

cni-255cf837-ca3c-f08d-730f-bbb359cb6ae1

inet6 fe80::8cf:bff:febb:85d7/64 scope link

valid_lft forever preferred_lft forever

1082: veth001513a2@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default

    link/ether 52:5c:64:41:c6:e1 brd ff:ff:ff:ff:ff:ff link-netns

cni-71490108-5f0a-a266-de1f-4bed565dcf88

inet6 fe80::605b:23ff:fe50:c22f/64 scope link

valid_lft forever preferred_lft forever

1083: veth4ae47008@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default

    link/ether be:0e:af:76:4d:66 brd ff:ff:ff:ff:ff:ff link-netns

cni-bb1a1e05-6fa1-e9dd-3de8-b3a4be47197a

inet6 fe80::8467:31ff:fec3:6b20/64 scope link

valid_lft forever preferred_lft forever

1084: veth07cd05c1@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default

    link/ether 5e:75:7b:bc:ee:1c brd ff:ff:ff:ff:ff:ff link-netns

cni-07e997d6-7918-0f38-b1e5-1b08e9ac0581

inet6 fe80::28e0:ceff:fe42:364d/64 scope link tentative
valid_lft forever preferred_lft forever

root@qemux86-64:~# kubectl get pods -n kube-system

NAME READY STATUS RESTARTS
AGE

local-path-provisioner-7ff9579c6-bd947 0/1 CrashLoopBackOff 8

    22m

coredns-66c464876b-76bdl 0/1 CrashLoopBackOff 8

    22m

helm-install-traefik-mvt5k 0/1 CrashLoopBackOff 8

    22m

metrics-server-7b4f8b595-ck5kz 0/1 CrashLoopBackOff 8

    22m

root@qemux86-64:~#

Those failures are a bit different than I used to see before going to 1.19

For whatever reason, my logs are empty, so I'm trying to see if my k3s host image is missing something.

Bruce

Quoted 52 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 6 lines · Joakim Roubert
Quoted 5 lines · Bruce Ashfield

But if you have an agent, that would be 2 nodes, right? One master and
one worker node?

Quoted 8 lines
Quoted 2 lines

In kubernetes terms, perhaps. But physical boxes .. no. This needs to be testable under a single instance (qemu in my case).

I see. Well, I have never tried that, but would in that case use one qemu instance for the master and a separate qemu instance for each worker node. (I should try to run both master and a worker on one machine and see what happens on my setup.)

I'm experimenting with two qemu sessions as well (I had to do this with meta-openstack), but then host networking becomes an issue and if that's also the box you are doing builds on (like I am), it can cause issues.

Either way, I'm fairly close now and hopefully will have this sorted out soon.

My build is still running, but I'll post specific logs later today.

Bruce

Quoted 9 lines

BR,

/Joakim

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
#53 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 8 lines · Bruce Ashfield
root@qemux86-64:~# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
local-path-provisioner-7ff9579c6-bd947 0/1 CrashLoopBackOff 822m
coredns-66c464876b-76bdl 0/1 CrashLoopBackOff 822m
helm-install-traefik-mvt5k 0/1 CrashLoopBackOff 822m
metrics-server-7b4f8b595-ck5kz 0/1 CrashLoopBackOff 822m
root@qemux86-64:~#

From my own experience with CrashLoopBackOff situations is that there may be serveral different things causing that. Can you trace down what is causing the CrashLoopBackOff here?

BR,

/Joakim

#54 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 10 lines · Joakim Roubert
Quoted 8 lines · Bruce Ashfield
root@qemux86-64:~# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
local-path-provisioner-7ff9579c6-bd947 0/1 CrashLoopBackOff 822m
coredns-66c464876b-76bdl 0/1 CrashLoopBackOff 822m
helm-install-traefik-mvt5k 0/1 CrashLoopBackOff 822m
metrics-server-7b4f8b595-ck5kz 0/1 CrashLoopBackOff 822m
root@qemux86-64:~#
Quoted 4 lines

From my own experience with CrashLoopBackOff situations is that there
may be serveral different things causing that. Can you trace down what is causing the CrashLoopBackOff here?

Not really. The build seems to have some sort of misconfiguration and no logs are created (and hence can't be accessed via kubectl).

Running the server on the command line just gets us reams of messages and a lot of false leads (too many nameservers, iptables --random-fully not supported, etc ).

I've tried a few variants today, but can't get any really simple logs out. I'll do more tomorrow.

I've been able to confirm that my packagegroup/image type runs docker, containerd, and runc launched containers just fine, so this is just some quirk/idiocy of k*s that is causing the problem.

Bruce

Quoted 8 lines

BR,

/Joakim

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
#55 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Hi Bruce and Joakim,

I recently saw a new branch master-next has been created in meta-virtualization. It mainly related to k3s and I saw k3s had been bumped to v1.19. I would like to know if the k3s patch will be merged into master branch soon. I remembered you mentioned k3s v1.18 has some agent registration issue and some had crash loop issues.

What is the current status of the k3s patch? Is there any further issues?

Best Regards,
Lance
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Thursday, November 19, 2020 4:39 AM
To: Joakim Roubert <[email protected]>
Cc: meta-virtualization <[email protected]>
Subject: Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

On Wed, Nov 18, 2020 at 1:27 PM Joakim Roubert <[email protected]<mailto:[email protected]>> wrote:

Quoted 8 lines · Bruce Ashfield
root@qemux86-64:~# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
local-path-provisioner-7ff9579c6-bd947 0/1 CrashLoopBackOff 822m
coredns-66c464876b-76bdl 0/1 CrashLoopBackOff 822m
helm-install-traefik-mvt5k 0/1 CrashLoopBackOff 822m
metrics-server-7b4f8b595-ck5kz 0/1 CrashLoopBackOff 822m
root@qemux86-64:~#

From my own experience with CrashLoopBackOff situations is that there
may be serveral different things causing that. Can you trace down what is causing the CrashLoopBackOff here?

Not really. The build seems to have some sort of misconfiguration and no logs are created (and hence can't be accessed via kubectl).

Running the server on the command line just gets us reams of messages and a lot of false leads (too many nameservers, iptables --random-fully not supported, etc ).

I've tried a few variants today, but can't get any really simple logs out. I'll do more tomorrow.

I've been able to confirm that my packagegroup/image type runs docker, containerd, and runc launched containers just fine, so this is just some quirk/idiocy of k*s that is causing the problem.

Bruce

BR,

/Joakim

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#56 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 10 lines · Lance Yang

Hi Bruce and Joakim,

I recently saw a new branch master-next has been created in meta-virtualization. It mainly related to k3s and I saw k3s had been bumped to v1.19. I would like to know if the k3s patch will be merged into master branch soon. I remembered you mentioned k3s v1.18 has some agent registration issue and some had crash loop issues.

What is the current status of the k3s patch? Is there any further issues?

It is just in master-next, to show that some cleanups have been done, and the supporting component version bumps that I've done, but no, it still doesn't work under qemux86-64, so it can't actually merge to master.

The node becomes ready, but all the core containers go into a crashbackoff loop, kubectl logs doesn't work, and we still can't register a client with it (likely because of symptom #1).

I'm also chasing down an iptables issue (that I had previously fixed), but should have that sorted shortly.

Bruce

Quoted 72 lines

Best Regards,

Lance

From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Thursday, November 19, 2020 4:39 AM
To: Joakim Roubert <[email protected]>
Cc: meta-virtualization <[email protected]>
Subject: Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

On Wed, Nov 18, 2020 at 1:27 PM Joakim Roubert <[email protected]> wrote:

Quoted 8 lines · Bruce Ashfield
root@qemux86-64:~# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
local-path-provisioner-7ff9579c6-bd947 0/1 CrashLoopBackOff 822m
coredns-66c464876b-76bdl 0/1 CrashLoopBackOff 822m
helm-install-traefik-mvt5k 0/1 CrashLoopBackOff 822m
metrics-server-7b4f8b595-ck5kz 0/1 CrashLoopBackOff 822m
root@qemux86-64:~#

From my own experience with CrashLoopBackOff situations is that there
may be serveral different things causing that. Can you trace down what is causing the CrashLoopBackOff here?

Not really. The build seems to have some sort of misconfiguration and no logs are created (and hence can't be accessed via kubectl).

Running the server on the command line just gets us reams of messages and a lot of false leads (too many nameservers, iptables --random-fully not supported, etc ).

I've tried a few variants today, but can't get any really simple logs out. I'll do more tomorrow.

I've been able to confirm that my packagegroup/image type runs docker, containerd, and runc launched containers just fine, so this is just some quirk/idiocy of k*s that is causing the problem.

Bruce

BR,

/Joakim

--

- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#57 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Hi Bruce,

Thanks for your quick reply.

Yesterday, I pulled meta-virtualization master-next branch to my local server and trying to build k3s with qemux86-64 image. In master-next branch, I found you had created DISTRO_FEATURE check for k3s and some package groups for k3s. E.g. packagegroup-k3s-node, packagegroup-k3s-host.

I configured my local.conf to install k3s with following configurations:

DISTRO_FEATURES_append = " k3s virtualization"
IMAGE_INSTALL_append = " packagegroup-k3s-node packagegroup-k3s-host cgroup-lite kernel-modules"

However, when I use:

bitbake core-image-minimal

to compile a minimal image to test k3s. I encountered the following error message:

gcc: error: unrecognized command line option ‘-fmacro-prefix-map=/path/to/build/tmp/work/core2-64-poky-linux/libvirt/6.3.0-r0=/usr/src/debug/libvirt/6.3.0-r0’

| error: command '/home/lance/repos/build/tmp/hosttools/gcc' failed with exit code 1
| WARNING: exit code 1 from a shell command.
| ERROR: Execution of '/path/to/build/tmp/work/core2-64-poky-linux/libvirt/6.3.0-r0/temp/run.do_compile.162048' failed with exit code 1:

It seems the error was related to libvirt. I two three questions here. If you need more log details, please let me know.

1. Do I mis-config anything so that the error above appeared? Could you please share your configuration when you compile the image with k3s?
2. Do we need libvirt when we install k3s? From my own experience, libvirt is for virtualization. When I run k3s on bare-metal, I don't think I install a libvirt component. And before, I ran a k3s successfully without libvirt on aarch64 Yocto Linux.

Once I successfully built the qemux86-64 image, I am willing to spare some effort to reproduce the bugs you mentioned in the previous email. Would you mind sharing your config to me?

Best Regards,
Lance

Quoted 128 lines

-----Original Message-----
From: Bruce Ashfield <[email protected]>
Sent: Friday, December 11, 2020 9:44 PM
To: Lance Yang <[email protected]>
Cc: Joakim Roubert <[email protected]>; meta-virtualization <meta- [email protected]>; nd <[email protected]>; Michael Zhao <[email protected]>
Subject: Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 6 lines · Lance Yang

Hi Bruce and Joakim,

I recently saw a new branch master-next has been created in meta-

virtualization. It mainly related to k3s and I saw k3s had been bumped to v1.19. I would like to know if the k3s patch will be merged into master branch soon. I remembered you mentioned k3s v1.18 has some agent registration issue and some had crash loop issues.

Quoted 4 lines

What is the current status of the k3s patch? Is there any further issues?

It is just in master-next, to show that some cleanups have been done, and the supporting component version bumps that I've done, but no, it still doesn't work under qemux86-64, so it can't actually merge to master.

The node becomes ready, but all the core containers go into a crashbackoff loop, kubectl logs doesn't work, and we still can't register a client with it (likely because of symptom #1).

I'm also chasing down an iptables issue (that I had previously fixed), but should have that sorted shortly.

Bruce

Quoted 22 lines

Best Regards,

Lance

From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Thursday, November 19, 2020 4:39 AM
To: Joakim Roubert <[email protected]>
Cc: meta-virtualization <[email protected]>
Subject: Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

On Wed, Nov 18, 2020 at 1:27 PM Joakim Roubert

Quoted 4 lines · <[email protected]> wrote:
Quoted 2 lines · Bruce Ashfield
root@qemux86-64:~# kubectl get pods -n kube-system NAME READY

STATUS

Quoted 7 lines
Quoted 6 lines

RESTARTS AGE
local-path-provisioner-7ff9579c6-bd947 0/1 CrashLoopBackOff 822m coredns-66c464876b-76bdl 0/1 CrashLoopBackOff 822m helm-install-traefik-mvt5k 0/1 CrashLoopBackOff 822m metrics-server-7b4f8b595-ck5kz 0/1 CrashLoopBackOff 822m root@qemux86-64:~#

Quoted 7 lines

From my own experience with CrashLoopBackOff situations is that there
may be serveral different things causing that. Can you trace down what is causing the CrashLoopBackOff here?

Not really. The build seems to have some sort of misconfiguration and no

logs are created (and hence can't be accessed via kubectl).

Quoted 4 lines

Running the server on the command line just gets us reams of messages

and a lot of false leads (too many nameservers, iptables --random-fully not supported, etc ).

Quoted 4 lines

I've tried a few variants today, but can't get any really simple logs out. I'll do

more tomorrow.

Quoted 4 lines

I've been able to confirm that my packagegroup/image type runs docker,

containerd, and runc launched containers just fine, so this is just some quirk/idiocy of k*s that is causing the problem.

Quoted 16 lines

Bruce

BR,

/Joakim

Quoted 7 lines

--

- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are

confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Quoted 4 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#58 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 18 lines · Lance Yang

Hi Bruce,

Thanks for your quick reply.

Yesterday, I pulled meta-virtualization master-next branch to my local server and trying to build k3s with qemux86-64 image. In master-next branch, I found you had created DISTRO_FEATURE check for k3s and some package groups for k3s. E.g. packagegroup-k3s-node, packagegroup-k3s-host.

I configured my local.conf to install k3s with following configurations:

DISTRO_FEATURES_append = " k3s virtualization"
IMAGE_INSTALL_append = " packagegroup-k3s-node packagegroup-k3s-host cgroup-lite kernel-modules"

However, when I use:

bitbake core-image-minimal

to compile a minimal image to test k3s. I encountered the following error message:

Quoted 8 lines

gcc: error: unrecognized command line option ‘-fmacro-prefix-map=/path/to/build/tmp/work/core2-64-poky-linux/libvirt/6.3.0-r0=/usr/src/debug/libvirt/6.3.0-r0’

| error: command '/home/lance/repos/build/tmp/hosttools/gcc' failed with exit code 1
| WARNING: exit code 1 from a shell command.
| ERROR: Execution of '/path/to/build/tmp/work/core2-64-poky-linux/libvirt/6.3.0-r0/temp/run.do_compile.162048' failed with exit code 1:

It seems the error was related to libvirt. I two three questions here. If you need more log details, please let me know.

1. Do I mis-config anything so that the error above appeared? Could you please share your configuration when you compile the image with k3s?

The packagegroups are still a work in progress, and I actually have some changes to them that aren't on that branch yet. In particular docker has been removed as a dependency and libvirt has been shuffled to an optional packagegroup.

That being said, I just built libvirt on two different machines on an up to date master branch for all layers, and didn't get that same error. So

You don't need the packagegroups for testing, but once I have a working configuration, it will be split and captured into package lists for re-use across many of the different configurations.

So I'd suggest dropping it and ignoring that failure for now (since) I can't reproduce it.

Quoted 4 lines

2. Do we need libvirt when we install k3s? From my own experience, libvirt is for virtualization. When I run k3s on bare-metal, I don't think I install a libvirt component. And before, I ran a k3s successfully without libvirt on aarch64 Yocto Linux.

Once I successfully built the qemux86-64 image, I am willing to spare some effort to reproduce the bugs you mentioned in the previous email. Would you mind sharing your config to me?

I've actually tracked this down to something in our build of the k3s binary. I can bring up a node by substituting just the k3s all-in-one executable. There have been issues with the go compiler reported in master, so I'm doing a full rebuild now, in case the linker is causing the problem. I've looked all through the build instructions from the k3s repository, and nothing obvious is different between the two (except possibly the compiler) .. I'm building the exact same git hash of the one that works.

For now, I'd suggest waiting a couple more days, since I may have it solved shortly.

Bruce

Quoted 131 lines

Best Regards,
Lance

Quoted 128 lines

-----Original Message-----
From: Bruce Ashfield <[email protected]>
Sent: Friday, December 11, 2020 9:44 PM
To: Lance Yang <[email protected]>
Cc: Joakim Roubert <[email protected]>; meta-virtualization <meta- [email protected]>; nd <[email protected]>; Michael Zhao <[email protected]>
Subject: Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 6 lines · Lance Yang

Hi Bruce and Joakim,

I recently saw a new branch master-next has been created in meta-

virtualization. It mainly related to k3s and I saw k3s had been bumped to v1.19. I would like to know if the k3s patch will be merged into master branch soon. I remembered you mentioned k3s v1.18 has some agent registration issue and some had crash loop issues.

Quoted 4 lines

What is the current status of the k3s patch? Is there any further issues?

It is just in master-next, to show that some cleanups have been done, and the supporting component version bumps that I've done, but no, it still doesn't work under qemux86-64, so it can't actually merge to master.

The node becomes ready, but all the core containers go into a crashbackoff loop, kubectl logs doesn't work, and we still can't register a client with it (likely because of symptom #1).

I'm also chasing down an iptables issue (that I had previously fixed), but should have that sorted shortly.

Bruce

Quoted 22 lines

Best Regards,

Lance

From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Thursday, November 19, 2020 4:39 AM
To: Joakim Roubert <[email protected]>
Cc: meta-virtualization <[email protected]>
Subject: Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

On Wed, Nov 18, 2020 at 1:27 PM Joakim Roubert

Quoted 4 lines · <[email protected]> wrote:
Quoted 2 lines · Bruce Ashfield
root@qemux86-64:~# kubectl get pods -n kube-system NAME READY

STATUS

Quoted 7 lines
Quoted 6 lines

RESTARTS AGE
local-path-provisioner-7ff9579c6-bd947 0/1 CrashLoopBackOff 822m coredns-66c464876b-76bdl 0/1 CrashLoopBackOff 822m helm-install-traefik-mvt5k 0/1 CrashLoopBackOff 822m metrics-server-7b4f8b595-ck5kz 0/1 CrashLoopBackOff 822m root@qemux86-64:~#

Quoted 7 lines

From my own experience with CrashLoopBackOff situations is that there
may be serveral different things causing that. Can you trace down what is causing the CrashLoopBackOff here?

Not really. The build seems to have some sort of misconfiguration and no

logs are created (and hence can't be accessed via kubectl).

Quoted 4 lines

Running the server on the command line just gets us reams of messages

and a lot of false leads (too many nameservers, iptables --random-fully not supported, etc ).

Quoted 4 lines

I've tried a few variants today, but can't get any really simple logs out. I'll do

more tomorrow.

Quoted 4 lines

I've been able to confirm that my packagegroup/image type runs docker,

containerd, and runc launched containers just fine, so this is just some quirk/idiocy of k*s that is causing the problem.

Quoted 16 lines

Bruce

BR,

/Joakim

Quoted 7 lines

--

- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are

confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Quoted 5 lines
Quoted 4 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

#59 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 3 lines · Bruce Ashfield

For now, I'd suggest waiting a couple more days, since I may have it solved shortly.

If there continues to be strange problems, perhaps the Rancher guys would be interested in helping out? They are both skilled and friendly, and given Rancher's focused agenda on being _the_ Kubernetes provider for edge with k3s, I guess having k3s in meta-virtualization would be attractive to them.

BR,

/Joakim

#60 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 11 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield

For now, I'd suggest waiting a couple more days, since I may have it solved shortly.

If there continues to be strange problems, perhaps the Rancher guys would be interested in helping out? They are both skilled and friendly, and given Rancher's focused agenda on being _the_ Kubernetes provider for edge with k3s, I guess having k3s in meta-virtualization would be attractive to them.

I have reached out to them, but had to put down my debug on this for a few days as I prepared 5.10 as the new reference kernel for oe-core.

I'm almost through that, and will cycle back to this shortly.

The bottom line is that only the rancher build k3s binary works with our constructed rootfs (in my tests).

If I use our built binary, or one that I build outside of yocto (natively, on my build server), we end up in crashloops on all the core containers. The binaries are built from the identical hash (for yocto) and using the official build container images / process on my workstation .. same result. So that rules out something in the kernel (and its config), the go version or something in our rootfs being missing. But as soon as you fetch the rancher binary, restart the services .. the containers come up just fine.

And that unfortunately means that it is difficult to get help from the project, since they don't have the right understanding of the runtime environment (but I'm trying!).

I'm diving deeper to try and sort it out.

Bruce

Quoted 5 lines

BR,

/Joakim

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#61 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 1 line · Bruce Ashfield

So that rules out something in the kernel (and its config), the go version or something in our rootfs being missing.

That surely narrows things down.

Quoted 2 lines

But as soon as you fetch the rancher binary, restart the services .. the containers come up just fine.

Is this the same thing that happened for the k3s v1.18.9 build too?

Quoted 1 line

I'm diving deeper to try and sort it out.

👍

BR,

/Joakim

#62 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 5 lines · Joakim Roubert
Quoted 2 lines · Bruce Ashfield

So that rules out something in the kernel (and its config), the go version or something in our rootfs being missing.

Quoted 7 lines

That surely narrows things down.

Quoted 2 lines

But as soon as you fetch the rancher binary, restart the services .. the containers come up just fine.

Is this the same thing that happened for the k3s v1.18.9 build too?

Yep. It is consistent with all versions that I've built, including the pre-release master branch as well. I was hoping for some "uprev magic" to fix it, but that wasn't the case :)

I'm back from the holidays now, and will be back into this shortly.

Bruce

Quoted 7 lines
Quoted 1 line

I'm diving deeper to try and sort it out.

BR,

/Joakim

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

A message that is not in the archive (sent off-list or not yet received): [email protected]

#64 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 12 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 10 lines · Joakim Roubert
Quoted 2 lines · Bruce Ashfield

So that rules out something in the kernel (and its config), the go version or something in our rootfs being missing.

That surely narrows things down.

Quoted 2 lines

But as soon as you fetch the rancher binary, restart the services .. the containers come up just fine.

Quoted 9 lines
Quoted 2 lines

Is this the same thing that happened for the k3s v1.18.9 build too?

Yep. It is consistent with all versions that I've built, including the pre-release master branch as well. I was hoping for some "uprev magic" to fix it, but that wasn't the case :)

I'm back from the holidays now, and will be back into this shortly.

I do have one more thing I'd like to try (I didn't get much debug done yesterday), do you happen to have a k3s binary that works in your environment ?

I'd like to toss it into my image and see if it behaves differently from my builds and the rancher builds of k3s.

Bruce

Quoted 19 lines

Bruce

Quoted 7 lines
Quoted 1 line

I'm diving deeper to try and sort it out.

BR,

/Joakim

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

A message that is not in the archive (sent off-list or not yet received): [email protected]

#66 lore raw

Re: [meta-virtualization][PATCH] k3s: Bump to v1.19.3+k3s3

Quoted 2 lines · Joakim Roubert

The current stable release installed when running the regular k3s install script is now v1.19.3+k3s3.

I launched a multi-arch cluster with this v1.19.3+k3s3, deployed Linkerd 2.9.0 on it and then meshed a small test application sending traffic back and forth between a client and a server, and it all seems to work quite nicely.

BR,

/Joakim

Signature · 7 lines
Joakim Roubert
Senior Engineer

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com
#67 lore raw
In reply to Lance Yang
Quoted 3 lines · Lance Yang

Problem 1: conflicting requests
  - nothing provides kernel-module-ip-set needed by ipset-6.38-r0

Do you have

CONFIG_IP_SET=m

in your kernel config?

BR,

/Joakim

#68 lore raw

Hi Joakim,

Please see the comments inline.

Quoted 7 lines

-----Original Message-----
From: Joakim Roubert <[email protected]>
Sent: Thursday, November 12, 2020 9:43 PM
To: Lance Yang <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 8 lines
Quoted 3 lines · Lance Yang

Problem 1: conflicting requests
- nothing provides kernel-module-ip-set needed by ipset-6.38-r0

Do you have

CONFIG_IP_SET=m

[ Uh! That works! I thought I set it but when I double checked the menuconfig, I found I forgot to set it.

As far as I know, ipset is a tool to simplify iptables configurations. (e.g. iptables -t nat -A PREROUTING -m set --match-set podnet dst -j xxx) which decreases the number of iptables rules.

Because of my limited knowledge of k3s/k8s, I have a questions about ipset.

Is it a must installed component for k3s/k8s?
]

Quoted 6 lines

in your kernel config?

BR,

/Joakim

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#69 lore raw
Quoted 20 lines · Lance Yang

Hi Joakim,

Please see the comments inline.

Quoted 7 lines

-----Original Message-----
From: Joakim Roubert <[email protected]>
Sent: Thursday, November 12, 2020 9:43 PM
To: Lance Yang <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 8 lines
Quoted 3 lines · Lance Yang

Problem 1: conflicting requests
- nothing provides kernel-module-ip-set needed by ipset-6.38-r0

Do you have

CONFIG_IP_SET=m

Quoted 7 lines

[ Uh! That works! I thought I set it but when I double checked the menuconfig, I found I forgot to set it.

As far as I know, ipset is a tool to simplify iptables configurations. (e.g. iptables -t nat -A PREROUTING -m set --match-set podnet dst -j xxx) which decreases the number of iptables rules.

Because of my limited knowledge of k3s/k8s, I have a questions about ipset.

Is it a must installed component for k3s/k8s?

Check my reviews on the mailing list, you'll see where it is discussed.

Bruce

Quoted 8 lines

]

Quoted 6 lines

in your kernel config?

BR,

/Joakim

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#70 lore raw
In reply to Bruce Ashfield
Quoted 3 lines · Bruce Ashfield

But if anyone else does have an all in one working and has some tips, feel free to share :D

When it comes to mount issues as were mentioned here earlier, I have seen those when I have has /var/lib/rancher/k3s mounted on overlayfs file systems (which does not work when the containers are unpacking their layers). Mounting another file system (e.g. ext4 or someting) on /var/lib/rancher/k3s has mitigated such errors.

BR,

/Joakim

A message that is not in the archive (sent off-list or not yet received): [email protected]

#72 lore raw
Quoted 40 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 26 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 4 lines · Lance Yang

Hi Bruce and Joakim,

Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.

The branch will be more functional shortly, I have quite a few changes to factor things for
k8s and generally more usable :D

        modified:   classes/cni_networking.bbclass
        modified:   conf/layer.conf
        modified:   recipes-containers/containerd/containerd-docker_git.bb
        modified:
recipes-containers/containerd/containerd-opencontainers_git.bb
        modified:   recipes-containers/k3s/README.md
        modified:   recipes-containers/k3s/k3s_git.bb
        modified:   recipes-kernel/linux/linux-yocto/kubernetes.cfg
        modified:   recipes-networking/cni/cni_git.bb

container-deploy.txt
recipes-core/packagegroups/

Quoted 3 lines

My Image: Linux qemuarm64 by yocto.

Quoted 9 lines
Quoted 2 lines

The master node can be ready after I started the k3s server. However, the pods in kube-system (which are essential components for k3s) cannot turn to ready state on qemuarm64.

That's interesting, since in my configuration, the master never comes ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
qemux86-64   NotReady   master   15h   v1.18.9-k3s1
Quoted 8 lines

Hah.

I finally got the node to show up as ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS   ROLES    AGE    VERSION
qemux86-64   Ready    master   112s   v1.18.9-k3s1

Lance,

What image type were you building ? I'm pulling in dependencies to packagegroups and the recipes themselves.

I'm not seeing the mount issue on my master/server node:

root@qemux86-64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS      RESTARTS   AGE
local-path-provisioner-6d59f47c7-h7lxk   1/1     Running     0          3m32s
metrics-server-7566d596c8-mwntr          1/1     Running     0          3m32s
helm-install-traefik-229v7               0/1     Completed   0          3m32s
coredns-7944c66d8d-9rfj7                 1/1     Running     0          3m32s
svclb-traefik-pb5j4                      2/2     Running     0          2m29s
traefik-758cd5fc85-lxpr8                 1/1     Running     0          2m29s

I'm going back to all-in-one node debugging, but can look into the mount issue more later.

Bruce

Quoted 146 lines

I'm attempting to build an all-in-one node, and that is likely causing me some issues.

I'm revisiting those potential conflicts now.

But if anyone else does have an all in one working and has some tips, feel free to share :D

Bruce

Quoted 116 lines

I've sorted out more of the dependencies, and have packagegroups to make them easier
now.

Hopefully, I can figure out what is now missing and keeping my master from moving into
ready today.

Bruce

Quoted 106 lines

After the master node itself turned to ready state, I check the pods with kubectl:

kubectl get nodes

NAME        STATUS   ROLES    AGE   VERSION
qemuarm64   Ready    master   11m   v1.18.9-k3s1
root@qemuarm64:~# ls
root@qemuarm64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS              RESTARTS   AGE
local-path-provisioner-6d59f47c7-xxvbl   0/1     ContainerCreating   0          12m
coredns-7944c66d8d-tlrm9                 0/1     ContainerCreating   0          12m
metrics-server-7566d596c8-svkff          0/1     ContainerCreating   0          12m
helm-install-traefik-s8p5g               0/1     ContainerCreating   0          12m

Then I describe the pods with:

Events:

  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    16m                  default-scheduler  Successfully assigned kube-system/coredns-7944c66d8d-tlrm9 to qemuarm64
  Warning  FailedMount  5m23s (x3 over 14m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token-b7nlh config-volume]: timed out waiting for the condition
  Warning  FailedMount  50s (x4 over 12m)    kubelet            Unable to attach or mount volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume coredns-token-b7nlh]: timed out waiting for the condition
  Warning  FailedMount  11s (x16 over 16m)   kubelet            MountVolume.SetUp failed for volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable file not found in $PATH

I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and mount binary on my qemuarm64 image:

root@qemuarm64:~# echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
root@qemuarm64:~# which mount
/bin/mount

When I type mount command, it worked fine:

/dev/root on / type ext4 (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
...
... (skipped the verbose output)

I would like to know whether you have met this "mount" issue ever?

Best Regards,
Lance

Quoted 60 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield

Ha!!!!

This applies.

Quoted 7 lines

Wonderful, thank you! I guess this is what is called "five times lucky"...

Quoted 3 lines

I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.

Excellent!

I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/log/?h=k3s-wip

That includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.

I did have a couple of questions on the k3s packaging itself, I was getting the following error:

ERROR: k3s-v1.18.9+k3s1-dirty-r0 do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3s

So I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?

Bruce

Quoted 8 lines

BR,

/Joakim
--
Joakim Roubert
Senior Engineer

Quoted 5 lines

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Quoted 11 lines
Quoted 1 line

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#73 lore raw

Hi Bruce,

Thanks for your reply. It took me a long time to address the $PATH issue.

I found the cause after I cleaned my environment. it is because I set my own $PATH before k3s startup which did not include "/bin" and some components in k3s will use "os.Getenv("PATH") to search the related tool.

Since some OS may not have systemd and the k3s-clean script seems not to delete all interfaces created by cni and umount all mount points related to k3s. I wrote a script for this situation and I will send it through a separate email based on your previous scripts.

Best Regards,
Lance

Quoted 241 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Tuesday, November 10, 2020 9:35 PM
To: Bruce Ashfield <[email protected]>
Cc: Lance Yang <[email protected]>; Joakim Roubert <[email protected]>; meta- [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 27 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 23 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 4 lines · Lance Yang

Hi Bruce and Joakim,

Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.

The branch will be more functional shortly, I have quite a few changes to factor things for k8s and generally more usable :D

        modified:   classes/cni_networking.bbclass
        modified:   conf/layer.conf
        modified:   recipes-containers/containerd/containerd-docker_git.bb
        modified:
recipes-containers/containerd/containerd-opencontainers_git.bb
        modified:   recipes-containers/k3s/README.md
        modified:   recipes-containers/k3s/k3s_git.bb
        modified:   recipes-kernel/linux/linux-yocto/kubernetes.cfg
        modified:   recipes-networking/cni/cni_git.bb

container-deploy.txt
recipes-core/packagegroups/

Quoted 1 line
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

My Image: Linux qemuarm64 by yocto.

The master node can be ready after I started the k3s server. However, the pods in kube-

system (which are essential components for k3s) cannot turn to ready state on qemuarm64.

Quoted 9 lines
Quoted 8 lines
Quoted 1 line

That's interesting, since in my configuration, the master never comes ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
qemux86-64   NotReady   master   15h   v1.18.9-k3s1
Quoted 8 lines

Hah.

I finally got the node to show up as ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS   ROLES    AGE    VERSION
qemux86-64   Ready    master   112s   v1.18.9-k3s1

Lance,

What image type were you building ? I'm pulling in dependencies to packagegroups and the recipes themselves.

I'm not seeing the mount issue on my master/server node:

root@qemux86-64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS      RESTARTS   AGE
local-path-provisioner-6d59f47c7-h7lxk   1/1     Running     0          3m32s
metrics-server-7566d596c8-mwntr          1/1     Running     0          3m32s
helm-install-traefik-229v7               0/1     Completed   0          3m32s
coredns-7944c66d8d-9rfj7                 1/1     Running     0          3m32s
svclb-traefik-pb5j4                      2/2     Running     0          2m29s
traefik-758cd5fc85-lxpr8                 1/1     Running     0          2m29s

I'm going back to all-in-one node debugging, but can look into the mount issue more later.

Bruce

Quoted 37 lines

I'm attempting to build an all-in-one node, and that is likely causing me some issues.

I'm revisiting those potential conflicts now.

But if anyone else does have an all in one working and has some tips, feel free to share :D

Bruce

Quoted 27 lines

I've sorted out more of the dependencies, and have packagegroups to make them easier now.

Hopefully, I can figure out what is now missing and keeping my master from moving into ready today.

Bruce

Quoted 19 lines

After the master node itself turned to ready state, I check the pods with kubectl:

kubectl get nodes

NAME        STATUS   ROLES    AGE   VERSION
qemuarm64   Ready    master   11m   v1.18.9-k3s1
root@qemuarm64:~# ls
root@qemuarm64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS              RESTARTS   AGE
local-path-provisioner-6d59f47c7-xxvbl   0/1     ContainerCreating   0          12m
coredns-7944c66d8d-tlrm9                 0/1     ContainerCreating   0          12m
metrics-server-7566d596c8-svkff          0/1     ContainerCreating   0          12m
helm-install-traefik-s8p5g               0/1     ContainerCreating   0          12m

Then I describe the pods with:

Events:

  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    16m                  default-scheduler  Successfully assigned kube-

system/coredns-7944c66d8d-tlrm9 to qemuarm64

Quoted 1 line
Quoted 1 line
Quoted 1 line

Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to attach or mount

volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token- b7nlh config-volume]: timed out waiting for the condition

Quoted 1 line
Quoted 1 line
Quoted 1 line

Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach or mount

volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume coredns-token-b7nlh]: timed out waiting for the condition

Quoted 1 line
Quoted 1 line
Quoted 1 line

Warning FailedMount 11s (x16 over 16m) kubelet MountVolume.SetUp failed

for volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable file not found in $PATH

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and

mount binary on my qemuarm64 image:

Quoted 81 lines
Quoted 81 lines
Quoted 81 lines
root@qemuarm64:~# echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
root@qemuarm64:~# which mount
/bin/mount

When I type mount command, it worked fine:

/dev/root on / type ext4 (rw,relatime) devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
...
... (skipped the verbose output)

I would like to know whether you have met this "mount" issue ever?

Best Regards,
Lance

Quoted 60 lines

-----Original Message-----
From: [email protected] <[email protected]>
On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield

Ha!!!!

This applies.

Quoted 7 lines

Wonderful, thank you! I guess this is what is called "five times lucky"...

Quoted 3 lines

I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.

Excellent!

I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/l
og/?h=k3s-wip

That includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.

I did have a couple of questions on the k3s packaging itself, I was getting the following
error:

ERROR: k3s-v1.18.9+k3s1-dirty-r0 do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3s

So I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?

Bruce

Quoted 13 lines

BR,

/Joakim
--
Joakim Roubert
Senior Engineer

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and

may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Quoted 12 lines
Quoted 10 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Quoted 7 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Quoted 3 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#74 lore raw
Quoted 9 lines · Lance Yang

Hi Bruce,

Thanks for your reply. It took me a long time to address the $PATH issue.

I found the cause after I cleaned my environment. it is because I set my own $PATH before k3s startup which did not include "/bin" and some components in k3s will use "os.Getenv("PATH") to search the related tool.

Since some OS may not have systemd and the k3s-clean script seems not to delete all interfaces created by cni and umount all mount points related to k3s. I wrote a script for this situation and I will send it through a separate email based on your previous scripts.

Sounds good.

You'll notice that I pushed an update to the k3s-wip branch, things are a bit closer to working there now.

I am interested to hear if flannel crashes if you (or anyone else) install (and use) the k3s-agent on the same node as the k3s-server.

As soon as you run any k3s-agent command (with the proper token and local host as the server), the node moves into NotReady with what looks like a CNI error.

I fetched and ran the binaries directly from rancher, and they showed the same behaviour as the meta-virt ones, so it isn't something fundamental with the integration.

Bruce

Quoted 247 lines

Best Regards,
Lance

Quoted 241 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Tuesday, November 10, 2020 9:35 PM
To: Bruce Ashfield <[email protected]>
Cc: Lance Yang <[email protected]>; Joakim Roubert <[email protected]>; meta- [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 27 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 23 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 4 lines · Lance Yang

Hi Bruce and Joakim,

Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.

The branch will be more functional shortly, I have quite a few changes to factor things for k8s and generally more usable :D

        modified:   classes/cni_networking.bbclass
        modified:   conf/layer.conf
        modified:   recipes-containers/containerd/containerd-docker_git.bb
        modified:
recipes-containers/containerd/containerd-opencontainers_git.bb
        modified:   recipes-containers/k3s/README.md
        modified:   recipes-containers/k3s/k3s_git.bb
        modified:   recipes-kernel/linux/linux-yocto/kubernetes.cfg
        modified:   recipes-networking/cni/cni_git.bb

container-deploy.txt
recipes-core/packagegroups/

Quoted 1 line
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

My Image: Linux qemuarm64 by yocto.

The master node can be ready after I started the k3s server. However, the pods in kube-

system (which are essential components for k3s) cannot turn to ready state on qemuarm64.

Quoted 9 lines
Quoted 8 lines
Quoted 1 line

That's interesting, since in my configuration, the master never comes ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
qemux86-64   NotReady   master   15h   v1.18.9-k3s1
Quoted 8 lines

Hah.

I finally got the node to show up as ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS   ROLES    AGE    VERSION
qemux86-64   Ready    master   112s   v1.18.9-k3s1

Lance,

What image type were you building ? I'm pulling in dependencies to packagegroups and the recipes themselves.

I'm not seeing the mount issue on my master/server node:

root@qemux86-64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS      RESTARTS   AGE
local-path-provisioner-6d59f47c7-h7lxk   1/1     Running     0          3m32s
metrics-server-7566d596c8-mwntr          1/1     Running     0          3m32s
helm-install-traefik-229v7               0/1     Completed   0          3m32s
coredns-7944c66d8d-9rfj7                 1/1     Running     0          3m32s
svclb-traefik-pb5j4                      2/2     Running     0          2m29s
traefik-758cd5fc85-lxpr8                 1/1     Running     0          2m29s

I'm going back to all-in-one node debugging, but can look into the mount issue more later.

Bruce

Quoted 37 lines

I'm attempting to build an all-in-one node, and that is likely causing me some issues.

I'm revisiting those potential conflicts now.

But if anyone else does have an all in one working and has some tips, feel free to share :D

Bruce

Quoted 27 lines

I've sorted out more of the dependencies, and have packagegroups to make them easier now.

Hopefully, I can figure out what is now missing and keeping my master from moving into ready today.

Bruce

Quoted 19 lines

After the master node itself turned to ready state, I check the pods with kubectl:

kubectl get nodes

NAME        STATUS   ROLES    AGE   VERSION
qemuarm64   Ready    master   11m   v1.18.9-k3s1
root@qemuarm64:~# ls
root@qemuarm64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS              RESTARTS   AGE
local-path-provisioner-6d59f47c7-xxvbl   0/1     ContainerCreating   0          12m
coredns-7944c66d8d-tlrm9                 0/1     ContainerCreating   0          12m
metrics-server-7566d596c8-svkff          0/1     ContainerCreating   0          12m
helm-install-traefik-s8p5g               0/1     ContainerCreating   0          12m

Then I describe the pods with:

Events:

  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    16m                  default-scheduler  Successfully assigned kube-

system/coredns-7944c66d8d-tlrm9 to qemuarm64

Quoted 1 line
Quoted 1 line
Quoted 1 line

Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to attach or mount

volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token- b7nlh config-volume]: timed out waiting for the condition

Quoted 1 line
Quoted 1 line
Quoted 1 line

Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach or mount

volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume coredns-token-b7nlh]: timed out waiting for the condition

Quoted 1 line
Quoted 1 line
Quoted 1 line

Warning FailedMount 11s (x16 over 16m) kubelet MountVolume.SetUp failed

for volume "coredns-token-b7nlh" : mount failed: exec: "mount": executable file not found in $PATH

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and

mount binary on my qemuarm64 image:

Quoted 81 lines
Quoted 81 lines
Quoted 81 lines
root@qemuarm64:~# echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
root@qemuarm64:~# which mount
/bin/mount

When I type mount command, it worked fine:

/dev/root on / type ext4 (rw,relatime) devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
...
... (skipped the verbose output)

I would like to know whether you have met this "mount" issue ever?

Best Regards,
Lance

Quoted 60 lines

-----Original Message-----
From: [email protected] <[email protected]>
On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield

Ha!!!!

This applies.

Quoted 7 lines

Wonderful, thank you! I guess this is what is called "five times lucky"...

Quoted 3 lines

I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.

Excellent!

I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualization/l
og/?h=k3s-wip

That includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.

I did have a couple of questions on the k3s packaging itself, I was getting the following
error:

ERROR: k3s-v1.18.9+k3s1-dirty-r0 do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3s

So I added them to the FILES of the k3s package itself (so both k3s-server and k3s-agent will get them), is that the split you were looking for ?

Bruce

Quoted 13 lines

BR,

/Joakim
--
Joakim Roubert
Senior Engineer

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and

may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Quoted 12 lines
Quoted 10 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Quoted 7 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Quoted 3 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#75 lore raw

Hi Bruce,

On my side, the flannel did not crash.

I started k3s server without agent and confirmed the node is ready. Then I copied the node_token and then ran k3s agent with that token and specified the server url. Then I can see node registered successfully.

The pods traefik and coredns will be not ready after I started k3s agent in a short while:

kube-system   traefik-758cd5fc85-8tqhf                 0/1     Running   0          24m
kube-system   coredns-7944c66d8d-584jt                 0/1     Running   0          21m

but they soon were back to ready state:

kube-system   coredns-7944c66d8d-584jt                 1/1     Running   0          21m
kube-system   traefik-758cd5fc85-8tqhf                 1/1     Running   0          24m

I did not see any errors related flannel crash. Hope this helped a little bit.

Best Regards,
Lance

Quoted 304 lines

-----Original Message-----
From: Bruce Ashfield <[email protected]>
Sent: Wednesday, November 11, 2020 9:40 PM
To: Lance Yang <[email protected]>
Cc: Joakim Roubert <[email protected]>; [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 6 lines · Lance Yang

Hi Bruce,

Thanks for your reply. It took me a long time to address the $PATH issue.

I found the cause after I cleaned my environment. it is because I set my own $PATH before k3s

startup which did not include "/bin" and some components in k3s will use "os.Getenv("PATH") to search the related tool.

Quoted 2 lines

Since some OS may not have systemd and the k3s-clean script seems not to delete all

interfaces created by cni and umount all mount points related to k3s. I wrote a script for this situation and I will send it through a separate email based on your previous scripts.

Quoted 1 line

Sounds good.

You'll notice that I pushed an update to the k3s-wip branch, things are a bit closer to working there now.

I am interested to hear if flannel crashes if you (or anyone else) install (and use) the k3s-agent on the same node as the k3s-server.

As soon as you run any k3s-agent command (with the proper token and local host as the server), the node moves into NotReady with what looks like a CNI error.

I fetched and ran the binaries directly from rancher, and they showed the same behaviour as the meta-virt ones, so it isn't something fundamental with the integration.

Bruce

Quoted 124 lines

Best Regards,
Lance

Quoted 122 lines

-----Original Message-----
From: [email protected] <[email protected]>
On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Tuesday, November 10, 2020 9:35 PM
To: Bruce Ashfield <[email protected]>
Cc: Lance Yang <[email protected]>; Joakim Roubert <[email protected]>; meta- [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 27 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 23 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 4 lines · Lance Yang

Hi Bruce and Joakim,

Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.

The branch will be more functional shortly, I have quite a few changes to factor things for k8s and generally more usable :D

        modified:   classes/cni_networking.bbclass
        modified:   conf/layer.conf
        modified:   recipes-containers/containerd/containerd-docker_git.bb
        modified:
recipes-containers/containerd/containerd-opencontainers_git.bb
        modified:   recipes-containers/k3s/README.md
        modified:   recipes-containers/k3s/k3s_git.bb
        modified:   recipes-kernel/linux/linux-yocto/kubernetes.cfg
        modified:   recipes-networking/cni/cni_git.bb

container-deploy.txt
recipes-core/packagegroups/

Quoted 1 line
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

My Image: Linux qemuarm64 by yocto.

The master node can be ready after I started the k3s server. However, the pods in kube-

system (which are essential components for k3s) cannot turn to ready state on qemuarm64.

Quoted 9 lines
Quoted 8 lines
Quoted 1 line

That's interesting, since in my configuration, the master never comes ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
qemux86-64   NotReady   master   15h   v1.18.9-k3s1
Quoted 8 lines

Hah.

I finally got the node to show up as ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS   ROLES    AGE    VERSION
qemux86-64   Ready    master   112s   v1.18.9-k3s1

Lance,

What image type were you building ? I'm pulling in dependencies to packagegroups and the recipes themselves.

I'm not seeing the mount issue on my master/server node:

root@qemux86-64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS      RESTARTS   AGE
local-path-provisioner-6d59f47c7-h7lxk   1/1     Running     0          3m32s
metrics-server-7566d596c8-mwntr          1/1     Running     0          3m32s
helm-install-traefik-229v7               0/1     Completed   0          3m32s
coredns-7944c66d8d-9rfj7                 1/1     Running     0          3m32s
svclb-traefik-pb5j4                      2/2     Running     0          2m29s
traefik-758cd5fc85-lxpr8                 1/1     Running     0          2m29s

I'm going back to all-in-one node debugging, but can look into the mount issue more later.

Bruce

Quoted 37 lines

I'm attempting to build an all-in-one node, and that is likely causing me some issues.

I'm revisiting those potential conflicts now.

But if anyone else does have an all in one working and has some tips, feel free to share :D

Bruce

Quoted 27 lines

I've sorted out more of the dependencies, and have packagegroups to make them easier now.

Hopefully, I can figure out what is now missing and keeping my master from moving into ready today.

Bruce

Quoted 19 lines

After the master node itself turned to ready state, I check the pods with kubectl:

kubectl get nodes

NAME        STATUS   ROLES    AGE   VERSION
qemuarm64   Ready    master   11m   v1.18.9-k3s1
root@qemuarm64:~# ls
root@qemuarm64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS              RESTARTS   AGE
local-path-provisioner-6d59f47c7-xxvbl   0/1     ContainerCreating   0          12m
coredns-7944c66d8d-tlrm9                 0/1     ContainerCreating   0          12m
metrics-server-7566d596c8-svkff          0/1     ContainerCreating   0          12m
helm-install-traefik-s8p5g               0/1     ContainerCreating   0          12m

Then I describe the pods with:

Events:

  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    16m                  default-scheduler  Successfully assigned kube-

system/coredns-7944c66d8d-tlrm9 to qemuarm64

Quoted 1 line
Quoted 1 line
Quoted 1 line

Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to attach or

mount

Quoted 8 lines
Quoted 8 lines

volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token- b7nlh config-volume]: timed out waiting for the condition

Quoted 1 line
Quoted 1 line
Quoted 1 line

Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach or mount

volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume
coredns-token-b7nlh]: timed out waiting for the condition

Quoted 1 line
Quoted 1 line
Quoted 1 line

Warning FailedMount 11s (x16 over 16m) kubelet MountVolume.SetUp

failed

Quoted 93 lines
Quoted 93 lines

for volume "coredns-token-b7nlh" : mount failed: exec: "mount":
executable file not found in $PATH

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and

mount binary on my qemuarm64 image:

Quoted 87 lines
Quoted 87 lines
Quoted 87 lines
root@qemuarm64:~# echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
root@qemuarm64:~# which mount
/bin/mount

When I type mount command, it worked fine:

/dev/root on / type ext4 (rw,relatime) devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) ...
... (skipped the verbose output)

I would like to know whether you have met this "mount" issue ever?

Best Regards,
Lance

Quoted 66 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield

Ha!!!!

This applies.

Quoted 7 lines

Wonderful, thank you! I guess this is what is called "five times lucky"...

Quoted 3 lines

I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.

Excellent!

I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualizati
on/l
og/?h=k3s-wip

That includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.

I did have a couple of questions on the k3s packaging itself, I was getting the following
error:

ERROR: k3s-v1.18.9+k3s1-dirty-r0 do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3s

So I added them to the FILES of the k3s package itself (so
both k3s-server and k3s-agent will get them), is that the split you were looking for ?

Bruce

Quoted 8 lines

BR,

/Joakim
--
Joakim Roubert
Senior Engineer

Quoted 5 lines

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and

may also be privileged. If you are not the intended recipient,
please notify the sender immediately and do not disclose the
contents to any other person, use it for any purpose, or store or copy the information in any

medium. Thank you.

Quoted 21 lines
Quoted 21 lines
Quoted 19 lines
Quoted 10 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Quoted 5 lines
Quoted 4 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may

also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Quoted 3 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#76 lore raw
Quoted 6 lines · Lance Yang

Hi Bruce,

On my side, the flannel did not crash.

I started k3s server without agent and confirmed the node is ready. Then I copied the node_token and then ran k3s agent with that token and specified the server url. Then I can see node registered successfully.

When you say 'copied', do you mean copied to a different machine ?

Quoted 11 lines

The pods traefik and coredns will be not ready after I started k3s agent in a short while:

kube-system   traefik-758cd5fc85-8tqhf                 0/1     Running   0          24m
kube-system   coredns-7944c66d8d-584jt                 0/1     Running   0          21m

but they soon were back to ready state:

kube-system   coredns-7944c66d8d-584jt                 1/1     Running   0          21m
kube-system   traefik-758cd5fc85-8tqhf                 1/1     Running   0          24m

I did not see any errors related flannel crash. Hope this helped a little bit.

unfortunately .. no.

I need a single node configuration to work, before I can actually merge the changes, since without it, I have no way to ensure that it continues working.

Cheers,

Bruce

Quoted 310 lines

Best Regards,
Lance

Quoted 304 lines

-----Original Message-----
From: Bruce Ashfield <[email protected]>
Sent: Wednesday, November 11, 2020 9:40 PM
To: Lance Yang <[email protected]>
Cc: Joakim Roubert <[email protected]>; [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 6 lines · Lance Yang

Hi Bruce,

Thanks for your reply. It took me a long time to address the $PATH issue.

I found the cause after I cleaned my environment. it is because I set my own $PATH before k3s

startup which did not include "/bin" and some components in k3s will use "os.Getenv("PATH") to search the related tool.

Quoted 2 lines

Since some OS may not have systemd and the k3s-clean script seems not to delete all

interfaces created by cni and umount all mount points related to k3s. I wrote a script for this situation and I will send it through a separate email based on your previous scripts.

Quoted 1 line

Sounds good.

You'll notice that I pushed an update to the k3s-wip branch, things are a bit closer to working there now.

I am interested to hear if flannel crashes if you (or anyone else) install (and use) the k3s-agent on the same node as the k3s-server.

As soon as you run any k3s-agent command (with the proper token and local host as the server), the node moves into NotReady with what looks like a CNI error.

I fetched and ran the binaries directly from rancher, and they showed the same behaviour as the meta-virt ones, so it isn't something fundamental with the integration.

Bruce

Quoted 124 lines

Best Regards,
Lance

Quoted 122 lines

-----Original Message-----
From: [email protected] <[email protected]>
On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Tuesday, November 10, 2020 9:35 PM
To: Bruce Ashfield <[email protected]>
Cc: Lance Yang <[email protected]>; Joakim Roubert <[email protected]>; meta- [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 27 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 23 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 4 lines · Lance Yang

Hi Bruce and Joakim,

Thanks for sharing this branch: k3s-wip. I have tested against my yocto build.

The branch will be more functional shortly, I have quite a few changes to factor things for k8s and generally more usable :D

        modified:   classes/cni_networking.bbclass
        modified:   conf/layer.conf
        modified:   recipes-containers/containerd/containerd-docker_git.bb
        modified:
recipes-containers/containerd/containerd-opencontainers_git.bb
        modified:   recipes-containers/k3s/README.md
        modified:   recipes-containers/k3s/k3s_git.bb
        modified:   recipes-kernel/linux/linux-yocto/kubernetes.cfg
        modified:   recipes-networking/cni/cni_git.bb

container-deploy.txt
recipes-core/packagegroups/

Quoted 1 line
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

My Image: Linux qemuarm64 by yocto.

The master node can be ready after I started the k3s server. However, the pods in kube-

system (which are essential components for k3s) cannot turn to ready state on qemuarm64.

Quoted 9 lines
Quoted 8 lines
Quoted 1 line

That's interesting, since in my configuration, the master never comes ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
qemux86-64   NotReady   master   15h   v1.18.9-k3s1
Quoted 8 lines

Hah.

I finally got the node to show up as ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS   ROLES    AGE    VERSION
qemux86-64   Ready    master   112s   v1.18.9-k3s1

Lance,

What image type were you building ? I'm pulling in dependencies to packagegroups and the recipes themselves.

I'm not seeing the mount issue on my master/server node:

root@qemux86-64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS      RESTARTS   AGE
local-path-provisioner-6d59f47c7-h7lxk   1/1     Running     0          3m32s
metrics-server-7566d596c8-mwntr          1/1     Running     0          3m32s
helm-install-traefik-229v7               0/1     Completed   0          3m32s
coredns-7944c66d8d-9rfj7                 1/1     Running     0          3m32s
svclb-traefik-pb5j4                      2/2     Running     0          2m29s
traefik-758cd5fc85-lxpr8                 1/1     Running     0          2m29s

I'm going back to all-in-one node debugging, but can look into the mount issue more later.

Bruce

Quoted 37 lines

I'm attempting to build an all-in-one node, and that is likely causing me some issues.

I'm revisiting those potential conflicts now.

But if anyone else does have an all in one working and has some tips, feel free to share :D

Bruce

Quoted 27 lines

I've sorted out more of the dependencies, and have packagegroups to make them easier now.

Hopefully, I can figure out what is now missing and keeping my master from moving into ready today.

Bruce

Quoted 19 lines

After the master node itself turned to ready state, I check the pods with kubectl:

kubectl get nodes

NAME        STATUS   ROLES    AGE   VERSION
qemuarm64   Ready    master   11m   v1.18.9-k3s1
root@qemuarm64:~# ls
root@qemuarm64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS              RESTARTS   AGE
local-path-provisioner-6d59f47c7-xxvbl   0/1     ContainerCreating   0          12m
coredns-7944c66d8d-tlrm9                 0/1     ContainerCreating   0          12m
metrics-server-7566d596c8-svkff          0/1     ContainerCreating   0          12m
helm-install-traefik-s8p5g               0/1     ContainerCreating   0          12m

Then I describe the pods with:

Events:

  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    16m                  default-scheduler  Successfully assigned kube-

system/coredns-7944c66d8d-tlrm9 to qemuarm64

Quoted 1 line
Quoted 1 line
Quoted 1 line

Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to attach or

mount

Quoted 8 lines
Quoted 8 lines

volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token- b7nlh config-volume]: timed out waiting for the condition

Quoted 1 line
Quoted 1 line
Quoted 1 line

Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach or mount

volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume
coredns-token-b7nlh]: timed out waiting for the condition

Quoted 1 line
Quoted 1 line
Quoted 1 line

Warning FailedMount 11s (x16 over 16m) kubelet MountVolume.SetUp

failed

Quoted 93 lines
Quoted 93 lines

for volume "coredns-token-b7nlh" : mount failed: exec: "mount":
executable file not found in $PATH

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and

mount binary on my qemuarm64 image:

Quoted 87 lines
Quoted 87 lines
Quoted 87 lines
root@qemuarm64:~# echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
root@qemuarm64:~# which mount
/bin/mount

When I type mount command, it worked fine:

/dev/root on / type ext4 (rw,relatime) devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) ...
... (skipped the verbose output)

I would like to know whether you have met this "mount" issue ever?

Best Regards,
Lance

Quoted 66 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 6 lines · Joakim Roubert
Quoted 3 lines · Bruce Ashfield

Ha!!!!

This applies.

Quoted 7 lines

Wonderful, thank you! I guess this is what is called "five times lucky"...

Quoted 3 lines

I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.

Excellent!

I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtualizati
on/l
og/?h=k3s-wip

That includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.

I did have a couple of questions on the k3s packaging itself, I was getting the following
error:

ERROR: k3s-v1.18.9+k3s1-dirty-r0 do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3s

So I added them to the FILES of the k3s package itself (so
both k3s-server and k3s-agent will get them), is that the split you were looking for ?

Bruce

Quoted 8 lines

BR,

/Joakim
--
Joakim Roubert
Senior Engineer

Quoted 5 lines

Axis Communications AB
Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and

may also be privileged. If you are not the intended recipient,
please notify the sender immediately and do not disclose the
contents to any other person, use it for any purpose, or store or copy the information in any

medium. Thank you.

Quoted 21 lines
Quoted 21 lines
Quoted 19 lines
Quoted 10 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Quoted 5 lines
Quoted 4 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may

also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Quoted 3 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#77 lore raw

Hi Bruce,

No, it is not. I just use the command:

k3s agent --token (copied token) --server https://localhost:6443

Best Regards,
Lance

Quoted 383 lines

-----Original Message-----
From: Bruce Ashfield <[email protected]>
Sent: Thursday, November 12, 2020 9:40 PM
To: Lance Yang <[email protected]>
Cc: Joakim Roubert <[email protected]>; [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>; nd <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 6 lines · Lance Yang

Hi Bruce,

On my side, the flannel did not crash.

I started k3s server without agent and confirmed the node is ready. Then I copied the

node_token and then ran k3s agent with that token and specified the server url. Then I can see node registered successfully.

When you say 'copied', do you mean copied to a different machine ?

Quoted 11 lines

The pods traefik and coredns will be not ready after I started k3s agent in a short while:

kube-system   traefik-758cd5fc85-8tqhf                 0/1     Running   0          24m
kube-system   coredns-7944c66d8d-584jt                 0/1     Running   0          21m

but they soon were back to ready state:

kube-system   coredns-7944c66d8d-584jt                 1/1     Running   0          21m
kube-system   traefik-758cd5fc85-8tqhf                 1/1     Running   0          24m

I did not see any errors related flannel crash. Hope this helped a little bit.

unfortunately .. no.

I need a single node configuration to work, before I can actually merge the changes, since without it, I have no way to ensure that it continues working.

Cheers,

Bruce

Quoted 26 lines

Best Regards,
Lance

Quoted 24 lines

-----Original Message-----
From: Bruce Ashfield <[email protected]>
Sent: Wednesday, November 11, 2020 9:40 PM
To: Lance Yang <[email protected]>
Cc: Joakim Roubert <[email protected]>; [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 7 lines · Lance Yang

Hi Bruce,

Thanks for your reply. It took me a long time to address the $PATH issue.

I found the cause after I cleaned my environment. it is because I set my own $PATH before k3s

startup which did not include "/bin" and some components in k3s will use "os.Getenv("PATH") to search the related tool.

Quoted 3 lines

Since some OS may not have systemd and the k3s-clean script seems not to delete all

interfaces created by cni and umount all mount points related to
k3s. I wrote a script for this situation and I will send it through a separate email

based on your previous scripts.

Quoted 4 lines
Quoted 4 lines
Quoted 1 line

Sounds good.

Quoted 8 lines
Quoted 8 lines

You'll notice that I pushed an update to the k3s-wip branch, things are a bit closer to working there now.

I am interested to hear if flannel crashes if you (or anyone else) install (and use) the k3s-agent on the same node as the k3s-server.

As soon as you run any k3s-agent command (with the proper token and local host as the server), the node moves into NotReady with what looks like a CNI

error.

Quoted 3 lines
Quoted 3 lines

I fetched and ran the binaries directly from rancher, and they
showed the same behaviour as the meta-virt ones, so it isn't something fundamental

with the integration.

Quoted 24 lines
Quoted 24 lines

Bruce

Quoted 21 lines

Best Regards,
Lance

Quoted 19 lines

-----Original Message-----
From: [email protected] <[email protected]>
On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Tuesday, November 10, 2020 9:35 PM
To: Bruce Ashfield <[email protected]>
Cc: Lance Yang <[email protected]>; Joakim Roubert <[email protected]>; meta- [email protected]; Michael Zhao <[email protected]>; Kaly Xin <[email protected]>
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

Quoted 5 lines · Bruce Ashfield via lists.yoctoproject.org
Quoted 1 line · Bruce Ashfield via lists.yoctoproject.org
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines · Lance Yang
Quoted 4 lines

Hi Bruce and Joakim,

Thanks for sharing this branch: k3s-wip. I have tested against my yocto

build.

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines

The branch will be more functional shortly, I have quite a few changes to factor things for k8s and generally more usable :D

        modified:   classes/cni_networking.bbclass
        modified:   conf/layer.conf
        modified:   recipes-containers/containerd/containerd-docker_git.bb
        modified:
recipes-containers/containerd/containerd-opencontainers_git.bb
        modified:   recipes-containers/k3s/README.md
        modified:   recipes-containers/k3s/k3s_git.bb
        modified:   recipes-kernel/linux/linux-yocto/kubernetes.cfg
        modified:   recipes-networking/cni/cni_git.bb

container-deploy.txt
recipes-core/packagegroups/

Quoted 1 line
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

My Image: Linux qemuarm64 by yocto.

The master node can be ready after I started the k3s server. However, the pods in kube-

system (which are essential components for k3s) cannot turn to ready state on

qemuarm64.

Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 9 lines
Quoted 8 lines
Quoted 1 line

That's interesting, since in my configuration, the master never comes ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
qemux86-64   NotReady   master   15h   v1.18.9-k3s1
Quoted 8 lines

Hah.

I finally got the node to show up as ready:

root@qemux86-64:~# kubectl get nodes
NAME         STATUS   ROLES    AGE    VERSION
qemux86-64   Ready    master   112s   v1.18.9-k3s1

Lance,

What image type were you building ? I'm pulling in dependencies to packagegroups and the recipes themselves.

I'm not seeing the mount issue on my master/server node:

root@qemux86-64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS      RESTARTS   AGE
local-path-provisioner-6d59f47c7-h7lxk   1/1     Running     0          3m32s
metrics-server-7566d596c8-mwntr          1/1     Running     0          3m32s
helm-install-traefik-229v7               0/1     Completed   0          3m32s
coredns-7944c66d8d-9rfj7                 1/1     Running     0          3m32s
svclb-traefik-pb5j4                      2/2     Running     0          2m29s
traefik-758cd5fc85-lxpr8                 1/1     Running     0          2m29s
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I'm going back to all-in-one node debugging, but can look into the mount issue

more later.

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines

Bruce

Quoted 13 lines

I'm attempting to build an all-in-one node, and that is likely causing me some issues.

I'm revisiting those potential conflicts now.

But if anyone else does have an all in one working and has some tips, feel free to share :D

Bruce

Quoted 3 lines

I've sorted out more of the dependencies, and have packagegroups to make them easier now.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Hopefully, I can figure out what is now missing and keeping my master from moving into ready today.

Bruce

Quoted 1 line

After the master node itself turned to ready state, I check the pods with

kubectl:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

kubectl get nodes

NAME        STATUS   ROLES    AGE   VERSION
qemuarm64   Ready    master   11m   v1.18.9-k3s1
root@qemuarm64:~# ls
root@qemuarm64:~# kubectl get pods -n kube-system
NAME                                     READY   STATUS              RESTARTS   AGE
local-path-provisioner-6d59f47c7-xxvbl   0/1     ContainerCreating   0

12m

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
coredns-7944c66d8d-tlrm9                 0/1     ContainerCreating   0          12m
metrics-server-7566d596c8-svkff          0/1     ContainerCreating   0

12m

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

helm-install-traefik-s8p5g 0/1 ContainerCreating 0 12m

Then I describe the pods with:

Events:

  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    16m                  default-scheduler  Successfully

assigned kube-

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

system/coredns-7944c66d8d-tlrm9 to qemuarm64

Quoted 1 line
Quoted 1 line
Quoted 1 line

Warning FailedMount 5m23s (x3 over 14m) kubelet Unable to

attach or

Quoted 5 lines
Quoted 5 lines

mount

Quoted 4 lines
Quoted 4 lines

volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[coredns-token- b7nlh config-volume]: timed out waiting for the condition

Quoted 1 line
Quoted 1 line
Quoted 1 line

Warning FailedMount 50s (x4 over 12m) kubelet Unable to attach

or mount

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

volumes: unmounted volumes=[coredns-token-b7nlh], unattached volumes=[config-volume
coredns-token-b7nlh]: timed out waiting for the condition

Quoted 1 line
Quoted 1 line
Quoted 1 line

Warning FailedMount 11s (x16 over 16m) kubelet

MountVolume.SetUp

Quoted 40 lines
Quoted 40 lines

failed

Quoted 39 lines
Quoted 39 lines

for volume "coredns-token-b7nlh" : mount failed: exec: "mount":
executable file not found in $PATH

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I found the "mount" binary is not found in $PATH. However, I confirmed the $PATH and

mount binary on my qemuarm64 image:

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
root@qemuarm64:~# echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sb
in root@qemuarm64:~# which mount /bin/mount

When I type mount command, it worked fine:

/dev/root on / type ext4 (rw,relatime) devtmpfs on /dev type devtmpfs (rw,relatime,size=2016212k,nr_inodes=504053,mode=755) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs
(rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) ...
... (skipped the verbose output)

I would like to know whether you have met this "mount" issue ever?

Best Regards,
Lance

Quoted 11 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Bruce Ashfield via lists.yoctoproject.org
Sent: Monday, October 26, 2020 11:46 PM
To: Joakim Roubert <[email protected]>
Cc: [email protected]
Subject: Re: [meta-virtualization][PATCH v5] Adding k3s recipe

On Wed, Oct 21, 2020 at 2:00 AM Joakim Roubert

Quoted 34 lines · <[email protected]> wrote:
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 6 lines · Bruce Ashfield
Quoted 3 lines

Ha!!!!

This applies.

Quoted 7 lines

Wonderful, thank you! I guess this is what is called "five times lucky"...

Quoted 3 lines

I'm now testing and completing some of my networking factoring, as well as importing / forking some recipes to avoid extra layer depends.

Excellent!

I've pushed some of my WIP to:
https://git.yoctoproject.org/cgit/cgit.cgi/meta-virtuali
zati
on/l
og/?h=k3s-wip

That includes the split of the networking, the import of some of the dependencies and some small tweaks I'm working on.

I did have a couple of questions on the k3s packaging itself, I was getting the following
error:

ERROR: k3s-v1.18.9+k3s1-dirty-r0 do_package: QA Issue: k3s:
Files/directories were installed but not shipped in any package:
/usr/local/bin/k3s-clean
/usr/local/bin/crictl
/usr/local/bin/kubectl
/usr/local/bin/k3s

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

So I added them to the FILES of the k3s package itself
(so both k3s-server and k3s-agent will get them), is that the split you were

looking for ?

Quoted 52 lines
Quoted 52 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines

Bruce

Quoted 8 lines

BR,

/Joakim
--
Joakim Roubert
Senior Engineer

Quoted 5 lines

Axis Communications AB Emdalavägen 14, SE-223 69 Lund, Sweden
Tel: +46 46 272 18 00, Tel (direct): +46 46 272 27 48
Fax: +46 46 13 61 30, www.axis.com

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and

may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any

medium. Thank you.

Quoted 25 lines
Quoted 25 lines
Quoted 12 lines
Quoted 10 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Quoted 7 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may

also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any

medium. Thank you.

Quoted 8 lines
Quoted 7 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential

and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Quoted 3 lines

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#78 lore raw
In reply to Lance Yang
Quoted 2 lines · Lance Yang

I started k3s server without agent and confirmed the node is ready. Then I copied the node_token and then ran k3s agent with that token and specified the server url.

Did you do this on the same device?
I always run the k3s master on one device and the k3s agents on other machines (and thought that was how it is supposed to work).

BR,

/Joakim

#79 lore raw
Quoted 11 lines · Joakim Roubert
Quoted 4 lines · Lance Yang

I started k3s server without agent and confirmed the node is ready. Then I copied the node_token and then ran k3s agent with that token and specified the server url.

Did you do this on the same device?
I always run the k3s master on one device and the k3s agents on other machines (and thought that was how it is supposed to work).

FYI: after literally 5 months of on and off (most off) debugging and integration, I have finally gotten to the bottom of this, and now can bring up my all in one node out of the box!

I can now pull k3s into lots of little bits and am confident that I can support it, given my recent findings and debug.

I'm cleaning up changes, and will push a fully updated k3s to meta-virt master in the coming week.

Cheers,

Bruce

Quoted 7 lines

BR,

/Joakim

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#80 lore raw
Quoted 5 lines · Bruce Ashfield
Quoted 4 lines · Joakim Roubert
Quoted 3 lines · Lance Yang

I started k3s server without agent and confirmed the node is ready. Then I copied the node_token and then ran k3s agent with that token and specified the server url.

Quoted 8 lines
Quoted 4 lines

Did you do this on the same device?
I always run the k3s master on one device and the k3s agents on other machines (and thought that was how it is supposed to work).

FYI: after literally 5 months of on and off (most off) debugging and integration, I have finally gotten to the bottom of this, and now can bring up my all in one node out of the box!

bravo kube-edge next, or starlingx simplex aio ---- meta-stx

which combines k8s and openstack into a single platform

which i thought would be the ultimate goal for OverC :)

just thinking to myself outloud...

Quoted 22 lines

I can now pull k3s into lots of little bits and am confident that I can support it, given my recent findings and debug.

I'm cleaning up changes, and will push a fully updated k3s to meta-virt master in the coming week.

Cheers,

Bruce

Quoted 7 lines

BR,

/Joakim

#81 lore raw
In reply to Bruce Ashfield
Quoted 7 lines · Bruce Ashfield

FYI: after literally 5 months of on and off (most off) debugging and integration, I have finally gotten to the bottom of this, and now can bring up my all in one node out of the box!

I can now pull k3s into lots of little bits and am confident that I can support it, given my recent findings and debug.

This is fantastic news, Bruce!

Quoted 2 lines

I'm cleaning up changes, and will push a fully updated k3s to meta-virt master in the coming week.

This officially makes you the king.

BR,

/Joakim

#82 lore raw
In reply to Joakim Roubert

FYI: This version of the patch hasn't been forgotten.

I'm just having some issues with containerd and alternate runtimes + the networking consolidation + preparing slides for ELC-e (x3!!) .. so progress has been slower than I wanted.

That being said, I expect to have it in place before branching the release.

Cheers,

Bruce

Quoted 448 lines · Joakim Roubert
---
  recipes-containers/k3s/README.md              |  26 +++++
  ...01-Finding-host-local-in-usr-libexec.patch |  27 +++++
  .../k3s/k3s/cni-containerd-net.conf           |  24 +++++
  recipes-containers/k3s/k3s/k3s-agent          | 100 ++++++++++++++++++
  recipes-containers/k3s/k3s/k3s-agent.service  |  26 +++++
  recipes-containers/k3s/k3s/k3s-clean          |  25 +++++
  recipes-containers/k3s/k3s/k3s.service        |  27 +++++
  recipes-containers/k3s/k3s_git.bb             |  75 +++++++++++++
  8 files changed, 330 insertions(+)

create mode 100644 recipes-containers/k3s/README.md
create mode 100644
recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch
create mode 100644 recipes-containers/k3s/k3s/cni-containerd-net.conf create mode 100755 recipes-containers/k3s/k3s/k3s-agent create mode 100644 recipes-containers/k3s/k3s/k3s-agent.service create mode 100755 recipes-containers/k3s/k3s/k3s-clean create mode 100644 recipes-containers/k3s/k3s/k3s.service create mode 100644 recipes-containers/k3s/k3s_git.bb

recipes-containers/k3s/README.md new file +7 -0
new file mode 100644
index 0000000..8a0a994
@@ -0,0 +1,26 @@
+# k3s: Lightweight Kubernetes
+
+Rancher's [k3s](https://k3s.io/), available under
+[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), provides
+lightweight Kubernetes suitable for small/edge devices. There are use cases
+where the
+[installation procedures provided by

Rancher](https://rancher.com/docs/k3s/latest/en/installation/)

recipes-containers/k3s/README.md new file +6 -0
+are not ideal but a bitbake-built version is what is needed. And only a few
+mods to the [k3s source code](https://github.com/rancher/k3s) is needed to
+accomplish that.
+
+## CNI
+By default, K3s will run with flannel as the CNI, using VXLAN as the

default
+backend. It is both possible to change the flannel backend and to change from

recipes-containers/k3s/README.md new file +3 -0
+flannel to another CNI.
+
+Please see

https://rancher.com/docs/k3s/latest/en/installation/network-options/

recipes-containers/k3s/README.md new file +4 -0
+for further k3s networking details.
+
+## Configure and run a k3s agent
+The convenience script `k3s-agent` can be used to set up a k3s agent

(service):

recipes-containers/k3s/README.md new file +5 -0
+
+    k3s-agent -t <token> -s https://<master>:6443
+
+(Here `<token>` is found in `/var/lib/rancher/k3s/server/node-token` at the
+k3s master.)

diff --git a/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch b/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch new file mode 100644
index 0000000..8205d73
--- /dev/null
+++
b/recipes-containers/k3s/k3s/0001-Finding-host-local-in-usr-libexec.patch

recipes-containers/k3s/README.md new file +27 -0
@@ -0,0 +1,27 @@
+From 4faf68d68c97cfd10947e1152f711acc59f39647 Mon Sep 17 00:00:00 2001
+From: Erik Jansson <[email protected]>
+Date: Wed, 16 Oct 2019 15:07:48 +0200
+Subject: [PATCH] Finding host-local in /usr/libexec
+
+Upstream-status: Inappropriate [embedded specific]
+Signed-off-by: <[email protected]>
+---
+ pkg/agent/config/config.go | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/pkg/agent/config/config.go b/pkg/agent/config/config.go
+index b4296f360a..6af9dab895 100644
+--- a/pkg/agent/config/config.go
++++ b/pkg/agent/config/config.go
+@@ -308,7 +308,7 @@ func get(envInfo *cmds.Agent) (*config.Node, error) {
+                return nil, err
+        }
+
+-      hostLocal, err := exec.LookPath("host-local")
++      hostLocal, err := exec.LookPath("/usr/libexec/cni/host-local")
+        if err != nil {
+                return nil, errors.Wrapf(err, "failed to find host-local")
+        }
+--
+2.11.0
+
recipes-containers/k3s/k3s/cni-containerd-net.conf new file +24 -0
new file mode 100644
index 0000000..ca434d6
@@ -0,0 +1,24 @@
+{
+  "cniVersion": "0.4.0",
+  "name": "containerd-net",
+  "plugins": [
+    {
+      "type": "bridge",
+      "bridge": "cni0",
+      "isGateway": true,
+      "ipMasq": true,
+      "promiscMode": true,
+      "ipam": {
+        "type": "host-local",
+        "subnet": "10.88.0.0/16",
+        "routes": [
+          { "dst": "0.0.0.0/0" }
+        ]
+      }
+    },
+    {
+      "type": "portmap",
+      "capabilities": {"portMappings": true}
+    }
+  ]
+}
recipes-containers/k3s/k3s/k3s-agent new file +11 -0
new file mode 100755
index 0000000..1bb4c78
@@ -0,0 +1,100 @@
+#!/bin/sh -eu
+# SPDX-License-Identifier: Apache-2.0
+
+ENV_CONF=/etc/systemd/system/k3s-agent.service.d/10-env.conf
+
+usage() {
+       echo "
+USAGE:
+    ${0##*/} [OPTIONS]
+OPTIONS:
+    --token value, -t value             Token to use for authentication

[\$K3S_TOKEN]
+ --token-file value Token file to use for
authentication [\$K3S_TOKEN_FILE]

recipes-containers/k3s/k3s/k3s-agent new file +3 -0
+    --server value, -s value            Server to connect to [\$K3S_URL]
+    --node-name value                   Node name [\$K3S_NODE_NAME]
+    --resolv-conf value                 Kubelet resolv.conf file

[\$K3S_RESOLV_CONF]
+ --cluster-secret value Shared secret used to bootstrap
a cluster [\$K3S_CLUSTER_SECRET]

recipes-containers/k3s/k3s/k3s-agent new file +84 -0
+    -h                                  print this
+"
+}
+
+[ $# -gt 0 ] || {
+       usage
+       exit
+}
+
+case $1 in
+       -*)
+               ;;
+       *)
+               usage
+               exit 1
+               ;;
+esac
+
+rm -f $ENV_CONF
+mkdir -p ${ENV_CONF%/*}
+echo [Service] > $ENV_CONF
+
+while getopts "t:s:-:h" opt; do
+       case $opt in
+               h)
+                       usage
+                       exit
+                       ;;
+               t)
+                       VAR_NAME=K3S_TOKEN
+                       ;;
+               s)
+                       VAR_NAME=K3S_URL
+                       ;;
+               -)
+                       [ $# -ge $OPTIND ] || {
+                               usage
+                               exit 1
+                       }
+                       opt=$OPTARG
+                       eval OPTARG='$'$OPTIND
+                       OPTIND=$(($OPTIND + 1))
+                       case $opt in
+                               token)
+                                       VAR_NAME=K3S_TOKEN
+                                       ;;
+                               token-file)
+                                       VAR_NAME=K3S_TOKEN_FILE
+                                       ;;
+                               server)
+                                       VAR_NAME=K3S_URL
+                                       ;;
+                               node-name)
+                                       VAR_NAME=K3S_NODE_NAME
+                                       ;;
+                               resolv-conf)
+                                       VAR_NAME=K3S_RESOLV_CONF
+                                       ;;
+                               cluster-secret)
+                                       VAR_NAME=K3S_CLUSTER_SECRET
+                                       ;;
+                               help)
+                                       usage
+                                       exit
+                                       ;;
+                               *)
+                                       usage
+                                       exit 1
+                                       ;;
+                       esac
+                       ;;
+               *)
+                       usage
+                       exit 1
+                       ;;
+       esac
+    echo Environment=$VAR_NAME=$OPTARG >> $ENV_CONF
+done
+
+chmod 0644 $ENV_CONF
+rm -rf /var/lib/rancher/k3s/agent
+systemctl daemon-reload
+systemctl restart k3s-agent
+systemctl enable k3s-agent.service
recipes-containers/k3s/k3s/k3s-agent.service new file +26 -0
new file mode 100644
index 0000000..9f9016d
@@ -0,0 +1,26 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes Agent
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=control-group
+Delegate=yes
+LimitNOFILE=infinity
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s agent
+ExecStopPost=/usr/local/bin/k3s-clean
+
recipes-containers/k3s/k3s/k3s-clean new file +25 -0
new file mode 100755
index 0000000..8eff829
@@ -0,0 +1,25 @@
+#!/bin/sh -eu
+# SPDX-License-Identifier: Apache-2.0
+do_unmount() {
+       [ $# -eq 2 ] || return
+       local mounts=
+       while read ignore mount ignore; do
+               case $mount in
+                       $1/*|$2/*)
+                               mounts="$mount $mounts"
+                               ;;
+               esac
+       done </proc/self/mounts
+       [ -z "$mounts" ] || umount $mounts
+}
+
+do_unmount /run/k3s /var/lib/rancher/k3s
+
+ip link show | grep 'master cni0' | while read ignore iface ignore; do
+    iface=${iface%%@*}
+    [ -z "$iface" ] || ip link delete $iface
+done
+
+ip link delete cni0
+ip link delete flannel.1
+rm -rf /var/lib/cni/
recipes-containers/k3s/k3s/k3s.service new file +15 -0
new file mode 100644
index 0000000..34c7a80
@@ -0,0 +1,27 @@
+# Derived from the k3s install.sh's create_systemd_service_file() function
+[Unit]
+Description=Lightweight Kubernetes
+Documentation=https://k3s.io
+Requires=containerd.service
+After=containerd.service
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=notify
+KillMode=process
+Delegate=yes
+# Having non-zero Limit*s causes performance problems due to accounting

overhead
+# in the kernel. We recommend using cgroups to do container-local accounting.

recipes-containers/k3s/k3s/k3s.service new file +11 -0
+LimitNOFILE=1048576
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+TimeoutStartSec=0
+Restart=always
+RestartSec=5s
+ExecStartPre=-/sbin/modprobe br_netfilter
+ExecStartPre=-/sbin/modprobe overlay
+ExecStart=/usr/local/bin/k3s server
+
recipes-containers/k3s/k3s_git.bb new file +2 -0
new file mode 100644
index 0000000..cfc2c64
@@ -0,0 +1,75 @@
+SUMMARY = "Production-Grade Container Scheduling and Management"
+DESCRIPTION = "Lightweight Kubernetes, intended to be a fully compliant

Kubernetes."

recipes-containers/k3s/k3s_git.bb new file +3 -0
+HOMEPAGE = "https://k3s.io/"
+LICENSE = "Apache-2.0"
+LIC_FILES_CHKSUM =

"file://${S}/src/import/LICENSE;md5=2ee41112a44fe7014dce33e26468ba93"

recipes-containers/k3s/k3s_git.bb new file +9 -0
+PV = "v1.18.9+k3s1-dirty"
+
+SRC_URI = "git://github.com/rancher/k3s.git;branch=release-1.18;name=k3s \
+           file://k3s.service \
+           file://k3s-agent.service \
+           file://k3s-agent \
+           file://k3s-clean \
+           file://cni-containerd-net.conf \
+

file://0001-Finding-host-local-in-usr-libexec.patch;patchdir=src/import \

recipes-containers/k3s/k3s_git.bb new file +13 -0
+          "
+SRC_URI[k3s.md5sum] = "363d3a08dc0b72ba6e6577964f6e94a5"
+SRCREV_k3s = "630bebf94b9dce6b8cd3d402644ed023b3af8f90"
+
+inherit go
+inherit goarch
+inherit systemd
+
+PACKAGECONFIG = ""
+PACKAGECONFIG[upx] = ",,upx-native"
+GO_IMPORT = "import"
+GO_BUILD_LDFLAGS = "-X github.com/rancher/k3s/pkg/version.Version=${PV} \
+                    -X

github.com/rancher/k3s/pkg/version.GitCommit=${@d.getVar('SRCREV_k3s', d, 1)[:8]} \

recipes-containers/k3s/k3s_git.bb new file +6 -0
+                    -w -s \
+                   "
+BIN_PREFIX ?= "${exec_prefix}/local"
+
+do_compile() {
+        export

GOPATH="${S}/src/import/.gopath:${S}/src/import/vendor:${STAGING_DIR_TARGET}/${prefix}/local/go"

recipes-containers/k3s/k3s_git.bb new file +12 -1
+        export CGO_ENABLED="1"
+        export GOFLAGS="-mod=vendor"
+        cd ${S}/src/import
+        ${GO} build -tags providerless -ldflags "${GO_BUILD_LDFLAGS}"
-o ./dist/artifacts/k3s ./cmd/server/main.go
+        # Use UPX if it is enabled (and thus exists) to compress binary
+        if command -v upx > /dev/null 2>&1; then
+                upx -9 ./dist/artifacts/k3s
+        fi
+}
+do_install() {
+        install -d "${D}${BIN_PREFIX}/bin"
+        install -m 755 "${S}/src/import/dist/artifacts/k3s"

"${D}${BIN_PREFIX}/bin"

recipes-containers/k3s/k3s_git.bb new file +5 -0
+        ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/crictl"
+        ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/ctr"
+        ln -sr "${D}/${BIN_PREFIX}/bin/k3s" "${D}${BIN_PREFIX}/bin/kubectl"
+        install -m 755 "${WORKDIR}/k3s-clean" "${D}${BIN_PREFIX}/bin"
+        install -D -m 0644 "${WORKDIR}/cni-containerd-net.conf"

"${D}/${sysconfdir}/cni/net.d/10-containerd-net.conf"
+ if
${@bb.utils.contains('DISTRO_FEATURES','systemd','true','false',d)}; then
+ install -D -m 0644 "${WORKDIR}/k3s.service"
"${D}${systemd_system_unitdir}/k3s.service"
+ install -D -m 0644 "${WORKDIR}/k3s-agent.service"
"${D}${systemd_system_unitdir}/k3s-agent.service"
+ sed -i
"s#\(Exec\)\(.*\)=\(.*\)\(k3s\)#\1\2=${BIN_PREFIX}/bin/\4#g" "${D}${systemd_system_unitdir}/k3s.service" "${D}${systemd_system_unitdir}/k3s-agent.service"
+ install -m 755 "${WORKDIR}/k3s-agent"
"${D}${BIN_PREFIX}/bin"

recipes-containers/k3s/k3s_git.bb new file +6 -0
+        fi
+}
+
+PACKAGES =+ "${PN}-server ${PN}-agent"
+
+SYSTEMD_PACKAGES =

"${@bb.utils.contains('DISTRO_FEATURES','systemd','${PN}-server ${PN}-agent','',d)}"
+SYSTEMD_SERVICE_${PN}-server = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s.service','',d)}" +SYSTEMD_SERVICE_${PN}-agent = "${@bb.utils.contains('DISTRO_FEATURES','systemd','k3s-agent.service','',d)}"

recipes-containers/k3s/k3s_git.bb new file +5 -0
+SYSTEMD_AUTO_ENABLE_${PN}-agent = "disable"
+
+FILES_${PN}-agent = "${BIN_PREFIX}/bin/k3s-agent"
+
+RDEPENDS_${PN} = "cni conntrack-tools coreutils findutils iproute2

ipset virtual/containerd"

recipes-containers/k3s/k3s_git.bb new file +7 -1
+RDEPENDS_${PN}-server = "${PN}"
+RDEPENDS_${PN}-agent = "${PN}"
+
+RCONFLICTS_${PN} = "kubectl"
+
+INHIBIT_PACKAGE_STRIP = "1"
+INSANE_SKIP_${PN} += "ldflags already-stripped"
--

2.20.1

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II