Skip to contentSkip to conversation

scripts/package: snap-pkg target

PATCH In review Paolo Pisati 4 participants 16 messages Nov 27, 2017 – Dec 12, 2017 1 message not in the archive Kbuild

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 20171114123834.xsjiy2ynvott4gae@brain

Paolo Pisati

author
#1 lore raw

PATCH v3 1/1 scripts/package: snap-pkg target

Following in footsteps of other targets like 'deb-pkg, 'rpm-pkg' and 'tar-pkg', this patch adds a 'snap-pkg' target for the creation of a Linux kernel snap package using the kbuild infrastructure.

A snap, in its general form, is a self contained, sandboxed, universal package and it is intended to work across multiple distributions and/or devices. A snap package is distributed as a single compressed squashfs filesystem.

A kernel snap is a snap package carrying the Linux kernel, kernel modules, accessory files (DTBs, System.map, etc) and a manifesto file. The purpose of a kernel snap is to carry the Linux kernel during the creation of a system image, eg. Ubuntu Core, and its subsequent upgrades.

For more information on snap packages: https://snapcraft.io/docs/

Signed-off-by Paolo Pisati <[email protected]>

---
Changes since v2:
- use a tarball to instead of cloning the src tree via git
- set 'kernel-with-firmware: false' to accomodate for 'firmware_install' removal

Changes since v1:
- slightly improve the messaging

---
 .gitignore                         |  5 +++++
 scripts/package/Makefile           | 16 ++++++++++++++++
 scripts/package/snapcraft.template | 14 ++++++++++++++
 3 files changed, 35 insertions(+)
 create mode 100644 scripts/package/snapcraft.template
.gitignore +5 -0
index f6050b8..705e099 100644
@@ -66,6 +66,11 @@ modules.builtin
 /debian/
 
 #
+# Snap directory (make snap-pkg)
+#
+/snap/
+
+#
 # tar directory (make tar*-pkg)
 #
 /tar-install/
scripts/package/Makefile +16 -0
index 9ed96aef..5996300 100644
@@ -95,6 +95,21 @@ bindeb-pkg: FORCE
 
 clean-dirs += $(objtree)/debian/
 
+# snap-pkg
+# ---------------------------------------------------------------------------
+snap-pkg: FORCE
+	rm -rf $(objtree)/snap
+	mkdir $(objtree)/snap
+	$(MAKE) clean
+	$(call cmd,src_tar,$(KERNELPATH))
+	sed "s@KERNELRELEASE@$(KERNELRELEASE)@; \
+		s@SRCTREE@$(shell realpath $(KERNELPATH).tar.gz)@" \
+		$(srctree)/scripts/package/snapcraft.template > \
+		$(objtree)/snap/snapcraft.yaml
+	cd $(objtree)/snap && \
+	snapcraft --target-arch=$(UTS_MACHINE)
+
+clean-dirs += $(objtree)/snap/
 
 # tarball targets
 # ---------------------------------------------------------------------------

@@ -139,6 +154,7 @@ help: FORCE
 	@echo '  binrpm-pkg          - Build only the binary kernel RPM package'
 	@echo '  deb-pkg             - Build both source and binary deb kernel packages'
 	@echo '  bindeb-pkg          - Build only the binary kernel deb package'
+	@echo '  snap-pkg            - Build only the binary kernel snap package (will connect to external hosts)'
 	@echo '  tar-pkg             - Build the kernel as an uncompressed tarball'
 	@echo '  targz-pkg           - Build the kernel as a gzip compressed tarball'
 	@echo '  tarbz2-pkg          - Build the kernel as a bzip2 compressed tarball'
scripts/package/snapcraft.template new file +14 -0
new file mode 100644
index 0000000..626d278e
@@ -0,0 +1,14 @@
+name: kernel
+version: KERNELRELEASE
+summary: Linux kernel
+description: The upstream Linux kernel
+grade: stable
+confinement: strict
+type: kernel
+
+parts:
+  kernel:
+    plugin: kernel
+    source: SRCTREE
+    source-type: tar
+    kernel-with-firmware: false
Signature · 1 line
2.7.4
#2 lore raw
Quoted 3 lines · Paolo Pisati

Following in footsteps of other targets like 'deb-pkg, 'rpm-pkg' and 'tar-pkg', this patch adds a 'snap-pkg' target for the creation of a Linux kernel snap package using the kbuild infrastructure.

With this patch applied to the 4.14 kernel source, and running make snap-pkg on a Ubuntu 16.04 VM with the latest 16.04 kernel config file, I ran into

Traceback (most recent call last):
File "/usr/lib/python3.5/shutil.py", line 538, in move
os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory:
'/home/jim/linux/snap/parts/kernel/install/lib/firmware' -> '/home/jim/linux/snap/parts/kernel/install/firmware'

What did I miss?

Signature · 1 line
Jim
author
#3 lore raw
Quoted 5 lines · Jim Davis

With this patch applied to the 4.14 kernel source, and running make snap-pkg on a Ubuntu 16.04 VM with the latest 16.04 kernel config file, I ran into

Quoted 8 lines

Traceback (most recent call last):
File "/usr/lib/python3.5/shutil.py", line 538, in move
os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory:
'/home/jim/linux/snap/parts/kernel/install/lib/firmware' -> '/home/jim/linux/snap/parts/kernel/install/firmware'

What did I miss?

You need a recent version of snapcraft, i landed a fix for that last week.

$ snap install --classic --edge snapcraft

or refresh it, in case you already installed via snaps:

$ snap refresh --edge snapcraft

and then 'make snap-pkg' again.
Beware of $PATH in case you have the snap and deb version installed.

Signature · 2 lines
bye,
p.
#4 lore raw
Quoted 17 lines · Paolo Pisati
Quoted 5 lines · Jim Davis

With this patch applied to the 4.14 kernel source, and running make snap-pkg on a Ubuntu 16.04 VM with the latest 16.04 kernel config file, I ran into

Quoted 8 lines

Traceback (most recent call last):
File "/usr/lib/python3.5/shutil.py", line 538, in move
os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory:
'/home/jim/linux/snap/parts/kernel/install/lib/firmware' -> '/home/jim/linux/snap/parts/kernel/install/firmware'

What did I miss?

You need a recent version of snapcraft, i landed a fix for that last week.

Quoted 8 lines
$ snap install --classic --edge snapcraft

or refresh it, in case you already installed via snaps:

$ snap refresh --edge snapcraft

and then 'make snap-pkg' again.
Beware of $PATH in case you have the snap and deb version installed.

Ok, after updating and setting ~/snapcraft/bin/snapcraft ahead of
/usr/bin in my $PATH, the build completed. Oddly both
/usr/bin/snapcraft --version and ~/snapcraft/bin/snapcraft --version return 2.34, though the build failed with /usr/bin/snapcraft.

Signature · 1 line
Jim
#5 lore raw

2017-11-29 4:35 GMT+09:00 Jim Davis <[email protected]>:

Quoted 33 lines
Quoted 17 lines · Paolo Pisati
Quoted 5 lines · Jim Davis

With this patch applied to the 4.14 kernel source, and running make snap-pkg on a Ubuntu 16.04 VM with the latest 16.04 kernel config file, I ran into

Quoted 8 lines

Traceback (most recent call last):
File "/usr/lib/python3.5/shutil.py", line 538, in move
os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory:
'/home/jim/linux/snap/parts/kernel/install/lib/firmware' -> '/home/jim/linux/snap/parts/kernel/install/firmware'

What did I miss?

You need a recent version of snapcraft, i landed a fix for that last week.

Quoted 8 lines
$ snap install --classic --edge snapcraft

or refresh it, in case you already installed via snaps:

$ snap refresh --edge snapcraft

and then 'make snap-pkg' again.
Beware of $PATH in case you have the snap and deb version installed.

Ok, after updating and setting ~/snapcraft/bin/snapcraft ahead of
/usr/bin in my $PATH, the build completed. Oddly both
/usr/bin/snapcraft --version and ~/snapcraft/bin/snapcraft --version return 2.34, though the build failed with /usr/bin/snapcraft.

Quoted 6 lines

--
Jim
--
To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Worked for me too,
after updating snapcraft.

Is it really impossible to check the snapcraft version?
What is the minimum version? 2.35 ?

Signature · 2 lines
Best Regards
Masahiro Yamada
author
#6 lore raw
Quoted 7 lines · Masahiro Yamada

Worked for me too,
after updating snapcraft.

Is it really impossible to check the snapcraft version?
What is the minimum version? 2.35 ?

The minimum version is 2.35+ - what is about to become 2.36 anytime soon now.

Signature · 2 lines
bye,
p.
#7 lore raw
Quoted 11 lines · Paolo Pisati
Quoted 7 lines · Masahiro Yamada

Worked for me too,
after updating snapcraft.

Is it really impossible to check the snapcraft version?
What is the minimum version? 2.35 ?

The minimum version is 2.35+ - what is about to become 2.36 anytime soon now.

If I run /snap/bin/snapcraft --version now on that Ubuntu 16.04 VM I am getting 2.36+git2.ae61453.

I noticed that if I tried to make snap-pkg with the O=/some/dir option the tar step failed but the rest of the build continued, and seems to have finished successfully. Should the snap-pkg target stop after a tar failure?

Signature · 1 line
Jim
author
#8 lore raw
Quoted 3 lines · Jim Davis

If I run /snap/bin/snapcraft --version now on that Ubuntu 16.04 VM I am getting 2.36+git2.ae61453.

Yep, the edge channel follows the bleeding edge:

$ snap info snapcraft
...
channels:
  stable:    2.35                (794) 53MB classic
  candidate: ↑
  beta:      2.36.1              (876) 53MB classic
  edge:      2.36.1+git6.3524514 (895) 53MB classic

you better switch to beta (and then stable when 2.36.1 gets promoted there):

$ snap refresh --beta snapcraft
Quoted 4 lines

I noticed that if I tried to make snap-pkg with the O=/some/dir option the tar step failed but the rest of the build continued, and seems to have finished successfully. Should the snap-pkg target stop after a tar failure?

That is weird, how did you do that?

My experience with the O=... option (that mimicks the other rpm-pkg and deb-pkg targets) is that, you first create the dir, copy there the .config and it builds fine:

~/linux $ mke defconfig
...
~/linux $ mkdir /tmp/foobar
~/linux $ mv .config /tmp/foobar
~/linux $ git clean -ffdx
~/linux $ make O=/tmp/foobar snap-pkg
...
Snapped kernel_4.14.0+_amd64.snap
~/linux $ ls -la /tmp/foobar/snap/

-rw-r--r-- 1 flag flag 14114816 dic  5 12:41 kernel_4.14.0+_amd64.snap
drwxrwxr-x 3 flag flag     4096 dic  5 12:38 parts
drwxrwxr-x 4 flag flag     4096 dic  5 12:41 prime
drwxrwxr-x 3 flag flag     4096 dic  5 12:38 snap
-rw-rw-r-- 1 flag flag      274 dic  5 12:38 snapcraft.yaml
drwxrwxr-x 3 flag flag     4096 dic  5 12:41 stage
Signature · 2 lines
bye,
p.
#9 lore raw
Quoted 6 lines · Paolo Pisati
Quoted 4 lines

I noticed that if I tried to make snap-pkg with the O=/some/dir option the tar step failed but the rest of the build continued, and seems to have finished successfully. Should the snap-pkg target stop after a tar failure?

That is weird, how did you do that?

make O=/some/dir defconfig (or your-config-of-choice) make O=/some/dir snap-pkg

The snap-pkg target is calling the src_tar command, which bails when $(objtree) isn't $(srctree).

Is $(objtree) = $(srctree) a necessary restriction with snap builds? It's a restriction for rpm builds for some reason or another, but I thought that was just a peculiarity of the rpm build process. If the snap's tarball could be built without using src_tar, like the buildtar script does, then Bob's your uncle.

Though if the source tree is an essential part of the snap build then perhaps the build should just exit if the tarball can't be generated for whatever reason, including users doing weird things :)

Signature · 1 line
Jim
#10 lore raw

2017-12-06 6:23 GMT+09:00 Jim Davis <[email protected]>:

Quoted 9 lines · Paolo Pisati
Quoted 7 lines
Quoted 5 lines

I noticed that if I tried to make snap-pkg with the O=/some/dir option the tar step failed but the rest of the build continued, and seems to have finished successfully. Should the snap-pkg target stop after a tar failure?

That is weird, how did you do that?

Quoted 8 lines

make O=/some/dir defconfig (or your-config-of-choice) make O=/some/dir snap-pkg

The snap-pkg target is calling the src_tar command, which bails when $(objtree) isn't $(srctree).

Is $(objtree) = $(srctree) a necessary restriction with snap builds? It's a restriction for rpm builds for some reason or another,

Rather, restriction for our build system.

Working directory is changed at the top Makefile if O= is given. This option is propagated to sub-make.

Quoted 4 lines

but I
thought that was just a peculiarity of the rpm build process. If the snap's tarball could be built without using src_tar, like the buildtar script does, then Bob's your uncle.

Probably, you must reset O=, KBUILD_OUTPUT, KBUILD_SRC, etc. before calling rpmbuild, snapcraft, etc.

This is not a peculiarity of rpmbuild or snapcraft.

I guess things are not so simple as they look at a glance.

Quoted 4 lines

Though if the source tree is an essential part of the snap build then perhaps the build should just exit if the tarball can't be generated for whatever reason, including users doing weird things :)

I agree, but this is not a problem specific to snap-pkg, so Paolo's work will not be blocked.

I sent this patch:
https://patchwork.kernel.org/patch/10095377/

Just one question to Paolo:

If some change like firmware removal happens in the future, patching for snapcraft is necessary?
(i.e. users are required to upgrade snapcraft from --edge ?)

I guess the answer is yes, because "plugin: kernel" needs to know the kernel-build system internal.

Signature · 2 lines
Best Regards
Masahiro Yamada
author
#11 lore raw
Quoted 8 lines · Masahiro Yamada

Just one question to Paolo:

If some change like firmware removal happens in the future, patching for snapcraft is necessary?
(i.e. users are required to upgrade snapcraft from --edge ?)

I guess the answer is yes, because "plugin: kernel" needs to know the kernel-build system internal.

Well, in this case, i had to fix the 'kernel-with-firmware' option to skip the 'firmware_install' step since
we took it by default - let's say that in the future the 'modules_install' target is removed or change name,
then i'll have to patch snapcraft.

Wrt updating from -edge, i told you to take it from there because snap packages land there more often and more quickly
compared to the canonicaldeb archive, so from a developer perspective it's easier&quicker to iterate over it, but sooner
or later (e.g. once snapcraft 2.36.1 reaches the snap -stable channel), it will be copied to the deb archive too -
IOW, you would get using apt-get only too.

Signature · 2 lines
bye,
p.
#12 lore raw

2017-12-07 3:14 GMT+09:00 Paolo Pisati <[email protected]>:

Quoted 17 lines
Quoted 8 lines · Masahiro Yamada

Just one question to Paolo:

If some change like firmware removal happens in the future, patching for snapcraft is necessary?
(i.e. users are required to upgrade snapcraft from --edge ?)

I guess the answer is yes, because "plugin: kernel" needs to know the kernel-build system internal.

Well, in this case, i had to fix the 'kernel-with-firmware' option to skip the 'firmware_install' step since
we took it by default - let's say that in the future the 'modules_install' target is removed or change name,
then i'll have to patch snapcraft.

Quoted 8 lines

Wrt updating from -edge, i told you to take it from there because snap packages land there more often and more quickly
compared to the canonicaldeb archive, so from a developer perspective it's easier&quicker to iterate over it, but sooner
or later (e.g. once snapcraft 2.36.1 reaches the snap -stable channel), it will be copied to the deb archive too -
IOW, you would get using apt-get only too.
--

I will pick up this for v4.16-rc1.
It sounds like this target is a bit fragile, but let's see. I hope a breakage will not happen...

Signature · 2 lines
Best Regards
Masahiro Yamada
#13 lore raw

2017-12-08 0:31 GMT+09:00 Masahiro Yamada <[email protected]>:

Quoted 28 lines

2017-12-07 3:14 GMT+09:00 Paolo Pisati <[email protected]>:

Quoted 17 lines
Quoted 8 lines · Masahiro Yamada

Just one question to Paolo:

If some change like firmware removal happens in the future, patching for snapcraft is necessary?
(i.e. users are required to upgrade snapcraft from --edge ?)

I guess the answer is yes, because "plugin: kernel" needs to know the kernel-build system internal.

Well, in this case, i had to fix the 'kernel-with-firmware' option to skip the 'firmware_install' step since
we took it by default - let's say that in the future the 'modules_install' target is removed or change name,
then i'll have to patch snapcraft.

Quoted 8 lines

Wrt updating from -edge, i told you to take it from there because snap packages land there more often and more quickly
compared to the canonicaldeb archive, so from a developer perspective it's easier&quicker to iterate over it, but sooner
or later (e.g. once snapcraft 2.36.1 reaches the snap -stable channel), it will be copied to the deb archive too -
IOW, you would get using apt-get only too.
--

Quoted 4 lines

I will pick up this for v4.16-rc1.
It sounds like this target is a bit fragile, but let's see. I hope a breakage will not happen...

Applied to linux-kbuild/misc. Thanks!

Signature · 2 lines
Best Regards
Masahiro Yamada
#14 lore raw
In reply to Paolo Pisati
Quoted 11 lines · Paolo Pisati
Quoted 7 lines · Masahiro Yamada

Worked for me too,
after updating snapcraft.

Is it really impossible to check the snapcraft version?
What is the minimum version? 2.35 ?

The minimum version is 2.35+ - what is about to become 2.36 anytime soon now.

Since you are already patching snapcraft for this, why not embed the whole support in snapcraft? Eg something like snapcraft --kernel /path/to/kernel

The key advantage you'll get is being able to support old kernel versions too.

Riku

author
#15 lore raw
Quoted 6 lines · Riku Voipio

Since you are already patching snapcraft for this, why not embed the whole support in snapcraft? Eg something like snapcraft --kernel /path/to/kernel

The key advantage you'll get is being able to support old kernel versions too.

There's a lot more you can do inside a snapcraft.yaml (bundle external files, override .config options, run pre or post scripts, etc) but by embedding everything inside the kernel/kbuild plugins
you would loose all these options, and people would loose visibility of these opportunities.

Besides, snapcraft requires a snapcraft.yaml by design, and i can't remove that (such a change would never be accepted).

Signature · 2 lines
bye,
p.
author
#16 lore raw
In reply to Jim Davis

Weird, are you sure you are checking the snap version?

$ /snap/bin/snapcraft --version
snapcraft, version 2.35+git26.0474d85
$ snap info snapcraft
name:      snapcraft
summary:   easily create snaps
publisher: canonical
contact:   [email protected]
description: |
  Snapcraft aims to make upstream developers' lives easier and as such is not a
  single toolset, but instead is a collection of tools that enable the natural
  workflow of an upstream to be extended with a simple release step into Snappy
  enabled system.
snap-id: vMTKRaLjnOJQetI78HjntT37VuoyssFE
commands:
  - snapcraft
tracking:    edge
installed:   2.35+git26.0474d85 (864) 53MB classic
refreshed:   2017-11-29 09:54:04 +0100 CET
channels:
  stable:    2.35               (794) 53MB classic
  candidate: ↑
  beta:      ↑
  edge:      2.35+git26.0474d85 (864) 53MB classic
Quoted 35 lines · Jim Davis
Quoted 17 lines · Paolo Pisati
Quoted 5 lines · Jim Davis

With this patch applied to the 4.14 kernel source, and running make snap-pkg on a Ubuntu 16.04 VM with the latest 16.04 kernel config file, I ran into

Quoted 8 lines

Traceback (most recent call last):
File "/usr/lib/python3.5/shutil.py", line 538, in move
os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory:
'/home/jim/linux/snap/parts/kernel/install/lib/firmware' -> '/home/jim/linux/snap/parts/kernel/install/firmware'

What did I miss?

You need a recent version of snapcraft, i landed a fix for that last week.

Quoted 8 lines
$ snap install --classic --edge snapcraft

or refresh it, in case you already installed via snaps:

$ snap refresh --edge snapcraft

and then 'make snap-pkg' again.
Beware of $PATH in case you have the snap and deb version installed.

Ok, after updating and setting ~/snapcraft/bin/snapcraft ahead of
/usr/bin in my $PATH, the build completed. Oddly both
/usr/bin/snapcraft --version and ~/snapcraft/bin/snapcraft --version return 2.34, though the build failed with /usr/bin/snapcraft.

--
Jim

Signature · 2 lines
bye,
p.