fs/common.mk: support lz4 compression
Subsystems (2)
PATCH 1/4 fs/common.mk: support lz4 compression
Similar to the other compressors. Notice that we use the -l (legacy format) for Linux kernel initrd compatibility.
Lz4 decompression is supported by the Linux kernel since 3.11.
--- fs/common.mk | 5 +++++ 1 file changed, 5 insertions(+)
fs/common.mk +5 -0
@@ -59,6 +59,11 @@ ROOTFS_$(2)_DEPENDENCIES += host-lzma ROOTFS_$(2)_COMPRESS_EXT = .lzma ROOTFS_$(2)_COMPRESS_CMD = $$(LZMA) -9 -c endif +ifeq ($$(BR2_TARGET_ROOTFS_$(2)_LZ4),y) +ROOTFS_$(2)_DEPENDENCIES += host-lz4 +ROOTFS_$(2)_COMPRESS_EXT = .lz4 +ROOTFS_$(2)_COMPRESS_CMD = lz4 -l -9 -c +endif ifeq ($$(BR2_TARGET_ROOTFS_$(2)_LZO),y) ROOTFS_$(2)_DEPENDENCIES += host-lzop ROOTFS_$(2)_COMPRESS_EXT = .lzo
Signature · 1 line
2.11.0
PATCH 2/4 fs/cpio: add option for lz4 compression
--- fs/cpio/Config.in | 5 +++++ 1 file changed, 5 insertions(+)
fs/cpio/Config.in +5 -0
@@ -31,6 +31,11 @@ config BR2_TARGET_ROOTFS_CPIO_BZIP2 help Do compress the cpio filesystem with bzip2. +config BR2_TARGET_ROOTFS_CPIO_LZ4 + bool "lz4" + help + Do compress the cpio filesystem with lz4. + config BR2_TARGET_ROOTFS_CPIO_LZMA bool "lzma" help
Signature · 1 line
2.11.0
PATCH 3/4 fs/ext2: add option for lz4 compression
In reply to Peter Korsgaard--- fs/ext2/Config.in | 5 +++++ 1 file changed, 5 insertions(+)
fs/ext2/Config.in +5 -0
@@ -108,6 +108,11 @@ config BR2_TARGET_ROOTFS_EXT2_BZIP2 help Do compress the ext2/3/4 filesystem with bzip2. +config BR2_TARGET_ROOTFS_EXT2_LZ4 + bool "lz4" + help + Do compress the ext2 filesystem with lz4. + config BR2_TARGET_ROOTFS_EXT2_LZMA bool "lzma" help
Signature · 1 line
2.11.0
PATCH 4/4 fs/tar: add option for lz4 compression
In reply to Peter Korsgaard--- fs/tar/Config.in | 5 +++++ 1 file changed, 5 insertions(+)
fs/tar/Config.in +5 -0
@@ -26,6 +26,11 @@ config BR2_TARGET_ROOTFS_TAR_BZIP2 help Do compress the tarball with bzip2. +config BR2_TARGET_ROOTFS_TAR_LZ4 + bool "lz4" + help + Do compress the tarball with lz4. + config BR2_TARGET_ROOTFS_TAR_LZMA bool "lzma" help
Signature · 1 line
2.11.0
PATCH 1/4 fs/common.mk: support lz4 compression
In reply to Peter KorsgaardHello,
Quoted 6 lines · Peter Korsgaard
Similar to the other compressors. Notice that we use the -l (legacy format) for Linux kernel initrd compatibility.
Lz4 decompression is supported by the Linux kernel since 3.11.
Signed-off-by Peter Korsgaard <[email protected]>
I don't know in which direction we want to go, but Yann has a series doing cleanups/improvements in the filesystem stuff, and as part of this, he is dropping the compression options:
https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/fs-2
(This has not been submitted so far I believe.)
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH 1/4 fs/common.mk: support lz4 compression
Peter, Thomas, All,
On 2017-10-14 00:05 +0200, Thomas Petazzoni spake thusly:
Quoted 10 lines · Peter Korsgaard
Quoted 6 lines
Similar to the other compressors. Notice that we use the -l (legacy format) for Linux kernel initrd compatibility.
Lz4 decompression is supported by the Linux kernel since 3.11.
Signed-off-by Peter Korsgaard <[email protected]>I don't know in which direction we want to go, but Yann has a series doing cleanups/improvements in the filesystem stuff,
I already pointed Peter to this on-going work 10 days ago, while discussing on IRC, and he said "ahh, nice!" ;-)
Quoted 4 lines
and as part of
this, he is dropping the compression options:https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/fs-2
Indeed. It does not make sense to get a compressedd ext2/3/4 (or whatever other uncompressed filesystem). That is semantically useless.
One may argue that for very big images, there might not be enough place to store it, but the image is first generated uncomoressed anyway, so we need at least that much space to start with. Compressing the image will not help for big images sanyway.
If one still needs to compress the image (e.g. to send to Q/A), then it is still possible to do it in a post-iamge script, which is even more versatile (it can sign and/or encrypt as well, for example).
The cpio case is a bit different, but nonetheless the outcome is the
same: we don't need to compress it.
When used as an initramfs, the kernel will compress it anyway, so that would be a double compression; bad. When used as an initrd with the iso9660 filesystem, it is it that should compress it.
The only questionable case that remains is the cpio as used as a pure initrd. Does that still exist? And for that single use-case, I would still suggest a post-image script, because I don't see the point of having support in the infra for compression, when only a single image uses it. OR at the very least, that compressing cpio is entirely moved to the cpio fs.
Quoted 1 line
(This has not been submitted so far I believe.)
I still need to do a few tests of it, and I shall submit it here by the end of the week (so we can discuss it during the dev-days).
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/4 fs/common.mk: support lz4 compression
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Peter, Thomas, All,
On 2017-10-14 00:05 +0200, Thomas Petazzoni spake thusly:Quoted 6 lines · Peter Korsgaard
Quoted 5 lines
Similar to the other compressors. Notice that we use the -l (legacy format) for Linux kernel initrd compatibility.
Lz4 decompression is supported by the Linux kernel since 3.11.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Signed-off-by Peter Korsgaard <[email protected]>I don't know in which direction we want to go, but Yann has a series doing cleanups/improvements in the filesystem stuff,
I already pointed Peter to this on-going work 10 days ago, while discussing on IRC, and he said "ahh, nice!" ;-)
Ahh, the discussion about fixing the issues around toplevel parallel build and filesystem creation? I haven't looked at your patches, I just commented that it was nice that you were taking care of it ;)
Quoted 2 lines
Indeed. It does not make sense to get a compressedd ext2/3/4 (or whatever other uncompressed filesystem). That is semantically useless.
I agree that we would most likely not introduce such options today, but they have been supported for a very long time. I have personally used the .gz option to save archival disk space for buildbot, so they are not completely useless ;)
It is not completely semantically useless as it can be used for old-style initrds (but yeah, using an initramfs makes more sense).
Quoted 4 lines
One may argue that for very big images, there might not be enough place to store it, but the image is first generated uncomoressed anyway, so we need at least that much space to start with. Compressing the image will not help for big images sanyway.
Except if you are archiving the output/images directory and want to limit long term disk space like explained above.
Quoted 3 lines
If one still needs to compress the image (e.g. to send to Q/A), then it is still possible to do it in a post-iamge script, which is even more versatile (it can sign and/or encrypt as well, for example).
Indeed.
Quoted 6 lines
The cpio case is a bit different, but nonetheless the outcome is the
same: we don't need to compress it.When used as an initramfs, the kernel will compress it anyway, so that would be a double compression; bad. When used as an initrd with the iso9660 filesystem, it is it that should compress it.
You are missing the external initramfs usecase (E.G. not built into the kernel) - How just about all PC distributions boot. This usecase probably doesn't make much sense for Buildroot, but such a setup does make sense when you use a dm-verity / dm-crypt rootfs and need to mount it in the initramfs.
For thes use cases, the compression options IMHO makes sense.
Quoted 2 lines
The only questionable case that remains is the cpio as used as a pure initrd. Does that still exist?
Yes, that is similar to the initramfs-inside-kernel usecase, but here the compression options do make sense. There may be some usecases where it is interesting to keep the rootfs and kernel separately from each other.
Quoted 4 lines
And for that single use-case, I would still suggest a post-image script, because I don't see the point of having support in the infra for compression, when only a single image uses it. OR at the very least, that compressing cpio is entirely moved to the cpio fs.
In concept I agree, but you could use that logic on a lot of the things Buildroot does. We have supported these options for a very long time, and they imho add very little complexity, so I suggest we keep them for compatibility reasons.
Regarding the question of if we should add lz4 as well. I'm fairly pragmatic about it. Most likely other people may also want it, so either we add it or create a visible (Config.in.host) host package so it can be used from post-image scripts.
For consistency I would prefer to add it to the compression options.
Quoted 4 lines
Quoted 1 line
(This has not been submitted so far I believe.)
I still need to do a few tests of it, and I shall submit it here by the end of the week (so we can discuss it during the dev-days).
Great, thanks!
Signature · 1 line
Bye, Peter Korsgaard
PATCH 1/4 fs/common.mk: support lz4 compression
Peter, All,
On 2017-10-15 11:03 +0200, Peter Korsgaard spake thusly:
Quoted 11 lines
Quoted 11 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Peter, Thomas, All,
On 2017-10-14 00:05 +0200, Thomas Petazzoni spake thusly:Quoted 7 lines · Peter Korsgaard
Quoted 6 lines
Similar to the other compressors. Notice that we use the -l (legacy format) for Linux kernel initrd compatibility.
Lz4 decompression is supported by the Linux kernel since 3.11.
Signed-off-by Peter Korsgaard <[email protected]>
[--SNIP--]
Quoted 7 lines
Quoted 2 lines
Indeed. It does not make sense to get a compressedd ext2/3/4 (or whatever other uncompressed filesystem). That is semantically useless.
I agree that we would most likely not introduce such options today, but they have been supported for a very long time. I have personally used the .gz option to save archival disk space for buildbot, so they are not completely useless ;)
But since the filesystem infra does not get rid of the uncompresed image, surely you have a post-image (or post-buildroot) script that gets rid of the uncompressed stuff, don't you? In which case that archival script could very well be the one doing the compresion...
And Buildroot should not be concerned about the archiving requirements of the users: there will be almost as many such archiving requirements as there are users.
Quoted 2 lines
It is not completely semantically useless as it can be used for old-style initrds (but yeah, using an initramfs makes more sense).
The external initrd is what I was talking about below, yes.
Quoted 7 lines
Quoted 4 lines
One may argue that for very big images, there might not be enough place to store it, but the image is first generated uncomoressed anyway, so we need at least that much space to start with. Compressing the image will not help for big images sanyway.
Except if you are archiving the output/images directory and want to limit long term disk space like explained above.
That should not be the responsibility of Buildroot. And if you want to archive something, then you have much more to take care of than just the images, so you'll anyway have to provide a post-image (or post-buildroot) script anyway...
Quoted 12 lines
Quoted 3 lines
If one still needs to compress the image (e.g. to send to Q/A), then it is still possible to do it in a post-iamge script, which is even more versatile (it can sign and/or encrypt as well, for example).
Indeed.
Quoted 6 lines
The cpio case is a bit different, but nonetheless the outcome is the
same: we don't need to compress it.When used as an initramfs, the kernel will compress it anyway, so that would be a double compression; bad. When used as an initrd with the iso9660 filesystem, it is it that should compress it.
Quoted 7 lines
You are missing the external initramfs usecase (E.G. not built into the kernel) - How just about all PC distributions boot. This usecase probably doesn't make much sense for Buildroot, but such a setup does make sense when you use a dm-verity / dm-crypt rootfs and need to mount it in the initramfs.
For thes use cases, the compression options IMHO makes sense.
Well, the compression is indeed only valid when using an initrd (cpio, ext, whatev') or an external initramfs.
But again, I still think that it's better to offload that to an external script.
Quoted 13 lines
Quoted 2 lines
The only questionable case that remains is the cpio as used as a pure initrd. Does that still exist?
Yes, that is similar to the initramfs-inside-kernel usecase, but here the compression options do make sense. There may be some usecases where it is interesting to keep the rootfs and kernel separately from each other.
Quoted 4 lines
And for that single use-case, I would still suggest a post-image script, because I don't see the point of having support in the infra for compression, when only a single image uses it. OR at the very least, that compressing cpio is entirely moved to the cpio fs.
Quoted 4 lines
In concept I agree, but you could use that logic on a lot of the things Buildroot does. We have supported these options for a very long time, and they imho add very little complexity, so I suggest we keep them for compatibility reasons.
Well, we historically had support for the toolchain on the target as well, but we did eventually get rid of it! ;-)
Quoted 6 lines
Regarding the question of if we should add lz4 as well. I'm fairly pragmatic about it. Most likely other people may also want it, so either we add it or create a visible (Config.in.host) host package so it can be used from post-image scripts.
For consistency I would prefer to add it to the compression options.
Note: I am not arguing against adding lz4. I'm arguing against keeping the compresion options altogether. Slight difference.
What I find cumbersome to maintain is the duplication of compression lists in each filesystem types.
Maybe a middle ground would be to remove all the compression options, and move them to the top-level filesystem menu, something like:
Image compression:
(X) none
( ) gzip
( ) bzip2
( ) xz
( ) lzo
( ) lz4
( ) lzpWith this help text:
For filesystems that inherently do not support compression (e.g. ext2, cpio, tarball...), compress them with this compresion algorithm (using the default compression level)
Filesystem that already support internal compression (e.g. squashfs, cramfs...) are not impacted by this option.
Note that we can *not* remove the uncompresed image, because it may serve as input to another filesystem.
Regards,
Yann E. MORIN.
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
(This has not been submitted so far I believe.)
I still need to do a few tests of it, and I shall submit it here by the end of the week (so we can discuss it during the dev-days).
Great, thanks!
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/4 fs/common.mk: support lz4 compression
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Hi,
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Indeed. It does not make sense to get a compressedd ext2/3/4 (or whatever other uncompressed filesystem). That is semantically useless.
I agree that we would most likely not introduce such options today, but they have been supported for a very long time. I have personally used the .gz option to save archival disk space for buildbot, so they are not completely useless ;)
Quoted 4 lines
But since the filesystem infra does not get rid of the uncompresed image, surely you have a post-image (or post-buildroot) script that gets rid of the uncompressed stuff, don't you? In which case that archival script could very well be the one doing the compresion...
Correct. This was just a (poor) example where I personally have used the option.
Quoted 8 lines
And Buildroot should not be concerned about the archiving requirements of the users: there will be almost as many such archiving requirements as there are users.
Quoted 2 lines
It is not completely semantically useless as it can be used for old-style initrds (but yeah, using an initramfs makes more sense).
The external initrd is what I was talking about below, yes.
Ok. Keep in mind that there's 3 variants:
- Initramfs built into the kernel
- External initramfs (separate cpio archive)
- External initrd (filesystem image, E.G. extN)
Quoted 6 lines
Quoted 6 lines
You are missing the external initramfs usecase (E.G. not built into the kernel) - How just about all PC distributions boot. This usecase probably doesn't make much sense for Buildroot, but such a setup does make sense when you use a dm-verity / dm-crypt rootfs and need to mount it in the initramfs.
Quoted 7 lines
Quoted 1 line
For thes use cases, the compression options IMHO makes sense.
Well, the compression is indeed only valid when using an initrd (cpio, ext, whatev') or an external initramfs.
But again, I still think that it's better to offload that to an external script.
Tradeoffs, a script is more flexible, but an option is more user friendly.
Quoted 7 lines
Quoted 4 lines
In concept I agree, but you could use that logic on a lot of the things Buildroot does. We have supported these options for a very long time, and they imho add very little complexity, so I suggest we keep them for compatibility reasons.
Well, we historically had support for the toolchain on the target as well, but we did eventually get rid of it! ;-)
The reason why we got rid of it was that it was hard to test, nontrivial to maintain and against the philosophy of Buildroot (cross compilation).
I don't think those things really applies here. Sure, it could be done by a script, but so could E.G. the root password option, remount rw, purge locales, ..
Quoted 9 lines
Quoted 6 lines
Regarding the question of if we should add lz4 as well. I'm fairly pragmatic about it. Most likely other people may also want it, so either we add it or create a visible (Config.in.host) host package so it can be used from post-image scripts.
For consistency I would prefer to add it to the compression options.
Note: I am not arguing against adding lz4. I'm arguing against keeping the compresion options altogether. Slight difference.
I understand that.
Quoted 2 lines
What I find cumbersome to maintain is the duplication of compression lists in each filesystem types.
I agree that it isn't very nice, but we only rarely add new filesystem types or compression algorithms (and if we do, the patches are trivial as this lz4 series).
Quoted 12 lines
Maybe a middle ground would be to remove all the compression options, and move them to the top-level filesystem menu, something like:
Image compression: (X) none ( ) gzip ( ) bzip2 ( ) xz ( ) lzo ( ) lz4 ( ) lzp
Quoted 8 lines
With this help text:
For filesystems that inherently do not support compression (e.g. ext2, cpio, tarball...), compress them with this compresion algorithm (using the default compression level)
Filesystem that already support internal compression (e.g. squashfs, cramfs...) are not impacted by this option.
Hmm, I don't really like this. It still breaks the build for existing users, and it doesn't really simplify anything in our infrastructure (except for a few Config.in entries).
Signature · 1 line
Bye, Peter Korsgaard
PATCH 1/4 fs/common.mk: support lz4 compression
In reply to Peter KorsgaardHello,
Quoted 9 lines · Peter Korsgaard
Similar to the other compressors. Notice that we use the -l (legacy format) for Linux kernel initrd compatibility.
Lz4 decompression is supported by the Linux kernel since 3.11.
Signed-off-by Peter Korsgaard <[email protected]>--- fs/common.mk | 5 +++++ 1 file changed, 5 insertions(+)
I've applied the series. I'm also not super convinced we need to support all those compression options, but we already have plenty of them, and Peter is just adding one more, which doesn't make the whole thing much more complicated.
We can continue the discussion on whether we want to keep this or not, especially in light of http://patchwork.ozlabs.org/patch/850055/ that proposes to allow compressing the sdcard.img generated by genimage.
If we allow compression rootfs.tar, why not sdcard.img ?
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com