manual: add section about dealing efficiently with big image files
PATCH 1/1 manual: add section about dealing efficiently with big image files
From: "Yann E. MORIN" <[email protected]>
As reported by Ryan, it is not well-known that most tools can deal efficiently with big sparse files.
Add a section in the manual about this, with three tools as examples, and a hint to the man pages for the others.
--- docs/manual/common-usage.txt | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
docs/manual/common-usage.txt +33 -0
@@ -100,3 +100,36 @@ or +g+++ for building helper-binaries on your host, then do -------------------- $ make HOSTCXX=g++-4.3-HEAD HOSTCC=gcc-4.3-HEAD -------------------- + +Dealing efficiently with filesystem images +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Filesystem images can get pretty big, depending on the filesystem you choose, +the number of packages, whether you provisionned free space... Yet, some +locations in the filesystems images may just be _empty_ (eg. a long run of +'zeroes'); such a file is called a _sparse_ file. + +Most tools can handle sparse files efficiently, and will only store or write +those parts of a sparse file that are not empty. + +For example: + +* +tar+ accepts the +-S+ option to tell it to only store non-zero blocks + of sparse files: +** +tar cf archive.tar -S [files...]+ will efficiently store sparse files + in a tarball +** +tar xf archive.tar -S+ will efficiently store sparse files extracted + from a tarball + +* +cp+ accepts the +--sparse=WHEN+ option (+WHEN+ is one of +auto+, + +never+ or +always+): +** +cp --sparse=always source.file dest.file+ will make +dest.file+ a + sparse file if +source.file+ has long runs of zeroes + +* +dd+ accepts the +sparse+ value in a +conv+ list (specifying the + block-size is recommended): +** +dd if=image.file of=/dev/device bs=131072 conv=sparse+ will only + write to +/dev/device+ the blocks of +image.file+ that are not made + only of zeroes + +Other tools may have similar options. Please consult their own man pages.
Signature · 1 line
1.8.1.2
PATCH 1/1 manual: add section about dealing efficiently with big image files
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
From: "Yann E. MORIN" <[email protected]>
As reported by Ryan, it is not well-known that most tools can deal efficiently with big sparse files.Add a section in the manual about this, with three tools as examples, and a hint to the man pages for the others.
...
Quoted 4 lines
docs/manual/common-usage.txt +4
-0+* +tar+ accepts the +-S+ option to tell it to only store non-zero blocks + of sparse files: +** +tar cf archive.tar -S [files...]+ will efficiently store sparse files + in a tarball
docs/manual/common-usage.txt or +g+++ for building helper-binaries on your host, then do · line 130
Quoted 10 lines
docs/manual/common-usage.txt +10
-0+** +tar xf archive.tar -S+ will efficiently store sparse files extracted + from a tarball + +* +cp+ accepts the +--sparse=WHEN+ option (+WHEN+ is one of +auto+, + +never+ or +always+): +** +cp --sparse=always source.file dest.file+ will make +dest.file+ a + sparse file if +source.file+ has long runs of zeroes + +* +dd+ accepts the +sparse+ value in a +conv+ list (specifying the + block-size is recommended):
We should probably note that these things are only supported by (recent versions of ) the big GNU versions of the tools, not the busybox variant (which people are likely to be using).
docs/manual/common-usage.txt or +g+++ for building helper-binaries on your host, then do · line 133
Quoted 3 lines
docs/manual/common-usage.txt +3
-0+** +dd if=image.file of=/dev/device bs=131072 conv=sparse+ will only + write to +/dev/device+ the blocks of +image.file+ that are not made + only of zeroes
Is that safe advise when writing to devices? If the image file contains 128K of spaces in the used part of the filesystem (E.G. as part of a file or in the block bitmap or wherever), then that data will not be written and you will instead read back whatever was at that location in the device before.
I think that you should be using bigger block sizes, tools that understand the filesystem layout or resize afterwards (E.G. resize2fs) instead.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 1/1 manual: add section about dealing efficiently with big image files
Peter, All,
On 2013-12-15 20:54 +0100, Peter Korsgaard spake thusly:
Quoted 21 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
From: "Yann E. MORIN" <[email protected]>
As reported by Ryan, it is not well-known that most tools can deal efficiently with big sparse files.Add a section in the manual about this, with three tools as examples, and a hint to the man pages for the others.
...
Quoted 10 lines
docs/manual/common-usage.txt +10
-0+* +tar+ accepts the +-S+ option to tell it to only store non-zero blocks + of sparse files: +** +tar cf archive.tar -S [files...]+ will efficiently store sparse files + in a tarball +** +tar xf archive.tar -S+ will efficiently store sparse files extracted + from a tarball + +* +cp+ accepts the +--sparse=WHEN+ option (+WHEN+ is one of +auto+, + +never+ or +always+): +** +cp --sparse=always source.file dest.file+ will make +dest.file+ a
Quoted 8 lines
docs/manual/common-usage.txt or +g+++ for building helper-binaries on your host, then do · line 130
Quoted 4 lines
docs/manual/common-usage.txt +4
-0+ sparse file if +source.file+ has long runs of zeroes + +* +dd+ accepts the +sparse+ value in a +conv+ list (specifying the + block-size is recommended):We should probably note that these things are only supported by (recent versions of ) the big GNU versions of the tools, not the busybox variant (which people are likely to be using).
As said on IRC: this is about handling those files at build time, not on the target.
Quoted 9 lines
docs/manual/common-usage.txt or +g+++ for building helper-binaries on your host, then do · line 133
Quoted 3 lines
docs/manual/common-usage.txt +3
-0+** +dd if=image.file of=/dev/device bs=131072 conv=sparse+ will only + write to +/dev/device+ the blocks of +image.file+ that are not made + only of zeroesIs that safe advise when writing to devices? If the image file contains 128K of spaces in the used part of the filesystem (E.G. as part of a file or in the block bitmap or wherever), then that data will not be written and you will instead read back whatever was at that location in the device before.
Yes, that's true. But those are just example on how to handle big sparse files, an in no way the recommended way of flashing a device.
I'll remove it before re-submitting. After all, we hint to the man pages of other tools, so it's thoroughly documented. And if anyone if foolish enough to try that, then we can say 'we never wrote that!' :-)
Quoted 3 lines
I think that you should be using bigger block sizes, tools that understand the filesystem layout or resize afterwards (E.G. resize2fs) instead.
Not sure I follow you on that one. What if the user enters a large number of blocks for his ext2 filesystem? Those will be empty (zero-filled), but the image file will not be made sparse. So there is no 'fs resize' or such in the process.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 manual: add section about dealing efficiently with big image files
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Hi,
Quoted 3 lines
I'll remove it before re-submitting. After all, we hint to the man pages of other tools, so it's thoroughly documented. And if anyone if foolish enough to try that, then we can say 'we never wrote that!' :-)
Indeed, thanks ;)
Quoted 8 lines
Quoted 3 lines
I think that you should be using bigger block sizes, tools that understand the filesystem layout or resize afterwards (E.G. resize2fs) instead.
Not sure I follow you on that one. What if the user enters a large number of blocks for his ext2 filesystem? Those will be empty (zero-filled), but the image file will not be made sparse. So there is no 'fs resize' or such in the process.
What I meant was simply that if you want to end up with a filesystem with lots of free space and don't want to waste time writing zeroes to the unused areas, it is safer to:
- create the filesystem spanning the entire partition yourself on the fly (mkfs + tar xf output/images/rootfs.tar)
- or resize fs to the full partition size after writing the image (dd if=output/images/rootfs.ext2 + resize2fs)
Signature · 1 line
Bye, Peter Korsgaard
PATCH 1/1 manual: add section about dealing efficiently with big image files
Peter, All,
On 2013-12-15 23:38 +0100, Peter Korsgaard spake thusly:
Quoted 14 lines
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
I think that you should be using bigger block sizes, tools that understand the filesystem layout or resize afterwards (E.G. resize2fs) instead.
Not sure I follow you on that one. What if the user enters a large number of blocks for his ext2 filesystem? Those will be empty (zero-filled), but the image file will not be made sparse. So there is no 'fs resize' or such in the process.
What I meant was simply that if you want to end up with a filesystem with lots of free space and don't want to waste time writing zeroes to the unused areas, it is safer to:
Quoted 5 lines
- create the filesystem spanning the entire partition yourself on the fly (mkfs + tar xf output/images/rootfs.tar)
- or resize fs to the full partition size after writing the image (dd if=output/images/rootfs.ext2 + resize2fs)
I see, but in that case, we should no offer the user the ability to specify the number of blocks to use for the ext2 filesystem, with BR2_TARGET_ROOTFS_EXT2_BLOCKS, for example (and ext2 if the only writable filesystem for which we do it).
Should I cook up a patch to remove BR2_TARGET_ROOTFS_EXT2_BLOCKS?
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 manual: add section about dealing efficiently with big image files
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
What I meant was simply that if you want to end up with a filesystem with lots of free space and don't want to waste time writing zeroes to the unused areas, it is safer to:
- create the filesystem spanning the entire partition yourself on the fly (mkfs + tar xf output/images/rootfs.tar)
Quoted 7 lines
Quoted 2 lines
- or resize fs to the full partition size after writing the image (dd if=output/images/rootfs.ext2 + resize2fs)
I see, but in that case, we should no offer the user the ability to specify the number of blocks to use for the ext2 filesystem, with
BR2_TARGET_ROOTFS_EXT2_BLOCKS, for example (and ext2 if the only writable filesystem for which we do it).
Historically, part of the reason why we had that option was that our size estimate wasn't perfect, so we sometimes didn't forsee enough space. Next to that, some people might be using it already for provide-me-a-bit-extra-space (I know I have used it in the past). It's not that the option is really bad, just that it is more efficient to not transfer / write all those zeroes for the unallocated space if you can.
Quoted 1 line
Should I cook up a patch to remove
BR2_TARGET_ROOTFS_EXT2_BLOCKS?
I would prefer to keep it. It doesn't add much complexity in Buildroot.
Signature · 1 line
Bye, Peter Korsgaard