Skip to contentSkip to conversation

manual: add section about dealing efficiently with big image files

PATCH New Yann E. MORIN 2 participants 6 messages Dec 15, 2013

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 manual: add section about dealing efficiently with big image files

From: "Yann E. MORIN" <[email protected]>

As reported by Ryan, it is not well-known that most tools can deal efficiently with big sparse files.

Add a section in the manual about this, with three tools as examples, and a hint to the man pages for the others.

---
 docs/manual/common-usage.txt | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
docs/manual/common-usage.txt +33 -0
index 1290dfc..46183ec 100644
@@ -100,3 +100,36 @@ or +g+++ for building helper-binaries on your host, then do
 --------------------
  $ make HOSTCXX=g++-4.3-HEAD HOSTCC=gcc-4.3-HEAD
 --------------------
+
+Dealing efficiently with filesystem images
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Filesystem images can get pretty big, depending on the filesystem you choose,
+the number of packages, whether you provisionned free space... Yet, some
+locations in the filesystems images may just be _empty_ (eg. a long run of
+'zeroes'); such a file is called a _sparse_ file.
+
+Most tools can handle sparse files efficiently, and will only store or write
+those parts of a sparse file that are not empty.
+
+For example:
+
+* +tar+ accepts the +-S+ option to tell it to only store non-zero blocks
+  of sparse files:
+** +tar cf archive.tar -S [files...]+ will efficiently store sparse files
+   in a tarball
+** +tar xf archive.tar -S+ will efficiently store sparse files extracted
+   from a tarball
+
+* +cp+ accepts the +--sparse=WHEN+ option (+WHEN+ is one of +auto+,
+  +never+ or +always+):
+** +cp --sparse=always source.file dest.file+ will make +dest.file+ a
+   sparse file if +source.file+ has long runs of zeroes
+
+* +dd+ accepts the +sparse+ value in a +conv+ list (specifying the
+  block-size is recommended):
+** +dd if=image.file of=/dev/device bs=131072 conv=sparse+ will only
+   write to +/dev/device+ the blocks of +image.file+ that are not made
+   only of zeroes
+
+Other tools may have similar options. Please consult their own man pages.
Signature · 1 line
1.8.1.2
#1 lore raw

PATCH 1/1 manual: add section about dealing efficiently with big image files

Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

From: "Yann E. MORIN" <[email protected]>
As reported by Ryan, it is not well-known that most tools can deal efficiently with big sparse files.

Add a section in the manual about this, with three tools as examples, and a hint to the man pages for the others.

...

Quoted 4 lines
docs/manual/common-usage.txt +4 -0
+* +tar+ accepts the +-S+ option to tell it to only store non-zero blocks
+  of sparse files:
+** +tar cf archive.tar -S [files...]+ will efficiently store sparse files
+   in a tarball

docs/manual/common-usage.txt or +g+++ for building helper-binaries on your host, then do · line 130

Quoted 10 lines
docs/manual/common-usage.txt +10 -0
+** +tar xf archive.tar -S+ will efficiently store sparse files extracted
+   from a tarball
+
+* +cp+ accepts the +--sparse=WHEN+ option (+WHEN+ is one of +auto+,
+  +never+ or +always+):
+** +cp --sparse=always source.file dest.file+ will make +dest.file+ a
+   sparse file if +source.file+ has long runs of zeroes
+
+* +dd+ accepts the +sparse+ value in a +conv+ list (specifying the
+  block-size is recommended):

We should probably note that these things are only supported by (recent versions of ) the big GNU versions of the tools, not the busybox variant (which people are likely to be using).

docs/manual/common-usage.txt or +g+++ for building helper-binaries on your host, then do · line 133

Quoted 3 lines
docs/manual/common-usage.txt +3 -0
+** +dd if=image.file of=/dev/device bs=131072 conv=sparse+ will only
+   write to +/dev/device+ the blocks of +image.file+ that are not made
+   only of zeroes

Is that safe advise when writing to devices? If the image file contains 128K of spaces in the used part of the filesystem (E.G. as part of a file or in the block bitmap or wherever), then that data will not be written and you will instead read back whatever was at that location in the device before.

I think that you should be using bigger block sizes, tools that understand the filesystem layout or resize afterwards (E.G. resize2fs) instead.

Signature · 1 line
Bye, Peter Korsgaard
#2 lore raw

PATCH 1/1 manual: add section about dealing efficiently with big image files

Peter, All,

On 2013-12-15 20:54 +0100, Peter Korsgaard spake thusly:

Quoted 21 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

From: "Yann E. MORIN" <[email protected]>
As reported by Ryan, it is not well-known that most tools can deal efficiently with big sparse files.

Add a section in the manual about this, with three tools as examples, and a hint to the man pages for the others.

...

Quoted 10 lines
docs/manual/common-usage.txt +10 -0
+* +tar+ accepts the +-S+ option to tell it to only store non-zero blocks
+  of sparse files:
+** +tar cf archive.tar -S [files...]+ will efficiently store sparse files
+   in a tarball
+** +tar xf archive.tar -S+ will efficiently store sparse files extracted
+   from a tarball
+
+* +cp+ accepts the +--sparse=WHEN+ option (+WHEN+ is one of +auto+,
+  +never+ or +always+):
+** +cp --sparse=always source.file dest.file+ will make +dest.file+ a
Quoted 8 lines

docs/manual/common-usage.txt or +g+++ for building helper-binaries on your host, then do · line 130

Quoted 4 lines
docs/manual/common-usage.txt +4 -0
+   sparse file if +source.file+ has long runs of zeroes
+
+* +dd+ accepts the +sparse+ value in a +conv+ list (specifying the
+  block-size is recommended):

We should probably note that these things are only supported by (recent versions of ) the big GNU versions of the tools, not the busybox variant (which people are likely to be using).

As said on IRC: this is about handling those files at build time, not on the target.

Quoted 9 lines

docs/manual/common-usage.txt or +g+++ for building helper-binaries on your host, then do · line 133

Quoted 3 lines
docs/manual/common-usage.txt +3 -0
+** +dd if=image.file of=/dev/device bs=131072 conv=sparse+ will only
+   write to +/dev/device+ the blocks of +image.file+ that are not made
+   only of zeroes

Is that safe advise when writing to devices? If the image file contains 128K of spaces in the used part of the filesystem (E.G. as part of a file or in the block bitmap or wherever), then that data will not be written and you will instead read back whatever was at that location in the device before.

Yes, that's true. But those are just example on how to handle big sparse files, an in no way the recommended way of flashing a device.

I'll remove it before re-submitting. After all, we hint to the man pages of other tools, so it's thoroughly documented. And if anyone if foolish enough to try that, then we can say 'we never wrote that!' :-)

Quoted 3 lines

I think that you should be using bigger block sizes, tools that understand the filesystem layout or resize afterwards (E.G. resize2fs) instead.

Not sure I follow you on that one. What if the user enters a large number of blocks for his ext2 filesystem? Those will be empty (zero-filled), but the image file will not be made sparse. So there is no 'fs resize' or such in the process.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#3 lore raw

PATCH 1/1 manual: add section about dealing efficiently with big image files

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Hi,

Quoted 3 lines

I'll remove it before re-submitting. After all, we hint to the man pages of other tools, so it's thoroughly documented. And if anyone if foolish enough to try that, then we can say 'we never wrote that!' :-)

Indeed, thanks ;)

Quoted 8 lines
Quoted 3 lines

I think that you should be using bigger block sizes, tools that understand the filesystem layout or resize afterwards (E.G. resize2fs) instead.

Not sure I follow you on that one. What if the user enters a large number of blocks for his ext2 filesystem? Those will be empty (zero-filled), but the image file will not be made sparse. So there is no 'fs resize' or such in the process.

What I meant was simply that if you want to end up with a filesystem with lots of free space and don't want to waste time writing zeroes to the unused areas, it is safer to:

- create the filesystem spanning the entire partition yourself on the fly (mkfs + tar xf output/images/rootfs.tar)

- or resize fs to the full partition size after writing the image (dd if=output/images/rootfs.ext2 + resize2fs)

Signature · 1 line
Bye, Peter Korsgaard
#4 lore raw

PATCH 1/1 manual: add section about dealing efficiently with big image files

Peter, All,

On 2013-12-15 23:38 +0100, Peter Korsgaard spake thusly:

Quoted 14 lines
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

I think that you should be using bigger block sizes, tools that understand the filesystem layout or resize afterwards (E.G. resize2fs) instead.

Not sure I follow you on that one. What if the user enters a large number of blocks for his ext2 filesystem? Those will be empty (zero-filled), but the image file will not be made sparse. So there is no 'fs resize' or such in the process.

What I meant was simply that if you want to end up with a filesystem with lots of free space and don't want to waste time writing zeroes to the unused areas, it is safer to:

Quoted 5 lines

- create the filesystem spanning the entire partition yourself on the fly (mkfs + tar xf output/images/rootfs.tar)

- or resize fs to the full partition size after writing the image (dd if=output/images/rootfs.ext2 + resize2fs)

I see, but in that case, we should no offer the user the ability to specify the number of blocks to use for the ext2 filesystem, with BR2_TARGET_ROOTFS_EXT2_BLOCKS, for example (and ext2 if the only writable filesystem for which we do it).

Should I cook up a patch to remove BR2_TARGET_ROOTFS_EXT2_BLOCKS?

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#5 lore raw

PATCH 1/1 manual: add section about dealing efficiently with big image files

Quoted 9 lines
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

What I meant was simply that if you want to end up with a filesystem with lots of free space and don't want to waste time writing zeroes to the unused areas, it is safer to:

- create the filesystem spanning the entire partition yourself on the fly (mkfs + tar xf output/images/rootfs.tar)

Quoted 7 lines
Quoted 2 lines

- or resize fs to the full partition size after writing the image (dd if=output/images/rootfs.ext2 + resize2fs)

I see, but in that case, we should no offer the user the ability to specify the number of blocks to use for the ext2 filesystem, with BR2_TARGET_ROOTFS_EXT2_BLOCKS, for example (and ext2 if the only writable filesystem for which we do it).

Historically, part of the reason why we had that option was that our size estimate wasn't perfect, so we sometimes didn't forsee enough space. Next to that, some people might be using it already for provide-me-a-bit-extra-space (I know I have used it in the past). It's not that the option is really bad, just that it is more efficient to not transfer / write all those zeroes for the unallocated space if you can.

Quoted 1 line

Should I cook up a patch to remove BR2_TARGET_ROOTFS_EXT2_BLOCKS?

I would prefer to keep it. It doesn't add much complexity in Buildroot.

Signature · 1 line
Bye, Peter Korsgaard