Feature request: e2fsck -z
Hi all,
This is something I've wanted to see for a very long time, and it finally occurred to me that perhaps I should say something about it!
It would be a very nice thing to have a flag to e2fsck, presumably -z, to zero out any unused data blocks, inodes and so on. The goal is to minimize the amount of space required after compressing a virtual disk image or similar, and to make sure any non-data isn't lying around.
-hpa
Quoted 5 lines · H. Peter Anvin
Hi all,
This is something I've wanted to see for a very long time, and it finally occurred to me that perhaps I should say something about it!
Quoted 7 lines
It would be a very nice thing to have a flag to e2fsck, presumably -z, to zero out any unused data blocks, inodes and so on. The goal is to minimize the amount of space required after compressing a virtual disk image or similar, and to make sure any non-data isn't lying around.
-hpa
Do you need it to be in the fsck tool?
If you have a sparsely allocated block map under your file system, doing a zero of all blocks could add hours for a big, slow S-ATA drives (2-3 hours for a 1TB drive).
An alternative for SSD's and devices that do TRIM/UNMAP would be to use one of the batched discard tools (that would make discarded data read back as zeroed).
Thanks!
Ric
Quoted 6 lines · Ric Wheeler
Quoted 5 lines · H. Peter Anvin
Hi all,
This is something I've wanted to see for a very long time, and it finally occurred to me that perhaps I should say something about it!
Quoted 8 lines
Quoted 4 lines
It would be a very nice thing to have a flag to e2fsck, presumably -z, to zero out any unused data blocks, inodes and so on. The goal is to minimize the amount of space required after compressing a virtual disk image or similar, and to make sure any non-data isn't lying around.
Do you need it to be in the fsck tool?
If you have a sparsely allocated block map under your file system, doing a zero of all blocks could add hours for a big, slow S-ATA drives (2-3 hours for a 1TB drive).
I think Ted has a tool that does this already. It should be relatively simple oo do, like "dd if=/dev/zero of=/mountpoint/temp_zero_file && rm /mountpoint/temp_zero_file.
Quoted 1 line
An alternative for SSD's and devices that do TRIM/UNMAP would be to use one of the batched discard tools (that would make discarded data read back as zeroed).
In fact, I thought Lukas has already made a tool for sending BLKDISCARD for all unused parts of the filesystem?
Cheers, Andreas
Quoted 7 lines · Andreas Dilger
Quoted 6 lines · Ric Wheeler
Quoted 5 lines · H. Peter Anvin
Hi all,
This is something I've wanted to see for a very long time, and it finally occurred to me that perhaps I should say something about it!
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
It would be a very nice thing to have a flag to e2fsck, presumably -z, to zero out any unused data blocks, inodes and so on. The goal is to minimize the amount of space required after compressing a virtual disk image or similar, and to make sure any non-data isn't lying around.
Do you need it to be in the fsck tool?
If you have a sparsely allocated block map under your file system, doing a zero of all blocks could add hours for a big, slow S-ATA drives (2-3 hours for a 1TB drive).
I think Ted has a tool that does this already. It should be relatively simple oo do, like "dd if=/dev/zero of=/mountpoint/temp_zero_file&& rm /mountpoint/temp_zero_file.
This will work but will be a potential multi-hour long process (and cause out of space errors at some point for other applications for a very brief window :))
Quoted 5 lines
Quoted 2 lines
An alternative for SSD's and devices that do TRIM/UNMAP would be to use one of the batched discard tools (that would make discarded data read back as zeroed).
In fact, I thought Lukas has already made a tool for sending BLKDISCARD for all unused parts of the filesystem?
Cheers, Andreas
Right, I think he has. That tool (not part of fsck) would do the job much quicker for enabled devices,
Ric
Quoted 29 lines · Ric Wheeler
Quoted 11 lines · Andreas Dilger
Quoted 10 lines · Ric Wheeler
Quoted 9 lines · H. Peter Anvin
Hi all,
This is something I've wanted to see for a very long time, and it finally occurred to me that perhaps I should say something about it!
It would be a very nice thing to have a flag to e2fsck, presumably -z, to zero out any unused data blocks, inodes and so on. The goal is to minimize the amount of space required after compressing a virtual disk image or similar, and to make sure any non-data isn't lying around.
Quoted 8 lines
Quoted 5 lines
Do you need it to be in the fsck tool?
If you have a sparsely allocated block map under your file system, doing a zero of all blocks could add hours for a big, slow S-ATA drives (2-3 hours for a 1TB drive).
I think Ted has a tool that does this already. It should be relatively simple oo do, like "dd if=/dev/zero of=/mountpoint/temp_zero_file&& rm /mountpoint/temp_zero_file.
This will work but will be a potential multi-hour long process (and cause out of space errors at some point for other applications for a very brief window :))
Quoted 4 lines
Quoted 4 lines
An alternative for SSD's and devices that do TRIM/UNMAP would be to use one of the batched discard tools (that would make discarded data read back as zeroed).
Quoted 7 lines
Quoted 4 lines
In fact, I thought Lukas has already made a tool for sending BLKDISCARD for all unused parts of the filesystem?
Cheers, Andreas
Right, I think he has. That tool (not part of fsck) would do the job much quicker for enabled devices,
You're right. It is a part of e2fsck and you can enable it with -E discard.
Thanks!
-Lukas
* Lukas Czerner:
Quoted 2 lines
You're right. It is a part of e2fsck and you can enable it with -E discard.
If the block device does not support TRIM, does this result in zeros being written instead?
Signature · 8 lines
Florian Weimer <[email protected]> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Florian Weimer
* Lukas Czerner:
Quoted 2 lines
You're right. It is a part of e2fsck and you can enable it with -E discard.
If the block device does not support TRIM, does this result in zeros being written instead?
Of course not, discard works only on devices which does support it. If the device does not support discard it will to nothing.
Thanks!
-Lukas
* Lukas Czerner:
Quoted 12 lines · Florian Weimer
Quoted 8 lines
* Lukas Czerner:
Quoted 2 lines
You're right. It is a part of e2fsck and you can enable it with -E discard.
If the block device does not support TRIM, does this result in zeros being written instead?
Of course not, discard works only on devices which does support it. If the device does not support discard it will to nothing.
Oh, then it doesn't address hpa's use case (improving compression).
Signature · 8 lines
Florian Weimer <[email protected]> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Florian Weimer
* Lukas Czerner:
Quoted 4 lines · Florian Weimer
Quoted 3 lines
* Lukas Czerner:
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
You're right. It is a part of e2fsck and you can enable it with -E discard.
If the block device does not support TRIM, does this result in zeros being written instead?
Of course not, discard works only on devices which does support it. If the device does not support discard it will to nothing.
Oh, then it doesn't address hpa's use case (improving compression).
I think that there are multiple ways to help compression.
One would be to extract only the used data blocks (like file system specific dump utilities can do), the other would be to zero with the discard or by writing zeros to the unused blocks.
Ric
Quoted 9 lines · Florian Weimer
* Lukas Czerner:
Quoted 7 lines · Florian Weimer
Quoted 6 lines
* Lukas Czerner:
Quoted 2 lines
You're right. It is a part of e2fsck and you can enable it with -E discard.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
If the block device does not support TRIM, does this result in zeros being written instead?
Of course not, discard works only on devices which does support it. If the device does not support discard it will to nothing.
Oh, then it doesn't address hpa's use case (improving compression).
It does if he will use SSD's with "discard_zeroes_data" capability. But my original answer was to simply confirm that e2fsck has that feature.
Thanks!
-Lukas
Quoted 2 lines · Ric Wheeler
Do you need it to be in the fsck tool?
As others have noted, (a) zerofree does this already, and (b) there's also -E discard.
My take on it though is that it's a reasonable request. It's much like there is "sort -u" even though you could do this via "sort | uniq". Part of the Unix philosophy is to use tools that are composable, yes --- but optimizing for common cases is also a good thing, and with the advent of virtualization being more and more popular, zeroing free blocks for virtualization images is good and useful thing.
This is also why I agitated for adding support so that e2fsprogs tools could operate directly on qemu-img files, and not just have support which is hacked into e2image. Yes, you can always take an qemu-img file, decompress and expand it into a raw file, run debugfs or e2fsck on it, and then convert it back to a qemu-img file --- but if lots of people are doing that, it does make sense to optimize for the most common use cases.
- Ted
Quoted 7 lines · Ted Ts'o
Quoted 2 lines · Ric Wheeler
Do you need it to be in the fsck tool?
As others have noted, (a) zerofree does this already, and (b) there's also -E discard.
Quoted 7 lines
My take on it though is that it's a reasonable request. It's much like there is "sort -u" even though you could do this via "sort | uniq". Part of the Unix philosophy is to use tools that are composable, yes --- but optimizing for common cases is also a good thing, and with the advent of virtualization being more and more popular, zeroing free blocks for virtualization images is good and useful thing.
And since we already have punch hole support in ext4 I am going to use that to "discard" - "punch out" unused blocks from the image. It is probably not exactly what you are talking about, but it can be useful.
Quoted 8 lines
This is also why I agitated for adding support so that e2fsprogs tools could operate directly on qemu-img files, and not just have support which is hacked into e2image. Yes, you can always take an qemu-img file, decompress and expand it into a raw file, run debugfs or e2fsck on it, and then convert it back to a qemu-img file --- but if lots of people are doing that, it does make sense to optimize for the most common use cases.
I am in favor if this, however I do not want to do it. I seems to me that it is just rewriting the support that qemu already has. However if qemu guys (or anyone) would create an library which we can use from both projects, that would be certainly very useful.
Thanks!
-Lukas
Quoted 7 lines
- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · H. Peter Anvin
It would be a very nice thing to have a flag to e2fsck, presumably -z, to zero out any unused data blocks, inodes and so on. The goal is to minimize the amount of space required after compressing a virtual disk image or similar, and to make sure any non-data isn't lying around.
I wrote a separate utility, zerofree[1], that zeroes out free blocks.
Ron
--
[1] http://intgat.tigress.co.uk/rmy/uml/