Skip to contentSkip to conversation

e2fsck: Discard free data and inode blocks.

PATCH In review Lukas Czerner 5 participants 24 messages Oct 21, 2010 – Oct 22, 2010

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 e2fsck: Discard free data and inode blocks.

In Pass 5 when we are checking block and inode bitmaps we have great opportunity to discard free space and unused inodes on the device, because bitmaps has just been verified as valid. This commit takes advantage of this opportunity and discards both, all free space and unused inodes.

I have added new option '-K' which when set, disables discard. Also when the underlying device does not support discard, or BLKDISCARD ioctl returns any kind of error, or when some errors occurred in bitmaps, the discard is disabled.

As an addition, when there is any not-yet-zeroed inode table and discard zeroes data, then inode table is marked as zeroed.

---
 e2fsck/e2fsck.8.in |    7 +++
 e2fsck/e2fsck.h    |    1 +
 e2fsck/pass5.c     |  119 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 e2fsck/unix.c      |   10 ++++-
 4 files changed, 135 insertions(+), 2 deletions(-)
e2fsck/e2fsck.8.in +7 -0
index 3fb15e6..507f7b1 100644
@@ -19,6 +19,9 @@ e2fsck \- check a Linux ext2/ext3/ext4 file system
 .I blocksize
 ]
 [
+.B \-K
+]
+[
 .BR \-l | \-L
 .I bad_blocks_file
 ]
@@ -204,6 +207,10 @@ time trials.
 @JDEV@Set the pathname where the external-journal for this filesystem can be
 @JDEV@found.
 .TP
+.BI \-K
+Keep, do not attempt to discard free blocks and unused inodes (discarding
+blocks is useful on solid state devices and sparse / thin-provisioned storage).
+.TP
 .BI \-k
 When combined with the 
 .B \-c
e2fsck/e2fsck.h +1 -0
index d4df5f3..a377246 100644
@@ -155,6 +155,7 @@ struct resource_track {
 #define E2F_OPT_WRITECHECK	0x0200
 #define E2F_OPT_COMPRESS_DIRS	0x0400
 #define E2F_OPT_FRAGCHECK	0x0800
+#define E2F_OPT_DISCARD		0x1000
 
 /*
  * E2fsck flags
e2fsck/pass5.c +119 -0

168 lines not on this page · raw message

e2fsck/unix.c +8 -1
index 7eb269c..15dd204 100644
@@ -667,7 +667,11 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx)
 		ctx->program_name = *argv;
 	else
 		ctx->program_name = "e2fsck";
-	while ((c = getopt (argc, argv, "panyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDk")) != EOF)
+
+	/* set defaults */
+	ctx->options |= E2F_OPT_DISCARD;
+
+	while ((c = getopt (argc, argv, "panyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDkK")) != EOF)
 		switch (c) {
 		case 'C':
 			ctx->progress = e2fsck_update_progress;
@@ -790,6 +794,9 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx)
 		case 'k':
 			keep_bad_blocks++;
 			break;
+		case 'K':
+			ctx->options &= ~E2F_OPT_DISCARD;
+			break;
 		default:
 			usage(ctx);
 		}
@@ -943,7 +950,6 @@ static errcode_t try_open_fs(e2fsck_t ctx, int flags, io_manager io_ptr,
 	return retval;
 }
 
#1 lore raw
Quoted 10 lines · Lukas Czerner

In Pass 5 when we are checking block and inode bitmaps we have great opportunity to discard free space and unused inodes on the device, because bitmaps has just been verified as valid. This commit takes advantage of this opportunity and discards both, all free space and unused inodes.

I have added new option '-K' which when set, disables discard. Also when the underlying device does not support discard, or BLKDISCARD ioctl returns any kind of error, or when some errors occurred in bitmaps, the discard is disabled.

I'm always a bit nervous with patches like this, that will prevent data recovery after an e2fsck run (which seems like the opposite of what we want from e2fsck).

Two suggestions:
- it probably makes sense to disable this by default, and allow it to be specified on the command-line and e2fsck.conf
- should we really have a short option, or a "-E discard" and "-E nodiscard" options, which allow us to change the default easily at some later time (which we can't do with a single -K flag)

Quoted 7 lines
e2fsck/pass5.c +7 -0
+static void e2fsck_discard_blocks(e2fsck_t ctx, blk_t start,
+				  blk_t count)
+{
+	fd = open64(fs->device_name, O_RDWR);
+	if (fd < 0) {
+		com_err("open", errno,
+			_("while opening %s for discarding"),

e2fsck/pass5.c void e2fsck_pass5(e2fsck_t ctx) · line 115

Quoted 10 lines
e2fsck/pass5.c +10 -0
+			ctx->device_name);
+		fatal_error(ctx, 0);
+	}
+
+	ret = ioctl(fd, BLKDISCARD, &range);
+	if (ret)
+		ctx->options &= ~E2F_OPT_DISCARD;
+
+	close(fd);
+}

If we are calling this ioctl for a lot of small block ranges, doing an open/close for each one could add significant overhead. The unix struct_io_manager already has an open file descriptor for this block device, maybe it is better to encapsulate this operation there? The ioctl also doesn't make sense for non-Linux platforms (though they may have a different ioctl that is equivalent) so that may be a better solution.

(defect) It makes sense to start with a blk64_t for this function, instead of a blk_t that needs to be fixed immediately for > 16TB filesystems, or the block number will be truncated and accidentally discard the wrong data. Oops.

Cheers, Andreas

#2 lore raw
Quoted 12 lines · Andreas Dilger
Quoted 10 lines · Lukas Czerner

In Pass 5 when we are checking block and inode bitmaps we have great opportunity to discard free space and unused inodes on the device, because bitmaps has just been verified as valid. This commit takes advantage of this opportunity and discards both, all free space and unused inodes.

I have added new option '-K' which when set, disables discard. Also when the underlying device does not support discard, or BLKDISCARD ioctl returns any kind of error, or when some errors occurred in bitmaps, the discard is disabled.

Quoted 8 lines

I'm always a bit nervous with patches like this, that will prevent data recovery after an e2fsck run (which seems like the opposite of what we want from e2fsck).

Two suggestions:
- it probably makes sense to disable this by default, and allow it to be specified on the command-line and e2fsck.conf
- should we really have a short option, or a "-E discard" and "-E nodiscard" options, which allow us to change the default easily at some later time (which we can't do with a single -K flag)

Right, I agree it would be probably better to disable this by default.

Quoted 12 lines
Quoted 12 lines
e2fsck/pass5.c +11 -0
+static void e2fsck_discard_blocks(e2fsck_t ctx, blk_t start,
+				  blk_t count)
+{
+	fd = open64(fs->device_name, O_RDWR);
+	if (fd < 0) {
+		com_err("open", errno,
+			_("while opening %s for discarding"),
+			ctx->device_name);
+		fatal_error(ctx, 0);
+	}
+
Quoted 8 lines

e2fsck/pass5.c void e2fsck_pass5(e2fsck_t ctx) · line 115

Quoted 6 lines
e2fsck/pass5.c +6 -0
+	ret = ioctl(fd, BLKDISCARD, &range);
+	if (ret)
+		ctx->options &= ~E2F_OPT_DISCARD;
+
+	close(fd);
+}

If we are calling this ioctl for a lot of small block ranges, doing an open/close for each one could add significant overhead. The unix struct_io_manager already has an open file descriptor for this block device, maybe it is better to encapsulate this operation there? The ioctl also doesn't make sense for non-Linux platforms (though they may have a different ioctl that is equivalent) so that may be a better solution.

That is why the #ifdef __linux__ is there. I agree with using struct_io_manager file descriptor.

Quoted 2 lines

(defect) It makes sense to start with a blk64_t for this function, instead of a blk_t that needs to be fixed immediately for > 16TB filesystems, or the block number will be truncated and accidentally discard the wrong data. Oops.

Oh, I have missed that out.

Quoted 3 lines

Cheers, Andreas

Thanks for review Andreas.

-Lukas

#3 lore raw
Quoted 15 lines · Lukas Czerner
Quoted 13 lines · Andreas Dilger
Quoted 10 lines · Lukas Czerner

In Pass 5 when we are checking block and inode bitmaps we have great opportunity to discard free space and unused inodes on the device, because bitmaps has just been verified as valid. This commit takes advantage of this opportunity and discards both, all free space and unused inodes.

I have added new option '-K' which when set, disables discard. Also when the underlying device does not support discard, or BLKDISCARD ioctl returns any kind of error, or when some errors occurred in bitmaps, the discard is disabled.

I'm always a bit nervous with patches like this, that will prevent data recovery after an e2fsck run (which seems like the opposite of what we want from e2fsck).

Quoted 9 lines
Quoted 6 lines

Two suggestions:
- it probably makes sense to disable this by default, and allow it to be specified on the command-line and e2fsck.conf
- should we really have a short option, or a "-E discard" and "-E nodiscard" options, which allow us to change the default easily at some later time (which we can't do with a single -K flag)

Right, I agree it would be probably better to disable this by default.

If we do disable it by default, I think that we might also want to be consistent and disable the discard support in mkfs by default as well?

thanks!

Ric

#4 lore raw
Quoted 29 lines · Ric Wheeler
Quoted 17 lines · Lukas Czerner
Quoted 15 lines · Andreas Dilger
Quoted 10 lines · Lukas Czerner

In Pass 5 when we are checking block and inode bitmaps we have great opportunity to discard free space and unused inodes on the device, because bitmaps has just been verified as valid. This commit takes advantage of this opportunity and discards both, all free space and unused inodes.

I have added new option '-K' which when set, disables discard. Also when the underlying device does not support discard, or BLKDISCARD ioctl returns any kind of error, or when some errors occurred in bitmaps, the discard is disabled.

I'm always a bit nervous with patches like this, that will prevent data recovery after an e2fsck run (which seems like the opposite of what we want from e2fsck).

Quoted 10 lines
Quoted 7 lines

Two suggestions:
- it probably makes sense to disable this by default, and allow it to be specified on the command-line and e2fsck.conf
- should we really have a short option, or a "-E discard" and "-E nodiscard"
options, which allow us to change the default easily at some later time (which we can't do with a single -K flag)

Right, I agree it would be probably better to disable this by default.

Quoted 7 lines

If we do disable it by default, I think that we might also want to be consistent and disable the discard support in mkfs by default as well?

thanks!

Ric

I think that this will not be necessary. There is a concern that it might prevent data recovery after fsck because it might be already discarded (some weird fs corruption?) in pass 5. However in my opinion this is a very small window (if there even is any), because we have already passed check 1-4 and we have just confirmed that group descriptors should be ok. But when there is an even slight chance this might happen I would suggest that we really disable it by default (at least for a while - we will see then).

On the other hand there is nothing to be afraid of in the case of mkfs, because we can not possibly lose any relevant data, because discard is done before the filesystem gets created.

-Lukas

#5 lore raw
Quoted 46 lines · Lukas Czerner
Quoted 28 lines · Ric Wheeler
Quoted 17 lines · Lukas Czerner
Quoted 15 lines · Andreas Dilger
Quoted 10 lines · Lukas Czerner

In Pass 5 when we are checking block and inode bitmaps we have great opportunity to discard free space and unused inodes on the device, because bitmaps has just been verified as valid. This commit takes advantage of this opportunity and discards both, all free space and unused inodes.

I have added new option '-K' which when set, disables discard. Also when the underlying device does not support discard, or BLKDISCARD ioctl returns any kind of error, or when some errors occurred in bitmaps, the discard is disabled.

I'm always a bit nervous with patches like this, that will prevent data recovery after an e2fsck run (which seems like the opposite of what we want from e2fsck).

Quoted 10 lines
Quoted 7 lines

Two suggestions:
- it probably makes sense to disable this by default, and allow it to be specified on the command-line and e2fsck.conf
- should we really have a short option, or a "-E discard" and "-E nodiscard"
options, which allow us to change the default easily at some later time (which we can't do with a single -K flag)

Right, I agree it would be probably better to disable this by default.

Quoted 7 lines

If we do disable it by default, I think that we might also want to be consistent and disable the discard support in mkfs by default as well?

thanks!

Ric

I think that this will not be necessary. There is a concern that it might prevent data recovery after fsck because it might be already discarded (some weird fs corruption?) in pass 5. However in my opinion this is a very small window (if there even is any), because we have already passed check 1-4 and we have just confirmed that group descriptors should be ok. But when there is an even slight chance this might happen I would suggest that we really disable it by default (at least for a while - we will see then).

Quoted 5 lines

On the other hand there is nothing to be afraid of in the case of mkfs, because we can not possibly lose any relevant data, because discard is done before the filesystem gets created.

-Lukas

My concern with mkfs is that we have seen several devices which don't handle this well.

We will be using this TRIM (or UNMAP, etc) on lots of old, creaky hardware with old firmware, so having it try on all devices is almost certainly going to cause breakages, hangs, etc in the field....

Ric

#6 lore raw
Quoted 58 lines · Ric Wheeler
Quoted 51 lines · Lukas Czerner
Quoted 33 lines · Ric Wheeler
Quoted 29 lines · Lukas Czerner
Quoted 18 lines · Andreas Dilger
Quoted 6 lines · Lukas Czerner

In Pass 5 when we are checking block and inode bitmaps we have great opportunity to discard free space and unused inodes on the device, because bitmaps has just been verified as valid. This commit takes advantage of this opportunity and discards both, all free space and unused inodes.

Quoted 6 lines

I have added new option '-K' which when set, disables discard. Also when
the underlying device does not support discard, or BLKDISCARD ioctl returns any kind of error, or when some errors occurred in bitmaps, the
discard is disabled.

I'm always a bit nervous with patches like this, that will prevent data
recovery after an e2fsck run (which seems like the opposite of what we want from e2fsck).

Quoted 9 lines

Two suggestions:
- it probably makes sense to disable this by default, and allow it to be
specified on the command-line and e2fsck.conf
- should we really have a short option, or a "-E discard" and "-E nodiscard"
options, which allow us to change the default easily at some later
time
(which we can't do with a single -K flag)

Quoted 3 lines

Right, I agree it would be probably better to disable this by default.

Quoted 7 lines

If we do disable it by default, I think that we might also want to be consistent and disable the discard support in mkfs by default as well?

thanks!

Ric

I think that this will not be necessary. There is a concern that it might prevent data recovery after fsck because it might be already discarded (some weird fs corruption?) in pass 5. However in my opinion this is a very small window (if there even is any), because we have already passed check 1-4 and we have just confirmed that group descriptors should be ok. But when there is an even slight chance this might happen I would suggest that we really disable it by default (at least for a while - we will see then).

Quoted 5 lines

On the other hand there is nothing to be afraid of in the case of mkfs, because we can not possibly lose any relevant data, because discard is done before the filesystem gets created.

-Lukas

Quoted 9 lines

My concern with mkfs is that we have seen several devices which don't handle this well.

We will be using this TRIM (or UNMAP, etc) on lots of old, creaky hardware with old firmware, so having it try on all devices is almost certainly going to cause breakages, hangs, etc in the field....

Ric

Well, so far the only breakages I have seen was with lots of small TRIMs (or UNMAPs, etc) issued in random pattern, never in case of mkfs which is quite a opposite - big sequential ranges.

Hangs should be covered by those two patches:

http://marc.info/?l=linux-ext4&m=128774558623608&w=2
http://marc.info/?l=linux-ext4&m=128767099123375&w=2

if, of course, they get upstream. Also there is a big win, when discard also zeroes data, because in that case we can just skip inode table initialization (zeroing) without any need of in-kernel lazyinit code enabled. And we get all this for free. It was introduced with Sandeens patch:

http://marc.info/?l=linux-ext4&m=128234048208327&w=2

So, I would rather leave it on by default.

-Lukas

#7 lore raw
Quoted 83 lines · Lukas Czerner
Quoted 57 lines · Ric Wheeler
Quoted 51 lines · Lukas Czerner
Quoted 33 lines · Ric Wheeler
Quoted 29 lines · Lukas Czerner
Quoted 18 lines · Andreas Dilger
Quoted 6 lines · Lukas Czerner

In Pass 5 when we are checking block and inode bitmaps we have great opportunity to discard free space and unused inodes on the device, because bitmaps has just been verified as valid. This commit takes advantage of this opportunity and discards both, all free space and unused inodes.

Quoted 6 lines

I have added new option '-K' which when set, disables discard. Also when
the underlying device does not support discard, or BLKDISCARD ioctl returns any kind of error, or when some errors occurred in bitmaps, the
discard is disabled.

I'm always a bit nervous with patches like this, that will prevent data
recovery after an e2fsck run (which seems like the opposite of what we want from e2fsck).

Quoted 9 lines

Two suggestions:
- it probably makes sense to disable this by default, and allow it to be
specified on the command-line and e2fsck.conf
- should we really have a short option, or a "-E discard" and "-E nodiscard"
options, which allow us to change the default easily at some later
time
(which we can't do with a single -K flag)

Quoted 3 lines

Right, I agree it would be probably better to disable this by default.

Quoted 7 lines

If we do disable it by default, I think that we might also want to be consistent and disable the discard support in mkfs by default as well?

thanks!

Ric

I think that this will not be necessary. There is a concern that it might prevent data recovery after fsck because it might be already discarded (some weird fs corruption?) in pass 5. However in my opinion this is a very small window (if there even is any), because we have already passed check 1-4 and we have just confirmed that group descriptors should be ok. But when there is an even slight chance this might happen I would suggest that we really disable it by default (at least for a while - we will see then).

Quoted 5 lines

On the other hand there is nothing to be afraid of in the case of mkfs, because we can not possibly lose any relevant data, because discard is done before the filesystem gets created.

-Lukas

Quoted 9 lines

My concern with mkfs is that we have seen several devices which don't handle this well.

We will be using this TRIM (or UNMAP, etc) on lots of old, creaky hardware with old firmware, so having it try on all devices is almost certainly going to cause breakages, hangs, etc in the field....

Ric

Well, so far the only breakages I have seen was with lots of small TRIMs (or UNMAPs, etc) issued in random pattern, never in case of mkfs which is quite a opposite - big sequential ranges.

Hangs should be covered by those two patches:

http://marc.info/?l=linux-ext4&m=128774558623608&w=2
http://marc.info/?l=linux-ext4&m=128767099123375&w=2

if, of course, they get upstream. Also there is a big win, when discard also zeroes data, because in that case we can just skip inode table initialization (zeroing) without any need of in-kernel lazyinit code enabled. And we get all this for free. It was introduced with Sandeens patch:

Quoted 5 lines

http://marc.info/?l=linux-ext4&m=128234048208327&w=2

So, I would rather leave it on by default.

-Lukas

You cannot 100% depend on discard zeroing blocks - that is not a universal requirement of devices that support it. Specifically, for ATA devices, I think that there are optional bits that specify how a device will behave when you read from a trimmed region.

Ric

#8 lore raw

Ric Wheeler wrote:

...

Quoted 17 lines
Quoted 17 lines

Well, so far the only breakages I have seen was with lots of small TRIMs (or UNMAPs, etc) issued in random pattern, never in case of mkfs which is quite a opposite - big sequential ranges.

Hangs should be covered by those two patches:

http://marc.info/?l=linux-ext4&m=128774558623608&w=2
http://marc.info/?l=linux-ext4&m=128767099123375&w=2

if, of course, they get upstream. Also there is a big win, when discard also zeroes data, because in that case we can just skip inode table initialization (zeroing) without any need of in-kernel lazyinit code enabled. And we get all this for free. It was introduced with Sandeens patch:

http://marc.info/?l=linux-ext4&m=128234048208327&w=2

Quoted 8 lines
Quoted 3 lines

So, I would rather leave it on by default.

-Lukas

You cannot 100% depend on discard zeroing blocks - that is not a universal requirement of devices that support it. Specifically, for ATA devices, I think that there are optional bits that specify how a device will behave when you read from a trimmed region.

But don't we have the ability to test whether discard -does- zero blocks, as advertised by the device? And honestly if the device mis-reports, that sounds like a device vendor problem to fix.

The proposal wasn't to discard and assume zero, but to check for that behavior:

http://kerneltrap.org/mailarchive/linux-ext4/2010/9/21/6885628/thread

+ if (!retval && mke2fs_discard_zeroes_data(fs)) {
+ if (verbose)
+ printf(_("Discard succeeded and will return 0s "
+ " - enabling lazy_itable_init\n"));
+ lazy_itable_init = 1;
+ lazy_itable_zeroed = 1;
+ }

so we're not depending on it zeroing blocks, we're just depending on it advertising correctly whether or not it -does- zero.

-Eric

Quoted 2 lines

Ric

#9 lore raw
Quoted 45 lines · Eric Sandeen

Ric Wheeler wrote:

...

Quoted 17 lines
Quoted 17 lines

Well, so far the only breakages I have seen was with lots of small TRIMs (or UNMAPs, etc) issued in random pattern, never in case of mkfs which is quite a opposite - big sequential ranges.

Hangs should be covered by those two patches:

http://marc.info/?l=linux-ext4&m=128774558623608&w=2
http://marc.info/?l=linux-ext4&m=128767099123375&w=2

if, of course, they get upstream. Also there is a big win, when discard also zeroes data, because in that case we can just skip inode table initialization (zeroing) without any need of in-kernel lazyinit code enabled. And we get all this for free. It was introduced with Sandeens patch:

http://marc.info/?l=linux-ext4&m=128234048208327&w=2

Quoted 7 lines
Quoted 3 lines

So, I would rather leave it on by default.

-Lukas

You cannot 100% depend on discard zeroing blocks - that is not a universal requirement of devices that support it. Specifically, for ATA devices, I think that there are optional bits that specify how a device will behave when you read from a trimmed region.

But don't we have the ability to test whether discard -does- zero blocks, as advertised by the device? And honestly if the device mis-reports, that sounds like a device vendor problem to fix.

The proposal wasn't to discard and assume zero, but to check for that behavior:

http://kerneltrap.org/mailarchive/linux-ext4/2010/9/21/6885628/thread

(quoted diff) +7 -0
+		if (!retval&&  mke2fs_discard_zeroes_data(fs)) {
+			if (verbose)
+				printf(_("Discard succeeded and will return 0s "
+					 " - enabling lazy_itable_init\n"));
+			lazy_itable_init = 1;
+			lazy_itable_zeroed = 1;
+		}
Quoted 6 lines

so we're not depending on it zeroing blocks, we're just depending on it advertising correctly whether or not it -does- zero.

-Eric

I think that ATA devices have historically not done this correctly, but the T13 committee is working on it. The question is whether the bit we check and rely on has the right semantics (and then if the device will reliably implement this).

Historically, array vendors did rely on SCSI commands like the old fashioned "WRITE_SAME" to initialize storage for them, but that takes a *long* time to run :)

Ric

#10 lore raw
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Ric" == Ric Wheeler <[email protected]> writes:

so we're not depending on it zeroing blocks, we're just depending on it advertising correctly whether or not it -does- zero.

If the relevant bits are set (ATA: DRAT and RZAT, SCSI: TPRZ) we'll set the bdev's discard_zeroes_data flag.

[root@test ~]# lsscsi | grep SSD | awk '{ print $7 }' /dev/sde
[root@test ~]# grep . /sys/block/sde/queue/discard_zeroes_data 1

The relevant ioctl is BLKDISCARDZEROES.

Ric> I think that ATA devices have historically not done this correctly,

I'm only aware of one drive that advertised RZAT and got it wrong. I believe a firmware update fixed it.

Generally we assume that if the firmware writers go through the effort of reporting things correctly then they have also implemented the feature. We have quite a few sanity checks in place in libata so we won't trigger if the firmware guys just put all ones in a word, for instance. There are several things that need to line up for us to actually set the discard flags.

PS. http://oss.oracle.com/~mkp/docs/linux-advanced-storage.pdf

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#11 lore raw
Quoted 27 lines · Martin K. Petersen
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Ric" == Ric Wheeler<[email protected]> writes:

so we're not depending on it zeroing blocks, we're just depending on it advertising correctly whether or not it -does- zero.

If the relevant bits are set (ATA: DRAT and RZAT, SCSI: TPRZ) we'll set the bdev's discard_zeroes_data flag.

[root@test ~]# lsscsi | grep SSD | awk '{ print $7 }' /dev/sde
[root@test ~]# grep . /sys/block/sde/queue/discard_zeroes_data 1

The relevant ioctl is BLKDISCARDZEROES.

Ric> I think that ATA devices have historically not done this correctly,

I'm only aware of one drive that advertised RZAT and got it wrong. I believe a firmware update fixed it.

Generally we assume that if the firmware writers go through the effort of reporting things correctly then they have also implemented the feature. We have quite a few sanity checks in place in libata so we won't trigger if the firmware guys just put all ones in a word, for instance. There are several things that need to line up for us to actually set the discard flags.

Quoted 2 lines

PS. http://oss.oracle.com/~mkp/docs/linux-advanced-storage.pdf

Hi Martin,

I think that the T13 people are still reworking (or have just finished?) the spec. For example,

http://www.t13.org/documents/UploadedDocuments/docs2010/e09158r2-Trim_Clarifications.pdf

has more detail on how to interpret those bits. I certainly cannot claim to have followed all of this recently, but do recall that early on the ATA devices specifically had various behaviors that caused concern.

Ric

#12 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Ric" == Ric Wheeler <[email protected]> writes:

Ric> I think that the T13 people are still reworking (or have just Ric> finished?) the spec. For example,

ACS-2 is pretty much set in stone (it's in ballot). The trim bits haven't changed for quite a while and we're compliant with the latest draft.

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#13 lore raw
In reply to Martin K. Petersen
Quoted 8 lines · Martin K. Petersen
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Ric" == Ric Wheeler <[email protected]> writes:

so we're not depending on it zeroing blocks, we're just depending on it advertising correctly whether or not it -does- zero.

If the relevant bits are set (ATA: DRAT and RZAT, SCSI: TPRZ) we'll set the bdev's discard_zeroes_data flag.

Quoted 6 lines

[root@test ~]# lsscsi | grep SSD | awk '{ print $7 }' /dev/sde
[root@test ~]# grep . /sys/block/sde/queue/discard_zeroes_data 1

The relevant ioctl is BLKDISCARDZEROES.

which is what that patch I pointed to uses, BTW...

I really think it's on firm footing here, we're not assuming discard == zeroed data, we are checking whether the drive reports exactly that behavior ...

-Eric

#14 lore raw
In reply to Ric Wheeler
Quoted 8 lines · Ric Wheeler
Quoted 7 lines · Lukas Czerner

Also there is a big win, when discard
also zeroes data, because in that case we can just skip inode table initialization (zeroing) without any need of in-kernel lazyinit code enabled. And we get all this for free. It was introduced with Sandeens patch:

Quoted 5 lines
Quoted 3 lines

http://marc.info/?l=linux-ext4&m=128234048208327&w=2

So, I would rather leave it on by default.

You cannot 100% depend on discard zeroing blocks - that is not a universal requirement of devices that support it. Specifically, for ATA devices, I think that there are optional bits that specify how a device will behave when you read from a trimmed region.

That patch also checks for the zeroing feature. When this patch was first under discussion, I proposed that we validate that the device is actually zeroed by doing a write a non-zero block to the disk and then calling discard+zero for that region, and reading back the block and verifying it.

Eric wasn't convinced that was necessary, maybe you can convince him more...

Cheers, Andreas

#15 lore raw
Quoted 13 lines · Andreas Dilger
Quoted 12 lines · Ric Wheeler
Quoted 10 lines · Lukas Czerner

Also there is a big win, when discard
also zeroes data, because in that case we can just skip inode table initialization (zeroing) without any need of in-kernel lazyinit code enabled. And we get all this for free. It was introduced with Sandeens patch:

http://marc.info/?l=linux-ext4&m=128234048208327&w=2

So, I would rather leave it on by default.

Quoted 7 lines
Quoted 1 line

You cannot 100% depend on discard zeroing blocks - that is not a universal requirement of devices that support it. Specifically, for ATA devices, I think that there are optional bits that specify how a device will behave when you read from a trimmed region.

That patch also checks for the zeroing feature. When this patch was first under discussion, I proposed that we validate that the device is actually zeroed by doing a write a non-zero block to the disk and then calling discard+zero for that region, and reading back the block and verifying it.

Eric wasn't convinced that was necessary, maybe you can convince him more...

Cheers, Andreas

One of the counter arguments was, that some devices does not preserve this behavior through power cycles. I think Ted was the one talking about that.

-Lukas

#16 lore raw
Quoted 7 lines · Lukas Czerner
Quoted 3 lines

That patch also checks for the zeroing feature. When this patch was first under discussion, I proposed that we validate that the device is actually zeroed by doing a write a non-zero block to the disk and then calling discard+zero for that region, and reading back the block and verifying it.

Eric wasn't convinced that was necessary, maybe you can convince him more...

One of the counter arguments was, that some devices does not preserve this behavior through power cycles. I think Ted was the one talking about that.

Sure, I don't think we can handle every pathology, but doing a write/discard/read of a few blocks (when it has the potential to avoid many GB of writes for zeroing) is surely easy and worthwhile?

In any case, I thought that discussion was about a device that didn't report BLKDISCARDSZEROES=1, but only that a normal DISCARD would read back zero until the next restart? That prevents optimizations like "read until we see non-zero data, then start writing zeroes", which would still be faster for many RAID devices (or older kernels that don't have DISCARD/ZERO support at all).

Cheers, Andreas

#17 lore raw
Quoted 4 lines · Andreas Dilger
Quoted 3 lines · Lukas Czerner
Quoted 3 lines

That patch also checks for the zeroing feature. When this patch was first under discussion, I proposed that we validate that the device is actually zeroed by doing a write a non-zero block to the disk and then calling discard+zero for that region, and reading back the block and verifying it.

Eric wasn't convinced that was necessary, maybe you can convince him more...

Quoted 8 lines
Quoted 3 lines

One of the counter arguments was, that some devices does not preserve this behavior through power cycles. I think Ted was the one talking about that.

Sure, I don't think we can handle every pathology, but doing a write/discard/read of a few blocks (when it has the potential to avoid many GB of writes for zeroing) is surely easy and worthwhile?

In any case, I thought that discussion was about a device that didn't report BLKDISCARDSZEROES=1, but only that a normal DISCARD would read back zero until the next restart? That prevents optimizations like "read until we see non-zero data, then start writing zeroes", which would still be faster for many RAID devices (or older kernels that don't have DISCARD/ZERO support at all).

Cheers, Andreas

Just to further confuse things, if we just want to zero a device, there is the (relatively old) WRITE_SAME command that arrays use. Note that it is quite a bit faster than doing this from the server since you only transfer over one block of data and the disk firmware does the rest - no data transfer for each block once you start.

It can certainly take a long, long time, but would be faster than zeroing a drive with write() system calls :)

ric

#18 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Ric" == Ric Wheeler <[email protected]> writes:

Ric> Just to further confuse things, if we just want to zero a device, Ric> there is the (relatively old) WRITE_SAME command that arrays Ric> use. Note that it is quite a bit faster than doing this from the Ric> server since you only transfer over one block of data and the disk Ric> firmware does the rest - no data transfer for each block once you Ric> start.

Ric> It can certainly take a long, long time, but would be faster than Ric> zeroing a drive with write() system calls :)

I took some stabs at this in the spring. And while it looked like a good idea on paper it turned out not to be a huge win unless the FC link was heavily congested due to traffic to other devices.

First of all many drives have a cap on the maximum number of blocks that can be written using one WRITE SAME command. Typically you can only write 16-32 megs at a time. So I needed to have a bunch of magic to scale down and retry while attempting to find the sweet spot.

Fred tried to convince T10 that it would be nice to have a field in the block limits VPD that would indicate the max WRITE SAME blocks a device supported. But T10 thought that was a bad idea and the proposal was rejected. Otherwise I would have wired that up and we could have handled generic WRITE SAME like we do the discard case.

The other problem is that the WRITE SAME may take a looong time. And so we need special timeouts in place to prevent regular error handling from kicking in while the drive is busy wiping stuff.

I guess we could just pick a number (16 MB, maybe) and define that as the max. Picking a low number also has the benefit of being less likely to interfere with timeouts.

If there's interest I'll be happy to revisit my patches...

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#19 lore raw
In reply to Andreas Dilger
Quoted 16 lines · Andreas Dilger
Quoted 7 lines · Ric Wheeler
Quoted 6 lines · Lukas Czerner

Also there is a big win, when discard also zeroes data, because in that case we can just skip inode table initialization (zeroing) without any need of in-kernel lazyinit code enabled. And we get all this for free. It was introduced with Sandeens patch:

Quoted 7 lines
Quoted 3 lines

http://marc.info/?l=linux-ext4&m=128234048208327&w=2

So, I would rather leave it on by default.

You cannot 100% depend on discard zeroing blocks - that is not a universal requirement of devices that support it. Specifically, for ATA devices, I think that there are optional bits that specify how a device will behave when you read from a trimmed region.

Quoted 8 lines

That patch also checks for the zeroing feature. When this patch was first under discussion, I proposed that we validate that the device is actually zeroed by doing a write a non-zero block to the disk and then calling discard+zero for that region, and reading back the block and verifying it.

Eric wasn't convinced that was necessary, maybe you can convince him more...

*grin* I wouldn't -veto- it, but I think we have to maintain a balance between defensive programming and trying to handle every possible piece of buggy hardware by doing too many extra gyrations in the code...

-Eric

Quoted 1 line

Cheers, Andreas

#20 lore raw
In reply to Andreas Dilger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Andreas" == Andreas Dilger <[email protected]> writes:

Andreas> When this patch was first under discussion, I proposed that we Andreas> validate that the device is actually zeroed by doing a write a Andreas> non-zero block to the disk and then calling discard+zero for Andreas> that region, and reading back the block and verifying it.

The problem is that the devices that get this wrong don't deterministically return zeroes. So they may return zeroes for the discarded test block but garbage for the stuff you actually depend on being cleared.

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#21 lore raw
In reply to Lukas Czerner
Quoted 5 lines · Lukas Czerner
Quoted 5 lines
Quoted 5 lines

There is a concern that discard might
prevent data recovery after fsck because it might be already discarded (some weird fs corruption?) in pass 5. However in my opinion this is a very small window (if there even is any), because we have already passed check 1-4 and we have just confirmed that group descriptors should be ok.

I don't totally agree. When users have a serious filesystem problem, the first thing they normally do is run e2fsck to see if it is corrected (it may even be done automatically at boot after errors=panic causing a reboot.

After that, they may want to recover some more data (e.g. with ext3grep, or restore an e2image of the metadata, and re-run e2fsck). If e2fsck will discard all of the data then any data recovery will be impossible.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

On the other hand there is nothing to be afraid of in the case of mkfs, because we can not possibly lose any relevant data, because discard is done before the filesystem gets created.

Well, I've worked several times with users that have accidentally repartitioned and/or reformatted over top of their important data, and it is usually possible to recover some data from this. Again, with discard that would be impossible.

I agree that with mke2fs there is less often a need to do that, and the normal intent of mke2fs is to destroy the previously-existing data, which is why I don't totally object to allowing discard by default for it. The intent of e2fsck is different however, since it is usually used for recovery purposes.

I'm trying to think of a good heuristic for when discard could be chosen automatically, but have a hard time doing so:

- the current heuristic of "no block bitmap errors" is not strong enough...
- even a completely clean e2fsck is not sufficient, because sometimes/often in the case of serious corruption a second e2fsck is run to ensure all the problems have been fixed
- after "N" consecutive clean e2fsck runs might be enough, but we don't have a counter for that today (but it isn't hard to add, if we are changing the e2fsck code anyway). That said, e2fsck is run so rarely on some systems that this would equate to "never" in some cases

Cheers, Andreas

#22 lore raw
Quoted 13 lines · Andreas Dilger
Quoted 6 lines · Lukas Czerner
Quoted 6 lines
Quoted 6 lines

There is a concern that discard might prevent data recovery after fsck because it might be already discarded (some weird fs corruption?) in pass 5. However in my opinion this is a very small window (if there even is any), because we have already passed check 1-4 and we have just confirmed that group descriptors should be ok.

I don't totally agree. When users have a serious filesystem problem, the first thing they normally do is run e2fsck to see if it is corrected (it may even be done automatically at boot after errors=panic causing a reboot.

Quoted 4 lines

After that, they may want to recover some more data (e.g. with ext3grep, or restore an e2image of the metadata, and re-run e2fsck). If e2fsck will discard all of the data then any data recovery will be impossible.

Could set it to only issue discard when the check was clean....

It could still be an option of course, but that might be safer still.

-Eric

#23 lore raw
Quoted 19 lines · Eric Sandeen
Quoted 13 lines · Andreas Dilger
Quoted 6 lines · Lukas Czerner
Quoted 6 lines
Quoted 6 lines

There is a concern that discard might prevent data recovery after fsck because it might be already discarded (some weird fs corruption?) in pass 5. However in my opinion this is a very small window (if there even is any), because we have already passed check 1-4 and we have just confirmed that group descriptors should be ok.

I don't totally agree. When users have a serious filesystem problem, the first thing they normally do is run e2fsck to see if it is corrected (it may even be done automatically at boot after errors=panic causing a reboot.

Quoted 4 lines

After that, they may want to recover some more data (e.g. with ext3grep, or restore an e2image of the metadata, and re-run e2fsck). If e2fsck will discard all of the data then any data recovery will be impossible.

Quoted 6 lines

Could set it to only issue discard when the check was clean....

It could still be an option of course, but that might be safer still.

-Eric

Well, this is the way I am doing it now: in pass 5 (which is the last one), after the group descriptors has been checked, so there is an assumption that it should be clean. But I agree, that it should be off by default in fsck.

-Lukas