Skip to contentSkip to conversation

dm mpath: disable WRITE SAME if it fails

PATCH In review Mike Snitzer 11 participants 25 messages Sep 19, 2013 – Sep 26, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 dm mpath: disable WRITE SAME if it fails

Workaround the SCSI layer's problematic WRITE SAME heuristics by disabling WRITE SAME in the DM multipath device's queue_limits if an underlying device disabled it.

The WRITE SAME heuristics, with both the original commit 5db44863b6eb ("[SCSI] sd: Implement support for WRITE SAME") and the updated commit 66c28f971 ("[SCSI] sd: Update WRITE SAME heuristics"), default to enabling WRITE SAME(10) even without successfully determining it is supported. After the first failed WRITE SAME the SCSI layer will disable WRITE SAME for the device (by setting sdkp->device->no_write_same which results in 'max_write_same_sectors' in device's queue_limits to be set to 0).

When a device is stacked ontop of such a SCSI device any changes to that SCSI device's queue_limits do not automatically propagate up the stack. As such, a DM multipath device will not have its WRITE SAME support disabled. This causes the block layer to continue to issue WRITE SAME requests to the mpath device which causes paths to fail and (if mpath IO isn't configured to queue when no paths are available) it will result in actual IO errors to the upper layers.

This fix doesn't help configurations that have additional devices stacked ontop of the mpath device (e.g. LVM created linear DM devices ontop). A proper fix that restacks all the queue_limits from the bottom of the device stack up will need to be explored if SCSI will continue to use this model of optimistically allowing op codes and then disabling them after they fail for the first time.

Before this patch:

EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts: (null)
device-mapper: multipath: XXX snitm debugging: got -EREMOTEIO (-121)
device-mapper: multipath: XXX snitm debugging: failing WRITE SAME IO with error=-121
end_request: critical target error, dev dm-6, sector 528
dm-6: WRITE SAME failed. Manually zeroing.
device-mapper: multipath: Failing path 8:112.
end_request: I/O error, dev dm-6, sector 4616
dm-6: WRITE SAME failed. Manually zeroing.
end_request: I/O error, dev dm-6, sector 4616
end_request: I/O error, dev dm-6, sector 5640
end_request: I/O error, dev dm-6, sector 6664
end_request: I/O error, dev dm-6, sector 7688
end_request: I/O error, dev dm-6, sector 524288
Buffer I/O error on device dm-6, logical block 65536
lost page write due to I/O error on dm-6
JBD2: Error -5 detected when updating journal superblock for dm-6-8.
end_request: I/O error, dev dm-6, sector 524296
Aborting journal on device dm-6-8.
end_request: I/O error, dev dm-6, sector 524288
Buffer I/O error on device dm-6, logical block 65536
lost page write due to I/O error on dm-6
JBD2: Error -5 detected when updating journal superblock for dm-6-8.

# cat /sys/block/sdh/queue/write_same_max_bytes
0
# cat /sys/block/dm-6/queue/write_same_max_bytes
33553920

After this patch:

EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts: (null)
device-mapper: multipath: XXX snitm debugging: got -EREMOTEIO (-121)
device-mapper: multipath: XXX snitm debugging: WRITE SAME I/O failed with error=-121
end_request: critical target error, dev dm-6, sector 528
dm-6: WRITE SAME failed. Manually zeroing.

# cat /sys/block/sdh/queue/write_same_max_bytes
0
# cat /sys/block/dm-6/queue/write_same_max_bytes
0

It should be noted that WRITE SAME support wasn't enabled in DM multipath until v3.10.

Cc Martin K. Petersen <[email protected]>
Cc Hannes Reinecke <[email protected]>
---
 drivers/md/dm-mpath.c         |   11 ++++++++++-
 drivers/md/dm.c               |   11 +++++++++++
 include/linux/device-mapper.h |    3 ++-
 3 files changed, 23 insertions(+), 2 deletions(-)
drivers/md/dm-mpath.c +10 -1
index 0b3955f..de570a5 100644
@@ -1299,8 +1299,17 @@ static int do_end_io(struct multipath *m, struct request *clone,
 	if (!error && !clone->errors)
 		return 0;	/* I/O complete */
 
-	if (noretry_error(error))
+	if (noretry_error(error)) {
+		if ((clone->cmd_flags & REQ_WRITE_SAME) &&
+		    !clone->q->limits.max_write_same_sectors) {
+			struct queue_limits *limits;
+
+			/* device doesn't really support WRITE SAME, disable it */
+			limits = dm_get_queue_limits(dm_table_get_md(m->ti->table));
+			limits->max_write_same_sectors = 0;
+		}
 		return error;
+	}
 
 	if (mpio->pgpath)
 		fail_path(mpio->pgpath);
drivers/md/dm.c +11 -0
index 033dc26..b3e26c7 100644
@@ -2323,6 +2323,17 @@ struct target_type *dm_get_immutable_target_type(struct mapped_device *md)
 }
 
 /*
+ * The queue_limits are only valid as long as you have a reference
+ * count on 'md'.
+ */
+struct queue_limits *dm_get_queue_limits(struct mapped_device *md)
+{
+	BUG_ON(!atomic_read(&md->holders));
+	return &md->queue->limits;
+}
+EXPORT_SYMBOL_GPL(dm_get_queue_limits);
+
+/*
  * Fully initialize a request-based queue (->elevator, ->request_fn, etc).
  */
 static int dm_init_request_based_queue(struct mapped_device *md)
include/linux/device-mapper.h +2 -1
index 653073d..ed419c6 100644
@@ -406,13 +406,14 @@ int dm_noflush_suspending(struct dm_target *ti);
 union map_info *dm_get_mapinfo(struct bio *bio);
 union map_info *dm_get_rq_mapinfo(struct request *rq);
 
+struct queue_limits *dm_get_queue_limits(struct mapped_device *md);
+
 /*
  * Geometry functions.
  */
 int dm_get_geometry(struct mapped_device *md, struct hd_geometry *geo);
 int dm_set_geometry(struct mapped_device *md, struct hd_geometry *geo);
 
-
 /*-----------------------------------------------------------------
  * Functions for manipulating device-mapper tables.
  *---------------------------------------------------------------*/
#1 lore raw

SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

Quoted 3 lines · Mike Snitzer

Workaround the SCSI layer's problematic WRITE SAME heuristics by disabling WRITE SAME in the DM multipath device's queue_limits if an underlying device disabled it.

...

Quoted 6 lines

This fix doesn't help configurations that have additional devices stacked ontop of the mpath device (e.g. LVM created linear DM devices ontop). A proper fix that restacks all the queue_limits from the bottom of the device stack up will need to be explored if SCSI will continue to use this model of optimistically allowing op codes and then disabling them after they fail for the first time.

I really don't think we can afford to keep SCSI's current heuristics for enabling WRITE SAME, re-stating them for the benefit of others:
1) check if WRITE SAME is supported by sending REPORT SUPPORTED OPERATION CODES to the device
2a) if REPORT SUPPORTED OPERATION CODES shows WRITE SAME is supported,
enable WRITE SAME
2b) if REPORT SUPPORTED OPERATION CODES shows WRITE SAME is not
supported, disable WRITE SAME
2c) if REPORT SUPPORTED OPERATION CODES isn't supported _and_ the device
doesn't have an ATA Information VPD page: enable WRITE SAME
- if/when WRITE SAME does fail, disable it on the failing device

AFAIK the reason for these heuristics is: devices that do support WRITE SAME cannot properly report as much because they don't support REPORT SUPPORTED OPERATION CODES -- this lack of RSOC support is apparently very common?

I can appreciate the idea behind the current heuristics but I think the prevelence of the other side of the spectrum (SCSI devices that don't support RSOC or WRITE SAME) was underestimated. As such we're seeing a fair amount of WRITE SAME error noise on these systems -- that noise is itself considered a bug to many.

Also, please see my comment in a related Fedora 19 BZ:
https://bugzilla.redhat.com/show_bug.cgi?id=995271#c23

As I say in that comment:
"A proper fix could be to make SCSI's default be to disable WRITE SAME for devices that don't properly report they support it. And possibly have a whitelist to opt-in to enabling WRITE SAME for select targets."

I'm open to any ideas you might have.

Thanks,
Mike

#2 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Mike" == Mike Snitzer <[email protected]> writes:

Mike,

Mike> AFAIK the reason for these heuristics is: devices that do support Mike> WRITE SAME cannot properly report as much because they don't Mike> support REPORT SUPPORTED OPERATION CODES -- this lack of RSOC Mike> support is apparently very common?

Only a handful of the very latest and greatest devices support RSOC. The number of devices that support WRITE SAME is orders of magnitude larger.

Last I checked I had exactly 1 out of about 100 devices in my lab that supported RSOC.

Mike> I can appreciate the idea behind the current heuristics but I Mike> think the prevelence of the other side of the spectrum (SCSI Mike> devices that don't support RSOC or WRITE SAME) was underestimated.

If you by "devices" mean vintage PCI RAID controllers that don't pass things through correctly, then yes. I don't think I have a single SCSI drive that doesn't support WRITE SAME. And all the controllers I tested with here worked fine.

Mike> As I say in that comment: "A proper fix could be to make SCSI's Mike> default be to disable WRITE SAME for devices that don't properly Mike> report they support it. And possibly have a whitelist to opt-in Mike> to enabling WRITE SAME for select targets."

The problem with the opt-in approach is that there are orders of magnitude more devices that would need to get it enabled than there are broken ones that need it disabled.

There are only a couple of handfuls of RAID controller drivers. We've been working through the issues on these on a case by case basis.

Yes, I totally agree it sucks. And I hate that things broke for people with Areca and 3ware. But we got those fixed. And it's way easier to blacklist "all devices hanging off RAID driver xyz" than it is to whitelist every SCSI drive known to man. It sucks in the short term but is better long term.

The major headache here of course is that WRITE SAME is inherently destructive. We can't just fire off one during discovery and see if it works. For WRITE you can issue a command with a transfer length of 0 to see if things work. But unfortunately for WRITE SAME a transfer length of zero means "wipe the entire device". Yikes!

I guess we could read one sector and try to write it back using WRITE SAME and a block count of one. But it's really icky. And I don't like the notion of actually writing things during discovery.

As far as being able to trigger a restacking of the queue limits I think it's inevitable. We see more and more devices that change properties after a firmware upgrade. I think we'll just have to bite the bullet and work on that...

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#3 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

Quoted 50 lines · Martin K. Petersen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Mike" == Mike Snitzer <[email protected]> writes:

Mike,

Mike> AFAIK the reason for these heuristics is: devices that do support Mike> WRITE SAME cannot properly report as much because they don't Mike> support REPORT SUPPORTED OPERATION CODES -- this lack of RSOC Mike> support is apparently very common?

Only a handful of the very latest and greatest devices support RSOC. The number of devices that support WRITE SAME is orders of magnitude larger.

Last I checked I had exactly 1 out of about 100 devices in my lab that supported RSOC.

Mike> I can appreciate the idea behind the current heuristics but I Mike> think the prevelence of the other side of the spectrum (SCSI Mike> devices that don't support RSOC or WRITE SAME) was underestimated.

If you by "devices" mean vintage PCI RAID controllers that don't pass things through correctly, then yes. I don't think I have a single SCSI drive that doesn't support WRITE SAME. And all the controllers I tested with here worked fine.

Mike> As I say in that comment: "A proper fix could be to make SCSI's Mike> default be to disable WRITE SAME for devices that don't properly Mike> report they support it. And possibly have a whitelist to opt-in Mike> to enabling WRITE SAME for select targets."

The problem with the opt-in approach is that there are orders of magnitude more devices that would need to get it enabled than there are broken ones that need it disabled.

There are only a couple of handfuls of RAID controller drivers. We've been working through the issues on these on a case by case basis.

Yes, I totally agree it sucks. And I hate that things broke for people with Areca and 3ware. But we got those fixed. And it's way easier to blacklist "all devices hanging off RAID driver xyz" than it is to whitelist every SCSI drive known to man. It sucks in the short term but is better long term.

The major headache here of course is that WRITE SAME is inherently destructive. We can't just fire off one during discovery and see if it works. For WRITE you can issue a command with a transfer length of 0 to see if things work. But unfortunately for WRITE SAME a transfer length of zero means "wipe the entire device". Yikes!

Quoted 8 lines

I guess we could read one sector and try to write it back using WRITE SAME and a block count of one. But it's really icky. And I don't like the notion of actually writing things during discovery.

As far as being able to trigger a restacking of the queue limits I think it's inevitable. We see more and more devices that change properties after a firmware upgrade. I think we'll just have to bite the bullet and work on that...

Would a closer examination of the available VPD pages help? For example support for the Logical Block Provisioning and Block Limits VPD pages. Given either of those two pages, even if the WRITE SAME specific fields in those pages are not set, it is unlikely that sending a WRITE SAME (when actually required rather than at discovery) would wedge the disk/controller.

If WSNZ is set in the Block Limits VPD pages then it should be "safe" ** to send a zero length WRITE SAME command to a LU. And that is another good reason to check the response of a VPD page request carefully (e.g. the echo-ed page_code in byte 1 and a sensible page_length in bytes 2 and 3) since crap devices often return a standard INQUIRY response to a VPD page request.

Doug Gilbert

** there is departure from the normal "do nothing when
transfer_length or number_of_LB fields are zero":
In the case of the WS command with number_of_LBs=0 and WSNZ=1 in the BLOCK LIMITS VPD page, the response should be ILLEGAL_REQUEST with INVALID FIELD IN CDB. I guess if it yielded status=GOOD in that case and you heard the disk clicking, you might get quite worried :-)

#4 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

In reply to Martin K. Petersen

On Fri, 2013-09-20 at 18:03 -0400, Martin K. Petersen wrote:
...

Quoted 5 lines

Only a handful of the very latest and greatest devices support RSOC. The number of devices that support WRITE SAME is orders of magnitude larger.

Last I checked I had exactly 1 out of about 100 devices in my lab that supported RSOC.

...

Quoted 9 lines

The major headache here of course is that WRITE SAME is inherently destructive. We can't just fire off one during discovery and see if it works. For WRITE you can issue a command with a transfer length of 0 to see if things work. But unfortunately for WRITE SAME a transfer length of zero means "wipe the entire device". Yikes!

I guess we could read one sector and try to write it back using WRITE SAME and a block count of one. But it's really icky. And I don't like the notion of actually writing things during discovery.

...

Just out of curiosity, what do the devices that support WRITE SAME report for the MAXIMUM WRITE SAME LENGTH field in VPD page B0? The spec says that this can be zero if there is no restriction, but is there any chance that most/all of them report some nonzero value?

Expanding on Doug's thinking, perhaps there is some combination of VPD page availability / field values that could be used to explicitly enable WRITE SAME? Or, have you been through that already?

-Ewan

#5 lore raw

Re: [dm-devel] SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

Quoted 19 lines · Ewan Milne

On Fri, 2013-09-20 at 18:03 -0400, Martin K. Petersen wrote:
...

Quoted 5 lines

Only a handful of the very latest and greatest devices support RSOC. The number of devices that support WRITE SAME is orders of magnitude larger.

Last I checked I had exactly 1 out of about 100 devices in my lab that supported RSOC.

...

Quoted 9 lines

The major headache here of course is that WRITE SAME is inherently destructive. We can't just fire off one during discovery and see if it works. For WRITE you can issue a command with a transfer length of 0 to see if things work. But unfortunately for WRITE SAME a transfer length of zero means "wipe the entire device". Yikes!

I guess we could read one sector and try to write it back using WRITE SAME and a block count of one. But it's really icky. And I don't like the notion of actually writing things during discovery.

...

Quoted 9 lines

Just out of curiosity, what do the devices that support WRITE SAME report for the MAXIMUM WRITE SAME LENGTH field in VPD page B0? The spec says that this can be zero if there is no restriction, but is there any chance that most/all of them report some nonzero value?

Expanding on Doug's thinking, perhaps there is some combination of VPD page availability / field values that could be used to explicitly enable WRITE SAME? Or, have you been through that already?

Hehe. Won't do any good.

My drives support 'report opcodes', and report that write same is supported:
...
93 16 Write same(16)
...

but no support for page 'b0'. And yes, these are real SAS drives.

Cheers,

Hannes

Signature · 8 lines
Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#20 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

In reply to Martin K. Petersen

Il 21/09/2013 00:03, Martin K. Petersen ha scritto:

Quoted 6 lines

The major headache here of course is that WRITE SAME is inherently destructive. We can't just fire off one during discovery and see if it works. For WRITE you can issue a command with a transfer length of 0 to see if things work. But unfortunately for WRITE SAME a transfer length of zero means "wipe the entire device". Yikes!

What about WRITE SAME with an out-of-range lba? Or lba equal to the size of the disk and #blocks equal to zero.

Paolo

#6 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

In reply to Hannes Reinecke
Quoted 31 lines · Hannes Reinecke
Quoted 19 lines · Ewan Milne

On Fri, 2013-09-20 at 18:03 -0400, Martin K. Petersen wrote:
...

Quoted 5 lines

Only a handful of the very latest and greatest devices support RSOC. The number of devices that support WRITE SAME is orders of magnitude larger.

Last I checked I had exactly 1 out of about 100 devices in my lab that supported RSOC.

...

Quoted 9 lines

The major headache here of course is that WRITE SAME is inherently destructive. We can't just fire off one during discovery and see if it works. For WRITE you can issue a command with a transfer length of 0 to see if things work. But unfortunately for WRITE SAME a transfer length of zero means "wipe the entire device". Yikes!

I guess we could read one sector and try to write it back using WRITE SAME and a block count of one. But it's really icky. And I don't like the notion of actually writing things during discovery.

...

Quoted 9 lines

Just out of curiosity, what do the devices that support WRITE SAME report for the MAXIMUM WRITE SAME LENGTH field in VPD page B0? The spec says that this can be zero if there is no restriction, but is there any chance that most/all of them report some nonzero value?

Expanding on Doug's thinking, perhaps there is some combination of VPD page availability / field values that could be used to explicitly enable WRITE SAME? Or, have you been through that already?

Hehe. Won't do any good.

Quoted 7 lines

My drives support 'report opcodes', and report that write same is supported:
...
93 16 Write same(16)
...

but no support for page 'b0'. And yes, these are real SAS drives.

Your drive would be fine then since it does support RSOC and specifies WRITE SAME is supported. Pretty sure Ewan was only suggesting to resort to looking at the VPD page if RSOC isn't supported.

So are there drives like this?:
1) don't support RSOC
2) do support WRITE SAME
3) do populate VPD page with either WRITE SAME w/ discard bit set or UNMAP?

maintainer
#21 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

In reply to Paolo Bonzini
Quoted 10 lines · Paolo Bonzini

Il 21/09/2013 00:03, Martin K. Petersen ha scritto:

Quoted 6 lines

The major headache here of course is that WRITE SAME is inherently destructive. We can't just fire off one during discovery and see if it works. For WRITE you can issue a command with a transfer length of 0 to see if things work. But unfortunately for WRITE SAME a transfer length of zero means "wipe the entire device". Yikes!

What about WRITE SAME with an out-of-range lba? Or lba equal to the size of the disk and #blocks equal to zero.

That's inviting undefined behaviour in a data destructive command ... not something we really want to do.

James

#7 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

In reply to Mike Snitzer
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Mike" == Mike Snitzer <[email protected]> writes:

Mike> So are there drives like this?:
Mike> 1) don't support RSOC
Mike> 2) do support WRITE SAME
Mike> 3) do populate VPD page with either WRITE SAME w/ discard bit set
Mike> or UNMAP?

Yes.

But again, the fundamental issue here is not the drives. It's the controller firmware. I am not aware of a single SPI/SAS/FC drive that does not support at least WRITE SAME(10).

For DIX and T10 PI I have a capabilities mask that each HBA driver fills out that tells the sd driver what the controller can do. And this is combined with whatever the drive reports to figure out whether integrity protection can be enabled.

I have been contemplating doing something similar for "fancy" SCSI commands. We could have a flag in the scsi host template that controls whether the device supports WRITE SAME, EXTENDED COPY, etc. The advantage being that we do the matching at discovery time instead of once a WRITE SAME is issued.

This would also permit HBA drivers to toggle the feature on a per instance basis. I.e. if "RAID controller firmware rev is lower than XYZ, do not support WRITE SAME".

I'll do a PoC later today...

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#8 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

Quoted 25 lines · Martin K. Petersen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Mike" == Mike Snitzer <[email protected]> writes:

Mike> So are there drives like this?:
Mike> 1) don't support RSOC
Mike> 2) do support WRITE SAME
Mike> 3) do populate VPD page with either WRITE SAME w/ discard bit set
Mike> or UNMAP?

Yes.

But again, the fundamental issue here is not the drives. It's the controller firmware. I am not aware of a single SPI/SAS/FC drive that does not support at least WRITE SAME(10).

For DIX and T10 PI I have a capabilities mask that each HBA driver fills out that tells the sd driver what the controller can do. And this is combined with whatever the drive reports to figure out whether integrity protection can be enabled.

I have been contemplating doing something similar for "fancy" SCSI commands. We could have a flag in the scsi host template that controls whether the device supports WRITE SAME, EXTENDED COPY, etc. The advantage being that we do the matching at discovery time instead of once a WRITE SAME is issued.

Quoted 5 lines

This would also permit HBA drivers to toggle the feature on a per instance basis. I.e. if "RAID controller firmware rev is lower than XYZ, do not support WRITE SAME".

I'll do a PoC later today...

Sounds promising, thanks!

maintainer
#22 lore raw

Re: [dm-devel] SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

In reply to Mike Snitzer
Quoted 7 lines · Mike Snitzer
Quoted 3 lines · Mike Snitzer

Workaround the SCSI layer's problematic WRITE SAME heuristics by disabling WRITE SAME in the DM multipath device's queue_limits if an underlying device disabled it.

Quoted 8 lines

...

Quoted 6 lines

This fix doesn't help configurations that have additional devices stacked ontop of the mpath device (e.g. LVM created linear DM devices ontop). A proper fix that restacks all the queue_limits from the bottom of the device stack up will need to be explored if SCSI will continue to use this model of optimistically allowing op codes and then disabling them after they fail for the first time.

Propagating the limits up the tree won't work anyway because of a race condition. Think of this case:

1. We test queue limits and find out that the device supports WRITE_SAME
2. We build a WRITE_SAME bio to be submitted.
3. The user reconfigures a device so that it no longer supports WRITE_SAME (for example, he adds a snapshot or converts linear volume to a mirror).
4. We submit the bio that we built in step 2.
5. The bio fails.

I think the correct solution is to specify that WRITE SAME request may fail anytime and the caller is responsible for retrying with normal WRITE request if WRITE SAME failed.

If the caller needs to send WRITE SAME, it should try to send it and if it fails, set a flag that the device doesn't support WRITE SAME and not send WRITE SAME in the future. (we could also reset that flag after some long time, in case the device was reconfigured to support WRITE SAME).

Quoted 31 lines

I really don't think we can afford to keep SCSI's current heuristics for enabling WRITE SAME, re-stating them for the benefit of others:
1) check if WRITE SAME is supported by sending REPORT SUPPORTED OPERATION CODES to the device
2a) if REPORT SUPPORTED OPERATION CODES shows WRITE SAME is supported,
enable WRITE SAME
2b) if REPORT SUPPORTED OPERATION CODES shows WRITE SAME is not
supported, disable WRITE SAME
2c) if REPORT SUPPORTED OPERATION CODES isn't supported _and_ the device
doesn't have an ATA Information VPD page: enable WRITE SAME
- if/when WRITE SAME does fail, disable it on the failing device

AFAIK the reason for these heuristics is: devices that do support WRITE SAME cannot properly report as much because they don't support REPORT SUPPORTED OPERATION CODES -- this lack of RSOC support is apparently very common?

I can appreciate the idea behind the current heuristics but I think the prevelence of the other side of the spectrum (SCSI devices that don't support RSOC or WRITE SAME) was underestimated. As such we're seeing a fair amount of WRITE SAME error noise on these systems -- that noise is itself considered a bug to many.

Also, please see my comment in a related Fedora 19 BZ:
https://bugzilla.redhat.com/show_bug.cgi?id=995271#c23

As I say in that comment:
"A proper fix could be to make SCSI's default be to disable WRITE SAME for devices that don't properly report they support it. And possibly have a whitelist to opt-in to enabling WRITE SAME for select targets."

Quoted 4 lines

I'm open to any ideas you might have.

Thanks,
Mike

What does happend if we send WRITE SAME to a disk that doesn't support it? If we only get "Invalid field in CDB" or similar sense code, we could just propagate it up and let the caller retry.

If we could get something worse (firmware crash), we must blacklist it.

Another idea: we could kill the REQ_WRITE_SAME flag entirely - if the some subsystem needs to clear some area of a disk, it builds a bio with all vector entries pointing to the zero page. Drivers that don't support WRITE SAME (such as IDE) will implement it doing DMA from the zero page. Drivers that support WRITE SAME can detect this special case and convert a normal WRITE request to WRITE SAME. The request is converted to WRITE SAME at the lowest level (in the SCSI driver), it passes as normal WRITE request through dm and md midlayers - thus, we magically get WRITE SAME support in all md and dm drivers without writing additional code for it.

Mikulas

#18 lore raw

Re: [dm-devel] SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

In reply to Hannes Reinecke
Quoted 4 lines · Hannes Reinecke

My drives support 'report opcodes', and report that write same is
supported: ... 93 16 Write same(16) ...

but no support for page 'b0'. And yes, these are real SAS drives.

So the question is, how many devices get the protect bit in the std inquiry
incorrect?

If that is mostly correct, how about std inquiry (SPC>2), protect bit, and then
VPD page 0x86 (or alternatively then do the READ CAPACITY, P_TYPE instead of page 0x86)?

After all, the set of valid read/write opcodes is limited by the protection
mode format, yes?

#19 lore raw

Re: [dm-devel] SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

Quoted 9 lines · Jeremy Linton
Quoted 4 lines · Hannes Reinecke

My drives support 'report opcodes', and report that write same is
supported: ... 93 16 Write same(16) ...

but no support for page 'b0'. And yes, these are real SAS drives.

So the question is, how many devices get the protect bit in the std inquiry
incorrect?

To that specific question: all of them get it right because if they don't the disk (LU) does not support PI (or at least that is the conclusion the application client should reach). What the sales blurb for the device may say is another thing.

More generally why are we using WRITE SAME:
a) associated with LB provisioning?
b) associated with PI (DIF)?
c) as an optimized multi-block write of the same contents?

I'm not aware there is any special relationship between WS and PI. That leaves a) and c) . I'm guessing it is more important for a) .

Quoted 6 lines

If that is mostly correct, how about std inquiry (SPC>2), protect bit, and then
VPD page 0x86 (or alternatively then do the READ CAPACITY, P_TYPE instead of page 0x86)?

After all, the set of valid read/write opcodes is limited by the protection
mode format, yes?

Yes: READ(32), WRITE(32), WRITE SAME(32), VERIFY(32) and WRITE AND VERIFY(32) are only valid for protection type 2 (sbc3r35h 4.22.2.2). I'm sure T10 have a good reason for that but they are keeping that reason to themselves.

Doug Gilbert

#23 lore raw

Re: [dm-devel] SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

In reply to Mikulas Patocka
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Mikulas" == Mikulas Patocka <[email protected]> writes:

Mikulas> Propagating the limits up the tree won't work anyway because of Mikulas> a race condition. Think of this case:

Mikulas> 1. We test queue limits and find out that the device supports
Mikulas> WRITE_SAME
Mikulas> 2. We build a WRITE_SAME bio to be submitted. Mikulas> 3. The user reconfigures a device so that it no longer supports
Mikulas> WRITE_SAME
Mikulas> (for example, he adds a snapshot or converts linear volume to a Mikulas> mirror).
Mikulas> 4. We submit the bio that we built in step 2. Mikulas> 5. The bio fails.

The existing "fall back to writes if write same fails" mechanism isn't going away. We absolutely need that.

The principal headache is caused by host adapters that perform SCSI-ATA or SCSI-<internal RAID implementation> translation and which do not correctly fail a WRITE SAME command that we send down to the disk.

Based on heuristics from querying the target we expect WRITE SAME to succeed so we enable it. But for whatever reason the controller is not able to handle the command. Most controllers/drivers will fail the command correctly and we'll disable WRITE SAME and fall back to manually zeroing. Some controllers, however, hang or fail the I/O in a way in which we do not pick up it's a WRITE SAME problem.

The other headache is what happens if a stacking driver gets -EIO or -EREMOTEIO on a WRITE SAME request. That's what Mike was trying to fix with his patch. Maybe it would make sense for us to use -EINVAL or something similar to single out the ILLEGAL REQUEST case. While -EREMOTEIO specifically calls out for "no retry" it might be helpful to indicate that we're dealing with an unsupported command. That way DM or MD can reconfigure themselves accordingly.

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#24 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

Quoted 7 lines · Martin K. Petersen

The other headache is what happens if a stacking driver gets -EIO or -EREMOTEIO on a WRITE SAME request. That's what Mike was trying to fix with his patch. Maybe it would make sense for us to use -EINVAL or something similar to single out the ILLEGAL REQUEST case. While -EREMOTEIO specifically calls out for "no retry" it might be helpful to indicate that we're dealing with an unsupported command. That way DM or MD can reconfigure themselves accordingly.

What I've committed as a v3.12 fix works, see:
http://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-linus&id=f84cb8a46a771f36a04a02c61ea635c968ed5f6a

"Workaround the SCSI layer's problematic WRITE SAME heuristics by disabling WRITE SAME in the DM multipath device's queue_limits if an underlying device disabled it."

So I'm trapping requests with REQ_WRITE_SAME whose underlying queue's queue_limits have disabled WRITE SAME. And as the before and after detailed in the patch header shows, it works.

I'm open to tweaking this later but will be leaving well enough alone for v3.12 by keeping this fix.

#9 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

In reply to Martin K. Petersen
Quoted 25 lines · Martin K. Petersen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Mike" == Mike Snitzer <[email protected]> writes:

Mike> So are there drives like this?:
Mike> 1) don't support RSOC
Mike> 2) do support WRITE SAME
Mike> 3) do populate VPD page with either WRITE SAME w/ discard bit set
Mike> or UNMAP?

Yes.

But again, the fundamental issue here is not the drives. It's the controller firmware. I am not aware of a single SPI/SAS/FC drive that does not support at least WRITE SAME(10).

For DIX and T10 PI I have a capabilities mask that each HBA driver fills out that tells the sd driver what the controller can do. And this is combined with whatever the drive reports to figure out whether integrity protection can be enabled.

I have been contemplating doing something similar for "fancy" SCSI commands. We could have a flag in the scsi host template that controls whether the device supports WRITE SAME, EXTENDED COPY, etc. The advantage being that we do the matching at discovery time instead of once a WRITE SAME is issued.

Quoted 6 lines

This would also permit HBA drivers to toggle the feature on a per instance basis. I.e. if "RAID controller firmware rev is lower than XYZ, do not support WRITE SAME".

I'll do a PoC later today...

Martin,

I'm afraid we have another problem. I'm currently working on to get discard working for our LSI2008 HBAs with attached sata-SSDs and the heuristics in sd_read_write_same with based on VPD page 0x89 is not correct for this HBA - its SATL supports write-same (although it does "Logical block address out of range" at the end of the device, I'm going to look into this tomorrow).

So allow LSI SATL or remove this check at all?

Thanks,
Bernd

#10 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

Quoted 35 lines · Bernd Schubert
Quoted 25 lines · Martin K. Petersen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Mike" == Mike Snitzer <[email protected]> writes:

Mike> So are there drives like this?:
Mike> 1) don't support RSOC
Mike> 2) do support WRITE SAME
Mike> 3) do populate VPD page with either WRITE SAME w/ discard bit set
Mike> or UNMAP?

Yes.

But again, the fundamental issue here is not the drives. It's the controller firmware. I am not aware of a single SPI/SAS/FC drive that does not support at least WRITE SAME(10).

For DIX and T10 PI I have a capabilities mask that each HBA driver fills out that tells the sd driver what the controller can do. And this is combined with whatever the drive reports to figure out whether integrity protection can be enabled.

I have been contemplating doing something similar for "fancy" SCSI commands. We could have a flag in the scsi host template that controls whether the device supports WRITE SAME, EXTENDED COPY, etc. The advantage being that we do the matching at discovery time instead of once a WRITE SAME is issued.

Quoted 6 lines

This would also permit HBA drivers to toggle the feature on a per instance basis. I.e. if "RAID controller firmware rev is lower than XYZ, do not support WRITE SAME".

I'll do a PoC later today...

Martin,

Quoted 8 lines

I'm afraid we have another problem. I'm currently working on to get discard working for our LSI2008 HBAs with attached sata-SSDs and the heuristics in sd_read_write_same with based on VPD page 0x89 is not correct for this HBA - its SATL supports write-same (although it does "Logical block address out of range" at the end of the device, I'm going to look into this tomorrow).

So allow LSI SATL or remove this check at all?

LSI implement their SATL in firmware inside their HBAs. Given their latest firmware release name (*Package_P17_IR_IT_Firmware* dated 9 August 2013) for their SAS-2 family *** you have 17 versions of that firmware potentially out there in the field. Updating that firmware is a fiddly process; I use a USB stick with DOS on it! LSI do fix things in their SATL when flaws are pointed out.

Generally speaking LSI's SATL is pretty good, at least compared to another SATL I can think of. They are both moving targets, and move independently.

Doug Gilbert

*** LSI's SAS-3 HBAs are still at "P1" which I assume is the
first publically released version.

#11 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

In reply to Bernd Schubert
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Bernd" == Bernd Schubert <[email protected]> writes:

Hey Bernd,

Bernd> I'm afraid we have another problem. I'm currently working on to Bernd> get discard working for our LSI2008 HBAs with attached sata-SSDs Bernd> and the heuristics in sd_read_write_same with based on VPD page Bernd> 0x89 is not correct for this HBA - its SATL supports write-same

This has nothing to do with the WRITE SAME heuristics.

It's true that depending on wind and whether we might issue WRITE SAME(10) or (16) with the UNMAP bit set to perform discard operations on the low level device. But we use a set of different (and somewhat more reliable) heuristics to decide which command to send down for that purpose.

For discards to a SATA device to work you need a recent phase LSI firmware. And you need the target mode firmware (IT). There is no UNMAP->DSM TRIM translation in the RAID (IR) firmware.

If your SATA SSDs reports DSM TRIM support, the LSI firmware will set LBPME=1 in READ CAPACITY(16) and the LOGICAL BLOCK PROVISIONING VPD page will indicate a preference for the UNMAP command (LBPU=1).

Also, LSI firmware is well-behaved in general and will report ILLEGAL REQUEST when you send down a command that can't be handled.

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#12 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

Quoted 21 lines · Martin K. Petersen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Bernd" == Bernd Schubert <[email protected]> writes:

Hey Bernd,

Bernd> I'm afraid we have another problem. I'm currently working on to Bernd> get discard working for our LSI2008 HBAs with attached sata-SSDs Bernd> and the heuristics in sd_read_write_same with based on VPD page Bernd> 0x89 is not correct for this HBA - its SATL supports write-same

This has nothing to do with the WRITE SAME heuristics.

It's true that depending on wind and whether we might issue WRITE SAME(10) or (16) with the UNMAP bit set to perform discard operations on the low level device. But we use a set of different (and somewhat more reliable) heuristics to decide which command to send down for that purpose.

For discards to a SATA device to work you need a recent phase LSI firmware. And you need the target mode firmware (IT). There is no UNMAP->DSM TRIM translation in the RAID (IR) firmware.

Quoted 6 lines

If your SATA SSDs reports DSM TRIM support, the LSI firmware will set LBPME=1 in READ CAPACITY(16) and the LOGICAL BLOCK PROVISIONING VPD page will indicate a preference for the UNMAP command (LBPU=1).

Also, LSI firmware is well-behaved in general and will report ILLEGAL REQUEST when you send down a command that can't be handled.

An example with a LSI 9212-4i4e running the latest firmware (P17) connected to a SATA SSD (via an expander):

# sg_vpd /dev/sg1 -p sinq
standard INQUIRY:
  PQual=0  Device_type=0  RMB=0  version=0x06  [SPC-4]
  [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=1  Resp_data_format=2
  SCCS=0  ACC=0  TPGS=0  3PC=0  Protect=0  [BQue=0]
  EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
  [RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
  Vendor_identification: ATA
  Product_identification: INTEL SSDSA2M080
  Product_revision_level: 02M3
# sg_vpd /dev/sg1 -p bl
Block limits VPD page (SBC):
  Write same no zero (WSNZ): 0
  Maximum compare and write length: 0 blocks
  Optimal transfer length granularity: 0 blocks
  Maximum transfer length: 0 blocks
  Optimal transfer length: 0 blocks
  Maximum prefetch length: 0 blocks
  Maximum unmap LBA count: 4194303
  Maximum unmap block descriptor count: 32
  Optimal unmap granularity: 1
  Unmap granularity alignment valid: 0
  Unmap granularity alignment: 0
  Maximum write same length: 0x0 blocks
# sg_vpd /dev/sg1 -p lbpv
Logical block provisioning VPD page (SBC):
  Unmap command supported (LBPU): 1
  Write same (16) with unmap bit supported (LBWS): 1
  Write same (10) with unmap bit supported (LBWS10): 0
  Logical block provisioning read zeros (LBPRZ): 0
  Anchored LBAs supported (ANC_SUP): 1
  Threshold exponent: 0
  Descriptor present (DP): 0
  Provisioning type: 0
# sg_opcodes -n /dev/sg1
Report supported operation codes: operation not supported

Room for improvement there. It also supports a useful set of mode pages (including some chageable fields) and two log pages.

Doug Gilbert

#13 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

Quoted 69 lines · Douglas Gilbert
Quoted 21 lines · Martin K. Petersen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Bernd" == Bernd Schubert <[email protected]> writes:

Hey Bernd,

Bernd> I'm afraid we have another problem. I'm currently working on to Bernd> get discard working for our LSI2008 HBAs with attached sata-SSDs Bernd> and the heuristics in sd_read_write_same with based on VPD page Bernd> 0x89 is not correct for this HBA - its SATL supports write-same

This has nothing to do with the WRITE SAME heuristics.

It's true that depending on wind and whether we might issue WRITE SAME(10) or (16) with the UNMAP bit set to perform discard operations on the low level device. But we use a set of different (and somewhat more reliable) heuristics to decide which command to send down for that purpose.

For discards to a SATA device to work you need a recent phase LSI firmware. And you need the target mode firmware (IT). There is no UNMAP->DSM TRIM translation in the RAID (IR) firmware.

Quoted 6 lines

If your SATA SSDs reports DSM TRIM support, the LSI firmware will set LBPME=1 in READ CAPACITY(16) and the LOGICAL BLOCK PROVISIONING VPD page will indicate a preference for the UNMAP command (LBPU=1).

Also, LSI firmware is well-behaved in general and will report ILLEGAL REQUEST when you send down a command that can't be handled.

An example with a LSI 9212-4i4e running the latest firmware (P17) connected to a SATA SSD (via an expander):

# sg_vpd /dev/sg1 -p sinq
standard INQUIRY:
   PQual=0  Device_type=0  RMB=0  version=0x06  [SPC-4]
   [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=1  Resp_data_format=2
   SCCS=0  ACC=0  TPGS=0  3PC=0  Protect=0  [BQue=0]
   EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
   [RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
   Vendor_identification: ATA
   Product_identification: INTEL SSDSA2M080
   Product_revision_level: 02M3
# sg_vpd /dev/sg1 -p bl
Block limits VPD page (SBC):
   Write same no zero (WSNZ): 0
   Maximum compare and write length: 0 blocks
   Optimal transfer length granularity: 0 blocks
   Maximum transfer length: 0 blocks
   Optimal transfer length: 0 blocks
   Maximum prefetch length: 0 blocks
   Maximum unmap LBA count: 4194303
   Maximum unmap block descriptor count: 32
   Optimal unmap granularity: 1
   Unmap granularity alignment valid: 0
   Unmap granularity alignment: 0
   Maximum write same length: 0x0 blocks
# sg_vpd /dev/sg1 -p lbpv
Logical block provisioning VPD page (SBC):
   Unmap command supported (LBPU): 1
   Write same (16) with unmap bit supported (LBWS): 1
   Write same (10) with unmap bit supported (LBWS10): 0
   Logical block provisioning read zeros (LBPRZ): 0
   Anchored LBAs supported (ANC_SUP): 1
   Threshold exponent: 0
   Descriptor present (DP): 0
   Provisioning type: 0
Quoted 8 lines
# sg_opcodes -n /dev/sg1
Report supported operation codes: operation not supported

Room for improvement there. It also supports a useful set of mode pages (including some chageable fields) and two log pages.

Both types of systems we have in-house neither block limits vpd nor READ_CAP16 return anything that would indicate discard is supported. But UNMAP and WRITE SAME unmap(*) just work fine.

I certainly don't want to cause any more write-same trouble, but as all layers initially have to assume write same is supported anyway and need to dynamically disable it if it fails, can't we also enable discard by default with WRITE SAME16 unmap? I'm going to send a PoC patch later on.

The older system I can play with for a few days has an Intel510 (SSDSC2MH25) connected to an LSI SAS9211-8i via a sas enclosure. Ignoring identificication string and revision level, sg_vdp output is almost the same here, but with the exception of

(wheezy)node02:~# sg_vpd /dev/sdb -p bl
Block limits VPD page (SBC):
[...]
  Maximum unmap LBA count: 0
  Maximum unmap block descriptor count: 0
  Optimal unmap granularity: 0
[...]

I think interesting for discard is also read cap 16:

Quoted 10 lines
(wheezy)node02:~# sg_readcap --16 /dev/sda
Read Capacity results:
   Protection: prot_en=0, p_type=0, p_i_exponent=0
   Logical block provisioning: lbpme=0, lbprz=0
   Last logical block address=490350671 (0x1d3a284f), Number of logical blocks=490350672
   Logical block length=512 bytes
   Logical blocks per physical block exponent=0
   Lowest aligned logical block address=0
Hence:
   Device size: 251059544064 bytes, 239429.0 MiB, 251.06 GB

So again no indication of discard support.

We also long ago flushed the IT fw and just recently updated to fw version 17

Quoted 2 lines

mpt2sas0: LSISAS2008: FWVersion(17.00.01.00), ChipRevision(0x03), BiosVersion(07.33.00.00)
mpt2sas0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)

Thanks,
Bernd

PS: LSI SATL with FWv17 seems to have an unmap bug - I cannot unmap the last sector:

Quoted 5 lines
(wheezy)node02:~# cat /sys/block/sdb/size
488397168
(wheezy)node02:~# sg_write_same --16 --unmap --verbose --lba=488397167 --num=1 /dev/sdb
Default data-out buffer set to 512 zeros

So write same works. But then unmap fails:

Quoted 6 lines
(wheezy)node02:~# sg_unmap --verbose --lba=488397167 --num=1 /dev/sdb
    unmap cdb: 42 00 00 00 00 00 00 00 18 00
unmap:  Fixed format, current;  Sense key: Illegal Request
 Additional sense: Logical block address out of range
  Info fld=0x1d1c596f [488397167]
bad field in UNMAP cdb

All sectors before that work fine:

Quoted 3 lines
(wheezy)node02:~# sg_unmap --verbose --lba=0 --num=488397167 /dev/sdb
    unmap cdb: 42 00 00 00 00 00 00 00 18 00
(
#14 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Bernd" == Bernd Schubert <[email protected]> writes:

Bernd,

Bernd> Both types of systems we have in-house neither block limits vpd Bernd> nor READ_CAP16 return anything that would indicate discard is Bernd> supported. But UNMAP and WRITE SAME unmap(*) just work fine.

I have a collection of different SSDs in a tray connected to an LSI SAS2008 ASIC. The 510 is the only drive that does not have LBPME=1. Chances are it's because DRAT and RZAT are not set but it could also be that the 510 is blacklisted.

Bernd> I certainly don't want to cause any more write-same trouble, but Bernd> as all layers initially have to assume write same is supported Bernd> anyway and need to dynamically disable it if it fails, can't we Bernd> also enable discard by default with WRITE SAME16 unmap?

No thanks :)

But you can force discards on by doing a:

 # echo -n unmap > /sys/class/scsi_disk/x:y:z:n/provisioning_mode

or

 # echo -n writesame_16 > /sys/class/scsi_disk/x:y:z:n/provisioning_mode

Create a udev rule if you like.

In any case I wouldn't recommend using TRIM on that drive...

Bernd> PS: LSI SATL with FWv17 seems to have an unmap bug - I cannot Bernd> unmap the last sector:

Yes, it appears there's an off-by-one bug in the UNMAP translation. Sumit, is this something you guys can look into?

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#15 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

Quoted 21 lines · Martin K. Petersen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Bernd" == Bernd Schubert <[email protected]> writes:

Bernd,

Bernd> Both types of systems we have in-house neither block limits vpd Bernd> nor READ_CAP16 return anything that would indicate discard is Bernd> supported. But UNMAP and WRITE SAME unmap(*) just work fine.

I have a collection of different SSDs in a tray connected to an LSI SAS2008 ASIC. The 510 is the only drive that does not have LBPME=1. Chances are it's because DRAT and RZAT are not set but it could also be that the 510 is blacklisted.

Bernd> I certainly don't want to cause any more write-same trouble, but Bernd> as all layers initially have to assume write same is supported Bernd> anyway and need to dynamically disable it if it fails, can't we Bernd> also enable discard by default with WRITE SAME16 unmap?

No thanks :)

Quoted 7 lines

But you can force discards on by doing a:

  # echo -n unmap > /sys/class/scsi_disk/x:y:z:n/provisioning_mode

or

  # echo -n writesame_16 > /sys/class/scsi_disk/x:y:z:n/provisioning_mode

D'oh, I didn't know about this parameter! THANKS a lot!

Quoted 2 lines

Create a udev rule if you like.

Definitely, we have rules anyway.

Quoted 2 lines

In any case I wouldn't recommend using TRIM on that drive...

Hmm yeah, the problem is that that these SSDs are used in our compute nodes (but we as file system group can also sometimes use it for benchmarking) and these SSDs slow down rather quickly without trim. The current procedure is to remove everything, to sg_unmap the full device and to re-create anything. So anything that would allow us to do it on the fly is very welcome. I don't think we need DRAT and RZAT, as we only use raid0 for these nodes.

So thanks a lot for your hint about the provisioning_mode!

Quoted 8 lines

Bernd> PS: LSI SATL with FWv17 seems to have an unmap bug - I cannot Bernd> unmap the last sector:

Yes, it appears there's an off-by-one bug in the UNMAP translation. Sumit, is this something you guys can look into?

Thanks again,
Bernd

#16 lore raw

Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

In reply to Martin K. Petersen
Quoted 31 lines · Martin K. Petersen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Bernd" == Bernd Schubert <[email protected]> writes:

Bernd,

Bernd> Both types of systems we have in-house neither block limits vpd Bernd> nor READ_CAP16 return anything that would indicate discard is Bernd> supported. But UNMAP and WRITE SAME unmap(*) just work fine.

I have a collection of different SSDs in a tray connected to an LSI SAS2008 ASIC. The 510 is the only drive that does not have LBPME=1. Chances are it's because DRAT and RZAT are not set but it could also be that the 510 is blacklisted.

Bernd> I certainly don't want to cause any more write-same trouble, but Bernd> as all layers initially have to assume write same is supported Bernd> anyway and need to dynamically disable it if it fails, can't we Bernd> also enable discard by default with WRITE SAME16 unmap?

No thanks :)

But you can force discards on by doing a:

  # echo -n unmap > /sys/class/scsi_disk/x:y:z:n/provisioning_mode

or

  # echo -n writesame_16 > /sys/class/scsi_disk/x:y:z:n/provisioning_mode

Create a udev rule if you like.

Quoted 8 lines

In any case I wouldn't recommend using TRIM on that drive...

Bernd> PS: LSI SATL with FWv17 seems to have an unmap bug - I cannot Bernd> unmap the last sector:

Yes, it appears there's an off-by-one bug in the UNMAP translation. Sumit, is this something you guys can look into?

Hi Sumit,
While you are looking at the HBA firmware could you fix this minor annoyance:

# lsscsi -H -t
.....
[6]    mpt3sas       sas:0x500605b006d3b510
# smp_rep_general /dev/mpt3ctl -s 0x500605b006d3b510
Report general response:
  expander change count: 0
  expander route indexes: 0
  long response: 0
  number of phys: 0
  zone configuring: 0
  self configuring: 0
  externally configurable route table: 0
  initiates SSP close: 0
  enclosure logical identifier (hex): b005065010b5d306

So that is a SMP REPORT GENERAL (RG) directed at the HBA itself. So either my code is incorrectly decoding the "enclosure logical identifier" or ... . That field was defined in SAS 1.1 so no excuses on that front. 0x0 would be better than a shuffled version of the HBA's own SAS address.

Same bug/feature in all versions of SAS-2 and now SAS-3 firmware that I have tried.

Doug Gilbert

BTW The bsg driver can send a RG to the HBA like this:
smp_rep_general /dev/bsg/sas_host6
So the intent is clearer and the response is the same.

#17 lore raw

RE: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

Quoted 51 lines

-----Original Message-----
From: Douglas Gilbert [mailto:[email protected]]
Sent: Thursday, September 26, 2013 9:17 PM
To: Martin K. Petersen; Bernd Schubert
Cc: Mike Snitzer; Hannes Reinecke; [email protected]; device-mapper development; [email protected]; Saxena, Sumit
Subject: Re: SCSI's heuristics for enabling WRITE SAME still need work [was: dm mpath: disable WRITE SAME if it fails]

Quoted 14 lines · Martin K. Petersen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Bernd" == Bernd Schubert <[email protected]> writes:

Bernd,

Bernd> Both types of systems we have in-house neither block limits vpd Bernd> nor READ_CAP16 return anything that would indicate discard is Bernd> supported. But UNMAP and WRITE SAME unmap(*) just work fine.

I have a collection of different SSDs in a tray connected to an LSI SAS2008 ASIC. The 510 is the only drive that does not have LBPME=1. Chances are it's because DRAT and RZAT are not set but it could also be that the 510 is blacklisted.

Quoted 4 lines

Bernd> I certainly don't want to cause any more write-same trouble, Bernd> but as all layers initially have to assume write same is Bernd> supported anyway and need to dynamically disable it if it Bernd> fails, can't we also enable discard by default with WRITE

SAME16 unmap?

Quoted 20 lines

No thanks :)

But you can force discards on by doing a:

  # echo -n unmap > /sys/class/scsi_disk/x:y:z:n/provisioning_mode

or

  # echo -n writesame_16 >
/sys/class/scsi_disk/x:y:z:n/provisioning_mode

Create a udev rule if you like.

In any case I wouldn't recommend using TRIM on that drive...

Bernd> PS: LSI SATL with FWv17 seems to have an unmap bug - I cannot Bernd> unmap the last sector:

Quoted 7 lines
Quoted 2 lines

Yes, it appears there's an off-by-one bug in the UNMAP translation. Sumit, is this something you guys can look into?

Hi Sumit,
While you are looking at the HBA firmware could you fix this minor annoyance:

Thibash, Can you please look at this?

Sumit

Quoted 35 lines
# lsscsi -H -t
.....
[6]    mpt3sas       sas:0x500605b006d3b510
# smp_rep_general /dev/mpt3ctl -s 0x500605b006d3b510 Report general
response:
  expander change count: 0
  expander route indexes: 0
  long response: 0
  number of phys: 0
  zone configuring: 0
  self configuring: 0
  externally configurable route table: 0
  initiates SSP close: 0
  enclosure logical identifier (hex): b005065010b5d306

So that is a SMP REPORT GENERAL (RG) directed at the HBA itself. So either my code is incorrectly decoding the "enclosure logical identifier" or ... . That field was defined in SAS 1.1 so no excuses on that front. 0x0 would be better than a shuffled version of the HBA's own SAS address.

Same bug/feature in all versions of SAS-2 and now SAS-3 firmware that I have tried.

Doug Gilbert

BTW The bsg driver can send a RG to the HBA like this:
smp_rep_general /dev/bsg/sas_host6
So the intent is clearer and the response is the same.