T10 WCE interpretation in Linux & device level access
For many years, we have used WCE as an indication that a device has a volatile write cache (not just a write cache) and used this as a trigger to send down SYNCHRONIZE_CACHE commands as needed.
Some arrays with non-volatile cache seem to have WCE set and simply ignore the command.
Some arrays with non-volatile cache seem to not set WCE.
Others arrays with non-volatile cache - our problem arrays - set WCE and do something horrible and slow when sent the SYNCHRONIZE_CACHE commands.
Note that for file systems, you can override this behavior by mounting with our barriers disabled (mount -o nobarrier .....). There is currently no way do disable this for anything using the device directly, not through the file system.
Some applications run against block devices - not through a file system - and want not to slow to a crawl when they have an array in my problem set.
Giving them a hook to ignore WCE seems to be a hack, but one that would resolve issues with users who won't want to wait months (years?) for us to convince the array vendors.
Is this a hook worth doing?
Have we hashed this out in the T10 committee?
Regards,
Ric
James Bottomley
maintainerQuoted 6 lines · Ric Wheeler
For many years, we have used WCE as an indication that a device has a volatile write cache (not just a write cache) and used this as a trigger to send down
SYNCHRONIZE_CACHEcommands as needed.Some arrays with non-volatile cache seem to have WCE set and simply ignore the command.
I bet they don't; they probably obey the spec. There's a SYNC_NV bit which if unset (which it is in our implementation) means only sync your non-NV cache. For a device with all NV, that equates to nop.
Quoted 4 lines
Some arrays with non-volatile cache seem to not set WCE.
Others arrays with non-volatile cache - our problem arrays - set WCE and do something horrible and slow when sent the
SYNCHRONIZE_CACHEcommands.
These arrays sound to be out of spec, so we should probably just blacklist them.
Quoted 4 lines
Note that for file systems, you can override this behavior by mounting with our barriers disabled (mount -o nobarrier .....). There is currently no way do disable this for anything using the device directly, not through the file system.
Quoted 8 lines
Some applications run against block devices - not through a file system - and want not to slow to a crawl when they have an array in my problem set.
Giving them a hook to ignore WCE seems to be a hack, but one that would resolve issues with users who won't want to wait months (years?) for us to convince the array vendors.
Is this a hook worth doing?
We already have the ability to set the cache type in sysfs. It tries to do a mode select back to the array, but the USB guys want it for the reverse problem (write back cache behind bridge declared as write through).
Quoted 1 line
Have we hashed this out in the T10 committee?
SBC-3 contains everything one could wish for about handling devices with volatile and NV cache, I thought.
James
Quoted 5 lines · Ric Wheeler
For many years, we have used WCE as an indication that a device has a volatile write cache (not just a write cache) and used this as a trigger to send down
SYNCHRONIZE_CACHEcommands as needed.
Quoted 7 lines
Some arrays with non-volatile cache seem to have WCE set and simply ignore the command.
Some arrays with non-volatile cache seem to not set WCE.
Others arrays with non-volatile cache - our problem arrays - set WCE and do something horrible and slow when sent the
SYNCHRONIZE_CACHEcommands.
Do the problematic arrays correctly set the NV_SUP** or V_SUP bits in the Extended Inquiry VPD pages? Does Linux take any notice of those bits? [Guess the answer.]
If the problematic arrays correctly set the NV_SUP and Linux sends SYNCHRONIZE_CACHE commands then Linux should be named and shamed.
Quoted 7 lines
Note that for file systems, you can override this behavior by mounting with our barriers disabled (mount -o nobarrier .....). There is currently no way do disable this for anything using the device directly, not through the file system.
Some applications run against block devices - not through a file system - and want not to slow to a crawl when they have an array in my problem set.
Quoted 7 lines
Giving them a hook to ignore WCE seems to be a hack, but one that would resolve issues with users who won't want to wait months (years?) for us to convince the array vendors.
Is this a hook worth doing?
Have we hashed this out in the T10 committee?
There is also a NV_DIS bit in the caching mode page that if supported would allow the NV cache to be disabled.
Looks like T10 supplies enough knobs, it is up to the vendor to set them appropriately and Linux to take notice.
Doug Gilbert
** What JB called SYNC_NV ...
Quoted 4 lines · James Bottomley
I bet they don't; they probably obey the spec. There's a
SYNC_NVbit which if unset (which it is in our implementation) means only sync your non-NV cache. For a device with all NV, that equates to nop.
Yes, linux leaves the SYNC_NV bit unset in scsi_setup_flush_cmnd().
The draft specs, and a couple others I have laying about says: says the device shall sync cache to medium for both volatile and non volatile cache data if SYNC_NV is _unset_.
With it set, the table could be more confusing!
For volatile cache blocks with SYNC_NV set "If a non-volatile cache is present, then the device server shall synchronize to non-volatile cache or to the medium. If a non-volatile cache is not present, then the device server shall synchronize to the medium".
And for Non-volatile cache with it set "No Requirement"
Which to me says, don't expect any particular behavior if you set this bit and have NV it could flush to medium, flush to NV cache, or do nothing at all. But it seems pretty clear that with it unset its probably going to get synchronized to the medium.
If T10 were to do something, maybe they could stop putting bits in the docs that aren't guaranteed to do anything (fill in rant).
As for linux, seems the state of the spec really doesn't leave any good options other than provide the user the ability to disable the flush_cmnd() if the NV_SUP bit is set. Or maybe a white list (ick!)...
If the writeback cache is enabled (per the WCE bit in the Caching mode page), prudent software uses the FUA bit in WRITE commands when writing metadata and/or sends the SYNCHRONIZE CACHE command at important checkpoints to ensure the data is not going to be lost due to a power loss. Some database software is particularly prolific at sending these commands.
Around 2003, many RAID controllers with non-volatile writeback caches honored the SYNCHRONIZE CACHE command, flushing the entire cache to the drives. This started causing timeouts as non-volatile write cache sizes grew. Recently, it's even causing trouble on individual disk drives with growing volatile write caches.
The intent of software using these commands and bits was unclear - it could be:
a) ensure data is in non-volatile cache (and will eventually be flushed) or on the medium; or
b) ensure data is on the medium (so the drives are ready for removal).
As a short-term fix, many RAID controllers assumed intent (a) and started interpreting the SYNCHRONIZE CACHE command as a NOP and ignoring the FUA bit.
Surprise removal of a drive from a RAID controller is risky even if software has run SYNCHRONIZE CACHE, since the RAID controller might be doing other activity in the background. So, there are other reasons to justify assuming that the user just won't do that.
Afraid of breaking software with intent (b) (which was more likely in the days of floppy disks, Bournelli Boxes, and other removable block devices), T10 chose to clarify that the original meaning was (b) and added new FUA_NV and SYNC_NV bits to let software express intent (a). The hope was that devices would implement the bits and software would start using them at appropriate times.
Unfortunately, the short-term fix worked well enough that it still prevails today, and most standalone removable media block devices have disappeared. There is not much software actually sending the FUA_NV and SYNC_NV bits and few devices honoring the bits per the standard.
As an SBC-3 letter ballot comment, I recently submitted T10 proposal 13-050 (see http://www.t10.org/doc13.htm) to obsolete the SYNC_NV and FUA_NV bits and change the meaning of the commands without those bits to intent (a), reflecting what the industry has actually done.
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jeremy Linton
Sent: Tuesday, April 23, 2013 5:40 PM
To: James Bottomley
Cc: Ric Wheeler; [email protected]; Martin K. Petersen; Jeff Moyer; Tejun Heo; Mike Snitzer; [email protected]
Subject: Re: T10 WCE interpretation in Linux & device level access
Quoted 4 lines · James Bottomley
I bet they don't; they probably obey the spec. There's a
SYNC_NVbit which if unset (which it is in our implementation) means only sync your non-NV cache. For a device with all NV, that equates to nop.
Yes, linux leaves the SYNC_NV bit unset in scsi_setup_flush_cmnd().
The draft specs, and a couple others I have laying about says: says the device shall sync cache to medium for both volatile and non volatile cache data if SYNC_NV is _unset_.
With it set, the table could be more confusing!
For volatile cache blocks with SYNC_NV set "If a non-volatile cache is present, then the device server shall synchronize to non-volatile cache or to the medium. If a non-volatile cache is not present, then the device server shall synchronize to the medium".
And for Non-volatile cache with it set "No Requirement"
Which to me says, don't expect any particular behavior if you set this bit and have NV it could flush to medium, flush to NV cache, or do nothing at all. But it seems pretty clear that with it unset its probably going to get synchronized to the medium.
If T10 were to do something, maybe they could stop putting bits in the docs that aren't guaranteed to do anything (fill in rant).
As for linux, seems the state of the spec really doesn't leave any good options other than provide the user the ability to disable the flush_cmnd() if the NV_SUP bit is set. Or maybe a white list (ick!)...
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Rob,
Comments inline below.
Quoted 12 lines · Robert (Server Storage)
If the writeback cache is enabled (per the WCE bit in the Caching mode page), prudent software uses the FUA bit in WRITE commands when writing metadata and/or sends the SYNCHRONIZE CACHE command at important checkpoints to ensure the data is not going to be lost due to a power loss. Some database software is particularly prolific at sending these commands.
Around 2003, many RAID controllers with non-volatile writeback caches honored the SYNCHRONIZE CACHE command, flushing the entire cache to the drives. This started causing timeouts as non-volatile write cache sizes grew. Recently, it's even causing trouble on individual disk drives with growing volatile write caches.
Quoted 4 lines
The intent of software using these commands and bits was unclear - it could be:
a) ensure data is in non-volatile cache (and will eventually be flushed) or on the medium; or
b) ensure data is on the medium (so the drives are ready for removal).
Linux issues SYNCHRONIZE_CACHE commands when we need to make sure that the data needs to be crash safe (after a transaction commit from a file system journal, an explicit fsync call or write system call with O_SYNC set).
If the cache is nonvolatile (i.e., the target will have it after a power outage or reboot), we are fine - pretty much your (a) clause above.
Not sure we have thought through (or can control) how an array would handle pulling a drive from behind a RAID controller that has not flushed its state.
Quoted 3 lines
As a short-term fix, many RAID controllers assumed intent (a) and started interpreting the SYNCHRONIZE CACHE command as a NOP and ignoring the FUA bit.
We have seen problems with some RAID controllers that leave the write cache enabled on back end drives - their cache is battery backed, but the cache on those backend drives is exposed to certain data loss on a power outage.
It would be nice if they always disabled the write cache on the backend drives *or* advertised WCE and propagated the SYNCHRONIZE_CACHE commands to each drive when we send them down.
Quoted 18 lines
Surprise removal of a drive from a RAID controller is risky even if software has run SYNCHRONIZE CACHE, since the RAID controller might be doing other activity in the background. So, there are other reasons to justify assuming that the user just won't do that.
Afraid of breaking software with intent (b) (which was more likely in the days of floppy disks, Bournelli Boxes, and other removable block devices), T10 chose to clarify that the original meaning was (b) and added new
FUA_NVandSYNC_NVbits to let software express intent (a). The hope was that devices would implement the bits and software would start using them at appropriate times.Unfortunately, the short-term fix worked well enough that it still prevails today, and most standalone removable media block devices have disappeared. There is not much software actually sending the
FUA_NVandSYNC_NVbits and few devices honoring the bits per the standard.
Quoted 4 lines
As an SBC-3 letter ballot comment, I recently submitted T10 proposal 13-050 (see http://www.t10.org/doc13.htm) to obsolete the
SYNC_NVandFUA_NVbits and change the meaning of the commands without those bits to intent (a), reflecting what the industry has actually done.
This is definitely something that we should review and take into account going forward.
It does sound like we have a lot of confusion around WCE meaning in the storage industry today though, which leads me to think that we will need to allow raw block accessing applications to manually override our flush settings (reluctantly!).
Regards,
Ric
Quoted 56 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jeremy Linton
Sent: Tuesday, April 23, 2013 5:40 PM
To: James Bottomley
Cc: Ric Wheeler; [email protected]; Martin K. Petersen; Jeff Moyer; Tejun Heo; Mike Snitzer; [email protected]
Subject: Re: T10 WCE interpretation in Linux & device level accessQuoted 3 lines · James Bottomley
I bet they don't; they probably obey the spec. There's a
SYNC_NVbit which if unset (which it is in our implementation) means only sync your non-NV cache. For a device with all NV, that equates to nop.Yes, linux leaves the
SYNC_NVbit unset inscsi_setup_flush_cmnd().The draft specs, and a couple others I have laying about says: says the device shall sync cache to medium for both volatile and non volatile cache data if
SYNC_NVis _unset_.With it set, the table could be more confusing!
For volatile cache blocks with
SYNC_NVset "If a non-volatile cache is present, then the device server shall synchronize to non-volatile cache or to the medium. If a non-volatile cache is not present, then the device server shall synchronize to the medium".And for Non-volatile cache with it set "No Requirement"
Which to me says, don't expect any particular behavior if you set this bit and have NV it could flush to medium, flush to NV cache, or do nothing at all. But it seems pretty clear that with it unset its probably going to get synchronized to the medium.
If T10 were to do something, maybe they could stop putting bits in the docs that aren't guaranteed to do anything (fill in rant).
As for linux, seems the state of the spec really doesn't leave any good options other than provide the user the ability to disable the
flush_cmnd()if theNV_SUPbit is set. Or maybe a white list (ick!)...--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Il 23/04/2013 22:07, James Bottomley ha scritto:
Quoted 11 lines · Ric Wheeler
Quoted 6 lines
For many years, we have used WCE as an indication that a device has a volatile write cache (not just a write cache) and used this as a trigger to send down
SYNCHRONIZE_CACHEcommands as needed.Some arrays with non-volatile cache seem to have WCE set and simply ignore the command.
I bet they don't; they probably obey the spec. There's a
SYNC_NVbit which if unset (which it is in our implementation) means only sync your non-NV cache. For a device with all NV, that equates to nop.
Isn't it the other way round?
SYNC_NV = 0 means "sync all your caches to the medium", and it's what we do.
SYNC_NV = 1 means "sync volatile to non-volatile", and it's what Ric wants.
So we should set SYNC_NV=1 if NV_SUP is set, perhaps only if the medium is non-removable just to err on the safe side.
Ric, can you check what your arrays set in VPD page 0x86 byte 6 bit 1?
Paolo
Quoted 39 lines
Quoted 4 lines
Some arrays with non-volatile cache seem to not set WCE.
Others arrays with non-volatile cache - our problem arrays - set WCE and do something horrible and slow when sent the
SYNCHRONIZE_CACHEcommands.These arrays sound to be out of spec, so we should probably just blacklist them.
Quoted 4 lines
Note that for file systems, you can override this behavior by mounting with our barriers disabled (mount -o nobarrier .....). There is currently no way do disable this for anything using the device directly, not through the file system.
Quoted 8 lines
Some applications run against block devices - not through a file system - and want not to slow to a crawl when they have an array in my problem set.
Giving them a hook to ignore WCE seems to be a hack, but one that would resolve issues with users who won't want to wait months (years?) for us to convince the array vendors.
Is this a hook worth doing?
We already have the ability to set the cache type in sysfs. It tries to do a mode select back to the array, but the USB guys want it for the reverse problem (write back cache behind bridge declared as write through).
Quoted 1 line
Have we hashed this out in the T10 committee?
SBC-3 contains everything one could wish for about handling devices with volatile and NV cache, I thought.
James
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 12 lines · James Bottomley
Quoted 6 lines · Ric Wheeler
For many years, we have used WCE as an indication that a device has a volatile write cache (not just a write cache) and used this as a trigger to send down
SYNCHRONIZE_CACHEcommands as needed.Some arrays with non-volatile cache seem to have WCE set and simply ignore the command.
I bet they don't; they probably obey the spec. There's a
SYNC_NVbit which if unset (which it is in our implementation) means only sync your non-NV cache. For a device with all NV, that equates to nop.
Quoted 8 lines
Quoted 4 lines
Some arrays with non-volatile cache seem to not set WCE.
Others arrays with non-volatile cache - our problem arrays - set WCE and do something horrible and slow when sent the
SYNCHRONIZE_CACHEcommands.These arrays sound to be out of spec, so we should probably just blacklist them.
Don't think so.
There is no time limit for the SYNCHRONIZE_CACHE command, so the array might take all day to write out the cache.
In fact, I've recently had a rather heated discussion with a certain storage vendor which reserved his right to take up to several seconds when flushing the cache.
Might be that we're in fact talking about the same one ... are we on a naming-and-shaming policy here ?
If so I could tell you some really _interesting_ details about their behaviour ...
Also note that SYNCHRONIZE_CACHE was always problematic; and as we're not even setting the LBA range we're even have cross-speak when issuing it to partitioned devices.
Quoted 11 lines
Quoted 11 lines
Note that for file systems, you can override this behavior by mounting with our barriers disabled (mount -o nobarrier .....). There is currently no way do disable this for anything using the device directly, not through the file system.
Some applications run against block devices - not through a file system - and want not to slow to a crawl when they have an array in my problem set.
Giving them a hook to ignore WCE seems to be a hack, but one that would resolve issues with users who won't want to wait months (years?) for us to convince the array vendors.
Quoted 7 lines
Quoted 1 line
Is this a hook worth doing?
We already have the ability to set the cache type in sysfs. It tries to do a mode select back to the array, but the USB guys want it for the reverse problem (write back cache behind bridge declared as write through).
You can always set the 'IMMED' bit for these arrays :-)
Cheers,
Hannes
Signature · 8 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 13 lines · Paolo Bonzini
Il 23/04/2013 22:07, James Bottomley ha scritto:
Quoted 11 lines · Ric Wheeler
Quoted 6 lines
For many years, we have used WCE as an indication that a device has a volatile write cache (not just a write cache) and used this as a trigger to send down
SYNCHRONIZE_CACHEcommands as needed.Some arrays with non-volatile cache seem to have WCE set and simply ignore the command.
I bet they don't; they probably obey the spec. There's a
SYNC_NVbit which if unset (which it is in our implementation) means only sync your non-NV cache. For a device with all NV, that equates to nop.
Quoted 9 lines
Isn't it the other way round?
SYNC_NV= 0 means "sync all your caches to the medium", and it's what we do.
SYNC_NV= 1 means "sync volatile to non-volatile", and it's what Ric wants.So we should set SYNC_NV=1 if
NV_SUPis set, perhaps only if the medium is non-removable just to err on the safe side.
Or use 'WRITE_AND_VERIFY' here; that's guaranteed to hit the disk. Plus it even has a guarantee about data consistency on the disk, which the normal WRITE command has not.
Cheers,
Hannes
Signature · 8 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Il 24/04/2013 14:07, Hannes Reinecke ha scritto:
Quoted 18 lines · Paolo Bonzini
Quoted 17 lines
Il 23/04/2013 22:07, James Bottomley ha scritto:
Quoted 11 lines · Ric Wheeler
Quoted 6 lines
For many years, we have used WCE as an indication that a device has a volatile write cache (not just a write cache) and used this as a trigger to send down
SYNCHRONIZE_CACHEcommands as needed.Some arrays with non-volatile cache seem to have WCE set and simply ignore the command.
I bet they don't; they probably obey the spec. There's a
SYNC_NVbit which if unset (which it is in our implementation) means only sync your non-NV cache. For a device with all NV, that equates to nop.Isn't it the other way round?
SYNC_NV= 0 means "sync all your caches to the medium", and it's what we do.
Quoted 8 lines
Quoted 4 lines
SYNC_NV= 1 means "sync volatile to non-volatile", and it's what Ric wants.So we should set SYNC_NV=1 if
NV_SUPis set, perhaps only if the medium is non-removable just to err on the safe side.Or use '
WRITE_AND_VERIFY' here; that's guaranteed to hit the disk. Plus it even has a guarantee about data consistency on the disk, which the normal WRITE command has not.
The point is to _avoid_ hitting the disk. :)
Paolo
Quoted 21 lines · Paolo Bonzini
Il 24/04/2013 14:07, Hannes Reinecke ha scritto:
Quoted 20 lines · Paolo Bonzini
Quoted 19 lines
Il 23/04/2013 22:07, James Bottomley ha scritto:
Quoted 11 lines · Ric Wheeler
Quoted 6 lines
For many years, we have used WCE as an indication that a device has a volatile write cache (not just a write cache) and used this as a trigger to send down
SYNCHRONIZE_CACHEcommands as needed.Some arrays with non-volatile cache seem to have WCE set and simply ignore the command.
I bet they don't; they probably obey the spec. There's a
SYNC_NVbit which if unset (which it is in our implementation) means only sync your non-NV cache. For a device with all NV, that equates to nop.Isn't it the other way round?
SYNC_NV= 0 means "sync all your caches to the medium", and it's what we do.
SYNC_NV= 1 means "sync volatile to non-volatile", and it's what Ric wants.
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
So we should set SYNC_NV=1 if
NV_SUPis set, perhaps only if the medium is non-removable just to err on the safe side.Or use '
WRITE_AND_VERIFY' here; that's guaranteed to hit the disk. Plus it even has a guarantee about data consistency on the disk, which the normal WRITE command has not.The point is to _avoid_ hitting the disk. :)
Ah. Really?
Why do we discuss SYNCHRONIZE CACHE then?
I was under the impression that we're talking various 'barriers' (or rather 'flush' nowadays) implementations.
Which require that some data needs to be written to disk before continuing.
Or did I somehow misread the thread?
Confused,
Hannes
Signature · 8 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Il 24/04/2013 14:12, Hannes Reinecke ha scritto:
Quoted 29 lines · Paolo Bonzini
Quoted 28 lines
Il 24/04/2013 14:07, Hannes Reinecke ha scritto:
Quoted 26 lines · Paolo Bonzini
Quoted 13 lines
Il 23/04/2013 22:07, James Bottomley ha scritto:
Quoted 11 lines · Ric Wheeler
Quoted 6 lines
For many years, we have used WCE as an indication that a device has a volatile write cache (not just a write cache) and used this as a trigger to send down
SYNCHRONIZE_CACHEcommands as needed.Some arrays with non-volatile cache seem to have WCE set and simply ignore the command.
I bet they don't; they probably obey the spec. There's a
SYNC_NVbit which if unset (which it is in our implementation) means only sync your non-NV cache. For a device with all NV, that equates to nop.Quoted 8 lines
Isn't it the other way round?
SYNC_NV= 0 means "sync all your caches to the medium", and it's what we do.
SYNC_NV= 1 means "sync volatile to non-volatile", and it's what Ric wants.So we should set SYNC_NV=1 if
NV_SUPis set, perhaps only if the medium is non-removable just to err on the safe side.Or use '
WRITE_AND_VERIFY' here; that's guaranteed to hit the disk. Plus it even has a guarantee about data consistency on the disk, which the normal WRITE command has not.
Quoted 5 lines
Quoted 2 lines
The point is to _avoid_ hitting the disk. :)
Ah. Really?
Why do we discuss SYNCHRONIZE CACHE then?
Because we do want the data to hit the non-volatile cache, just in case the disk has both a volatile and a non-volatile cache.
Paolo
Quoted 33 lines · Hannes Reinecke
Quoted 21 lines · Paolo Bonzini
Il 24/04/2013 14:07, Hannes Reinecke ha scritto:
Quoted 20 lines · Paolo Bonzini
Quoted 19 lines
Il 23/04/2013 22:07, James Bottomley ha scritto:
Quoted 11 lines · Ric Wheeler
Quoted 6 lines
For many years, we have used WCE as an indication that a device has a volatile write cache (not just a write cache) and used this as a trigger to send down
SYNCHRONIZE_CACHEcommands as needed.Some arrays with non-volatile cache seem to have WCE set and simply ignore the command.
I bet they don't; they probably obey the spec. There's a
SYNC_NVbit which if unset (which it is in our implementation) means only sync your non-NV cache. For a device with all NV, that equates to nop.Isn't it the other way round?
SYNC_NV= 0 means "sync all your caches to the medium", and it's what we do.
SYNC_NV= 1 means "sync volatile to non-volatile", and it's what Ric wants.Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
So we should set SYNC_NV=1 if
NV_SUPis set, perhaps only if the medium is non-removable just to err on the safe side.Or use '
WRITE_AND_VERIFY' here; that's guaranteed to hit the disk. Plus it even has a guarantee about data consistency on the disk, which the normal WRITE command has not.The point is to _avoid_ hitting the disk. :)
Ah. Really?
Quoted 7 lines
Why do we discuss SYNCHRONIZE CACHE then?
I was under the impression that we're talking various 'barriers' (or rather 'flush' nowadays) implementations.
Which require that some data needs to be written to disk before continuing.Or did I somehow misread the thread?
This thread was motivated by the fact that the storage is reporting WCE=1 and OracleDB (with ASM) is issuing SYNCHRONIZE CACHE (via REQ_FLUSH) which the array in question handles _very_ slowly (even though it is battery backed).
So the question Ric had is: should we expose a new knob that allows admins to impose WCE=0 behavior (avoiding the SYNCHRONIZE CACHE).
I'm concerned such a knob will be abused for the benefit of speed and all data integrity caution will get thrown to the wind (much like the nobarrier FS mount option).
Mike
Quoted 19 lines · Paolo Bonzini
Il 24/04/2013 14:07, Hannes Reinecke ha scritto:
Quoted 18 lines · Paolo Bonzini
Quoted 17 lines
Il 23/04/2013 22:07, James Bottomley ha scritto:
Quoted 10 lines · Ric Wheeler
Quoted 6 lines
For many years, we have used WCE as an indication that a device has a volatile write cache (not just a write cache) and used this as a trigger to send down
SYNCHRONIZE_CACHEcommands as needed.Some arrays with non-volatile cache seem to have WCE set and simply ignore the command.
I bet they don't; they probably obey the spec. There's a
SYNC_NVbit which if unset (which it is in our implementation) means only sync your non-NV cache. For a device with all NV, that equates to nop.Isn't it the other way round?
SYNC_NV= 0 means "sync all your caches to the medium", and it's what we do.
SYNC_NV= 1 means "sync volatile to non-volatile", and it's what Ric wants.
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
So we should set SYNC_NV=1 if
NV_SUPis set, perhaps only if the medium is non-removable just to err on the safe side.Or use '
WRITE_AND_VERIFY' here; that's guaranteed to hit the disk. Plus it even has a guarantee about data consistency on the disk, which the normal WRITE command has not.The point is to _avoid_ hitting the disk. :)
Paolo
The point is to have a crash-proof version of the data acknowledged by the target device while letting data sit in volatile state as long as possible. To be even clearer, we would love to do this for a sub-range of the device but currently use a "big hammer" to flush the entire cache (possibly for multiple file systems on one target storage device).
Once we use the SYNCHRONIZE_CACHE (or CACHE_FLUSH_EXT) command, we want the data on that target device to be there if someone loses power.
If the device can promise this, we don't care (and don't know) how it manages that promise. It can leave the data on battery backed DRAM, can archive it to flash or any other scheme that works.
Just as importantly, we don't want to "destage" data to the back end drives if that is not required since it is really, really slow.
The confusion here is that various storage devices have used the standard bits in arbitrary ways which makes it very hard to have one clear set of rules.
Even harder to explain to end users when to use a work around (like mount -o nobarrier) or the proposed "ignore flushes" block level call :)
Regards,
Ric
Il 24/04/2013 14:27, Ric Wheeler ha scritto:
Quoted 8 lines
Quoted 1 line
The point is to _avoid_ hitting the disk. :)
The point is to have a crash-proof version of the data acknowledged by the target device while letting data sit in volatile state as long as possible. To be even clearer, we would love to do this for a sub-range of the device but currently use a "big hammer" to flush the entire cache (possibly for multiple file systems on one target storage device).
Quoted 6 lines
Once we use the
SYNCHRONIZE_CACHE(orCACHE_FLUSH_EXT) command, we want the data on that target device to be there if someone loses power.If the device can promise this, we don't care (and don't know) how it manages that promise. It can leave the data on battery backed DRAM, can archive it to flash or any other scheme that works.
That's exactly the point of SYNC_NV=1.
Quoted 6 lines
Just as importantly, we don't want to "destage" data to the back end drives if that is not required since it is really, really slow.
The confusion here is that various storage devices have used the standard bits in arbitrary ways which makes it very hard to have one clear set of rules.
Also that we have ignored the problem for long, and it's worked surprisingly well. :)
Quoted 2 lines
Even harder to explain to end users when to use a work around (like mount -o nobarrier) or the proposed "ignore flushes" block level call :)
Hoping Thunderbird doesn't mangle the patch too badly, code might be worth a thousand words... see after the sig, compile-tested only. I have no access to these controllers, neither the good ones nor the bad ones. :)
Paolo
--------------------- 8< -------------------------
From: Paolo Bonzini <[email protected]>
Subject: [PATCH] scsi: only make REQ_FLUSH flush to non-volatile cache
The point of REQ_FLUSH is to have a crash-proof version of the data acknowledged by the target device. We want the data on that target device to be there if someone loses power, but we don't care (and don't want to know) how it manages that promise. It can leave the data on battery backed DRAM, can archive it to flash or any other scheme that works.
This is exactly what SYNC_NV=1 does. Instead, SYNC_NV=0 should flush the data to the medium, which is not desirable when we have a non-volatile cache (except perhaps if the medium is removable).
---
drivers/scsi/sd.c +29 -0
@@ -800,9 +800,17 @@ static int sd_setup_write_same_cmnd(struct scsi_device *sdp, struct request *rq)static int scsi_setup_flush_cmnd (struct scsi_device *sdp,struct request *rq) { +struct scsi_disk *sdkp =scsi_disk (rq->rq_disk); + rq->timeout =SD_FLUSH_TIMEOUT ; rq->retries =SD_MAX_RETRIES ; rq->cmd[0 ] =SYNCHRONIZE_CACHE ; + +/* No need to synchronize to medium if we have a non-volatile cache, +* but be safe if the medium could just go away. +*/ +if (sdkp->nv_sup && !sdp->removable) + rq->cmd[1 ] |=4 ;/* SYNC_NV */ rq->cmd_len =10 ;return scsi_setup_blk_pc_cmnd (sdp, rq);
@@ -2511,6 +2519,26 @@ static void sd_read_app_tag_own(struct scsi_disk *sdkp, unsigned char *buffer) }/** +* sd_read_extended_inquiry - Query disk device for non-volatile cache. +* @disk: disk to query +*/ +static void sd_read_extended_inquiry (struct scsi_disk *sdkp) +{ +const int vpd_len =64 ; +unsigned char *buffer =kmalloc (vpd_len,GFP_KERNEL ); + +if (!buffer || +/* Block Limits VPD */ +scsi_get_vpd_page (sdkp->device,0x86 , buffer, vpd_len)) + goto out; + + sdkp->nv_sup = (buffer[6 ] &0x02 ) !=0 ; + + out: +kfree (buffer); +} + +/** * sd_read_block_limits - Query disk device for preferred I/O sizes. * @disk: disk to query */
@@ -2684,6 +2712,7 @@ static int sd_revalidate_disk(struct gendisk *disk)sd_read_capacity (sdkp, buffer);if (sd_try_extended_inquiry (sdp)) { +sd_read_extended_inquiry (sdkp);sd_read_block_provisioning (sdkp);sd_read_block_limits (sdkp);sd_read_block_characteristics (sdkp);
drivers/scsi/sd.h +1 -0
@@ -84,6 +84,7 @@ struct scsi_disk { unsigned lbpws10 : 1; unsigned lbpvpd : 1; unsigned ws16 : 1; + unsigned nv_sup : 1; };#define to_scsi_disk (obj) container_of(obj,struct scsi_disk,dev)
Quoted 4 lines
Regards,
Ric
Quoted 5 lines · Paolo Bonzini
Quoted 3 lines
If the device can promise this, we don't care (and don't know) how it manages that promise. It can leave the data on battery backed DRAM, can archive it to flash or any other scheme that works.
That's exactly the point of SYNC_NV=1.
Well its the point, but the specification is written such that the vendors can
choose to implement it any way they wish, especially for split cache systems where there is both volatile and non volatile cache.
Flushing the NV cache to medium (as is the current behavior) may not be a bad
idea anyway.
Thats because I know of a large vendors array where the non-volatile cache
might be better described as the "sometimes" non-volatile cache. That is because a failure to flush the volatile portions results in the non-volatile portions being considered invalid when power is restored. This fences the volume, and the usual method for recovering the array is to call support and have them invalidate the NV portions of the cache. Thereby negating the whole reason for having a NV cache. I'm sure they don't tell customers this fact when they sell the array, when it happened in our lab I was in a state of shock for about a week.
Quoted 20 lines · Ric Wheeler
For many years, we have used WCE as an indication that a device has a volatile write cache (not just a write cache) and used this as a trigger to send down
SYNCHRONIZE_CACHEcommands as needed.Some arrays with non-volatile cache seem to have WCE set and simply ignore the command.
Some arrays with non-volatile cache seem to not set WCE.
Others arrays with non-volatile cache - our problem arrays - set WCE and do something horrible and slow when sent the
SYNCHRONIZE_CACHEcommands.Note that for file systems, you can override this behavior by mounting with our barriers disabled (mount -o nobarrier .....). There is currently no way do disable this for anything using the device directly, not through the file system.
Some applications run against block devices - not through a file system - and want not to slow to a crawl when they have an array in my problem set.
Quoted 7 lines
Giving them a hook to ignore WCE seems to be a hack, but one that would resolve issues with users who won't want to wait months (years?) for us to convince the array vendors.
Is this a hook worth doing?
Have we hashed this out in the T10 committee?
Naturally I'm biased, but I tend to think the user space is usually smarter than the kernel. That assumes skilled users.
So if the user space issues a SYNCHRONIZE_CACHE with the IMMED bit set and for the whole disk then the user should have a way of forcing that command to be issued. The assumption here is that the skilled user is about to power down that array or pull some disks or SSDs *.
The more questionable cases are when a file system or the block layer is issuing a barrier or some such that translates to a SYNCHRONIZE_CACHE. That should be ignored in some cases already discussed in this thread.
While working with SoCs I have noticed an interesting technique. Sub-system sized sections of the memory mapped IO space (e.g. a bank of GPIOs) can be write protected by a simple ASCII sequence **. Attempts to change configuration registers after write protect are ignored and an error is noted (if anyone cares). The same ACSII sequence can be used to un-write protect those sub-system configuration registers. Typically on a SoC if the GPIOs are randomly re-configured, it's game over.
Back to the SCSI world: a better solution might be if an LLD could be informed of the reason a SCSI control command is being issued (a sort of "come from" field). Failing, or it addition to that, a sysfs interface could be added to filter out "dangerous" SCSI commands:
echo "SC" > /sys/class/scsi_device/8:0:0:0/device/filter
cat /sys/class/scsi_device/8:0:0:0/device/filter
FU SC
If, for whatever reason, we did ignore a SYNCHRONIZE_CACHE command we could use vendor specific sense data (vendor=Linux) to indicate that a command had been ignored. That could be extended to all SCSI commands that are filtered out ***; better that than EIO, EACCES etc.
Doug Gilbert
* and if Linux doesn't permit this, then user might be
advised to run another, more obedient, host OS with Linux running as a VM. A "pass-by" rather than a "pass-through" ...
** only the configuration registers are write protected, so
data can still be written to the GPIOs
*** like me, many pass-through users cannot see why SCSI
commands injected to the SCSI subsystem (e.g. via sg or bsg) are filtered out silently by the block layer.
Jeremy,
It looks like, you, Paolo and Ric have hit the nail on the head here - this is a nice summary, IMHO:
Quoted 10 lines · Paolo Bonzini
Quoted 5 lines
Quoted 2 lines
If the device can promise this, we don't care (and don't know) how it manages that promise. It can leave the data on battery backed DRAM, can
archive it to flash or any other scheme that works.
That's exactly the point of SYNC_NV=1.
Well its the point, but the specification is written such that the vendors can
choose to implement it any way they wish, especially for split cache systems where there is both volatile and non volatile cache.
Independent of T10's best intentions at the time, the implementations aren't doing what's needed or intended, and I'd guess that the SYNC_NV bit is not being set to 1 by [other people's ;-) ] software that should be setting it to 1 if it were paying attention to the standard.
This is further complicated by it being completely legitimate wrt the SCSI standard to put non-volatile cache in a system and not have the SCSI interface admit that the non-volatile cache exists (WCE=0, SYNCHRONIZE CACHE is a no-op independent of the value of SYNC_NV).
I believe that Rob Elliot's 13-050 proposal to obsolete SYNC_NV and re-specify SYNCHRONIZE CACHE to make all data non-volatile by whatever means the target chooses is what T10 should do, and that matches Ric's summary:
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
If the device can promise this, we don't care (and don't know) how it manages that promise. It can leave the data on battery backed DRAM, can
archive it to flash or any other scheme that works.
Beyond that, attempting to manage drive removal from storage systems via the SCSI interface with standard commands is a waste of time and effort, IMHO. In a serious storage array (and even some fairly simple RAID controllers), some vendor-specific "magic" is needed to get the array (or controller) to prepare so that the drive can be removed cleanly. To oversimplify, it's not enough to flush data to the drive; the array or controller is stateful, and hence has to be told to "forget" the drive, where "forget" involves things that are rather implementation-specific.
Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 [email protected] Mobile: +1 (978) 394-7754
----------------------------------------------------
Quoted 25 lines
-----Original Message-----
From: Jeremy Linton [mailto:[email protected]]
Sent: Wednesday, April 24, 2013 10:36 AM
To: Paolo Bonzini
Cc: Ric Wheeler; Hannes Reinecke; James Bottomley; [email protected]; Martin K. Petersen; Jeff Moyer; Tejun Heo; Mike Snitzer; Black, David; Elliott, Robert (Server Storage); Knight, Frederick
Subject: Re: T10 WCE interpretation in Linux & device level accessQuoted 5 lines · Paolo Bonzini
Quoted 3 lines
If the device can promise this, we don't care (and don't know) how it manages that promise. It can leave the data on battery backed DRAM, can archive it to flash or any other scheme that works.
That's exactly the point of SYNC_NV=1.
Well its the point, but the specification is written such that the
vendors can
choose to implement it any way they wish, especially for split cache systems where there is both volatile and non volatile cache.Flushing the NV cache to medium (as is the current behavior) may not be
a bad
idea anyway.
Quoted 21 lines
Thats because I know of a large vendors array where the non-volatile
cache
might be better described as the "sometimes" non-volatile cache. That is because
a failure to flush the volatile portions results in the non-volatile portions being considered invalid when power is restored. This fences the volume, and the
usual method for recovering the array is to call support and have them invalidate the NV portions of the cache. Thereby negating the whole reason for having a NV cache. I'm sure they don't tell customers this fact when they sell the array, when it happened in our lab I was in a state of shock for about a week.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 40 lines · David
Jeremy,
It looks like, you, Paolo and Ric have hit the nail on the head here - this is a nice summary, IMHO:
Quoted 9 lines · Paolo Bonzini
Quoted 5 lines
Quoted 2 lines
If the device can promise this, we don't care (and don't know) how it manages that promise. It can leave the data on battery backed DRAM, can
archive it to flash or any other scheme that works.
That's exactly the point of SYNC_NV=1.
Well its the point, but the specification is written such that the vendors can
choose to implement it any way they wish, especially for split cache systems where there is both volatile and non volatile cache.Independent of T10's best intentions at the time, the implementations aren't doing what's needed or intended, and I'd guess that the
SYNC_NVbit is not being set to 1 by [other people's ;-) ] software that should be setting it to 1 if it were paying attention to the standard.This is further complicated by it being completely legitimate wrt the SCSI standard to put non-volatile cache in a system and not have the SCSI interface admit that the non-volatile cache exists (WCE=0, SYNCHRONIZE CACHE is a no-op independent of the value of
SYNC_NV).I believe that Rob Elliot's 13-050 proposal to obsolete
SYNC_NVand re-specify SYNCHRONIZE CACHE to make all data non-volatile by whatever means the target chooses is what T10 should do, and that matches Ric's summary:Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
If the device can promise this, we don't care (and don't know) how it manages that promise. It can leave the data on battery backed DRAM, can
archive it to flash or any other scheme that works.
Beyond that, attempting to manage drive removal from storage systems via the SCSI interface with standard commands is a waste of time and effort, IMHO. In a serious storage array (and even some fairly simple RAID controllers), some vendor-specific "magic" is needed to get the array (or controller) to prepare so that the drive can be removed cleanly. To oversimplify, it's not enough to flush data to the drive; the array or controller is stateful, and hence has to be told to "forget" the drive, where "forget" involves things that are rather implementation-specific.
Quoted 3 lines
Thanks,
--David
So I think that leaves us with some arrays that might benefit from Paolo's proposed patch, but almost certainly still will need to be able to "ignore flushes" for some block device accessing DB's, etc....
Ric
James Bottomley
maintainerQuoted 41 lines · Ric Wheeler
Quoted 40 lines · David
Jeremy,
It looks like, you, Paolo and Ric have hit the nail on the head here - this is a nice summary, IMHO:
Quoted 9 lines · Paolo Bonzini
Quoted 5 lines
Quoted 2 lines
If the device can promise this, we don't care (and don't know) how it manages that promise. It can leave the data on battery backed DRAM, can
archive it to flash or any other scheme that works.
That's exactly the point of SYNC_NV=1.
Well its the point, but the specification is written such that the vendors can
choose to implement it any way they wish, especially for split cache systems where there is both volatile and non volatile cache.Independent of T10's best intentions at the time, the implementations aren't doing what's needed or intended, and I'd guess that the
SYNC_NVbit is not being set to 1 by [other people's ;-) ] software that should be setting it to 1 if it were paying attention to the standard.This is further complicated by it being completely legitimate wrt the SCSI standard to put non-volatile cache in a system and not have the SCSI interface admit that the non-volatile cache exists (WCE=0, SYNCHRONIZE CACHE is a no-op independent of the value of
SYNC_NV).I believe that Rob Elliot's 13-050 proposal to obsolete
SYNC_NVand re-specify SYNCHRONIZE CACHE to make all data non-volatile by whatever means the target chooses is what T10 should do, and that matches Ric's summary:Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
If the device can promise this, we don't care (and don't know) how it manages that promise. It can leave the data on battery backed DRAM, can
archive it to flash or any other scheme that works.
Beyond that, attempting to manage drive removal from storage systems via the SCSI interface with standard commands is a waste of time and effort, IMHO. In a serious storage array (and even some fairly simple RAID controllers), some vendor-specific "magic" is needed to get the array (or controller) to prepare so that the drive can be removed cleanly. To oversimplify, it's not enough to flush data to the drive; the array or controller is stateful, and hence has to be told to "forget" the drive, where "forget" involves things that are rather implementation-specific.
Quoted 7 lines
Quoted 3 lines
Thanks,
--DavidSo I think that leaves us with some arrays that might benefit from Paolo's proposed patch, but almost certainly still will need to be able to "ignore flushes" for some block device accessing DB's, etc....
That just leaves us with random standards behaviour. Lets permit the deterministic thing instead for the distros. It kills two birds with one stone because we can set WCE for the stupid UAS devices that clear it wrongly as well.
For those who don't read code well, you add a temporary prefix to the cache set in
echo xxx > /sys/class/scsi_disk/<disk>/cache_type
and it will set the flags for the lifetime of the current kernel, but won't try to do a mode select to make them permanent.
James
---
drivers/scsi/sd.c +20 -0
@@ -142,6 +142,7 @@ sd_store_cache_type(struct device *dev, struct device_attribute *attr,char *buffer_data;struct scsi_mode_data data;struct scsi_sense_hdr sshdr; +const char *temp ="temporary " ;int len;if (sdp->type !=TYPE_DISK )
@@ -150,6 +151,13 @@ sd_store_cache_type(struct device *dev, struct device_attribute *attr,* it's not worth the risk */ return -EINVAL ; +if (strncmp (buf, temp,sizeof (temp) -1 ) ==0 ) { + buf +=sizeof (temp) -1 ; + sdkp->cache_override =1 ; + }else { + sdkp->cache_override =0 ; + } +for (i =0 ; i <ARRAY_SIZE (sd_cache_types); i++) { len =strlen (sd_cache_types[i]); if (strncmp(sd_cache_types[i], buf, len) == 0 &&
@@ -162,6 +170,13 @@ sd_store_cache_type(struct device *dev, struct device_attribute *attr,return -EINVAL ; rcd = ct &0x01 ?1 :0 ; wce = ct &0x02 ?1 :0 ; + +if (sdkp->cache_override) { + sdkp->WCE = wce; + sdkp->RCD = rcd; +return count; + } +if (scsi_mode_sense (sdp,0x08 ,8 , buffer,sizeof (buffer),SD_TIMEOUT ,SD_MAX_RETRIES , &data,NULL ))return -EINVAL ;
@@ -2319,6 +2334,10 @@ sd_read_cache_type(struct scsi_disk *sdkp, unsigned char *buffer)int old_rcd = sdkp->RCD;int old_dpofua = sdkp->DPOFUA; + +if (sdkp->cache_override) +return ; + first_len =4 ;if (sdp->skip_ms_page_8) {if (sdp->type ==TYPE_RBC )
@@ -2812,6 +2831,7 @@ static void sd_probe_async(void *data, async_cookie_t cookie) sdkp->capacity =0 ; sdkp->media_present =1 ; sdkp->write_prot =0 ; + sdkp->cache_override =0 ; sdkp->WCE =0 ; sdkp->RCD =0 ; sdkp->ATO =0 ;
drivers/scsi/sd.h +1 -0
@@ -73,6 +73,7 @@ struct scsi_disk {u8 protection_type;/* Data Integrity Field */ u8 provisioning_mode; unsigned ATO : 1; /* state of disk ATO bit */ + unsigned cache_override : 1; /* temp override of WCE,RCD */ unsigned WCE : 1; /* state of disk WCE bit */ unsigned RCD : 1; /* state of disk RCD bit, unused */ unsigned DPOFUA : 1; /* state of disk DPOFUA bit */
Il 24/04/2013 23:02, James Bottomley ha scritto:
Quoted 5 lines
That just leaves us with random standards behaviour. Lets permit the deterministic thing instead for the distros. It kills two birds with one stone because we can set WCE for the stupid UAS devices that clear it wrongly as well.
Quoted 7 lines
For those who don't read code well, you add a temporary prefix to the cache set in
echo xxx > /sys/class/scsi_disk/<disk>/cache_type
and it will set the flags for the lifetime of the current kernel, but won't try to do a mode select to make them permanent.
Having the knob is useful indeed. I don't like the "temporary" name though, because "temporary write-through" doesn't sound like it can eat data on a power loss. What about "force" or "assume"?
Also, this would be in addition to my patch (when tested), right?
Paolo
James Bottomley
maintainerQuoted 9 lines · Paolo Bonzini
Il 24/04/2013 23:02, James Bottomley ha scritto:
Quoted 8 lines
That just leaves us with random standards behaviour. Lets permit the deterministic thing instead for the distros. It kills two birds with one stone because we can set WCE for the stupid UAS devices that clear it wrongly as well.
For those who don't read code well, you add a temporary prefix to the cache set in
Quoted 8 lines
Quoted 4 lines
echo xxx > /sys/class/scsi_disk/<disk>/cache_type
and it will set the flags for the lifetime of the current kernel, but won't try to do a mode select to make them permanent.
Having the knob is useful indeed. I don't like the "temporary" name though, because "temporary write-through" doesn't sound like it can eat data on a power loss. What about "force" or "assume"?
I'm fairly ambivalent, except not force. The default behaviour is to do the mode select, so force seems to imply that as well, except it won't. I don't see a difference between assume and temporary.
Quoted 1 line
Also, this would be in addition to my patch (when tested), right?
Not really ... given T10s deprecation I don't think we want to touch anything to do with SYNC_NV because it just adds to the uncertainty about what will actually happen. Giving the ability to control WCE (and RCD) fixes all the problems raised so far.
James
Quoted 21 lines · James Bottomley
Quoted 9 lines · Paolo Bonzini
Il 24/04/2013 23:02, James Bottomley ha scritto:
Quoted 8 lines
That just leaves us with random standards behaviour. Lets permit the deterministic thing instead for the distros. It kills two birds with one stone because we can set WCE for the stupid UAS devices that clear it wrongly as well.
For those who don't read code well, you add a temporary prefix to the cache set in
Quoted 7 lines
Quoted 4 lines
echo xxx > /sys/class/scsi_disk/<disk>/cache_type
and it will set the flags for the lifetime of the current kernel, but won't try to do a mode select to make them permanent.
Having the knob is useful indeed. I don't like the "temporary" name though, because "temporary write-through" doesn't sound like it can eat data on a power loss. What about "force" or "assume"?
I'm fairly ambivalent, except not force. The default behaviour is to do the mode select, so force seems to imply that as well, except it won't. I don't see a difference between assume and temporary.
Quoted 9 lines
Quoted 1 line
Also, this would be in addition to my patch (when tested), right?
Not really ... given T10s deprecation I don't think we want to touch anything to do with
SYNC_NVbecause it just adds to the uncertainty about what will actually happen. Giving the ability to control WCE (and RCD) fixes all the problems raised so far.James
Why are we turning off the RCD bit in this? Not sure it matters, but we only should care about WCE (and the dirty write cache data)?
Thanks!
Ric
James Bottomley
maintainerQuoted 23 lines · Ric Wheeler
Quoted 22 lines · James Bottomley
Quoted 9 lines · Paolo Bonzini
Il 24/04/2013 23:02, James Bottomley ha scritto:
Quoted 8 lines
That just leaves us with random standards behaviour. Lets permit the deterministic thing instead for the distros. It kills two birds with one stone because we can set WCE for the stupid UAS devices that clear it wrongly as well.
For those who don't read code well, you add a temporary prefix to the cache set in
Quoted 7 lines
Quoted 4 lines
echo xxx > /sys/class/scsi_disk/<disk>/cache_type
and it will set the flags for the lifetime of the current kernel, but won't try to do a mode select to make them permanent.
Having the knob is useful indeed. I don't like the "temporary" name though, because "temporary write-through" doesn't sound like it can eat data on a power loss. What about "force" or "assume"?
I'm fairly ambivalent, except not force. The default behaviour is to do the mode select, so force seems to imply that as well, except it won't. I don't see a difference between assume and temporary.
Quoted 1 line
Also, this would be in addition to my patch (when tested), right?
Quoted 8 lines
Quoted 5 lines
Not really ... given T10s deprecation I don't think we want to touch anything to do with
SYNC_NVbecause it just adds to the uncertainty about what will actually happen. Giving the ability to control WCE (and RCD) fixes all the problems raised so far.Why are we turning off the RCD bit in this? Not sure it matters, but we only should care about WCE (and the dirty write cache data)?
Well, it's in the code. Cache policy is a combination of those two bits. The cache type takes a cache policy string, ergo it must update both. We don't do anything with it because having a write back cache and no cache at all is transparent to us.
James
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"James" == James Bottomley <[email protected]> writes:
James> I'm fairly ambivalent, except not force. The default behaviour James> is to do the mode select, so force seems to imply that as well, James> except it won't. I don't see a difference between assume and James> temporary.
I'm ok with your patch. And a strong believer in not altering the SYNCHRONIZE CACHE behavior that's been rigorously tested in the field by adding SYNC_NV to the mix.
James> Not really ... given T10s deprecation I don't think we want to James> touch anything to do with SYNC_NV because it just adds to the James> uncertainty about what will actually happen.
Yep.
James> Giving the ability to control WCE (and RCD) fixes all the James> problems raised so far.
If there are devices that would truly benefit from SYNC_NV we could add a sync_nv parameter to scsi_disk's sysfs that could be used to set that bit when issuing flush_cmnd.
But it would be something we would do manually on a per-device basis and not something that is automatically keyed off of NV_SUP (SYNC_NV doesn't require NV_SUP, btw., so that's not even a valid check).
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Quoted 29 lines · James Bottomley
Quoted 28 lines · Ric Wheeler
Quoted 27 lines · James Bottomley
Quoted 9 lines · Paolo Bonzini
Il 24/04/2013 23:02, James Bottomley ha scritto:
Quoted 8 lines
That just leaves us with random standards behaviour. Lets permit the deterministic thing instead for the distros. It kills two birds with one stone because we can set WCE for the stupid UAS devices that clear it wrongly as well.
For those who don't read code well, you add a temporary prefix to the cache set in
Quoted 7 lines
Quoted 4 lines
echo xxx > /sys/class/scsi_disk/<disk>/cache_type
and it will set the flags for the lifetime of the current kernel, but won't try to do a mode select to make them permanent.
Having the knob is useful indeed. I don't like the "temporary" name though, because "temporary write-through" doesn't sound like it can eat data on a power loss. What about "force" or "assume"?
I'm fairly ambivalent, except not force. The default behaviour is to do the mode select, so force seems to imply that as well, except it won't. I don't see a difference between assume and temporary.
Quoted 1 line
Also, this would be in addition to my patch (when tested), right?
Not really ... given T10s deprecation I don't think we want to touch anything to do with
SYNC_NVbecause it just adds to the uncertainty about what will actually happen. Giving the ability to control WCE (and RCD) fixes all the problems raised so far.
Quoted 10 lines
Quoted 2 lines
Why are we turning off the RCD bit in this? Not sure it matters, but we only should care about WCE (and the dirty write cache data)?
Well, it's in the code. Cache policy is a combination of those two bits. The cache type takes a cache policy string, ergo it must update both. We don't do anything with it because having a write back cache and no cache at all is transparent to us.
James
It was pointed out to me that RCD is "Read Cache Disable" so by setting it to zero, we are enabling the read cache (not that we ever look at this bit or send it down). The WCE bit is "write cache enable" so the polarity of the bits is inverted.
Should be fine regardless :)
Ric
James Bottomley
maintainerQuoted 9 lines · Ric Wheeler
It was pointed out to me that RCD is "Read Cache Disable" so by setting it to
zero, we are enabling the read cache (not that we ever look at this bit or send
it down). The WCE bit is "write cache enable" so the polarity of the bits is
inverted.Should be fine regardless :)
Just look at the code. We already do the right thing.
The whole reason for combining both into cache policy is that half the world got the individual bits wrong. We translate from an understandable cache policy to and from WCE/RCD speak.
James
Il 25/04/2013 03:32, Martin K. Petersen ha scritto:
Quoted 3 lines
I'm ok with your patch. And a strong believer in not altering the SYNCHRONIZE CACHE behavior that's been rigorously tested in the field by adding
SYNC_NVto the mix.
SYNC_NV is absolutely necessary for targets that (a) have both volatile and non-volatile cache, and (b) actually follow the standards behavior for SYNC_NV=0.
I used NV_SUP as a guess that the SYNC_NV bit is supported, perhaps V_SUP && NV_SUP is a better guess.
Paolo
James Bottomley
maintainerQuoted 12 lines · Robert (Server Storage)
If the writeback cache is enabled (per the WCE bit in the Caching mode page), prudent software uses the FUA bit in WRITE commands when writing metadata and/or sends the SYNCHRONIZE CACHE command at important checkpoints to ensure the data is not going to be lost due to a power loss. Some database software is particularly prolific at sending these commands.
Around 2003, many RAID controllers with non-volatile writeback caches honored the SYNCHRONIZE CACHE command, flushing the entire cache to the drives. This started causing timeouts as non-volatile write cache sizes grew. Recently, it's even causing trouble on individual disk drives with growing volatile write caches.
Quoted 4 lines
The intent of software using these commands and bits was unclear - it could be:
a) ensure data is in non-volatile cache (and will eventually be flushed) or on the medium; or
b) ensure data is on the medium (so the drives are ready for removal).
Just from looking at the Linux code (and the code in other operating systems like BSD or Solaris), you can see that for non-removable media our intent is always a).
For removable media, you can argue the OS needs b), but I don't actually know of any removable hard disks that actually have a NV cache (that's exclusively the province of the array vendors), so it's a bit moot.
Quoted 7 lines
As a short-term fix, many RAID controllers assumed intent (a) and started interpreting the SYNCHRONIZE CACHE command as a NOP and ignoring the FUA bit.
Surprise removal of a drive from a RAID controller is risky even if software has run SYNCHRONIZE CACHE, since the RAID controller might be doing other activity in the background. So, there are other reasons to justify assuming that the user just won't do that.
Right. In fact surprise removal of array disks is something most admins quickly learn never to do. The only use case for deliberately damaging your array like this is drive replacement, and that's where you remove a potentially failing device and ask for a rebuild but since the array keeps running, there are no cache issues involved.
Quoted 6 lines
Afraid of breaking software with intent (b) (which was more likely in the days of floppy disks, Bournelli Boxes, and other removable block devices), T10 chose to clarify that the original meaning was (b) and added new
FUA_NVandSYNC_NVbits to let software express intent (a). The hope was that devices would implement the bits and software would start using them at appropriate times.
Just for future learning, does T10 see the mistake here? Even if we assume the b) case (which I think everyone can agree is the wrong one), Operating Systems are slow to change, so arrays have to continue with current behaviour. Even in the b) case, the only way to update the standard to codify existing behaviour and enable the b) case is to say that current SYNCHRONIZE CACHE may now choose not to flush the NV cache but here's a new bit to signal intent to flush NV cache as well (i.e. the new flag should have forced flush of volatile + Non Volatile cache).
By doing the opposite, T10 effectively piled confusion onto the situation because array vendors worried about flush latencies were always going to ignore the flush and new entrants were going to get confused about what the OS is doing, leading to what you say below:
Quoted 4 lines
Unfortunately, the short-term fix worked well enough that it still prevails today, and most standalone removable media block devices have disappeared. There is not much software actually sending the
FUA_NVandSYNC_NVbits and few devices honoring the bits per the standard.
And the arrays that did actually honour the standard are now the ones people are complaining about ...
Quoted 4 lines
As an SBC-3 letter ballot comment, I recently submitted T10 proposal 13-050 (see http://www.t10.org/doc13.htm) to obsolete the
SYNC_NVandFUA_NVbits and change the meaning of the commands without those bits to intent (a), reflecting what the industry has actually done.
I think that works. If an admin is concerned about the b) case, they'll ask the array management software to do the offline rather than the OS, so I don't actually see any use case where we have to worry in the OS about the NV cache.
James