Why is O_DSYNC on linux so slow / what's wrong with my SSD?
Hello,
while struggling about an application beeing so slow on my SSD and having high I/O Waits while the app is using the raw block device i've detected that this is caused by open the block device with O_DSYNC.
I've used dd and fio with oflags=direct,dsync / --direct=1 and --sync=1
and got these "strange" results:
fio --sync=1:
WRITE: io=1694.0MB, aggrb=57806KB/s, minb=57806KB/s, maxb=57806KB/s, mint=30008msec, maxt=30008msec
fio --sync=0:
WRITE: io=5978.0MB, aggrb=204021KB/s, minb=204021KB/s, maxb=204021KB/s, mint=30004msec, maxt=30004msec
I get the same results on a crucial m4 as on my intel 530 ssd.
I also tried the same under FreeBSD 9.1 which shows around the same results for sync=0 as sync=1:
sync=0:
WRITE: io=5984.0MB, aggrb=204185KB/s, minb=204185KB/s, maxb=204185KB/s, mint=30010msec, maxt=30010msec
sync=1:
WRITE: io=5843.0MB, aggrb=199414KB/s, minb=199414KB/s, maxb=199414KB/s, mint=30004msec, maxt=30004msec
Can anyone explain to me why O_DSYNC for my app on linux is so slow?
used kernel is vanilla 3.10.19
Thanks!
Greets Stefan
Quoted 1 line · Stefan Priebe - Profihost AG
Can anyone explain to me why
O_DSYNCfor my app on linux is so slow?
Because FreeBSD ignores O_DSYNC on block devices, it never sends a FLUSH to the device.
Hi Stefan,
Christoph is bang on right. To further elaborate upon this, here is what is happening in the above case :
By using DIRECT, SYNC/DSYNC flags on a block device (i.e. bypassing the file-systems layer), essentially you are enforcing a CMD_FLUSH on each I/O command sent to the disk. This is by design of the block-device driver in the Linux kernel. This severely degrades the performance.
A detailed walk-through of the various I/O scenarios is available at thecodeartist.blogspot.com/2012/08/hdd-filesystems-osync.html
Note that SYNC/DSYNC on a filesystem(eg. ext2/3/4) does NOT issue a CMD_FLUSH. The "SYNC" via filesystem, simply guarantees that the data is sent to the disk and not really flushed to the disk. It will continue to reside in the internal cache on the disk, waiting to be written to the disk platter in a optimum manner (bunch of writes re-ordered to be sequential on-disk and clubbed together in one go). This can affect performance to a large extent on modern HDDs with NCQ support (CMD_FLUSH simply cancels all performance benefits of NCQ).
In case of SSDs, the huge IOPS number for the disk (40,000 in case of Crucial M4) is again typically observed with write-cache enabled. For Crucial M4 SSDs,
http://www.crucial.com/pdf/tech_specs-letter_crucial_m4_ssd_v3-11-11_online.pdf
Footnote1 - "Typical I/O performance numbers as measured using Iometer with a queue depth of 32 and write cache enabled. Iometer measurements are performed on a 8GB span. 4k transfers used for Read/Write latency values."
To simply disable this behaviour and make the SYNC/DSYNC behaviour and performance on raw block-device I/O resemble the standard filesystem I/O you may want to apply the following patch to your kernel -
https://gist.github.com/TheCodeArtist/93dddcd6a21dc81414ba
The above patch simply disables the CMD_FLUSH command support even on disks that claim to support it.
regards
ChinmayVS
Quoted 10 lines · Christoph Hellwig
Quoted 1 line · Stefan Priebe - Profihost AG
Can anyone explain to me why
O_DSYNCfor my app on linux is so slow?Because FreeBSD ignores
O_DSYNCon block devices, it never sends a FLUSH to the device.--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · Chinmay V S
Note that SYNC/DSYNC on a filesystem(eg. ext2/3/4) does NOT issue a
CMD_FLUSH. The "SYNC" via filesystem, simply guarantees that the data is sent to the disk and not really flushed to the disk.
While this used to be the case for ext2 and ext3 this has never been true for the modern filesystem, and has been fixed for ext3 quite a while ago.
Hi ChinmayVS,
Am 20.11.2013 14:34, schrieb Chinmay V S:
Quoted 22 lines
Hi Stefan,
Christoph is bang on right. To further elaborate upon this, here is what is happening in the above case :
By using DIRECT, SYNC/DSYNC flags on a block device (i.e. bypassing the file-systems layer), essentially you are enforcing aCMD_FLUSHon each I/O command sent to the disk. This is by design of the block-device driver in the Linux kernel. This severely degrades the performance.A detailed walk-through of the various I/O scenarios is available at thecodeartist.blogspot.com/2012/08/hdd-filesystems-osync.html
Note that SYNC/DSYNC on a filesystem(eg. ext2/3/4) does NOT issue a
CMD_FLUSH. The "SYNC" via filesystem, simply guarantees that the data is sent to the disk and not really flushed to the disk. It will continue to reside in the internal cache on the disk, waiting to be written to the disk platter in a optimum manner (bunch of writes re-ordered to be sequential on-disk and clubbed together in one go). This can affect performance to a large extent on modern HDDs with NCQ support (CMD_FLUSHsimply cancels all performance benefits of NCQ).
Quoted 8 lines
In case of SSDs, the huge IOPS number for the disk (40,000 in case of Crucial M4) is again typically observed with write-cache enabled. For Crucial M4 SSDs,
http://www.crucial.com/pdf/tech_specs-letter_crucial_m4_ssd_v3-11-11_online.pdf
Footnote1 - "Typical I/O performance numbers as measured using Iometer with a queue depth of 32 and write cache enabled. Iometer measurements are performed on a 8GB span. 4k transfers used for Read/Write latency values."
thanks for your great and detailed reply. I'm just wondering why an intel 520 ssd degrades the speed just by 2% in case of O_SYNC. intel 530 the newer model and replacement for the 520 degrades speed by 75% like the crucial m4.
The Intel DC S3500 instead delivers also nearly 98% of it's performance even under O_SYNC.
Quoted 7 lines
To simply disable this behaviour and make the SYNC/DSYNC behaviour and performance on raw block-device I/O resemble the standard filesystem I/O you may want to apply the following patch to your kernel -
https://gist.github.com/TheCodeArtist/93dddcd6a21dc81414baThe above patch simply disables the
CMD_FLUSHcommand support even on disks that claim to support it.
Is this the right one? By assing ahci_dummy_read_id we disable the CMD_FLUSH?
What is the risk of that one?
Thanks!
Stefan
Hi Stefan,
Quoted 7 lines
thanks for your great and detailed reply. I'm just wondering why an intel 520 ssd degrades the speed just by 2% in case of
O_SYNC. intel 530 the newer model and replacement for the 520 degrades speed by 75% like the crucial m4.The Intel DC S3500 instead delivers also nearly 98% of it's performance even under
O_SYNC.
If you have confirmed the performance numbers, then it indicates that the Intel 530 controller is more advanced and makes better use of the internal disk-cache to achieve better performance (as compared to the Intel 520). Thus forcing CMD_FLUSH on each IOP (negating the benefits of the disk write-cache and not allowing any advanced disk controller optimisations) has a more pronouced effect of degrading the performance on Intel 530 SSDs. (Someone with some actual info on Intel SSDs kindly confirm this.)
Quoted 5 lines
Quoted 5 lines
To simply disable this behaviour and make the SYNC/DSYNC behaviour and performance on raw block-device I/O resemble the standard filesystem I/O you may want to apply the following patch to your kernel -
https://gist.github.com/TheCodeArtist/93dddcd6a21dc81414ba
Quoted 7 lines
Quoted 2 lines
The above patch simply disables the
CMD_FLUSHcommand support even on disks that claim to support it.Is this the right one? By assing
ahci_dummy_read_idwe disable theCMD_FLUSH?What is the risk of that one?
Yes, https://gist.github.com/TheCodeArtist/93dddcd6a21dc81414ba is the right one. The dummy read_id() provides a hook into the initial disk-properties discovery process when the disk is plugged-in. By explicitly negating the bits that indicate cache and flush-cache(CMD_FLUSH) support, we can ensure that the block driver does NOT issue CMD_FLUSH commands to the disk. Note that this does NOT disable the write-cache on the disk itself i.e. performance improves due to the on-disk write-cache in the absence of any CMD_FLUSH commands from the host-PC.
Theoretically, it increases the chances of data loss i.e. if power is removed while the write is in progress from the app. Personally though i have found that the impact of this is minimal because SYNC on a raw block device with CMD_FLUSH does NOT guarantee atomicity in case of a power-loss. Hence, in the event of a power loss, applications cannot rely on SYNC(with CMD_FLUSH) for data integrity. Rather they have to maintain other data-structures with redundant disk metadata (which is precisely what modern file-systems do). Thus, removing CMD_FLUSH doesn't really result in a downside as such.
The main thing to consider when applying the above simple patch is that it is system-wide. The above patch prevents the host-PC from issuing CMD_FLUSH for ALL drives enumerated via SATA/SCSI on the system.
If this patch works for you, then to restrict the change in behaviour to a specific disk, you will need to:
1. Identify the disk by its model number within the dummy read_id().
2. Zero the bits ONLY for your particular disk.
3. Return without modifying anything for all other disks.
Try out the above patch and let me know if you have any further issues.
regards
ChinmayVS
Quoted 9 lines · Chinmay V S
If you have confirmed the performance numbers, then it indicates that the Intel 530 controller is more advanced and makes better use of the internal disk-cache to achieve better performance (as compared to the Intel 520). Thus forcing
CMD_FLUSHon each IOP (negating the benefits of the disk write-cache and not allowing any advanced disk controller optimisations) has a more pronouced effect of degrading the performance on Intel 530 SSDs. (Someone with some actual info on Intel SSDs kindly confirm this.)
You might also want to do some power fail testing to make sure that the SSD is actually flusing all of its internal Flash Translation Layer (FTL) metadata to stable storage on every CMD_FLUSH command.
There are lots of flash media that don't do this, with the result that I get lots of users whining at me when their file system stored on an SD card has massive corruption after a power fail event.
Historically, Intel has been really good about avoiding this, but since they've moved to using 3rd party flash controllers, I now advise everyone who plans to use any flash storage, regardless of the manufacturer, to do their own explicit power fail testing (hitting the reset button is not good enough, you need to kick the power plug out of the wall, or better yet, use a network controlled power switch you so you can repeat the power fail test dozens or hundreds of times for your qualification run) before being using flash storage in a mission critical situation where you care about data integrity after a power fail event.
IOW, make sure that the SSD isn't faster because it's playing fast and loose with the FTL metadata....
Cheers,
- Ted
Quoted 11 lines · Theodore Ts'o
Quoted 9 lines · Chinmay V S
If you have confirmed the performance numbers, then it indicates that the Intel 530 controller is more advanced and makes better use of the internal disk-cache to achieve better performance (as compared to the Intel 520). Thus forcing
CMD_FLUSHon each IOP (negating the benefits of the disk write-cache and not allowing any advanced disk controller optimisations) has a more pronouced effect of degrading the performance on Intel 530 SSDs. (Someone with some actual info on Intel SSDs kindly confirm this.)
Quoted 3 lines
You might also want to do some power fail testing to make sure that the SSD is actually flusing all of its internal Flash Translation Layer (FTL) metadata to stable storage on every
CMD_FLUSHcommand.
Some SSD's are also claim the ability to flush the cache on power loss:
Which should in theory let them respond immediately to flush requests, right? Except they only seem to advertise it as a safety (rather than a performance) feature, so I probably misunderstand something.
And the 520 doesn't claim this feature (look for "enhanced power loss protection" at http://ark.intel.com/products/66248), so that wouldn't explain these results anyway.
--b.
Quoted 26 lines
There are lots of flash media that don't do this, with the result that I get lots of users whining at me when their file system stored on an SD card has massive corruption after a power fail event.
Historically, Intel has been really good about avoiding this, but since they've moved to using 3rd party flash controllers, I now advise everyone who plans to use any flash storage, regardless of the manufacturer, to do their own explicit power fail testing (hitting the reset button is not good enough, you need to kick the power plug out of the wall, or better yet, use a network controlled power switch you so you can repeat the power fail test dozens or hundreds of times for your qualification run) before being using flash storage in a mission critical situation where you care about data integrity after a power fail event.
IOW, make sure that the SSD isn't faster because it's playing fast and loose with the FTL metadata....
Cheers,
- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Theodore Ts'o
Historically, Intel has been really good about avoiding this, but since they've moved to using 3rd party flash controllers, I now advise everyone who plans to use any flash storage, regardless of the manufacturer, to do their own explicit power fail testing (hitting the reset button is not good enough, you need to kick the power plug out of the wall, or better yet, use a network controlled power switch you so you can repeat the power fail test dozens or hundreds of times for your qualification run) before being using flash storage in a mission critical situation where you care about data integrity after a power fail event.
Speaking of which, what would you use to automate this sort of test? I'm thinking an SSD connected by eSATA, with an external power supply, and the host running inside a VM. Drop power to the drive at the same time as doing a kill -9 on the VM, then you can resume the VM pretty quickly instead of waiting for a full reboot sequence.
Signature · 4 lines
-- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Quoted 11 lines · J. Bruce Fields
Some SSD's are also claim the ability to flush the cache on power loss:
Which should in theory let them respond immediately to flush requests, right? Except they only seem to advertise it as a safety (rather than a performance) feature, so I probably misunderstand something.
And the 520 doesn't claim this feature (look for "enhanced power loss protection" at http://ark.intel.com/products/66248), so that wouldn't explain these results anyway.
FYI, nowhere does Intel imply that the CMD_FLUSH is instantaneous. The product brief for Intel 320 SSDs (above link), explains that it is implemented by a power-fail detection circuit that detects drop in power-supply, following which the on-disk controller issues an internal CMD_FLUSH equivalent command to ensure data is moved to the non-volatile area from the disk-cache. Large secondary capacitors ensure backup supply for this brief duration.
Thus applications can always perform asynchronous I/O upon the disk, taking comfort in the fact that the physical disk ensures that all data in the volatile disk-cache is automatically transferred to the non-volatile area even in the event of an external power-failure. Thus the host never has to worry about issuing a CMD_FLUSH (which is still a terribly expensive performance bottleneck, even on the Intel 320 SSDs).
Quoted 21 lines · Chinmay V S
Quoted 11 lines · J. Bruce Fields
Some SSD's are also claim the ability to flush the cache on power loss:
Which should in theory let them respond immediately to flush requests, right? Except they only seem to advertise it as a safety (rather than a performance) feature, so I probably misunderstand something.
And the 520 doesn't claim this feature (look for "enhanced power loss protection" at http://ark.intel.com/products/66248), so that wouldn't explain these results anyway.
FYI, nowhere does Intel imply that the
CMD_FLUSHis instantaneous. The product brief for Intel 320 SSDs (above link), explains that it is implemented by a power-fail detection circuit that detects drop in power-supply, following which the on-disk controller issues an internalCMD_FLUSHequivalent command to ensure data is moved to the non-volatile area from the disk-cache. Large secondary capacitors ensure backup supply for this brief duration.
Quoted 7 lines
Thus applications can always perform asynchronous I/O upon the disk, taking comfort in the fact that the physical disk ensures that all data in the volatile disk-cache is automatically transferred to the non-volatile area even in the event of an external power-failure. Thus the host never has to worry about issuing a
CMD_FLUSH(which is still a terribly expensive performance bottleneck, even on the Intel 320 SSDs).
So why is it up to the application to do this and not the drive? Naively I'd've thought it would be simpler if the protocol allowed the drive to respond instantly if it knows it can do so safely, and then you could always issue flush requests, and save some poor admin from having to read spec sheets to figure out if they can safely mount "nobarrier".
Is it that you want to eliminate CMD_FLUSH entirely because the protocol still has some significant overhead even if the drive responds to it quickly?
--b.
Quoted 30 lines · J. Bruce Fields
Quoted 21 lines · Chinmay V S
Quoted 11 lines · J. Bruce Fields
Some SSD's are also claim the ability to flush the cache on power loss:
Which should in theory let them respond immediately to flush requests, right? Except they only seem to advertise it as a safety (rather than a performance) feature, so I probably misunderstand something.
And the 520 doesn't claim this feature (look for "enhanced power loss protection" at http://ark.intel.com/products/66248), so that wouldn't explain these results anyway.
FYI, nowhere does Intel imply that the
CMD_FLUSHis instantaneous. The product brief for Intel 320 SSDs (above link), explains that it is implemented by a power-fail detection circuit that detects drop in power-supply, following which the on-disk controller issues an internalCMD_FLUSHequivalent command to ensure data is moved to the non-volatile area from the disk-cache. Large secondary capacitors ensure backup supply for this brief duration.Quoted 7 lines
Thus applications can always perform asynchronous I/O upon the disk, taking comfort in the fact that the physical disk ensures that all data in the volatile disk-cache is automatically transferred to the non-volatile area even in the event of an external power-failure. Thus the host never has to worry about issuing a
CMD_FLUSH(which is still a terribly expensive performance bottleneck, even on the Intel 320 SSDs).
Quoted 5 lines
So why is it up to the application to do this and not the drive? Naively I'd've thought it would be simpler if the protocol allowed the drive to respond instantly if it knows it can do so safely, and then you could always issue flush requests, and save some poor admin from having to read spec sheets to figure out if they can safely mount "nobarrier".
Strictly speaking CMD_FLUSH implies that the app/driver wants to ensure data IS in-fact on the non-volatile area. Also the time-penalty associated with it on majority of disks is a known fact and hence CMD_FLUSHes are not issued unless absolutely necessary. During IO upon a raw block device, as this is the ONLY data barrier available, it is mapped to the SYNC command.
The Intel 320 SSD is an exception where the disk does NOT need a CMD_FLUSH as it can guarantee that the cache is always flushed to the non-volatile area automatically in case of a power loss. However, a CMD_FLUSH is an explicit command to write to non-volatile area and is implemented accordingly. Practically though it is could have been made a no-op on the Intel 320 series(and other similar battery-backed disks, but not for all disks). Unfortunately this is not how the on-disk controller firmware is implemented and hence it is up to the app/kernel-driver to avoid issuing CMD_FLUSHes which are clearly unnecessary as discussed above.
Quoted 3 lines
Is it that you want to eliminate
CMD_FLUSHentirely because the protocol still has some significant overhead even if the drive responds to it quickly?
1. Most drives do NOT respond to CMD_FLUSH immediately i.e. they wait until the data is actually moved to the non-volatile media (which is the right behaviour) i.e. performance drops.
2. Some drives may implement CMD_FLUSH to return immediately i.e. no guarantee the data is actually on disk.
3. Anyway, CMD_FLUSH does NOT guarantee atomicity. (Consider power failure in the middle of an ongoing CMD_FLUSH on non battery-backed disks).
4. Throughput using CMD_FLUSH is so less that an app generating large amount of I/O will have to buffer most of it in the app layer itself i.e. it is lost in case of a power-outage.
Considering the above 4 facts, ASYNC IO is almost always better on raw block devices. This pushes the data to the disk as fast as possible and an occasional CMD_FLUSH will ensure it is flushed to the non-volatile area periodically.
In case the application cannot be modified to perform ASYNC IO, there exists a way to disable the behaviour of issuing a CMD_FLUSH for each sync() within the block device driver for SATA/SCSI disks. This is what is described by
https://gist.github.com/TheCodeArtist/93dddcd6a21dc81414ba
Just to be clear, i am NOT recommending that this change be mainlined; rather it is a reference to improve performance in the rare cases(like in the OP Stefan's case) where both the app performing DIRECT SYNC block IO and the disk firmware implementing CMD_FLUSH can NOT be modified. In which case the standard block driver behaviour of issuing a CMD_FLUSH with each write is too restrictive and thus modified using the patch.
Quoted 4 lines
1. Most drives do NOT respond to
CMD_FLUSHimmediately i.e. they wait until the data is actually moved to the non-volatile media (which is the right behaviour) i.e. performance drops.
Which is what the specification sais they must do.
Quoted 2 lines
2. Some drives may implement
CMD_FLUSHto return immediately i.e. no guarantee the data is actually on disk.
In which case they aren't spec complicant. While I've seen countless data integrity bugs on lower end ATA SSDs I've not seen one that simpliy ingnores flush. If you'd want to cheat that bluntly you'd be better of just claiming to not have a writeback cache.
Quoted 3 lines
3. Anyway,
CMD_FLUSHdoes NOT guarantee atomicity. (Consider power failure in the middle of an ongoingCMD_FLUSHon non battery-backed disks).
It does not guarantee atomicy by itself, but it's the only low-level primitive a filesystem or database can use build atomic transaction at a higher level on an ATA disk with the writeback cache enabled.
Quoted 5 lines
In case the application cannot be modified to perform ASYNC IO, there exists a way to disable the behaviour of issuing a
CMD_FLUSHfor eachsync()within the block device driver for SATA/SCSI disks. This is what is described by
https://gist.github.com/TheCodeArtist/93dddcd6a21dc81414ba
Which is utterly broken, and your insistance on pushing it shows you do not understand the problem space.
You solve your performance problem by completely disabling any chance of having data integrity guarantees, and do so in a way that is not detectable for applications or users.
If you have a workload with lots of small synchronous writes disabling the writeback cache on the disk does indeed often help, especially with the non-queueable FLUSH on all but the most recent ATA devices.
If you do have workloads where you do lots of synchronous writes
Quoted 7 lines
Just to be clear, i am NOT recommending that this change be mainlined; rather it is a reference to improve performance in the rare cases(like in the OP Stefan's case) where both the app performing DIRECT SYNC block IO and the disk firmware implementing
CMD_FLUSHcan NOT be modified. In which case the standard block driver behaviour of issuing aCMD_FLUSHwith each write is too restrictive and thus modified using the patch.
Again, what your patch does is to explicitly ignore the data integrity request from the application. While this will usually be way faster, it will also cause data loss. Simply disabling the writeback cache feature of the disk using hdparm will give you much better performance than issueing all the FLUSH command, especially if they are non-queued, but without breaking the gurantee to the application.
Am 20.11.2013 16:22, schrieb Chinmay V S:
Quoted 32 lines
Hi Stefan,
Quoted 7 lines
thanks for your great and detailed reply. I'm just wondering why an intel 520 ssd degrades the speed just by 2% in case of
O_SYNC. intel 530 the newer model and replacement for the 520 degrades speed by 75% like the crucial m4.The Intel DC S3500 instead delivers also nearly 98% of it's performance even under
O_SYNC.If you have confirmed the performance numbers, then it indicates that the Intel 530 controller is more advanced and makes better use of the internal disk-cache to achieve better performance (as compared to the Intel 520). Thus forcing
CMD_FLUSHon each IOP (negating the benefits of the disk write-cache and not allowing any advanced disk controller optimisations) has a more pronouced effect of degrading the performance on Intel 530 SSDs. (Someone with some actual info on Intel SSDs kindly confirm this.)Quoted 5 lines
Quoted 5 lines
To simply disable this behaviour and make the SYNC/DSYNC behaviour and performance on raw block-device I/O resemble the standard filesystem I/O you may want to apply the following patch to your kernel -
https://gist.github.com/TheCodeArtist/93dddcd6a21dc81414baQuoted 7 lines
Quoted 2 lines
The above patch simply disables the
CMD_FLUSHcommand support even on disks that claim to support it.Is this the right one? By assing
ahci_dummy_read_idwe disable theCMD_FLUSH?What is the risk of that one?
Quoted 9 lines
Yes, https://gist.github.com/TheCodeArtist/93dddcd6a21dc81414ba is the right one. The dummy
read_id()provides a hook into the initial disk-properties discovery process when the disk is plugged-in. By explicitly negating the bits that indicate cache and flush-cache(CMD_FLUSH) support, we can ensure that the block driver does NOT issueCMD_FLUSHcommands to the disk. Note that this does NOT disable the write-cache on the disk itself i.e. performance improves due to the on-disk write-cache in the absence of anyCMD_FLUSHcommands from the host-PC.
ah OK thanks.
Quoted 9 lines
Theoretically, it increases the chances of data loss i.e. if power is removed while the write is in progress from the app. Personally though i have found that the impact of this is minimal because SYNC on a raw block device with
CMD_FLUSHdoes NOT guarantee atomicity in case of a power-loss. Hence, in the event of a power loss, applications cannot rely onSYNC(with CMD_FLUSH)for data integrity. Rather they have to maintain other data-structures with redundant disk metadata (which is precisely what modern file-systems do). Thus, removingCMD_FLUSHdoesn't really result in a downside as such.
In my production system i've crucial m500 which have a capicitor so in a case of power loss they flush their data to disk automatically.
Quoted 5 lines
The main thing to consider when applying the above simple patch is that it is system-wide. The above patch prevents the host-PC from issuing
CMD_FLUSHfor ALL drives enumerated via SATA/SCSI on the system.
Quoted 7 lines
If this patch works for you, then to restrict the change in behaviour to a specific disk, you will need to:
1. Identify the disk by its model number within the dummyread_id().
2. Zero the bits ONLY for your particular disk.
3. Return without modifying anything for all other disks.Try out the above patch and let me know if you have any further issues.
The best thing would be a a flag under/sys/bock/sdc/device/
for ssds with capictor - so everybody can decide on their own.
Stefan
Am 20.11.2013 16:55, schrieb J. Bruce Fields:
Quoted 16 lines · Theodore Ts'o
Quoted 11 lines · Chinmay V S
Quoted 9 lines
If you have confirmed the performance numbers, then it indicates that the Intel 530 controller is more advanced and makes better use of the internal disk-cache to achieve better performance (as compared to the Intel 520). Thus forcing
CMD_FLUSHon each IOP (negating the benefits of the disk write-cache and not allowing any advanced disk controller optimisations) has a more pronouced effect of degrading the performance on Intel 530 SSDs. (Someone with some actual info on Intel SSDs kindly confirm this.)Quoted 3 lines
You might also want to do some power fail testing to make sure that the SSD is actually flusing all of its internal Flash Translation Layer (FTL) metadata to stable storage on every
CMD_FLUSHcommand.
Quoted 7 lines
Some SSD's are also claim the ability to flush the cache on power loss:
Which should in theory let them respond immediately to flush requests, right? Except they only seem to advertise it as a safety (rather than a performance) feature, so I probably misunderstand something.
Yes but they all should make use and support CMD_FLUSH so it's slow on them too.
Quoted 3 lines
And the 520 doesn't claim this feature (look for "enhanced power loss protection" at http://ark.intel.com/products/66248), so that wouldn't explain these results anyway.
Correct i think intel simply ignores CMD_FLUSH on that drive - no idea why an they fixed this for their 330, 530, DC S3500 (all tested)
Quoted 28 lines
--b.
Quoted 26 lines
There are lots of flash media that don't do this, with the result that I get lots of users whining at me when their file system stored on an SD card has massive corruption after a power fail event.
Historically, Intel has been really good about avoiding this, but since they've moved to using 3rd party flash controllers, I now advise everyone who plans to use any flash storage, regardless of the manufacturer, to do their own explicit power fail testing (hitting the reset button is not good enough, you need to kick the power plug out of the wall, or better yet, use a network controlled power switch you so you can repeat the power fail test dozens or hundreds of times for your qualification run) before being using flash storage in a mission critical situation where you care about data integrity after a power fail event.
IOW, make sure that the SSD isn't faster because it's playing fast and loose with the FTL metadata....
Cheers,
- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Christoph,
Am 21.11.2013 11:11, schrieb Christoph Hellwig:
Quoted 9 lines
Quoted 3 lines
2. Some drives may implement
CMD_FLUSHto return immediately i.e. no guarantee the data is actually on disk.In which case they aren't spec complicant. While I've seen countless data integrity bugs on lower end ATA SSDs I've not seen one that simpliy ingnores flush. If you'd want to cheat that bluntly you'd be better of just claiming to not have a writeback cache.
Quoted 7 lines
You solve your performance problem by completely disabling any chance of having data integrity guarantees, and do so in a way that is not detectable for applications or users.
If you have a workload with lots of small synchronous writes disabling the writeback cache on the disk does indeed often help, especially with the non-queueable FLUSH on all but the most recent ATA devices.
But this isn't correct for drives with capicitors like Crucial m500, Intel DC S3500, DC S3700 isn't it? Shouldn't the linux kernel has an option to disable this for drives like these? /sys/block/sdX/device/ignore_flush
Quoted 7 lines
Again, what your patch does is to explicitly ignore the data integrity request from the application. While this will usually be way faster, it will also cause data loss. Simply disabling the writeback cache feature of the disk using hdparm will give you much better performance than issueing all the FLUSH command, especially if they are non-queued, but without breaking the gurantee to the application.
Quoted 15 lines · Stefan Priebe
Hi Christoph,
Am 21.11.2013 11:11, schrieb Christoph Hellwig:Quoted 13 lines
Quoted 3 lines
2. Some drives may implement
CMD_FLUSHto return immediately i.e. no guarantee the data is actually on disk.In which case they aren't spec complicant. While I've seen countless data integrity bugs on lower end ATA SSDs I've not seen one that simpliy ingnores flush. If you'd want to cheat that bluntly you'd be better of just claiming to not have a writeback cache.
You solve your performance problem by completely disabling any chance of having data integrity guarantees, and do so in a way that is not detectable for applications or users.
Quoted 6 lines
Quoted 3 lines
If you have a workload with lots of small synchronous writes disabling the writeback cache on the disk does indeed often help, especially with the non-queueable FLUSH on all but the most recent ATA devices.
But this isn't correct for drives with capicitors like Crucial m500, Intel DC S3500, DC S3700 isn't it? Shouldn't the linux kernel has an option to disable this for drives like these?
/sys/block/sdX/device/ignore_flush
If you know 100% for sure that your drive has a non-volatile write cache, you can run the file system without the flushing by mounting "-o nobarrier". With most devices, this is not needed since they tend to simply ignore the flushes if they know they are power failure safe.
Block level, we did something similar for users who are not running through a file system for SCSI devices - James added support to echo "temporary" into the sd's device's cache_type field:
See:
Ric
Quoted 9 lines
Quoted 8 lines
Again, what your patch does is to explicitly ignore the data integrity request from the application. While this will usually be way faster, it will also cause data loss. Simply disabling the writeback cache feature of the disk using hdparm will give you much better performance than issueing all the FLUSH command, especially if they are non-queued, but without breaking the gurantee to the application.
Hi Ric,
Am 22.11.2013 21:37, schrieb Ric Wheeler:
Quoted 25 lines · Stefan Priebe
Quoted 15 lines
Hi Christoph,
Am 21.11.2013 11:11, schrieb Christoph Hellwig:Quoted 13 lines
Quoted 3 lines
2. Some drives may implement
CMD_FLUSHto return immediately i.e. no guarantee the data is actually on disk.In which case they aren't spec complicant. While I've seen countless data integrity bugs on lower end ATA SSDs I've not seen one that simpliy ingnores flush. If you'd want to cheat that bluntly you'd be better of just claiming to not have a writeback cache.
You solve your performance problem by completely disabling any chance of having data integrity guarantees, and do so in a way that is not detectable for applications or users.
Quoted 8 lines
Quoted 3 lines
If you have a workload with lots of small synchronous writes disabling the writeback cache on the disk does indeed often help, especially with the non-queueable FLUSH on all but the most recent ATA devices.
But this isn't correct for drives with capicitors like Crucial m500, Intel DC S3500, DC S3700 isn't it? Shouldn't the linux kernel has an option to disable this for drives like these?
/sys/block/sdX/device/ignore_flush
Quoted 4 lines
If you know 100% for sure that your drive has a non-volatile write cache, you can run the file system without the flushing by mounting "-o nobarrier". With most devices, this is not needed since they tend to simply ignore the flushes if they know they are power failure safe.
Thanks - but i have raw block devices the data goes to .
Quoted 7 lines
Block level, we did something similar for users who are not running through a file system for SCSI devices - James added support to echo "temporary" into the sd's device's
cache_typefield:See:
So i have to switch to write through but i'm still using the wb cache of the device?
echo temporary write through > /sys/class/scsi_disk/<disk>/cache_type
Thanks!
Stefan
Hi Ric,
Am 22.11.2013 21:37, schrieb Ric Wheeler:
Quoted 30 lines · Stefan Priebe
Quoted 15 lines
Hi Christoph,
Am 21.11.2013 11:11, schrieb Christoph Hellwig:Quoted 13 lines
Quoted 3 lines
2. Some drives may implement
CMD_FLUSHto return immediately i.e. no guarantee the data is actually on disk.In which case they aren't spec complicant. While I've seen countless data integrity bugs on lower end ATA SSDs I've not seen one that simpliy ingnores flush. If you'd want to cheat that bluntly you'd be better of just claiming to not have a writeback cache.
You solve your performance problem by completely disabling any chance of having data integrity guarantees, and do so in a way that is not detectable for applications or users.
Quoted 8 lines
Quoted 3 lines
If you have a workload with lots of small synchronous writes disabling the writeback cache on the disk does indeed often help, especially with the non-queueable FLUSH on all but the most recent ATA devices.
But this isn't correct for drives with capicitors like Crucial m500, Intel DC S3500, DC S3700 isn't it? Shouldn't the linux kernel has an option to disable this for drives like these?
/sys/block/sdX/device/ignore_flushIf you know 100% for sure that your drive has a non-volatile write cache, you can run the file system without the flushing by mounting "-o nobarrier". With most devices, this is not needed since they tend to simply ignore the flushes if they know they are power failure safe.
Quoted 7 lines
Block level, we did something similar for users who are not running through a file system for SCSI devices - James added support to echo "temporary" into the sd's device's
cache_typefield:See:
At least to me this does not work. I get the same awful speed as before - also the I/O waits stay the same. I'm still seeing CMD flushes going to the devices.
Is there any way to check whether the temporary got accepted and works?
I simply executed:
for i in /sys/class/scsi_disk/*/cache_type; do echo $i; echo temporary write back >$i; done
Stefan
Quoted 42 lines · Stefan Priebe
Hi Ric,
Am 22.11.2013 21:37, schrieb Ric Wheeler:
Quoted 30 lines · Stefan Priebe
Quoted 15 lines
Hi Christoph,
Am 21.11.2013 11:11, schrieb Christoph Hellwig:Quoted 13 lines
Quoted 3 lines
2. Some drives may implement
CMD_FLUSHto return immediately i.e. no guarantee the data is actually on disk.In which case they aren't spec complicant. While I've seen countless data integrity bugs on lower end ATA SSDs I've not seen one that simpliy ingnores flush. If you'd want to cheat that bluntly you'd be better of just claiming to not have a writeback cache.
You solve your performance problem by completely disabling any chance of having data integrity guarantees, and do so in a way that is not detectable for applications or users.
Quoted 8 lines
Quoted 3 lines
If you have a workload with lots of small synchronous writes disabling the writeback cache on the disk does indeed often help, especially with the non-queueable FLUSH on all but the most recent ATA devices.
But this isn't correct for drives with capicitors like Crucial m500, Intel DC S3500, DC S3700 isn't it? Shouldn't the linux kernel has an option to disable this for drives like these?
/sys/block/sdX/device/ignore_flushIf you know 100% for sure that your drive has a non-volatile write cache, you can run the file system without the flushing by mounting "-o nobarrier". With most devices, this is not needed since they tend to simply ignore the flushes if they know they are power failure safe.
Quoted 8 lines
Block level, we did something similar for users who are not running through a file system for SCSI devices - James added support to echo "temporary" into the sd's device's
cache_typefield:See:
Quoted 8 lines
At least to me this does not work. I get the same awful speed as before - also the I/O waits stay the same. I'm still seeing CMD flushes going to the devices.
Is there any way to check whether the temporary got accepted and works?
I simply executed:
for i in/sys/class/scsi_disk/*/cache_type; do echo $i; echo temporary write back >$i; doneStefan
What kernel are you running? This is a new addition....
Also, you can "cat" the same file to see what it says.
Regards,
Ric
Hi Ric,
Am 23.11.2013 20:35, schrieb Ric Wheeler:
Quoted 51 lines · Stefan Priebe
Quoted 50 lines
Hi Ric,
Am 22.11.2013 21:37, schrieb Ric Wheeler:
Quoted 32 lines · Stefan Priebe
Quoted 21 lines
Hi Christoph,
Am 21.11.2013 11:11, schrieb Christoph Hellwig:Quoted 10 lines
Quoted 3 lines
2. Some drives may implement
CMD_FLUSHto return immediately i.e. no guarantee the data is actually on disk.In which case they aren't spec complicant. While I've seen countless data integrity bugs on lower end ATA SSDs I've not seen one that simpliy
ingnores flush. If you'd want to cheat that bluntly you'd be better of just claiming to not have a writeback cache.Quoted 8 lines
You solve your performance problem by completely disabling any chance of having data integrity guarantees, and do so in a way that is not detectable for applications or users.
If you have a workload with lots of small synchronous writes disabling the writeback cache on the disk does indeed often help, especially with
the non-queueable FLUSH on all but the most recent ATA devices.Quoted 4 lines
But this isn't correct for drives with capicitors like Crucial m500, Intel DC S3500, DC S3700 isn't it? Shouldn't the linux kernel has an option to disable this for drives like these?
/sys/block/sdX/device/ignore_flushIf you know 100% for sure that your drive has a non-volatile write cache, you can run the file system without the flushing by mounting "-o nobarrier". With most devices, this is not needed since they tend to simply ignore the flushes if they know they are power failure safe.
Quoted 8 lines
Block level, we did something similar for users who are not running through a file system for SCSI devices - James added support to echo "temporary" into the sd's device's
cache_typefield:See:
At least to me this does not work. I get the same awful speed as before - also the I/O waits stay the same. I'm still seeing CMD flushes going to the devices.
Is there any way to check whether the temporary got accepted and works?
Quoted 7 lines
Quoted 5 lines
I simply executed:
for i in/sys/class/scsi_disk/*/cache_type; do echo $i; echo temporary write back >$i; doneStefan
What kernel are you running? This is a new addition....
3.10.19
Quoted 1 line
Also, you can "cat" the same file to see what it says.
Sure:
[cloud2-1338: ~]# for i in /sys/class/scsi_disk/*/cache_type; do echo $i; echo temporary write through >$i; done /sys/class/scsi_disk/1:0:0:0/cache_type
/sys/class/scsi_disk/2:0:0:0/cache_type
/sys/class/scsi_disk/3:0:0:0/cache_type
/sys/class/scsi_disk/4:0:0:0/cache_type
/sys/class/scsi_disk/5:0:0:0/cache_type
/sys/class/scsi_disk/6:0:0:0/cache_type
[cloud2-1338: ~]# for i in /sys/class/scsi_disk/*/cache_type; do echo $i; cat $i; done /sys/class/scsi_disk/1:0:0:0/cache_type
write through
/sys/class/scsi_disk/2:0:0:0/cache_type
write through
/sys/class/scsi_disk/3:0:0:0/cache_type
write through
/sys/class/scsi_disk/4:0:0:0/cache_type
write through
/sys/class/scsi_disk/5:0:0:0/cache_type
write through
/sys/class/scsi_disk/6:0:0:0/cache_type
write through
[cloud2-1338: ~]# for i in /sys/class/scsi_disk/*/cache_type; do echo $i; echo temporary write back >$i; done /sys/class/scsi_disk/1:0:0:0/cache_type
/sys/class/scsi_disk/2:0:0:0/cache_type
/sys/class/scsi_disk/3:0:0:0/cache_type
/sys/class/scsi_disk/4:0:0:0/cache_type
/sys/class/scsi_disk/5:0:0:0/cache_type
/sys/class/scsi_disk/6:0:0:0/cache_type
[cloud2-1338: ~]# for i in /sys/class/scsi_disk/*/cache_type; do echo $i; cat $i; done /sys/class/scsi_disk/1:0:0:0/cache_type
write back
/sys/class/scsi_disk/2:0:0:0/cache_type
write back
/sys/class/scsi_disk/3:0:0:0/cache_type
write back
/sys/class/scsi_disk/4:0:0:0/cache_type
write back
/sys/class/scsi_disk/5:0:0:0/cache_type
write back
/sys/class/scsi_disk/6:0:0:0/cache_type
write back
[cloud2-1338: ~]#
Stefan
Quoted 12 lines · Howard Chu
Quoted 10 lines · Theodore Ts'o
Historically, Intel has been really good about avoiding this, but since they've moved to using 3rd party flash controllers, I now advise everyone who plans to use any flash storage, regardless of the manufacturer, to do their own explicit power fail testing (hitting the reset button is not good enough, you need to kick the power plug out of the wall, or better yet, use a network controlled power switch you so you can repeat the power fail test dozens or hundreds of times for your qualification run) before being using flash storage in a mission critical situation where you care about data integrity after a power fail event.
Quoted 5 lines
Speaking of which, what would you use to automate this sort of test? I'm thinking an SSD connected by eSATA, with an external power supply, and the host running inside a VM. Drop power to the drive at the same time as doing a kill -9 on the VM, then you can resume the VM pretty quickly instead of waiting for a full reboot sequence.
I was just pulling power on sata drive.
It uncovered "interesting" stuff. I plugged power back, and kernel re-estabilished communication with that drive, but any settings with hdparm were forgotten. I'd say there's some room for improvement there...
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 17 lines · Pavel Machek
Quoted 11 lines · Howard Chu
Quoted 10 lines · Theodore Ts'o
Historically, Intel has been really good about avoiding this, but since they've moved to using 3rd party flash controllers, I now advise everyone who plans to use any flash storage, regardless of the manufacturer, to do their own explicit power fail testing (hitting the reset button is not good enough, you need to kick the power plug out of the wall, or better yet, use a network controlled power switch you so you can repeat the power fail test dozens or hundreds of times for your qualification run) before being using flash storage in a mission critical situation where you care about data integrity after a power fail event.
Quoted 5 lines
Speaking of which, what would you use to automate this sort of test? I'm thinking an SSD connected by eSATA, with an external power supply, and the host running inside a VM. Drop power to the drive at the same time as doing a kill -9 on the VM, then you can resume the VM pretty quickly instead of waiting for a full reboot sequence.
Quoted 8 lines
I was just pulling power on sata drive.
It uncovered "interesting" stuff. I plugged power back, and kernel re-estabilished communication with that drive, but any settings with hdparm were forgotten. I'd say there's some room for improvement there...
Pavel
Hi Pavel,
When you drop power, your drive normally loses temporary settings (like a change to write cache, etc).
Depending on the class of the device, there are ways to make that permanent (look at hdparm or sdparm for details).
This is a feature of the drive and its firmware, not something we reset in the device each time it re-appears.
Ric
Quoted 6 lines
Quoted 3 lines
And the 520 doesn't claim this feature (look for "enhanced power loss protection" at http://ark.intel.com/products/66248), so that wouldn't explain these results anyway.
Correct i think intel simply ignores
CMD_FLUSHon that drive - no idea why an they fixed this for their 330, 530, DC S3500 (all tested)
You are not as I read the standard allowed to "ignore" it. In fact if you advertise the property you are obliged to implement it. The late Andre Hedrick made sure the standard was phrased the way it was to stop it being abused for benchmarketing. The goal was that anyone cheating would be non-compliant.
Now its entirely possible to do clever stuff and treat it merely as a write barrier, providing you can't lose what is queued up. What the actual drives do I don't know.. all deep magic and not my department.
A second thing to be careful about is that certain kinds of I/O barriers and atomic write patterns that force lots of commits to flash and erase cycles are going to wear the drive out faster and I've been told by manufacturers that drives do respond to such patterns by limiting the transaction rate in self defence (and presumably in the hope the OS will then begin to block stuff up better).
Pavel - what is lost/kept over the reset of a device is also fairly clearly defined in the standard. Much is lost because if you committed a permanent configuration change that the controller couldnt support you would be a bit screwed!
If you are driving an SSD I'd work very hard to avoid the need for any kind of O_SYNC or O_DSYNC type behaviour for exactly the same reason you avoid uncached memory accesses - the hardware can't do its job properly without the needed freedom. Use minimal barriers and proper sync points and your performance will be far higher.
Alan
Quoted 27 lines · Ric Wheeler
Quoted 17 lines · Pavel Machek
Quoted 11 lines · Howard Chu
Quoted 10 lines · Theodore Ts'o
Historically, Intel has been really good about avoiding this, but since they've moved to using 3rd party flash controllers, I now advise everyone who plans to use any flash storage, regardless of the manufacturer, to do their own explicit power fail testing (hitting the reset button is not good enough, you need to kick the power plug out of the wall, or better yet, use a network controlled power switch you so you can repeat the power fail test dozens or hundreds of times for your qualification run) before being using flash storage in a mission critical situation where you care about data integrity after a power fail event.
Quoted 5 lines
Speaking of which, what would you use to automate this sort of test? I'm thinking an SSD connected by eSATA, with an external power supply, and the host running inside a VM. Drop power to the drive at the same time as doing a kill -9 on the VM, then you can resume the VM pretty quickly instead of waiting for a full reboot sequence.
Quoted 6 lines
I was just pulling power on sata drive.
It uncovered "interesting" stuff. I plugged power back, and kernel re-estabilished communication with that drive, but any settings with hdparm were forgotten. I'd say there's some room for improvement there...
Hi Pavel,
Quoted 8 lines
When you drop power, your drive normally loses temporary settings (like a change to write cache, etc).
Depending on the class of the device, there are ways to make that permanent (look at hdparm or sdparm for details).
This is a feature of the drive and its firmware, not something we reset in the device each time it re-appears.
Yes, and I'm arguing that is a bug (as in, < 0.01% people are using hdparm correctly).
So you used hparm to disable write cache so that ext3 can be safely used on your hdd. Now you have glitch on power. Then, system continues to operate in dangerous mode until reboot.
I guess it would be safer not to reattach drives after power fail... (also I wonder what this does to data integrity. Drive lost content of its writeback cache, but kernel continues... Journal will not prevent data corruption in this case).
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 2 lines
Yes, and I'm arguing that is a bug (as in, < 0.01% people are using hdparm correctly).
Generally speaking if you are using hdparm for tuning it means we need to fix something in the ATA layer so you don't have to !
Quoted 4 lines
I guess it would be safer not to reattach drives after power fail... (also I wonder what this does to data integrity. Drive lost content of its writeback cache, but kernel continues... Journal will not prevent data corruption in this case).
For good or bad its very hard to tell if a drive randomly powers off or we merely get a bus reset in the ATA case. In the SATA case we do at least get the relevant events to handle it nicely as we *should* see a DevExch event. We also can't tell a power fail event from a hot drive swap, so we most definitely want to re-attach the drive, just so long as we ensure that it comes back on a different device node.
Alan
Quoted 30 lines · Pavel Machek
Quoted 29 lines · Ric Wheeler
Quoted 17 lines · Pavel Machek
Quoted 11 lines · Howard Chu
Quoted 10 lines · Theodore Ts'o
Historically, Intel has been really good about avoiding this, but since they've moved to using 3rd party flash controllers, I now advise everyone who plans to use any flash storage, regardless of the manufacturer, to do their own explicit power fail testing (hitting the reset button is not good enough, you need to kick the power plug out of the wall, or better yet, use a network controlled power switch you so you can repeat the power fail test dozens or hundreds of times for your qualification run) before being using flash storage in a mission critical situation where you care about data integrity after a power fail event.
Quoted 5 lines
Speaking of which, what would you use to automate this sort of test? I'm thinking an SSD connected by eSATA, with an external power supply, and the host running inside a VM. Drop power to the drive at the same time as doing a kill -9 on the VM, then you can resume the VM pretty quickly instead of waiting for a full reboot sequence.
Quoted 6 lines
I was just pulling power on sata drive.
It uncovered "interesting" stuff. I plugged power back, and kernel re-estabilished communication with that drive, but any settings with hdparm were forgotten. I'd say there's some room for improvement there...
Hi Pavel,
When you drop power, your drive normally loses temporary settings (like a change to write cache, etc).
Quoted 7 lines
Quoted 5 lines
Depending on the class of the device, there are ways to make that permanent (look at hdparm or sdparm for details).
This is a feature of the drive and its firmware, not something we reset in the device each time it re-appears.
Yes, and I'm arguing that is a bug (as in, < 0.01% people are using hdparm correctly).
Almost no end users use hdparm. Those who do should read the man page and add the -K flag :)
Or system scripts that tweak should invoke it with the right flags.....
Ric
Quoted 10 lines
So you used hparm to disable write cache so that ext3 can be safely used on your hdd. Now you have glitch on power. Then, system continues to operate in dangerous mode until reboot.
I guess it would be safer not to reattach drives after power fail... (also I wonder what this does to data integrity. Drive lost content of its writeback cache, but kernel continues... Journal will not prevent data corruption in this case).
Pavel
Hi Ric,
Am 23.11.2013 20:35, schrieb Ric Wheeler:
Quoted 57 lines · Stefan Priebe
Quoted 48 lines
Hi Ric,
Am 22.11.2013 21:37, schrieb Ric Wheeler:
Quoted 32 lines · Stefan Priebe
Quoted 21 lines
Hi Christoph,
Am 21.11.2013 11:11, schrieb Christoph Hellwig:Quoted 10 lines
Quoted 3 lines
2. Some drives may implement
CMD_FLUSHto return immediately i.e. no guarantee the data is actually on disk.In which case they aren't spec complicant. While I've seen countless data integrity bugs on lower end ATA SSDs I've not seen one that simpliy
ingnores flush. If you'd want to cheat that bluntly you'd be better of just claiming to not have a writeback cache.Quoted 8 lines
You solve your performance problem by completely disabling any chance of having data integrity guarantees, and do so in a way that is not detectable for applications or users.
If you have a workload with lots of small synchronous writes disabling the writeback cache on the disk does indeed often help, especially with
the non-queueable FLUSH on all but the most recent ATA devices.Quoted 4 lines
But this isn't correct for drives with capicitors like Crucial m500, Intel DC S3500, DC S3700 isn't it? Shouldn't the linux kernel has an option to disable this for drives like these?
/sys/block/sdX/device/ignore_flushIf you know 100% for sure that your drive has a non-volatile write cache, you can run the file system without the flushing by mounting "-o nobarrier". With most devices, this is not needed since they tend to simply ignore the flushes if they know they are power failure safe.
Quoted 8 lines
Block level, we did something similar for users who are not running through a file system for SCSI devices - James added support to echo "temporary" into the sd's device's
cache_typefield:See:
At least to me this does not work. I get the same awful speed as before - also the I/O waits stay the same. I'm still seeing CMD flushes going to the devices.
Quoted 7 lines
Is there any way to check whether the temporary got accepted and works?
I simply executed:
for i in/sys/class/scsi_disk/*/cache_type; do echo $i; echo temporary write back >$i; doneStefan
Quoted 8 lines
What kernel are you running? This is a new addition....
Also, you can "cat" the same file to see what it says.
Regards,
Ric
Is the output i sent to you fine? Anything wrong?
Stefan
slow sync performance on LSI / Broadcom MegaRaid performance with battery cache
In reply to Ric WheelerHello list,
while we used adaptec controller with battery cache for years we recently switched to dell hw using the perc controllers which are rebranded lsi/broadcom controllers.
We're running btrfs subvolume / snapshot workloads and while those are very fast on btrfs using a btrfs raid 0 on top of several raid 5 running on adaptec (battery backed up) in write back mode.
The performance really sucks on those LSI controllers even the one i have has 8GB cache instead of just 1GB at adaptec.
Especially sync / fsync are awfully slow taking sometimes 30-45 minutes while btrfs is doing snapshots. The workload on all machines is the same and the disks are ok.
Is there a way to disable FLUSH / sync at all for those devices? Just to test?
I'm already using nobarrier mount option on btrfs but this does not help either.
Thanks!
Greets,
Stefan