default mount options
Is it possible for the 'mount' man page to be enhanced to show what the defaults are? Or if that's not possible, maybe the xfs(5) manpage?
Also, I'm again "unclear" on barriers.
The xfs(5) man page says:
"barrier|nobarrier - Enables/disables the use of block layer write barriers... this allows for drive level write caching to be enabled. Barriers are enabled by default.
This seems to say that barriers are enabled. Does that mean the the barriers are implemented in the HW of the disk, or that SW adds "barriers" for disks that don't have them implemented in HW?
It also says drives may enable write-caching -- but this should only be done if they support write barriers. How is this "decided"? I.e is it done "automatically" in HW? in SW? Or should the user "know"?
Is this related to whether or not the drives support "state" over power interruptions? By having non-volatile "write-cache" memory, battery-backed cache, or backed by a UPS? Wouldn't SSD's be considers safe for this purpose (because their state is non-volatile?).
I seem to "get" this topic periodically, but after some time passes, and upon rereading the associated manpages, I realize I'm not real clear which way is what and realize the lack of defaults being specified and whether or not SSD's and/or UPS-backed disks were safe whether barriers were present or not was still vague.
Thanks!
-linda
Quoted 4 lines · L.A. Walsh
Is it possible for the 'mount' man page to be enhanced to show what the defaults are? Or if that's not possible, maybe the xfs(5) manpage?
xfs(5) covers xfs mount options now. AFAIK, defaults are clearly stated. Is something missing?
i.e. -
"Barriers are enabled by default."
"For this reason, nodiscard is the default."
"For kernel v3.7 and later, inode64 is the default."
"noikeep is the default."
etc.
If there's something missing, please let us know (or send a patch).
Quoted 11 lines
Also, I'm again "unclear" on barriers.
The xfs(5) man page says:
"barrier|nobarrier - Enables/disables the use of block layer write barriers... this allows for drive level write caching to be enabled. Barriers are enabled by default.
This seems to say that barriers are enabled. Does that mean the the barriers are implemented in the HW of the disk, or that SW adds "barriers" for disks that don't have them implemented in HW?
It means that the xfs "barrier" mount option is enabled by default. ;)
There is no "barrier implemented in hardware" - having barriers turned on means that XFS will requeseet block device flushes at appropriate times to maintain consistency, even with write caching.
Quoted 3 lines
It also says drives may enable write-caching -- but this should only be done if they support write barriers. How is this "decided"? I.e is it done "automatically" in HW? in SW? Or should the user "know"?
What this means is that if barriers are enabled, write-caching on the drive can be safely enabled.
The user should leave barriers on. Devices which don't need them should ignore them.
Simplified, if turning barriers /off/ made your workload go faster, that means you should have left them on in the first place. If it didn't, then there was no reason to turn the knob in the first place...
Quoted 4 lines
Is this related to whether or not the drives support "state" over power interruptions? By having non-volatile "write-cache" memory, battery-backed cache, or backed by a UPS? Wouldn't SSD's be considers safe for this purpose (because their state is non-volatile?).
I'm not sure there is any universal answer for what SSDs may do on a power loss, but I think it's certainly common for them to have a volatile write cache as well.
Quoted 5 lines
I seem to "get" this topic periodically, but after some time passes, and upon rereading the associated manpages, I realize I'm not real clear which way is what and realize the lack of defaults being specified and whether or not SSD's and/or UPS-backed disks were safe whether barriers were present or not was still vague.
Just leave the option at the default, and you'll be fine. There is rarely, if ever, a reason to change it.
-Eric
Quoted 3 lines
Thanks!
-linda
Quoted 5 lines · Eric Sandeen
Quoted 3 lines · L.A. Walsh
Is it possible for the 'mount' man page to be enhanced to show what the defaults are? Or if that's not possible, maybe the xfs(5) manpage?
Quoted 8 lines
xfs(5) covers xfs mount options now. AFAIK, defaults are clearly stated. Is something missing?
i.e. -
"Barriers are enabled by default."
"For this reason, nodiscard is the default."
"For kernel v3.7 and later, inode64 is the default." "noikeep is the default."
etc.
----
Most of the text is the same as on the manpage, with small
insertions that weren't obvious at the time I sent the email, however logbsize -- default for v2 logs is MAX(32768, log_sunit). What does 'log' mean? As in the arithmetic function?
"noalign" is only relevant to filesystems created with
non-zero data alignment parms by mkfs. Does it apply if the container that xfs is in is not zero aligned? If the partitions weren't created on boundaries or the "volumes" on top of the partitions weren't created on boundaries, how would one specify the overall file system alignment -- especially when, say, lvm's on-disk allocs at he beginning of a volume may not be a multiple of a strip-size (had 768K w/a 3-stripe, 4-data disk RAID5 (RAID50).
"noquota" to force off all quota. So I don't specify
any quotas, is that the same as "noquota" -- does that mean it is the default?
It seems one can figure things out if one makes certain
assumptions, but that makes me uneasy.
Quoted 4 lines
If there's something missing, please let us know (or send a patch).
Quoted 1 line
Also, I'm again "unclear" on barriers.
It means that the xfs "barrier" mount option is enabled by default. ;)
---
Then why doesn't it say "the barrier option" telling
xfs to add barriers, is the default".
Quoted 3 lines
There is no "barrier implemented in hardware" - having barriers turned on means that XFS will requeseet block device flushes at appropriate times to maintain consistency, even with write caching.
---
"requeseet"? Is that "request"? (Seriously, I had
to google it to be sure).
Quoted 10 lines
Quoted 4 lines
It also says drives may enable write-caching -- but this should only be done if they support write barriers. How is this "decided"? I.e is it done "automatically" in HW? in SW? Or should the user "know"?
What this means is that if barriers are enabled, write-caching on the drive can be safely enabled.
The user should leave barriers on. Devices which don't need them should ignore them.
====
Not my experience. Devices with non-volatile cache
or UPS-backed cache, in the past, have been considered "not needing barriers". Has this changed? Why?
Quoted 4 lines
Simplified, if turning barriers /off/ made your workload go faster, that means you should have left them on in the first place. If it didn't, then there was no reason to turn the knob in the first place...
====
Not my experience. Devices with non-volatile cache
or UPS-backed cache, in the past, have been considered "not needing barriers". But those systems also, sometimes, change runtime behavior based on the UPS or battery state -- using write-back on a full-healthy battery, or write-through when it wouldn't be safe.
In that case, it seems nobarrier would be a better choice
for those volumes -- letting the controller decide.
Quoted 8 lines
Quoted 4 lines
Is this related to whether or not the drives support "state" over power interruptions? By having non-volatile "write-cache" memory, battery-backed cache, or backed by a UPS? Wouldn't SSD's be considers safe for this purpose (because their state is non-volatile?).
I'm not sure there is any universal answer for what SSDs may do on a power loss, but I think it's certainly common for them to have a volatile write cache as well.
---
I've yet to see one that does. Not saying they couldn't
exist, but just that I've yet to see one -- with the behavior being that if it accepts the write and returns, the data is on the SSD.
Quoted 2 lines
Just leave the option at the default, and you'll be fine. There is rarely, if ever, a reason to change it.
---
Fine isn't what I asked. I wanted to know if the switch
specified that xfs should add barriers or that barriers were already handled in the backing store for those file systems. If the prior then I would want nobarrier on some file systems, if the latter, I might want the default. But it sounds like the switch applies to the former -- meaning I don't want them for partitions that don't need them.
Quoted 16 lines · L A Walsh
Quoted 5 lines · Eric Sandeen
Quoted 3 lines · L.A. Walsh
Is it possible for the 'mount' man page to be enhanced to show what the defaults are? Or if that's not possible, maybe the xfs(5) manpage?
Quoted 8 lines
xfs(5) covers xfs mount options now. AFAIK, defaults are clearly stated. Is something missing?
i.e. -
"Barriers are enabled by default."
"For this reason, nodiscard is the default."
"For kernel v3.7 and later, inode64 is the default." "noikeep is the default."
etc.
Quoted 2 lines
----
Most of the text is the same as on the manpage,
Right, I was quoting from the manpage... (xfs(5)).
Quoted 4 lines
with small
insertions that weren't obvious at the time I sent the email, however logbsize -- default for v2 logs isMAX(32768, log_sunit). What does 'log' mean? As in the arithmetic function?
v2 logs are version 2 xfs logs - i.e. the metadata log. log_sunit is the stripe unit of the log. I can see that that's not super clear, as it's not used or defined anywhere else.
Quoted 8 lines
"noalign" is only relevant to filesystems created with
non-zero data alignment parms by mkfs. Does it apply if the container that xfs is in is not zero aligned? If the partitions weren't created on boundaries or the "volumes" on top of the partitions weren't created on boundaries, how would one specify the overall file system alignment -- especially when, say, lvm's on-disk allocs at he beginning of a volume may not be a multiple of a strip-size (had 768K w/a 3-stripe, 4-data disk RAID5 (RAID50).
alignment only applies /within/ the filesystem, it has no view outside of that. If you create an unaligned partition, there is no magical re-alignment that can happen within the filesystem.
Quoted 4 lines
"noquota" to force off all quota. So I don't specify
any quotas, is that the same as "noquota" -- does that mean it is
the default? It seems one can figure things out if one makes certain
assumptions, but that makes me uneasy.
I'm not actually sure why one would ever use "noquota."
Quoted 7 lines
Quoted 4 lines
If there's something missing, please let us know (or send a patch).
Quoted 1 line
Also, I'm again "unclear" on barriers.
It means that the xfs "barrier" mount option is enabled by default. ;)
---
Then why doesn't it say "the barrier option" telling
xfs to add barriers, is the default".
I guess we assumed that it could be inferred readily from the phrase "Barriers are enabled by default" in the barrier/nobarrier section.
Quoted 7 lines
Quoted 4 lines
There is no "barrier implemented in hardware" - having barriers turned on means that XFS will requeseet block device flushes at appropriate times to maintain consistency, even with write caching.
---
"requeseet"? Is that "request"? (Seriously, I had
to google it to be sure).
typo.
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
It also says drives may enable write-caching -- but this should only be done if they support write barriers. How is this "decided"? I.e is it done "automatically" in HW? in SW? Or should the user "know"?
What this means is that if barriers are enabled, write-caching on the drive can be safely enabled.
Quoted 6 lines
Quoted 2 lines
The user should leave barriers on. Devices which don't need them should ignore them.
====
Not my experience. Devices with non-volatile cache
or UPS-backed cache, in the past, have been considered "not needing barriers". Has this changed? Why?
Devices with ups-backed cache will in general ignore barrier requests. I didn't mean to say that they did need barriers, I meant to imply that such devices will /ignore/ barriers.
Quoted 8 lines
Quoted 4 lines
Simplified, if turning barriers /off/ made your workload go faster, that means you should have left them on in the first place. If it didn't, then there was no reason to turn the knob in the first place...
====
Not my experience. Devices with non-volatile cache
or UPS-backed cache, in the past, have been considered "not needing barriers".
This is true.
Quoted 6 lines
But those systems also, sometimes, change runtime
behavior based on the UPS or battery state -- using write-back on a full-healthy battery, or write-through when it wouldn't be safe.In that case, it seems nobarrier would be a better choice
for those volumes -- letting the controller decide.
No. Because then xfs will /never/ send barriers requests, even if the battery dies. So I think you have that backwards.
If you leave them at the default, i.e. barriers /enabled,/ then the device is free to ignore the barrier operations if the battery is healthy, or to honor them if it fails.
If you turn it off at mount time, xfs will /never/ send such requests, and the storage will be unsafe if the battery fails, and you will be at risk for corruption or data loss.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Is this related to whether or not the drives support "state" over power interruptions? By having non-volatile "write-cache" memory, battery-backed cache, or backed by a UPS? Wouldn't SSD's be considers safe for this purpose (because their state is non-volatile?).
Quoted 7 lines
Quoted 3 lines
I'm not sure there is any universal answer for what SSDs may do on a power loss, but I think it's certainly common for them to have a volatile write cache as well.
---
I've yet to see one that does. Not saying they couldn't exist, but just that I've yet to see one -- with the behavior
being that if it accepts the write and returns, the data is
on the SSD.
*shrug* I'm not going to tell anyone to turn off barriers for ssds. :)
Quoted 10 lines
Quoted 2 lines
Just leave the option at the default, and you'll be fine. There is rarely, if ever, a reason to change it.
---
Fine isn't what I asked. I wanted to know if the switch
specified that xfs should add barriers or that barriers were already handled in the backing store for those file systems. If the prior then I would want nobarrier on some file systems, if the latter, I might want the default. But it sounds like the switch applies to the former -- meaning I don't want them for partitions that don't need them.
"barrier" means "the xfs filesystem will send barrier requests to the storage." It does this at critical points during updates to ensure that data is /permanently/ stored on disk when required - for metadata consistency and/or for data permanence.
If the storage doesn't need barriers, they'll simply be ignored. "partitions that don't need them" should be /unaffected/ by their presence, so there's no use in turning them off.
Turning them off risks corruption.
-Eric
Quoted 10 lines · Eric Sandeen
Quoted 7 lines
But those systems also, sometimes, change runtime
behavior based on the UPS or battery state -- using write-back on a full-healthy battery, or write-through when it wouldn't be safe.In that case, it seems nobarrier would be a better choice
for those volumes -- letting the controller decide.No. Because then xfs will /never/ send barriers requests, even if the battery dies. So I think you have that backwards.
---
If the battery dies, then the controller shifts
to write-through and no longer uses its write cache. This is documented and observed behavior.
Quoted 9 lines
If you leave them at the default, i.e. barriers /enabled,/ then the device is free to ignore the barrier operations if the battery is healthy, or to honor them if it fails.
If you turn it off at mount time, xfs will /never/ send such requests, and the storage will be unsafe if the battery fails, and you will be at risk for corruption or data loss.
---
I know what the device does in regards to its battery.
I don't know that the device responds to xfs-drivers in a way that xfs will know to change barrier usage.
Quoted 11 lines
Quoted 10 lines
Quoted 2 lines
Just leave the option at the default, and you'll be fine. There is rarely, if ever, a reason to change it.
---
Fine isn't what I asked. I wanted to know if the switch
specified that xfs should add barriers or that barriers were already handled in the backing store for those file systems. If the prior then I would want nobarrier on some file systems, if the latter, I might want the default. But it sounds like the switch applies to the former -- meaning I don't want them for partitions that don't need them.
Quoted 6 lines
"barrier" means "the xfs filesystem will send barrier requests to the storage." It does this at critical points during updates to ensure that data is /permanently/ stored on disk when required - for metadata consistency and/or for data permanence.
If the storage doesn't need barriers, they'll simply be ignored.
---
How can that be determined? If xfs was able to determine
the need for barriers or not, then why can't it determine something as simple as disk alignment and ensure writes are on optimal boundaries?
Quoted 4 lines
"partitions that don't need them" should be /unaffected/ by their presence, so there's no use in turning them off.
Turning them off risks corruption.
---
The only corrupt devices I've had w/xfs were ones
that had them turned on. Those were > 5 years ago. That says to me that there are likely other risks that have had a greater possibility for causing corruption than that caused by lack or presence of barriers.
-l
Quoted 11 lines · L A Walsh
Quoted 8 lines · Eric Sandeen
Quoted 7 lines
But those systems also, sometimes, change runtime
behavior based on the UPS or battery state -- using write-back on a full-healthy battery, or write-through when it wouldn't be safe.In that case, it seems nobarrier would be a better choice
for those volumes -- letting the controller decide.
Quoted 6 lines
Quoted 2 lines
No. Because then xfs will /never/ send barriers requests, even if the battery dies. So I think you have that backwards.
---
If the battery dies, then the controller shifts
to write-through and no longer uses its write cache. This is documented and observed behavior.
Ok, right, sorry.
In that case barriers may not be /needed,/ but turning them off should offer no benefit.
Quoted 6 lines
Quoted 6 lines
If you leave them at the default, i.e. barriers /enabled,/ then the device is free to ignore the barrier operations if the battery is healthy, or to honor them if it fails.
Quoted 7 lines
Quoted 3 lines
If you turn it off at mount time, xfs will /never/ send such requests, and the storage will be unsafe if the battery fails, and you will be at risk for corruption or data loss.
---
I know what the device does in regards to its battery.
I don't know that the device responds to xfs-drivers in a way that xfs will know to change barrier usage.
xfs does not change its barrier usage. It is determined solely at mount time.
Quoted 16 lines
Quoted 16 lines
Quoted 10 lines
Quoted 3 lines
Just leave the option at the default, and you'll be fine. There is rarely, if ever, a reason to change it.
--- Fine isn't what I asked. I wanted to know if the switch
specified that xfs should add barriers or that barriers were already handled in the backing store for those file systems. If the prior then I would want nobarrier on some file systems, if the latter, I might want the default. But it sounds like the switch applies to the former -- meaning I don't want them for partitions that don't need them."barrier" means "the xfs filesystem will send barrier requests to the storage." It does this at critical points during updates to ensure that data is /permanently/ stored on disk when required - for metadata consistency and/or for data permanence.
Quoted 5 lines
Quoted 1 line
If the storage doesn't need barriers, they'll simply be ignored.
---
How can that be determined? If xfs was able to determine
the need for barriers or not, then why can't it determine something as simple as disk alignment and ensure writes are on optimal boundaries?
xfs does not determine the need for barriers or not, it is governed solely by the specified mount option.
If they are sent by xfs, the device can choose to ignore them or not.
(ignoring the alignment non-sequitor for now)
Quoted 10 lines
Quoted 4 lines
"partitions that don't need them" should be /unaffected/ by their presence, so there's no use in turning them off.
Turning them off risks corruption.
---
The only corrupt devices I've had w/xfs were ones
that had them turned on. Those were > 5 years ago. That says to me that there are likely other risks that have had a greater possibility for causing corruption than that caused by lack or presence of barriers.
*shrug*
You seem to really want to turn barriers off in some cases. I certainly can't /make/ you leave it at the safe-and-harmless "on" default. :)
-Eric
Quoted 8 lines · L A Walsh
Quoted 5 lines · Eric Sandeen
Quoted 5 lines
But those systems also, sometimes, change runtime behavior based on the UPS or battery state -- using write-back on a full-healthy battery, or write-through when it wouldn't be safe.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
In that case, it seems nobarrier would be a better choice
for those volumes -- letting the controller decide.No. Because then xfs will /never/ send barriers requests, even if the battery dies. So I think you have that backwards.
Let's just get somethign straight first - there is no "barrier" operation that is sent to the storage, and Linux does not have "barriers" anymore. What we now do is strictly order our IO at the filesystem level and issue cache flush requests to ensure all IO prior to the cache flush request is on stable storage. We also make use of FUA writes, which guarantee that a specific write hits stable storage before the filesystem is told that it is complete (FUA is emulated with post-IO cache flush requests on devices that don't support FUA).
This is why "barriers" no longer have a performance cost - we don't need to empty the IO pipeline to guarantee integrity anymore. And it should be clear why hardware that has non-volatile caches don't care whether "barriers" are enabled or not because all writes are FUA and cache flushes are no-ops.
IOWs, "barriers" are an outdated concept and we only still have it hanging around because we were stupid enough to name a mount option after an implementation, rather than the feature it provided.
Quoted 4 lines
---
If the battery dies, then the controller shifts
to write-through and no longer uses its write cache. This is documented and observed behavior.
For /some/ RAID controllers in /some/ modes. For example, the megaraid driver that has been ignoring cache flushes for over 9 years because in RAID mode it doesn't need it. However, in JBOD mode, that same controller requires cache flushes to be sent because it turns off sane cache management behaviour in JBOD mode:
ommit 1e793f6fc0db920400574211c48f9157a37e3945
Author: Kashyap Desai <[email protected]> Date: Fri Oct 21 06:33:32 2016 -0700
scsi: megaraid_sas: Fix data integrity failure for JBOD (passthrough) devices
Commit 02b01e010afe ("megaraid_sas: return sync cache call with success") modified the driver to successfully complete SYNCHRONIZE_CACHE commands without passing them to the controller. Disk drive caches are only explicitly managed by controller firmware when operating in RAID mode. So this commit effectively disabled writeback cache flushing for any drives used in JBOD mode, leading to data integrity failures.
This is a clear example of why "barriers" should always be on and cache flushes always passed through to the storage - because we just don't know WTF the storage is actually doing with it's caches.
Quite frankly, I think it's time we marked the "barrier/nobarrier" mount options as deprecated and simply always issue the required cache flushes.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 5 lines · Dave Chinner
For /some/ RAID controllers in /some/ modes. For example, the megaraid driver that has been ignoring cache flushes for over 9 years because in RAID mode it doesn't need it. However, in JBOD mode, that same controller requires cache flushes to be sent because it turns off sane cache management behaviour in JBOD mode...
---
Lovely. For better or worse, none of my HW-based LSI-raid cards have
been able to do JBOD.
do JBOD.
Quoted 3 lines
This is a clear example of why "barriers" should always be on and cache flushes always passed through to the storage - because we just don't know WTF the storage is actually doing with it's caches.
----
When it comes to JBOD, its not so clear about where caching
helps the most.
Related -- wondering about how external journals would
affect need for barriers. Haven't thought about it much, but it seems
like one would get alot of bang-for-the-buck to put journals on
SSD's. _If_ data written to SSD's becomes "safe" as soon as it is accepted by SSDs (not saying it *is*, but _if_ it is), then how "needed" is ordering of writes for rotating media -- apart from
special use/need cases like apps maintaining their own data-integrity
journals like DB's or such? Just wondering about how increasing use
of SSD's might affect the need for barriers.
Anyway, I have the illusion that I am mostly clear about
current params (at least until my next disillusioning)... ;-)
-l
Quoted 2 lines · L A Walsh
Related -- wondering about how external journals would affect need
for barriers.
XFS correctly orders the cache flushes for it's different devices to ensure integrity. e.g. it flushes the caches of the data device which contains the filesytsem metadata before it issues the the FUA write to the external log device.
And it does all the right cache flushes for both data and real-time devices on fsync to ensure data updates are stable, too.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]