XFS speculative preallocation
PATCH v2 1/1 XFS speculative preallocation
Hi all,
This is v2 of the speculative preallocation FAQ bits. The initial proposal was here:
http://oss.sgi.com/archives/xfs/2014-03/msg00316.html
This version includes some updates based on review from arekm and dchinner. Most notably, the content has been broken down into a few more questions. Unless there are further major changes required, I'll plan to post something along these lines to the wiki when my account is approved. Thanks for the feedback!
Brian
---
Q: Why do files on XFS use more data blocks than expected?
A:
The XFS speculative preallocation algorithm allocates extra blocks beyond end of file (EOF) to minimise file fragmentation during buffered write workloads. Workloads that benefit from this behaviour include slowly growing files, concurrent writers and mixed reader/writer workloads. It also provides fragmentation resistence in situations where memory pressure prevents adequate buffering of dirty data to allow formation of large contiguous regions of data in memory.
This post-EOF block allocation is accounted identically to blocks within EOF. It is visible in 'st_blocks' counts via stat() system calls, accounted as globally allocated space and against quotas that apply to the associated file. The space is reported by various userspace utilities (stat, du, df, ls) and thus provides a common source of confusion for administrators. Post-EOF blocks are temporary in most situations and are usually reclaimed via several possible mechanisms in XFS.
See the FAQ entry on speculative preallocation for details.
Q: What is speculative preallocation?
A:
XFS speculatively preallocates post-EOF blocks on file extending writes in anticipation of future extending writes. The size of a preallocation is dynamic and depends on the runtime state of the file and fs. Generally speaking, preallocation is disabled for very small files and preallocation sizes grow as files grow larger.
Preallocations are capped to the maximum extent size supported by the filesystem. Preallocation size is throttled automatically as the filesystem approaches low free space conditions or other allocation limits on a file (such as a quota).
In most cases, speculative preallocation is automatically reclaimed when a file is closed. Preallocation may also persist beyond the lifecycle of the file descriptor. Certain application behaviors that are known to cause fragmentation, such as file server workloads, slowly growing files, etc., benefit from this and delay the removal of preallocated blocks beyond fd close.
Q: How can I speed up or avoid delayed removal of speculative preallocation?
A:
Remove the inode from the VFS cache or unmount the filesystem to remove speculative preallocations associated with an inode.
Linux 3.8 (and later) includes a scanner to perform background trimming of files with lingering post-EOF preallocations. The scanner bypasses dirty files to avoid interference with ongoing writes. A 5 minute scan interval is used by default and can be adjusted via the following file (value in seconds):
/proc/sys/fs/xfs/speculative_prealloc_lifetime
Q: Is speculative preallocation permanent?
A:
Although speculative preallocation can lead to reports of excess space usage, the preallocated space is not permanent unless explicitly made so via fallocate or a similar interface. Preallocated space can also be encoded permanently in situations where file size is extended beyond a range of post-EOF blocks (i.e., via truncate). Otherwise, preallocated blocks are reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides.
Q: My workload has known characteristics - can I tune speculative preallocation to an optimal fixed size?
A:
The 'allocsize=' mount option configures the XFS block allocation algorithm to use a fixed allocation size. Speculative preallocation is not dynamically resized when the allocsize mount option is set and thus the potential for fragmentation is increased. XFS historically set allocsize to 64k by default.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 15 lines · Brian Foster
Hi all,
This is v2 of the speculative preallocation FAQ bits. The initial proposal was here:
http://oss.sgi.com/archives/xfs/2014-03/msg00316.html
This version includes some updates based on review from arekm and dchinner. Most notably, the content has been broken down into a few more questions. Unless there are further major changes required, I'll plan to post something along these lines to the wiki when my account is approved. Thanks for the feedback!
Brian
Quoted 8 lines
---
Q: Why do files on XFS use more data blocks than expected?
A:
The XFS speculative preallocation algorithm allocates extra blocks beyond end of file (EOF) to minimise file fragmentation during buffered
s/minimise/minimize/
Quoted 3 lines
write workloads. Workloads that benefit from this behaviour include slowly growing files, concurrent writers and mixed reader/writer workloads. It also provides fragmentation resistence in situations where
s/resistence/resistance/
Quoted 11 lines
memory pressure prevents adequate buffering of dirty data to allow formation of large contiguous regions of data in memory.
This post-EOF block allocation is accounted identically to blocks within EOF. It is visible in '
st_blocks' counts viastat()system calls, accounted as globally allocated space and against quotas that apply to the associated file. The space is reported by various userspace utilities (stat, du, df, ls) and thus provides a common source of confusion for administrators. Post-EOF blocks are temporary in most situations and are usually reclaimed via several possible mechanisms in XFS.
"usually reclaimed" - is it ever "never" reclaimed, then?
Quoted 17 lines
See the FAQ entry on speculative preallocation for details.
Q: What is speculative preallocation?
A:
XFS speculatively preallocates post-EOF blocks on file extending writes in anticipation of future extending writes. The size of a preallocation is dynamic and depends on the runtime state of the file and fs. Generally speaking, preallocation is disabled for very small files and preallocation sizes grow as files grow larger.
Preallocations are capped to the maximum extent size supported by the filesystem. Preallocation size is throttled automatically as the filesystem approaches low free space conditions or other allocation limits on a file (such as a quota).
Quoted 6 lines
In most cases, speculative preallocation is automatically reclaimed when a file is closed. Preallocation may also persist beyond the lifecycle of the file descriptor. Certain application behaviors that are known to cause fragmentation, such as file server workloads, slowly growing files, etc., benefit from this and delay the removal of preallocated blocks beyond fd close.
this is a little handwavy. "It's reclaimed when it's closed, except when it's not?" Can we say something more informative here?
Quoted 7 lines
Q: How can I speed up or avoid delayed removal of speculative preallocation?
A:
Remove the inode from the VFS cache or unmount the filesystem to remove speculative preallocations associated with an inode.
How does a user remove an inode from the VFS cache? ;)
So far the answer to this question sounds like "no."
We can't remove a single inode; drop_caches is way too heavy weight, and unmount isn't really viable in most cases.
Quoted 10 lines
Linux 3.8 (and later) includes a scanner to perform background trimming of files with lingering post-EOF preallocations. The scanner bypasses dirty files to avoid interference with ongoing writes. A 5 minute scan interval is used by default and can be adjusted via the following file (value in seconds):
/proc/sys/fs/xfs/speculative_prealloc_lifetimeQ: Is speculative preallocation permanent?
Quoted 7 lines
A:
Although speculative preallocation can lead to reports of excess space usage, the preallocated space is not permanent unless explicitly made so via fallocate or a similar interface. Preallocated space can also be encoded permanently in situations where file size is extended beyond a range of post-EOF blocks (i.e., via truncate). Otherwise, preallocated
(maybe "an extending truncate")
Quoted 6 lines
blocks are reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides.
Q: My workload has known characteristics - can I tune speculative preallocation to an optimal fixed size?
Quoted 7 lines
A:
The 'allocsize=' mount option configures the XFS block allocation algorithm to use a fixed allocation size. Speculative preallocation is not dynamically resized when the allocsize mount option is set and thus the potential for fragmentation is increased. XFS historically set allocsize to 64k by default.
Thanks,
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 7 lines · Brian Foster
Q: How can I speed up or avoid delayed removal of speculative preallocation?
A:
Remove the inode from the VFS cache or unmount the filesystem to remove speculative preallocations associated with an inode.
"Remove all inodes from the VFS cache" + ? AFAIK there is no way to remove single inode from cache. Example would be nice, too.
Signature · 6 lines
Arkadiusz Miśkiewicz, arekm / maven.pl _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 18 lines · Eric Sandeen
Quoted 17 lines · Brian Foster
Hi all,
This is v2 of the speculative preallocation FAQ bits. The initial proposal was here:
http://oss.sgi.com/archives/xfs/2014-03/msg00316.html
This version includes some updates based on review from arekm and dchinner. Most notably, the content has been broken down into a few more questions. Unless there are further major changes required, I'll plan to post something along these lines to the wiki when my account is approved. Thanks for the feedback!
Brian
---
Quoted 9 lines
Quoted 6 lines
Q: Why do files on XFS use more data blocks than expected?
A:
The XFS speculative preallocation algorithm allocates extra blocks beyond end of file (EOF) to minimise file fragmentation during buffered
s/minimise/minimize/
Fixed.
Quoted 6 lines
Quoted 3 lines
write workloads. Workloads that benefit from this behaviour include slowly growing files, concurrent writers and mixed reader/writer workloads. It also provides fragmentation resistence in situations where
s/resistence/resistance/
Fixed.
Quoted 12 lines
Quoted 11 lines
memory pressure prevents adequate buffering of dirty data to allow formation of large contiguous regions of data in memory.
This post-EOF block allocation is accounted identically to blocks within EOF. It is visible in '
st_blocks' counts viastat()system calls, accounted as globally allocated space and against quotas that apply to the associated file. The space is reported by various userspace utilities (stat, du, df, ls) and thus provides a common source of confusion for administrators. Post-EOF blocks are temporary in most situations and are usually reclaimed via several possible mechanisms in XFS.
Quoted 2 lines
"usually reclaimed" - is it ever "never" reclaimed, then?
I worded it that way because of the several little corner cases that can turn preallocations permanent. E.g., the extending truncate case and IIRC, an fallocate on an inode means the space won't be trimmed either.
Quoted 24 lines
Quoted 17 lines
See the FAQ entry on speculative preallocation for details.
Q: What is speculative preallocation?
A:
XFS speculatively preallocates post-EOF blocks on file extending writes in anticipation of future extending writes. The size of a preallocation is dynamic and depends on the runtime state of the file and fs. Generally speaking, preallocation is disabled for very small files and preallocation sizes grow as files grow larger.
Preallocations are capped to the maximum extent size supported by the filesystem. Preallocation size is throttled automatically as the filesystem approaches low free space conditions or other allocation limits on a file (such as a quota).
Quoted 6 lines
In most cases, speculative preallocation is automatically reclaimed when a file is closed. Preallocation may also persist beyond the lifecycle of the file descriptor. Certain application behaviors that are known to cause fragmentation, such as file server workloads, slowly growing files, etc., benefit from this and delay the removal of preallocated blocks beyond fd close.
Quoted 3 lines
this is a little handwavy. "It's reclaimed when it's closed, except when it's not?" Can we say something more informative here?
This used to say:
"In most cases, speculative preallocation is automatically reclaimed when a file is closed. The preallocation may persist after file close if an open, write, close pattern is repeated on a file. In this scenario, post-EOF preallocation is trimmed once the inode is reclaimed from cache or the filesystem unmounted."
The point I want to get it across here is simply that the default case is to reclaim on close. The delayed reclaim scenario is the exception based on a heuristic. How about this?
"In most cases, speculative preallocation is automatically reclaimed when a file is closed. Applications that repeatedly trigger preallocation and reclaim cycles (e.g., this is common in file server or log file workloads) can cause fragmentation. Therefore, this pattern is detected and causes the preallocation to persist beyond the lifecycle of the file descriptor."
Quoted 8 lines
Quoted 7 lines
Q: How can I speed up or avoid delayed removal of speculative preallocation?
A:
Remove the inode from the VFS cache or unmount the filesystem to remove speculative preallocations associated with an inode.
Quoted 7 lines
How does a user remove an inode from the VFS cache? ;)
So far the answer to this question sounds like "no."
We can't remove a single inode;
drop_cachesis way too heavy weight, and unmount isn't really viable in most cases.
I guess there's a fine line between informing what mechanisms remove the preallocations and what is potentially recommending people take inappropriate actions to clear preallocated blocks. My initial intent was to simply inform that the traditional post-eof preallocation is not permanent (e.g. "don't worry, in the worst case this space is reclaimed on inode reclaim or umount"). Given that and this is a user FAQ, I'm sympathetic to nuking the "remove from cache" bit.
The answer to this question becomes "use the scanner" (as described below) and the bits about reclaim/umount remain referenced indirectly in the answer to the next question. Thoughts?
Quoted 12 lines
Quoted 12 lines
Linux 3.8 (and later) includes a scanner to perform background trimming of files with lingering post-EOF preallocations. The scanner bypasses dirty files to avoid interference with ongoing writes. A 5 minute scan interval is used by default and can be adjusted via the following file (value in seconds):
/proc/sys/fs/xfs/speculative_prealloc_lifetimeQ: Is speculative preallocation permanent?
A:
Quoted 8 lines
Quoted 5 lines
Although speculative preallocation can lead to reports of excess space usage, the preallocated space is not permanent unless explicitly made so via fallocate or a similar interface. Preallocated space can also be encoded permanently in situations where file size is extended beyond a range of post-EOF blocks (i.e., via truncate). Otherwise, preallocated
(maybe "an extending truncate")
Ok. Thanks for the feedback.
Brian
Quoted 14 lines
Quoted 6 lines
blocks are reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides.
Q: My workload has known characteristics - can I tune speculative preallocation to an optimal fixed size?
Quoted 7 lines
A:
The 'allocsize=' mount option configures the XFS block allocation algorithm to use a fixed allocation size. Speculative preallocation is not dynamically resized when the allocsize mount option is set and thus the potential for fragmentation is increased. XFS historically set allocsize to 64k by default.
Quoted 7 lines
Thanks,
-Eric_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 5 lines · Arkadiusz Miśkiewicz
Quoted 3 lines · Brian Foster
Q: How can I speed up or avoid delayed removal of speculative preallocation?
Quoted 8 lines
Quoted 4 lines
A:
Remove the inode from the VFS cache or unmount the filesystem to remove speculative preallocations associated with an inode.
"Remove all inodes from the VFS cache" + ? AFAIK there is no way to remove single inode from cache. Example would be nice, too.
Yeah, that's confusing. If people prefer this bit to stay (I prefer to nuke it, as noted in my previous reply), then I'll have to word it more accurately and cautiously. Thanks.
Brian
Quoted 7 lines
Signature · 6 lines
Arkadiusz Miśkiewicz, arekm / maven.pl _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 15 lines · Brian Foster
Hi all,
This is v2 of the speculative preallocation FAQ bits. The initial proposal was here:
http://oss.sgi.com/archives/xfs/2014-03/msg00316.html
This version includes some updates based on review from arekm and dchinner. Most notably, the content has been broken down into a few more questions. Unless there are further major changes required, I'll plan to post something along these lines to the wiki when my account is approved. Thanks for the feedback!
Brian
Quoted 8 lines
---
Q: Why do files on XFS use more data blocks than expected?
A:
The XFS speculative preallocation algorithm allocates extra blocks beyond end of file (EOF) to minimise file fragmentation during buffered
^^^ beyond here and then later adopt post-EOF phrasing.
...
Quoted 10 lines
See the FAQ entry on speculative preallocation for details.
Q: What is speculative preallocation?
A:
XFS speculatively preallocates post-EOF blocks on file extending writes in anticipation of future extending writes. The size of a preallocation is dynamic and depends on the runtime state of the file and fs. Generally speaking, preallocation is disabled for very small files and
vague what is very small? ^^^
...
Quoted 11 lines
Q: Is speculative preallocation permanent?
A:
Although speculative preallocation can lead to reports of excess space usage, the preallocated space is not permanent unless explicitly made so via fallocate or a similar interface. Preallocated space can also be encoded permanently in situations where file size is extended beyond a range of post-EOF blocks (i.e., via truncate). Otherwise, preallocated blocks are reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides.
Switch order?
Normally, preallocated
blocks are reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides. They can be explictly made permanent .
Quoted 10 lines
Q: My workload has known characteristics - can I tune speculative preallocation to an optimal fixed size?
A:
The 'allocsize=' mount option configures the XFS block allocation algorithm to use a fixed allocation size. Speculative preallocation is not dynamically resized when the allocsize mount option is set and thus the potential for fragmentation is increased. XFS historically set
sets the
Quoted 2 lines
allocsize to 64k by default.
Q: Can I disable S-P-A ?
-Mark.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 18 lines · Mark Tinguely
Quoted 17 lines · Brian Foster
Hi all,
This is v2 of the speculative preallocation FAQ bits. The initial proposal was here:
http://oss.sgi.com/archives/xfs/2014-03/msg00316.html
This version includes some updates based on review from arekm and dchinner. Most notably, the content has been broken down into a few more questions. Unless there are further major changes required, I'll plan to post something along these lines to the wiki when my account is approved. Thanks for the feedback!
Brian
---
Quoted 8 lines
Quoted 6 lines
Q: Why do files on XFS use more data blocks than expected?
A:
The XFS speculative preallocation algorithm allocates extra blocks beyond end of file (EOF) to minimise file fragmentation during buffered
^^^ beyond here and then later adopt post-EOF phrasing.
I think you're suggesting a broader terminology change, but I'm not quite following. Could you be specific about what "later" bits should change? What phrasing in particular..?
Quoted 6 lines
...
Quoted 4 lines
See the FAQ entry on speculative preallocation for details.
Q: What is speculative preallocation?
Quoted 8 lines
Quoted 6 lines
A:
XFS speculatively preallocates post-EOF blocks on file extending writes in anticipation of future extending writes. The size of a preallocation is dynamic and depends on the runtime state of the file and fs. Generally speaking, preallocation is disabled for very small files and
vague what is very small? ^^^
...
I originally pointed out 64k, but that and other heuristic details that are subject to change were purged in v2. I'm personally not against including something that indicates the default and the notion that it's subject to change. I don't feel too strongly about it either way. Thoughts appreciated.
Quoted 14 lines
Quoted 6 lines
Q: Is speculative preallocation permanent?
A:
Quoted 7 lines
Although speculative preallocation can lead to reports of excess space usage, the preallocated space is not permanent unless explicitly made so via fallocate or a similar interface. Preallocated space can also be encoded permanently in situations where file size is extended beyond a range of post-EOF blocks (i.e., via truncate). Otherwise, preallocated blocks are reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides.
Quoted 7 lines
Switch order?
Normally, preallocated
blocks are reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides. They can be explictly made permanent .
Thoughts on the following?
"Preallocated blocks are normally reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides. They can be explicitly made permanent via fallocate or a similar interface. They can be implicitly made permanent in situations where file size is extended beyond a range of post-EOF blocks (i.e., via an extending truncate)."
Quoted 11 lines
Quoted 10 lines
Q: My workload has known characteristics - can I tune speculative preallocation to an optimal fixed size?
A:
The 'allocsize=' mount option configures the XFS block allocation algorithm to use a fixed allocation size. Speculative preallocation is not dynamically resized when the allocsize mount option is set and thus the potential for fragmentation is increased. XFS historically set
Quoted 8 lines
sets the
Quoted 2 lines
allocsize to 64k by default.
Q: Can I disable S-P-A ?
A: No..? ;)
Are you proposing this with the similar intent to the previous Q (i.e., "what's the alternative to the default behavior?"), or with the notion that Dave pointed out how technically preallocation is not really "off?" Or something else? If the former, we could modify the question:
"My workload has known characteristics - can I disable speculative preallocation or tune it to an optimal fixed size?"
Or something along those lines. Would anybody object to also pointing out that 'allocsize=4k' (or allocsize=<blocksize>?) could be considered "speculative preallocation == off" from the user's perspective?
Thanks for the feedback.
Brian
Quoted 6 lines
-Mark.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 23 lines · Brian Foster
Quoted 22 lines · Mark Tinguely
Quoted 21 lines · Brian Foster
Hi all,
This is v2 of the speculative preallocation FAQ bits. The initial proposal was here:
http://oss.sgi.com/archives/xfs/2014-03/msg00316.html
This version includes some updates based on review from arekm and dchinner. Most notably, the content has been broken down into a few more questions. Unless there are further major changes required, I'll plan to post something along these lines to the wiki when my account is approved. Thanks for the feedback!
Brian
---
Q: Why do files on XFS use more data blocks than expected?
A:
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
The XFS speculative preallocation algorithm allocates extra blocks beyond end of file (EOF) to minimise file fragmentation during buffered
^^^ beyond here and then later adopt post-EOF phrasing.
I think you're suggesting a broader terminology change, but I'm not quite following. Could you be specific about what "later" bits should change? What phrasing in particular..?
You use "blocks beyond end of file (EOF)" here and then later use the terminology of "post-EOF" through the rest of the document. Just pointing out the change in terminology.
Quoted 13 lines
Quoted 13 lines
...
Quoted 10 lines
See the FAQ entry on speculative preallocation for details.
Q: What is speculative preallocation?
A:
XFS speculatively preallocates post-EOF blocks on file extending writes in anticipation of future extending writes. The size of a preallocation is dynamic and depends on the runtime state of the file and fs. Generally speaking, preallocation is disabled for very small files and
Quoted 8 lines
Quoted 2 lines
vague what is very small? ^^^
...I originally pointed out 64k, but that and other heuristic details that are subject to change were purged in v2. I'm personally not against including something that indicates the default and the notion that it's subject to change. I don't feel too strongly about it either way. Thoughts appreciated.
I think the details are good since everyone has a different idea on "very small". The FAQ can be changed with the code. You can expect the TOT FAQ to represent Linux 3.0-stable.
Quoted 22 lines
Quoted 14 lines
Quoted 6 lines
Q: Is speculative preallocation permanent?
A:
Quoted 7 lines
Although speculative preallocation can lead to reports of excess space usage, the preallocated space is not permanent unless explicitly made so via fallocate or a similar interface. Preallocated space can also be encoded permanently in situations where file size is extended beyond a range of post-EOF blocks (i.e., via truncate). Otherwise, preallocated blocks are reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides.
Quoted 7 lines
Switch order?
Normally, preallocated
blocks are reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides. They can be explictly made permanent .
Quoted 9 lines
Thoughts on the following?
"Preallocated blocks are normally reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides. They can be explicitly made permanent via fallocate or a similar interface. They can be implicitly made permanent in situations where file size is extended beyond a range of post-EOF blocks (i.e., via an extending truncate)."
Looks good to me.
Quoted 27 lines
Quoted 11 lines
Quoted 10 lines
Q: My workload has known characteristics - can I tune speculative preallocation to an optimal fixed size?
A:
The 'allocsize=' mount option configures the XFS block allocation algorithm to use a fixed allocation size. Speculative preallocation is not dynamically resized when the allocsize mount option is set and thus the potential for fragmentation is increased. XFS historically set
Quoted 8 lines
sets the
Quoted 2 lines
allocsize to 64k by default.
Q: Can I disable S-P-A ?
A: No..? ;)
Are you proposing this with the similar intent to the previous Q (i.e., "what's the alternative to the default behavior?"), or with the notion that Dave pointed out how technically preallocation is not really "off?" Or something else? If the former, we could modify the question:
Quoted 7 lines
"My workload has known characteristics - can I disable speculative preallocation or tune it to an optimal fixed size?"
Or something along those lines. Would anybody object to also pointing out that 'allocsize=4k' (or allocsize=<blocksize>?) could be considered "speculative preallocation == off" from the user's perspective?
That sounds good to me. If they know it is there, eventually someone will ask "can I turn it off?". I would be happy with the answer of "no, but it can be tuned" and don't tell them how to effectively turn it off.
Quoted 4 lines
Thanks for the feedback.
Brian
Thanks for the FAQ.
--Mark.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 12 lines · Brian Foster
Quoted 11 lines · Mark Tinguely
Quoted 10 lines · Brian Foster
See the FAQ entry on speculative preallocation for details.
Q: What is speculative preallocation?
A:
XFS speculatively preallocates post-EOF blocks on file extending writes in anticipation of future extending writes. The size of a preallocation is dynamic and depends on the runtime state of the file and fs. Generally speaking, preallocation is disabled for very small files and
Quoted 8 lines
Quoted 2 lines
vague what is very small? ^^^
...I originally pointed out 64k, but that and other heuristic details that are subject to change were purged in v2. I'm personally not against including something that indicates the default and the notion that it's subject to change. I don't feel too strongly about it either way. Thoughts appreciated.
As i said in the original - if we put specific values in here or describe the exact heuristics, then it will be wrong the moment we change the code. If someone wants to know the exact details on how it works, then they can look at the code for the kernel they are running, because the behaviour can (and does) change from kernel to kernel.
Quoted 20 lines
Quoted 12 lines
Quoted 11 lines
Q: Is speculative preallocation permanent?
A:
Although speculative preallocation can lead to reports of excess space usage, the preallocated space is not permanent unless explicitly made so via fallocate or a similar interface. Preallocated space can also be encoded permanently in situations where file size is extended beyond a range of post-EOF blocks (i.e., via truncate). Otherwise, preallocated blocks are reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides.
Quoted 7 lines
Switch order?
Normally, preallocated
blocks are reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides. They can be explictly made permanent .
Quoted 8 lines
Thoughts on the following?
"Preallocated blocks are normally reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides. They can be explicitly made permanent via fallocate or a similar interface. They can be implicitly made permanent in situations where file size is extended beyond a range of post-EOF blocks (i.e., via an extending truncate)."
Speculative prealloc may end up permanent on a crash, because the in-memory state used to track and reclaim the speculative preallocation is lost.
Quoted 12 lines
Quoted 12 lines
Quoted 9 lines
Q: My workload has known characteristics - can I tune speculative preallocation to an optimal fixed size?
A:
The 'allocsize=' mount option configures the XFS block allocation algorithm to use a fixed allocation size. Speculative preallocation is not dynamically resized when the allocsize mount option is set and thus the potential for fragmentation is increased. XFS historically set
sets the
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
allocsize to 64k by default.
Q: Can I disable S-P-A ?
A: No..? ;)
That's the correct answer ;)
Quoted 7 lines
Are you proposing this with the similar intent to the previous Q (i.e., "what's the alternative to the default behavior?"), or with the notion that Dave pointed out how technically preallocation is not really "off?" Or something else? If the former, we could modify the question:
"My workload has known characteristics - can I disable speculative preallocation or tune it to an optimal fixed size?"
Yup, I'd change the questions like that.
Quoted 5 lines
Or something along those lines. Would anybody object to also pointing out that 'allocsize=4k' (or allocsize=<blocksize>?) could be considered "speculative preallocation == off" from the user's perspective?
From a users perspective, fixed size preallocation with anything
less than allocsize=64k would be "off". That was the old default, and no user ever noticed that speculative prealloc was occurring except on large files when it failed to prevent fragmentation....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 17 lines · Mark Tinguely
Quoted 6 lines · Brian Foster
Quoted 5 lines · Mark Tinguely
Quoted 4 lines · Brian Foster
XFS speculatively preallocates post-EOF blocks on file extending writes in anticipation of future extending writes. The size of a preallocation is dynamic and depends on the runtime state of the file and fs. Generally speaking, preallocation is disabled for very small files and
Quoted 8 lines
Quoted 2 lines
vague what is very small? ^^^
...I originally pointed out 64k, but that and other heuristic details that are subject to change were purged in v2. I'm personally not against including something that indicates the default and the notion that it's subject to change. I don't feel too strongly about it either way. Thoughts appreciated.
Quoted 3 lines
I think the details are good since everyone has a different idea on "very small". The FAQ can be changed with the code. You can expect the TOT FAQ to represent Linux 3.0-stable.
What's that supposed to mean? The FAQ on the xfs.org website does not represent a specific release. It is supposed to contain the most up-to-date information we have about various topics. If there's something specific to a kernel version we need to mention, then that's explicitly stated in the FAQ entry....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 29 lines · Mark Tinguely
Quoted 28 lines · Brian Foster
Quoted 26 lines · Mark Tinguely
Quoted 15 lines · Brian Foster
Hi all,
This is v2 of the speculative preallocation FAQ bits. The initial proposal was here:
http://oss.sgi.com/archives/xfs/2014-03/msg00316.html
This version includes some updates based on review from arekm and dchinner. Most notably, the content has been broken down into a few more questions. Unless there are further major changes required, I'll plan to post something along these lines to the wiki when my account is approved. Thanks for the feedback!
Brian
Quoted 8 lines
---
Q: Why do files on XFS use more data blocks than expected?
A:
The XFS speculative preallocation algorithm allocates extra blocks beyond end of file (EOF) to minimise file fragmentation during buffered
^^^ beyond here and then later adopt post-EOF phrasing.
Quoted 8 lines
Quoted 3 lines
I think you're suggesting a broader terminology change, but I'm not quite following. Could you be specific about what "later" bits should change? What phrasing in particular..?
You use "blocks beyond end of file (EOF)" here and then later use the terminology of "post-EOF" through the rest of the document. Just pointing out the change in terminology.
Quoted 1 line
Ok. I was just trying to be more descriptive here, this being the initial question so to speak (i.e., spelling out "end of file"). The remainder uses the abbreviation introduced here.
Brian
Quoted 100 lines
Quoted 12 lines
Quoted 12 lines
...
Quoted 10 lines
See the FAQ entry on speculative preallocation for details.
Q: What is speculative preallocation?
A:
XFS speculatively preallocates post-EOF blocks on file extending writes in anticipation of future extending writes. The size of a preallocation is dynamic and depends on the runtime state of the file and fs. Generally speaking, preallocation is disabled for very small files and
Quoted 8 lines
Quoted 2 lines
vague what is very small? ^^^
...I originally pointed out 64k, but that and other heuristic details that are subject to change were purged in v2. I'm personally not against including something that indicates the default and the notion that it's subject to change. I don't feel too strongly about it either way. Thoughts appreciated.
I think the details are good since everyone has a different idea on "very small". The FAQ can be changed with the code. You can expect the TOT FAQ to represent Linux 3.0-stable.
Quoted 22 lines
Quoted 14 lines
Quoted 6 lines
Q: Is speculative preallocation permanent?
A:
Quoted 7 lines
Although speculative preallocation can lead to reports of excess space usage, the preallocated space is not permanent unless explicitly made so via fallocate or a similar interface. Preallocated space can also be encoded permanently in situations where file size is extended beyond a range of post-EOF blocks (i.e., via truncate). Otherwise, preallocated blocks are reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides.
Quoted 7 lines
Switch order?
Normally, preallocated
blocks are reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides. They can be explictly made permanent .Quoted 9 lines
Thoughts on the following?
"Preallocated blocks are normally reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides. They can be explicitly made permanent via fallocate or a similar interface. They can be implicitly made permanent in situations where file size is extended beyond a range of post-EOF blocks (i.e., via an extending truncate)."
Looks good to me.
Quoted 27 lines
Quoted 11 lines
Quoted 10 lines
Q: My workload has known characteristics - can I tune speculative preallocation to an optimal fixed size?
A:
The 'allocsize=' mount option configures the XFS block allocation algorithm to use a fixed allocation size. Speculative preallocation is not dynamically resized when the allocsize mount option is set and thus the potential for fragmentation is increased. XFS historically set
Quoted 8 lines
sets the
Quoted 2 lines
allocsize to 64k by default.
Q: Can I disable S-P-A ?
A: No..? ;)
Are you proposing this with the similar intent to the previous Q (i.e., "what's the alternative to the default behavior?"), or with the notion that Dave pointed out how technically preallocation is not really "off?" Or something else? If the former, we could modify the question:
Quoted 7 lines
"My workload has known characteristics - can I disable speculative preallocation or tune it to an optimal fixed size?"
Or something along those lines. Would anybody object to also pointing out that 'allocsize=4k' (or allocsize=<blocksize>?) could be considered "speculative preallocation == off" from the user's perspective?
That sounds good to me. If they know it is there, eventually someone will ask "can I turn it off?". I would be happy with the answer of "no, but it can be tuned" and don't tell them how to effectively turn it off.
Quoted 9 lines
Quoted 4 lines
Thanks for the feedback.
Brian
Thanks for the FAQ.
--Mark.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 13 lines · Brian Foster
Hi all,
This is v2 of the speculative preallocation FAQ bits. The initial proposal was here:
http://oss.sgi.com/archives/xfs/2014-03/msg00316.html
This version includes some updates based on review from arekm and dchinner. Most notably, the content has been broken down into a few more questions. Unless there are further major changes required, I'll plan to post something along these lines to the wiki when my account is approved. Thanks for the feedback!
Quoted 3 lines
Brian
---
I've updated the wiki with this content plus the feedback in this thread. The new FAQs are here:
http://xfs.org/index.php/XFS_FAQ#Q:_Why_do_files_on_XFS_use_more_data_blocks_than_expected.3F
http://xfs.org/index.php/XFS_FAQ#Q:_What_is_speculative_preallocation.3F
http://xfs.org/index.php/XFS_FAQ#Q:_How_can_I_speed_up_or_avoid_delayed_removal_of_speculative_preallocation.3F
http://xfs.org/index.php/XFS_FAQ#Q:_Is_speculative_preallocation_permanent.3F
http://xfs.org/index.php/XFS_FAQ#Q:_My_workload_has_known_characteristics_-_can_I_disable_speculative_preallocation_or_tune_it_to_an_optimal_fixed_size.3F
Thanks for all of the reviews and feedback. If there are any further suggestions... well, it's wiki! Feel free to modify it. ;)
Brian
Quoted 85 lines
Q: Why do files on XFS use more data blocks than expected?
A:
The XFS speculative preallocation algorithm allocates extra blocks beyond end of file (EOF) to minimise file fragmentation during buffered write workloads. Workloads that benefit from this behaviour include slowly growing files, concurrent writers and mixed reader/writer workloads. It also provides fragmentation resistence in situations where memory pressure prevents adequate buffering of dirty data to allow formation of large contiguous regions of data in memory.
This post-EOF block allocation is accounted identically to blocks within EOF. It is visible in '
st_blocks' counts viastat()system calls, accounted as globally allocated space and against quotas that apply to the associated file. The space is reported by various userspace utilities (stat, du, df, ls) and thus provides a common source of confusion for administrators. Post-EOF blocks are temporary in most situations and are usually reclaimed via several possible mechanisms in XFS.See the FAQ entry on speculative preallocation for details.
Q: What is speculative preallocation?
A:
XFS speculatively preallocates post-EOF blocks on file extending writes in anticipation of future extending writes. The size of a preallocation is dynamic and depends on the runtime state of the file and fs. Generally speaking, preallocation is disabled for very small files and preallocation sizes grow as files grow larger.
Preallocations are capped to the maximum extent size supported by the filesystem. Preallocation size is throttled automatically as the filesystem approaches low free space conditions or other allocation limits on a file (such as a quota).
In most cases, speculative preallocation is automatically reclaimed when a file is closed. Preallocation may also persist beyond the lifecycle of the file descriptor. Certain application behaviors that are known to cause fragmentation, such as file server workloads, slowly growing files, etc., benefit from this and delay the removal of preallocated blocks beyond fd close.
Q: How can I speed up or avoid delayed removal of speculative preallocation?
A:
Remove the inode from the VFS cache or unmount the filesystem to remove speculative preallocations associated with an inode.
Linux 3.8 (and later) includes a scanner to perform background trimming of files with lingering post-EOF preallocations. The scanner bypasses dirty files to avoid interference with ongoing writes. A 5 minute scan interval is used by default and can be adjusted via the following file (value in seconds):
/proc/sys/fs/xfs/speculative_prealloc_lifetimeQ: Is speculative preallocation permanent?
A:
Although speculative preallocation can lead to reports of excess space usage, the preallocated space is not permanent unless explicitly made so via fallocate or a similar interface. Preallocated space can also be encoded permanently in situations where file size is extended beyond a range of post-EOF blocks (i.e., via truncate). Otherwise, preallocated blocks are reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides.
Q: My workload has known characteristics - can I tune speculative preallocation to an optimal fixed size?
A:
The 'allocsize=' mount option configures the XFS block allocation algorithm to use a fixed allocation size. Speculative preallocation is not dynamically resized when the allocsize mount option is set and thus the potential for fragmentation is increased. XFS historically set allocsize to 64k by default.
Quoted 4 lines
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs