fallocate vs ENOSPC
I was wondering about adding fallocate() to cp,
where one of the benefits would be immediate indication of ENOSPC. I'm now wondering though might fallocate() fail to allocate an extent with ENOSPC, but there could be fragmented space available to write()?
If the above was true, then perhaps we could assume that any file system returning ENOSPC from fallocate(), might provide accurate fstatvfs().f_b{free,avail} values for a subsequent check, though needing to do that seems hacky.
cheers,
Pádraig.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · P??draig Brady
I was wondering about adding
fallocate()to cp,
where one of the benefits would be immediate indication of ENOSPC. I'm now wondering though mightfallocate()fail to allocate an extent with ENOSPC, but there could be fragmented space available towrite()?
fallocate isn't guaranteed to allocate a single or even contiguous extents, it just allocate the given amount of space, and if the fs isn't too fragmented and the allocator not braindead it will be sufficiently contiguous. Also all Linux implementation may actually still fail a write later if extreme corner cases when btree splits or other metadata operations during unwritten extent conversions go over the space limit.
Quoted 6 lines · Christoph Hellwig
Quoted 4 lines · P??draig Brady
I was wondering about adding
fallocate()to cp,
where one of the benefits would be immediate indication of ENOSPC. I'm now wondering though mightfallocate()fail to allocate an extent with ENOSPC, but there could be fragmented space available towrite()?
Quoted 6 lines
fallocate isn't guaranteed to allocate a single or even contiguous extents, it just allocate the given amount of space, and if the fs isn't too fragmented and the allocator not braindead it will be sufficiently contiguous. Also all Linux implementation may actually still fail a write later if extreme corner cases when btree splits or other metadata operations during unwritten extent conversions go over the space limit.
While this is true, *usually* fallocate will allocate enough space, but as Cirstoph has said, you still have to check the error returns for the write(2) and close(2) system call, and deal appropriately with any errors.
The other reason to use fallocate is if you are copying a huge number of files, it's possible you'll get better block allocation layout, depending on the file system, and how insane the writeback code for a particular kernel version might be. (Some versions of the kernel had writeback algorithms that would write 4MB of one file, then 4MB for another file, then 4MB for yet another file, then 4MB of the first file, etc. --- and some file systems can deal with this kind of write pattern better than others.) Using fallocate if you know the size of the file up front won't hurt, and on some systems it might help.
- Ted
Quoted 19 lines · Ted Ts'o
Quoted 6 lines · Christoph Hellwig
Quoted 4 lines · P??draig Brady
I was wondering about adding
fallocate()to cp,
where one of the benefits would be immediate indication of ENOSPC. I'm now wondering though mightfallocate()fail to allocate an extent with ENOSPC, but there could be fragmented space available towrite()?Quoted 6 lines
fallocate isn't guaranteed to allocate a single or even contiguous extents, it just allocate the given amount of space, and if the fs isn't too fragmented and the allocator not braindead it will be sufficiently contiguous. Also all Linux implementation may actually still fail a write later if extreme corner cases when btree splits or other metadata operations during unwritten extent conversions go over the space limit.
While this is true, *usually* fallocate will allocate enough space, but as Cirstoph has said, you still have to check the error returns for the write(2) and close(2) system call, and deal appropriately with any errors.
Quoted 8 lines
The other reason to use fallocate is if you are copying a huge number of files, it's possible you'll get better block allocation layout, depending on the file system, and how insane the writeback code for a particular kernel version might be. (Some versions of the kernel had writeback algorithms that would write 4MB of one file, then 4MB for another file, then 4MB for yet another file, then 4MB of the first file, etc. --- and some file systems can deal with this kind of write pattern better than others.)
Right, but....
Quoted 2 lines
Using fallocate if you know the size of
the file up front won't hurt, and on some systems it might help.
... this is - as a generalisation - wrong. Up front fallocate() can and does hurt performance, even when you know the size of the file ahead of time.
Why? Because it defeats the primary, seek reducing writeback optimisation that filesystems have these days: delayed allocation. This has been mentioned before in previous threads where you've been considering adding fallocate to cp. e.g:
http://www.mail-archive.com/[email protected]/msg10819.html
fallocate() style (or non-delalloc, write syscall time) allocation leads to non-optimal file layouts and slower writeback because the location that blocks are allocated in no way matches the writeback pattern, hence causing an increase in seeks during writeback of large numbers of files.
Further, filesytsems that are alignment aware (e.g. XFS) will align every fallocate() based allocation, greatly fragmenting free space when used on small files and the filesystem is on a RAID array. However, in XFS, delayed allocation will actually pack the allocation across files tightly on disk, resulting in full stripe writes (even for sub-stripe unit/width files) during writeback.
Delayed allocation allows workloads such as cp to run as a bandwidth bound operation because allocation is optimised to cause sequential write IO, whereas up-front fallocate() causes it to run as an IOPS bound option because file layout does not match the writeback pattern. And on large, high performance RAID arrays, bandwidth capacity is much, much higher than IOPS capacity, so delayed allocation is going to be far faster and have less long term impact on the filesystem than using fallocate.
IOWs, use of fallocate() -by default- will speed filesystem aging because it removes the benefits delayed allocation has on both short and long term filesystem performance.
The three major Linux filesystems (XFS, BTRFS and ext4) use delayed allocation, and hence do not need fallocate() to be used by userspace utilities like cp, tar, etc. to avoid fragmentation. If a given filesystem is still prone to fragmentation of data extents when copying data via cp or tar, then that is a problem with the filesystem that needs to be fixed, not worked around in the userspace utilities in a manner that is detrimental to other filesystems that don't suffer from those problems...
Yes, fallocate can help reduce fragmentation and increase performance in some situations, so making it an -option- for people who know what they are doing is a good idea. However, it should not be made the default for all of the reasons above.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 43 lines · Dave Chinner
Quoted 19 lines · Ted Ts'o
Quoted 6 lines · Christoph Hellwig
Quoted 4 lines · P??draig Brady
I was wondering about adding
fallocate()to cp,
where one of the benefits would be immediate indication of ENOSPC. I'm now wondering though mightfallocate()fail to allocate an extent with ENOSPC, but there could be fragmented space available towrite()?Quoted 6 lines
fallocate isn't guaranteed to allocate a single or even contiguous extents, it just allocate the given amount of space, and if the fs isn't too fragmented and the allocator not braindead it will be sufficiently contiguous. Also all Linux implementation may actually still fail a write later if extreme corner cases when btree splits or other metadata operations during unwritten extent conversions go over the space limit.
While this is true, *usually* fallocate will allocate enough space, but as Cirstoph has said, you still have to check the error returns for the write(2) and close(2) system call, and deal appropriately with any errors.
Quoted 8 lines
The other reason to use fallocate is if you are copying a huge number of files, it's possible you'll get better block allocation layout, depending on the file system, and how insane the writeback code for a particular kernel version might be. (Some versions of the kernel had writeback algorithms that would write 4MB of one file, then 4MB for another file, then 4MB for yet another file, then 4MB of the first file, etc. --- and some file systems can deal with this kind of write pattern better than others.)
Right, but....
Quoted 2 lines
Using fallocate if you know the size of
the file up front won't hurt, and on some systems it might help.... this is - as a generalisation - wrong. Up front
fallocate()can and does hurt performance, even when you know the size of the file ahead of time.Why? Because it defeats the primary, seek reducing writeback optimisation that filesystems have these days: delayed allocation. This has been mentioned before in previous threads where you've been considering adding fallocate to cp. e.g:
Quoted 7 lines
http://www.mail-archive.com/[email protected]/msg10819.html
fallocate()style (or non-delalloc, write syscall time) allocation leads to non-optimal file layouts and slower writeback because the location that blocks are allocated in no way matches the writeback pattern, hence causing an increase in seeks during writeback of large numbers of files.
I'm interpreting the above to mean that,
in the presence of concurrent writes to multiple files, fallocate() may cause slower _writes_, due to bypassing the delalloc write scheduler.
Subsequent reads of the file should be no slower though, and perhaps faster, due to the greater likelihood of all the blocks for the file being contiguous.
Quoted 6 lines
Further, filesytsems that are alignment aware (e.g. XFS) will align every
fallocate()based allocation, greatly fragmenting free space when used on small files and the filesystem is on a RAID array. However, in XFS, delayed allocation will actually pack the allocation across files tightly on disk, resulting in full stripe writes (even for sub-stripe unit/width files) during writeback.
Interesting. So what are the typical alignments involved. If you had to, what would you choose as a default min file size to enable fallocate() for?
Quoted 8 lines
Delayed allocation allows workloads such as cp to run as a bandwidth bound operation because allocation is optimised to cause sequential write IO, whereas up-front
fallocate()causes it to run as an IOPS bound option because file layout does not match the writeback pattern. And on large, high performance RAID arrays, bandwidth capacity is much, much higher than IOPS capacity, so delayed allocation is going to be far faster and have less long term impact on the filesystem than using fallocate.
So the consequences are the same as those in the first paragraph? Though I don't understand the detrimental "long term impact" you mention.
Quoted 13 lines
IOWs, use of
fallocate()-by default- will speed filesystem aging because it removes the benefits delayed allocation has on both short and long term filesystem performance.The three major Linux filesystems (XFS, BTRFS and ext4) use delayed allocation, and hence do not need
fallocate()to be used by userspace utilities like cp, tar, etc. to avoid fragmentation. If a given filesystem is still prone to fragmentation of data extents when copying data via cp or tar, then that is a problem with the filesystem that needs to be fixed, not worked around in the userspace utilities in a manner that is detrimental to other filesystems that don't suffer from those problems...
Quoted 4 lines
Yes, fallocate can help reduce fragmentation and increase performance in some situations, so making it an -option- for people who know what they are doing is a good idea. However, it should not be made the default for all of the reasons above.
thanks for the excellent info,
Pádraig.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Dave Chinner
fallocate()style (or non-delalloc, write syscall time) allocation leads to non-optimal file layouts and slower writeback because the location that blocks are allocated in no way matches the writeback pattern, hence causing an increase in seeks during writeback of large numbers of files.
Quoted 6 lines
Further, filesytsems that are alignment aware (e.g. XFS) will align every
fallocate()based allocation, greatly fragmenting free space when used on small files and the filesystem is on a RAID array. However, in XFS, delayed allocation will actually pack the allocation across files tightly on disk, resulting in full stripe writes (even for sub-stripe unit/width files) during write back.
Well, the question is whether you're optimizing for writing the files,
or reading the files. In some cases, files are write once, read never
(well, almost never) --- i.e., the backup case. In other cases, the files are write once, read many --- i.e., when installing software.
In that case, optimizing for the file reading might mean that you want to make the files aligned on RAID stripes, although it will fragment free
space. It all depends on what you're optimizing for.
I didn't realize that XFS was not aligning to RAID stripes when doing
delayed allocation writes. I'm curious --- does it do this only when
there are multiple files outstanding for delayed allocation in an
allocation group? If someone does a singleton cp of a large file
without using fallocate, will XFS try to align the write?
Also, if we are going to use fallocate() as a way of implicitly signaling to the file system that the file should be optimized for reads, as opposed to the write, maybe we should explicitly document it as such in the fallocate(2) man page, so that application programmers understand that this is the semantics they should expect.
-- Ted
Quoted 52 lines · Pádraig Brady
Quoted 43 lines · Dave Chinner
Quoted 19 lines · Ted Ts'o
Quoted 6 lines · Christoph Hellwig
Quoted 4 lines · P??draig Brady
I was wondering about adding
fallocate()to cp,
where one of the benefits would be immediate indication of ENOSPC. I'm now wondering though mightfallocate()fail to allocate an extent with ENOSPC, but there could be fragmented space available towrite()?Quoted 6 lines
fallocate isn't guaranteed to allocate a single or even contiguous extents, it just allocate the given amount of space, and if the fs isn't too fragmented and the allocator not braindead it will be sufficiently contiguous. Also all Linux implementation may actually still fail a write later if extreme corner cases when btree splits or other metadata operations during unwritten extent conversions go over the space limit.
While this is true, *usually* fallocate will allocate enough space, but as Cirstoph has said, you still have to check the error returns for the write(2) and close(2) system call, and deal appropriately with any errors.
Quoted 8 lines
The other reason to use fallocate is if you are copying a huge number of files, it's possible you'll get better block allocation layout, depending on the file system, and how insane the writeback code for a particular kernel version might be. (Some versions of the kernel had writeback algorithms that would write 4MB of one file, then 4MB for another file, then 4MB for yet another file, then 4MB of the first file, etc. --- and some file systems can deal with this kind of write pattern better than others.)
Right, but....
Quoted 2 lines
Using fallocate if you know the size of
the file up front won't hurt, and on some systems it might help.... this is - as a generalisation - wrong. Up front
fallocate()can and does hurt performance, even when you know the size of the file ahead of time.Why? Because it defeats the primary, seek reducing writeback optimisation that filesystems have these days: delayed allocation. This has been mentioned before in previous threads where you've been considering adding fallocate to cp. e.g:
Quoted 7 lines
http://www.mail-archive.com/[email protected]/msg10819.html
fallocate()style (or non-delalloc, write syscall time) allocation leads to non-optimal file layouts and slower writeback because the location that blocks are allocated in no way matches the writeback pattern, hence causing an increase in seeks during writeback of large numbers of files.
Quoted 4 lines
I'm interpreting the above to mean that,
in the presence of concurrent writes to multiple files,fallocate()may cause slower _writes_, due to bypassing the delalloc write scheduler.
It's not even concurrent writes. A single process writing multiple files into cache serially does not necessarily result in writeback 30s later writing the data to disk in the same order.
Quoted 3 lines
Subsequent reads of the file should be no slower though, and perhaps faster, due to the greater likelihood of all the blocks for the file being contiguous.
If delayed allocation does it's job, the files will be contiguous and unfragmented and no slower to read.
Quoted 8 lines
Quoted 6 lines
Further, filesytsems that are alignment aware (e.g. XFS) will align every
fallocate()based allocation, greatly fragmenting free space when used on small files and the filesystem is on a RAID array. However, in XFS, delayed allocation will actually pack the allocation across files tightly on disk, resulting in full stripe writes (even for sub-stripe unit/width files) during writeback.Interesting. So what are the typical alignments involved.
Typical range of alignments can be anything from 16k through to 16MB or larger.
Consider this - a 1TB filesystem with a 1MB alignment unit (stripe unit in the case of XFS) doing 16k aligned allocation per file will run out of aligned allocation slots after ~1,000,000 files have been created. At that point, the largest contiguous free space in the filesytsem is now under 16MB. When you want to create that multi-GB file now, it's going to have lots of extents rather than 1-2 because the preallocation has spread the small file data all over the place.
If you used delayed allocation, the small file data will be packed close together without alignment, leaving large, multi-GB free space extents for the large file you then want to create....
Quoted 2 lines
If you had to, what would you choose as a default min file size to enable
fallocate()for?
I would not enable fallocate by default at all.
Quoted 11 lines
Quoted 8 lines
Delayed allocation allows workloads such as cp to run as a bandwidth bound operation because allocation is optimised to cause sequential write IO, whereas up-front
fallocate()causes it to run as an IOPS bound option because file layout does not match the writeback pattern. And on large, high performance RAID arrays, bandwidth capacity is much, much higher than IOPS capacity, so delayed allocation is going to be far faster and have less long term impact on the filesystem than using fallocate.So the consequences are the same as those in the first paragraph? Though I don't understand the detrimental "long term impact" you mention.
Free space fragmentation will result in severe degradation of performance as soon as all > alignment sized free spaces are partially consumed. Then fragmentation will result from any large allocation. i.e. small aligned preallocations accelerate filesystem aging effects by imcreasing free space fragmentation. This typically won't be noticed for months until the fragmentation starts causing noticable performance issues - at which point it will be difficult if not impossible to correct without a backup/mkfs/restore cycle....
Cheers,
Dave.
Signature · 6 lines
Dave Chinner [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 16 lines · Theodore Tso
Quoted 6 lines · Dave Chinner
fallocate()style (or non-delalloc, write syscall time) allocation leads to non-optimal file layouts and slower writeback because the location that blocks are allocated in no way matches the writeback pattern, hence causing an increase in seeks during writeback of large numbers of files.Quoted 6 lines
Further, filesytsems that are alignment aware (e.g. XFS) will align every
fallocate()based allocation, greatly fragmenting free space when used on small files and the filesystem is on a RAID array. However, in XFS, delayed allocation will actually pack the allocation across files tightly on disk, resulting in full stripe writes (even for sub-stripe unit/width files) during write back.
Quoted 4 lines
Well, the question is whether you're optimizing for writing the files,
or reading the files. In some cases, files are write once, read never
(well, almost never) --- i.e., the backup case. In other cases, the files are write once, read many --- i.e., when installing software.
Doesn't matter. If delayed allocation is doing it's job properly, then you'll get unfragemented files when they are written. delayed allocation is supposed to make up front preallocation of disk space -unnecessary- to prevent fragmentation. Using preallocation instead of dealyed allocation implies your dealyed allocation implementation is sub-optimal and needs to be fixed.
Indeed, there is no guarantee that preallocation will even lay the files out in a sane manner that will give you good read speeds across multiple files - it may place them so far apart that the seek penalty between files is worse than having a few fragments...
Quoted 4 lines
In that case, optimizing for the file reading might mean that you want to make the files aligned on RAID stripes, although it will
fragment free space. It all depends on what you're optimizing
for.
If you want to optimise for read speed - especially for small files or random IO patterns - you want to *avoid* alignment to RAID stripes. Doing so overloads the first disk in the RAID stripe because all small file reads (and writes) hit that disk/LUN in the stripe. Indeed, if you have RAID5/6 and lots of small files, it is recommended that you turn off filesystem alignment at mkfs time for XFS.
SGI hit this problem back in the early 90s, and is one of the reasons that XFS lays it's metadata out such that it does not hot-spot one drive in a RAID stripe trying to read/write frequently accessed metadata (e.g. AG headers).
Quoted 2 lines
I didn't realize that XFS was not aligning to RAID stripes when doing delayed allocation writes.
It certainly does do alignment during delayed allocation.
/me waits for the "but you said..."
That's because XFS does -selective- alignment during delayed allocation.... :)
What people seem to forget about delayed allocation is that when delayed allocation occurs, we have lots of information about the data being written that is not available in the fallocate() context
- how big the delalloc extent is, how large the file currently is, how much more data needs to be written, whether the file is still growing, etc, and so delayed allocation can make a much more informed decision about how to allocate the data extents compared to fallocate().
For example, if the allocation is for offset zero of the file, the filesystem is using aligned allocation and the file size is larger than the stripe unit, the allocation will be stripe unit aligned.
Hence, if you've got lots of small files, they get packed because aligned allocation is not triggered and each allocation gets peeled from the front edge of the same free space extent.
If you've got large files, then they get aligned, leaving space between them for the fiel to potentially grow and fill full stripe units and widths.
And if you've got really large files still being written to, they get aligned and over-allocated thanks to the speculative prealloc beyond EOF, which effectively prevents fragmentation of large files due to interleaving allocations between files when many files are being written concurrently by writeback.....
Quoted 3 lines
I'm curious --- does it do this only when
there are multiple files outstanding for delayed allocation in an allocation group?
Irrelevant - the consideration is solely to do with the state of the current inode the allocation is being done for. If you're only writing a single file, then it doesn't matter for perfromance whether it is aligned or not. But it will matter for a freespace management POV, and hence how the filesytem ages.
Quoted 2 lines
If someone does a singleton cp of a large file without using fallocate, will XFS try to align the write?
The above should hopefully answer that question, especially with respect to why delayed allocation should not be short-circuited by using fallocate by default in generic system utilities.
Quoted 5 lines
Also, if we are going to use
fallocate()as a way of implicitly signaling to the file system that the file should be optimized for reads, as opposed to the write, maybe we should explicitly document it as such in the fallocate(2) man page, so that application programmers understand that this is the semantics they should expect.
Preallocation is for preventing fragmentation that leads to performance problems. Use of fallocate() does not imply the file layout has been optimised for read access and, IMO, never should.
Quite frankly, if system utilities like cp and tar start to abuse fallocate() by default so they can get "upfront ENOSPC detection", then I will seriously consider making XFS use delayed allocation for fallocate rather than unwritten extents so we don't lose the past 15 years worth of IO and aging optimisations that delayed allocation provides us with....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 6 lines · Dave Chinner
Quite frankly, if system utilities like cp and tar start to abuse
fallocate()by default so they can get "upfront ENOSPC detection", then I will seriously consider making XFS use delayed allocation for fallocate rather than unwritten extents so we don't lose the past 15 years worth of IO and aging optimisations that delayed allocation provides us with....
For the record I was considering fallocate() for these reasons.
1. Improved file layout for subsequent access
2. Immediate indication of ENOSPC
3. Efficient writing of NUL portions
You lucidly detailed issues with 1. which I suppose could be somewhat mitigated by not fallocating < say 1MB, though I suppose file systems could be smarter here and not preallocate small chunks (or when otherwise not appropriate). We can already get ENOSPC from a write() after an fallocate() in certain edge cases, so it would probably make sense to expand those cases.
cheers,
Pádraig.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 22 lines
-----Original Message-----
From: [email protected] [mailto:linux-fsdevel- [email protected]] On Behalf Of Pádraig Brady
Sent: Monday, November 28, 2011 4:55 PM
To: Dave Chinner
Cc: Theodore Tso; Christoph Hellwig; [email protected]
Subject: Re: fallocate vs ENOSPCQuoted 6 lines · Dave Chinner
Quite frankly, if system utilities like cp and tar start to abuse
fallocate()by default so they can get "upfront ENOSPC detection", then I will seriously consider making XFS use delayed allocation for fallocate rather than unwritten extents so we don't lose the past 15 years worth of IO and aging optimisations that delayed allocation provides us with....For the record I was considering
fallocate()for these reasons.1. Improved file layout for subsequent access
2. Immediate indication of ENOSPC
3. Efficient writing of NUL portions
Quoted 7 lines
You lucidly detailed issues with 1. which I suppose could be somewhat mitigated by not fallocating < say 1MB, though I suppose file systems could be smarter here and not preallocate small chunks (or when otherwise not appropriate). We can already get ENOSPC from a
write()after anfallocate()in certain edge cases, so it would probably make sense to expand those cases.
Just out of curiosity, how is it going to work with sparse files? By default, cp uses --sparse=auto. And for sparse files, it avoids some disk allocation automatically. With fallocate(), do you plan to change the semantics?
Cheers,
Tao
Quoted 6 lines
cheers,
Pádraig.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line · On 11/28/2011 10:41 AM, [email protected] wrote:
Just out of curiosity, how is it going to work with sparse files? By default, cp uses --sparse=auto. And for sparse files, it avoids some disk allocation automatically. With
fallocate(), do you plan to change the semantics?
With sparse files, coreutils currently uses fiemap (with a sync), and thus has full details of what is sparse or empty etc. So as detailed¹ on the coreutils list, the conversions would be:
--sparse=auto => 'Empty' -> 'Empty'
--sparse=always => 'Empty' -> 'Hole'
--sparse=never => 'Hole' -> 'Empty'cheers,
Pádraig.
¹ http://debbugs.gnu.org/9500#14
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Pádraig Brady
You lucidly detailed issues with 1. which I suppose could be somewhat mitigated by not fallocating < say 1MB, though I suppose file systems could be smarter here and not preallocate small chunks (or when otherwise not appropriate). We can already get ENOSPC from a
write()after anfallocate()in certain edge cases, so it would probably make sense to expand those cases.
I'm curious -- why are you so worried about ENOSPC?
You need to check the error returns on write(2) anyway (and it's good programming practice anyways --- don't forget to check on close because some network file systems only push to the network on close, and in some cases they might only get quota errors on the close), so I don't see why using fallocate() to get an early ENOSPC is so interesting for you.
-- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Theodore Tso
Quoted 7 lines · Pádraig Brady
You lucidly detailed issues with 1. which I suppose could be somewhat mitigated by not fallocating < say 1MB, though I suppose file systems could be smarter here and not preallocate small chunks (or when otherwise not appropriate). We can already get ENOSPC from a
write()after anfallocate()in certain edge cases, so it would probably make sense to expand those cases.
Quoted 7 lines
I'm curious -- why are you so worried about ENOSPC?
You need to check the error returns on write(2) anyway (and it's good programming practice anyways --- don't forget to check on close because some network file systems only push to the network on close, and in some cases they might only get quota errors on the close), so I don't see why using
fallocate()to get an early ENOSPC is so interesting for you.
It would be better to indicate ENOSPC _before_ copying a (potentially large) file to a (potentially slow) device. If the implementation complexity and side effects of doing this are sufficiently small, then it's worth doing. These discussions are to quantify the side effects.
cheers,
Pádraig.
p.s. You can be sure that cp deals with errors from write() and close(). --
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Theodore Tso
Quoted 7 lines · Pádraig Brady
You lucidly detailed issues with 1. which I suppose could be somewhat mitigated by not fallocating < say 1MB, though I suppose file systems could be smarter here and not preallocate small chunks (or when otherwise not appropriate). We can already get ENOSPC from a
write()after anfallocate()in certain edge cases, so it would probably make sense to expand those cases.
Quoted 7 lines
I'm curious -- why are you so worried about ENOSPC?
You need to check the error returns on write(2) anyway (and it's good programming practice anyways --- don't forget to check on close because some network file systems only push to the network on close, and in some cases they might only get quota errors on the close), so I don't see why using
fallocate()to get an early ENOSPC is so interesting for you.
Unfortunately for Samba, Windows clients will *only* report ENOSPC to the userspace apps if the initial fallocation fails. Most of the Windows apps don't bother to check for write() fails after the initial allocation succeeds.
We check for and report them back to the Windows client anyway of course, but most Windows apps just silently corrupt their data in this case.
That's why we use fallocate() in Samba :-(.
Jeremy.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · Pádraig Brady
It would be better to indicate ENOSPC _before_ copying a (potentially large) file to a (potentially slow) device. If the implementation complexity and side effects of doing this are sufficiently small, then it's worth doing. These discussions are to quantify the side effects.
In that case, why not use statfs(2) as a first class approximation? You won't know for user how much fs metadata will be required, but for the common case where someone trying to fit 10 pounds of horse manure in a 5 pound bag, that can be caught very readily without needing to use fallocate(2).
Regards,
- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Ted Ts'o
Quoted 4 lines · Pádraig Brady
It would be better to indicate ENOSPC _before_ copying a (potentially large) file to a (potentially slow) device. If the implementation complexity and side effects of doing this are sufficiently small, then it's worth doing. These discussions are to quantify the side effects.
In that case, why not use statfs(2) as a first class approximation? You won't know for user how much fs metadata will be required, but for the common case where someone trying to fit 10 pounds of horse manure in a 5 pound bag, that can be caught very readily without needing to use fallocate(2).
Yeah, we do that too, if the fallocate call fails.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Pádraig Brady
Quoted 6 lines · Dave Chinner
Quite frankly, if system utilities like cp and tar start to abuse
fallocate()by default so they can get "upfront ENOSPC detection", then I will seriously consider making XFS use delayed allocation for fallocate rather than unwritten extents so we don't lose the past 15 years worth of IO and aging optimisations that delayed allocation provides us with....For the record I was considering
fallocate()for these reasons.
Quoted 8 lines
1. Improved file layout for subsequent access
2. Immediate indication of ENOSPC
3. Efficient writing of NUL portionsYou lucidly detailed issues with 1. which I suppose could be somewhat mitigated by not fallocating < say 1MB, though I suppose file systems could be smarter here and not preallocate small chunks (or when otherwise not appropriate).
When you consider that some high end filesystem deployments have alignment characteristics over 50MB (e.g. so each uncompressed 4k resolution video frame is located on a different set of non-overlapping disks), arbitrary "don't fallocate below this amount" heuristics will always have unforseen failure cases...
In short: leave optimising general allocation strategies to the filesytems and their developers - there is no One True Solution for optimal file layout in a given filesystem, let alone across different filesytems. In fact, I don't even want to think about the mess fallocate() on everything would make of btrfs because of it's COW structure - it seems to me to guarantee worse fragmentation than using delayed allocation...
Quoted 3 lines
We can already get ENOSPC from a
write()
after anfallocate()in certain edge cases, so it would probably make sense to expand those cases.
fallocate is for preallocation, not for ENOSPC detection. If you want efficient and effective ENOSPC detection before writing anything, then you really want a space -reservation- extension to fallocate. Filesystems that use delayed allocation already have a space reservation subsystem - it how they account for space that is reserved by delayed allocation prior to the real allocation being done. IMO, allowing userspace some level of access to those reservations would be more appropriate for early detection of ENOSPC than using preallocation for everything...
As to efficient writing of NULL ranges - that's what sparse files are for - you do not need to write or even preallocate NULL ranges when copying files. Indeed, the most efficient way of dealing with NULL ranges is to punch a hole and let the filesystem deal with it.....
Cheers,
Dave.
Signature · 6 lines
Dave Chinner [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 25 lines · Jeremy Allison
Quoted 11 lines · Theodore Tso
Quoted 7 lines · Pádraig Brady
You lucidly detailed issues with 1. which I suppose could be somewhat mitigated by not fallocating < say 1MB, though I suppose file systems could be smarter here and not preallocate small chunks (or when otherwise not appropriate). We can already get ENOSPC from a
write()after anfallocate()in certain edge cases, so it would probably make sense to expand those cases.Quoted 7 lines
I'm curious -- why are you so worried about ENOSPC?
You need to check the error returns on write(2) anyway (and it's good programming practice anyways --- don't forget to check on close because some network file systems only push to the network on close, and in some cases they might only get quota errors on the close), so I don't see why using
fallocate()to get an early ENOSPC is so interesting for you.Unfortunately for Samba, Windows clients will *only* report ENOSPC to the userspace apps if the initial fallocation fails. Most of the Windows apps don't bother to check for
write()fails after the initial allocation succeeds.
Quoted 5 lines
We check for and report them back to the Windows client anyway of course, but most Windows apps just silently corrupt their data in this case.
That's why we use
fallocate()in Samba :-(.
IOWs, what you really want is a space reservation mechanism. You've only got this preallocate hammer, so you use it, yes?
Cheers,
Dave.
Signature · 6 lines
Dave Chinner [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 26 lines · Dave Chinner
Quoted 25 lines · Jeremy Allison
Quoted 11 lines · Theodore Tso
Quoted 7 lines · Pádraig Brady
You lucidly detailed issues with 1. which I suppose could be somewhat mitigated by not fallocating < say 1MB, though I suppose file systems could be smarter here and not preallocate small chunks (or when otherwise not appropriate). We can already get ENOSPC from a
write()after anfallocate()in certain edge cases, so it would probably make sense to expand those cases.Quoted 7 lines
I'm curious -- why are you so worried about ENOSPC?
You need to check the error returns on write(2) anyway (and it's good programming practice anyways --- don't forget to check on close because some network file systems only push to the network on close, and in some cases they might only get quota errors on the close), so I don't see why using
fallocate()to get an early ENOSPC is so interesting for you.Unfortunately for Samba, Windows clients will *only* report ENOSPC to the userspace apps if the initial fallocation fails. Most of the Windows apps don't bother to check for
write()fails after the initial allocation succeeds.
Quoted 8 lines
Quoted 5 lines
We check for and report them back to the Windows client anyway of course, but most Windows apps just silently corrupt their data in this case.
That's why we use
fallocate()in Samba :-(.IOWs, what you really want is a space reservation mechanism. You've only got this preallocate hammer, so you use it, yes?
Yes, absolutely. We're just trying to provide the Windows semantics the clients expect.
Jeremy.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 20 lines · Dave Chinner
Quoted 10 lines · Pádraig Brady
Quoted 6 lines · Dave Chinner
Quite frankly, if system utilities like cp and tar start to abuse
fallocate()by default so they can get "upfront ENOSPC detection", then I will seriously consider making XFS use delayed allocation for fallocate rather than unwritten extents so we don't lose the past 15 years worth of IO and aging optimisations that delayed allocation provides us with....For the record I was considering
fallocate()for these reasons.Quoted 8 lines
1. Improved file layout for subsequent access
2. Immediate indication of ENOSPC
3. Efficient writing of NUL portionsYou lucidly detailed issues with 1. which I suppose could be somewhat mitigated by not fallocating < say 1MB, though I suppose file systems could be smarter here and not preallocate small chunks (or when otherwise not appropriate).
Quoted 5 lines
When you consider that some high end filesystem deployments have alignment characteristics over 50MB (e.g. so each uncompressed 4k resolution video frame is located on a different set of non-overlapping disks), arbitrary "don't fallocate below this amount" heuristics will always have unforseen failure cases...
So about this alignment policy, I don't understand the issues so I'm guessing here. You say delalloc packs files, while fallocate() will align on XFS according to the stripe config. Is that assuming that when writing lots of files, that they will be more likely to be read together, rather than independently. That's a big assumption if true. Also the converse is a big assumption, that fallocate() should be aligned, as that's more likely to be read independently.
Quoted 12 lines
In short: leave optimising general allocation strategies to the filesytems and their developers - there is no One True Solution for optimal file layout in a given filesystem, let alone across different filesytems. In fact, I don't even want to think about the mess
fallocate()on everything would make of btrfs because of it's COW structure - it seems to me to guarantee worse fragmentation than using delayed allocation...Quoted 3 lines
We can already get ENOSPC from a
write()
after anfallocate()in certain edge cases, so it would probably make sense to expand those cases.
Quoted 9 lines
fallocate is for preallocation, not for ENOSPC detection. If you want efficient and effective ENOSPC detection before writing anything, then you really want a space -reservation- extension to fallocate. Filesystems that use delayed allocation already have a space reservation subsystem - it how they account for space that is reserved by delayed allocation prior to the real allocation being done. IMO, allowing userspace some level of access to those reservations would be more appropriate for early detection of ENOSPC than using preallocation for everything...
Fair enough, so fallocate() would be a superset of reserve(), though I'm having a hard time thinking of why one might ever need to fallocate() then.
Quoted 5 lines
As to efficient writing of NULL ranges - that's what sparse files are for - you do not need to write or even preallocate NULL ranges when copying files. Indeed, the most efficient way of dealing with NULL ranges is to punch a hole and let the filesystem deal with it.....
well not for cp --sparse=never which might be used so that processing of the copy will not result in ENOSPC.
I'm also linking here to a related discussion.
http://oss.sgi.com/archives/xfs/2011-06/msg00064.html
Note also that the gold linker does fallocate() on output files by default.
cheers,
Pádraig.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Jeremy Allison
Quoted 6 lines · Ted Ts'o
Quoted 4 lines · Pádraig Brady
It would be better to indicate ENOSPC _before_ copying a (potentially large) file to a (potentially slow) device. If the implementation complexity and side effects of doing this are sufficiently small, then it's worth doing. These discussions are to quantify the side effects.
Quoted 7 lines
Quoted 5 lines
In that case, why not use statfs(2) as a first class approximation? You won't know for user how much fs metadata will be required, but for the common case where someone trying to fit 10 pounds of horse manure in a 5 pound bag, that can be caught very readily without needing to use fallocate(2).
Yeah, we do that too, if the fallocate call fails.
That seems backwards to me; if fallocate fails, statfs(2) isn't going to reveal more space, is it? (modulo metadata issues, anyway?)
-Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 14 lines · Eric Sandeen
Quoted 13 lines · Jeremy Allison
Quoted 11 lines · Ted Ts'o
Quoted 4 lines · Pádraig Brady
It would be better to indicate ENOSPC _before_ copying a (potentially large) file to a (potentially slow) device. If the implementation complexity and side effects of doing this are sufficiently small, then it's worth doing. These discussions are to quantify the side effects.
In that case, why not use statfs(2) as a first class approximation? You won't know for user how much fs metadata will be required, but for the common case where someone trying to fit 10 pounds of horse manure in a 5 pound bag, that can be caught very readily without needing to use fallocate(2).
Quoted 4 lines
Quoted 1 line
Yeah, we do that too, if the fallocate call fails.
That seems backwards to me; if fallocate fails, statfs(2) isn't going to reveal more space, is it? (modulo metadata issues, anyway?)
It might if fallocate fails with ENOSYS :-).
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 15 lines · Jeremy Allison
Quoted 14 lines · Eric Sandeen
Quoted 13 lines · Jeremy Allison
Quoted 11 lines · Ted Ts'o
Quoted 4 lines · Pádraig Brady
It would be better to indicate ENOSPC _before_ copying a (potentially large) file to a (potentially slow) device. If the implementation complexity and side effects of doing this are sufficiently small, then it's worth doing. These discussions are to quantify the side effects.
In that case, why not use statfs(2) as a first class approximation? You won't know for user how much fs metadata will be required, but for the common case where someone trying to fit 10 pounds of horse manure in a 5 pound bag, that can be caught very readily without needing to use fallocate(2).
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Yeah, we do that too, if the fallocate call fails.
That seems backwards to me; if fallocate fails, statfs(2) isn't going to reveal more space, is it? (modulo metadata issues, anyway?)
It might if fallocate fails with ENOSYS :-).
Doh. Sorry, was not thinking of that failure. :)
-Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 21 lines · Pádraig Brady
Quoted 20 lines · Dave Chinner
Quoted 18 lines · Pádraig Brady
Quoted 6 lines · Dave Chinner
Quite frankly, if system utilities like cp and tar start to abuse
fallocate()by default so they can get "upfront ENOSPC detection", then I will seriously consider making XFS use delayed allocation for fallocate rather than unwritten extents so we don't lose the past 15 years worth of IO and aging optimisations that delayed allocation provides us with....For the record I was considering
fallocate()for these reasons.1. Improved file layout for subsequent access
2. Immediate indication of ENOSPC
3. Efficient writing of NUL portionsYou lucidly detailed issues with 1. which I suppose could be somewhat mitigated by not fallocating < say 1MB, though I suppose file systems could be smarter here and not preallocate small chunks (or when otherwise not appropriate).
Quoted 7 lines
Quoted 5 lines
When you consider that some high end filesystem deployments have alignment characteristics over 50MB (e.g. so each uncompressed 4k resolution video frame is located on a different set of non-overlapping disks), arbitrary "don't fallocate below this amount" heuristics will always have unforseen failure cases...
So about this alignment policy, I don't understand the issues so I'm guessing here.
Which, IMO, is exactly why you shouldn't be using fallocate() by default. Every filesystem behaves differently, and is optimises allocation differently to be tuned for the filesystem's unique structure and capability. fallocate() is a big hammer that ensures filesystems cannot optimise allocation to match observed operational patterns.
Quoted 3 lines
You say delalloc packs files, while
fallocate()will align on XFS according to the stripe config. Is that assuming that when writing lots of files, that they will be more likely to be read together, rather than independently.
No, it's assuming that preallocation is used for enabling extremely high performance, high bandwidth IO. This is what it has been used for in XFS for the past 10+ years, and so that is what the implementation in XFS is optimised for (and will continue to be optimised for). In this environment, even when the file size is smaller than the alignment unit, we want allocation alignment to be done.
A real world example for you: supporting multiple, concurrent, realtime file-per-frame uncompressed 2k res video streams (@ ~12.5MB per frame). Systems doing this sort of work are made from lots of HW RAID5/6 Luns (often spread across multiple arrays) that will have a stripe width of 14MB. XFS will be configured with a stripe unit of 14MB. 4-6 of these Luns will be striped together to give a stripe width of 56-84MB from a filesystem perspective. Each file that is preallocated needs to be aligned to a 16MB stripe unit so that each frame IO goes to a different RAID Lun. Each frame write can be done as a full stripe write without a RMW cycle in the back end array, and each frame read loads all the disks in the LUN evenly. i.e. the load is distributed evenly, optimally and deterministically across all the back end storage.
This is the sort of application that cannot be done effectively with a lot of filesystem allocator support (indeed, XFS has the special filestreams allocation policy for this workload), and it's this sort of high peformance application that what we optimise preallocation for.
In short, what XFS is doing here is optimising allocation patterns for high performance, RAID based storage. If your write pattern triggers repeated RMW cycles in a RAID array, your write performance will fall by an order of magnitude or more. Large files don't need packing because the writeback flusher threads can do full stripe writes which avoids RMW cycles in the RAID array if the files are aligned to the underlying RAID stripes. But small files need tight packing to enable them to be aggregated into full stripe writes in the elevator and/or RAID controller BBWC. This aggregation then avoids RMW cycles in the RAID array and hence writeback performance for both small and large files is similar (i.e. close to maximum IO bandwidth). If you don't pack small files tightly (and XFs won't if you use preallocation), then each file write will cause a RMW cycle in the RAID array and the throughput is effective going to be about half the IOPS of a random write workload....
Quoted 2 lines
That's a big assumption if true. Also the converse is a big assumption, that
fallocate()should be aligned, as that's more likely to be read independently.
You're guessing, making assumptions, etc all about how one filesystem works and what the impact of the change is going to be. What about ext4, or btrfs? They are very different structurally to XFS, and hence have different sets of issues when you start preallocating everything. It is not a simple problem: allocation optimisation is, IMO, the single most difficult and complex area of filesystems, with many different, non-obvious, filesystem specific trade-offs to be made....
Quoted 10 lines
Quoted 9 lines
fallocate is for preallocation, not for ENOSPC detection. If you want efficient and effective ENOSPC detection before writing anything, then you really want a space -reservation- extension to fallocate. Filesystems that use delayed allocation already have a space reservation subsystem - it how they account for space that is reserved by delayed allocation prior to the real allocation being done. IMO, allowing userspace some level of access to those reservations would be more appropriate for early detection of ENOSPC than using preallocation for everything...
Quoted 3 lines
Fair enough, so
fallocate()would be a superset ofreserve(), though I'm having a hard time thinking of why one might ever need tofallocate()then.
Exactly my point - the number of applications that actually need -preallocation- for performance reasons is actually quite small.
I'd suggest that we'd implement a reservation mechanism as a separate fallocate() flag, to tell fallocate() to reserve the space over the given range rather than needing to preallocate it. I'd also suggest that a reservation is not persistent (e.g. only guaranteed to last for the life of the file descriptor the reservation was made for). That would make it simple to implement in memory for all filesystems, and provide you with the short-term ENOSPC-or-success style reservation you are looking for...
Does that sound reasonable?
Quoted 11 lines
Quoted 5 lines
As to efficient writing of NULL ranges - that's what sparse files are for - you do not need to write or even preallocate NULL ranges when copying files. Indeed, the most efficient way of dealing with NULL ranges is to punch a hole and let the filesystem deal with it.....
well not for
cp --sparse=neverwhich might be used so that processing of the copy will not result in ENOSPC.I'm also linking here to a related discussion.
http://oss.sgi.com/archives/xfs/2011-06/msg00064.html
Right, and from that discussion you can see exactly why delayed allocation in XFS significantly improves both data and metadata allocation and IO patterns for operations like tar, cp, rsync, etc whilst also minimising long term aging effects as compared to preallocation:
http://oss.sgi.com/archives/xfs/2011-06/msg00092.html
Quoted 1 line
Note also that the gold linker does
fallocate()on output files by default.
"He's doing it, so we should do it" is not a very convincing technical argument.
Cheers,
Dave.
Signature · 6 lines
Dave Chinner [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 123 lines · Dave Chinner
Quoted 21 lines · Pádraig Brady
Quoted 20 lines · Dave Chinner
Quoted 18 lines · Pádraig Brady
Quoted 6 lines · Dave Chinner
Quite frankly, if system utilities like cp and tar start to abuse
fallocate()by default so they can get "upfront ENOSPC detection", then I will seriously consider making XFS use delayed allocation for fallocate rather than unwritten extents so we don't lose the past 15 years worth of IO and aging optimisations that delayed allocation provides us with....For the record I was considering
fallocate()for these reasons.1. Improved file layout for subsequent access
2. Immediate indication of ENOSPC
3. Efficient writing of NUL portionsYou lucidly detailed issues with 1. which I suppose could be somewhat mitigated by not fallocating < say 1MB, though I suppose file systems could be smarter here and not preallocate small chunks (or when otherwise not appropriate).
Quoted 7 lines
Quoted 5 lines
When you consider that some high end filesystem deployments have alignment characteristics over 50MB (e.g. so each uncompressed 4k resolution video frame is located on a different set of non-overlapping disks), arbitrary "don't fallocate below this amount" heuristics will always have unforseen failure cases...
So about this alignment policy, I don't understand the issues so I'm guessing here.
Which, IMO, is exactly why you shouldn't be using
fallocate()by default. Every filesystem behaves differently, and is optimises allocation differently to be tuned for the filesystem's unique structure and capability.fallocate()is a big hammer that ensures filesystems cannot optimise allocation to match observed operational patterns.Quoted 3 lines
You say delalloc packs files, while
fallocate()will align on XFS according to the stripe config. Is that assuming that when writing lots of files, that they will be more likely to be read together, rather than independently.No, it's assuming that preallocation is used for enabling extremely high performance, high bandwidth IO. This is what it has been used for in XFS for the past 10+ years, and so that is what the implementation in XFS is optimised for (and will continue to be optimised for). In this environment, even when the file size is smaller than the alignment unit, we want allocation alignment to be done.
A real world example for you: supporting multiple, concurrent, realtime file-per-frame uncompressed 2k res video streams (@ ~12.5MB per frame). Systems doing this sort of work are made from lots of HW RAID5/6 Luns (often spread across multiple arrays) that will have a stripe width of 14MB. XFS will be configured with a stripe unit of 14MB. 4-6 of these Luns will be striped together to give a stripe width of 56-84MB from a filesystem perspective. Each file that is preallocated needs to be aligned to a 16MB stripe unit so that each frame IO goes to a different RAID Lun. Each frame write can be done as a full stripe write without a RMW cycle in the back end array, and each frame read loads all the disks in the LUN evenly. i.e. the load is distributed evenly, optimally and deterministically across all the back end storage.
This is the sort of application that cannot be done effectively with a lot of filesystem allocator support (indeed, XFS has the special filestreams allocation policy for this workload), and it's this sort of high peformance application that what we optimise preallocation for.
In short, what XFS is doing here is optimising allocation patterns for high performance, RAID based storage. If your write pattern triggers repeated RMW cycles in a RAID array, your write performance will fall by an order of magnitude or more. Large files don't need packing because the writeback flusher threads can do full stripe writes which avoids RMW cycles in the RAID array if the files are aligned to the underlying RAID stripes. But small files need tight packing to enable them to be aggregated into full stripe writes in the elevator and/or RAID controller BBWC. This aggregation then avoids RMW cycles in the RAID array and hence writeback performance for both small and large files is similar (i.e. close to maximum IO bandwidth). If you don't pack small files tightly (and XFs won't if you use preallocation), then each file write will cause a RMW cycle in the RAID array and the throughput is effective going to be about half the IOPS of a random write workload....
Quoted 2 lines
That's a big assumption if true. Also the converse is a big assumption, that
fallocate()should be aligned, as that's more likely to be read independently.You're guessing, making assumptions, etc all about how one filesystem works and what the impact of the change is going to be. What about ext4, or btrfs? They are very different structurally to XFS, and hence have different sets of issues when you start preallocating everything. It is not a simple problem: allocation optimisation is, IMO, the single most difficult and complex area of filesystems, with many different, non-obvious, filesystem specific trade-offs to be made....
Quoted 10 lines
Quoted 9 lines
fallocate is for preallocation, not for ENOSPC detection. If you want efficient and effective ENOSPC detection before writing anything, then you really want a space -reservation- extension to fallocate. Filesystems that use delayed allocation already have a space reservation subsystem - it how they account for space that is reserved by delayed allocation prior to the real allocation being done. IMO, allowing userspace some level of access to those reservations would be more appropriate for early detection of ENOSPC than using preallocation for everything...
Quoted 3 lines
Fair enough, so
fallocate()would be a superset ofreserve(), though I'm having a hard time thinking of why one might ever need tofallocate()then.Exactly my point - the number of applications that actually need -preallocation- for performance reasons is actually quite small.
I'd suggest that we'd implement a reservation mechanism as a separate
fallocate()flag, to tellfallocate()to reserve the space over the given range rather than needing to preallocate it. I'd also suggest that a reservation is not persistent (e.g. only guaranteed to last for the life of the file descriptor the reservation was made for). That would make it simple to implement in memory for all filesystems, and provide you with the short-term ENOSPC-or-success style reservation you are looking for...
Quoted 1 line
Does that sound reasonable?
But then posix_fallocate() would always be slow I think, requiring one to actually write the NULs.
TBH, it sounds like the best/minimal change is to the uncommon case. I.E. add an ALIGN flag to fallocate() which specialised apps like described above can use.
Quoted 18 lines
Quoted 11 lines
Quoted 5 lines
As to efficient writing of NULL ranges - that's what sparse files are for - you do not need to write or even preallocate NULL ranges when copying files. Indeed, the most efficient way of dealing with NULL ranges is to punch a hole and let the filesystem deal with it.....
well not for
cp --sparse=neverwhich might be used so that processing of the copy will not result in ENOSPC.I'm also linking here to a related discussion.
http://oss.sgi.com/archives/xfs/2011-06/msg00064.htmlRight, and from that discussion you can see exactly why delayed allocation in XFS significantly improves both data and metadata allocation and IO patterns for operations like tar, cp, rsync, etc whilst also minimising long term aging effects as compared to preallocation:
Quoted 6 lines
http://oss.sgi.com/archives/xfs/2011-06/msg00092.html
Quoted 1 line
Note also that the gold linker does
fallocate()on output files by default."He's doing it, so we should do it" is not a very convincing technical argument.
Just FYI.
cheers,
Pádraig.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · Pádraig Brady
But then
posix_fallocate()would always be slow I think, requiring one to actually write the NULs.
Almost no one should ever use posix_fallocate(); it's can be a performance disaster because you don't know whether or not the file system will really do fallocate, or will do the slow "write zeros" thing.
You really should use fallocate(), take the failure if the file system doesn't support fallocate, and then you can decide what the appropriate thing to do might be.
- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · Ted Ts'o
Quoted 3 lines · Pádraig Brady
But then
posix_fallocate()would always be slow I think, requiring one to actually write the NULs.
Quoted 8 lines
Almost no one should ever use
posix_fallocate(); it's can be a performance disaster because you don't know whether or not the file system will really do fallocate, or will do the slow "write zeros" thing.You really should use
fallocate(), take the failure if the file system doesn't support fallocate, and then you can decide what the appropriate thing to do might be.
s/posix_fallocate()/functionality provided by &/
I.E. copy --sparse=never could use that,
and it would be beneficial if it was as fast as possible.
I looked for a couple of minutes on the XFS preallocate behaviour, and it seems that these ioctls pre date fallocate().
http://linux.die.net/man/3/xfsctl
I see XFS_IOC_ALLOCSP and XFS_IOC_RESVSP.
So fallocate() support was directly mapped on top of the existing ALLOCSP. I think the specialised alignment behavior should be restricted to direct calls to XFS_IOC_ALLOCSP to be called by xfs_mkfile(1) or whatever. Better would be to provide generic access to that functionality through an ALIGN option to fallocate()
cheers,
Pádraig.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Pádraig Brady
I looked for a couple of minutes on the XFS preallocate behaviour, and it seems that these ioctls pre date
fallocate().
http://linux.die.net/man/3/xfsctl
I seeXFS_IOC_ALLOCSPandXFS_IOC_RESVSP.
Sofallocate()support was directly mapped on top of the existing ALLOCSP. I think the specialised alignment behavior should be restricted to direct calls toXFS_IOC_ALLOCSPto be called byxfs_mkfile(1)or whatever. Better would be to provide generic access to that functionality through an ALIGN option tofallocate()
Well, XFS_IOC_RESVSP is the same as fallocate with the FALLOC_FL_KEEP_SIZE flag. That is to say, blocks are allocated and attached to the inode --- that is, which blocks out of the pool of free blocks should be selected is decided at the time that you call fallocate() with the KEEP_SIZE flag or use the XFS_IOC_RESVSP ioctl (which by the way works on any file system that supports fallocate on modern kernels --- the kernel provides the translation from XFS_IOC_RESVSP to fallocate/KEEP_SIZE in fs/ioctl.c's ioctl_preallocate() function.)
What Dave was talking about is something different. He's suggesting a new call which reserves space, but which does not actually make the block allocation decision until the time of the write. He suggested tieing it to the file descriptor, but I wonder if it's actually more functional to tie it to the process --- that is, the process says, "guarantee that I will be able to write 5MB", and writes made by that process get counted against that 5MB reservation. When the process exits, any reservation made by that process evaporates.
Whether we tie this space reservation to a fd or a process, we also would need to decide up front whether this space shows up as "missing" by statfs(2)/df or not.
- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Ted Ts'o
What Dave was talking about is something different. He's suggesting a new call which reserves space, but which does not actually make the block allocation decision until the time of the write. He suggested tieing it to the file descriptor, but I wonder if it's actually more functional to tie it to the process --- that is, the process says, "guarantee that I will be able to write 5MB", and writes made by that process get counted against that 5MB reservation. When the process exits, any reservation made by that process evaporates.
It needs to be tied to the inode in some way - there's metadata reservations that need to be made per inode that delayed allocation reserations are made for to take into account the potential need to allocate extent tree blocks as well. If we on't do this, then we'll get ENOSPC reported for writes during writeback that should have succeeded. And that is a Bad Thing.
Further, you need to track all the ranges that have space reserved like a special type of delayed allocation extent. That way, when the write() comes along into the reserved range, you don't account for it a second time as delayed allocation as the space usage has already been accounted for.
And then there is the problem of freeing space that you don't use. Close the fd and you automatically terminate the reservation. fiemap can be used to find unused reserved ranges. You could probably even release them by punching the range.
If you have a per-process pool, how do you only use it for the write() calls you want, on the file you want, over the range you wanted reserved? And when you have finished writing to that file, how do you release any unused reservation? How do you know that you've got reservations remaining?
Then the interesting questions start - how does per-process reservation interact with quotas? The quota needs to be checked whenthe reservation is made, and without knowing what file it is being made for this canot be done sanely. Especially for project quotas....
Also, per-process reservatin pools can't really be managed through existing APIs, so we'd need new ones. And then we'd be asking application developers to use two different models for almost identical functionality, which means they'll just use the one that is most effective for their purpose (i.e. fallocate() because they already have a fd open on the file they are going to write to).
IOWs, all I see from an implementation persepctive of per-process reservation pools is complexity and nasty corner cases. And from the user persepctive, an API that doesn't match up with the operations at hand. i.e. that of writing a file....
Quoted 3 lines
Whether we tie this space reservation to a fd or a process, we also would need to decide up front whether this space shows up as "missing" by statfs(2)/df or not.
IMO, reserved space is used space - it's not free for just anyone to use anymore, and it has to be checked and accounted against quotas even before it gets used....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 11 lines · Ted Ts'o
Quoted 9 lines · Pádraig Brady
I looked for a couple of minutes on the XFS preallocate behaviour, and it seems that these ioctls pre date
fallocate().
http://linux.die.net/man/3/xfsctl
I seeXFS_IOC_ALLOCSPandXFS_IOC_RESVSP.
Sofallocate()support was directly mapped on top of the existing ALLOCSP. I think the specialised alignment behavior should be restricted to direct calls toXFS_IOC_ALLOCSPto be called byxfs_mkfile(1)or whatever. Better would be to provide generic access to that functionality through an ALIGN option tofallocate()
Quoted 9 lines
Well,
XFS_IOC_RESVSPis the same as fallocate with theFALLOC_FL_KEEP_SIZEflag. That is to say, blocks are allocated and attached to the inode --- that is, which blocks out of the pool of free blocks should be selected is decided at the time that you callfallocate()with theKEEP_SIZEflag or use theXFS_IOC_RESVSPioctl (which by the way works on any file system that supports fallocate on modern kernels --- the kernel provides the translation fromXFS_IOC_RESVSPto fallocate/KEEP_SIZE in fs/ioctl.c'sioctl_preallocate()function.)
Thanks for the clarification.
My main point is that these related ioctls existed before fallocate.
Quoted 3 lines
What Dave was talking about is something different. He's suggesting a new call which reserves space, but which does not actually make the block allocation decision until the time of the write.
Yes that was clear.
I'm still not sure it's needed TBH.
The separation of functionality is needed for the reasons Dave detailed, but it might be better to add an ALIGN flag to fallocate for that special use case.
I'm not trying to enforce my argument with repetition here, just trying to be clear.
cheers,
Pádraig.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 29 lines · Pádraig Brady
Quoted 11 lines · Ted Ts'o
Quoted 9 lines · Pádraig Brady
I looked for a couple of minutes on the XFS preallocate behaviour, and it seems that these ioctls pre date
fallocate().
http://linux.die.net/man/3/xfsctl
I seeXFS_IOC_ALLOCSPandXFS_IOC_RESVSP.
Sofallocate()support was directly mapped on top of the existing ALLOCSP. I think the specialised alignment behavior should be restricted to direct calls toXFS_IOC_ALLOCSPto be called byxfs_mkfile(1)or whatever. Better would be to provide generic access to that functionality through an ALIGN option tofallocate()Quoted 9 lines
Well,
XFS_IOC_RESVSPis the same as fallocate with theFALLOC_FL_KEEP_SIZEflag. That is to say, blocks are allocated and attached to the inode --- that is, which blocks out of the pool of free blocks should be selected is decided at the time that you callfallocate()with theKEEP_SIZEflag or use theXFS_IOC_RESVSPioctl (which by the way works on any file system that supports fallocate on modern kernels --- the kernel provides the translation fromXFS_IOC_RESVSPto fallocate/KEEP_SIZE in fs/ioctl.c'sioctl_preallocate()function.)Thanks for the clarification.
My main point is that these related ioctls existed before fallocate.Quoted 3 lines
What Dave was talking about is something different. He's suggesting a new call which reserves space, but which does not actually make the block allocation decision until the time of the write.
Quoted 7 lines
Yes that was clear.
I'm still not sure it's needed TBH.
The separation of functionality is needed for the reasons Dave detailed, but it might be better to add an ALIGN flag to fallocate for that special use case.
I'm not trying to enforce my argument with repetition here, just trying to be clear.
Is XFS the only file system that overloads this alignment behavior on fallocate()? Why I ask, is because if that was the case, then perhaps XFS could change to using the FALLOC_FL_ALIGN flag for this (or its existing ioctl), and so would not be negatively impacted by tools which start using fallocate(), unaware of the subtle performance implications on XFS.
cheers,
Pádraig.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html