file system defragmentation
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi, you mentioned delayed allocation. What is the size of the buffer holding the data, before they're actually written to disk? How can it tackle growing files?
If I have a DVR system of 16 channels. They keep writing data to the disk in pieces of video files. I read some spec of xfs. Apart from extend-based allocation, there're allocaiton groups in xfs. I would like to ask, does XFS's allocation groups work similar as JFS's, which would lock an allocation group for individual file write? How does the allocation group in XFS work? And how would it help the fragmentation problem?
Thankyou!
Signature · 2 lines
View this message in context: http://www.nabble.com/file-system-defragmentation-tf255485.html#a5356806 Sent from the Xfs - General forum at Nabble.com.
Quoted 3 lines · Cosmo Nova
Hi, you mentioned delayed allocation. What is the size of the buffer holding the data, before they're actually written to disk? How can it tackle growing files?
I'll leave the details for Nathan/Dave...
Quoted 8 lines
If I have a DVR system of 16 channels. They keep writing data to the disk in pieces of video files. I read some spec of xfs. Apart from extend-based allocation, there're allocaiton groups in xfs. I would like to ask, does XFS's allocation groups work similar as JFS's, which would lock an allocation group for individual file write? How does the allocation group in XFS work? And how would it help the fragmentation problem?
Locking down an allocation group is part of what a new "filestream" mount option will provide. This feature is already in use at a number of IRIX sites with media apps (mainly in conjunction with CXFS since you have more writers in a clustered filesystem and more need to lock down contiguous space), and is coming to Linux XFS soon.
The best way to tackle fragementation with streaming data is to preallocate space for the files. This allows the allocator to try to find the space you require using as large contiguous chunks as it can find. See xfsctl(3).
David
Signature · 3 lines
David Chatterton Phone: +61 3 9963 1934 XFS Engineering Manager Mobile: +61 409 154 121 SGI Australia http://www.sgi.com/products/storage
Quoted 4 lines · Cosmo Nova
Hi, you mentioned delayed allocation. What is the size of the buffer holding the data, before they're actually written to disk? How can it tackle growing files?
No special buffering is needed in XFS as the page cache does it all for us. delalloc means that XFS does not allocate until the data gets written to disk (i.e. flushed from the cache). The size of the allocation is dependent on the amount of contiguous dirty data in the file. That is, The more sequential dirty pages in the page cache for a file, the larger the allocation...
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Hi, can I summarize by saying that, XFS still partition the volume into different allocation groups, but there is NO locking. Two files may write in the same allocation group and introduce fragments?
What is the maximum and typical number of allocation groups please? I can't find the numbers in the source code...
And for the buffer, Is preallocation the same as delayed allocation? page cache, as I remember belongs to Linux VFS. What are the memory / disk requirement and sizing for the cache please? If a file space allocation is delayed, what is the upper bound of the delayed size? Can files of MBs, GBs being delayed allocate, and multiple channels/threads files all store in memory?? How XFS's allocation works (with number please if possible) to help tackle fragmentation?
Thankyou!
Signature · 2 lines
View this message in context: http://www.nabble.com/file-system-defragmentation-tf255485.html#a5374022 Sent from the Xfs - General forum at Nabble.com.
Quoted 2 lines · Cosmo Nova
Hi, can I summarize by saying that, XFS still partition the volume into different allocation groups, but there is NO locking.
the allocator has per-AG locks
Quoted 2 lines
Two files may write in the same allocation group and introduce fragments?
i'm not sure what you mean, but any number of files can be writing to a single AG at once, the degree of fragmentation you will see depends on the IO patterns and where there are writing
in the case of writing multiple large files in parallel in the same directory the blocks do tend to interleave
Quoted 2 lines
What is the maximum and typical number of allocation groups please? I can't find the numbers in the source code...
It's a 32-bit value, I'm not sure how practical very large values are though.
Quoted 1 line
And for the buffer, Is preallocation the same as delayed allocation?
No, preallocation allocates space a head of time on disk so that later when it's written the allocation process is simpler (or not needed) and ideally also to reduce fragmentation by doing as much allocation as possible at once.
Delayed allocation is simply the idea of doing the allocation as late as possible so when you do have to flush the data to disk you can try to make better decisions about where to place the blocks.
Quoted 2 lines
If a file space allocation is delayed, what is the upper bound of the delayed size?
It depends on the platform and how often data is flushed from the OS.
If there are AG locks, but multiple files can write in the same AG, how "AG lock" is interpreted by XFS then? What does it really do?
For pre-allocation and delayed allocation, do they belong to the feature set of XFS? Or are they application dependant?
I am trying to compare filesystems, especially xfs vs jfs. I found that jfs's per AG locking would only allow one file to be written per AG, which helps a lot to prevent fragmentation (according to experiment results). There're no pre-allocation and delayed allocation in JFS though. Comparing the experiment results, XFS is doing a good job, giving 6-10 fragments (compare to majority of single fragment in JFS...). So answers of the above questions with numbers would help a lot. Thanks!
Signature · 2 lines
View this message in context: http://www.nabble.com/file-system-defragmentation-tf255485.html#a5393901 Sent from the Xfs - General forum at Nabble.com.
Quoted 2 lines · Cosmo Nova
If I have a DVR system of 16 channels. They keep writing data to the disk in pieces of video files.
I did some work for someone who does a similar thing (they write 96 channels in parallel and have to be able to do read back up to 32 of them at the same time of something).
By default concurrent writes into the same directory will cause the files to get badly interleaved, and trying to get one file per-ag doesn't work so well if the agcount < n-files.
I ended up getting them to change their code to preallocate and AFAIK it works very well now (you can throw away any space you over allocate when you're closing the file too).
A message that is not in the archive (sent off-list or not yet received): E4E12CD035E14A4AB8C36E0F4B7143F7277DED@iu-mssg-mbx09.exchange.iu.edu
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
does the preallocation belong to xfs layer? or it's a "suggested feature" for applications?
Signature · 2 lines
View this message in context: http://www.nabble.com/file-system-defragmentation-tf255485.html#a5408744 Sent from the Xfs - General forum at Nabble.com.
A message that is not in the archive (sent off-list or not yet received): [email protected]
what is the maximum tolerance for the delayed allocations pls? I guess it's not possible to buf a GB size file and delay its allocation?
Signature · 2 lines
View this message in context: http://www.nabble.com/file-system-defragmentation-tf255485.html#a5408752 Sent from the Xfs - General forum at Nabble.com.
Quoted 3 lines · Cosmo Nova
what is the maximum tolerance for the delayed allocations pls? I guess it's not possible to buf a GB size file and delay its allocation?
Sure it is. The limit on how much is buffered before writeback (and therefore allocation) is determined by how much memory you have and your /proc/sys/vm/dirty* tunable settings. This can range from a few MB to a few TB of data depending on how big your machine is ;)
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group