XFS on 2.6.26: reading the first 4K of a large file takes ages
We've got a couple of rather large files, and with a cold cache, reading the first 4K bytes of the file (e.g., just running "head --bytes 4096" on it) takes ages, up to several minutes, sometimes triggering the hang check timer.
I wonder if XFS reads the whole extent information into RAM when the file is opened. Is this the case, at least on 2.6.26? Has this changed in later versions, perhaps?
The files in question are heavily fragmented (they have been created with holes first, and the holes have been filled in subsequently). I'll try to run xfs_fsr on those files, but it's going to be tough. 8-/
Signature · 9 lines
Florian Weimer <[email protected]> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · Florian Weimer
We've got a couple of rather large files, and with a cold cache, reading the first 4K bytes of the file (e.g., just running "head --bytes 4096" on it) takes ages, up to several minutes, sometimes triggering the hang check timer.
I wonder if XFS reads the whole extent information into RAM when the file is opened. Is this the case, at least on 2.6.26? Has this changed in later versions, perhaps?
Yes, XFS always reads in the extent map, and no this hasn't changed recently.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Florian Weimer
We've got a couple of rather large files, and with a cold cache, reading the first 4K bytes of the file (e.g., just running "head --bytes 4096" on it) takes ages, up to several minutes, sometimes triggering the hang check timer.
Quoted 6 lines
Quoted 3 lines
I wonder if XFS reads the whole extent information into RAM when the file is opened. Is this the case, at least on 2.6.26? Has this changed in later versions, perhaps?
Yes, XFS always reads in the extent map, and no this hasn't changed recently.
And demand paging the in-memory information is hard. It's on a to-do list somewhere 'round here....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
* Christoph Hellwig:
Quoted 6 lines · Florian Weimer
Quoted 5 lines
We've got a couple of rather large files, and with a cold cache, reading the first 4K bytes of the file (e.g., just running "head --bytes 4096" on it) takes ages, up to several minutes, sometimes triggering the hang check timer.
Quoted 6 lines
Quoted 3 lines
I wonder if XFS reads the whole extent information into RAM when the file is opened. Is this the case, at least on 2.6.26? Has this changed in later versions, perhaps?
Yes, XFS always reads in the extent map, and no this hasn't changed recently.
Okay, defragmenting seems to improve things considerably. But it's going to take a while: "extents before:5309152 after:13" *sigh*
Thanks for confirming my hunch. I don't think it's worth fixing this in XFS. The database should call posix_fallocate() before flushing its internal cache to the file in essentially random order, but it's difficult to get upstream to implement this (the source code is a bit hard to follow, unfortunately).
Signature · 9 lines
Florian Weimer <[email protected]> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 5 lines · Florian Weimer
Thanks for confirming my hunch. I don't think it's worth fixing this in XFS. The database should call
posix_fallocate()before flushing its internal cache to the file in essentially random order, but it's difficult to get upstream to implement this (the source code is a bit hard to follow, unfortunately).
Which database?
You could always mount with allocsize or use other tools to do the preallocation before things got too bad.
Signature · 6 lines
Stewart Smith _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
* Stewart Smith:
Quoted 8 lines · Florian Weimer
Quoted 5 lines
Thanks for confirming my hunch. I don't think it's worth fixing this in XFS. The database should call
posix_fallocate()before flushing its internal cache to the file in essentially random order, but it's difficult to get upstream to implement this (the source code is a bit hard to follow, unfortunately).Which database?
Oracle Berkeley DB.
Quoted 1 line
You could always mount with allocsize
This happens with "allocsize=4194304".
Quoted 2 lines
or use other tools to do the preallocation before things got too bad.
Is there a way to transparently preallocate a few GB after the current end of the file? That would be helpful because Berkeley DB wouldn't have to know about it.
It's a legacy system, otherwise I would invest more effort into putting some sort of preallocation somewhere deep into Berkeley DB.
Signature · 9 lines
Florian Weimer <[email protected]> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Florian Weimer
* Stewart Smith:
Quoted 7 lines · Florian Weimer
Quoted 5 lines
Thanks for confirming my hunch. I don't think it's worth fixing this in XFS. The database should call
posix_fallocate()before flushing its internal cache to the file in essentially random order, but it's difficult to get upstream to implement this (the source code is a bit hard to follow, unfortunately).
Quoted 7 lines
Quoted 1 line
Which database?
Oracle Berkeley DB.
Quoted 1 line
You could always mount with allocsize
This happens with "allocsize=4194304".
Because allocsize only works for allocations extending the file.
Quoted 6 lines
Quoted 2 lines
or use other tools to do the preallocation before things got too bad.
Is there a way to transparently preallocate a few GB after the current end of the file? That would be helpful because Berkeley DB wouldn't have to know about it.
Yes. the fallocate() syscall has a mode that allows allocation beyond the current end of file, as does the XFS_IOC_RESVSP ioctl.
Or, even easier, with xfs_io:
$ stat /mnt/test/foo File: `/mnt/test/foo' Size: 0 Blocks: 0 IO Block: 4096 regular empty file .... $ xfs_io -f -c "resvsp 0 1048576" /mnt/test/foo $ stat /mnt/test/foo File: `/mnt/test/foo' Size: 0 Blocks: 2048 IO Block: 4096 regular empty file .... $ xfs_bmap -vp /mnt/test/foo /mnt/test/foo: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS 0: [0..2047]: 171912..173959 0 (171912..173959) 2048 10000 $
/mnt/test/foo still a zero length file but has 1MB of extents allocated.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs