optimal file order for reading from disk
I know this will be dependent on filesystem, I/O scheduler, ... but given a list of files, what is the best (filesystem agnostic) order to read from disk (to minimise seeks).
Should I sort by path, inode number, getdents, or something else?
thanks,
Pádraig.
Quoted 5 lines · On Tue, Jun 14, 2005 at 12:21:40PM +0100, [email protected] wrote:
I know this will be dependent on filesystem, I/O scheduler, ... but given a list of files, what is the best (filesystem agnostic) order to read from disk (to minimise seeks).
Should I sort by path, inode number, getdents, or something else?
I know several projects that sort on inode number and benefit from that, sometimes in a big way. The effect of this will probably be less on a matured filesystem image.
I can't really explain why it helps though. I don't think the kernel will do 'crossfile readahead', although your disk might do so.
Google on 'orlov allocator', is enlightning.
Signature · 2 lines
http://www.PowerDNS.com Open source, database driven DNS Software http://netherlabs.nl Open and Closed source services
Quoted 11 lines · bert hubert
Quoted 5 lines · On Tue, Jun 14, 2005 at 12:21:40PM +0100, [email protected] wrote:
I know this will be dependent on filesystem, I/O scheduler, ... but given a list of files, what is the best (filesystem agnostic) order to read from disk (to minimise seeks).
Should I sort by path, inode number, getdents, or something else?
I know several projects that sort on inode number and benefit from that, sometimes in a big way. The effect of this will probably be less on a matured filesystem image.
Thanks for that. Yep I'm torn between sorting by inode which should be good for new filesystems, but maybe sorting by path would be better for mature filesystems?
Quoted 4 lines
I can't really explain why it helps though. I don't think the kernel will do 'crossfile readahead', although your disk might do so.
Google on 'orlov allocator', is enlightning.
I found some interesting into here thanks:
http://kerneltrap.org/node/2157
cheers,
Pádraig.
Quoted 8 lines · [email protected] writes:
Quoted 7 lines · bert hubert
Quoted 5 lines · On Tue, Jun 14, 2005 at 12:21:40PM +0100, [email protected] wrote:
I know this will be dependent on filesystem, I/O scheduler, ... but given a list of files, what is the best (filesystem agnostic) order to read from disk (to minimise seeks).
Should I sort by path, inode number, getdents, or something else?
Quoted 8 lines
Quoted 4 lines
I know several projects that sort on inode number and benefit from that,
sometimes in a big way. The effect of this will probably be less on a matured filesystem image.Thanks for that. Yep I'm torn between sorting by inode which should be good for new filesystems, but maybe sorting by path would be better for mature filesystems?
Sorting by either inode number, or file name is not "file system agnostic" way to minimize seeks.
You should call fibmap ioctl on all files, to obtain lists of block numbers used by them, and then sort file list to minimize seeks.
Quoted 10 lines
Quoted 4 lines
I can't really explain why it helps though. I don't think the kernel will do 'crossfile readahead', although your disk might do so.
Google on 'orlov allocator', is enlightning.I found some interesting into here thanks:
http://kerneltrap.org/node/2157cheers,
Pádraig.
Nikita.
Quoted 2 lines · Nikita Danilov
You should call fibmap ioctl on all files, to obtain lists of block numbers used by them, and then sort file list to minimize seeks.
hey that's a cool hack...
looking at FIBMAP i see it uses an int to store the block number... any plans for a FIBMAP64 api?
-dean