Slow filesystem.
Hi
My work (as in what brings in money to buy things from) file-system is now "slow as hell". Last time this happened you guys told me that this was because I had millions of hardlinks and way too many files.
Filesystem Inodes IUsed IFree IUse% Mounted on/dev/md0 815M 4.1M 811M 1% /recover2
I have only 4.1M files. Only 1% inodes "in use".
The filesystem was filling up lately, but i've managed to create some free space.
Filesystem Size Used Avail Use% Mounted on /dev/md0 6.3T 5.3T 720G 89% /recover2
The question is: How do I get the filesystem to perform
normally again?
All mkdir calls are taking around 62 milliseconds:
14:57:02 mkdir("...ME_25c8e/", 0777) = 0 <0.062312> 14:57:02 mkdir("...ME_112ff/", 0777) = 0 <0.062658> 14:57:02 mkdir("...ME_183cf/", 0777) = 0 <0.062344>
while on my workstation where i'm not supposed to be working with client-data I get:
14:58:18 mkdir("...ME_2a6c6/", 0777) = 0 <0.000033> 14:58:18 mkdir("...ME_2b3d9/", 0777) = 0 <0.000031> 14:58:18 mkdir("...ME_2c63f/", 0777) = 0 <0.000034>
So, things on my big datastorage drive are about 2000 times slower than on my workstation.
Roger.
Signature · 6 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 ** ** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 ** *-- BitWizard writes Linux device drivers for any device you may have! --* Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. Does it sit on the couch all day? Is it unemployed? Please be specific! Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
Quoted 23 lines · Rogier Wolff
Hi
My work (as in what brings in money to buy things from) file-system is now "slow as hell". Last time this happened you guys told me that this was because I had millions of hardlinks and way too many files.
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md0815M 4.1M 811M 1% /recover2I have only 4.1M files. Only 1% inodes "in use".
The filesystem was filling up lately, but i've managed to create some free space.
Filesystem Size Used Avail Use% Mounted on /dev/md0 6.3T 5.3T 720G 89% /recover2The question is: How do I get the filesystem to perform
normally again?
Quoted 5 lines
All mkdir calls are taking around 62 milliseconds:
14:57:02
mkdir("...ME_25c8e/", 0777)= 0 <0.062312> 14:57:02mkdir("...ME_112ff/", 0777)= 0 <0.062658> 14:57:02mkdir("...ME_183cf/", 0777)= 0 <0.062344>
I don't know what the workload is, and "slow" isn't very well characterized here, but from this can I assume that mkdir is the bottleneck?
Did you ever try that patch I sent you for this issue in March?
Using oprofile to see where the fs is spending its time might also be useful.
-Eric
Quoted 12 lines
while on my workstation where i'm not supposed to be working with client-data I get:
14:58:18
mkdir("...ME_2a6c6/", 0777)= 0 <0.000033> 14:58:18mkdir("...ME_2b3d9/", 0777)= 0 <0.000031> 14:58:18mkdir("...ME_2c63f/", 0777)= 0 <0.000034>So, things on my big datastorage drive are about 2000 times slower than on my workstation.
Roger.
Quoted 7 lines · Rogier Wolff
My work (as in what brings in money to buy things from) file-system is now "slow as hell". Last time this happened you guys told me that this was because I had millions of hardlinks and way too many files.
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md0815M 4.1M 811M 1% /recover2
Quoted 8 lines
I have only 4.1M files. Only 1% inodes "in use".
The filesystem was filling up lately, but i've managed to create some free space.
Filesystem Size Used Avail Use% Mounted on /dev/md0 6.3T 5.3T 720G 89% /recover2
I suspect that the free inodes in your filesystem are very fragmented, due to the filesystem filling up and then deleting a lot of files that create "holes" of free inodes within the inode table blocks. That means that possibly finding free inodes is a bunch of work, and then possibly only one or two inodes are allocated from each block.
Quoted 2 lines
The question is: How do I get the filesystem to perform normally again?
It is probably worthwhile to do a blktrace to see what the IO is looking like, and as Eric suggested the oprofile information may be interesting if a lot of CPU is being used.
Quoted 9 lines
All mkdir calls are taking around 62 milliseconds:
14:57:02
mkdir("...ME_25c8e/", 0777)= 0 <0.062312> 14:57:02mkdir("...ME_112ff/", 0777)= 0 <0.062658> 14:57:02mkdir("...ME_183cf/", 0777)= 0 <0.062344>while on my workstation where i'm not supposed to be working with client-data I get:
Quoted 20 lines
14:58:18
mkdir("...ME_2a6c6/", 0777)= 0 <0.000033> 14:58:18mkdir("...ME_2b3d9/", 0777)= 0 <0.000031> 14:58:18mkdir("...ME_2c63f/", 0777)= 0 <0.000034>So, things on my big datastorage drive are about 2000 times slower than on my workstation.
Roger.
Signature · 10 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 ** ** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 ** *-- BitWizard writes Linux device drivers for any device you may have! --* Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. Does it sit on the couch all day? Is it unemployed? Please be specific! Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Cheers, Andreas
Quoted 17 lines · Andreas Dilger
Quoted 7 lines · Rogier Wolff
My work (as in what brings in money to buy things from) file-system is now "slow as hell". Last time this happened you guys told me that this was because I had millions of hardlinks and way too many files.
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md0815M 4.1M 811M 1% /recover2Quoted 8 lines
I have only 4.1M files. Only 1% inodes "in use".
The filesystem was filling up lately, but i've managed to create some free space.
Filesystem Size Used Avail Use% Mounted on /dev/md0 6.3T 5.3T 720G 89% /recover2
Quoted 6 lines
I suspect that the free inodes in your filesystem are very fragmented, due to the filesystem filling up and then deleting a lot of files that create "holes" of free inodes within the inode table blocks. That means that possibly finding free inodes is a bunch of work, and then possibly only one or two inodes are allocated from each block.
How can the free inodes be fragmented?!? 99% of the available inodes ARE free.
Quoted 6 lines
Quoted 2 lines
The question is: How do I get the filesystem to perform normally again?
It is probably worthwhile to do a blktrace to see what the IO is looking like, and as Eric suggested the oprofile information may be interesting if a lot of CPU is being used.
A lot of CPU time is being used, yes. The 62 milliseconds are almost completely CPU time.
I was running a big "backup" (image of a 500Gb disk) onto the partition. Now that this has finished I'll try booting into a recent kernel. Next I'll try finding and applying the patch....
Roger.
Signature · 6 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 ** ** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 ** *-- BitWizard writes Linux device drivers for any device you may have! --* Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. Does it sit on the couch all day? Is it unemployed? Please be specific! Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ