Skip to contentSkip to conversation

Slow filesystem.

DISCUSSION Rogier Wolff 3 participants 4 messages May 13, 2011 – May 14, 2011

lore
Share & tools
lore
#0 lore raw

Hi

My work (as in what brings in money to buy things from) file-system is now "slow as hell". Last time this happened you guys told me that this was because I had millions of hardlinks and way too many files.

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md0 815M 4.1M 811M 1% /recover2

I have only 4.1M files. Only 1% inodes "in use".

The filesystem was filling up lately, but i've managed to create some free space.

Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              6.3T  5.3T  720G  89% /recover2

The question is: How do I get the filesystem to perform
normally again?

All mkdir calls are taking around 62 milliseconds:

14:57:02 mkdir("...ME_25c8e/", 0777) = 0 <0.062312> 14:57:02 mkdir("...ME_112ff/", 0777) = 0 <0.062658> 14:57:02 mkdir("...ME_183cf/", 0777) = 0 <0.062344>

while on my workstation where i'm not supposed to be working with client-data I get:

14:58:18 mkdir("...ME_2a6c6/", 0777) = 0 <0.000033> 14:58:18 mkdir("...ME_2b3d9/", 0777) = 0 <0.000031> 14:58:18 mkdir("...ME_2c63f/", 0777) = 0 <0.000034>

So, things on my big datastorage drive are about 2000 times slower than on my workstation.

Roger.

Signature · 6 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
#1 lore raw
Quoted 23 lines · Rogier Wolff

Hi

My work (as in what brings in money to buy things from) file-system is now "slow as hell". Last time this happened you guys told me that this was because I had millions of hardlinks and way too many files.

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md0 815M 4.1M 811M 1% /recover2

I have only 4.1M files. Only 1% inodes "in use".

The filesystem was filling up lately, but i've managed to create some free space.

Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              6.3T  5.3T  720G  89% /recover2

The question is: How do I get the filesystem to perform
normally again?

Quoted 5 lines

All mkdir calls are taking around 62 milliseconds:

14:57:02 mkdir("...ME_25c8e/", 0777) = 0 <0.062312> 14:57:02 mkdir("...ME_112ff/", 0777) = 0 <0.062658> 14:57:02 mkdir("...ME_183cf/", 0777) = 0 <0.062344>

I don't know what the workload is, and "slow" isn't very well characterized here, but from this can I assume that mkdir is the bottleneck?

Did you ever try that patch I sent you for this issue in March?

Using oprofile to see where the fs is spending its time might also be useful.

-Eric

Quoted 12 lines

while on my workstation where i'm not supposed to be working with client-data I get:

14:58:18 mkdir("...ME_2a6c6/", 0777) = 0 <0.000033> 14:58:18 mkdir("...ME_2b3d9/", 0777) = 0 <0.000031> 14:58:18 mkdir("...ME_2c63f/", 0777) = 0 <0.000034>

So, things on my big datastorage drive are about 2000 times slower than on my workstation.

Roger.

#2 lore raw
In reply to Rogier Wolff
Quoted 7 lines · Rogier Wolff

My work (as in what brings in money to buy things from) file-system is now "slow as hell". Last time this happened you guys told me that this was because I had millions of hardlinks and way too many files.

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md0 815M 4.1M 811M 1% /recover2

Quoted 8 lines

I have only 4.1M files. Only 1% inodes "in use".

The filesystem was filling up lately, but i've managed to create some free space.

Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              6.3T  5.3T  720G  89% /recover2

I suspect that the free inodes in your filesystem are very fragmented, due to the filesystem filling up and then deleting a lot of files that create "holes" of free inodes within the inode table blocks. That means that possibly finding free inodes is a bunch of work, and then possibly only one or two inodes are allocated from each block.

Quoted 2 lines

The question is: How do I get the filesystem to perform normally again?

It is probably worthwhile to do a blktrace to see what the IO is looking like, and as Eric suggested the oprofile information may be interesting if a lot of CPU is being used.

Quoted 9 lines

All mkdir calls are taking around 62 milliseconds:

14:57:02 mkdir("...ME_25c8e/", 0777) = 0 <0.062312> 14:57:02 mkdir("...ME_112ff/", 0777) = 0 <0.062658> 14:57:02 mkdir("...ME_183cf/", 0777) = 0 <0.062344>

while on my workstation where i'm not supposed to be working with client-data I get:

Quoted 20 lines

14:58:18 mkdir("...ME_2a6c6/", 0777) = 0 <0.000033> 14:58:18 mkdir("...ME_2b3d9/", 0777) = 0 <0.000031> 14:58:18 mkdir("...ME_2c63f/", 0777) = 0 <0.000034>

So, things on my big datastorage drive are about 2000 times slower than on my workstation.

Roger.

Signature · 10 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Cheers, Andreas

#3 lore raw
Quoted 17 lines · Andreas Dilger
Quoted 7 lines · Rogier Wolff

My work (as in what brings in money to buy things from) file-system is now "slow as hell". Last time this happened you guys told me that this was because I had millions of hardlinks and way too many files.

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md0 815M 4.1M 811M 1% /recover2

Quoted 8 lines

I have only 4.1M files. Only 1% inodes "in use".

The filesystem was filling up lately, but i've managed to create some free space.

Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              6.3T  5.3T  720G  89% /recover2
Quoted 6 lines

I suspect that the free inodes in your filesystem are very fragmented, due to the filesystem filling up and then deleting a lot of files that create "holes" of free inodes within the inode table blocks. That means that possibly finding free inodes is a bunch of work, and then possibly only one or two inodes are allocated from each block.

How can the free inodes be fragmented?!? 99% of the available inodes ARE free.

Quoted 6 lines
Quoted 2 lines

The question is: How do I get the filesystem to perform normally again?

It is probably worthwhile to do a blktrace to see what the IO is looking like, and as Eric suggested the oprofile information may be interesting if a lot of CPU is being used.

A lot of CPU time is being used, yes. The 62 milliseconds are almost completely CPU time.

I was running a big "backup" (image of a 500Gb disk) onto the partition. Now that this has finished I'll try booting into a recent kernel. Next I'll try finding and applying the patch....

Roger.

Signature · 6 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ