Skip to contentSkip to conversation

2.4.25 - large inode_cache

DISCUSSION Jakob Oestergaard 4 participants 9 messages Feb 26, 2004 – Feb 27, 2004

lore
Share & tools
lore
#0 lore raw

Dear list,

I have this dual athlon box with 1G memory and a 150G filesystem (four IDE disks on promise controllers, SW RAID-0+1, ext3fs, user quotas, HIGHMEM set, plain 2.4.25).

A fresh boot after an unclean shutdown, caused it to run quotacheck on the filesystem, nothing odd about that.

However, after quotacheck completed, I got "3 order allocation failed" messages. They kept coming, about one per second. This was happening as the box entered single user mode - and the messages continued.

Quoted 1 line

From /proc/slabinfo, I got:

inode_cache 1208571 1208571 512 172653 172653 1 : 124 62
dentry_cache 736 268680 128 27 8956 1 : 252 126

To me it looks like this could be at least a part of the explanation for the memory shortage - am I completely off track here?

After a clean boot with no quotacheck, it looks like:

inode_cache         3829   3829    512  547  547    1 :  124   62
dentry_cache        4710   4710    128  157  157    1 :  252  126

Besides, after a few days of running, the machine will use about 100MB of memory for cache, 100MB for buffers, about 100MB for userspace, and the remaining 600-700 MB of memory for inode_cache and dentry_cache.

It's a file server, and its performance is far from stellar. After seeing that only about 200MB total was used for cache/buffers, I started digging into slabinfo.

Is this a known problem? (yes, I know that there's been quite a bit back and forward on this list about the slabs, but I'm not sure what the current status is - as far as I know, the only known long-standing problems with the 2.4 series VM should have been fixed in 2.4.25).

If not, is there anything I can do to actually find out what the cause
of my poor cache sizes are? I believe that a box which does almost
strictly NFS file serving and has 1G of memory, should use more than
100M for cache. Or is that just me? (no, there is no significant
amount of free memory, virtually all is used - but not by cache and not by userspace).

If it is a known problem - any workarounds?

Would 2.6 solve it?

Thanks, for any input you may have,

/ jakob

#1 lore raw
Quoted 3 lines · Jakob Oestergaard

Besides, after a few days of running, the machine will use about 100MB of memory for cache, 100MB for buffers, about 100MB for userspace, and the remaining 600-700 MB of memory for inode_cache and dentry_cache.

I have the same problem (it's an dual PIII NFS fileserver with promise sx6000 raid, 320 GB ext3 filesystem and only 512 MB Ram).

After only 2 days running the bloatmeter output looks like:

   inode_cache:   336585KB   357234KB   94.21
  dentry_cache:    50305KB    56523KB   88.99
       size-32:     1516KB     1695KB   89.46

free output is this:

             total       used       free     shared    buffers  cached
Mem:        515980     506464       9516          0    2272      19204

-/+ buffers/cache: 484988 30992
Swap: 1951856 7992 1943864

Regards

Christian Leber

Signature · 3 lines
  "Omnis enim res, quae dando non deficit, dum habetur et non datur,
   nondum habetur, quomodo habenda est."       (Aurelius Augustinus)
  Translation: <http://gnuhh.org/work/fsf-europe/augustinus.html>
#2 lore raw
Quoted 13 lines · Christian Leber
Quoted 3 lines · Jakob Oestergaard

Besides, after a few days of running, the machine will use about 100MB of memory for cache, 100MB for buffers, about 100MB for userspace, and the remaining 600-700 MB of memory for inode_cache and dentry_cache.

I have the same problem (it's an dual PIII NFS fileserver with promise sx6000 raid, 320 GB ext3 filesystem and only 512 MB Ram).

After only 2 days running the bloatmeter output looks like:

   inode_cache:   336585KB   357234KB   94.21
  dentry_cache:    50305KB    56523KB   88.99
       size-32:     1516KB     1695KB   89.46
Quoted 5 lines

free output is this:

             total       used       free     shared    buffers  cached
Mem:        515980     506464       9516          0    2272      19204

-/+ buffers/cache: 484988 30992
Swap: 1951856 7992 1943864

This should be normal behaviour -- the i/d caches grew because of file system activitity. This memory will be reclaimed in case theres pressure.

Is the behaviour different from previous 2.4 or 2.6 kernels?

#3 lore raw

On Thu, Feb 26, 2004 at 10:08:23AM -0300, Marcelo Tosatti wrote:
...

Quoted 9 lines
Quoted 6 lines

free output is this:

             total       used       free     shared    buffers  cached
Mem:        515980     506464       9516          0    2272      19204

-/+ buffers/cache: 484988 30992
Swap: 1951856 7992 1943864

This should be normal behaviour -- the i/d caches grew because of file system activitity. This memory will be reclaimed in case theres pressure.

But how is "pressure" defined?

Will a heap of busy knfsd processes doing reads or writes exert
pressure? Or is it only local userspace that can pressurize the VM (by
either anonymously backed memory or file I/O).

This server happily serves large home directories over NFS, at really poor speeds. It will happily serve tens or hundreds of gigabytes, read and write, over the course of a day, and *still* only cache about 100MB NFS to/from the server is slow. It's common to see 10 knfsd processes in D state while vmstat tells me the array works with about 4-6MB/sec sustained throughput (where hdparm -t would give me more than 70MB/sec on the md device).

The files read and written are commonly in the 20-60 MB range, so it's not just because I'm loading the server with small seeks. Many files are read multiple times within a few minutes, so the cache usage of 100MB is completely bogus the way that I see it - but maybe there's just
something I don't know about the caching? :)

Quoted 2 lines

Is the behaviour different from previous 2.4 or 2.6 kernels?

I never investigated the slabinfo on earlier 2.4. But the performance on this server has been "under expectations" for as long as I can remember. So, from the performance experience on this server I would say that 2.4.25 is not any worse than older kernels.

Since this is a production system I have been reluctant to jump on the 2.6 wagon - but my other experiences with 2.6.X have been good, so I'm probably going to soften up and give it a try in a not too distant future.

However, if this dcache/icache problem is well known and is (or at least should be) solved in 2.6, then I can do the test this weekend.

Any enlightenment or suggestions are greatly appreciated :)

Thanks,

/ jakob

#4 lore raw
Quoted 9 lines · Jakob Oestergaard

On Thu, Feb 26, 2004 at 10:08:23AM -0300, Marcelo Tosatti wrote:
...

Quoted 7 lines
Quoted 6 lines

free output is this:

             total       used       free     shared    buffers  cached
Mem:        515980     506464       9516          0    2272      19204

-/+ buffers/cache: 484988 30992
Swap: 1951856 7992 1943864

Quoted 8 lines
Quoted 2 lines

This should be normal behaviour -- the i/d caches grew because of file system activitity. This memory will be reclaimed in case theres pressure.

But how is "pressure" defined?

Will a heap of busy knfsd processes doing reads or writes exert
pressure? Or is it only local userspace that can pressurize the VM (by
either anonymously backed memory or file I/O).

Any allocator will cause VM pressure.

Quoted 8 lines

This server happily serves large home directories over NFS, at really poor speeds. It will happily serve tens or hundreds of gigabytes, read and write, over the course of a day, and *still* only cache about 100MB NFS to/from the server is slow. It's common to see 10 knfsd processes in D state while vmstat tells me the array works with about 4-6MB/sec sustained throughput (where hdparm -t would give me more than 70MB/sec on the md device).

Quoted 5 lines

The files read and written are commonly in the 20-60 MB range, so it's not just because I'm loading the server with small seeks. Many files are read multiple times within a few minutes, so the cache usage of 100MB is completely bogus the way that I see it - but maybe there's just
something I don't know about the caching? :)

It sounds the cache could grow larger for better performance, indeed.

Quoted 12 lines
Quoted 1 line

Is the behaviour different from previous 2.4 or 2.6 kernels?

I never investigated the slabinfo on earlier 2.4. But the performance on this server has been "under expectations" for as long as I can remember. So, from the performance experience on this server I would say that 2.4.25 is not any worse than older kernels.

Since this is a production system I have been reluctant to jump on the 2.6 wagon - but my other experiences with 2.6.X have been good, so I'm probably going to soften up and give it a try in a not too distant future.

Quoted 4 lines

However, if this dcache/icache problem is well known and is (or at least should be) solved in 2.6, then I can do the test this weekend.

Any enlightenment or suggestions are greatly appreciated :)

What you can try is to increase the VM tunable vm_vfs_scan_ratio. This is the proportion of VFS unused d/i caches that will try to be in one VM freeing pass. The default is 6. Try 4 or 3.

/proc/sys/vm/vm_vfs_scan_ratio

You can also play with

/proc/sys/vm/vm_cache_scan_ratio (which is the percentage of cache which will be scanned in one go).

#5 lore raw

On Thu, Feb 26, 2004 at 11:23:46AM -0300, Marcelo Tosatti wrote:
...

Quoted 5 lines
Quoted 3 lines

Will a heap of busy knfsd processes doing reads or writes exert
pressure? Or is it only local userspace that can pressurize the VM (by
either anonymously backed memory or file I/O).

Any allocator will cause VM pressure.

And I suppose that a busy knfsd qualifies as an "allocator" :)

...

Quoted 7 lines
Quoted 1 line

Any enlightenment or suggestions are greatly appreciated :)

What you can try is to increase the VM tunable vm_vfs_scan_ratio. This is the proportion of VFS unused d/i caches that will try to be in one VM freeing pass. The default is 6. Try 4 or 3.

/proc/sys/vm/vm_vfs_scan_ratio

Done! Set to 3 now - I will let the box run with this setting until tomorrow, and report back how things look.

Quoted 4 lines

You can also play with

/proc/sys/vm/vm_cache_scan_ratio (which is the percentage of cache which will be scanned in one go).

I'm leaving this one be for now (one variable at a time). But let's see what tomorrow brings.

Judging from the code, it seems that it's the vm_vfs_scan_ratio that directly affects the icache/dcache and dquot - but I'm sure that there are subtle interactions far beyond what I can possibly hope to comprehend ;)

Thanks a lot for your suggestions Marcelo!

/ jakob

#6 lore raw
In reply to Marcelo Tosatti
Quoted 5 lines · Marcelo Tosatti
Quoted 4 lines · Jakob Oestergaard
Quoted 2 lines · Marcelo Tosatti

This should be normal behaviour -- the i/d caches grew because of file system activitity. This memory will be reclaimed in case theres pressure.

Quoted 7 lines
Quoted 5 lines

But how is "pressure" defined?

Will a heap of busy knfsd processes doing reads or writes exert
pressure? Or is it only local userspace that can pressurize the VM (by
either anonymously backed memory or file I/O).

Any allocator will cause VM pressure.

But won't all of the knfsd allocations be by necessity GFP_NOFS to avoid deadlocks, so they will be unable to clear inodes or dentries? Both shrink_icache_memory() and shrink_dcache_memory() do nothing if __GFP_FS isn't set so if there is no user-space allocation pressure we will never get into the dcache/icache freeing paths from knfsd allocations.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

#7 lore raw
Quoted 12 lines · Andreas Dilger
Quoted 11 lines · Marcelo Tosatti
Quoted 9 lines · Jakob Oestergaard
Quoted 2 lines · Marcelo Tosatti

This should be normal behaviour -- the i/d caches grew because of file system activitity. This memory will be reclaimed in case theres pressure.

But how is "pressure" defined?

Will a heap of busy knfsd processes doing reads or writes exert
pressure? Or is it only local userspace that can pressurize the VM (by
either anonymously backed memory or file I/O).

Quoted 7 lines
Quoted 1 line

Any allocator will cause VM pressure.

But won't all of the knfsd allocations be by necessity GFP_NOFS to avoid deadlocks, so they will be unable to clear inodes or dentries? Both shrink_icache_memory() and shrink_dcache_memory() do nothing if __GFP_FS isn't set so if there is no user-space allocation pressure we will never get into the dcache/icache freeing paths from knfsd allocations.

Hi Andreas,

AFAICS knfsd does not allocate using GFP_NOFS.

#8 lore raw
Quoted 1 line · Marcelo Tosatti

...

Quoted 11 lines
Quoted 7 lines
Quoted 1 line

Any allocator will cause VM pressure.

But won't all of the knfsd allocations be by necessity GFP_NOFS to avoid deadlocks, so they will be unable to clear inodes or dentries? Both shrink_icache_memory() and shrink_dcache_memory() do nothing if __GFP_FS isn't set so if there is no user-space allocation pressure we will never get into the dcache/icache freeing paths from knfsd allocations.

Hi Andreas,

AFAICS knfsd does not allocate using GFP_NOFS.

So far, it looks like the vm_vfs_scan_ratio setting of '3' (instead of '6' which was the default) made a big difference.

Currently the machine is using about ~600MB for cache (up from ~100MB), and about 150MB for buffers (up from 80-100MB).

I'm going to let it run a little longer just with this setting, before experimenting further.

I'll ask around about the perceived performance of the machine, and pay attention to it myself. I'll post the results monday or tuesday (not a lot of interactive users during the weekend ;)

Again, thanks a lot for the quick feedback!

/ jakob