10GB memorys occupied by XFS

Hello folks,

I used xfs file system in kernel-2.6.32-220.13.1.el6.x86_64 for store pictures. About 100 days system memorys is lost and some nginx process is killed by oom-killer.So,I looked /proc/meminfo and find memorys is lost.Finally, I try to umount xfs system and 10GB memorys is coming back. l look xfs bugzilla no such BUG.I have no idea for it.

Cheers,

Guochao.

some memorys info:

0> free -m
total used free shared buffers cached

Mem:         11887      11668        219          0          0          2
-/+ buffers/cache:      11665        222

Swap: 0 0 0

130> cat /proc/meminfo

MemTotal:       12173268 kB
MemFree:          223044 kB
Buffers:             244 kB
Cached:             4540 kB
SwapCached:            0 kB
Active:             1700 kB
Inactive:           5312 kB
Active(anon):       1616 kB
Inactive(anon):     1128 kB
Active(file):         84 kB
Inactive(file):     4184 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          2556 kB
Mapped:             1088 kB
Shmem:               196 kB
Slab:             509708 kB
SReclaimable:       7596 kB
SUnreclaim:       502112 kB
KernelStack:        1096 kB
PageTables:          748 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6086632 kB
Committed_AS:       9440 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      303488 kB
VmallocChunk:   34359426132 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        6152 kB
DirectMap2M:     2070528 kB
DirectMap1G:    10485760 kB

--
View this message in context: http://xfs.9218.n7.nabble.com/10GB-memorys-occupied-by-XFS-tp35015.html
Sent from the Xfs - General mailing list archive at Nabble.com.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Hello,

Note that your problems are not XFS specific, but can occur with any Linux filesystem.

First problem:  no swap
Second problem: cache is not being reclaimed

Read vfs_cache_pressure at:
https://www.kernel.org/doc/Documentation/sysctl/vm.txt

You've likely set this value to zero. Changing it to 200 should prompt the kernel to reclaim dentries and inodes aggressively, preventing the oom-killer from kicking in.

Cheers,

Stan

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Dear Stan,
Thank you for your kind assistance.

In accordance with your suggestion, we executed "echo 3 > /proc/sysm/drop_caches" for trying to release vfs dentries and inodes. Really, our lost memory came back. But we learned that the memory of vfs dentries and inodes is distributed from slab. Please check our system "Slab:  509708 kB" from /proc/meminfo, and it seems only be took up 500MB and xfs_buf take up 450MB among. And /proc/meminfo indicated that our system memory is anomalous, there is about 10GB out of the statistics. We want to know how the system could observe the usage amount of vfs dentries and iodes through the system interface. If the memory usage of system is not reflected in /proc/meminfo as we can not find the statistics, and we thought it as a bug of xfs.

My  vm.vfs_cache_pressure of linux system is 100. We think that the system will proactively take the memory back when the memory is not enough, rather than oom-killer kills our work process. Our datas of /proc/meminfo occurred during the system problem as below:
130> cat /proc/meminfo
MemTotal:       12173268 kB
MemFree:          223044 kB
Buffers:             244 kB
Cached:             4540 kB
SwapCached:            0 kB
Active:             1700 kB
Inactive:           5312 kB
Active(anon):       1616 kB
Inactive(anon):     1128 kB
Active(file):         84 kB
Inactive(file):     4184 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          2556 kB
Mapped:             1088 kB
Shmem:               196 kB
Slab:             509708 kB
SReclaimable:       7596 kB
SUnreclaim:       502112 kB
KernelStack:        1096 kB
PageTables:          748 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6086632 kB
Committed_AS:       9440 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      303488 kB
VmallocChunk:   34359426132 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        6152 kB
DirectMap2M:     2070528 kB
DirectMap1G:    10485760 kB

I look forward to hearing from you and thank you very much for your kind assistance.

Best Regards,

Guochao

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

To free pagecache:
echo 1 > /proc/sys/vm/drop_caches
To free reclaimable slab objects (includes dentries and inodes):
echo 2 > /proc/sys/vm/drop_caches
To free slab objects and pagecache:
echo 3 > /proc/sys/vm/drop_caches

It seems much of this 10 GB of memory is being consumed by pagecache, not dentries and inodes. So the question is:

Why is pagecache not being reclaimed without manual intervention?

Except most of your slab is reported as not reclaimable, see below.

4.5 MB is reported use by the pagecache. But some 10 GB is being consumed by the page cache and not reported here.

This indicates that your slab is not being reclaimed, but not why.

Unfortunately I don't have solid answers for you at this point, nor a solution. This is beyond my expertise. Maybe someone else with more knowledge/experience will jump in. I suspect your application may be doing something a bit unusual.

Cheers,

Stan

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Dear Stan, I can't send email to you.So I leave a message here.I hope not to bother you.
Thank you for your kind assistance.

In accordance with your suggestion, we executed "echo 3 > /proc/sysm/drop_caches" for trying to release vfs dentries and inodes. Really,
our lost memory came back. But we learned that the memory of vfs dentries and inodes is distributed from slab. Please check our system "Slab: 509708 kB" from /proc/meminfo, and it seems only be took up 500MB and xfs_buf take up 450MB among. And /proc/meminfo indicated that our system memory is anomalous, there is about 10GB out of the statistics. We want to know how the system could observe the usage amount of vfs dentries and iodes through the system interface. If the memory usage of system is not reflected in /proc/meminfo as we can not find the statistics, and we thought it as a bug of xfs.

My vm.vfs_cache_pressure of linux system is 100. We think that the system will proactively take the memory back when the memory is not enough, rather than oom-killer kills our work process. Our datas of /proc/meminfo occurred during the system problem as below:
130> cat /proc/meminfo

MemTotal:       12173268 kB 
MemFree:          223044 kB 
Buffers:             244 kB 
Cached:             4540 kB 
SwapCached:            0 kB 
Active:             1700 kB 
Inactive:           5312 kB 
Active(anon):       1616 kB 
Inactive(anon):     1128 kB 
Active(file):         84 kB 
Inactive(file):     4184 kB 
Unevictable:           0 kB 
Mlocked:               0 kB 
SwapTotal:             0 kB 
SwapFree:              0 kB 
Dirty:                 0 kB 
Writeback:             0 kB 
AnonPages:          2556 kB 
Mapped:             1088 kB 
Shmem:               196 kB 
Slab:             509708 kB 
SReclaimable:       7596 kB 
SUnreclaim:       502112 kB 
KernelStack:        1096 kB 
PageTables:          748 kB 
NFS_Unstable:          0 kB 
Bounce:                0 kB 
WritebackTmp:          0 kB 
CommitLimit:     6086632 kB 
Committed_AS:       9440 kB 
VmallocTotal:   34359738367 kB 
VmallocUsed:      303488 kB 
VmallocChunk:   34359426132 kB 
HardwareCorrupted:     0 kB 
AnonHugePages:         0 kB 
HugePages_Total:       0 
HugePages_Free:        0 
HugePages_Rsvd:        0 
HugePages_Surp:        0 
Hugepagesize:       2048 kB 
DirectMap4k:        6152 kB 
DirectMap2M:     2070528 kB 
DirectMap1G:    10485760 kB

Best Regards,

Guochao

--
View this message in context: http://xfs.9218.n7.nabble.com/10GB-memorys-occupied-by-XFS-tp35015p35016.html
Sent from the Xfs - General mailing list archive at Nabble.com.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

That's where your memory is - in metadata buffers. The xfs_buf slab entries are just the handles - the metadata pages in the buffers usually take much more space and it's not accounted to the slab cache nor the page cache.

Can you post the output of /proc/slabinfo, and what is the output of xfs_info on the filesystem in question? Also, a description of your workload that is resulting in large amounts of cached metadata buffers but no inodes or dentries would be helpful.

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

I received all of the ones you sent to the list and that should always be the case. One that you sent directly to me was rejected but I think I've fixed that now. And I think my delayed reply made things seem worse than they are.

Anyway, Dave replied while I was typing my last response. He'll be much more able to assist you. Your problem seems beyond the edge of my knowledge.

Cheers,

Stan

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs