missing memory counter introduced by xfs
Hi all nice xfs folks,
I'm a rookie and really fresh new in xfs and currently I ran into an issue same as the following link described:
http://oss.sgi.com/archives/xfs/2014-04/msg00058.html
In my box(running cephfs osd using xfs kernel 2.6.32-358) and I sum all possible memory counter can be find but it seems that nearlly 26GB memory has gone and they are back after I echo 2 > /proc/sys/vm/drop_caches, so seems these memory can be reclaimed by slab. But it my box kernel use swap instead reclaim slab until I run echo 2 > /proc/sys/vm/drop_caches.
Following memory usage is caught by my shell script, and the slabinfo/meminfo pasted at the end of this mail.
-----
before echo 1 > /proc/sys/vm/drop_caches
Analysis: all processes rss + buffer+cached + slabs + free, total Rss: 39863308 K
total used free shared buffers cached
Mem: 65963504 58230212 7733292 0 31284 6711912
-/+ buffers/cache: 51487016 14476488
Swap: 8388600 0 8388600
after echo 1 > /proc/sys/vm/drop_caches
Analysis: all processes rss + buffer+cached + slabs + free, total Rss:
39781110 K
free info:
total used free shared buffers cached
Mem: 65963504 51666124 14297380 0 3376 55704 -/+ buffers/cache: 51607044 14356460
Swap: 8388600 0 8388600
after echo 2 > /proc/sys/vm/drop_caches
Analysis: all processes rss + buffer+cached + slabs + free, total Rss:
65259244 K
total used free shared buffers cached
Mem: 65963504 17194480 48769024 0 7948 53216 -/+ buffers/cache: 17133316 48830188
Swap: 8388600 0 8388600
-----
And according to what David said replying in the list:
David's mail contents quotes start
-----
Quoted 11 lines · daiguochao
Dear Stan, I can't send email to you.So I leave a message here.I hope not to bother you.
Thank you for your kind assistance.In accordance with your suggestion, we executed "echo 3 >
/proc/sysm/drop_caches" for trying to release vfs dentries and inodes. Really,
our lost memory came back. But we learned that the memory of vfs dentries and inodes is distributed from slab. Please check our system "Slab: 509708 kB" from/proc/meminfo, and it seems only be took up 500MB andxfs_buftake up 450MB among.
That's where your memory is - in metadata buffers. The xfs_buf slab entries are just the handles - the metadata pages in the buffers usually take much more space and it's not accounted to the slab cache nor the page cache.
Can you post the output of /proc/slabinfo, and what is the output of xfs_info on the filesystem in question? Also, a description of your workload that is resulting in large amounts of cached metadata buffers but no inodes or dentries would be helpful.
-----
David's mail contents quotes end
After some research it seems that after this patch(commit 0e6e847ffe37) we use alloc_page in xfs_buf_allocate_memory() instead of original find_or_create_page. Ps. mainline kernel still uses alloc_page.
So if my speculation was right, my problem is if there a way to find out how much memory that xfs_buf_t->b_pages are using or if xfs has already exported such counter to user space.
thanks in advance.
linfeng
-----
commit 0e6e847ffe37436e331c132639f9f872febce82e
Author: Dave Chinner <[email protected]> Date: Sat Mar 26 09:16:45 2011 +1100
xfs: stop using the page cache to back the buffer cache
Now that the buffer cache has it's own LRU, we do not need to use the page cache to provide persistent caching and reclaim infrastructure. Convert the buffer cache to use alloc_pages() instead of the page cache. This will remove all the overhead of page cache management from setup and teardown of the buffers, as well as needing to mark pages accessed as we find buffers in the buffer cache.
...
- retry:
- page = find_or_create_page(mapping, first + i, gfp_mask);
+retry:
+ page = alloc_page(gfp_mask);
-----
slabtop info:
Active / Total Objects (% used) : 27396369 / 27446160 (99.8%) Active / Total Slabs (% used) : 2371663 / 2371729 (100.0%) Active / Total Caches (% used) : 112 / 197 (56.9%) Active / Total Size (% used) : 9186047.17K / 9202410.61K (99.8%)
Minimum / Average / Maximum Object : 0.02K / 0.33K / 4096.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
4383036 4383014 99% 1.00K 1095759 4 4383036K xfs_inode 5394610 5394544 99% 0.38K 539461 10 2157844K xfs_buf 6448560 6448451 99% 0.19K 322428 20 1289712K dentry 1083285 1062902 98% 0.55K 154755 7 619020K radix_tree_node 3015600 3015546 99% 0.12K 100520 30 402080K size-128 4379806 4379430 99% 0.06K 74234 59 296936K xfs_ifork 687640 687144 99% 0.19K 34382 20 137528K size-192 1833130 1833089 99% 0.06K 31070 59 124280K size-64 1060 1059 99% 16.00K 1060 1 16960K size-16384 196 196 100% 32.12K 196 1 12544K kmem_cache 4332 4316 99% 2.59K 1444 3 11552K task_struct 15900 15731 98% 0.62K 2650 6 10600K proc_inode_cache 8136 7730 95% 1.00K 2034 4 8136K size-1024 9930 9930 100% 0.58K 1655 6 6620K inode_cache 20700 14438 69% 0.19K 1035 20 4140K filp 3704 3691 99% 1.00K 926 4 3704K ext4_inode_cache 17005 15631 91% 0.20K 895 19 3580K vm_area_struct 18090 18043 99% 0.14K 670 27 2680K sysfs_dir_cache 1266 1254 99% 1.94K 633 2 2532K TCP 885 885 100% 2.06K 295 3 2360K sighand_cache
/proc/meminfo
MemTotal: 65963504 kB MemFree: 14296540 kB Buffers: 3380 kB Cached: 55700 kB SwapCached: 0 kB Active: 15717512 kB Inactive: 306828 kB Active(anon): 15699604 kB Inactive(anon): 268724 kB Active(file): 17908 kB Inactive(file): 38104 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 8388600 kB SwapFree: 8388600 kB Dirty: 72 kB Writeback: 0 kB AnonPages: 15966248 kB Mapped: 33668 kB Shmem: 3056 kB Slab: 9521800 kB SReclaimable: 6314860 kB SUnreclaim: 3206940 kB KernelStack: 32680 kB PageTables: 51504 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 61159400 kB Committed_AS: 29734944 kB VmallocTotal: 34359738367 kB VmallocUsed: 389896 kB VmallocChunk: 34324818076 kB HardwareCorrupted: 0 kB AnonHugePages: 407552 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 5504 kB DirectMap2M: 2082816 kB DirectMap1G: 65011712 kB
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 11 lines · Lin Feng
Hi all nice xfs folks,
I'm a rookie and really fresh new in xfs and currently I ran into an issue same as the following link described:
http://oss.sgi.com/archives/xfs/2014-04/msg00058.htmlIn my box(running cephfs osd using xfs kernel 2.6.32-358) and I sum all possible memory counter can be find but it seems that nearlly 26GB memory has gone and they are back after I echo 2 >
/proc/sys/vm/drop_caches, so seems these memory can be reclaimed by slab.
It isn't "reclaimed by slab". The XFS metadata buffer cache is reclaimed by a memory shrinker, which are for reclaiming objects from caches that aren't the page cache. "echo 2 > /proc/sys/vm/drop_caches" runs the memory shrinkers rather than page cache reclaim. Many slab caches are backed by memory shrinkers, which is why it is thought that "2" is "slab reclaim"....
Quoted 1 line
And according to what David said replying in the list:
..
Quoted 4 lines
That's where your memory is - in metadata buffers. The
xfs_bufslab entries are just the handles - the metadata pages in the buffers usually take much more space and it's not accounted to the slab cache nor the page cache.
That's exactly the case.
Quoted 5 lines
Minimum / Average / Maximum Object : 0.02K / 0.33K / 4096.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
4383036 4383014 99% 1.00K 1095759 4 4383036K xfs_inode 5394610 5394544 99% 0.38K 539461 10 2157844K xfs_buf
So, you have *5.4 million* active metadata buffers. Each buffer will hold 1 or 2 4k pages on your kernel, so simple math says 4M * 4k + 1.4M * 8k = 26G. There's no missing counter here....
Obviously your workload is doing something extremely metadata intensive to have a cache footprint like this - you have more cached buffers than inodes, dentries, etc. That in itself is very unusual - can you describe what is stored on that filesystem and how large the attributes being stored in each inode are?
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Hi Dave,
Thank you for your fast reply, look beblow please.
Quoted 31 lines · Dave Chinner
Quoted 11 lines · Lin Feng
Hi all nice xfs folks,
I'm a rookie and really fresh new in xfs and currently I ran into an issue same as the following link described:
http://oss.sgi.com/archives/xfs/2014-04/msg00058.htmlIn my box(running cephfs osd using xfs kernel 2.6.32-358) and I sum all possible memory counter can be find but it seems that nearlly 26GB memory has gone and they are back after I echo 2 >
/proc/sys/vm/drop_caches, so seems these memory can be reclaimed by slab.It isn't "reclaimed by slab". The XFS metadata buffer cache is reclaimed by a memory shrinker, which are for reclaiming objects from caches that aren't the page cache. "echo 2 >
/proc/sys/vm/drop_caches" runs the memory shrinkers rather than page cache reclaim. Many slab caches are backed by memory shrinkers, which is why it is thought that "2" is "slab reclaim"....Quoted 1 line
And according to what David said replying in the list:
..
Quoted 4 lines
That's where your memory is - in metadata buffers. The
xfs_bufslab entries are just the handles - the metadata pages in the buffers usually take much more space and it's not accounted to the slab cache nor the page cache.That's exactly the case.
Quoted 2 lines
Minimum / Average / Maximum Object : 0.02K / 0.33K / 4096.00K
Quoted 7 lines
Quoted 3 lines
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
4383036 4383014 99% 1.00K 1095759 4 4383036K xfs_inode 5394610 5394544 99% 0.38K 539461 10 2157844K xfs_bufSo, you have *5.4 million* active metadata buffers. Each buffer will hold 1 or 2 4k pages on your kernel, so simple math says 4M * 4k + 1.4M * 8k = 26G. There's no missing counter here....
Does xattr contribute to such metadata buffers or there is something else? After consulting to my teammate, who told me that in our case small files (there are a looot, look below) always use xattr.
Another thing is do we need to export such thing or we have to make the computation every time to figure out if we leak memory.
And more important is that seems these memory has a low priority to be reclaimed by memory reclaim mechanism, does it due to most of the slab objects are active?
Quoted 3 lines
Quoted 3 lines
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
4383036 4383014 99% 1.00K 1095759 4 4383036K xfs_inode 5394610 5394544 99% 0.38K 539461 10 2157844K xfs_buf
In fact xfs eats a lot of my ram and I will never know where it goes without diving into xfs source, at least I'm the second extreme user ;-)
Quoted 6 lines
Obviously your workload is doing something extremely metadata intensive to have a cache footprint like this - you have more cached buffers than inodes, dentries, etc. That in itself is very unusual - can you describe what is stored on that filesystem and how large the attributes being stored in each inode are?
The fs-user behavior is that ceph-osd daemon will intensively pull/synchronize/update files from other osd when the server is up.
In our case cephfs osd stores a lot of small pictures in the filesystem, and I do some simple analysis, there are nearly 3,000,000 files on each disk and there are 10 such disk. [root@wzdx49 osd.670]# find current -type f -size -512k | wc -l
2668769
[root@wzdx49 ~]# find /data/osd/osd.67 -type f | wc -l
2682891
[root@wzdx49 ~]# find /data/osd/osd.67 -type d | wc -l
109760
thanks,
linfeng
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 40 lines · Lin Feng
Hi Dave,
Thank you for your fast reply, look beblow please.
Quoted 35 lines · Dave Chinner
Quoted 11 lines · Lin Feng
Hi all nice xfs folks,
I'm a rookie and really fresh new in xfs and currently I ran into an issue same as the following link described:
http://oss.sgi.com/archives/xfs/2014-04/msg00058.htmlIn my box(running cephfs osd using xfs kernel 2.6.32-358) and I sum all possible memory counter can be find but it seems that nearlly 26GB memory has gone and they are back after I echo 2 >
/proc/sys/vm/drop_caches, so seems these memory can be reclaimed by slab.It isn't "reclaimed by slab". The XFS metadata buffer cache is reclaimed by a memory shrinker, which are for reclaiming objects from caches that aren't the page cache. "echo 2 >
/proc/sys/vm/drop_caches" runs the memory shrinkers rather than page cache reclaim. Many slab caches are backed by memory shrinkers, which is why it is thought that "2" is "slab reclaim"....Quoted 1 line
And according to what David said replying in the list:
..
Quoted 4 lines
That's where your memory is - in metadata buffers. The
xfs_bufslab entries are just the handles - the metadata pages in the buffers usually take much more space and it's not accounted to the slab cache nor the page cache.That's exactly the case.
Quoted 5 lines
Minimum / Average / Maximum Object : 0.02K / 0.33K / 4096.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
4383036 4383014 99% 1.00K 1095759 4 4383036K xfs_inode 5394610 5394544 99% 0.38K 539461 10 2157844K xfs_buf
Quoted 5 lines
Quoted 3 lines
So, you have *5.4 million* active metadata buffers. Each buffer will hold 1 or 2 4k pages on your kernel, so simple math says 4M * 4k + 1.4M * 8k = 26G. There's no missing counter here....
Does xattr contribute to such metadata buffers or there is something else?
xattrs are metadata, so if they don't fit in line in the inode (typical for ceph because it uses xattrs larger than 256 bytes) then they are held in external blocks which are cached in the buffer cache.
Quoted 2 lines
After consulting to my teammate, who told me that in our case small files (there are a looot, look below) always use xattr.
Which means that if you have 4.4M cached inodes, you probably have ~4.4M xattr metadata buffers in cache for those inodes, too.
Quoted 5 lines
Another thing is do we need to export such thing or we have to make the computation every time to figure out if we leak memory. And more important is that seems these memory has a low priority to be reclaimed by memory reclaim mechanism, does it due to most of the slab objects are active?
"active" slab objects simply mean they are allocated. It does not mean they are cached or imply anything else about the object's life cycle.
Quoted 15 lines
Quoted 3 lines
Quoted 3 lines
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
4383036 4383014 99% 1.00K 1095759 4 4383036K xfs_inode 5394610 5394544 99% 0.38K 539461 10 2157844K xfs_bufIn fact xfs eats a lot of my ram and I will never know where it goes without diving into xfs source, at least I'm the second extreme user ;-)
Quoted 6 lines
Obviously your workload is doing something extremely metadata intensive to have a cache footprint like this - you have more cached buffers than inodes, dentries, etc. That in itself is very unusual - can you describe what is stored on that filesystem and how large the attributes being stored in each inode are?
Quoted 11 lines
The fs-user behavior is that ceph-osd daemon will intensively pull/synchronize/update files from other osd when the server is up. In our case cephfs osd stores a lot of small pictures in the filesystem, and I do some simple analysis, there are nearly 3,000,000 files on each disk and there are 10 such disk. [root@wzdx49 osd.670]# find current -type f -size -512k | wc -l 2668769
[root@wzdx49 ~]# find /data/osd/osd.67 -type f | wc -l 2682891
[root@wzdx49 ~]# find /data/osd/osd.67 -type d | wc -l 109760
Yup, that's a pretty good indication that you have a high metadata to data ratio in each filesystem, and that ceph is accessing the metadata more intensively than the data. The fact that the metadata buffer count roughly matches the cached inode count tells me that the memory reclaim code is being fairly balanced about what it reclaims under memory pressure - I think the problem here is more that you didn't know where the memory was being used than anything else....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Hi Dave,
A final not-clear concept about XFS, look beblow please.
Quoted 45 lines · Dave Chinner
Quoted 44 lines · Lin Feng
Hi Dave,
Thank you for your fast reply, look beblow please.
Quoted 38 lines · Dave Chinner
Quoted 11 lines · Lin Feng
Hi all nice xfs folks,
I'm a rookie and really fresh new in xfs and currently I ran into an issue same as the following link described:
http://oss.sgi.com/archives/xfs/2014-04/msg00058.htmlIn my box(running cephfs osd using xfs kernel 2.6.32-358) and I sum all possible memory counter can be find but it seems that nearlly 26GB memory has gone and they are back after I echo 2 >
/proc/sys/vm/drop_caches, so seems these memory can be reclaimed by slab.It isn't "reclaimed by slab". The XFS metadata buffer cache is reclaimed by a memory shrinker, which are for reclaiming objects from caches that aren't the page cache. "echo 2 >
/proc/sys/vm/drop_caches" runs the memory shrinkers rather than page cache reclaim. Many slab caches are backed by memory shrinkers, which is why it is thought that "2" is "slab reclaim"....Quoted 1 line
And according to what David said replying in the list:
..
Quoted 4 lines
That's where your memory is - in metadata buffers. The
xfs_bufslab entries are just the handles - the metadata pages in the buffers usually take much more space and it's not accounted to the slab cache nor the page cache.That's exactly the case.
Quoted 5 lines
Minimum / Average / Maximum Object : 0.02K / 0.33K / 4096.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
4383036 4383014 99% 1.00K 1095759 4 4383036K xfs_inode 5394610 5394544 99% 0.38K 539461 10 2157844K xfs_bufSo, you have *5.4 million* active metadata buffers. Each buffer will hold 1 or 2 4k pages on your kernel, so simple math says 4M * 4k + 1.4M * 8k = 26G. There's no missing counter here....
Quoted 7 lines
Quoted 1 line
Does xattr contribute to such metadata buffers or there is something else?
xattrs are metadata, so if they don't fit in line in the inode (typical for ceph because it uses xattrs larger than 256 bytes) then they are held in external blocks which are cached in the buffer cache.
So the 'buffer cache' here you mean is the pages handled by xfs_buf struct, used to hold the xattrs if the inode inline data space overflows, not the 'beffer/cache' seen via free command, they won't reflect in cache field by free command, right?
Quoted 12 lines
Quoted 2 lines
After consulting to my teammate, who told me that in our case small files (there are a looot, look below) always use xattr.
Which means that if you have 4.4M cached inodes, you probably have ~4.4M xattr metadata buffers in cache for those inodes, too.
Quoted 5 lines
Another thing is do we need to export such thing or we have to make the computation every time to figure out if we leak memory. And more important is that seems these memory has a low priority to be reclaimed by memory reclaim mechanism, does it due to most of the slab objects are active?
Quoted 3 lines
"active" slab objects simply mean they are allocated. It does not mean they are cached or imply anything else about the object's life cycle.
Sorry, I mistake the concept for active in slab, thanks your explanation.
Quoted 28 lines
Quoted 16 lines
Quoted 4 lines
Quoted 4 lines
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 4383036 4383014 99% 1.00K 1095759 4 4383036K xfs_inode 5394610 5394544 99% 0.38K 539461 10 2157844K xfs_bufIn fact xfs eats a lot of my ram and I will never know where it goes without diving into xfs source, at least I'm the second extreme user ;-)
Quoted 6 lines
Obviously your workload is doing something extremely metadata intensive to have a cache footprint like this - you have more cached buffers than inodes, dentries, etc. That in itself is very unusual - can you describe what is stored on that filesystem and how large the attributes being stored in each inode are?
Quoted 11 lines
The fs-user behavior is that ceph-osd daemon will intensively pull/synchronize/update files from other osd when the server is up. In our case cephfs osd stores a lot of small pictures in the filesystem, and I do some simple analysis, there are nearly 3,000,000 files on each disk and there are 10 such disk. [root@wzdx49 osd.670]# find current -type f -size -512k | wc -l 2668769
[root@wzdx49 ~]# find /data/osd/osd.67 -type f | wc -l 2682891
[root@wzdx49 ~]# find /data/osd/osd.67 -type d | wc -l 109760
Quoted 8 lines
Yup, that's a pretty good indication that you have a high metadata to data ratio in each filesystem, and that ceph is accessing the metadata more intensively than the data. The fact that the metadata buffer count roughly matches the cached inode count tells me that the memory reclaim code is being fairly balanced about what it reclaims under memory pressure - I think the problem here is more that you didn't know where the memory was being used than anything else....
Yes, that's exactly why I sent this mail.
Again, thanks for your detailed explanation.
Best regards,
linfeng
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 6 lines · Lin Feng
Hi Dave,
A final not-clear concept about XFS, look beblow please.
Quoted 1 line · Dave Chinner
On Thu, Sep 08, 2016 at 06:07:45PM +0800, Lin Feng wrote:
....
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
So, you have *5.4 million* active metadata buffers. Each buffer will hold 1 or 2 4k pages on your kernel, so simple math says 4M * 4k + 1.4M * 8k = 26G. There's no missing counter here....
Does xattr contribute to such metadata buffers or there is something else?
Quoted 8 lines
Quoted 5 lines
xattrs are metadata, so if they don't fit in line in the inode (typical for ceph because it uses xattrs larger than 256 bytes) then they are held in external blocks which are cached in the buffer cache.
So the 'buffer cache' here you mean is the pages handled by
xfs_bufstruct,
Yes.
Quoted 2 lines
used to hold the xattrs if the inode inline data space overflows,
And all other cached metadata that is accessed via struct xfs_buf.
Quoted 2 lines
not the 'beffer/cache' seen via free command, they won't reflect in cache field by free command, right?
Correct. From the "free" man page:
buffers
Memory used by kernel buffers (Buffers in /proc/meminfo)
cache
Memory used by the page cache and slabs (Cached and Slab in /proc/meminfo)
buff/cache
Sum of buffers and cache
So, "Buffers" is the amount of cached block device pages - this is always zero for XFS filesystems as we don't use the block device page cache at all (IIRC, that's where ext4 caches it's metadata). "cache" is obvious, but it does not include memory attached to slab objects. Hence it will account for struct xfs_buf memory usage, but not the pages attached to each xfs_buf....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs