ipc/shm.c: add RSS and swap size information to /proc/sysvipc/shm

The kernel currently provides no functionality to analyze the RSS and swap space usage of each individual sysvipc shared memory segment.

This patch add this info for each existing shm segment by extending the output of /proc/sysvipc/shm by two columns for RSS and swap.

Since shmctl(SHM_INFO) already provides a similiar calculation (it currently sums up all RSS/swap info for all segments), I did split out a static function which is now used by the /proc/sysvipc/shm output and shmctl(SHM_INFO).

---

 shm.c |   63 ++++++++++++++++++++++++++++++++++++++++++---------------------
 1 file changed, 42 insertions(+), 21 deletions(-)

Quoted 6 lines · On Wed, 11 Aug 2010 22:13:45 +0200 Helge Deller<[email protected]> wrote:

The kernel currently provides no functionality to analyze the RSS and swap space usage of each individual sysvipc shared memory segment.

This patch add this info for each existing shm segment by extending the output of /proc/sysvipc/shm by two columns for RSS and swap.

Quoted 5 lines

Since shmctl(SHM_INFO) already provides a similiar calculation (it currently sums up all RSS/swap info for all segments), I did split out a static function which is now used by the /proc/sysvipc/shm output and shmctl(SHM_INFO).

I suppose that could be useful, although it would be most interesting to hear why _you_ consider it useful?

But is it useful enough to risk breaking existing code which parses that file? The risk is not great, but it's there.

Quoted 6 lines

---

  shm.c |   63 ++++++++++++++++++++++++++++++++++++++++++---------------------
  1 file changed, 42 insertions(+), 21 deletions(-)

ipc/shm.c void __init shm_init (void) · line 114

Quoted 12 lines

ipc/shm.c +4 -1

@@ -108,7 +108,11 @@ void __init shm_init (void)
  {
  	shm_init_ns(&init_ipc_ns);
  	ipc_init_proc_interface("sysvipc/shm",
-				"       key      shmid perms       size  cpid  lpid nattch   uid   gid  cuid  cgid      atime      dtime      ctime\n",
+#if BITS_PER_LONG<= 32
+				"       key      shmid perms       size  cpid  lpid nattch   uid   gid  cuid  cgid      atime      dtime      ctime        RSS       swap\n",
+#else
+				"       key      shmid perms                  size  cpid  lpid nattch   uid   gid  cuid  cgid      atime      dtime      ctime                   RSS                  swap\n",

This adds 11 new spaces between "perms" and "size", only on 64-bit machines. That was unchangelogged and adds another (smaller) risk of breaking things. Please explain.

This interface is really old and crufty and horrid, but I guess that there's not a lot we can do about that :(

Quoted 9 lines · Andrew Morton

Quoted 6 lines · On Wed, 11 Aug 2010 22:13:45 +0200 Helge Deller<[email protected]> wrote:

The kernel currently provides no functionality to analyze the RSS and swap space usage of each individual sysvipc shared memory segment.

This patch add this info for each existing shm segment by extending the output of /proc/sysvipc/shm by two columns for RSS and swap.

Quoted 8 lines

Quoted 5 lines

Since shmctl(SHM_INFO) already provides a similiar calculation (it currently sums up all RSS/swap info for all segments), I did split out a static function which is now used by the /proc/sysvipc/shm output and shmctl(SHM_INFO).

I suppose that could be useful, although it would be most interesting to hear why _you_ consider it useful?

A reasonable question, and I really should have explained when I did send this patch.

In my job I do work for SAP in the SAP LinuxLab
(http://www.sap.com/linux) and take care of the SAP ERP enterprise software on Linux.
SAP products (esp. the SAP Netweaver ABAP Kernel) uses lots of big shared memory segments (we often have Linux systems with >= 16GB shm usage). Sometimes we get customer reports about "slow" system responses and while looking into their configurations we often find massive swapping activity on the system. With this patch it's now easy to see from the command line if and which shm segments gets swapped out (and how much) and can more easily give recommendations for system tuning. Without the patch it's currently not possible to do such shm analysis at all.

So, my patch actually does fix a real-world problem.

By the way - I found another bug/issue in /proc/<pid>/smaps as well. The kernel currently does not adds swapped-out shm pages to the swap size value correctly. The swap size value always stays zero for shm pages. I'm currently preparing a small patch to fix that, which I will send to linux-mm for review soon.

Quoted 2 lines

But is it useful enough to risk breaking existing code which parses that file? The risk is not great, but it's there.

Sure. The only positive argument is maybe, that I added the new info to the end of the lines. IMHO existing applications which parse /proc files should always take into account, that more text could follow with newer Linux kernels...?

Quoted 3 lines

This adds 11 new spaces between "perms" and "size", only on 64-bit machines. That was unchangelogged and adds another (smaller) risk of breaking things. Please explain.

Yes, I did added some spaces in front of the "size" field for 64bit kernels to get the columns correct if you cat the contents of the file. In sysvipc_shm_proc_show() the kernel prints the size value in "SPEC_SIZE" format, which is defined like this:

#if BITS_PER_LONG <= 32
#define SIZE_SPEC "%10lu"
#else
#define SIZE_SPEC "%21lu"
#endif

So, if the header is not adjusted, the columns are not correctly aligned. I actually tested this on 32- and 64-bit and it seems correct now.

Quoted 9 lines · Andrew Morton

Quoted 6 lines · On Wed, 11 Aug 2010 22:13:45 +0200 Helge Deller<[email protected]> wrote:

The kernel currently provides no functionality to analyze the RSS and swap space usage of each individual sysvipc shared memory segment.

This patch add this info for each existing shm segment by extending the output of /proc/sysvipc/shm by two columns for RSS and swap.

Quoted 8 lines

Quoted 5 lines

Since shmctl(SHM_INFO) already provides a similiar calculation (it currently sums up all RSS/swap info for all segments), I did split out a static function which is now used by the /proc/sysvipc/shm output and shmctl(SHM_INFO).

I suppose that could be useful, although it would be most interesting to hear why _you_ consider it useful?

A reasonable question, and I really should have explained when I did send this patch.

In my job I do work for SAP in the SAP LinuxLab (http://www.sap.com/linux) and take care of the SAP ERP enterprise software on Linux.
SAP products (esp. the SAP Netweaver ABAP Kernel) uses lots of big shared memory segments (we often have Linux systems with >= 16GB shm usage). Sometimes we get customer reports about "slow" system responses and while looking into their configurations we often find massive swapping activity on the system. With this patch it's now easy to see from the command line if and which shm segments gets swapped out (and how much) and can more easily give recommendations for system tuning.
Without the patch it's currently not possible to do such shm analysis at all.

So, my patch actually does fix a real-world problem.

By the way - I found another bug/issue in /proc/<pid>/smaps as well. The kernel currently does not adds swapped-out shm pages to the swap size value correctly. The swap size value always stays zero for shm pages. I'm currently preparing a small patch to fix that, which I will send to linux-mm for review soon.

Quoted 3 lines

But is it useful enough to risk breaking existing code which parses that file? The risk is not great, but it's there.

Sure. The only positive argument is maybe, that I added the new info to the end of the lines. IMHO existing applications which parse /proc files should always take into account, that more text could follow with newer Linux kernels...?

Quoted 17 lines

Quoted 17 lines

---

  shm.c |   63
++++++++++++++++++++++++++++++++++++++++++---------------------
  1 file changed, 42 insertions(+), 21 deletions(-)

ipc/shm.c +0 -1

@@ -108,7 +108,11 @@ void __init shm_init (void)
  {
  	shm_init_ns(&init_ipc_ns);
  	ipc_init_proc_interface("sysvipc/shm",
-				"       key      shmid perms       size  cpid
 lpid nattch   uid   gid  cuid  cgid      atime      dtime      ctime\n",

Quoted 8 lines

Quoted 8 lines

ipc/shm.c +2 -0

+#if BITS_PER_LONG<= 32
+				"       key      shmid perms       size  cpid

lpid nattch uid gid cuid cgid atime dtime ctime
RSS swap\n",

ipc/shm.c +2 -0

+#else
+				"       key      shmid perms

size cpid lpid nattch uid gid cuid cgid atime dtime
ctime RSS swap\n",

Quoted 5 lines

By the way - I found another bug/issue in /proc/<pid>/smaps as well. The kernel currently does not adds swapped-out shm pages to the swap size value correctly. The swap size value always stays zero for shm pages. I'm currently preparing a small patch to fix that, which I will send to linux-mm for review soon.

I certainly wouldn't call smaps's present behaviour on it a bug: but given your justification above, I can see that it would be more useful to you, and probably to others, for it to be changed in the way that you suggest, to reveal the underlying swap.

Hmm, I wonder what that patch is going to look like...

From my tests, a swapped-out shm segment

I tried quite hard to implement rss/swap accounting for shm segments inside smaps_pte_range() which is a callback function of walk_page_range() in show_smap().

Given the fact that I'm no linux-mm expert, I might have overseen other possibilities, but my experiments inside smaps_pte_range() were not very successful:
From my tests, a swapped-out shm segment
- fails on the "is_swap_pte()" test, and
- succeeds on the "!pte_present()" test (since it's swapped out).

So, here would it be possible to add such accounting for swap, but how can I then see that this pte is
a) belonging to a shm segment?, and
b) see if this page/pte was really swapped out and not just not
yet written to at all?

As answers I found:

	a) (vma->vm_flags & VM_MAYSHARE) is true for shm segments (is
		this check sufficient?)

b) no idea.

But if I add this page to the mss.swap entry, all pages including such which haven't been touched yet at all are suddenly counted as swapped-out...?

Any hints here would be great...

As an alternative solution, I created the following patch. This one works nicely, but it's just a fix-up of the mss.resident and mss.swap values after walk_page_range() was called.
It's mostly a copy of the shm_add_rss_swap() function from my previous patch (http://marc.info/?l=linux-mm&m=128171161101817&w=2).
Do you think such a fix-up-afterwards-approach is acceptable at all? If yes, a new patch on top of my ipc/shm.c patch would be easy (and small).

Quoted 4 lines · Helge Deller

I tried quite hard to implement rss/swap accounting for shm segments inside smaps_pte_range() which is a callback function of walk_page_range() in show_smap().

Sorry, I think the short answer will be that you should give up on this:
reasons below.

Quoted 8 lines

Given the fact that I'm no linux-mm expert, I might have overseen other possibilities, but my experiments inside smaps_pte_range() were not very successful:
From my tests, a swapped-out shm segment
- fails on the "is_swap_pte()" test, and
- succeeds on the "!pte_present()" test (since it's swapped out).

Yes.

Quoted 5 lines

So, here would it be possible to add such accounting for swap, but how can I then see that this pte is
a) belonging to a shm segment?, and
b) see if this page/pte was really swapped out and not just not
yet written to at all?

You would have to add a function in mm/shmem.c to do this: it would need to check vma->vm_file to work out if this vma belongs to it, and use shmem_swp_alloc() to check if the page there is on swap. OTOH I'm not sure if you could call it while holding page table lock or not.

Quoted 3 lines

As answers I found:

	a) (vma->vm_flags&  VM_MAYSHARE) is true for shm segments (is
		this check sufficient?)

No, VM_MAYSHARE is set on many other kinds of mapping too; and is not set on all mappings of shmem objects - there is no good reason to include SysV shm segments here, yet omit other kinds of shmem object (/dev/shm POSIX shared memory, shared-anonymous mappings, mappings of tmpfs files).

Quoted 9 lines

b) no idea.

But if I add this page to the mss.swap entry, all pages including such which haven't been touched yet at all are suddenly counted as swapped-out...?

Any hints here would be great...

Quoted 8 lines

As an alternative solution, I created the following patch. This one works nicely, but it's just a fix-up of the mss.resident and mss.swap values after walk_page_range() was called.
It's mostly a copy of the shm_add_rss_swap() function from my previous patch (http://marc.info/?l=linux-mm&m=128171161101817&w=2).
Do you think such a fix-up-afterwards-approach is acceptable at all? If yes, a new patch on top of my ipc/shm.c patch would be easy (and small).

Not acceptable, I'm afraid. Nothing wrong with a fix-up-afterwards approach as such, but it's assuming that the vma covers the full extent of the shmem object. That is very often the case, but by no means necessarily so (whereas it is always the case that one vma cannot cover more than one object). So you do have to count pageslot by pageslot.

There are two reasons why I think you have to abandon this. One is that /proc/<pid>/smaps is reporting on the userspace mappings, saying where swap is instanced in them. Some of those mappings may be of shmem objects, and some of those shmem objects may use swap backing themselves, but that's different from the mapping using swap directly.

One can argue about that distinction, but it is how all this is designed, and blurring that distinction tends to get into trouble. (It's reasonable to think of anonymous mappings as mappings of anon objects, which just happen to find room for the swp_entry in the page
table: but then it's a happy accident that smaps can see them.)

The second reason is that since 2.6.34, /proc/<pid>/status shows
VmSwap: we would not want a huge discrepancy between what it shows in swap and what /proc/<pid>/smaps shows in swap, but nor would we want to make /proc/<pid>/status scan through page tables enquiring of shmem.

All this stands in contrast to your /proc/sysvipc/shm patch, which is rightly dealing with one class of shmem object, not via mappings of those objects.

There is a case for a "where has my swap gone" tool, which examines the different kinds of object involved (anonymous mappings as well as shmem objects), and shows them all somehow. But that's a lot more work than just extending an existing stats display.