Skip to contentSkip to conversation

Strange memory usage reporting

DISCUSSION Måns Rullgård 5 participants 8 messages Aug 26, 2003 – Aug 27, 2003

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I was a little surprised to see top tell me this:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
10642 mru       11   0 23200  81m 2740 S  0.0 37.0   0:00.07 tcvp              

It didn't make sense that RES > VIRT, so I check /proc/pid/*. Their contents are below. Am I missing something? Note that they are not consistent with the 'top' line above, since they were copied at a different time. The effect is easily reproducible. It happens every time I run my music player with using ALSA.

The memory usage summary by top, also doesn't agree:

Mem:    224140k total,   219236k used,     4904k free,     7184k buffers
Swap:   524280k total,      224k used,   524056k free,   125976k cached

Subtracting the cached memory and some heavy processes doesn't leave room for another 80 MB.

/proc/pid/stat:
10642 (tcvp) S 926 10642 926 34816 10642 0 68 0 2556 0 4 3 0 0 11 0 0 0 1162589 25665536 18902 4294967295 134512640 134523016 3221216832 3221215328 1074519393 0 0 0 2 3222307553 0 0 17 0 0 0

/proc/pid/statm:
6266 18902 685 4 0 6262 0

/proc/pid/status:
Name: tcvp
State: S (sleeping)
Tgid: 10642
Pid: 10642
PPid: 926
TracerPid: 0
Uid: 51770 51770 51770 51770
Gid: 100 100 100 100
FDSize: 256
Groups: 100 102
VmSize: 25064 kB
VmLck: 0 kB
VmRSS: 75608 kB
VmData: 22308 kB
VmStk: 16 kB
VmExe: 12 kB
VmLib: 2492 kB
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 8000000000000000
SigCgt: 0000000380000002
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000

Signature · 2 lines
Måns Rullgård
[email protected]
#1 lore raw
Quoted 9 lines · M?ns Rullg?rd

I was a little surprised to see top tell me this:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
10642 mru       11   0 23200  81m 2740 S  0.0 37.0   0:00.07 tcvp              

It didn't make sense that RES > VIRT, so I check /proc/pid/*. Their contents are below. Am I missing something? Note that they are not consistent with the 'top' line above, since they were copied at a different time. The effect is easily reproducible. It happens every time I run my music player with using ALSA.
The memory usage summary by top, also doesn't agree:

What kernel version?

-- wli

#2 lore raw
Quoted 11 lines · William Lee Irwin III
Quoted 9 lines · M?ns Rullg?rd

I was a little surprised to see top tell me this:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
10642 mru       11   0 23200  81m 2740 S  0.0 37.0   0:00.07 tcvp              

It didn't make sense that RES > VIRT, so I check /proc/pid/*. Their contents are below. Am I missing something? Note that they are not consistent with the 'top' line above, since they were copied at a different time. The effect is easily reproducible. It happens every time I run my music player with using ALSA.
The memory usage summary by top, also doesn't agree:

Quoted 1 line

What kernel version?

Sorry, I forgot that. It's 2.6.0-test4 with Nick Piggins' v7 scheduler patch. The machine I'm running on is a Pentium 4 based laptop.

Signature · 2 lines
Måns Rullgård
[email protected]
#3 lore raw
In reply to Måns Rullgård
Quoted 10 lines · On Tue, 26 Aug 2003, [iso-8859-1] Mĺns Rullgĺrd wrote:

I was a little surprised to see top tell me this:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
10642 mru       11   0 23200  81m 2740 S  0.0 37.0   0:00.07 tcvp              

It didn't make sense that RES > VIRT, so I check /proc/pid/*. Their contents are below. Am I missing something? Note that they are not consistent with the 'top' line above, since they were copied at a different time. The effect is easily reproducible. It happens every time I run my music player with using ALSA.

I have exactly same behaviour with 2.4.21 kernel. It seems that VmRSS grows with the mmap2 syscalls although appropriate munmap is called. I'm investigating a possible problem with the memory accounting.

Jaroslav

-----
Jaroslav Kysela <[email protected]>
Linux Kernel Sound Maintainer
ALSA Project, SuSE Labs

#4 lore raw
Quoted 13 lines · Jaroslav Kysela
Quoted 10 lines · On Tue, 26 Aug 2003, [iso-8859-1] Mĺns Rullgĺrd wrote:

I was a little surprised to see top tell me this:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
10642 mru       11   0 23200  81m 2740 S  0.0 37.0   0:00.07 tcvp              

It didn't make sense that RES > VIRT, so I check /proc/pid/*. Their contents are below. Am I missing something? Note that they are not consistent with the 'top' line above, since they were copied at a different time. The effect is easily reproducible. It happens every time I run my music player with using ALSA.

Quoted 3 lines

I have exactly same behaviour with 2.4.21 kernel. It seems that VmRSS grows with the mmap2 syscalls although appropriate munmap is called. I'm investigating a possible problem with the memory accounting.

Yes, it seems so. The do_no_page() function in mm/memory.c does accounting for reserved pages (++mm->rss), but in zap_pte_range() there is a check preventing increase the count of freed pages.

Here is a patch for VM gurus to review (for 2.4 kernel, but it should apply to 2.6 as well):

===== mm/memory.c 1.57 vs edited =====

mm/memory.c +2 -1
@@ -1306,7 +1306,8 @@
 	 */
 	/* Only go through if we didn't race with anybody else... */
 	if (pte_none(*page_table)) {
-		++mm->rss;
+		if (!PageReserved(new_page))
+			++mm->rss;
 		flush_page_to_ram(new_page);
 		flush_icache_page(vma, new_page);
 		entry = mk_pte(new_page, vma->vm_page_prot);

Jaroslav

-----
Jaroslav Kysela <[email protected]>
Linux Kernel Sound Maintainer
ALSA Project, SuSE Labs

#5 lore raw
Quoted 12 lines · Jaroslav Kysela

Yes, it seems so. The do_no_page() function in mm/memory.c does accounting for reserved pages (++mm->rss), but in zap_pte_range() there is a check preventing increase the count of freed pages.

Here is a patch for VM gurus to review (for 2.4 kernel, but it should apply to 2.6 as well):

===== mm/memory.c 1.57 vs edited =====

mm/memory.c line 1313

Quoted 9 lines
mm/memory.c +2 -1
@@ -1306,7 +1306,8 @@
 	 */
 	/* Only go through if we didn't race with anybody else... */
 	if (pte_none(*page_table)) {
-		++mm->rss;
+		if (!PageReserved(new_page))
+			++mm->rss;
 		flush_page_to_ram(new_page);
 		flush_icache_page(vma, new_page);
 		entry = mk_pte(new_page, vma->vm_page_prot);

You're right (but please rediff against 2.4.22 when you send Marcelo).

You may wonder how this has taken so long to show up: because usually drivers which mmap Reserved pages use remap_page_range on them, and so never fault to do_no_page.

Which is the driver involved? Though it's not wrong to give do_no_page a Reserved page, beware of the the page->count accounting: while it's Reserved, get_page or page_cache_get raises the count, but put_page or page_cache_release does not decrement it - very easy to end up with the page never freed.

Hugh

#6 lore raw

Hi,

Quoted 5 lines · Hugh Dickins

Which is the driver involved? Though it's not wrong to give do_no_page a Reserved page, beware of the the page->count accounting: while it's Reserved, get_page or page_cache_get raises the count, but put_page or page_cache_release does not decrement it - very easy to end up with the page never freed.

Why is this so asymetric? I would understand ignoring these pages in the freeing logic, but why exclude them also from refcounting?

Regards

Ingo Oeser
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#7 lore raw
Quoted 9 lines · Ingo Oeser
Quoted 5 lines · Hugh Dickins

Which is the driver involved? Though it's not wrong to give do_no_page a Reserved page, beware of the the page->count accounting: while it's Reserved, get_page or page_cache_get raises the count, but put_page or page_cache_release does not decrement it - very easy to end up with the page never freed.

Why is this so asymetric? I would understand ignoring these pages in the freeing logic, but why exclude them also from refcounting?

I don't think there's a _good_ reason, it just evolved that way.

The real answer is to get rid of PageReserved completely, which I'll embark on again in 2.7 (I did start a couple of times in 2.5, but each time it was too late).

There was a halfway-house suggestion in 2.5 about three months ago, inspired (as usual) by Reserved page problems in AIO's get_user_pages, to do as you suggest: submit them to normal refcounting. I don't know what became of that, I didn't have much time to get involved.

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>