processes with shared vm

PATCH In review Terje Eggestad 5 participants 10 messages Aug 17, 2001 – Aug 19, 2001

Share & tools

Terje Eggestad

author

2001-08-17 07:50 UTC #0 lore raw

PATCH 1/1 processes with shared vm

I figured out that it's difficult to find out from /proc which processes that share VM (created with clone(CLONE_VM)).

made a patch that adds in /proc/<pid>/status a VmClones field that tells how many proc that uses the same VM (mm_struct). if there are clones I add another field VmFirstClone with the pid of clone with the lowest pid.

Needed for things like gtop that adds mem usage of groups of proc, or else they add up the usage of SIZE and RSS of threads.

The patch need to be applied to linux/fs/proc/array.c

NB: developed on 2.4.3

Signature · 10 lines

_________________________________________________________________________

Terje Eggestad                  [email protected]
Scali Scalable Linux Systems    http://www.scali.com

Olaf Helsets Vei 6              tel:    +47 22 62 89 61 (OFFICE)
P.O.Box 70 Bogerud                      +47 975 31 574  (MOBILE)
N-0621 Oslo                     fax:    +47 22 62 89 51
NORWAY            
_________________________________________________________________________

*** array.c Thu Aug 16 16:33:56 2001
--- array.c.orig Mon Mar 19 21:34:55 2001
***************
*** 50,61 ****
* Al Viro & Jeff Garzik : moved most of the thing into base.c and
* : proc_misc.c. The rest may eventually go into
* : base.c too.

-  *
-  * Terje Eggestad    :  added in /proc/<pid>/status a VmClones: n
-  *                   :  that tells how many proc that uses the same VM (mm_struct).
-  *                   :  if there are clones add another field VmFirstClone with the
-  *                   :  clone with the lowest pid. Needed for things like gtop that adds 
-  *                   :  mem usage of groups of proc, or else they add up the usage of threads.

#include <linux/config.h>
--- 50,55 ----
***************
*** 184,190 ****
static inline char * task_mem(struct mm_struct *mm, char *buffer) {
struct vm_area_struct * vma;
! unsigned long data = 0, stack = 0;
unsigned long exec = 0, lib = 0;

down_read(&mm->mmap_sem);
--- 178,184 ----
static inline char * task_mem(struct mm_struct *mm, char *buffer) {
struct vm_area_struct * vma;
! unsigned long data = 0, stack = 0;
unsigned long exec = 0, lib = 0;

down_read(&mm->mmap_sem);
***************
*** 212,235 ****
"VmData:\t%8lu kB\n"
"VmStk:\t%8lu kB\n"
"VmExe:\t%8lu kB\n"
! "VmLib:\t%8lu kB\n"
! "VmClones:\t%d\n",
mm->total_vm << (PAGE_SHIFT-10),
mm->locked_vm << (PAGE_SHIFT-10),
mm->rss << (PAGE_SHIFT-10),
data - stack, stack,
! exec - lib, lib,
! mm->mm_users.counter-2);
! /* if we've vm clones, find the lowest/first pid of the clones */
! if (mm->mm_users.counter > 2) {

! 	  struct task_struct *p;
! 	  read_lock(&tasklist_lock);
! 	  for_each_task(p) {
! 	    if (p->mm == mm) break;
! 	  };
! 	  buffer += sprintf(buffer, "VmFirstClone:\t%d\n", p->pid);
! 	  read_unlock(&tasklist_lock);

! };
up_read(&mm->mmap_sem);
return buffer;
}
--- 206,217 ----
"VmData:\t%8lu kB\n"
"VmStk:\t%8lu kB\n"
"VmExe:\t%8lu kB\n"
! "VmLib:\t%8lu kB\n",
mm->total_vm << (PAGE_SHIFT-10),
mm->locked_vm << (PAGE_SHIFT-10),
mm->rss << (PAGE_SHIFT-10),
data - stack, stack,
! exec - lib, lib);
up_read(&mm->mmap_sem);
return buffer;
}

Robert Love

2001-08-17 07:56 UTC #1 lore raw

Quoted 2 lines · Terje Eggestad

I figured out that it's difficult to find out from /proc which processes that share VM (created with clone(CLONE_VM)).

good idea, but use diff -u

Signature · 3 lines

Robert M. Love
rml at ufl.edu
rml at tech9.net

Terje Eggestad

author

2001-08-17 08:04 UTC #2 lore raw

Den 17 Aug 2001 03:56:49 -0400, skrev Robert Love:

Quoted 10 lines · Terje Eggestad

Quoted 2 lines
I figured out that it's difficult to find out from /proc which processes that share VM (created with clone(CLONE_VM)).
good idea, but use diff -u
Signature · 3 lines
Robert M. Love
rml at ufl.edu
rml at tech9.net

array.c.orig Mon Mar 19 21:34:55 2001 renamed from array.c Thu Aug 16 16:33:56 2001 +0 -3

@@ -50,12 +50,6 @@
  * Al Viro & Jeff Garzik :  moved most of the thing into base.c and
  *                      :  proc_misc.c. The rest may eventually go into
  *                      :  base.c too.
- *
- * Terje Eggestad    :  added in /proc/<pid>/status a VmClones: n
- *                   :  that tells how many proc that uses the same VM

(mm_struct).
- * : if there are clones add another field
VmFirstClone with the
- * : clone with the lowest pid. Needed for things
like gtop that adds
- * : mem usage of groups of proc, or else they add
up the usage of threads.
*/

#include <linux/config.h>
@@ -184,7 +178,7 @@
static inline char * task_mem(struct mm_struct *mm, char *buffer)

 {
        struct vm_area_struct * vma;
-       unsigned long data = 0, stack = 0;
+       unsigned long data = 0, stack = 0;
        unsigned long exec = 0, lib = 0;
 
        down_read(&mm->mmap_sem);
@@ -212,24 +206,12 @@
                "VmData:\t%8lu kB\n"
                "VmStk:\t%8lu kB\n"
                "VmExe:\t%8lu kB\n"
-               "VmLib:\t%8lu kB\n"
-               "VmClones:\t%d\n",
+               "VmLib:\t%8lu kB\n",
                mm->total_vm << (PAGE_SHIFT-10),
                mm->locked_vm << (PAGE_SHIFT-10),
                mm->rss << (PAGE_SHIFT-10),
                data - stack, stack,

- exec - lib, lib,
- mm->mm_users.counter-2);
- /* if we've vm clones, find the lowest/first pid of the clones
*/
- if (mm->mm_users.counter > 2) {

-         struct task_struct *p;
-         read_lock(&tasklist_lock);
-         for_each_task(p) {
-           if (p->mm == mm) break;
-         };
-         buffer += sprintf(buffer, "VmFirstClone:\t%d\n", p->pid);
-         read_unlock(&tasklist_lock);
-       };
+               exec - lib, lib);
        up_read(&mm->mmap_sem);
        return buffer;

}

Signature · 10 lines

_________________________________________________________________________

Terje Eggestad                  [email protected]
Scali Scalable Linux Systems    http://www.scali.com

Olaf Helsets Vei 6              tel:    +47 22 62 89 61 (OFFICE)
P.O.Box 70 Bogerud                      +47 975 31 574  (MOBILE)
N-0621 Oslo                     fax:    +47 22 62 89 51
NORWAY            
_________________________________________________________________________

array.c.orig renamed from array.c +3 ~~-21~~

@@ -50,12 +50,6 @@
  * Al Viro & Jeff Garzik :  moved most of the thing into base.c and
  *			 :  proc_misc.c. The rest may eventually go into
  *			 :  base.c too.
- *
- * Terje Eggestad    :  added in /proc/<pid>/status a VmClones: n
- *                   :  that tells how many proc that uses the same VM (mm_struct).
- *                   :  if there are clones add another field VmFirstClone with the
- *                   :  clone with the lowest pid. Needed for things like gtop that adds 
- *                   :  mem usage of groups of proc, or else they add up the usage of threads.
  */
 
 #include <linux/config.h>

@@ -184,7 +178,7 @@
 static inline char * task_mem(struct mm_struct *mm, char *buffer)
 {
 	struct vm_area_struct * vma;
- 	unsigned long data = 0, stack = 0;
+	unsigned long data = 0, stack = 0;
 	unsigned long exec = 0, lib = 0;
 
 	down_read(&mm->mmap_sem);

@@ -212,24 +206,12 @@
 		"VmData:\t%8lu kB\n"
 		"VmStk:\t%8lu kB\n"
 		"VmExe:\t%8lu kB\n"
-		"VmLib:\t%8lu kB\n"
-		"VmClones:\t%d\n",
+		"VmLib:\t%8lu kB\n",
 		mm->total_vm << (PAGE_SHIFT-10),
 		mm->locked_vm << (PAGE_SHIFT-10),
 		mm->rss << (PAGE_SHIFT-10),
 		data - stack, stack,
-		exec - lib, lib, 
-		mm->mm_users.counter-2);
-	/* if we've vm clones, find the lowest/first pid of the clones */	
-	if (mm->mm_users.counter > 2) {
-	  struct task_struct *p;
-	  read_lock(&tasklist_lock);
-	  for_each_task(p) {
-	    if (p->mm == mm) break;
-	  };
-	  buffer += sprintf(buffer, "VmFirstClone:\t%d\n", p->pid);
-	  read_unlock(&tasklist_lock);
-	};
+		exec - lib, lib);
 	up_read(&mm->mmap_sem);
 	return buffer;
 }

Robert Love

2001-08-17 08:09 UTC #3 lore raw

Quoted 1 line · Terje Eggestad

OK

the patch is backwards :) i think your original one was too

diff -u <old> <new>

Signature · 3 lines

Robert M. Love
rml at ufl.edu
rml at tech9.net

Terje Eggestad

author

2001-08-17 08:15 UTC #4 lore raw

Man......

Is there any other way I can do this wrong!?!?, guess I need to write more patches :-)

array.c renamed from array.c.orig +21 -3

@@ -50,6 +50,12 @@
  * Al Viro & Jeff Garzik :  moved most of the thing into base.c and
  *			 :  proc_misc.c. The rest may eventually go into
  *			 :  base.c too.
+ *
+ * Terje Eggestad    :  added in /proc/<pid>/status a VmClones: n
+ *                   :  that tells how many proc that uses the same VM (mm_struct).
+ *                   :  if there are clones add another field VmFirstClone with the
+ *                   :  clone with the lowest pid. Needed for things like gtop that adds 
+ *                   :  mem usage of groups of proc, or else they add up the usage of threads.
  */
 
 #include <linux/config.h>

@@ -178,7 +184,7 @@
 static inline char * task_mem(struct mm_struct *mm, char *buffer)
 {
 	struct vm_area_struct * vma;
-	unsigned long data = 0, stack = 0;
+ 	unsigned long data = 0, stack = 0;
 	unsigned long exec = 0, lib = 0;
 
 	down_read(&mm->mmap_sem);

@@ -206,12 +212,24 @@
 		"VmData:\t%8lu kB\n"
 		"VmStk:\t%8lu kB\n"
 		"VmExe:\t%8lu kB\n"
-		"VmLib:\t%8lu kB\n",
+		"VmLib:\t%8lu kB\n"
+		"VmClones:\t%d\n",
 		mm->total_vm << (PAGE_SHIFT-10),
 		mm->locked_vm << (PAGE_SHIFT-10),
 		mm->rss << (PAGE_SHIFT-10),
 		data - stack, stack,
-		exec - lib, lib);
+		exec - lib, lib, 
+		mm->mm_users.counter-2);
+	/* if we've vm clones, find the lowest/first pid of the clones */	
+	if (mm->mm_users.counter > 2) {
+	  struct task_struct *p;
+	  read_lock(&tasklist_lock);
+	  for_each_task(p) {
+	    if (p->mm == mm) break;
+	  };
+	  buffer += sprintf(buffer, "VmFirstClone:\t%d\n", p->pid);
+	  read_unlock(&tasklist_lock);
+	};
 	up_read(&mm->mmap_sem);
 	return buffer;
 }

Den 17 Aug 2001 04:08:53 -0400, skrev Robert Love:

Quoted 17 lines · Terje Eggestad

Quoted 1 line

OK

the patch is backwards :) i think your original one was too

diff -u <old> <new>

Signature · 9 lines

Robert M. Love
rml at ufl.edu
rml at tech9.net

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Signature · 10 lines

_________________________________________________________________________

Terje Eggestad                  [email protected]
Scali Scalable Linux Systems    http://www.scali.com

Olaf Helsets Vei 6              tel:    +47 22 62 89 61 (OFFICE)
P.O.Box 70 Bogerud                      +47 975 31 574  (MOBILE)
N-0621 Oslo                     fax:    +47 22 62 89 51
NORWAY            
_________________________________________________________________________

array.c renamed from array.c.orig +21 -3

@@ -50,6 +50,12 @@
  * Al Viro & Jeff Garzik :  moved most of the thing into base.c and
  *			 :  proc_misc.c. The rest may eventually go into
  *			 :  base.c too.
+ *
+ * Terje Eggestad    :  added in /proc/<pid>/status a VmClones: n
+ *                   :  that tells how many proc that uses the same VM (mm_struct).
+ *                   :  if there are clones add another field VmFirstClone with the
+ *                   :  clone with the lowest pid. Needed for things like gtop that adds 
+ *                   :  mem usage of groups of proc, or else they add up the usage of threads.
  */
 
 #include <linux/config.h>

@@ -178,7 +184,7 @@
 static inline char * task_mem(struct mm_struct *mm, char *buffer)
 {
 	struct vm_area_struct * vma;
-	unsigned long data = 0, stack = 0;
+ 	unsigned long data = 0, stack = 0;
 	unsigned long exec = 0, lib = 0;
 
 	down_read(&mm->mmap_sem);

@@ -206,12 +212,24 @@
 		"VmData:\t%8lu kB\n"
 		"VmStk:\t%8lu kB\n"
 		"VmExe:\t%8lu kB\n"
-		"VmLib:\t%8lu kB\n",
+		"VmLib:\t%8lu kB\n"
+		"VmClones:\t%d\n",
 		mm->total_vm << (PAGE_SHIFT-10),
 		mm->locked_vm << (PAGE_SHIFT-10),
 		mm->rss << (PAGE_SHIFT-10),
 		data - stack, stack,
-		exec - lib, lib);
+		exec - lib, lib, 
+		mm->mm_users.counter-2);
+	/* if we've vm clones, find the lowest/first pid of the clones */	
+	if (mm->mm_users.counter > 2) {
+	  struct task_struct *p;
+	  read_lock(&tasklist_lock);
+	  for_each_task(p) {
+	    if (p->mm == mm) break;
+	  };
+	  buffer += sprintf(buffer, "VmFirstClone:\t%d\n", p->pid);
+	  read_unlock(&tasklist_lock);
+	};
 	up_read(&mm->mmap_sem);
 	return buffer;
 }

[email protected]

2001-08-17 12:27 UTC #5 lore raw

In reply to Terje Eggestad

Why not just print out the address of the 'mm_struct'?

That lets 'ps' treat the address as a cookie, and
thus count the number of occurences of each vm.

This has the additional advantages of:

* moving the intelligence out to the app
* reducing the kernel size, and
* make it easy to find out exactly which processes are using which vm. (you just search for all occurences of the cookie).

Michael.

Quoted 16 lines · Terje Eggestad

--=-L6iLOYILsDzljNLIi035
Content-Type: text/plain
Content-Transfer-Encoding: 7bit

array.c renamed from array.c.orig +6 -0

@@ -50,6 +50,12 @@
  * Al Viro & Jeff Garzik :  moved most of the thing into base.c and
  *			 :  proc_misc.c. The rest may eventually go into
  *			 :  base.c too.
+ *
+ * Terje Eggestad    :  added in /proc/<pid>/status a VmClones: n
+ *                   :  that tells how many proc that uses the same VM (mm_struct).
+ *                   :  if there are clones add another field VmFirstClone with the
+ *                   :  clone with the lowest pid. Needed for things like gtop that adds 
+ *                   :  mem usage of groups of proc, or else they add up the usage of threads.
  */

Terje Eggestad

2001-08-18 14:15 UTC #6 lore raw

Quoted 9 lines · On 17 Aug 2001 [email protected] wrote:

Why not just print out the address of the 'mm_struct'?
That lets 'ps' treat the address as a cookie, and
thus count the number of occurences of each vm.
This has the additional advantages of:

Quoted 8 lines

* moving the intelligence out to the app
* reducing the kernel size, and
* make it easy to find out exactly which processes are using which vm. (you just search for all occurences of the cookie).
Michael.

Not a bad idea, One reason is that I've an inate distrust of using addresses as anything remotely useful. BTW, do you want the tag in hex or dec??
keep in mind 32 and 64 bit machines, it must actually be a 64 bit!

What I really wanted was a list of pids of the clones. ps/top/gtop could then use it as an exclude list for futher processing... Trouble is that returning more than 4kb in a file in proc is a pain, and there is no guarantee that someone make will not a 1000 clones. ref the recent problem with maps exceeding 4kb.

(I might be paranoid, you get ~170 (1024/6) pids in 1kb, assuming 16bit pid.

using the lowest pid seems a good compromise.

I still think the overhead is neglible.
What's the upper practial limit of procs ~64k? (more like 4k.) How many instructions to tranvers the task list and test mm_struct pointer for equality? O(10) per task.? assuming all clones we're talking about ~650k instructions, and with 100mips machines (with 64k task, that's slow1) thats 1/200 second overhead every time you do cat /proc/[0-9]*/status. I can live with that.

But I guess I can live with using the address.....

Quoted 24 lines

Quoted 6 lines · Terje Eggestad
--=-L6iLOYILsDzljNLIi035
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
array.c line 59
Quoted 10 lines
array.c renamed from array.c.orig +6 -0
@@ -50,6 +50,12 @@
  * Al Viro & Jeff Garzik :  moved most of the thing into base.c and
  *			 :  proc_misc.c. The rest may eventually go into
  *			 :  base.c too.
+ *
+ * Terje Eggestad    :  added in /proc/<pid>/status a VmClones: n
+ *                   :  that tells how many proc that uses the same VM (mm_struct).
+ *                   :  if there are clones add another field VmFirstClone with the
+ *                   :  clone with the lowest pid. Needed for things like gtop that adds
+ *                   :  mem usage of groups of proc, or else they add up the usage of threads.
  */
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 10 lines

_________________________________________________________________________

Terje Eggestad                  [email protected]
Scali Scalable Linux Systems    http://www.scali.com

Olaf Helsets Vei 6              tel:    +47 22 62 89 61 (OFFICE)
P.O.Box 70 Bogerud                      +47 975 31 574  (MOBILE)
N-0621 Oslo                     fax:    +47 22 62 89 51
NORWAY
_________________________________________________________________________

Albert D. Cahalan

2001-08-19 06:25 UTC #7 lore raw

Quoted 6 lines · Terje Eggestad

Quoted 4 lines · On 17 Aug 2001 [email protected] wrote:
Why not just print out the address of the 'mm_struct'?
That lets 'ps' treat the address as a cookie, and
thus count the number of occurences of each vm.

No, I won't make 'ps' do that. Ever wonder why 'ps' doesn't sort processes by default? It isn't OK to suck up lots of memory or reparse the files. This is bad for performance and causes extra trouble when a kernel bug causes /proc/42/status to freeze the 'ps' process.

Also your proposal would require 'ps' to _always_ read the data for _every_ process in the system.

Quoted 3 lines

Not a bad idea, One reason is that I've an inate distrust of using addresses as anything remotely useful. BTW, do you want the tag in hex or dec??

Either... hex is nice I guess.

Quoted 1 line

keep in mind 32 and 64 bit machines, it must actually be a 64 bit!

Nah, this gets you enough: (unsigned)(ptr_to_mm>>sizeof(long))

Quoted 2 lines

What I really wanted was a list of pids of the clones. ps/top/gtop could then use it as an exclude list for futher processing...

Nope. This is not enough for sane thread support in 'ps'. Information is lost across the kernel-user boundry. It would be relatively easy for the kernel to provide a /proc directory listing that groups processes by mm_struct. Without this, 'ps' would have to regenerate the lost information in an inefficient way.

BTW, 'ps' is now here: http://procps.sourceforge.net/

Quoted 6 lines

Trouble is that returning more than 4kb in a file in proc is a pain, and there is no guarantee that someone make will not a 1000 clones. ref the recent problem with maps exceeding 4kb.
(I might be paranoid, you get ~170 (1024/6) pids in 1kb, assuming 16bit pid.

Yes, one has to assume that some cracker or sicko will do that.

Quoted 1 line

using the lowest pid seems a good compromise.

The PID may wrap.

Quoted 2 lines

I still think the overhead is neglible.
What's the upper practial limit of procs ~64k? (more like 4k.)

On an SSI cluster, way more I'd guess.

Quoted 5 lines

How many instructions to tranvers the task list and test mm_struct pointer for equality? O(10) per task.? assuming all clones we're talking about ~650k instructions, and with 100mips machines (with 64k task, that's slow1) thats 1/200 second overhead every time you do cat /proc/[0-9]*/status. I can live with that.

You also dirty the cache and suffer load misses.

Terje Eggestad

2001-08-19 12:23 UTC #8 lore raw

Quoted 8 lines · Albert D. Cahalan

Quoted 6 lines · Terje Eggestad
Quoted 4 lines · On 17 Aug 2001 [email protected] wrote:
Why not just print out the address of the 'mm_struct'?
That lets 'ps' treat the address as a cookie, and
thus count the number of occurences of each vm.

Quoted 9 lines

No, I won't make 'ps' do that. Ever wonder why 'ps' doesn't sort processes by default? It isn't OK to suck up lots of memory or reparse the files. This is bad for performance and causes extra trouble when a kernel bug causes /proc/42/status to freeze the 'ps' process.
Also your proposal would require 'ps' to _always_ read the data for _every_ process in the system.

BTW, Why is ps reading the status file at all???? what info is there that is not in cmndline/stat/statm??

Quoted 10 lines

Quoted 3 lines
Not a bad idea, One reason is that I've an inate distrust of using addresses as anything remotely useful. BTW, do you want the tag in hex or dec??
Either... hex is nice I guess.
Quoted 1 line
keep in mind 32 and 64 bit machines, it must actually be a 64 bit!
Nah, this gets you enough: (unsigned)(ptr_to_mm>>sizeof(long))

hmmmmm, the MSB 32 bit??? that would almost always be the same for all kernel pointers, surely you mean (unsigned) (ptr_to_mm & 0xffffffff) ??

A point though, guess the kernel is never going to use > 4Gb of VM.

Quoted 9 lines

Quoted 2 lines
What I really wanted was a list of pids of the clones. ps/top/gtop could then use it as an exclude list for futher processing...
Nope. This is not enough for sane thread support in 'ps'. Information is lost across the kernel-user boundry. It would be relatively easy for the kernel to provide a /proc directory listing that groups processes by mm_struct. Without this, 'ps' would have to regenerate the lost information in an inefficient way.

Not sure what you're getting at if you use the address of mm_struct as a clone tag, you *do* have to keep track of all read clone tags, right?

What DO you need for sane thread support.....

Quoted 6 lines

BTW, 'ps' is now here: http://procps.sourceforge.net/
Quoted 4 lines
Trouble is that returning more than 4kb in a file in proc is a pain, and there is no guarantee that someone make will not a 1000 clones. ref the recent problem with maps exceeding 4kb.

Quoted 9 lines

Quoted 2 lines
(I might be paranoid, you get ~170 (1024/6) pids in 1kb, assuming 16bit pid.
Yes, one has to assume that some cracker or sicko will do that.
Quoted 1 line
using the lowest pid seems a good compromise.
The PID may wrap.

kept thinking that the first clone is unlikely to ever die, but you're right, new clones may very well have a lower pid, which will of course throw of ps and likes. (damn)

Quoted 5 lines

Quoted 2 lines
I still think the overhead is neglible.
What's the upper practial limit of procs ~64k? (more like 4k.)
On an SSI cluster, way more I'd guess.

Quoted 7 lines

Quoted 5 lines
How many instructions to tranvers the task list and test mm_struct pointer for equality? O(10) per task.? assuming all clones we're talking about ~650k instructions, and with 100mips machines (with 64k task, that's slow1) thats 1/200 second overhead every time you do cat /proc/[0-9]*/status. I can live with that.
You also dirty the cache and suffer load misses.

hmmm, put it like that I also prevent future cache misses, ps is likely to travers the entire task_struct list anyway...
But, sure, fine, 1/20 of a second, prefetching algo's is getting pretty good.
64k tasks.... aren't we both getting slightly farfetched here...

Signature · 10 lines

_________________________________________________________________________

Terje Eggestad                  [email protected]
Scali Scalable Linux Systems    http://www.scali.com

Olaf Helsets Vei 6              tel:    +47 22 62 89 61 (OFFICE)
P.O.Box 70 Bogerud                      +47 975 31 574  (MOBILE)
N-0621 Oslo                     fax:    +47 22 62 89 51
NORWAY
_________________________________________________________________________

Albert D. Cahalan

2001-08-19 23:26 UTC #9 lore raw

Quoted 15 lines · Terje Eggestad

Quoted 14 lines · Albert D. Cahalan
Quoted 6 lines · Terje Eggestad
Quoted 4 lines · On 17 Aug 2001 [email protected] wrote:
Why not just print out the address of the 'mm_struct'?
That lets 'ps' treat the address as a cookie, and
thus count the number of occurences of each vm.
No, I won't make 'ps' do that. Ever wonder why 'ps' doesn't sort processes by default? It isn't OK to suck up lots of memory or reparse the files. This is bad for performance and causes extra trouble when a kernel bug causes /proc/42/status to freeze the 'ps' process.

Quoted 5 lines

Quoted 2 lines
Also your proposal would require 'ps' to _always_ read the data for _every_ process in the system.
BTW, Why is ps reading the status file at all???? what info is there that is not in cmndline/stat/statm??

all 4 UIDs
all 64 signals
misc. VM stats (maybe)

Quoted 6 lines

Quoted 6 lines
Quoted 3 lines
Not a bad idea, One reason is that I've an inate distrust of using addresses as anything remotely useful. BTW, do you want the tag in hex or dec??
Either... hex is nice I guess.

Quoted 8 lines

Quoted 3 lines
Quoted 1 line
keep in mind 32 and 64 bit machines, it must actually be a 64 bit!
Nah, this gets you enough: (unsigned)(ptr_to_mm>>sizeof(long))
hmmmmm, the MSB 32 bit??? that would almost always be the same for all kernel pointers, surely you mean (unsigned) (ptr_to_mm & 0xffffffff) ??
A point though, guess the kernel is never going to use > 4Gb of VM.

sizeof(long) is 4 or 8

the struct is several hundred bytes on 64-bit hardware

shifting by 8 would allow for a terabyte of kernel memory

Quoted 10 lines

Quoted 9 lines
Quoted 2 lines
What I really wanted was a list of pids of the clones. ps/top/gtop could then use it as an exclude list for futher processing...
Nope. This is not enough for sane thread support in 'ps'. Information is lost across the kernel-user boundry. It would be relatively easy for the kernel to provide a /proc directory listing that groups processes by mm_struct. Without this, 'ps' would have to regenerate the lost information in an inefficient way.

Quoted 3 lines

Not sure what you're getting at if you use the address of mm_struct as a clone tag, you *do* have to keep track of all read clone tags, right?

There wouldn't be a need to keep track of all read clone tags if the kernel would group processes by clone tag. Then 'ps' would only need to remember the current clone tag and the accumulated info.

Quoted 1 line

What DO you need for sane thread support.....

Avoiding the "thread" and "process" terms here:

The above would do. When 'ps' looks at a task, 'ps' must be able to determine if the task should be grouped with the previous or next one. That, and the partial ordering implied, would be enough to collect tasks into groups. I don't expect much trouble selecting the leader, since I have the "PPid" (parent TASK identifier!) value.

Quoted 11 lines

Quoted 11 lines
Quoted 2 lines
I still think the overhead is neglible.
What's the upper practial limit of procs ~64k? (more like 4k.)
On an SSI cluster, way more I'd guess.
Quoted 5 lines
How many instructions to tranvers the task list and test mm_struct pointer for equality? O(10) per task.? assuming all clones we're talking about ~650k instructions, and with 100mips machines (with 64k task, that's slow1) thats 1/200 second overhead every time you do cat /proc/[0-9]*/status. I can live with that.

Quoted 7 lines

Quoted 1 line
You also dirty the cache and suffer load misses.
hmmm, put it like that I also prevent future cache misses, ps is likely to travers the entire task_struct list anyway...
But, sure, fine, 1/20 of a second, prefetching algo's is getting pretty good.
64k tasks.... aren't we both getting slightly farfetched here...

I don't know. Ask on the linux-cluster list.