ps performance sucks (was Re: dcache_rcu [performance results])
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Martin" == Martin J Bligh <[email protected]> writes:
And i'd still keep environ seperate. I'm inclined to think ps should never have presented it in the first place. This is the direction i (for what it's worth) favor.
Martin> If it doesn't need it then sure, otherwise just dump whatever Martin> it needs in there. The seperate files would still be there Martin> too.
Why not make an mmapable file in /proc with everything in it? It can be sparse, so have the first part a bit map to say what processes are active, then just look at the bits you need. That gets rid of all but the open and mmap system call.
Peter C
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
And i'd still keep environ seperate. I'm inclined to think ps should never have presented it in the first place. This is the direction i (for what it's worth) favor.
Quoted 8 lines
Martin> If it doesn't need it then sure, otherwise just dump whatever Martin> it needs in there. The seperate files would still be there Martin> too.
Why not make an mmapable file in /proc with everything in it? It can be sparse, so have the first part a bit map to say what processes are active, then just look at the bits you need. That gets rid of all but the open and mmap system call.
The locking of walking the tasklist seems non-trivial, but we may well end up with something like that. By the time you finish, it looks more like a /dev device thing than /proc (which I'm fine with), and looks rather like the patch that was posted earlier in this thread. Ideally I'd like fewer ioctls than that, and lower syscall overhead, but it seems much closer to functional than the current system.
M.
Quoted 3 lines · Martin J. Bligh
The locking of walking the tasklist seems non-trivial, but we may well end up with something like that. By the time you finish, it looks more like a /dev device thing than /proc (which I'm fine with), and looks
Can people just oprofile this instead of guessing? Opening a file is not very expensive anymore, so if ps is noticeably slow, it must be something else.
'To measure is to know'
Regards,
bert
Signature · 2 lines
http://www.PowerDNS.com Versatile DNS Software & Services http://lartc.org Linux Advanced Routing & Traffic Control HOWTO
Quoted 9 lines
Quoted 3 lines
The locking of walking the tasklist seems non-trivial, but we may well end up with something like that. By the time you finish, it looks more like a /dev device thing than /proc (which I'm fine with), and looks
Can people just oprofile this instead of guessing? Opening a file is not very expensive anymore, so if ps is noticeably slow, it must be something else.
'To measure is to know'
Errm... we have profiled it. Look at the subject line ... this started off as a dcache_rcu discussion. The dcache lookup ain't cheap, for starters, but that's not really the problem ... it's O(number of tasks), which sucks.
M.
Quoted 6 lines · Martin J. Bligh
Quoted 1 line
'To measure is to know'
Errm... we have profiled it. Look at the subject line ... this started off as a
dcache_rcudiscussion. The dcache lookup ain't cheap, for starters, but that's not really the problem ... it's O(number of tasks), which sucks.
Ok - but if opening a few files is the problem, the solution is not to roll those files into one but to figure out why opening the files is slow in the first place.
Apologies for misinterpreting some of the comments I saw, I'm a bit hyper from my futex documenting spree.
Regards,
bert
Signature · 2 lines
http://www.PowerDNS.com Versatile DNS Software & Services http://lartc.org Linux Advanced Routing & Traffic Control HOWTO
Quoted 8 lines
Quoted 4 lines
Errm... we have profiled it. Look at the subject line ... this started off as a
dcache_rcudiscussion. The dcache lookup ain't cheap, for starters, but that's not really the problem ... it's O(number of tasks), which sucks.Ok - but if opening a few files is the problem, the solution is not to roll those files into one but to figure out why opening the files is slow in the first place.
It's not a few files if you have large numbers of tasks. It's an interface that fundamentally wasn't designed to scale, and futzing around tweaking the thing isn't going to cut it, it needs a different design. I'm not proposing throwing out any of the old simple interfaces, just providing something efficient as a data gathering interface for those people who wish to use it.
M.
Quoted 6 lines · Martin J. Bligh
It's not a few files if you have large numbers of tasks. It's an interface that fundamentally wasn't designed to scale, and futzing around tweaking the thing isn't going to cut it, it needs a different design. I'm not proposing throwing out any of the old simple interfaces, just providing something efficient as a data gathering interface for those people who wish to use it.
That's odd, to say the least. Userland side is at least linear by number of tasks, regardless of the way you gather information. So I really wonder how opening O(number of tasks) files can show up when you scale the things up - pure userland parts will grow at least as fast as that.
Quoted 9 lines · On Tue, 5 Nov 2002 19:34:47 -0500 (EST) Alexander Viro <[email protected]>…
Quoted 6 lines · Martin J. Bligh
It's not a few files if you have large numbers of tasks. It's an interface that fundamentally wasn't designed to scale, and futzing around tweaking the thing isn't going to cut it, it needs a different design. I'm not proposing throwing out any of the old simple interfaces, just providing something efficient as a data gathering interface for those people who wish to use it.
Quoted 5 lines
That's odd, to say the least. Userland side is at least linear by number of tasks, regardless of the way you gather information. So I really wonder how opening O(number of tasks) files can show up when you scale the things up - pure userland parts will grow at least as fast as that.
And I look forward to your "du" interface the kernel, too. It uses this terrible method of statting every file!
Now, according to wli, there's a real problem with starvation by saturating the read side of the tasklist_lock so eg. the write_lock_irq(&tasklist_lock) in exit.c's release_task causes a CPU to spin for ages (forever?) with interrupts off. This needs fixing, be it RCU or making that particular lock give way to writers, or some other effect.
But the "ps takes too much time from my benchmarks" is a "don't do that". Or you may really hate the /dev/kmem hack, but it's simple and effective. The "hundreds of users doing ps/top" is easily solved with a "ps" daemon.
Fixing top is probably a worthwhile thing to do, and you'll note that after the first iteration it should be rarely neccessary to read every "stat" proc file.
Optimize userspace before putting a hack into the kernel, *please*!
Rusty.
Signature · 2 lines
there are those who do and those who hang on and you don't see too many doers quoting their contemporaries. -- Larry McVoy
Quoted 5 lines · Rusty Russell
Now, according to wli, there's a real problem with starvation by saturating the read side of the
tasklist_lockso eg. thewrite_lock_irq(&tasklist_lock)in exit.c'srelease_taskcauses a CPU to spin for ages (forever?) with interrupts off. This needs fixing, be it RCU or making that particular lock give way to writers, or some other effect.
One way to at least "postpone" having to do things like making a fair tasklist_lock is to make readers well-behaved. /proc/ is the worst remaining offender left with its quadratic (!) get_pid_list(). After "kernel, you're being bad and spinning in near-infinite loops with the tasklist_lock readlocked" is (completely?) solved, then we can wait for boxen with higher cpu counts to catch fire anyway when the arrival rate of readers * hold time of readers > 1, which will happen because arrival rates are O(cpus), and cpus will grow without bound as machines advance.
I'm not sure RCU would help this any; I'd be very much afraid of the writes being postponed indefinitely or just too long in the presence of what's essentially perpetually in-progress read access. Does RCU have a guarantee of forward progress for writers?
Thanks,
Bill
Quoted 9 lines · William Lee Irwin III
One way to at least "postpone" having to do things like making a fair
tasklist_lockis to make readers well-behaved./proc/is the worst remaining offender left with its quadratic (!)get_pid_list(). After "kernel, you're being bad and spinning in near-infinite loops with thetasklist_lockreadlocked" is (completely?) solved, then we can wait for boxen with higher cpu counts to catch fire anyway when the arrival rate of readers * hold time of readers > 1, which will happen because arrival rates are O(cpus), and cpus will grow without bound as machines advance.
Quoted 4 lines
I'm not sure RCU would help this any; I'd be very much afraid of the writes being postponed indefinitely or just too long in the presence of what's essentially perpetually in-progress read access. Does RCU have a guarantee of forward progress for writers?
I am not sure I like the idea of RCU for the tasklist_lock.
I do agree 100% with your first point, though - the problem is ill-behaved readers. I think the writing vs. reading is such that the rw-lock we have now is fine, we just need to make e.g. /proc play way way more fair.
Robert Love
Quoted 16 lines · Robert Love
Quoted 9 lines · William Lee Irwin III
One way to at least "postpone" having to do things like making a fair
tasklist_lockis to make readers well-behaved./proc/is the worst remaining offender left with its quadratic (!)get_pid_list(). After "kernel, you're being bad and spinning in near-infinite loops with thetasklist_lockreadlocked" is (completely?) solved, then we can wait for boxen with higher cpu counts to catch fire anyway when the arrival rate of readers * hold time of readers > 1, which will happen because arrival rates are O(cpus), and cpus will grow without bound as machines advance.Quoted 4 lines
I'm not sure RCU would help this any; I'd be very much afraid of the writes being postponed indefinitely or just too long in the presence of what's essentially perpetually in-progress read access. Does RCU have a guarantee of forward progress for writers?
Quoted 6 lines
I am not sure I like the idea of RCU for the
tasklist_lock.I do agree 100% with your first point, though - the problem is ill-behaved readers. I think the writing vs. reading is such that the rw-lock we have now is fine, we just need to make e.g. /proc play way way more fair.
If you consider the atomicity that readdir() in /proc needs (that is:
almost none; it needs a guarantee that it will not miss a task which is alive both before and after the opendir; but that's about it) then there should be an easy way to augment the pidhash for this sort of search.
Signature · 2 lines
Daniel Jacobowitz MontaVista Software Debian GNU/Linux Developer
Quoted 4 lines · William Lee Irwin III
Quoted 4 lines
I'm not sure RCU would help this any; I'd be very much afraid of the writes being postponed indefinitely or just too long in the presence of what's essentially perpetually in-progress read access. Does RCU have a guarantee of forward progress for writers?
Quoted 5 lines · Robert Love
I am not sure I like the idea of RCU for the
tasklist_lock. I do agree 100% with your first point, though - the problem is ill-behaved readers. I think the writing vs. reading is such that the rw-lock we have now is fine, we just need to make e.g. /proc play way way more fair.
This is only feasible for small numbers of cpus. Any compensation provided by algorithmic improvements on the read-side is outweighed by NR_CPUS. Making readers well-behaved only solves half of the issue. Whether the other half is addressible in a 2.6.x time scale is an open question, but a question I'd like to see answered in favor of fixing the livelocks sooner rather than later (esp. as 2.7+ issues are unlikely to be resolved in line with hardware release schedules). I have a strong bias toward code which works everywhere, all the time.
Bill