I/O statistics per process
Hello list,
it`s great that linux now has i/o re-nice with cfq now, but how can the admin determine HOW MUCH i/o a process is actually generating ?
have seen this on windows (process explorer from sysinternals) and on solaris: (psio http://users.tpg.com.au/bdgcvb/psio.html and pio
http://www.stormloader.com/yonghuang/freeware/pio.html), but what`s the Linux (commandline) equivalent ?
is there a modified top/ps with i/o column, or is there yet missing something at the kernel level for getting that counters from ?
regards
Roland K.
systems engineer.
Quoted 2 lines · roland
is there a modified top/ps with i/o column, or is there yet missing something at the kernel level for getting that counters from ?
Red Flag(http://www.redflag-linux.com/eindex.html) has developed an iotop based on kprobes/systemtap. You can contact them if necessary.
thanks. tried to contact redflag, but they don`t answer. maybe support is being on holiday.... !?
linux kernel hackers - there is really no standard way to watch i/o metrics (bytes read/written) at process level?
it`s extremly hard for the admin to track down, what process is hogging the disk - especially if there is more than one task consuming cpu.
meanwhile i found blktrace and read into the documenation. looks really cool and seems to be very powerful tool - but it it`s seems a little bit "oversized" and not the right tool for this. seems to be for tracing/debugging/analysis
what about http://lkml.org/lkml/2005/9/12/89 "with following patch, userspace processes/utilities will be able to access per process I/O statistics. for example, top like utilites can use this information" which has been posted to lkml one year ago ? any update on this ?
roland
----- Original Message ----- From: "Fengguang Wu" <[email protected]>
To: "roland" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, September 24, 2006 5:04 AM
Subject: Re: I/O statistics per process
Quoted 6 lines · roland
Quoted 2 lines
is there a modified top/ps with i/o column, or is there yet missing something at the kernel level for getting that counters from ?
Red Flag(http://www.redflag-linux.com/eindex.html) has developed an iotop based on kprobes/systemtap. You can contact them if necessary.
Quoted 5 lines · On Wed, 27 Sep 2006 23:22:02 +0200 "roland" <[email protected]> wrote:
thanks. tried to contact redflag, but they don`t answer. maybe support is being on holiday.... !?
linux kernel hackers - there is really no standard way to watch i/o metrics (bytes read/written) at process level?
The patch csa-accounting-taskstats-update.patch in current -mm kernels (whcih is planned for 2.6.19) does have per-process chars-read and chars-written accounting ("Extended accounting fields"). That's probably not waht you really want, although it might tell you what you want to know.
Quoted 2 lines
it`s extremly hard for the admin to track down, what process is hogging the disk - especially if there is more than one task consuming cpu.
Sure. Doing this is actually fairly tricky because disk writes are almost always deferred. We'd need to remember which process dirtied some memory, then track that info all the way down to the disk IO level, then perform the accounting operations at IO submit-time or completion time, on a per-page basis. It isn't rocket-science, but it's a lot of stuff and some overhead.
Quoted 9 lines
meanwhile i found blktrace and read into the documenation. looks really cool and seems to be very powerful tool - but it it`s seems a little bit "oversized" and not the right tool for this. seems to be for tracing/debugging/analysis
what about http://lkml.org/lkml/2005/9/12/89 "with following patch, userspace processes/utilities will be able to access per process I/O statistics. for example, top like utilites can use this information" which has been posted to lkml one year ago ? any update on this ?
csa-accounting-taskstats-update.patch makes that information available to userspace.
But it's approximate, because
- it doesn't account for disk readahead
- it doesn't account for pagefault-initiated reads (althought it easily could - Jay?)
- it overaccounts for a process writing to an already-dirty page.
(We could fix this too: nuke the existing stuff and do
current->wchar += PAGE_CACHE_SIZE;
in __set_page_dirty_[no]buffers().) (But that ends up being wrong if someone truncates the file before it got written)
- it doesn't account for file readahead (although it easily could)
- it doesn't account for pagefault-initiated readahead (it could)
hm. There's actually quite a lot we could do here to make these fields more accurate and useful. A lot of this depends on what the definition of these fields _is_. Is is just for disk IO? Is it supposed to include console IO, or what?
Quoted 6 lines · Andrew Morton
Quoted 3 lines · On Wed, 27 Sep 2006 23:22:02 +0200 "roland" <[email protected]> wrote:
thanks. tried to contact redflag, but they don`t answer. maybe support is being on holiday.... !?
Quoted 10 lines
Quoted 2 lines
linux kernel hackers - there is really no standard way to watch i/o metrics (bytes read/written) at process level?
The patch csa-accounting-taskstats-update.patch in current -mm kernels (whcih is planned for 2.6.19) does have per-process chars-read and chars-written accounting ("Extended accounting fields"). That's probably not waht you really want, although it might tell you what you want to know.
Quoted 2 lines
it`s extremly hard for the admin to track down, what process is hogging the disk - especially if there is more than one task consuming cpu.
Rolend,
The per-process chars-read and chars-writeen accounting is made available through taskstats interface (see Documentation/accounting/ taskstats.txt) in 2.6.18-mm1 kernel. Unfortunately, the user-space CSA package is still a few months away. You may, for now, write your own taskstats application or go a long way to port the in-kernel implementation of pagg/job/csa.
However, the "Externded acocunting fields" patch does not provide you straight forward answer. The patch provides accounting data only at process termination (just like the BSD accounting) and it seems that you want to see which run-away application (ie, alive) eating up your disk. The taskstats interface offers a query mode (command-response), but currently only delayacct uses that mode. We would need to make those data available in the query mode in order for application to see accounting data of live processes.
Of course, you can always kill all possible offenders.:) Then, you can find out which one was the bad guy with the data.
Thanks,
- jay
Quoted 46 lines
Sure. Doing this is actually fairly tricky because disk writes are almost always deferred. We'd need to remember which process dirtied some memory, then track that info all the way down to the disk IO level, then perform the accounting operations at IO submit-time or completion time, on a per-page basis. It isn't rocket-science, but it's a lot of stuff and some overhead.
Quoted 9 lines
meanwhile i found blktrace and read into the documenation. looks really cool and seems to be very powerful tool - but it it`s seems a little bit "oversized" and not the right tool for this. seems to be for tracing/debugging/analysis
what about http://lkml.org/lkml/2005/9/12/89 "with following patch, userspace processes/utilities will be able to access per process I/O statistics. for example, top like utilites can use this information" which has been posted to lkml one year ago ? any update on this ?
csa-accounting-taskstats-update.patch makes that information available to userspace.
But it's approximate, because
- it doesn't account for disk readahead
- it doesn't account for pagefault-initiated reads (althought it easily could - Jay?)
- it overaccounts for a process writing to an already-dirty page.
(We could fix this too: nuke the existing stuff and do
current->wchar += PAGE_CACHE_SIZE;in __set_page_dirty_[no]buffers().) (But that ends up being wrong if someone truncates the file before it got written)
- it doesn't account for file readahead (although it easily could)
- it doesn't account for pagefault-initiated readahead (it could)
hm. There's actually quite a lot we could do here to make these fields more accurate and useful. A lot of this depends on what the definition of these fields _is_. Is is just for disk IO? Is it supposed to include console IO, or what?
Quoted 27 lines · On Thu, 28 Sep 2006 11:55:38 -0700 Jay Lan <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 3 lines · On Wed, 27 Sep 2006 23:22:02 +0200 "roland" <[email protected]> wrote:
thanks. tried to contact redflag, but they don`t answer. maybe support is being on holiday.... !?
Quoted 10 lines
Quoted 2 lines
linux kernel hackers - there is really no standard way to watch i/o metrics (bytes read/written) at process level?
The patch csa-accounting-taskstats-update.patch in current -mm kernels (whcih is planned for 2.6.19) does have per-process chars-read and chars-written accounting ("Extended accounting fields"). That's probably not waht you really want, although it might tell you what you want to know.
Quoted 2 lines
it`s extremly hard for the admin to track down, what process is hogging the disk - especially if there is more than one task consuming cpu.
Rolend,
The per-process chars-read and chars-writeen accounting is made available through taskstats interface (see
Documentation/accounting/taskstats.txt) in 2.6.18-mm1 kernel. Unfortunately, the user-space CSA package is still a few months away. You may, for now, write your own taskstats application or go a long way to port the in-kernel implementation of pagg/job/csa.
Quoted 8 lines
However, the "Externded acocunting fields" patch does not provide you straight forward answer. The patch provides accounting data only at process termination (just like the BSD accounting) and it seems that you want to see which run-away application (ie, alive) eating up your disk. The taskstats interface offers a query mode (command-response), but currently only delayacct uses that mode. We would need to make those data available in the query mode in order for application to see accounting data of live processes.
ow. That is a rather important enhancement to have.
Quoted 22 lines
Quoted 22 lines
csa-accounting-taskstats-update.patch makes that information available to userspace.
But it's approximate, because
- it doesn't account for disk readahead
- it doesn't account for pagefault-initiated reads (althought it easily could - Jay?)
- it overaccounts for a process writing to an already-dirty page.
(We could fix this too: nuke the existing stuff and do
current->wchar += PAGE_CACHE_SIZE;in __set_page_dirty_[no]buffers().) (But that ends up being wrong if someone truncates the file before it got written)
- it doesn't account for file readahead (although it easily could)
Quoted 7 lines
Quoted 7 lines
- it doesn't account for pagefault-initiated readahead (it could)
hm. There's actually quite a lot we could do here to make these fields more accurate and useful. A lot of this depends on what the definition of these fields _is_. Is is just for disk IO? Is it supposed to include console IO, or what?
I'd be interested in your opinions on all the above, please.
andrew, jay - thank you very much for giving your comments on this issue and discussing this!
Quoted 1 line
I'd be interested in your opinions on all the above, please.
just kidding here a little bit (pardon!) - but from a user/admin perspective we "just" want something like this:
http://www.my-vserver.de/stuff/linux/process-explorer.png (especially mind the I/O columns in the middle of the screenshot)
awesome tool - but unfortunately mr. russinovich is working for the wrong company/developing for the wrong operating system.... ;)
regards
roland
----- Original Message ----- From: "Andrew Morton" <[email protected]>
To: "Jay Lan" <[email protected]>
Cc: "roland" <[email protected]>; "Fengguang Wu" <[email protected]>; <[email protected]>; <[email protected]>
Sent: Thursday, September 28, 2006 9:09 PM
Subject: Re: I/O statistics per process
Quoted 73 lines
Quoted 27 lines · On Thu, 28 Sep 2006 11:55:38 -0700 Jay Lan <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 3 lines · On Wed, 27 Sep 2006 23:22:02 +0200 "roland" <[email protected]> wrote:
thanks. tried to contact redflag, but they don`t answer. maybe support is being on holiday.... !?
Quoted 10 lines
Quoted 2 lines
linux kernel hackers - there is really no standard way to watch i/o metrics (bytes read/written) at process level?
The patch csa-accounting-taskstats-update.patch in current -mm kernels (whcih is planned for 2.6.19) does have per-process chars-read and chars-written accounting ("Extended accounting fields"). That's probably not waht you really want, although it might tell you what you want to know.
Quoted 2 lines
it`s extremly hard for the admin to track down, what process is hogging the disk - especially if there is more than one task consuming cpu.
Rolend,
The per-process chars-read and chars-writeen accounting is made available through taskstats interface (see
Documentation/accounting/taskstats.txt) in 2.6.18-mm1 kernel. Unfortunately, the user-space CSA package is still a few months away. You may, for now, write your own taskstats application or go a long way to port the in-kernel implementation of pagg/job/csa.Quoted 8 lines
However, the "Externded acocunting fields" patch does not provide you straight forward answer. The patch provides accounting data only at process termination (just like the BSD accounting) and it seems that you want to see which run-away application (ie, alive) eating up your disk. The taskstats interface offers a query mode (command-response), but currently only delayacct uses that mode. We would need to make those data available in the query mode in order for application to see accounting data of live processes.
ow. That is a rather important enhancement to have.
Quoted 22 lines
Quoted 22 lines
csa-accounting-taskstats-update.patch makes that information available to userspace.
But it's approximate, because
- it doesn't account for disk readahead
- it doesn't account for pagefault-initiated reads (althought it easily could - Jay?)
- it overaccounts for a process writing to an already-dirty page.
(We could fix this too: nuke the existing stuff and do
current->wchar+=PAGE_CACHE_SIZE;in __set_page_dirty_[no]buffers().) (But that ends up being wrong if someone truncates the file before it got written)
- it doesn't account for file readahead (although it easily could)
Quoted 7 lines
Quoted 7 lines
- it doesn't account for pagefault-initiated readahead (it could)
hm. There's actually quite a lot we could do here to make these fields more accurate and useful. A lot of this depends on what the definition of these fields _is_. Is is just for disk IO? Is it supposed to include console IO, or what?
I'd be interested in your opinions on all the above, please.
Quoted 37 lines · Andrew Morton
Quoted 25 lines · On Thu, 28 Sep 2006 11:55:38 -0700 Jay Lan <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 3 lines · On Wed, 27 Sep 2006 23:22:02 +0200 "roland" <[email protected]> wrote:
thanks. tried to contact redflag, but they don`t answer. maybe support is being on holiday.... !?
Quoted 9 lines
Quoted 2 lines
linux kernel hackers - there is really no standard way to watch i/o metrics (bytes read/written) at process level?
The patch csa-accounting-taskstats-update.patch in current -mm kernels (whcih is planned for 2.6.19) does have per-process chars-read and chars-written accounting ("Extended accounting fields"). That's probably not waht you really want, although it might tell you what you want to know.
Quoted 2 lines
it`s extremly hard for the admin to track down, what process is hogging the disk - especially if there is more than one task consuming cpu.
Rolend,
The per-process chars-read and chars-writeen accounting is made available through taskstats interface (see
Documentation/accounting/taskstats.txt) in 2.6.18-mm1 kernel. Unfortunately, the user-space CSA package is still a few months away. You may, for now, write your own taskstats application or go a long way to port the in-kernel implementation of pagg/job/csa.Quoted 8 lines
However, the "Externded acocunting fields" patch does not provide you straight forward answer. The patch provides accounting data only at process termination (just like the BSD accounting) and it seems that you want to see which run-away application (ie, alive) eating up your disk. The taskstats interface offers a query mode (command-response), but currently only delayacct uses that mode. We would need to make those data available in the query mode in order for application to see accounting data of live processes.
Quoted 1 line
ow. That is a rather important enhancement to have.
Yes, it is needed to provide accounting on live processes. Both BSD and CSA traditionally focused on completed processes. I guess that was the difference between a system accounting and system monitoring?
I certainly can make this enhancement. :)
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
csa-accounting-taskstats-update.patch makes that information available to userspace.
But it's approximate, because
- it doesn't account for disk readahead
- it doesn't account for pagefault-initiated reads (althought it easily could - Jay?)
- it overaccounts for a process writing to an already-dirty page.
(We could fix this too: nuke the existing stuff and do
current->wchar += PAGE_CACHE_SIZE;
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
in __set_page_dirty_[no]buffers().) (But that ends up being wrong if someone truncates the file before it got written)
- it doesn't account for file readahead (although it easily could)
- it doesn't account for pagefault-initiated readahead (it could)
Mmm, i am not a true FS I/O person. The data collection patches i submitted in Nov 2004 was the code i inherited and has been used in production system by our CSA customers. We lost a bit in contents and accuracy when CSA was ported from IRIX to Linux. I am sure there is room for improvement without much overhead. Maybe FS I/O guys can chip in?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
hm. There's actually quite a lot we could do here to make these fields more accurate and useful. A lot of this depends on what the definition of these fields _is_. Is is just for disk IO? Is it supposed to include console IO, or what?
Yes, the char_read and char_written are only for disk I/O.
Quoted 2 lines
I'd be interested in your opinions on all the above, please.
Sorry i can not answer you on data colleciton code.
Thanks,
- jay
Quoted 1 line
Quoted 5 lines · On Thu, 28 Sep 2006 15:00:17 -0700 Jay Lan <[email protected]> wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
in __set_page_dirty_[no]buffers().) (But that ends up being wrong if someone truncates the file before it got written)
- it doesn't account for file readahead (although it easily could)
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
- it doesn't account for pagefault-initiated readahead (it could)
Mmm, i am not a true FS I/O person. The data collection patches i submitted in Nov 2004 was the code i inherited and has been used in production system by our CSA customers. We lost a bit in contents and accuracy when CSA was ported from IRIX to Linux. I am sure there is room for improvement without much overhead.
OK, well it sounds like we're free to define these in any way we like. So we actually get to make them mean something useful - how nice.
I hereby declare: "approxmiately equal to the number of filesystem bytes which this task has caused to occur, or which shall occur in the near future".
Quoted 2 lines
Maybe FS
I/O guys can chip in?
I used to be one of them. I can take a look at doing this. Given the lack of any applciation to read the darn numbers out I guess I'll need to expose them in /proc for now. Yes, that monitoring patch (and an application to read from it!) would be appreciated, thanks.
hi!
didnt discover that there is anything new about this (andrew? jay?) or if some other person sent a patch , but id like to report that i came across a really nice tool which would immediately benefit from per-process i/o statistics feature.
please - this mail is not meant to clamor for such feature - it`s just to show up some more benefits if this feature would exist.
vmktree at http://vmktree.org/ is some really nice monitoring tool being able to graph performance statistics for a host running vmware virtual machines (closed source - evil - i know ;) - and it can break that statistics down to each virtual machine.
what`s hurting mostly here is that you have no clue how much I/O each of those virtual machine is generating - you may give sort of a "guess" that a machine with 100% idle cpu will not generate any significant amount of I/O, but vmktree would be so much more powerful with a per-process I/O statistic, since you can use your systems more efficient, because you would know about you I/O hogs, too.
having the ability to take such information from /proc would be a real killer feature - good for troubleshooting and also good for getting important statistics!
roland
ps:
this is another person, desperately seeking for a tool providing that information: http://www.tek-tips.com/viewthread.cfm?qid=1284288&page=4
----- Original Message ----- From: "Andrew Morton" <[email protected]>
To: "Jay Lan" <[email protected]>
Cc: "roland" <[email protected]>; "Fengguang Wu" <[email protected]>; <[email protected]>; <[email protected]>
Sent: Thursday, September 28, 2006 11:14 PM
Subject: Re: I/O statistics per process
Quoted 35 lines
Quoted 6 lines · On Thu, 28 Sep 2006 15:00:17 -0700 Jay Lan <[email protected]> wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
in __set_page_dirty_[no]buffers().) (But that ends up being wrong
if
someone truncates the file before it got written)- it doesn't account for file readahead (although it easily could)
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
- it doesn't account for pagefault-initiated readahead (it could)
Mmm, i am not a true FS I/O person. The data collection patches i submitted in Nov 2004 was the code i inherited and has been used in production system by our CSA customers. We lost a bit in contents and accuracy when CSA was ported from IRIX to Linux. I am sure there is room for improvement without much overhead.
OK, well it sounds like we're free to define these in any way we like. So we actually get to make them mean something useful - how nice.
I hereby declare: "approxmiately equal to the number of filesystem bytes which this task has caused to occur, or which shall occur in the near future".
Quoted 2 lines
Maybe FS
I/O guys can chip in?I used to be one of them. I can take a look at doing this. Given the lack
of any applciation to read the darn numbers out I guess I'll need to expose
them in /proc for now. Yes, that monitoring patch (and an application to read from it!) would be appreciated, thanks.
Hi,
Quoted 5 lines · roland
didn
t discover that there is anything new about this (andrew? jay?) or if some other person sent a patch , but id like to report that i came across a really nice tool which would immediately benefit from per-process i/o statistics feature.
Andrew has added kernel support to it in -mm tree. Check this commit log:
http://www.mail-archive.com/[email protected]/msg02975.html
io-accounting-core-statistics.patch
io-accounting-write-accounting.patch
io-accounting-write-cancel-accounting.patch
io-accounting-read-accounting-2.patch
io-accounting-read-accounting-nfs-fix.patch
io-accounting-read-accounting-cifs-fix.patch
io-accounting-direct-io.patch
io-accounting-report-in-procfs.patch
io-accounting-via-taskstats.patch
io-accounting-add-to-getdelays.patch
Regards,
Fengguang Wu
this is really great news!
thank you!
----- Original Message ----- From: "Fengguang Wu" <[email protected]>
To: "roland" <[email protected]>
Cc: "Andrew Morton" <[email protected]>; "Jay Lan" <[email protected]>; <[email protected]>; <[email protected]>
Sent: Friday, December 08, 2006 2:22 AM
Subject: Re: I/O statistics per process
Quoted 26 lines
Hi,
Quoted 5 lines · roland
didn
t discover that there is anything new about this (andrew? jay?) or if some other person sent a patch , but id like to report that i came across a really nice tool which would immediately benefit from per-process i/o statistics feature.Andrew has added kernel support to it in -mm tree. Check this commit log:
http://www.mail-archive.com/[email protected]/msg02975.htmlio-accounting-core-statistics.patch
io-accounting-write-accounting.patch
io-accounting-write-cancel-accounting.patch
io-accounting-read-accounting-2.patch
io-accounting-read-accounting-nfs-fix.patch
io-accounting-read-accounting-cifs-fix.patch
io-accounting-direct-io.patch
io-accounting-report-in-procfs.patch
io-accounting-via-taskstats.patch
io-accounting-add-to-getdelays.patchRegards,
Fengguang Wu