New Metrics to measure Load average
Hi Peter,
(1) Yes it would be a good idea to separate them for each CPU. The we have that as an information too.
Out project is as follows:
Our project which is conducted as part of CEL(and CSIRO) requires to separate the load along individual user lines. This project is a GRID computing project.
We have developed a prediction model called "Free Load Profile Model for Load Profile Prediction" for the purpose of scheduling in the Grid.
Irrespective of number of CPUs in the system we need to know the total load which has been exerted on all CPUs (on the system) by individual users and system/root.
As required by the prediction model then we need to know the break down of that load along the user lines.
It is up to the prediction model to handle the issues with number CPUs as it has such a capability.
(2) I am working at Electrical Engineering. My number until next Tuesday would be 93517227. IF you are living close to the Sydney Uni we will be able to discuss things later.
I had a discussion with Rusty Russel and he had few suggestions for me too.
Thanks
Sena Seneviratne
Computer Engineering Lab
School of Electrical and Information Engineering
Sydney University
Australia
Quoted 39 lines · you
Quoted 11 lines · sena seneviratne
Hi Shailabh,Vadim,
Zoltan, Peter Williams,
David Osojnik, Nishanth Aravamudan,
Balbir Singh, Andrew Morton(1) Please give us your valuable comments on this important change to introduce a new Metric to measure Load average.
Currently/proc/loadavgreports only the resultant value.
We have done the scheduling in a Grid project. As a part of that we had to do some changes to the Linux kernel as well.
The problem with the load metric of current Linux/Unix is that it measures CPU load and Disk load without indicating the true nature of the load, thereby creating some confusion among the readers. For example, if a CPU bound task switches on to read a large chunk of disk data, then the load average value would still continue to indicate this activity as a load, yet the true CPU load during this period would have been zero. This situation triggered us to make necessary additions to the kernel so that CPU load and Disk load could be reported separately. Further the specialisation of load helped our model to perform predictions when there is interference between CPU and Disk IO loads. In the user mode, a new proc file called/proc/loadavguswould collect the new data according to a new format which would look like the following,
CPU DiskQuoted 7 lines
500 0.7 0 501 0.9 1 502 0.9 0 503 1.03 1 504 0.93 0 505 1.0 0What do you think about this change?
I think that it is useful to have the separate components (cpu and disk I/O) of the load average available but, while I have no objection to the data being made available on a per user basis, I think that it should primarily be provided on a per CPU and total system basis. This is because most people are only interested in the loads for the system as a whole while per CPU data is a good indication of how well load balancing is working.
This latter observation leads me to suggest that "weighted" CPU load per CPU would also be useful. So output like:
CPU0 0.3 0.45 0.9
CPU1 0.4 0.5 0.8
TOTAL 0.7 0.95 1.7would be nice to have where the columns were raw CPU load, weighted CPU load and disk I/O load respectively.
In summary, I like the idea but think that you should do it properly and provide the data that will interest most users and then add the per user data that meets your specific needs as an optional extra.
Peter
PS I live very close to Sydney University if you want to discuss these ideas in person. --
Peter Williams [email protected]"Learning, n. The kind of ignorance distinguishing the studious."
-- Ambrose Bierce
Hi Andrew,
Hi Nagar,
Thanks for your reply and for your time.
Few years ago the few additions I have explained, have been implemented in linux 2.4.18 kernel at the Sydney Uni as part of a large research project.
Also few years ago I have posted many posts about this topic to this forum. Few very experience hands were advising me in coding at that time. To prove this I have all the e-mails with me. Yet then I had not explained to this forum about the research project which involved prediction algorithm as then it had not been published.
Thereafter few research papers have been published with regard to this. 1 in US.
Currently I am preparing the changes to a more recent linux patch.
In fact my question in the post was about performance testing after the changes being done.
--2) Now about the tests
--As I have documented all this yet need to perform some standard tests for the sake of completion. --What tests should I carry out to prove that the system is still intact?
--
--Please tell me whether the below is correct?
--(a) As suggested by the http://kernel-perf.sourceforge.net/ the lmbench and re-aim-7 test packages can be used to test the ----performance of the kernel before making changes and after. (Not done as yet)
--(-b) Further tests have been carried out to check the response time of short tasks before making changes and after making --changes. The results indicated that there was no difference in the response time after introducing the changes to the kernel (done)
---(c) Thereafter the tests have been carried out to check the runtime of long tasks before and after making changes. The results of the tests revealed that there is no change in reported runtime in both occasions.(done)
Thanks
Sena Seneviratne
Computer Engineering Lab
School of Electrical and Information Engineering
Sydney University
Australia
Quoted 20 lines · you
Quoted 2 lines · On Fri, 16 Jun 2006 21:46:08 +0530 Balbir Singh <[email protected]> wrote:
I think it is easier to make the changes to be per-task and then in user-space account all information for the user (using the per-task data).
Yes please try to do this. There are all sorts of ways in which we could combine these stats on behalf of a particular application scenario. Each scheme involves some sort of data loss, so each application needs new code to get the information which _it_ wants.
We really should work on presenting the relevant information to userspace in a complete, efficient and un-post-processed manner so that application-specific userspace code can combine it in the manner which it desires.
Balbir's new code (in -mm) is supposed to be the basis of _all_ new per-task accounting, so you should look at what additional information is needed and then find a way to transport it to userspace via Balbir's proposed framework, thanks.
Quoted 8 lines · sena seneviratne
In fact my question in the post was about performance testing after the changes being done.
--2) Now about the tests
--As I have documented all this yet need to perform some standard tests for the sake of completion.
--What tests should I carry out to prove that the system is still intact?
Quoted 7 lines
--Please tell me whether the below is correct?
--(a) As suggested by the http://kernel-perf.sourceforge.net/ the lmbench and re-aim-7 test packages can be used to test the ----performance of the kernel before making changes and after. (Not done as yet)
To measure impact of patches for a kernel tree, Contest (available from http://freshmeat.net/projects/contest/)
is a good start. lmbench is also useful.
Quoted 6 lines
--(-b) Further tests have been carried out to check the response time
of short tasks before making changes and after making --changes. The results indicated that there was no difference in the response time after introducing the changes to the kernel (done)---(c) Thereafter the tests have been carried out to check the runtime
of long tasks before and after making changes. The results of the tests revealed that there is no change in reported runtime in both occasions.(done)
Why is there a distinction between short and long running tasks when overall performance overhead of the kernel needs to be verified ?
Hi Nagar,
Thanks for your response and time.
Yes a good question.
(a) The short tasks to measure the response time itself after applying the division of load and before.
(b) The long tasks means HPC tasks to measure the load signal after applying the division of load and before.
I have run it for 600s, 1000s, and 1500s.
The both these tests were successful.
What I want to document is some standard tests just like using lmbench and re-aim-7.
I will go through your web site as well
Thanks
Sena Seneviratne
Computer Engineering Lab
School of Electrical and Information Engineering
Sydney University
Australia
Quoted 30 lines · you
Quoted 8 lines · sena seneviratne
In fact my question in the post was about performance testing after the changes being done.
--2) Now about the tests
--As I have documented all this yet need to perform some standard tests for the sake of completion.
--What tests should I carry out to prove that the system is still intact?Quoted 4 lines
--Please tell me whether the below is correct?
--(a) As suggested by the http://kernel-perf.sourceforge.net/ the lmbench
and re-aim-7 test packages can be used to test the ----performance of the kernel before making changes and after. (Not done as yet)To measure impact of patches for a kernel tree, Contest (available from http://freshmeat.net/projects/contest/)
is a good start. lmbench is also useful.Quoted 5 lines
--(-b) Further tests have been carried out to check the response time of
short tasks before making changes and after making --changes. The results indicated that there was no difference in the response time after introducing the changes to the kernel (done)---(c) Thereafter the tests have been carried out to check the runtime of
long tasks before and after making changes. The results of the tests revealed that there is no change in reported runtime in both occasions.(done)Why is there a distinction between short and long running tasks when overall performance overhead of the kernel needs to be verified ?
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/