end-of-proces handling for acct-csa
PATCH 1/1 end-of-proces handling for acct-csa
Since my idea of providing an accounting framework was considered 'overkill', here i submit a tiny patch just to allow CSA to handle end-of-process (eop) situation by saving off accounting data before a task_struct is disposed.
This patch is to modify the acct_process() in acct.c, which is invoked from do_exit() to handle eop for BSD accounting. Now the acct_process() wrapper will also take care of CSA, if it has been loaded. If the CSA module has been loaded, a CSA routine will be invoked to construct a CSA job record and to write the record to the CSA accounting file.
This patch only touchs one file: kernel/acct.c.
kernel/acct.c +11 -1
@@ -73,6 +73,11 @@ int acct_parm[3] = {4, 2, 30};/* * External references and all of the globals. */ + +/* do_exit hook used by CSA */ +void (*do_exit_csa)(int ,struct task_struct *) =NULL ; +EXPORT_SYMBOL_GPL (do_exit_csa); +static void do_acct_process (long ,struct file *);/*
@@ -504,12 +509,17 @@ static void do_acct_process(long exitcod }/* -* acct_process - now just a wrapper around do_acct_process +* acct_process - now just a wrapper around +* do_acct_process - for BSD accounting +* do_exit_csa - for CSA */ void acct_process (long exitcode) {struct file *file =NULL ; +if (do_exit_csa !=NULL ) +do_exit_csa (exitcode, current); +/* * accelerate the common fastpath: */
Quoted 6 lines · Jay Lan
Since my idea of providing an accounting framework was considered
'overkill', here i submit a tiny patch just to allow CSA to handle end-of-process (eop) situation by saving off accounting data before atask_structis disposed.
Quoted 6 lines
This patch is to modify the
acct_process()inacct.c, which is invoked fromdo_exit()to handle eop for BSD accounting. Now theacct_process()wrapper will also take care of CSA, if it has been loaded. If the CSA module has been loaded, a CSA routine will be invoked to construct a CSA job record and to write the record to the CSA accounting file.
I really don't want to have to (and shouldn't need to) become an accounting person, but as there seems to be a communication problem somewhere..
Please, you guys are the subject matter experts. Put your heads together and come up with something.
We shouldn't be putting CSA stuff over here and ELSA stuff over there.
We should be putting our heads together and coming up with a sensible and adequate accounting solution for Linux. Or the kernel, at least.
The ELSA team appear to be able to do that in userspace with the existing process accounting machinery and a single outcall from fork(). That's great.
Why is that not adequate for CSA? What end-user features does CSA offer that ELSA can not? Could those features be implemented (in userspace) if we also added an exit() outcall?
Quoted 14 lines · Andrew Morton
Quoted 6 lines · Jay Lan
Since my idea of providing an accounting framework was considered
'overkill', here i submit a tiny patch just to allow CSA to handle end-of-process (eop) situation by saving off accounting data before atask_structis disposed.Quoted 6 lines
This patch is to modify the
acct_process()inacct.c, which is invoked fromdo_exit()to handle eop for BSD accounting. Now theacct_process()wrapper will also take care of CSA, if it has been loaded. If the CSA module has been loaded, a CSA routine will be invoked to construct a CSA job record and to write the record to the CSA accounting file.
Quoted 5 lines
I really don't want to have to (and shouldn't need to) become an accounting person, but as there seems to be a communication problem somewhere..
Please, you guys are the subject matter experts. Put your heads together and come up with something.
I completely agree with that and we will continue this conversation in private with Jay and all people involved to come up with an appropriate solution.
Guillaume
Hi Andrew,
You asked:
Quoted 4 lines
In other words: given that ELSA can do its thing via existing accounting interfaces and a fork notifier, why does CSA need to add lots more kernel code?
And i explained:
Quoted 12 lines
Here are some codes from
do_exit()starting line 813 (based on 2.6.11-rc4-mm1):813 acct_update_integrals(tsk); 814 update_mem_hiwater(tsk); 815 group_dead = atomic_dec_and_test(&tsk->signal->live); 816 if (group_dead) {817
del_timer_sync(&tsk->signal->real_timer);
818acct_process(code);
819 }
820exit_mm(tsk);
Quoted 5 lines
The
acct_process()is called to save off BSD accounting data at line 818. The next statement at 820,tsk->mmis disposed and all data saved attsk->mmis gone, including memory hiwater marks information saved at line 814. The complete tsk is disposed before exit ofdo_exit()routine.
I was hoping Guilluame could jump in himself...
Quoted 9 lines · he
Quoted 6 lines
(Jay asked)
I spent some time trying to understand how ELSA save the per-process accounting data before thetask_structdata structure gets disposed, but failed to find anything. My assumption would be that ELSA does not collection those data in the kernel itself? Instead, it would read the pacct file created by either BSD or CSA?Yes you're right, ELSA reads accounting file created by BSD or CSA. We don't make accounting. I think that the communication problem is here
Guilluame's reply should answer your question. ELSA does not collect accounting data in the kernel as BSD/CSA does. It uses accounting data collected by BSD/CSA.
The exit hook is essential for CSA to save off data before the data is gone, A netlink type of thing does not help. BSD is in the same situation. You can not replace the acct_process() call with a netlink. If ELSA is to use the enhanced accounting data, it needs the CSA eop handling at exit as well.
Thanks,
- jay
Quoted 28 lines · Guillaume Thouvenin
Quoted 14 lines · Andrew Morton
Quoted 11 lines · Jay Lan
Since my idea of providing an accounting framework was considered 'overkill', here i submit a tiny patch just to allow CSA to handle end-of-process (eop) situation by saving off accounting data before a
task_structis disposed.This patch is to modify the
acct_process()inacct.c, which is invoked fromdo_exit()to handle eop for BSD accounting. Now theacct_process()wrapper will also take care of CSA, if it has been loaded. If the CSA module has been loaded, a CSA routine will be invoked to construct a CSA job record and to write the record to the CSA accounting file.Quoted 5 lines
I really don't want to have to (and shouldn't need to) become an accounting person, but as there seems to be a communication problem somewhere..
Please, you guys are the subject matter experts. Put your heads together and come up with something.
I completely agree with that and we will continue this conversation in private with Jay and all people involved to come up with an appropriate solution.
Guillaume
Quoted 5 lines · Jay Lan
The exit hook is essential for CSA to save off data before the data is gone, A netlink type of thing does not help. BSD is in the same situation. You can not replace the
acct_process()call with a netlink. If ELSA is to use the enhanced accounting data, it needs the CSA eop handling at exit as well.
Why replace the acct_process()? The problem here is to add a new hook in the do_fork() and you can use the BSD accounting hook acct_process() which is already in the exit() routine. We don't need to replace it with a netlink because today there are no user space applications that need it.
Best regards,
Guillaume
Sorry I was not clear on my point.
I was trying to point out that, an exit hook for BSD and CSA is essential to save accounting data before the data is gone. That can not be done with a netlink.
So, my patch was to keep acct_process as a wrapper, which would then call do_exit_csa() for CSA and call do_acct_process for BSD.
Thanks,
- jay
Quoted 17 lines · Guillaume Thouvenin
Quoted 5 lines · Jay Lan
The exit hook is essential for CSA to save off data before the data is gone, A netlink type of thing does not help. BSD is in the same situation. You can not replace the
acct_process()call with a netlink. If ELSA is to use the enhanced accounting data, it needs the CSA eop handling at exit as well.Why replace the
acct_process()? The problem here is to add a new hook in thedo_fork()and you can use the BSD accounting hookacct_process()which is already in theexit()routine. We don't need to replace it with a netlink because today there are no user space applications that need it.Best regards,
Guillaume
Quoted 9 lines · Jay Lan
Sorry I was not clear on my point.
I was trying to point out that, an exit hook for BSD and CSA is essential to save accounting data before the data is gone. That can not be done with a netlink.
So, my patch was to keep
acct_processas a wrapper, which would then calldo_exit_csa()for CSA and calldo_acct_processfor BSD.
Is it possible to merge BSD and CSA? I mean with CSA, there is a part that does per-process accounting. For exemple in the linux-2.6.9.acct_mm.patch the two functions update_mem_hiwater() and csa_update_integrals() update fields in the current (and parent) process. So maybe you can improve the BSD per-process accounting or maybe CSA can replace the BSD per-process accounting?
Guillaume
Quoted 6 lines · Guillaume Thouvenin
Is it possible to merge BSD and CSA? I mean with CSA, there is a part that does per-process accounting. For exemple in the linux-2.6.9.acct_mm.patch the two functions
update_mem_hiwater()andcsa_update_integrals()update fields in the current (and parent) process. So maybe you can improve the BSD per-process accounting or maybe CSA can replace the BSD per-process accounting?
The BSD accounting tools will expect the data to be written in a certain format, so we can't change that. We could, however, unify the data collection under CONFIG_ACCOUNTING or something, that collects all the data available (which would be the sum of the data collected by the BSD and CSA calls) and then throw away data when writing to the BSD log so its format remains the same.
That would simplify data collection since there would just be one set of calls, and data reporting could be driven by userspace (whether it's in old-style sys_acct format, or new-style data that CSA/ELSA defines).
Jesse
I did not look into the userspace commands supported in BSD accounting on the dependency on the format of /var/account/pacct file.
The accounting exit hook allows BSD/CSA to save accounting data stored in task_struct to internally kept data structure and then writes to their respective accounting file. The format of the accounting output file of the two are different, with CSA being the superset of BSD and also with CSA carrying concept of grouping of processes.
It would be an interesting project to merge BSD and CSA, but it is going to take some surgical work on both products, not an easy one.
- jay
Quoted 21 lines · Guillaume Thouvenin
Quoted 9 lines · Jay Lan
Sorry I was not clear on my point.
I was trying to point out that, an exit hook for BSD and CSA is essential to save accounting data before the data is gone. That can not be done with a netlink.
So, my patch was to keep
acct_processas a wrapper, which would then calldo_exit_csa()for CSA and calldo_acct_processfor BSD.Is it possible to merge BSD and CSA? I mean with CSA, there is a part that does per-process accounting. For exemple in the linux-2.6.9.acct_mm.patch the two functions
update_mem_hiwater()andcsa_update_integrals()update fields in the current (and parent) process. So maybe you can improve the BSD per-process accounting or maybe CSA can replace the BSD per-process accounting?Guillaume
Quoted 6 lines · Guillaume Thouvenin
Is it possible to merge BSD and CSA? I mean with CSA, there is a part that does per-process accounting. For exemple in the linux-2.6.9.acct_mm.patch the two functions
update_mem_hiwater()andcsa_update_integrals()update fields in the current (and parent) process. So maybe you can improve the BSD per-process accounting or maybe CSA can replace the BSD per-process accounting?
Yes, that was also my preferred direction - make CSA able to also write BSD acct format, and replace the existing BSD accounting with CSA. However it seems this will still increase the amount of kernel code quite a bit.
Sorry for not going into any details, I have to leave right now and will be offline for two weeks.
Tim
The patch i propose is tiny, simple and straight forward. It touches only one file and leaves the CSA code in a configurable loadable module. It broke nobody's code and it does not need to redesign existing BSD kernel code and utilities.
If we are to merge the code, there are some detailed discussion needed to happen on implementation deail. We are talking about supporting two different internal formats by one piece of code. We need to maintain BSD acct format because BSD utilities count on it.
If we are to combine two formats into one, we then need to modify BSD utilies to understand the new format. How about the backwards compatibility?
Now this is really an overkill. All i asked for was only adding a few lines to acct.c.
Thanks,
- jay
Quoted 18 lines · Tim Schmielau
Quoted 6 lines · Guillaume Thouvenin
Is it possible to merge BSD and CSA? I mean with CSA, there is a part that does per-process accounting. For exemple in the linux-2.6.9.acct_mm.patch the two functions
update_mem_hiwater()andcsa_update_integrals()update fields in the current (and parent) process. So maybe you can improve the BSD per-process accounting or maybe CSA can replace the BSD per-process accounting?Yes, that was also my preferred direction - make CSA able to also write BSD acct format, and replace the existing BSD accounting with CSA. However it seems this will still increase the amount of kernel code quite a bit.
Sorry for not going into any details, I have to leave right now and will be offline for two weeks.
Tim