Skip to contentSkip to conversation

end-of-proces handling for acct-csa

PATCH In review Jay Lan 5 participants 11 messages Feb 25, 2005 – Mar 8, 2005

lore
Share & tools
lore

Jay Lan

author
#0 lore raw

PATCH 1/1 end-of-proces handling for acct-csa

Since my idea of providing an accounting framework was considered 'overkill', here i submit a tiny patch just to allow CSA to handle end-of-process (eop) situation by saving off accounting data before a task_struct is disposed.

This patch is to modify the acct_process() in acct.c, which is invoked from do_exit() to handle eop for BSD accounting. Now the acct_process() wrapper will also take care of CSA, if it has been loaded. If the CSA module has been loaded, a CSA routine will be invoked to construct a CSA job record and to write the record to the CSA accounting file.

This patch only touchs one file: kernel/acct.c.

kernel/acct.c +11 -1
Index: linux/kernel/acct.c
===================================================================
@@ -73,6 +73,11 @@ int acct_parm[3] = {4, 2, 30};
 /*
  * External references and all of the globals.
  */
+
+/* do_exit hook used by CSA */
+void (*do_exit_csa)(int, struct task_struct *) = NULL;
+EXPORT_SYMBOL_GPL(do_exit_csa);
+
 static void do_acct_process(long, struct file *);
 
 /*
@@ -504,12 +509,17 @@ static void do_acct_process(long exitcod
 }
 
 /*
- * acct_process - now just a wrapper around do_acct_process
+ * acct_process - now just a wrapper around
+ *	do_acct_process	- for BSD accounting
+ *	do_exit_csa	- for CSA
  */
 void acct_process(long exitcode)
 {
 	struct file *file = NULL;
 
+	if (do_exit_csa != NULL)
+		do_exit_csa(exitcode, current);
+
 	/*
 	 * accelerate the common fastpath:
 	 */
#1 lore raw
Quoted 6 lines · Jay Lan

Since my idea of providing an accounting framework was considered
'overkill', here i submit a tiny patch just to allow CSA to handle end-of-process (eop) situation by saving off accounting data before a task_struct is disposed.

Quoted 6 lines

This patch is to modify the acct_process() in acct.c, which is invoked from do_exit() to handle eop for BSD accounting. Now the acct_process() wrapper will also take care of CSA, if it has been loaded. If the CSA module has been loaded, a CSA routine will be invoked to construct a CSA job record and to write the record to the CSA accounting file.

I really don't want to have to (and shouldn't need to) become an accounting person, but as there seems to be a communication problem somewhere..

Please, you guys are the subject matter experts. Put your heads together and come up with something.

We shouldn't be putting CSA stuff over here and ELSA stuff over there.

We should be putting our heads together and coming up with a sensible and adequate accounting solution for Linux. Or the kernel, at least.

The ELSA team appear to be able to do that in userspace with the existing process accounting machinery and a single outcall from fork(). That's great.

Why is that not adequate for CSA? What end-user features does CSA offer that ELSA can not? Could those features be implemented (in userspace) if we also added an exit() outcall?

#2 lore raw
Quoted 14 lines · Andrew Morton
Quoted 6 lines · Jay Lan

Since my idea of providing an accounting framework was considered
'overkill', here i submit a tiny patch just to allow CSA to handle end-of-process (eop) situation by saving off accounting data before a task_struct is disposed.

Quoted 6 lines

This patch is to modify the acct_process() in acct.c, which is invoked from do_exit() to handle eop for BSD accounting. Now the acct_process() wrapper will also take care of CSA, if it has been loaded. If the CSA module has been loaded, a CSA routine will be invoked to construct a CSA job record and to write the record to the CSA accounting file.

Quoted 5 lines

I really don't want to have to (and shouldn't need to) become an accounting person, but as there seems to be a communication problem somewhere..

Please, you guys are the subject matter experts. Put your heads together and come up with something.

I completely agree with that and we will continue this conversation in private with Jay and all people involved to come up with an appropriate solution.

Guillaume

author
#3 lore raw

Hi Andrew,

You asked:

Quoted 4 lines

In other words: given that ELSA can do its thing via existing accounting interfaces and a fork notifier, why does CSA need to add lots more kernel code?

And i explained:

Quoted 12 lines

Here are some codes from do_exit() starting line 813 (based on 2.6.11-rc4-mm1):

813        acct_update_integrals(tsk);
814        update_mem_hiwater(tsk);
815        group_dead = atomic_dec_and_test(&tsk->signal->live);
816        if (group_dead) {

817 del_timer_sync(&tsk->signal->real_timer);
818 acct_process(code);
819 }
820 exit_mm(tsk);

Quoted 5 lines

The acct_process() is called to save off BSD accounting data at line 818. The next statement at 820, tsk->mm is disposed and all data saved at tsk->mm is gone, including memory hiwater marks information saved at line 814. The complete tsk is disposed before exit of do_exit() routine.

I was hoping Guilluame could jump in himself...

Quoted 9 lines · he
Quoted 6 lines

(Jay asked)
I spent some time trying to understand how ELSA save the per-process accounting data before the task_struct data structure gets disposed, but failed to find anything. My assumption would be that ELSA does not collection those data in the kernel itself? Instead, it would read the pacct file created by either BSD or CSA?

Yes you're right, ELSA reads accounting file created by BSD or CSA. We don't make accounting. I think that the communication problem is here

Guilluame's reply should answer your question. ELSA does not collect accounting data in the kernel as BSD/CSA does. It uses accounting data collected by BSD/CSA.

The exit hook is essential for CSA to save off data before the data is gone, A netlink type of thing does not help. BSD is in the same situation. You can not replace the acct_process() call with a netlink. If ELSA is to use the enhanced accounting data, it needs the CSA eop handling at exit as well.

Thanks,
- jay

Quoted 28 lines · Guillaume Thouvenin
Quoted 14 lines · Andrew Morton
Quoted 11 lines · Jay Lan

Since my idea of providing an accounting framework was considered 'overkill', here i submit a tiny patch just to allow CSA to handle end-of-process (eop) situation by saving off accounting data before a task_struct is disposed.

This patch is to modify the acct_process() in acct.c, which is invoked from do_exit() to handle eop for BSD accounting. Now the acct_process() wrapper will also take care of CSA, if it has been loaded. If the CSA module has been loaded, a CSA routine will be invoked to construct a CSA job record and to write the record to the CSA accounting file.

Quoted 5 lines

I really don't want to have to (and shouldn't need to) become an accounting person, but as there seems to be a communication problem somewhere..

Please, you guys are the subject matter experts. Put your heads together and come up with something.

I completely agree with that and we will continue this conversation in private with Jay and all people involved to come up with an appropriate solution.

Guillaume

#4 lore raw
Quoted 5 lines · Jay Lan

The exit hook is essential for CSA to save off data before the data is gone, A netlink type of thing does not help. BSD is in the same situation. You can not replace the acct_process() call with a netlink. If ELSA is to use the enhanced accounting data, it needs the CSA eop handling at exit as well.

Why replace the acct_process()? The problem here is to add a new hook in the do_fork() and you can use the BSD accounting hook acct_process() which is already in the exit() routine. We don't need to replace it with a netlink because today there are no user space applications that need it.

Best regards,
Guillaume

author
#5 lore raw

Sorry I was not clear on my point.

I was trying to point out that, an exit hook for BSD and CSA is essential to save accounting data before the data is gone. That can not be done with a netlink.

So, my patch was to keep acct_process as a wrapper, which would then call do_exit_csa() for CSA and call do_acct_process for BSD.

Thanks,
- jay

Quoted 17 lines · Guillaume Thouvenin
Quoted 5 lines · Jay Lan

The exit hook is essential for CSA to save off data before the data is gone, A netlink type of thing does not help. BSD is in the same situation. You can not replace the acct_process() call with a netlink. If ELSA is to use the enhanced accounting data, it needs the CSA eop handling at exit as well.

Why replace the acct_process()? The problem here is to add a new hook in the do_fork() and you can use the BSD accounting hook acct_process() which is already in the exit() routine. We don't need to replace it with a netlink because today there are no user space applications that need it.

Best regards,
Guillaume

#6 lore raw
Quoted 9 lines · Jay Lan

Sorry I was not clear on my point.

I was trying to point out that, an exit hook for BSD and CSA is essential to save accounting data before the data is gone. That can not be done with a netlink.

So, my patch was to keep acct_process as a wrapper, which would then call do_exit_csa() for CSA and call do_acct_process for BSD.

Is it possible to merge BSD and CSA? I mean with CSA, there is a part that does per-process accounting. For exemple in the linux-2.6.9.acct_mm.patch the two functions update_mem_hiwater() and csa_update_integrals() update fields in the current (and parent) process. So maybe you can improve the BSD per-process accounting or maybe CSA can replace the BSD per-process accounting?

Guillaume

#7 lore raw
Quoted 6 lines · Guillaume Thouvenin

Is it possible to merge BSD and CSA? I mean with CSA, there is a part that does per-process accounting. For exemple in the linux-2.6.9.acct_mm.patch the two functions update_mem_hiwater() and csa_update_integrals() update fields in the current (and parent) process. So maybe you can improve the BSD per-process accounting or maybe CSA can replace the BSD per-process accounting?

The BSD accounting tools will expect the data to be written in a certain format, so we can't change that. We could, however, unify the data collection under CONFIG_ACCOUNTING or something, that collects all the data available (which would be the sum of the data collected by the BSD and CSA calls) and then throw away data when writing to the BSD log so its format remains the same.

That would simplify data collection since there would just be one set of calls, and data reporting could be driven by userspace (whether it's in old-style sys_acct format, or new-style data that CSA/ELSA defines).

Jesse

author
#8 lore raw
In reply to Guillaume Thouvenin

I did not look into the userspace commands supported in BSD accounting on the dependency on the format of /var/account/pacct file.

The accounting exit hook allows BSD/CSA to save accounting data stored in task_struct to internally kept data structure and then writes to their respective accounting file. The format of the accounting output file of the two are different, with CSA being the superset of BSD and also with CSA carrying concept of grouping of processes.

It would be an interesting project to merge BSD and CSA, but it is going to take some surgical work on both products, not an easy one.

- jay

Quoted 21 lines · Guillaume Thouvenin
Quoted 9 lines · Jay Lan

Sorry I was not clear on my point.

I was trying to point out that, an exit hook for BSD and CSA is essential to save accounting data before the data is gone. That can not be done with a netlink.

So, my patch was to keep acct_process as a wrapper, which would then call do_exit_csa() for CSA and call do_acct_process for BSD.

Is it possible to merge BSD and CSA? I mean with CSA, there is a part that does per-process accounting. For exemple in the linux-2.6.9.acct_mm.patch the two functions update_mem_hiwater() and csa_update_integrals() update fields in the current (and parent) process. So maybe you can improve the BSD per-process accounting or maybe CSA can replace the BSD per-process accounting?

Guillaume

#9 lore raw
In reply to Guillaume Thouvenin
Quoted 6 lines · Guillaume Thouvenin

Is it possible to merge BSD and CSA? I mean with CSA, there is a part that does per-process accounting. For exemple in the linux-2.6.9.acct_mm.patch the two functions update_mem_hiwater() and csa_update_integrals() update fields in the current (and parent) process. So maybe you can improve the BSD per-process accounting or maybe CSA can replace the BSD per-process accounting?

Yes, that was also my preferred direction - make CSA able to also write BSD acct format, and replace the existing BSD accounting with CSA. However it seems this will still increase the amount of kernel code quite a bit.

Sorry for not going into any details, I have to leave right now and will be offline for two weeks.

Tim

author
#10 lore raw

The patch i propose is tiny, simple and straight forward. It touches only one file and leaves the CSA code in a configurable loadable module. It broke nobody's code and it does not need to redesign existing BSD kernel code and utilities.

If we are to merge the code, there are some detailed discussion needed to happen on implementation deail. We are talking about supporting two different internal formats by one piece of code. We need to maintain BSD acct format because BSD utilities count on it.

If we are to combine two formats into one, we then need to modify BSD utilies to understand the new format. How about the backwards compatibility?

Now this is really an overkill. All i asked for was only adding a few lines to acct.c.

Thanks,
- jay

Quoted 18 lines · Tim Schmielau
Quoted 6 lines · Guillaume Thouvenin

Is it possible to merge BSD and CSA? I mean with CSA, there is a part that does per-process accounting. For exemple in the linux-2.6.9.acct_mm.patch the two functions update_mem_hiwater() and csa_update_integrals() update fields in the current (and parent) process. So maybe you can improve the BSD per-process accounting or maybe CSA can replace the BSD per-process accounting?

Yes, that was also my preferred direction - make CSA able to also write BSD acct format, and replace the existing BSD accounting with CSA. However it seems this will still increase the amount of kernel code quite a bit.

Sorry for not going into any details, I have to leave right now and will be offline for two weeks.

Tim