A common layer for Accounting packages

Since the need of Linux system accounting has gone beyond what BSD accounting provides, i think it is a good idea to create a thin layer of common code for various accounting packages, such as BSD accounting, CSA, ELSA, etc. The hook to do_exit() at exit.c was changed to invoke a routine in the common code which would then invoke those accounting packages that register to the acct_common to handle do_exit situation.

Quoted 7 lines · Jay Lan

Since the need of Linux system accounting has gone beyond what BSD accounting provides, i think it is a good idea to create a thin layer of common code for various accounting packages, such as BSD accounting, CSA, ELSA, etc. The hook to do_exit() at exit.c was changed to invoke a routine in the common code which would then invoke those accounting packages that register to the acct_common to handle do_exit situation.

This all seems to be heading in the wrong direction. Do we really want to have lots of different system accounting packages all hooking into a generic we-cant-decide-what-to-do-so-we-added-some-pointless-overhead framework?

Can't we get _one_ accounting system in there, get it right, avoid the framework?

Quoted 6 lines · Jay Lan

Since the need of Linux system accounting has gone beyond what BSD accounting provides, i think it is a good idea to create a thin layer of common code for various accounting packages, such as BSD accounting, CSA, ELSA, etc. The hook to do_exit() at exit.c was changed to invoke a routine in the common code which would then invoke those accounting packages that register to the acct_common to handle do_exit situation.

This all seems to be heading in the wrong direction. Do we really want to have lots of different system accounting packages all hooking into a generic we-cant-decide-what-to-do-so-we-added-some-pointless-overhead framework?

Can't we get _one_ accounting system in there, get it right, avoid the framework?

Quoted 14 lines · Andrew Morton

Quoted 6 lines · Jay Lan

Since the need of Linux system accounting has gone beyond what BSD accounting provides, i think it is a good idea to create a thin layer of common code for various accounting packages, such as BSD accounting, CSA, ELSA, etc. The hook to do_exit() at exit.c was changed to invoke a routine in the common code which would then invoke those accounting packages that register to the acct_common to handle do_exit situation.

This all seems to be heading in the wrong direction. Do we really want to have lots of different system accounting packages all hooking into a generic we-cant-decide-what-to-do-so-we-added-some-pointless-overhead framework?

Quoted 2 lines

Can't we get _one_ accounting system in there, get it right, avoid the framework?

Is it possible to just merge the BSD accounting and the CSA accounting
by adding in the current BSD per-process accounting structure some missing fields like the mm integral provided by the CSA patch?

ELSA is just a user of the accounting data. We need a hook in the do_fork() routine to manage group of processes, not to do accounting.

Guillaume

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Lse-tech mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/lse-tech

Hello, everyone.

Quoted 11 lines · Andrew Morton

Quoted 6 lines · Jay Lan

Since the need of Linux system accounting has gone beyond what BSD accounting provides, i think it is a good idea to create a thin layer of common code for various accounting packages, such as BSD accounting, CSA, ELSA, etc. The hook to do_exit() at exit.c was changed to invoke a routine in the common code which would then invoke those accounting packages that register to the acct_common to handle do_exit situation.

This all seems to be heading in the wrong direction. Do we really

want to

Quoted 4 lines

have lots of different system accounting packages all hooking into a generic we-cant-decide-what-to-do-so-we-added-some-pointless-overhead framework?

Quoted 2 lines

Can't we get _one_ accounting system in there, get it right, avoid the framework?

I think there are two issues about system accounting framework.

Issue: 1) How to define the appropriate unit for accounting ? Current BSD-accountiong make a collection per process accounting information. CSA make additionally a collection per process-aggregation accounting.

It is appropriate to make the fork-exit event handling framework for definition of the process-aggregation, such as PAGG.

This system-accounting per process-aggregation is quite useful, thought I tried the SGI's implementation named 'job' in past days.

Issue: 2) What items should be collected for accounting information ? BSD-accounting collects PID/UID/GID, User/Sys/Elapsed-Time, and # of minor/major page faults. SGI's CSA collects VM/RSS size on exit time, Integral-VM/RSS, and amount of block-I/O additionally.

I think it's hard to implement the accounting-engine as a kernel loadable module using any kinds of framework. Because, we must put callback functions into all around the kernel for this purpose.

Thus, I make a proposion as follows:
We should separate the process-aggregation functionality and collecting accounting informations.

Something of framework to implement process-aggregation is necessary. And, making a collection of accounting information should be merged into BSD-accounting and implemented as a part of monolithic kernel as Guillaume said.

Thanks.

Quoted 6 lines

I think there are two issues about system accounting framework.

Issue: 1) How to define the appropriate unit for accounting ? Current BSD-accountiong make a collection per process accounting information.
CSA make additionally a collection per process-aggregation accounting.

The 'enhanced acct data collection' patches that were added to 2-6-11-rc* tree still do collection of per process data.

CSA added those per-process data to per-aggregation ("job") data structure at do_exit() time when a process termintes.

Quoted 5 lines

It is appropriate to make the fork-exit event handling framework for definition
of the process-aggregation, such as PAGG.

Quoted 8 lines

This system-accounting per process-aggregation is quite useful, thought I tried the SGI's implementation named 'job' in past days.

Issue: 2) What items should be collected for accounting information ? BSD-accounting collects PID/UID/GID, User/Sys/Elapsed-Time, and # of minor/major page faults. SGI's CSA collects VM/RSS size on exit time, Integral-VM/RSS, and amount of block-I/O additionally.

These data are now collected in 2.6.11-rc* code. Note that these data are still per-process.

Quoted 9 lines

I think it's hard to implement the accounting-engine as a kernel loadable module using any kinds of framework. Because, we must put callback functions
into all around the kernel for this purpose.

Thus, I make a proposion as follows:
We should separate the process-aggregation functionality and collecting accounting informations.

I totally agree with this! Actually that was what we have done. The data collection part of code has been unified.

Quoted 4 lines

Something of framework to implement process-aggregation is necessary. And, making a collection of accounting information should be merged into BSD-accounting and implemented as a part of monolithic kernel as Guillaume said.

This sounds good. I am interested in learning how ELSA saves off the per-process accounting data before the data got disposed. If that scheme works for CSA, we would be very happy to adopt the scheme. The current BSD scheme is very insufficient. The code is very BSD centric and it provides no way to handle process-aggregation.

Thanks,
- jay

Quoted 2 lines

Thanks.

The common agreement for the method of dealing with process aggregation has not been constructed yet, I understood. And, we will not able to integrate each process aggregation model because of its diverseness.

For example, a process which belong to JOB-A must not belong any other 'JOB-X' in CSA-model. But, In ELSA-model, a process in BANK-B can concurrently belong to BANK-B1 which is a child of BANK-B.

And, there are other defferences:
Whether a process not to belong to any process-aggregation is permitted or not ? Whether a process-aggregation should be inherited to child process or not ? (There is possibility not to be inherited in a rule-based process aggregation like CKRM)

Some process-aggregation model have own philosophy and implemantation, so it's hard to integrate. Thus, I think that common 'fork/exec/exit' event handling framework to implement any kinds of process-aggregation.

How ELSA adds per process accounting data
to your grouping (banks) when a process exit? How do you save accounting data you need in task_struct before it is disposed? BSD handles that through acct_process() hook at do_exit(). CSA also depends on a hook at do_exit() to merge per-process data to per-job data. How does ELSA handle this without a need of a do_exit() hook?

Quoted 9 lines · Kaigai Kohei

The common agreement for the method of dealing with process aggregation has not been constructed yet, I understood. And, we will not able to integrate each process aggregation model because of its diverseness.

For example, a process which belong to JOB-A must not belong any other 'JOB-X' in CSA-model. But, In ELSA-model, a process in BANK-B can concurrently belong to BANK-B1 which is a child of BANK-B.

Quoted 8 lines

And, there are other defferences:
Whether a process not to belong to any process-aggregation is permitted or not ? Whether a process-aggregation should be inherited to child process or not ? (There is possibility not to be inherited in a rule-based process aggregation like CKRM)

Some process-aggregation model have own philosophy and implemantation, so it's hard to integrate. Thus, I think that common 'fork/exec/exit' event handling framework to implement any kinds of process-aggregation.

We really want to avoid doing such stuff in-kernel if at all possible, of course.

Is it not possible to implement the fork/exec/exit notifications to userspace so that a daemon can track the process relationships and perform aggregation based upon individual tasks' accounting? That's what one of the accounting systems is proposing doing, I believe.

(In fact, why do we even need the notifications? /bin/ps can work this stuff out).

...

Quoted 7 lines

We really want to avoid doing such stuff in-kernel if at all possible, of course.

Is it not possible to implement the fork/exec/exit notifications to userspace so that a daemon can track the process relationships and perform aggregation based upon individual tasks' accounting? That's what one of the accounting systems is proposing doing, I believe.

It's what I'm proposing. The problem is to be alerted when a new process is created in order to add it in the correct group of processes if the parent belongs to one (or several) groups. The notification can be done with the fork connector patch.

Quoted 2 lines

(In fact, why do we even need the notifications? /bin/ps can work this stuff out).

Yes it can but the risk is to lose some forks no?
I think that /bin/ps is using the /proc interface. If we're polling the /proc to catch process creation we may lost some of them. With the fork connector we catch all forks and we can check that by using the sequence number (incremented by each fork) of the message.

Quoted 4 lines

It's what I'm proposing. The problem is to be alerted when a new process is created in order to add it in the correct group of processes if the parent belongs to one (or several) groups. The notification can be done with the fork connector patch.

Yes, it sounds sane.

The 2.6.8.1 ELSA patch adds quite a bit of kernel code, but from what you're saying it seems like most of that has become redundant, and all you now need is the fork notifier. Is that correct?

I will run benchmarks found at http://bulk.fefe.de/scalability/ to see
how the fork connector impacts on the kernel.

All stuff that was previously done in kernel space and provided by the
2.6.8.1 ELSA patch has been moved in the ELSA user space daemon called "jobd".

We really want to avoid doing such stuff in-kernel if at all possible, of course.

Is it not possible to implement the fork/exec/exit notifications to userspace so that a daemon can track the process relationships and perform aggregation based upon individual tasks' accounting? That's what one of the accounting systems is proposing doing, I believe.

(In fact, why do we even need the notifications? /bin/ps can work this stuff out).

Quoted 3 lines

Some process-aggregation model have own philosophy and implemantation, so it's hard to integrate. Thus, I think that common 'fork/exec/exit' event handling framework to implement any kinds of process-aggregation.

We really want to avoid doing such stuff in-kernel if at all possible, of course.

Is it not possible to implement the fork/exec/exit notifications to userspace so that a daemon can track the process relationships and perform aggregation based upon individual tasks' accounting? That's what one of the accounting systems is proposing doing, I believe.

(In fact, why do we even need the notifications? /bin/ps can work this stuff out).

Quoted 10 lines · Andrew Morton

Quoted 8 lines · Kaigai Kohei

The common agreement for the method of dealing with process aggregation has not been constructed yet, I understood. And, we will not able to integrate each process aggregation model because of its diverseness.

For example, a process which belong to JOB-A must not belong any other 'JOB-X' in CSA-model. But, In ELSA-model, a process in BANK-B can concurrently belong to BANK-B1 which is a child of BANK-B.

Quoted 8 lines

Quoted 8 lines

And, there are other defferences:
Whether a process not to belong to any process-aggregation is permitted or not ? Whether a process-aggregation should be inherited to child process or not ? (There is possibility not to be inherited in a rule-based process aggregation like CKRM)

Some process-aggregation model have own philosophy and implemantation, so it's hard to integrate. Thus, I think that common 'fork/exec/exit' event handling framework to implement any kinds of process-aggregation.

I can add "policies". With ELSA, a process belongs to one or several groups and if a process is removed from one group, its children still belong to the group. Thus a good idea could be to associate a "philosophy" to a group. For exemple, when a group of processes is created it can be tagged as UNIQUE or SHARED. UNIQUE means that a process that belongs to it could not be added in another group by opposition to SHARED. It's not needed inside the kernel.

Quoted 9 lines

We really want to avoid doing such stuff in-kernel if at all possible, of course.

Is it not possible to implement the fork/exec/exit notifications to userspace so that a daemon can track the process relationships and perform aggregation based upon individual tasks' accounting? That's what one of the accounting systems is proposing doing, I believe.

It's what I'm proposing. The problem is to be alerted when a new process is created in order to add it in the correct group of processes if the parent belongs to one (or several) groups. The notification can be done with the fork connector patch.

Quoted 4 lines

(In fact, why do we even need the notifications? /bin/ps can work this stuff out).

Yes it can but the risk is to lose some forks no? I think that /bin/ps is using the /proc interface. If we're polling the /proc to catch process creation we may lost some of them. With the
fork connector we catch all forks and we can check that by using the
sequence number (incremented by each fork) of the message.

Guillaume

Hi, Thanks for your comments.

Quoted 8 lines

Quoted 6 lines

I think there are two issues about system accounting framework.

Issue: 1) How to define the appropriate unit for accounting ? Current BSD-accountiong make a collection per process accounting information.
CSA make additionally a collection per process-aggregation accounting.

Quoted 2 lines

The 'enhanced acct data collection' patches that were added to 2-6-11-rc* tree still do collection of per process data.

Hmm, I have not noticed this extension. But I made sure about it. The following your two patches implements enhanced data collection, didn't it?

- ChangeLog for 2.6.11-rc1
[PATCH] enhanced I/O accounting data patch [PATCH] enhanced Memory accounting data collection

Since making a collection per process accounting is unified to the stock kernel, I want to have a discussion about remaining half, "How to define the appropriate unit for accounting ?"
We can agree that only per process-accounting is so rigid, I think. Then, process-aggregation should be provided in one way or another.

[1] Is it necessary 'fork/exec/exit' event handling framework ?

The common agreement for the method of dealing with process aggregation has not been constructed yet, I understood. And, we will not able to integrate each process aggregation model because of its diverseness.

For example, a process which belong to JOB-A must not belong any other 'JOB-X' in CSA-model. But, In ELSA-model, a process in BANK-B can concurrently belong to BANK-B1 which is a child of BANK-B.

And, there are other defferences:
Whether a process not to belong to any process-aggregation is permitted or not ? Whether a process-aggregation should be inherited to child process or not ? (There is possibility not to be inherited in a rule-based process aggregation like CKRM)

Some process-aggregation model have own philosophy and implemantation, so it's hard to integrate. Thus, I think that common 'fork/exec/exit' event handling framework to implement any kinds of process-aggregation.

[2] What implementation should be adopted ?

I think registerable hooks on fork/execve/exit is necessary, not only exit() hook. Because a rule or policy based process-aggregation model requirees to catch the transition of a process status.
It might be enough to hook the exit() event only in process-accounting, but it's not kind for another customer.

Thus, I recommend SGI's PAGG.

In my understanding, the reason for not to include such a framework is that increase of unidentifiable (proprietary) modules is worried.

But, SI can divert LSM to implemente process-aggregation if they ignore the LSM's original purpose, for example.
# I'm strongly opposed to such a movement as a SELinux's user :-)

So, I think such a fork/execve/exit hooks is harmless now. Is this the time to unify it?

Thanks.

Quoted 12 lines

CSA added those per-process data to per-aggregation ("job") data structure at do_exit() time when a process termintes.

Quoted 9 lines

It is appropriate to make the fork-exit event handling framework for definition
of the process-aggregation, such as PAGG.

This system-accounting per process-aggregation is quite useful, thought I tried the SGI's implementation named 'job' in past days.

Quoted 11 lines

Quoted 4 lines

Issue: 2) What items should be collected for accounting information ? BSD-accounting collects PID/UID/GID, User/Sys/Elapsed-Time, and # of minor/major page faults. SGI's CSA collects VM/RSS size on exit time, Integral-VM/RSS, and amount of block-I/O additionally.

These data are now collected in 2.6.11-rc* code. Note that these data are still per-process.

Quoted 2 lines

I think it's hard to implement the accounting-engine as a kernel

loadable

Quoted 29 lines

Quoted 7 lines

module using any kinds of framework. Because, we must put callback functions
into all around the kernel for this purpose.

Thus, I make a proposion as follows:
We should separate the process-aggregation functionality and collecting accounting informations.

I totally agree with this! Actually that was what we have done. The data collection part of code has been unified.

Quoted 4 lines

Something of framework to implement process-aggregation is necessary. And, making a collection of accounting information should be merged into BSD-accounting and implemented as a part of monolithic kernel as Guillaume said.

This sounds good. I am interested in learning how ELSA saves off the per-process accounting data before the data got disposed. If that scheme works for CSA, we would be very happy to adopt the scheme. The current BSD scheme is very insufficient. The code is very BSD centric and it provides no way to handle process-aggregation.

Thanks,
- jay

Quoted 2 lines

Thanks.

So, I think such a fork/execve/exit hooks is harmless now.

Quoted 1 line

So, I think such a fork/execve/exit hooks is harmless now.

I don't recall seeing any microbenchmarking of the impact on fork/exit of such hooks. You might find such a benchmark in lmbench, or at
http://bulk.fefe.de/scalability/.

I think the microbenchmarking your link provides is irrelevant.

Quoted 4 lines · Guillaume Thouvenin

It's what I'm proposing. The problem is to be alerted when a new process is created in order to add it in the correct group of processes if the parent belongs to one (or several) groups. The notification can be done with the fork connector patch.

I am not quite comfortable of ELSA requesting a fork hook this way. How many hooks in the stock kernel that are related to accounting? Can anyone answer this question? I know of 'acct_process()' in exit.c used by the BSD accounting and ELSA is requesting a hook in fork. If people raise the same question again a few years later, how many people will still remember this ELSA hook?

Quoted 10 lines · Andrew Morton

We really want to avoid doing such stuff in-kernel if at all possible, of course.

Is it not possible to implement the fork/exec/exit notifications to userspace so that a daemon can track the process relationships and perform aggregation based upon individual tasks' accounting? That's what one of the accounting systems is proposing doing, I believe.

(In fact, why do we even need the notifications? /bin/ps can work this stuff out).

I had started a proof of concept implementation that could reconstruct the whole process tree from userspace just from the BSD accounting currently in the kernel (+ the conceptual bug-fix that I misnamed "[RFC] "biological parent" pid"). This could do the whole job ID thing from userspace. Unfortunately, I haven't had time to work on it recently.

Also, doing per-job accounting might actually be more lightweight than per-process accounting, so I'm not at all opposed to unifying CSA and BSD accounting into one mechanism that just writes different file formats.

A complete framework seems like overkill to me, too.

Tim

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Lse-tech mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/lse-tech

Quoted 1 line

[1] Is it necessary 'fork/exec/exit' event handling framework ?

Quoted 4 lines

Some process-aggregation model have own philosophy and implemantation, so it's hard to integrate. Thus, I think that common 'fork/exec/exit' event handling
framework to implement any kinds of process-aggregation.

BSD needs an exit hook and ELSA needs a fork hook. I am still evaluating whether CSA can use the ELSA module. If CSA can use the ELSA module, CSA maybe would be fine with the fork hook.

In my understanding, what Andrew Morton said is "If target functionality can
implement in user space only, then we should not modify the kernel-tree".

Quoted 2 lines · Kaigai Kohei

In my understanding, what Andrew Morton said is "If target functionality can implement in user space only, then we should not modify the kernel-tree".

fork, exec and exit upcalls sound pretty good to me. As long as

a) they use the same common machinery and

b) they are next-to-zero cost if something is listening on the netlink socket but no accounting daemon is running.

Question is: is this sufficient for CSA?

Quoted 9 lines · Andrew Morton

Quoted 3 lines · Kaigai Kohei

In my understanding, what Andrew Morton said is "If target functionality can
implement in user space only, then we should not modify the kernel-tree".

fork, exec and exit upcalls sound pretty good to me. As long as

Quoted 6 lines

a) they use the same common machinery and

b) they are next-to-zero cost if something is listening on the netlink socket but no accounting daemon is running.

Question is: is this sufficient for CSA?

Yes, fork, exec, and exit upcalls are sufficient for CSA.

Quoted 16 lines · * Jay Lan ([email protected]) wrote:

Quoted 14 lines · Andrew Morton

Quoted 2 lines · Kaigai Kohei

In my understanding, what Andrew Morton said is "If target functionality can implement in user space only, then we should not modify the kernel-tree".

fork, exec and exit upcalls sound pretty good to me. As long as

a) they use the same common machinery and

b) they are next-to-zero cost if something is listening on the netlink socket but no accounting daemon is running.

Quoted 3 lines

Quoted 1 line

Question is: is this sufficient for CSA?

Yes, fork, exec, and exit upcalls are sufficient for CSA.

As soon as you want to throttle tasks at the Job level, this would be insufficient. But, IIRC, that's not one of PAGG/Job/CSA's requirements right?

thanks,
-chris

Quoted 13 lines · Andrew Morton

Quoted 2 lines · Kaigai Kohei

In my understanding, what Andrew Morton said is "If target functionality can implement in user space only, then we should not modify the kernel-tree".

fork, exec and exit upcalls sound pretty good to me. As long as

a) they use the same common machinery and

b) they are next-to-zero cost if something is listening on the netlink socket but no accounting daemon is running.

Quoted 1 line

Question is: is this sufficient for CSA?

Yes, fork, exec, and exit upcalls are sufficient for CSA.

The framework i proposed earlier should satisfy your requirement a and b, and provides upcalls needed by BSD, ELSA and CSA. Maybe i misunderstood your concern of the 'very light weight' framework i proposed besides being "overkill"?

Quoted 14 lines · Jay Lan

Quoted 13 lines · Andrew Morton

Quoted 2 lines · Kaigai Kohei

In my understanding, what Andrew Morton said is "If target functionality can implement in user space only, then we should not modify the kernel-tree".

fork, exec and exit upcalls sound pretty good to me. As long as

a) they use the same common machinery and

b) they are next-to-zero cost if something is listening on the netlink socket but no accounting daemon is running.

Quoted 8 lines

Quoted 1 line

Question is: is this sufficient for CSA?

Yes, fork, exec, and exit upcalls are sufficient for CSA.

The framework i proposed earlier should satisfy your requirement a and b, and provides upcalls needed by BSD, ELSA and CSA. Maybe i misunderstood your concern of the 'very light weight' framework i proposed besides being "overkill"?

"upcall" is poorly defined.

What I meant was that ELSA can perform its function when the kernel merely sends asynchronous notifications of forks out to userspace via netlink.

Further, I'm wondering if CSA can perform its function with the same level of kernel support, perhaps with the addition of netlink-based notification of exec and exit as well.

The framework patch which you sent was designed to permit the addition of more kernel accounting code, which is heading in the opposite direction.

In other words: given that ELSA can do its thing via existing accounting interfaces and a fork notifier, why does CSA need to add lots more kernel code?

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Lse-tech mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/lse-tech

Quoted 3 lines · Kaigai Kohei

In my understanding, what Andrew Morton said is "If target functionality can
implement in user space only, then we should not modify the kernel-tree".

fork, exec and exit upcalls sound pretty good to me. As long as

a) they use the same common machinery and

b) they are next-to-zero cost if something is listening on the netlink socket but no accounting daemon is running.

Quoted 9 lines

Quoted 2 lines · Kaigai Kohei

In my understanding, what Andrew Morton said is "If target functionality can implement in user space only, then we should not modify the kernel-tree".

fork, exec and exit upcalls sound pretty good to me. As long as

a) they use the same common machinery and

Quoted 2 lines

b) they are next-to-zero cost if something is listening on the netlink socket but no accounting daemon is running.

b) would involved being able to avoid sending netlink messages in case there are no listeners. AFAIK that isnt possible currently, netlink sends packets unconditionally.

Am I wrong?

Hi,

Quoted 14 lines

Quoted 12 lines

Quoted 3 lines · Kaigai Kohei

In my understanding, what Andrew Morton said is "If target functionality can
implement in user space only, then we should not modify the kernel-tree".

fork, exec and exit upcalls sound pretty good to me. As long as

a) they use the same common machinery and

b) they are next-to-zero cost if something is listening on the netlink socket but no accounting daemon is running.

Quoted 5 lines

b) would involved being able to avoid sending netlink messages in case there are no listeners. AFAIK that isnt possible currently, netlink sends packets unconditionally.

Am I wrong?

In current implementaion, you might be right.
But we should make an effort to achieve the requirement-(b) from now.

And, why can't netlink packets send always?
If there are fork/exec/exit hooks, and they call CSA or other process-grouping modules,
then those modules will decide whether packets for interaction with the daemon should be
sent or not.

In most considerable case, CSA's kernel-loadable-module using such hooks will not be loaded
when no accounting daemon is running. Adversely, this module must be loaded when accounting
daemon needs CSA's netlink packets.
Thus, it is only necessary to refer flag valiable and to execute conditional-jump
when no-accounting daemon is running.

In my estimation, we must pay additional cost for an increment-operation, an decrement-op,
an comparison-op and an conditional jump-op. It's enough lightweight, I think.

For example:
If CSA's module isn't loaded, 'privates_for_grouping' will be empty.

inline int on_fork_hook(task_struct *parent, task_struct *newtask){
rcu_read_lock();

  if( !list_empty(&parent->privates_for_grouping) ){
    ..<Calling to any process grouping module>..;

}
rcu_read_unlock();
}

Yep, the netlink people should be able to help - they known what would be required for not sending messages in case there is no listener registered.

Yep, the netlink people should be able to help - they known what would be required for not sending messages in case there is no listener registered.

Maybe its already possible? I have never used netlink myself.

Quoted 6 lines

And, why can't netlink packets send always?
If there are fork/exec/exit hooks, and they call CSA or other process-grouping modules,
then those modules will decide whether packets for interaction with the daemon should be
sent or not.

The netlink data will be sent to userspace at fork/exec/exit hooks - one wants to avoid that if there are no listeners, so setups which dont want to run the accounting daemon dont pay the cost of building and sending the information through netlink.

Thats what Andrew asked for if I understand correctly.

Quoted 8 lines

In most considerable case, CSA's kernel-loadable-module using such hooks will not be loaded
when no accounting daemon is running. Adversely, this module must be loaded when accounting
daemon needs CSA's netlink packets.
Thus, it is only necessary to refer flag valiable and to execute conditional-jump
when no-accounting daemon is running.

That would be one hack, although it is uglier than the pure netlink selection.

Quoted 8 lines

In my estimation, we must pay additional cost for an increment-operation, an decrement-op,
an comparison-op and an conditional jump-op. It's enough lightweight, I think.

For example:
If CSA's module isn't loaded, 'privates_for_grouping' will be empty.

Quoted 7 lines

inline int on_fork_hook(task_struct *parent, task_struct *newtask){
rcu_read_lock();

 if( !list_empty(&parent->privates_for_grouping) ){
   ..<Calling to any process grouping module>..;

}
rcu_read_unlock();
}

Andrew has been talking about sending data over netlink to implement the accounting at userspace, so this piece of code is out of the game, no?

Quoted 4 lines

Yep, the netlink people should be able to help - they known what would be required for not sending messages in case there is no listener registered.

Maybe its already possible? I have never used netlink myself.

Hello,

Quoted 3 lines · Marcelo Tosatti

Yep, the netlink people should be able to help - they known what would be required for not sending messages in case there is no listener registered.

Quoted 1 line

Maybe its already possible? I have never used netlink myself.

If we notify the fork/exec/exit-events to user-space directly as you said, I don't think some hackings on netlink is necessary.
For example, such packets is sent only when /proc/sys/.../process_grouping is set, and user-side daemon set this value, and unset when daemon will exit. It's not necessary to take too seriously.

Quoted 4 lines · Marcelo Tosatti

Yep, the netlink people should be able to help - they known what would be required for not sending messages in case there is no listener registered.

Maybe its already possible? I have never used netlink myself.

If we notify the fork/exec/exit-events to user-space directly as you said, I don't think some hackings on netlink is necessary.
For example, such packets is sent only when /proc/sys/.../process_grouping is set, and user-side daemon set this value, and unset when daemon will exit. It's not necessary to take too seriously.

Ok the protocol is maybe too "basic" but with this mechanism the user
space application that uses the fork connector can start and stop the send of messages. This implementation needs somme improvements because currently, if two application are using the fork connector one can enable it and the other don't know if it is enable or not, but the idea is here I think.

Quoted 7 lines · Guillaume Thouvenin

Ok the protocol is maybe too "basic" but with this mechanism the user
space application that uses the fork connector can start and stop the send of messages. This implementation needs somme improvements because currently, if two application are using the fork connector one can enable it and the other don't know if it is enable or not, but the idea is here I think.

Yes. But this problem can be solved in userspace, with a little library function and a bit of locking.

IOW: use the library to enable/disable the fork connector rather than directly doing syscalls.

It has the problem that if a client of that library crashes, the counter gets out of whack, but really, it's not all _that_ important, and to handle this properly in-kernel each client would need an open fd against some object so we can do the close-on-exit thing properly. You'd need to create a separate netlink socket for the purpose.

Havent seen the beginnings of this thread. But whatever you are trying to do seems to suggest some complexity that you are trying to workaround. What was wrong with just going ahead and just always invoking your netlink_send()?

If there are nobody in user space (or
kernel) listening, it wont go anywhere.

cheers,
jamal

Quoted 20 lines · Andrew Morton

Quoted 7 lines · Guillaume Thouvenin

Ok the protocol is maybe too "basic" but with this mechanism the user
space application that uses the fork connector can start and stop the send of messages. This implementation needs somme improvements because currently, if two application are using the fork connector one can enable it and the other don't know if it is enable or not, but the idea is here I think.

Yes. But this problem can be solved in userspace, with a little library function and a bit of locking.

IOW: use the library to enable/disable the fork connector rather than directly doing syscalls.

It has the problem that if a client of that library crashes, the counter gets out of whack, but really, it's not all _that_ important, and to handle this properly in-kernel each client would need an open fd against some object so we can do the close-on-exit thing properly. You'd need to create a separate netlink socket for the purpose.

* jamal <[email protected]> 2005-02-28 08:40

Quoted 3 lines

netlink broadcast or a wrapper around it.
Why even bother doing the check with netlink_has_listeners()?

To implement the master enable/disable switch they want. The messages don't get send out anyway but why bother doing all the work if nothing will get send out in the end? It implements a well defined flag controlled by open/close on fds (thus handles dying applications) stating whether the whole code should be enabled or disabled.

It is of course not needed to avoid sending unnecessary messages.

Quoted 7 lines · Guillaume Thouvenin

Ok the protocol is maybe too "basic" but with this mechanism the user
space application that uses the fork connector can start and stop the send of messages. This implementation needs somme improvements because currently, if two application are using the fork connector one can enable it and the other don't know if it is enable or not, but the idea is here I think.

Yes. But this problem can be solved in userspace, with a little library function and a bit of locking.

IOW: use the library to enable/disable the fork connector rather than directly doing syscalls.

It has the problem that if a client of that library crashes, the counter gets out of whack, but really, it's not all _that_ important, and to handle this properly in-kernel each client would need an open fd against some object so we can do the close-on-exit thing properly. You'd need to create a separate netlink socket for the purpose.

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Lse-tech mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/lse-tech

Havent seen the beginnings of this thread. But whatever you are trying to do seems to suggest some complexity that you are trying to workaround. What was wrong with just going ahead and just always invoking your netlink_send()?

If there are nobody in user space (or kernel) listening, it wont go anywhere.

Quoted 4 lines

Havent seen the beginnings of this thread. But whatever you are trying to do seems to suggest some complexity that you are trying to workaround. What was wrong with just going ahead and just always invoking your netlink_send()?

I guess parts of the wheel are broken and need to be reinvented ;->

Quoted 1 line

If there are nobody in user space (or kernel) listening, it wont go anywhere.

Additional you may want to extend netlink a bit to check whether there is a listener before creating the messages. The method to do so depends on whether you use netlink_send() or netlink_brodacast(). The latter is more flexiable because you can add more groups later on and the userspace applications can decicde which ones they want to listen to. Both methods handle dying clients perfectly fine, the association to the netlink socket gets destroyed as soon as the socket is closed. Therefore you can simply check mc_list of the netlink protocol you use to see if there are any listeners registered:

static inline int netlink_has_listeners(struct sock *sk)

{
	int ret;

	read_lock(&nl_table_lock);
	ret = list_empty(&nl_table[sk->sk_protocol].mc_list)
	read_unlock(&nl_table_lock);

	return !ret;
}

This is simplified and ignores the actual group assignments, i.e. you might want to extend it to have it check if there are listeners for a certain group.

netlink broadcast or a wrapper around it.
Why even bother doing the check with netlink_has_listeners()?

* jamal <[email protected]> 2005-02-28 08:40

Quoted 3 lines

netlink broadcast or a wrapper around it.
Why even bother doing the check with netlink_has_listeners()?

To implement the master enable/disable switch they want. The messages don't get send out anyway but why bother doing all the work if nothing will get send out in the end? It implements a well defined flag controlled by open/close on fds (thus handles dying applications) stating whether the whole code should be enabled or disabled. It is of course not needed to avoid sending unnecessary messages.

Quoted 11 lines

* jamal <[email protected]> 2005-02-28 08:40

Quoted 3 lines

netlink broadcast or a wrapper around it.
Why even bother doing the check with netlink_has_listeners()?

To implement the master enable/disable switch they want. The messages don't get send out anyway but why bother doing all the work if nothing will get send out in the end? It implements a well defined flag controlled by open/close on fds (thus handles dying applications) stating whether the whole code should be enabled or disabled. It is of course not needed to avoid sending unnecessary messages.

To justify writting the new code, I am assuming someone has actually sat down and in the minimal stuck their finger in the air
and said "yes, there is definetely wind there".

Which leadsto Marcello's question in other email:
Theres some overhead.
- Message needs to be built with skbs allocated (not the cn_xxx thing that seems to be invoked - I suspect that thing will build the skbs);
- the netlink table needs to be locked
-and searched and only then do you find theres nothing to send to.

The point i was making is if you actually had to post this question, then you must be running into some problems of complexity ;-> which implies to me that the delta overhead maybe worth it compared to introducing the complexity or any new code.
I wasnt involved in the discussion - I just woke up and saw the posting and was bored. So the justification for the optimization has probably been explained and it may be worth doing the check (but probably such check should go into whatever that cn_xxx is).

* jamal <[email protected]> 2005-02-28 09:10

Quoted 3 lines

To justify writting the new code, I am assuming someone has actually sat down and in the minimal stuck their finger in the air
and said "yes, there is definetely wind there".

I did, not for this problem though. The code this idea comes from sends batched events

Quoted 1 line · Thomas Graf

* jamal <[email protected]> 2005-02-28 09:10

[..]

Quoted 6 lines

Quoted 3 lines

To justify writting the new code, I am assuming someone has actually sat down and in the minimal stuck their finger in the air
and said "yes, there is definetely wind there".

I did, not for this problem though. The code this idea comes from sends batched events

I would bet the benefit you are seeing has to do with batching rather than such an optimization flag. Different ballgame.
I relooked at their code snippet, they dont even have skbs built nor even figured out what sock or PID. That work still needs to be done it seems in cn_netlink_send(). So probably all they need to do is move the check in cn_netlink_send() instead. This is assuming they are not scratching their heads with some realted complexities.

I am gonna disapear for a while; hopefully the original posters have gathered some ideas from what we discussed.

cheers,
jamal

Quoted 7 lines · On 28 Feb 2005 10:31:33 -0500 jamal <[email protected]> wrote:

I would bet the benefit you are seeing has to do with batching rather than such an optimization flag. Different ballgame.
I relooked at their code snippet, they dont even have skbs built nor even figured out what sock or PID. That work still needs to be done it seems in cn_netlink_send(). So probably all they need to do is move the check in cn_netlink_send() instead. This is assuming they are not scratching their heads with some realted complexities.

[...]
As connector author, I still doubt it worth copying several lines from netlink_broadcast() before skb allocation in cn_netlink_send(). Of course it is easy and can be done, but I do not see any profit here. Atomic allocation is fast, if it succeds, but there are no groups/socket to send, skb will be freed, if allocation fails, then group check is useless.

I would prefer Guillaume Thouvenin as fork connector author to test his current implementation and show that connector's cost is negligible both with and without userspace listeners.
As far as I remember it is first entry in fork connector's TODO list.

I tested without user space listeners and the cost is negligible. I will test with a user space listeners and see the results. I'm going to run the test this week after improving the mechanism that switch on/off the sending of the message.

Quoted 4 lines

I tested without user space listeners and the cost is negligible. I will test with a user space listeners and see the results. I'm going to run the test this week after improving the mechanism that switch on/off the sending of the message.

I'm also trying to mesure the process-creation/destruction performance on following three environment.
Archtechture: i686 / Distribution: Fedora Core 3
* Kernel Preemption is DISABLE
* SMP kernel but UP-machine / Not Hyper Threading
[1] 2.6.11-rc4-mm1 normal
[2] 2.6.11-rc4-mm1 with PAGG based Process Accounting Module
[3] 2.6.11-rc4-mm1 with fork-connector notification (it's enabled)

When 367th-fork() was called after fork-connector notification, kernel was locked up. (User-Space-Listener has been also run until 366th-fork() notification was received)

Quoted 13 lines · Kaigai Kohei

Quoted 4 lines

I tested without user space listeners and the cost is negligible. I will test with a user space listeners and see the results. I'm going to run the test this week after improving the mechanism that switch on/off the sending of the message.

I'm also trying to mesure the process-creation/destruction performance on following three environment.
Archtechture: i686 / Distribution: Fedora Core 3
* Kernel Preemption is DISABLE
* SMP kernel but UP-machine / Not Hyper Threading
[1] 2.6.11-rc4-mm1 normal
[2] 2.6.11-rc4-mm1 with PAGG based Process Accounting Module
[3] 2.6.11-rc4-mm1 with fork-connector notification (it's enabled)

Quoted 2 lines

When 367th-fork() was called after fork-connector notification, kernel was locked up. (User-Space-Listener has been also run until 366th-fork() notification was received)

I don't see this limit on my computer. I'm currently running the lmbench with a new fork connector patch (one that enable/disable fork connector) on an SMP computer. I will send results and the new patch tomorrow because the test takes a while...

I'm using a small patch provided by Evgeniy and not included in the 2.6.11-rc4-mm1 tree.

Best regards,
Guillaume

connector.c +4 -5

@@ -168,12 +168,11 @@
        group = NETLINK_CB((skb)).groups;
        msg = (struct cn_msg *)NLMSG_DATA(nlh);
 
-       if (msg->len != nlh->nlmsg_len - sizeof(*msg) - sizeof(*nlh)) {
+       if (NLMSG_SPACE(msg->len + sizeof(*msg)) != nlh->nlmsg_len) {
                printk(KERN_ERR "skb does not have enough length: "
-                               "requested msg->len=%u[%u], nlh->nlmsg_len=%u[%u], skb->len=%u[must be %u].\n",
-                               msg->len, NLMSG_SPACE(msg->len),
-                               nlh->nlmsg_len, nlh->nlmsg_len - sizeof(*nlh),
-                               skb->len, msg->len + sizeof(*msg));
+                               "requested msg->len=%u[%u], nlh->nlmsg_len=%u, skb->len=%u.\n",
+                               msg->len, NLMSG_SPACE(msg->len + sizeof(*msg)),
+                               nlh->nlmsg_len, skb->len);
                kfree_skb(skb);
                return -EINVAL;
        }

What was wrong with just going ahead and just always invoking your netlink_send()?

b) they are next-to-zero cost if something is listening on the netlink socket but no accounting daemon is running.

the process-creation/destruction performance on following three environment.

kernel was also locked up after 366th-fork()

Quoted 4 lines

I tested without user space listeners and the cost is negligible. I will test with a user space listeners and see the results. I'm going to run the test this week after improving the mechanism that switch on/off the sending of the message.

I'm also trying to mesure the process-creation/destruction performance on following three environment.
Archtechture: i686 / Distribution: Fedora Core 3
* Kernel Preemption is DISABLE
* SMP kernel but UP-machine / Not Hyper Threading
[1] 2.6.11-rc4-mm1 normal
[2] 2.6.11-rc4-mm1 with PAGG based Process Accounting Module
[3] 2.6.11-rc4-mm1 with fork-connector notification (it's enabled)

When 367th-fork() was called after fork-connector notification, kernel was locked up. (User-Space-Listener has been also run until 366th-fork() notification was received)

So I ran the lmbench with three different kernels with the fork
connector patch I just sent. Results are attached at the end of the mail and there are three different lines which are:

(quoted diff) +0 -0

 	o First line is  a linux-2.6.11-rc4-mm1-cnfork
 	o Second line is a linux-2.6.11-rc4-mm1
 	o Third line is  a linux-2.6.11-rc4-mm1-cnfork with a user space
           application. The user space application listened during 15h 
           and received 6496 messages.
 

Quoted 11 lines · Guillaume Thouvenin

So I ran the lmbench with three different kernels with the fork
connector patch I just sent. Results are attached at the end of the mail and there are three different lines which are:

(quoted diff) +0 -0

 	o First line is  a linux-2.6.11-rc4-mm1-cnfork
 	o Second line is a linux-2.6.11-rc4-mm1
 	o Third line is  a linux-2.6.11-rc4-mm1-cnfork with a user space
           application. The user space application listened during 15h 
           and received 6496 messages.
 

Review

Quoted 10 lines

(quoted diff) +0 -0

 Each test has been ran only once. 
 
...
 ------------------------------------------------------------------------------
 Host                 OS  Mhz null null      open slct sig  sig  fork exec sh  
                              call  I/O stat clos TCP  inst hndl proc proc proc
 --------- ------------- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
 account   Linux 2.6.11- 2765 0.17 0.26 3.57 4.19 16.9 0.51 2.31 162. 629. 2415
 account   Linux 2.6.11- 2765 0.16 0.26 3.56 4.17 17.6 0.50 2.30 163. 628. 2417
 account   Linux 2.6.11- 2765 0.16 0.27 3.67 4.25 17.6 0.51 2.28 176. 664. 2456

This is the interesting bit, yes? 5-10% slowdown on fork is expected, but why was exec slower?

What does "The user space application listened during 15h" mean?

5-10% slowdown on fork is expected, but
why was exec slower?