Skip to contentSkip to conversation

ON/OFF control of taskstats accounting data at do_exit

DISCUSSION Jay Lan 5 participants 11 messages Jun 14, 2006 – Jun 15, 2006

lore
Share & tools
lore

Jay Lan

author
#0 lore raw

Hi Balbir and Shailabh,

I propose we add the capability to turn ON/OFF the multicase of taskstats accounting data at do_exit().

This would allow 'chkconfig taskstats' like of control similar to 'chkconfig acct' for BSD accounting. Sometimes sysadmins would wish to turn off sending accounting data to the multicast socket. We have seen many situations that our CSA customers need to turn off CSA for a period of time.

I think this feature is very important to this new interface.

Thanks,
- jay

#1 lore raw
Quoted 7 lines · Jay Lan

Hi Balbir and Shailabh,

I propose we add the capability to turn ON/OFF the multicase of taskstats accounting data at do_exit().

Quoted 6 lines

This would allow 'chkconfig taskstats' like of control similar to 'chkconfig acct' for BSD accounting. Sometimes sysadmins would wish to turn off sending accounting data to the multicast socket. We have seen many situations that our CSA customers need to turn off CSA for a period of time.

What happens to other clients listening to the data ? This sort of configuration option would again be along the lines of a systemwide change prompted by needs of one subsystem but affecting all others - something Andrew had recommended against doing while we were discussing the per-tgid turnoff.

The existing way to solve this is for the listeners to close the socket and reopen again later when they are interested. If there are no listeners at all, data won't be sent by the kernel anyway. Won't that work for you ?

--Shailabh

Quoted 6 lines

I think this feature is very important to this new interface.

Thanks,
- jay

author
#2 lore raw
Quoted 25 lines · Shailabh Nagar
Quoted 7 lines · Jay Lan

Hi Balbir and Shailabh,

I propose we add the capability to turn ON/OFF the multicase of taskstats accounting data at do_exit().

Quoted 6 lines

This would allow 'chkconfig taskstats' like of control similar to 'chkconfig acct' for BSD accounting. Sometimes sysadmins would wish to turn off sending accounting data to the multicast socket. We have seen many situations that our CSA customers need to turn off CSA for a period of time.

What happens to other clients listening to the data ? This sort of configuration option would
again be along the lines of a systemwide change prompted by needs of one subsystem but affecting
all others - something Andrew had recommended against doing while we were discussing the per-tgid
turnoff.

Quoted 5 lines

The existing way to solve this is for the listeners to close the socket and reopen again later when they
are interested. If there are no listeners at all, data won't be sent by the kernel anyway. Won't that work
for you ?

I was talking about turning off system-wise taskstats data preparation and delivery when every task exits. Sometimes customers like to do some benchmarking and need to turn off nonessential features.

Are you saying if there is no listener, data will not be assembled and sent by the kernel? I thought kernel would always send no matter whether there is listener? I apologize for the noise if i made a mistake.

Regards,
- jay

Quoted 3 lines

--Shailabh

#3 lore raw
Quoted 4 lines · Jay Lan

Are you saying if there is no listener, data will not be assembled and sent by the kernel? I thought kernel would always send no matter whether there is listener? I apologize for the noise if i made a mistake.

It will be assembled by taskstats. But afaict, the sending of data by the netlink layer will be halted within netlink_broadcast(), which is a couple of calls down the call path from genlmsg_multicast(), which is used by taskstats uses to send data out.

Lets discuss turning off of taskstats assembly separately.

Jamal, Thomas,

If there are no listeners on a genetlink socket, do you think overhead of kernel paths that continue to attempt sending multicast data on genetlink sockets is an issue ?

--Shailabh

#4 lore raw
In reply to Jay Lan
Quoted 4 lines · Jay Lan

I was talking about turning off system-wise taskstats data preparation and delivery when every task exits. Sometimes customers like to do some benchmarking and need to turn off nonessential features.

Lets go through the implications of turning on/off collection, assembly and delivery of the per-task accounting data.

Collection is defined by different subsystems using taskstats.
For atleast one of these (delay accounting), turning on/off dynamically has been tried and deemed to cause a lot of overhead (due to accumalated nature of data) and also be prone to races. Complexity of code added did not justify the value so on/off was restricted to a boot time decision.

Assembly and delivery of data is done by the taskstats code, calling subsystem-specific functions to fill the commonly used struct taskstats and relying on genetlink to do the delivery.
This can be turned on/off using a dynamic parameter such as /sys/kernel/taskstats_enable which sets some internal variable that is used to do early exits from relevant functions (mainly taskstats_send_stats and taskstats_exit_send)
Doing so will have impact on
a) queries sent to the kernel by monitoring applications
b) task exit data sent by kernel to apps listening on the multicast socket

For consistency, I'm assuming both a) and b) will have to be affected when taskstats is turned off. Also, I'm assuming monitoring applications aren't aware of the turn off.

What happens to case a) ? Apps will need to get some error message as a reply. Some assembly overhead is saved (since such an error reply can be sent right away as soon as a query command is seen) but no substantial saving on the delivery part.

For case b), we can save on assembly and delivery by exiting early from taskstats_exit_send(). But won't we need to send some message (if not periodically, atleast once) to listening apps that they shouldn't expect any exit data ? Semantics of suddenly not seeing any exit data could be misinterpreted ?

Its easy enough to implement...just concerned about the semantics of doing so (as far as userspace apps are concerned) and utility in general settings. Utility in case where only CSA is running (delay accounting and other users turned off) is clear.

Thoughts ?

--Shailabh

#5 lore raw
Quoted 9 lines · Shailabh Nagar
Quoted 3 lines · Jay Lan

I was talking about turning off system-wise taskstats data preparation and delivery when every task exits. Sometimes customers like to do some benchmarking and need to turn off nonessential features.

Quoted 8 lines

Lets go through the implications of turning on/off collection, assembly and delivery of the per-task accounting data.

Collection is defined by different subsystems using taskstats.
For atleast one of these (delay accounting), turning on/off dynamically has been tried and deemed to cause a lot of overhead (due to accumalated nature of data) and also be prone to races. Complexity of code added did not justify the value so on/off was restricted to a boot time decision.

Agreed.

Quoted 10 lines

Assembly and delivery of data is done by the taskstats code, calling subsystem-specific functions to fill the commonly used struct taskstats and relying on genetlink to do the delivery.
This can be turned on/off using a dynamic parameter such as /sys/kernel/taskstats_enable which sets some internal variable that is used to do early exits from relevant functions (mainly taskstats_send_stats and taskstats_exit_send)
Doing so will have impact on
a) queries sent to the kernel by monitoring applications
b) task exit data sent by kernel to apps listening on the multicast socket

Quoted 2 lines

For consistency, I'm assuming both a) and b) will have to be affected when taskstats is turned off. Also, I'm assuming monitoring applications aren't aware of the turn off.

I do not see impacts of both cases above since it would not happen. I expect the event of turning off taskstats feature is coordidated by the system adminstrator, so all users are notified in advance.

For that reason, i think exposing the switch at sysfs is not a good idea. Instead, /etc/init.d/taskstats script would be right for this purpose. At kernel side, we would need to make this possible.

What do you think?

Regards,
- jay

Quoted 24 lines

What happens to case a) ? Apps will need to get some error message as a reply. Some assembly overhead is saved (since such an error reply can be sent right away as soon as a query command is seen) but no substantial saving on the delivery part.

For case b), we can save on assembly and delivery by exiting early from taskstats_exit_send(). But won't we need to send some message (if not periodically, atleast once) to listening apps that they shouldn't expect any exit data ? Semantics of suddenly not seeing any exit data could be misinterpreted ?

Its easy enough to implement...just concerned about the semantics of doing so (as far as userspace apps are concerned) and utility in general settings. Utility in case where only CSA is running (delay accounting and other users turned off) is clear.

Thoughts ?

--Shailabh

#6 lore raw
Quoted 4 lines

For that reason, i think exposing the switch at sysfs is not a good idea. Instead, /etc/init.d/taskstats script would be right for this purpose. At kernel side, we would need to make this possible.

When you say we need to make this possible at the kernel side, do you want the kernel not to accumulate taskstats in the do_exit() path (avoid calling fill_xxxx routines) if there are no listeners on TASKSTATS_LISTEN_GROUP? As Shailabh, mentioned sending out does not happen if there are no listeners.

Quoted 6 lines

What do you think?

Regards,
- jay

Signature · 3 lines
	Balbir Singh,
	Linux Technology Center,
	IBM Software Labs
#7 lore raw
Quoted 11 lines · Balbir Singh
Quoted 4 lines

For that reason, i think exposing the switch at sysfs is not a good idea. Instead, /etc/init.d/taskstats script would be right for this purpose. At kernel side, we would need to make this possible.

When you say we need to make this possible at the kernel side, do you want the kernel not to accumulate taskstats in the do_exit() path (avoid calling fill_xxxx routines) if there are no listeners on TASKSTATS_LISTEN_GROUP? As Shailabh, mentioned sending out does not happen if there are no listeners.

I did not understand that when i made the request because i did not see from taskstats code how you can control that. Shailabh later said that it was done at netlink layer.

I do not know how deep down in netlink layer to exit the proces, but it may turn out that the need of this feature not that importan...

Yet, we still allocate memory, fill taskstats struct, invoke netlink, and free memory. Ideally when taskstats is turned off, we do not need to do any of these. So, if we can fail memory allocation when taskstats is turned off, the taskstats_exit_send() will return right away.

To answer Shailabh's discussion in a separate email, yes, i meant the second case that all taskstats applications will be stopped or suspended. The different between a /etc/init.d/taskstats and a sysfs/procfs control of taskstats is that the system will not be activated at init time. However, i admit customers can always turn off taskstats via sysfs before starts whatever they like to do.

I do not know what is community's preference: a /etc/init.d mechnism or a sysfs mechnism. I would be happy either way. Thanks!

Regards,
jay

Quoted 7 lines
Quoted 7 lines

What do you think?

Regards,
- jay

#8 lore raw
In reply to Jay Lan
Quoted 17 lines · Jay Lan
Quoted 15 lines · Shailabh Nagar

Assembly and delivery of data is done by the taskstats code, calling subsystem-specific functions to fill
the commonly used struct taskstats and relying on genetlink to do the delivery.
This can be turned on/off using a dynamic parameter such as /sys/kernel/taskstats_enable which sets
some internal variable that is used to do early exits from relevant functions (mainly taskstats_send_stats
and taskstats_exit_send)
Doing so will have impact on
a) queries sent to the kernel by monitoring applications
b) task exit data sent by kernel to apps listening on the multicast socket

Quoted 8 lines
Quoted 3 lines

For consistency, I'm assuming both a) and b) will have to be affected when taskstats is turned off.
Also, I'm assuming monitoring applications aren't aware of the turn off.

I do not see impacts of both cases above since it would not happen. I expect the event of turning off taskstats feature is coordidated by the system adminstrator, so all users are notified in advance.

Agreed that all users will be notified. Once that happens, what do you expect these users will do to the applications they are running that
a) query per-task data periodically, or
b) monitor exit data

Either these applications will continue running (querying, listening) and have to understand that they cannot expect more data
or they will be stopped(or suspended) from doing the monitoring/querying.

If its the latter, then we don't really need kernel changes since we only save on a little bit of assembling data overhead.

If its the former, then yes, it will be useful to have the kernel changes.

Quoted 4 lines

For that reason, i think exposing the switch at sysfs is not a good idea. Instead, /etc/init.d/taskstats script would be right for this purpose.

I don't get the difference between these two options as far as kernel is concerned. A systemwide switch that can be set by only by a privileged user is what is needed either way, right ? Whether it is in sys/kernel (through sysfs) or /proc/sys/kernel (through /proc and via sysctl) ? Or is there something else that an /etc/init.d/taskstats script can do to set a kernel parameter ?

Quoted 39 lines

At kernel side, we would need to make this possible.

What do you think?

Regards,
- jay

Quoted 31 lines

What happens to case a) ? Apps will need to get some error message as a reply. Some assembly overhead
is saved (since such an error reply can be sent right away as soon as a query command is seen) but no
substantial saving on the delivery part.

For case b), we can save on assembly and delivery by exiting early from taskstats_exit_send(). But won't
we need to send some message (if not periodically, atleast once) to listening apps that they shouldn't
expect any exit data ? Semantics of suddenly not seeing any exit data could be misinterpreted ?

Its easy enough to implement...just concerned about the semantics of doing so (as far as userspace
apps are concerned) and utility in general settings. Utility in case where only CSA is running (delay
accounting and other users turned off) is clear.

Thoughts ?

--Shailabh

#9 lore raw
In reply to Shailabh Nagar
Quoted 36 lines · Shailabh Nagar
Quoted 3 lines · Jay Lan

I was talking about turning off system-wise taskstats data preparation and delivery when every task exits. Sometimes customers like to do some benchmarking and need to turn off nonessential features.

Lets go through the implications of turning on/off collection, assembly and delivery of the per-task accounting data.

Collection is defined by different subsystems using taskstats.
For atleast one of these (delay accounting), turning on/off dynamically has been tried and deemed to cause a lot of overhead (due to accumalated nature of data) and also be prone to races. Complexity of code added did not justify the value so on/off was restricted to a boot time decision.

Assembly and delivery of data is done by the taskstats code, calling subsystem-specific functions to fill the commonly used struct taskstats and relying on genetlink to do the delivery.
This can be turned on/off using a dynamic parameter such as /sys/kernel/taskstats_enable which sets some internal variable that is used to do early exits from relevant functions (mainly taskstats_send_stats and taskstats_exit_send)
Doing so will have impact on
a) queries sent to the kernel by monitoring applications
b) task exit data sent by kernel to apps listening on the multicast socket

For consistency, I'm assuming both a) and b) will have to be affected when taskstats is turned off. Also, I'm assuming monitoring applications aren't aware of the turn off.

What happens to case a) ? Apps will need to get some error message as a reply. Some assembly overhead is saved (since such an error reply can be sent right away as soon as a query command is seen) but no substantial saving on the delivery part.

For case b), we can save on assembly and delivery by exiting early from taskstats_exit_send(). But won't we need to send some message (if not periodically, atleast once) to listening apps that they shouldn't expect any exit data ? Semantics of suddenly not seeing any exit data could be misinterpreted ?

Quoted 5 lines

Its easy enough to implement...just concerned about the semantics of doing so (as far as userspace apps are concerned) and utility in general settings. Utility in case where only CSA is running (delay accounting and other users turned off) is clear.

Thoughts ?

I would have thought that it was obvious that turning on/off the collection of statistics concerning the ACCRUED time that tasks spend in various states (which is what we're talking about) would cause the data to be horribly corrupted.

I also suspect that the overhead of the tests associated with checking whether the mechanism is on or off is probably almost as big as the overhead of gathering the statistics anyway. Assuming that the clock time is already available (true for most places in the CPU scheduler where you'd want to gather stats, for one) the cost of gathering the stats is a subtraction, an addition and (possibly) an assignment to a time stamp (I say possibly here as you may get the time stamp updated for free as well).

Peter

Signature · 4 lines
Peter Williams                                   [email protected]

"Learning, n. The kind of ignorance distinguishing the studious."
 -- Ambrose Bierce
#10 lore raw
In reply to Jay Lan
Quoted 10 lines · Jay Lan

Hi Balbir and Shailabh,

I propose we add the capability to turn ON/OFF the multicase of taskstats accounting data at do_exit().

This would allow 'chkconfig taskstats' like of control similar to 'chkconfig acct' for BSD accounting. Sometimes sysadmins would wish to turn off sending accounting data to the multicast socket. We have seen many situations that our CSA customers need to turn off CSA for a period of time.

If we need to turn the entire sending of stats off, then closing the socket listening on the multicast group should turn off sending of accounting information. Am I missing something?

Quoted 6 lines

I think this feature is very important to this new interface.

Thanks,
- jay

Signature · 4 lines
	Regards,
	Balbir Singh,
	Linux Technology Center,
	IBM Software Labs