Skip to contentSkip to conversation

rlimits: Print more information when CPU/RT limits are exceeded

DISCUSSION Dave Jones 2 participants 3 messages May 2, 2017 – May 4, 2017 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Dave Jones

author
#1 lore raw
Quoted 17 lines · Linux Kernel
Web:        https://git.kernel.org/torvalds/c/e7ea7c9806a2681807257ea89085339d33f7fa0b
Commit:     e7ea7c9806a2681807257ea89085339d33f7fa0b
Parent:     4495c08e84729385774601b5146d51d9e5849f81
Refname:    refs/heads/master
Author:     Arun Raghavan <[email protected]>
AuthorDate: Wed Mar 1 20:23:09 2017 +0530
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Mon Mar 13 21:32:15 2017 +0100

rlimits: Print more information when CPU/RT limits are exceeded

When a process is sent a SIGKILL because it exceeded CPU or RT limits, the cause may not be obvious in userspace -- daemonised processes just get killed, and even foreground process just see a 'Killed' message. The lack of any information on why this might be happening in logs can be confusing to users who are not aware of this mechanism.

Quoted 7 lines

Add messages which dump the process name and tid in dmesg when a process exceeds its CPU or RT limits (soft and hard) in order to make it clearer to people debugging such issues.

This needs to be configurable, because this is really obnoxious..

[  121.042170] RT Watchdog Timeout (hard): trinity-c40[7533]
[  125.670948] RT Watchdog Timeout (hard): trinity-c29[1612]
[  200.631968] CPU Watchdog Timeout (soft): trinity-c33[11454]
[  200.644308] CPU Watchdog Timeout (soft): trinity-c33[11454]
[  200.656551] CPU Watchdog Timeout (soft): trinity-c33[11454]
[  213.454504] CPU Watchdog Timeout (soft): trinity-c33[11454]
[  276.787116] CPU Watchdog Timeout (soft): trinity-c22[23943]
[  285.857773] CPU Watchdog Timeout (soft): trinity-c33[24908]
[  287.236710] CPU Watchdog Timeout (soft): trinity-c22[23943]
[  295.186400] CPU Watchdog Timeout (soft): trinity-c33[24908]
[  296.464352] CPU Watchdog Timeout (soft): trinity-c22[23943]
[  305.164011] CPU Watchdog Timeout (soft): trinity-c33[24908]
[  367.123564] CPU Watchdog Timeout (soft): trinity-c8[1472]
[  373.950321] CPU Watchdog Timeout (soft): trinity-c8[1472]
[  381.415054] CPU Watchdog Timeout (soft): trinity-c8[1472]
[  389.621759] CPU Watchdog Timeout (soft): trinity-c8[1472]
[  463.940996] CPU Watchdog Timeout (soft): trinity-c29[7725]
[  463.952215] CPU Watchdog Timeout (soft): trinity-c29[7725]
[  463.963306] CPU Watchdog Timeout (soft): trinity-c29[7725]
[  555.264401] RT Watchdog Timeout (hard): trinity-c58[19175]
[  610.536159] RT Watchdog Timeout (hard): trinity-c2[28363]
[  741.785688] CPU Watchdog Timeout (soft): trinity-c46[5034]
[  741.796600] CPU Watchdog Timeout (soft): trinity-c46[5034]
[  741.807384] CPU Watchdog Timeout (soft): trinity-c46[5034]
[  741.972679] CPU Watchdog Timeout (soft): trinity-c46[5034]
[  743.115630] CPU Watchdog Timeout (soft): trinity-c46[5034]
[  743.128628] CPU Watchdog Timeout (soft): trinity-c12[2803]
[  743.139032] CPU Watchdog Timeout (soft): trinity-c12[2803]
[  743.149276] CPU Watchdog Timeout (soft): trinity-c12[2803]
[  823.866183] CPU Watchdog Timeout (soft): trinity-c21[15684]
[  892.151230] CPU Watchdog Timeout (soft): trinity-c56[22668]
[  892.161239] CPU Watchdog Timeout (soft): trinity-c56[22668]
[  899.818894] CPU Watchdog Timeout (soft): trinity-c4[22072]
[  899.828718] CPU Watchdog Timeout (soft): trinity-c4[22072]
[  899.838439] CPU Watchdog Timeout (soft): trinity-c4[22072]
[  905.253660] CPU Watchdog Timeout (soft): trinity-c56[22668]
[  907.297573] CPU Watchdog Timeout (soft): trinity-c39[24134]
[  907.307170] CPU Watchdog Timeout (soft): trinity-c39[24134]
[  907.519560] CPU Watchdog Timeout (soft): trinity-c4[22072]
[  940.624120] RT Watchdog Timeout (hard): trinity-c34[30478]
[  959.189311] CPU Watchdog Timeout (soft): trinity-c61[31012]
[  968.873887] CPU Watchdog Timeout (soft): trinity-c61[31012]
[  980.305390] CPU Watchdog Timeout (soft): trinity-c61[31012]
[  992.532852] CPU Watchdog Timeout (soft): trinity-c61[31012]
[ 1032.126118] CPU Watchdog Timeout (soft): trinity-c34[5861]
[ 1036.723920] CPU Watchdog Timeout (soft): trinity-c34[5861]
[ 1046.628487] CPU Watchdog Timeout (soft): trinity-c34[5861]
[ 1054.169156] CPU Watchdog Timeout (soft): trinity-c34[5861]
[ 1064.102718] CPU Watchdog Timeout (soft): trinity-c34[5861]
[ 1076.722166] CPU Watchdog Timeout (soft): trinity-c34[5861]
[ 1129.844831] CPU Watchdog Timeout (soft): trinity-c4[14714]
[ 1134.980606] CPU Watchdog Timeout (soft): trinity-c4[14714]
[ 1146.512098] CPU Watchdog Timeout (soft): trinity-c4[14714]
[ 1158.420576] CPU Watchdog Timeout (soft): trinity-c4[14714]
[ 1170.275054] CPU Watchdog Timeout (soft): trinity-c4[14714]

Dave

#2 lore raw
Quoted 22 lines · Dave Jones
Quoted 21 lines · Linux Kernel
Web:        https://git.kernel.org/torvalds/c/e7ea7c9806a2681807257ea89085339d33f7fa0b
Commit:     e7ea7c9806a2681807257ea89085339d33f7fa0b
Parent:     4495c08e84729385774601b5146d51d9e5849f81
Refname:    refs/heads/master
Author:     Arun Raghavan <[email protected]>
AuthorDate: Wed Mar 1 20:23:09 2017 +0530
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Mon Mar 13 21:32:15 2017 +0100

rlimits: Print more information when CPU/RT limits are exceeded

When a process is sent a SIGKILL because it exceeded CPU or RT limits, the cause may not be obvious in userspace -- daemonised processes just get killed, and even foreground process just see a 'Killed' message. The lack of any information on why this might be happening in logs can be confusing to users who are not aware of this mechanism.

Add messages which dump the process name and tid in dmesg when a process exceeds its CPU or RT limits (soft and hard) in order to make it clearer to people debugging such issues.

Quoted 5 lines
Quoted 3 lines

This needs to be configurable, because this is really obnoxious..

Is there an example of hos this is done elsewhere that I can work off?

Thanks,
Arun

author
#3 lore raw
Quoted 23 lines · Arun Raghavan
Quoted 22 lines · Dave Jones
Quoted 21 lines · Linux Kernel
Web:        https://git.kernel.org/torvalds/c/e7ea7c9806a2681807257ea89085339d33f7fa0b
Commit:     e7ea7c9806a2681807257ea89085339d33f7fa0b
Parent:     4495c08e84729385774601b5146d51d9e5849f81
Refname:    refs/heads/master
Author:     Arun Raghavan <[email protected]>
AuthorDate: Wed Mar 1 20:23:09 2017 +0530
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Mon Mar 13 21:32:15 2017 +0100

rlimits: Print more information when CPU/RT limits are exceeded

When a process is sent a SIGKILL because it exceeded CPU or RT limits, the cause may not be obvious in userspace -- daemonised processes just get killed, and even foreground process just see a 'Killed' message. The lack of any information on why this might be happening in logs can be confusing to users who are not aware of this mechanism.

Add messages which dump the process name and tid in dmesg when a process exceeds its CPU or RT limits (soft and hard) in order to make it clearer to people debugging such issues.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

This needs to be configurable, because this is really obnoxious..

Is there an example of hos this is done elsewhere that I can work off?

The obvious one to me that comes to mind is /proc/sys/kernel/print-fatal-signals

Dave