Skip to contentSkip to conversation

arm64: defconfig: Disable fine-grained task level IRQ time accounting

RFC PATCH In review Alison Wang 10 participants 31 messages Jul 29, 2020 – Aug 6, 2020

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 arm64: defconfig: Disable fine-grained task level IRQ time accounting

In the current arm64 defconfig, CONFIG_IRQ_TIME_ACCOUNTING is enabled as default. According to my tests on NXP's LayerScape and i.MX platforms, the system hangs when running the command "stress-ng --hrtimers 1" with CONFIG_IRQ_TIME_ACCOUNTING enabled. Disabling this option, the issue disappears. CONFIG_IRQ_TIME_ACCOUNTING causes serious performance impact when running hrtimer stress test at the same time.

---
 arch/arm64/configs/defconfig | 1 -
 1 file changed, 1 deletion(-)
arch/arm64/configs/defconfig +0 -1
index e0f33826819f..ff1c11d8b10b 100644
@@ -4,7 +4,6 @@ CONFIG_AUDIT=y
 CONFIG_NO_HZ_IDLE=y
 CONFIG_HIGH_RES_TIMERS=y
 CONFIG_PREEMPT=y
-CONFIG_IRQ_TIME_ACCOUNTING=y
 CONFIG_BSD_PROCESS_ACCT=y
 CONFIG_BSD_PROCESS_ACCT_V3=y
 CONFIG_TASK_XACCT=y
Signature · 1 line
2.17.1
#1 lore raw

Hi Alison,

Quoted 6 lines · Alison Wang

In the current arm64 defconfig, CONFIG_IRQ_TIME_ACCOUNTING is enabled as default. According to my tests on NXP's LayerScape and i.MX platforms, the system hangs when running the command "stress-ng --hrtimers 1" with CONFIG_IRQ_TIME_ACCOUNTING enabled. Disabling this option, the issue disappears. CONFIG_IRQ_TIME_ACCOUNTING causes serious performance impact when running hrtimer stress test at the same time.

I think instead of disabling the option for all arm64 devices, it might be better to analyze the root-cause why the hrtimer test hangs when this option is enabled.

+Cc hrtimer maintainers: Thomas and Anna-Maria

Thanks,
Kurt

#2 lore raw

Hi, Kurt,

Quoted 7 lines · Alison Wang
Quoted 6 lines

In the current arm64 defconfig, CONFIG_IRQ_TIME_ACCOUNTING is enabled as default. According to my tests on NXP's LayerScape and i.MX platforms, the system hangs when running the command "stress-ng --hrtimers 1" with CONFIG_IRQ_TIME_ACCOUNTING enabled. Disabling this option, the issue disappears. CONFIG_IRQ_TIME_ACCOUNTING causes serious performance impact when running hrtimer stress test at the same

Quoted 8 lines

time.

I think instead of disabling the option for all arm64 devices, it might be better to analyze the root-cause why the hrtimer test hangs when this option is enabled.

+Cc hrtimer maintainers: Thomas and Anna-Maria

[Alison] Yes, I agree. I hope others can give me some clues for the root cause.

Best Regards,
Alison Wang

#3 lore raw
In reply to Kurt Kanzenbach
Quoted 10 lines · Kurt Kanzenbach

Hi Alison,

Quoted 6 lines · Alison Wang

In the current arm64 defconfig, CONFIG_IRQ_TIME_ACCOUNTING is enabled as default. According to my tests on NXP's LayerScape and i.MX platforms, the system hangs when running the command "stress-ng --hrtimers 1" with CONFIG_IRQ_TIME_ACCOUNTING enabled. Disabling this option, the issue disappears. CONFIG_IRQ_TIME_ACCOUNTING causes serious performance impact when running hrtimer stress test at the same time.

Quoted 8 lines

I think instead of disabling the option for all arm64 devices, it might be better to analyze the root-cause why the hrtimer test hangs when this option is enabled.

+Cc hrtimer maintainers: Thomas and Anna-Maria

Thanks,
Kurt

For more context, here is my original report of the issue:
https://lkml.org/lkml/2020/6/4/1062

Just like you, I could not reproduce the RCU stalls and system hang on a 5.6-rt kernel, just on mainline and derivatives, using the plain defconfig.

The issue is not specific to Layerscape or i.MX8, but rather I was able to see the same behavior on Marvell Armada 37xx as well as Qualcomm MSM8976.

So, while of course I agree that disabling IRQ time accounting for arm64 isn't a real solution, it isn't by far an exaggerated proposal either. Nonetheless, the patch is just a RFC and should be treated as such. We are at a loss when it comes to debugging this any further and we would appreciate some pointers.

Thanks,
-Vladimir

#4 lore raw

Hi Vladimir,

Quoted 11 lines · Vladimir Oltean

For more context, here is my original report of the issue:
https://lkml.org/lkml/2020/6/4/1062

Just like you, I could not reproduce the RCU stalls and system hang on a 5.6-rt kernel, just on mainline and derivatives, using the plain defconfig.

The issue is not specific to Layerscape or i.MX8, but rather I was able to see the same behavior on Marvell Armada 37xx as well as Qualcomm MSM8976.

Quoted 5 lines

So, while of course I agree that disabling IRQ time accounting for arm64 isn't a real solution, it isn't by far an exaggerated proposal either. Nonetheless, the patch is just a RFC and should be treated as such. We are at a loss when it comes to debugging this any further and we would appreciate some pointers.

Yeah, sure. I'll try to reproduce this issue first. So it triggers with:

* arm64
* mainline, not -rt kernel
* opened serial console
* irq accounting enabled

Anything else?

Thanks,
Kurt

#5 lore raw
Quoted 27 lines · Kurt Kanzenbach

Hi Vladimir,

Quoted 11 lines · Vladimir Oltean

For more context, here is my original report of the issue:
https://lkml.org/lkml/2020/6/4/1062

Just like you, I could not reproduce the RCU stalls and system hang on a 5.6-rt kernel, just on mainline and derivatives, using the plain defconfig.

The issue is not specific to Layerscape or i.MX8, but rather I was able to see the same behavior on Marvell Armada 37xx as well as Qualcomm MSM8976.

Quoted 5 lines

So, while of course I agree that disabling IRQ time accounting for arm64 isn't a real solution, it isn't by far an exaggerated proposal either. Nonetheless, the patch is just a RFC and should be treated as such. We are at a loss when it comes to debugging this any further and we would appreciate some pointers.

Yeah, sure. I'll try to reproduce this issue first. So it triggers with:

* arm64
* mainline, not -rt kernel
* opened serial console
* irq accounting enabled

Quoted 4 lines

Anything else?

Thanks,
Kurt

Thanks for giving a helping hand, Kurt. The defconfig should be enough. In the interest of full disclosure, the only arm64 device on which we didn't reproduce this was the 16-core LX2160A. But we did reproduce on that with maxcpus=1 though. And also on msm8976 with all 8 cores booted. Just mentioning this in case you're testing on a 16-core system, you might want to reduce the number a bit.

Thanks,
-Vladimir

#6 lore raw
Quoted 31 lines · Vladimir Oltean
Quoted 25 lines · Kurt Kanzenbach
Quoted 11 lines · Vladimir Oltean

For more context, here is my original report of the issue:
https://lkml.org/lkml/2020/6/4/1062

Just like you, I could not reproduce the RCU stalls and system hang on a 5.6-rt kernel, just on mainline and derivatives, using the plain defconfig.

The issue is not specific to Layerscape or i.MX8, but rather I was able to see the same behavior on Marvell Armada 37xx as well as Qualcomm MSM8976.

Quoted 5 lines

So, while of course I agree that disabling IRQ time accounting for arm64 isn't a real solution, it isn't by far an exaggerated proposal either. Nonetheless, the patch is just a RFC and should be treated as such. We are at a loss when it comes to debugging this any further and we would appreciate some pointers.

Yeah, sure. I'll try to reproduce this issue first. So it triggers with:

* arm64
* mainline, not -rt kernel
* opened serial console
* irq accounting enabled

Quoted 4 lines

Anything else?

Thanks,
Kurt

Quoted 6 lines

Thanks for giving a helping hand, Kurt. The defconfig should be enough. In the interest of full disclosure, the only arm64 device on which we didn't reproduce this was the 16-core LX2160A. But we did reproduce on that with maxcpus=1 though. And also on msm8976 with all 8 cores booted. Just mentioning this in case you're testing on a 16-core system, you might want to reduce the number a bit.

OK. I've reproduced it on a Marvell Armada SoC with v5.6 mainline. See splats below. Running with irq time accounting enabled, kills the machine immediately. However, I'm not getting the possible deadlock warnings in 8250 as you did. So that might be unrelated.

Unfortunately I have no idea what to debug here.

Thanks,
Kurt

Splats:

|root@marvell ~ # [  150.996487] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
|[  150.999764]  (detected by 0, t=2602 jiffies, g=9697, q=24)
|[  151.005403] rcu: All QSes seen, last rcu_sched kthread activity 2602 (4294952389-4294949787), jiffies_till_next_fqs=1, root ->qsmask 0x0
|[  151.018034] stress-ng-hrtim R  running task        0  2251   2248 0x00000001
|[  151.025290] Call trace:
|[  151.027808]  dump_backtrace+0x0/0x198
|[  151.031563]  show_stack+0x1c/0x28
|[  151.034971]  sched_show_task+0x224/0x248
|[  151.039002]  rcu_sched_clock_irq+0x7c4/0x840
|[  151.043392]  update_process_times+0x34/0x70
|[  151.047694]  tick_sched_handle.isra.0+0x34/0x70
|[  151.052350]  tick_sched_timer+0x50/0xa0
|[  151.056295]  __hrtimer_run_queues+0x18c/0x5b0
|[  151.060773]  hrtimer_interrupt+0xec/0x248
|[  151.064896]  arch_timer_handler_phys+0x38/0x48
|[  151.069465]  handle_percpu_devid_irq+0xd0/0x3c8
|[  151.074124]  generic_handle_irq+0x2c/0x40
|[  151.078246]  __handle_domain_irq+0x68/0xc0
|[  151.082456]  gic_handle_irq+0x64/0x150
|[  151.086308]  el1_irq+0xbc/0x140
|[  151.089536]  lock_acquire+0xdc/0x250
|[  151.093210]  __might_fault+0x68/0x88
|[  151.096880]  setup_rt_frame+0x2a4/0xee8
|[  151.100823]  do_notify_resume+0x3b0/0x488
|[  151.104945]  work_pending+0x8/0x14
|[  151.108443] rcu: rcu_sched kthread starved for 2613 jiffies! g9697 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
|[  151.118741] rcu: RCU grace-period kthread stack dump:
|[  151.123939] rcu_sched       R  running task        0    10      2 0x00000028
|[  151.131196] Call trace:
|[  151.133707]  __switch_to+0x104/0x170
|[  151.137384]  __schedule+0x36c/0x860
|[  151.140966]  schedule+0x7c/0x108
|[  151.144280]  schedule_timeout+0x204/0x4c0
|[  151.148403]  rcu_gp_kthread+0x5b0/0x1798
|[  151.152435]  kthread+0x110/0x140
|[  151.155749]  ret_from_fork+0x10/0x18
|[  181.665482] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 55s!
|[  181.671022] BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 56s!
|[  181.679198] BUG: workqueue lockup - pool cpus=0-1 flags=0x4 nice=0 stuck for 56s!
|[  181.686902] Showing busy workqueues and worker pools:
|[  181.692059] workqueue events: flags=0x0
|[  181.696226]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3
|[  181.703039]     pending: dbs_work_handler, mv88e6xxx_ptp_overflow_check
|[  181.709826]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3
|[  181.716783]     pending: vmstat_shepherd, switchdev_deferred_process_work
|[  181.723804] workqueue events_unbound: flags=0x2
|[  181.728526]   pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/512 refcnt=3
|[  181.735029]     pending: flush_to_ldisc
|[  181.738947] workqueue events_freezable: flags=0x4
|[  181.743755]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
|[  181.750741]     pending: mmc_rescan
|[  181.754359] workqueue events_power_efficient: flags=0x82
|[  181.759825]   pwq 4: cpus=0-1 flags=0x4 nice=0 active=5/256 refcnt=7
|[  181.766329]     pending: gc_worker, neigh_periodic_work, neigh_periodic_work, do_cache_clean, check_lifetime
|[  181.776488] workqueue mm_percpu_wq: flags=0x8
|[  181.780942]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
|[  181.787924]     pending: vmstat_update
|[  181.791808] workqueue pm: flags=0x4
|[  181.795368]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
|[  181.802349]     pending: pm_runtime_work
|[  181.806412] workqueue writeback: flags=0x4a
|[  181.810686]   pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/256 refcnt=3
|[  181.817224]     pending: wb_workfn
|[  181.820756] workqueue ipv6_addrconf: flags=0x40008
|[  181.825684]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=2
|[  181.832456]     pending: addrconf_verify_work
|[  181.836968] workqueue nfsd4: flags=0x2
|[  181.840794]   pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/256 refcnt=3
|[  181.847329]     pending: laundromat_main
|[  229.043792] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
|[  229.047053]  (detected by 0, t=10407 jiffies, g=9697, q=29)
|[  229.052785] rcu: All QSes seen, last rcu_sched kthread activity 10407 (4294960194-4294949787), jiffies_till_next_fqs=1, root ->qsmask 0x0
|[  229.065505] stress-ng-hrtim R  running task        0  2251   2248 0x00000001
|[  229.072763] Call trace:
|[  229.075276]  dump_backtrace+0x0/0x198
|[  229.079037]  show_stack+0x1c/0x28
|[  229.082443]  sched_show_task+0x224/0x248
|[  229.086476]  rcu_sched_clock_irq+0x7c4/0x840
|[  229.090866]  update_process_times+0x34/0x70
|[  229.095165]  tick_sched_handle.isra.0+0x34/0x70
|[  229.099824]  tick_sched_timer+0x50/0xa0
|[  229.103767]  __hrtimer_run_queues+0x18c/0x5b0
|[  229.108246]  hrtimer_interrupt+0xec/0x248
|[  229.112368]  arch_timer_handler_phys+0x38/0x48
|[  229.116938]  handle_percpu_devid_irq+0xd0/0x3c8
|[  229.121597]  generic_handle_irq+0x2c/0x40
|[  229.125718]  __handle_domain_irq+0x68/0xc0
|[  229.129930]  gic_handle_irq+0x64/0x150
|[  229.133781]  el1_irq+0xbc/0x140
|[  229.137008]  lock_acquire+0xdc/0x250
|[  229.140682]  __might_fault+0x68/0x88
|[  229.144354]  preserve_fpsimd_context+0x160/0x2f8
|[  229.149101]  setup_rt_frame+0x48c/0xee8
|[  229.153044]  do_notify_resume+0x3b0/0x488
|[  229.157167]  work_pending+0x8/0x14
|[  229.160666] rcu: rcu_sched kthread starved for 10418 jiffies! g9697 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
|[  229.171053] rcu: RCU grace-period kthread stack dump:
|[  229.176250] rcu_sched       R  running task        0    10      2 0x00000028
|[  229.183506] Call trace:
|[  229.186019]  __switch_to+0x104/0x170
|[  229.189697]  __schedule+0x36c/0x860
|[  229.193277]  schedule+0x7c/0x108
|[  229.196592]  schedule_timeout+0x204/0x4c0
|[  229.200714]  rcu_gp_kthread+0x5b0/0x1798
|[  229.204747]  kthread+0x110/0x140
|[  229.208061]  ret_from_fork+0x10/0x18
#7 lore raw
Quoted 39 lines · Kurt Kanzenbach
Quoted 31 lines · Vladimir Oltean
Quoted 25 lines · Kurt Kanzenbach
Quoted 11 lines · Vladimir Oltean

For more context, here is my original report of the issue:
https://lkml.org/lkml/2020/6/4/1062

Just like you, I could not reproduce the RCU stalls and system hang on a 5.6-rt kernel, just on mainline and derivatives, using the plain defconfig.

The issue is not specific to Layerscape or i.MX8, but rather I was able to see the same behavior on Marvell Armada 37xx as well as Qualcomm MSM8976.

Quoted 5 lines

So, while of course I agree that disabling IRQ time accounting for arm64 isn't a real solution, it isn't by far an exaggerated proposal either. Nonetheless, the patch is just a RFC and should be treated as such. We are at a loss when it comes to debugging this any further and we would appreciate some pointers.

Yeah, sure. I'll try to reproduce this issue first. So it triggers with:

* arm64
* mainline, not -rt kernel
* opened serial console
* irq accounting enabled

Quoted 4 lines

Anything else?

Thanks,
Kurt

Quoted 6 lines

Thanks for giving a helping hand, Kurt. The defconfig should be enough. In the interest of full disclosure, the only arm64 device on which we didn't reproduce this was the 16-core LX2160A. But we did reproduce on that with maxcpus=1 though. And also on msm8976 with all 8 cores booted. Just mentioning this in case you're testing on a 16-core system, you might want to reduce the number a bit.

Quoted 5 lines

OK. I've reproduced it on a Marvell Armada SoC with v5.6 mainline. See splats below. Running with irq time accounting enabled, kills the machine immediately. However, I'm not getting the possible deadlock warnings in 8250 as you did. So that might be unrelated.

Yes, the console lockdep warnings are unrelated. They are discussed here:
https://lore.kernel.org/lkml/CAHQZ30BnfX+gxjPm1DUd5psOTqbyDh4EJE=2=VAMW_VDafctkA@mail.gmail.com/

Quoted 4 lines

Unfortunately I have no idea what to debug here.

Thanks,
Kurt

So, this means we could submit a formal version of this patch? :)

Thanks,
-Vladimir

#8 lore raw
Quoted 50 lines · Vladimir Oltean
Quoted 39 lines · Kurt Kanzenbach
Quoted 31 lines · Vladimir Oltean
Quoted 25 lines · Kurt Kanzenbach
Quoted 11 lines · Vladimir Oltean

For more context, here is my original report of the issue:
https://lkml.org/lkml/2020/6/4/1062

Just like you, I could not reproduce the RCU stalls and system hang on a 5.6-rt kernel, just on mainline and derivatives, using the plain defconfig.

The issue is not specific to Layerscape or i.MX8, but rather I was able to see the same behavior on Marvell Armada 37xx as well as Qualcomm MSM8976.

Quoted 5 lines

So, while of course I agree that disabling IRQ time accounting for arm64 isn't a real solution, it isn't by far an exaggerated proposal either. Nonetheless, the patch is just a RFC and should be treated as such. We are at a loss when it comes to debugging this any further and we would appreciate some pointers.

Yeah, sure. I'll try to reproduce this issue first. So it triggers with:

* arm64
* mainline, not -rt kernel
* opened serial console
* irq accounting enabled

Quoted 4 lines

Anything else?

Thanks,
Kurt

Quoted 6 lines

Thanks for giving a helping hand, Kurt. The defconfig should be enough. In the interest of full disclosure, the only arm64 device on which we didn't reproduce this was the 16-core LX2160A. But we did reproduce on that with maxcpus=1 though. And also on msm8976 with all 8 cores booted. Just mentioning this in case you're testing on a 16-core system, you might want to reduce the number a bit.

Quoted 5 lines

OK. I've reproduced it on a Marvell Armada SoC with v5.6 mainline. See splats below. Running with irq time accounting enabled, kills the machine immediately. However, I'm not getting the possible deadlock warnings in 8250 as you did. So that might be unrelated.

Yes, the console lockdep warnings are unrelated. They are discussed here:
https://lore.kernel.org/lkml/CAHQZ30BnfX+gxjPm1DUd5psOTqbyDh4EJE=2=VAMW_VDafctkA@mail.gmail.com/

Quoted 6 lines
Quoted 4 lines

Unfortunately I have no idea what to debug here.

Thanks,
Kurt

So, this means we could submit a formal version of this patch? :)

Defconfig is absolutely not the appropriate way to work around bugs - it's merely a starting point for users and distros to set up their own kernel, and if they can still enable this option and render their system unusable then patching some other config that they aren't using is pointless. To usefully mitigate a problem you'd need to make sure the offending option cannot be selected at all (i.e. prohibit HAVE_IRQ_TASK_ACCOUNTING as well).

Having glanced across another thread that mentions IRQ accounting recently[1], I wonder if the underlying bug here might have something do to with the stuff that Marc's trying to clean up.

Robin.

[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/

#12 lore raw
In reply to Vladimir Oltean
Quoted 4 lines · Vladimir Oltean
Quoted 1 line · Kurt Kanzenbach

Unfortunately I have no idea what to debug here.

So, this means we could submit a formal version of this patch? :)

To paper over the underlying problem. Oh well...

#13 lore raw
In reply to Kurt Kanzenbach

Kurt,

Quoted 7 lines · Kurt Kanzenbach

On Thu Jul 30 2020, Vladimir Oltean wrote:
OK. I've reproduced it on a Marvell Armada SoC with v5.6 mainline. See splats below. Running with irq time accounting enabled, kills the machine immediately. However, I'm not getting the possible deadlock warnings in 8250 as you did. So that might be unrelated.

Unfortunately I have no idea what to debug here.

lets look at the facts we have so far.

1) When irq accounting is disabled, RT throttling kicks in as expected.

2) With irq accounting the RT throttler does not kick in and the RCU stall/lockups happen.

Not much, but there is clearly interaction between irq time accounting and scheduler accounting.

Can you please reduce /proc/sys/kernel/sched_rt_runtime_us in steps of 50000? At least in theory, reduction should bring the throttling back.

Thanks,

tglx

#9 lore raw
In reply to Robin Murphy
Quoted 8 lines · Robin Murphy

Having glanced across another thread that mentions IRQ accounting recently[1], I wonder if the underlying bug here might have something do to with the stuff that Marc's trying to clean up.

Robin.

[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/

Thanks Robin. I've applied Marc's "[PATCH v2 00/17] arm/arm64: Turning IPIs into normal interrupts" series and the LS1028A I'm debugging hangs in absolutely the same way.

-Vladimir

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#14 lore raw
In reply to Thomas Gleixner

Hi Thomas,

Quoted 22 lines · Thomas Gleixner

Kurt,

Quoted 7 lines · Kurt Kanzenbach

On Thu Jul 30 2020, Vladimir Oltean wrote:
OK. I've reproduced it on a Marvell Armada SoC with v5.6 mainline. See splats below. Running with irq time accounting enabled, kills the machine immediately. However, I'm not getting the possible deadlock warnings in 8250 as you did. So that might be unrelated.

Unfortunately I have no idea what to debug here.

lets look at the facts we have so far.

1) When irq accounting is disabled, RT throttling kicks in as expected.

2) With irq accounting the RT throttler does not kick in and the RCU stall/lockups happen.

Not much, but there is clearly interaction between irq time accounting and scheduler accounting.

Quoted 7 lines

Can you please reduce /proc/sys/kernel/sched_rt_runtime_us in steps of 50000? At least in theory, reduction should bring the throttling back.

Thanks,

tglx

We already know from my original report that reducing sched_rt_runtime_us to 80% (800000) gives us a stable system.
https://lkml.org/lkml/2020/6/4/1062

What is this telling us?

Thanks,
-Vladimir

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#10 lore raw
In reply to Vladimir Oltean
Quoted 6 lines · Vladimir Oltean
Quoted 5 lines · Robin Murphy

Having glanced across another thread that mentions IRQ accounting recently[1], I wonder if the underlying bug here might have something do to with the stuff that Marc's trying to clean up.

Quoted 8 lines
Quoted 3 lines

Robin.

[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/

Thanks Robin. I've applied Marc's "[PATCH v2 00/17] arm/arm64: Turning IPIs into normal interrupts" series and the LS1028A I'm debugging hangs in absolutely the same way.

I'm not too surprised by that, wrt accounting this mostly changes where the stores go to and barely shuffles when they happen (slightly earlier on the IPI handling path).

FWIW I've had 'stress-ng --hrtimers 1' running on my Juno and eMAG for ~15 minutes and haven't had a splat yet.

Quoted 1 line

-Vladimir

#11 lore raw
Quoted 11 lines · Valentin Schneider
Quoted 10 lines · Vladimir Oltean
Quoted 8 lines · Robin Murphy

Having glanced across another thread that mentions IRQ accounting recently[1], I wonder if the underlying bug here might have something do to with the stuff that Marc's trying to clean up.

Robin.

[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/

Quoted 8 lines
Quoted 4 lines

Thanks Robin. I've applied Marc's "[PATCH v2 00/17] arm/arm64: Turning IPIs into normal interrupts" series and the LS1028A I'm debugging hangs in absolutely the same way.

I'm not too surprised by that, wrt accounting this mostly changes where the stores go to and barely shuffles when they happen (slightly earlier on the IPI handling path).

Indeed. This series is just a "let's make things be the way they should be", and isn't really fixing any bug. It actually may introduce a couple...

Quoted 2 lines

FWIW I've had 'stress-ng --hrtimers 1' running on my Juno and eMAG for ~15 minutes and haven't had a splat yet.

I've started a couple of VMs with that workload too. 400K irq/s on an 8 vcpu guest, nothing to report so far.

But removing IRQ_TIME_ACCOUNTING from defconfig is a resounding "no way!". It looks like we have a bug, and it should be squashed, not glanced over.

        M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#15 lore raw
In reply to Vladimir Oltean
Quoted 9 lines · Vladimir Oltean
Quoted 9 lines

1) When irq accounting is disabled, RT throttling kicks in as expected.

2) With irq accounting the RT throttler does not kick in and the RCU stall/lockups happen.

Not much, but there is clearly interaction between irq time accounting and scheduler accounting.

Quoted 6 lines
Quoted 2 lines

Can you please reduce /proc/sys/kernel/sched_rt_runtime_us in steps of 50000? At least in theory, reduction should bring the throttling back.

We already know from my original report that reducing sched_rt_runtime_us to 80% (800000) gives us a stable system.
https://lkml.org/lkml/2020/6/4/1062

Oh. Somehow missed that. Btw, can you please use links in the form of:

https://lore.kernel.org/r/$MSGID

lkml.org is spotty and having the message id allows to search in the mail client as well.

Quoted 1 line

What is this telling us?

It seems that the fine grained irq time accounting affects the runtime accounting in some way which I haven't figured out yet.

Thanks,

tglx

#16 lore raw
Quoted 11 lines · Thomas Gleixner
Quoted 6 lines · Vladimir Oltean
Quoted 5 lines

1) When irq accounting is disabled, RT throttling kicks in as expected.

2) With irq accounting the RT throttler does not kick in and the RCU stall/lockups happen.

What is this telling us?

It seems that the fine grained irq time accounting affects the runtime accounting in some way which I haven't figured out yet.

With IRQ_TIME_ACCOUNTING, rq_clock_task() will always be incremented by a lesser-or-equal value than when not having the option; you start with the same delta_exec but slice some for the IRQ accounting, and leave the rest for the rq_clock_task() (+paravirt).

IIUC this means that if you spend e.g. 10% of the time in IRQ and 90% of the time running the stress-ng RT tasks, despite having RT tasks hogging the entirety of the "available time" it is still only 90% runtime, which is below the 95% default and the throttling doesn't happen.

I don't know if considering IRQ time in some way or another in sched_rt_runtime_exceeded() really is a way out here.

Quoted 3 lines

Thanks,

tglx

#17 lore raw
Quoted 19 lines · Valentin Schneider
Quoted 11 lines · Thomas Gleixner
Quoted 6 lines · Vladimir Oltean
Quoted 5 lines

1) When irq accounting is disabled, RT throttling kicks in as expected.

2) With irq accounting the RT throttler does not kick in and the RCU stall/lockups happen.

What is this telling us?

It seems that the fine grained irq time accounting affects the runtime accounting in some way which I haven't figured out yet.

With IRQ_TIME_ACCOUNTING, rq_clock_task() will always be incremented by a lesser-or-equal value than when not having the option; you start with the same delta_exec but slice some for the IRQ accounting, and leave the rest for the rq_clock_task() (+paravirt).

Quoted 12 lines

IIUC this means that if you spend e.g. 10% of the time in IRQ and 90% of the time running the stress-ng RT tasks, despite having RT tasks hogging the entirety of the "available time" it is still only 90% runtime, which is below the 95% default and the throttling doesn't happen.

I don't know if considering IRQ time in some way or another in sched_rt_runtime_exceeded() really is a way out here.

Quoted 3 lines

Thanks,

tglx

Not sure it this is relevant at all to the conversation we're having (sorry if it isn't), but FWIW, on my armv7 board, enabling CONFIG_IRQ_TIME_ACCOUNTING makes it completely unable to boot (the log below is just earlyprintk):

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0xf00
[    0.000000] Linux version 5.8.0-rc7-02154-g1a0b51626a14-dirty (tigrisor@skbuf) (arm-linux-gnueabihf-gcc (Linaro GCC 7.3-2018.05) 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701], GNU ld (Linaro_Binutils-2018.05) 2.28.2.20170706) #383 SMP Mon Aug 3 19:03:50 EEST 2020
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: NXP LS1021A-TSN Board
[    0.000000] earlycon: uart8250 at MMIO 0x021c0500 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] efi: UEFI not found.
[    0.000000] Ignoring RAM at 0xb0000000-0xc0000000
[    0.000000] Consider using a HIGHMEM enabled kernel.
[    0.000000] cma: Reserved 64 MiB at 0xac000000
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000080000000-0x00000000afffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000afffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000afffffff]
[    0.000000] percpu: Embedded 20 pages/cpu s51660 r8192 d22068 u81920
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 194880
[    0.000000] Kernel command line: console=ttyS0,115200 earlycon=uart8250,mmio,0x21c0500 root=/dev/mmcblk0p2 rootwait rw
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 687944K/786432K available (12288K kernel code, 1026K rwdata, 6980K rodata, 1024K init, 279K bss, 32952K reserved, 65536K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=16 to nr_cpu_ids=2.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] random: get_random_bytes called from start_kernel+0x36c/0x54c with crng_init=0
[    0.000000] timer_probe: no matching timers found
[    0.000000] Console: colour dummy device 80x30
[    0.000000] sched_clock: 32 bits at 100 Hz, resolution 10000000ns, wraps every 21474836475000000ns

Thanks,
-Vladimir

#18 lore raw
In reply to Valentin Schneider

Valentin,

Quoted 18 lines · Valentin Schneider
Quoted 11 lines · Thomas Gleixner
Quoted 6 lines · Vladimir Oltean
Quoted 5 lines

1) When irq accounting is disabled, RT throttling kicks in as expected.

2) With irq accounting the RT throttler does not kick in and the RCU stall/lockups happen.

What is this telling us?

It seems that the fine grained irq time accounting affects the runtime accounting in some way which I haven't figured out yet.

With IRQ_TIME_ACCOUNTING, rq_clock_task() will always be incremented by a lesser-or-equal value than when not having the option; you start with the same delta_exec but slice some for the IRQ accounting, and leave the rest for the rq_clock_task() (+paravirt).

Quoted 4 lines

IIUC this means that if you spend e.g. 10% of the time in IRQ and 90% of the time running the stress-ng RT tasks, despite having RT tasks hogging the entirety of the "available time" it is still only 90% runtime, which is below the 95% default and the throttling doesn't happen.

totaltime = irqtime + tasktime

Ignoring irqtime and pretending that totaltime is what the scheduler can control and deal with is naive at best.

Thanks,

tglx

#19 lore raw
Quoted 26 lines · Thomas Gleixner

Valentin,

Quoted 18 lines · Valentin Schneider
Quoted 11 lines · Thomas Gleixner
Quoted 6 lines · Vladimir Oltean
Quoted 5 lines

1) When irq accounting is disabled, RT throttling kicks in as expected.

2) With irq accounting the RT throttler does not kick in and the RCU stall/lockups happen.

What is this telling us?

It seems that the fine grained irq time accounting affects the runtime accounting in some way which I haven't figured out yet.

With IRQ_TIME_ACCOUNTING, rq_clock_task() will always be incremented by a lesser-or-equal value than when not having the option; you start with the same delta_exec but slice some for the IRQ accounting, and leave the rest for the rq_clock_task() (+paravirt).

Quoted 4 lines

IIUC this means that if you spend e.g. 10% of the time in IRQ and 90% of the time running the stress-ng RT tasks, despite having RT tasks hogging the entirety of the "available time" it is still only 90% runtime, which is below the 95% default and the throttling doesn't happen.

Quoted 5 lines

totaltime = irqtime + tasktime

Ignoring irqtime and pretending that totaltime is what the scheduler can control and deal with is naive at best.

Agreed, however AFAICT rt_time is only incremented by rq_clock_task() deltas, which don't include IRQ time with IRQ_TIME_ACCOUNTING=y. That would then be directly compared to the sysctl runtime.

Adding some prints in sched_rt_runtime_exceeded() and running this test case on my Juno, I get:

# IRQ_TIME_ACCOUNTING=y
cpu=2 rt_time=713455220 runtime=950000000 rq->avg_irq.util_avg=265 (rt_time oscillates between [70.1e7, 75.1e7]; avg_irq between [220, 270])

  # IRQ_TIME_ACCOUNTING=n
  cpu=2 rt_time=963035300 runtime=949951811
  (rt_time oscillates between [94.1e7, 96.1e7];

Throttling happens for IRQ_TIME_ACCOUNTING=n and doesn't for IRQ_TIME_ACCOUNTING=y - clearly the accounted rt_time isn't high enough for that to happen, and it does look like what is missing in rt_time (or what should be subtracted from the available runtime) is there in the avg_irq.

Or is that another case where I shouldn't have been writing emails at this hour?

Quoted 3 lines

Thanks,

tglx

#20 lore raw
Quoted 44 lines · Valentin Schneider
Quoted 26 lines · Thomas Gleixner

Valentin,

Quoted 18 lines · Valentin Schneider
Quoted 11 lines · Thomas Gleixner
Quoted 6 lines · Vladimir Oltean
Quoted 5 lines

1) When irq accounting is disabled, RT throttling kicks in as expected.

2) With irq accounting the RT throttler does not kick in and the RCU stall/lockups happen.

What is this telling us?

It seems that the fine grained irq time accounting affects the runtime accounting in some way which I haven't figured out yet.

With IRQ_TIME_ACCOUNTING, rq_clock_task() will always be incremented by a lesser-or-equal value than when not having the option; you start with the same delta_exec but slice some for the IRQ accounting, and leave the rest for the rq_clock_task() (+paravirt).

Quoted 4 lines

IIUC this means that if you spend e.g. 10% of the time in IRQ and 90% of the time running the stress-ng RT tasks, despite having RT tasks hogging the entirety of the "available time" it is still only 90% runtime, which is below the 95% default and the throttling doesn't happen.

Quoted 5 lines

totaltime = irqtime + tasktime

Ignoring irqtime and pretending that totaltime is what the scheduler can control and deal with is naive at best.

Agreed, however AFAICT rt_time is only incremented by rq_clock_task() deltas, which don't include IRQ time with IRQ_TIME_ACCOUNTING=y. That would then be directly compared to the sysctl runtime.

Adding some prints in sched_rt_runtime_exceeded() and running this test case on my Juno, I get:
# IRQ_TIME_ACCOUNTING=y
cpu=2 rt_time=713455220 runtime=950000000 rq->avg_irq.util_avg=265 (rt_time oscillates between [70.1e7, 75.1e7]; avg_irq between [220, 270])

Quoted 8 lines
  # IRQ_TIME_ACCOUNTING=n
  cpu=2 rt_time=963035300 runtime=949951811
  (rt_time oscillates between [94.1e7, 96.1e7];

Throttling happens for IRQ_TIME_ACCOUNTING=n and doesn't for IRQ_TIME_ACCOUNTING=y - clearly the accounted rt_time isn't high enough for that to happen, and it does look like what is missing in rt_time (or what should be subtracted from the available runtime) is there in the avg_irq.

I agree that w/ IRQ_TIME_ACCOUNTING=y rt_rq->rt_time isn't high enough in this testcase.

stress-ng-hrtim-1655 [001] 462.897733: bprint: update_curr_rt:
rt_rq->rt_time=416716900 rt_rq->rt_runtime=950000000 rt_b->rt_runtime=950000000

The 5% reservation (1 - sched_rt_runtime_us/sched_rt_period_us) for CFS is massively eclipsed by irqtime.

It's true that avg_irq tracks 'irq_delta + steal' time but it is meant to potentially reduce cpu capacity. It's also cpu and frequency invariant (your CPU2 is a big CPU so no issue here).

Could a rq_clock(rq) derived rt_rq signal been used to compare against rt_runtime?

BTW, DL already influences rt_rq->rt_time.

[...]

#21 lore raw
In reply to Thomas Gleixner
Quoted 4 lines · Thomas Gleixner

totaltime = irqtime + tasktime

Ignoring irqtime and pretending that totaltime is what the scheduler can control and deal with is naive at best.

Well no, that's what we call system overhead and is assumed to be included in the 'error margin'.

The way things are set up is that we say that, by default, RT tasks can consume 95% of cputime and the remaining 5% is sufficient to keep the system alive.

Those 5% include all system overhead, IRQs, RCU, !RT workqueues etc..

Obviously IRQ_TIME accounting changes the balance a bit, but that's what it is. We can't really do anything better.

Apparently this SoC has significant IRQ time for some reason. Also, relying on RT throttling for 'correct' behaviour is also wrong. What needs to be done is find who is using all this RT time and why, that isn't right.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#22 lore raw
Quoted 10 lines · On 05/08/20 14:40, [email protected] wrote:
Quoted 4 lines · Thomas Gleixner

totaltime = irqtime + tasktime

Ignoring irqtime and pretending that totaltime is what the scheduler can control and deal with is naive at best.

Well no, that's what we call system overhead and is assumed to be included in the 'error margin'.

Quoted 9 lines

The way things are set up is that we say that, by default, RT tasks can consume 95% of cputime and the remaining 5% is sufficient to keep the system alive.

Those 5% include all system overhead, IRQs, RCU, !RT workqueues etc..

Obviously IRQ_TIME accounting changes the balance a bit, but that's what it is. We can't really do anything better.

I'm starting to think that as well. I tried some fugly hack of injecting avg_irq into sched_rt_runtime_exceeded() with something along the lines of:

  irq_time = (rq->avg_irq.util_avg * sched_rt_period(rt_rq)) >> SCHED_CAPACITY_SHIFT;

It's pretty bad for a few reasons; one is that avg_irq already has its own period (PELT-based). Another is that it is, as Dietmar pointed out, CPU and freq invariant, so falls over on big.LITTLE.

Making update_curr_rt() use rq_clock() rather than rq_clock_task() makes it "work" but goes against all the good reasons there were to introduce rq_clock_task() in the first place.

Quoted 4 lines

Apparently this SoC has significant IRQ time for some reason. Also, relying on RT throttling for 'correct' behaviour is also wrong. What needs to be done is find who is using all this RT time and why, that isn't right.

I've been tempted to say the test case is a bit bogus, but am not familiar enough with the RT throttling details to stand that ground. That said, from both looking at the execution and the stress-ng source code, it seems to unconditionally spawn 32 FIFO-50 tasks (there's even an option to make these FIFO-99!!!), which is quite a crowd on monoCPU systems.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#23 lore raw
Quoted 5 lines · Valentin Schneider

I've been tempted to say the test case is a bit bogus, but am not familiar enough with the RT throttling details to stand that ground. That said, from both looking at the execution and the stress-ng source code, it seems to unconditionally spawn 32 FIFO-50 tasks (there's even an option to make these FIFO-99!!!), which is quite a crowd on monoCPU systems.

Oh, so it's a case of: we do stupid without tuning and the system falls over. I can live with that.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#30 lore raw
In reply to [email protected]
Quoted 16 lines · [email protected] writes:
Quoted 4 lines · Thomas Gleixner

totaltime = irqtime + tasktime

Ignoring irqtime and pretending that totaltime is what the scheduler can control and deal with is naive at best.

Well no, that's what we call system overhead and is assumed to be included in the 'error margin'.

The way things are set up is that we say that, by default, RT tasks can consume 95% of cputime and the remaining 5% is sufficient to keep the system alive.

Those 5% include all system overhead, IRQs, RCU, !RT workqueues etc..

Quoted 7 lines

Obviously IRQ_TIME accounting changes the balance a bit, but that's what it is. We can't really do anything better.

Apparently this SoC has significant IRQ time for some reason. Also, relying on RT throttling for 'correct' behaviour is also wrong. What needs to be done is find who is using all this RT time and why, that isn't right.

It's a test case and we know already who is using the time. But that's not the point.

A runaway RT task resulting in a RCU stall or whatever lockup of the system is definitely not the right answer.

The throttler, as much as it's a horrible hack, is there to prevent this and to give the admin a chance to pinpoint and kill that thing instead of having to press the reset button and scratching head what might have caused this.

Thanks,

tglx

#24 lore raw
In reply to [email protected]
Quoted 10 lines · [email protected] writes:
Quoted 5 lines · Valentin Schneider

I've been tempted to say the test case is a bit bogus, but am not familiar enough with the RT throttling details to stand that ground. That said, from both looking at the execution and the stress-ng source code, it seems to unconditionally spawn 32 FIFO-50 tasks (there's even an option to make these FIFO-99!!!), which is quite a crowd on monoCPU systems.

Oh, so it's a case of: we do stupid without tuning and the system falls over. I can live with that.

It's not a question of whether you can live with that behaviour for a particular silly test case.

The same happens with a single RT runaway task with enough interrupt load on a UP machine. Just validated that. And that has nothing to do with a silly test case. Sporadic runaways due to a bug in a once per week code path simply can happen and having the safety net working depending on a config option selected or not is just wrong.

Thanks,

tglx

#25 lore raw
Quoted 9 lines · Thomas Gleixner
Quoted 8 lines · [email protected] writes:
Quoted 5 lines · Valentin Schneider

I've been tempted to say the test case is a bit bogus, but am not familiar enough with the RT throttling details to stand that ground. That said, from both looking at the execution and the stress-ng source code, it seems to unconditionally spawn 32 FIFO-50 tasks (there's even an option to make these FIFO-99!!!), which is quite a crowd on monoCPU systems.

Quoted 8 lines
Quoted 2 lines

Oh, so it's a case of: we do stupid without tuning and the system falls over. I can live with that.

It's not a question of whether you can live with that behaviour for a particular silly test case.

The same happens with a single RT runaway task with enough interrupt load on a UP machine. Just validated that.

Of course.

Quoted 4 lines

And that has nothing to do
with a silly test case. Sporadic runaways due to a bug in a once per week code path simply can happen and having the safety net working depending on a config option selected or not is just wrong.

The safety thing is concerned with RT tasks. It doesn't pretend to help with runnaway IRQs, never has, never will.

The further extreme is an interrupt storm, those have always taken a machine down.

Accounting unrelated IRQ time to RT tasks is equally wrong, the task execution is unrelated to the IRQs. The config option at least offers insight into where time goes -- and it's a config option because doing time accounting on interrupts adds overhead :/

This really is a no-win all round.

The only 'sensible' option here is threaded IRQs, where the IRQ line gets disabled until the handler thread has ran, that also helps with IRQ storms.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#26 lore raw
Quoted 21 lines · On Thu, Aug 06, 2020 at 01:45:45PM +0200, [email protected] wrote:
Quoted 9 lines · Thomas Gleixner
Quoted 8 lines · [email protected] writes:
Quoted 5 lines · Valentin Schneider

I've been tempted to say the test case is a bit bogus, but am not familiar enough with the RT throttling details to stand that ground. That said, from both looking at the execution and the stress-ng source code, it seems to unconditionally spawn 32 FIFO-50 tasks (there's even an option to make these FIFO-99!!!), which is quite a crowd on monoCPU systems.

Quoted 8 lines
Quoted 2 lines

Oh, so it's a case of: we do stupid without tuning and the system falls over. I can live with that.

It's not a question of whether you can live with that behaviour for a particular silly test case.

The same happens with a single RT runaway task with enough interrupt load on a UP machine. Just validated that.

Of course.

Quoted 7 lines
Quoted 4 lines

And that has nothing to do
with a silly test case. Sporadic runaways due to a bug in a once per week code path simply can happen and having the safety net working depending on a config option selected or not is just wrong.

The safety thing is concerned with RT tasks. It doesn't pretend to help with runnaway IRQs, never has, never will.

Getting into the time machine back to the 1990s...

DYNIX/ptx had a discretionary mechanism to deal with excessive interrupts. There was a function that long-running interrupt handlers were supposed to call periodically that would return false if the system felt that the CPU had done enough interrupts for the time being. In that case, the interrupt handler was supposed to schedule itself for a later time, but leave the interrupt unacknowledged in order to prevent retriggering in the meantime.

Of course, this mechanism would be rather less helpful in Linux.

For one, Linux has way more device drivers and way more oddball devices. In contrast, the few devices that DYNIX/ptx supported were carefully selected, and the selection criteria included being able to put up with this sort of thing. Also, the fact that there was but a handful of device drivers meant that changes like this could be more easily propagated through all drivers.

Also, Linux supports way more workloads. In contrast, DYNIX/ptx could pick a small percentage of each CPU that would be permitted to be used by hardware interrupt handlers. As in there are probably Linux workloads that run >90% of some poor CPU within hardware interrupt handlers.

But reminiscing anyway on the off-chance that this inspires someone to come up with an idea that would work well in the Linux environment.

Thanx, Paul

Quoted 8 lines

The further extreme is an interrupt storm, those have always taken a machine down.

Accounting unrelated IRQ time to RT tasks is equally wrong, the task execution is unrelated to the IRQs. The config option at least offers insight into where time goes -- and it's a config option because doing time accounting on interrupts adds overhead :/

Quoted 5 lines

This really is a no-win all round.

The only 'sensible' option here is threaded IRQs, where the IRQ line gets disabled until the handler thread has ran, that also helps with IRQ storms.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#29 lore raw
In reply to [email protected]

Peter,

Quoted 8 lines · [email protected] writes:
Quoted 4 lines · Thomas Gleixner

And that has nothing to do
with a silly test case. Sporadic runaways due to a bug in a once per week code path simply can happen and having the safety net working depending on a config option selected or not is just wrong.

The safety thing is concerned with RT tasks. It doesn't pretend to help with runnaway IRQs, never has, never will.

Of course not. But without irq accounting the runtime is accounted on the runaway task which causes it to throttle.

Quoted 2 lines

The further extreme is an interrupt storm, those have always taken a machine down.

If every interrupt is actually handled, then yes.

Quoted 4 lines

Accounting unrelated IRQ time to RT tasks is equally wrong, the task execution is unrelated to the IRQs. The config option at least offers insight into where time goes -- and it's a config option because doing time accounting on interrupts adds overhead :/

Right, but it's not totally out of the world either to make the throttler do:

if (rt_runtime + irq_time > threshold)
try_to_keep_the_box_alive()

Quoted 1 line

This really is a no-win all round.

That's not the question here :)

Quoted 3 lines

The only 'sensible' option here is threaded IRQs, where the IRQ line gets disabled until the handler thread has ran, that also helps with IRQ storms.

I'm not against enforcing threaded IRQs. :)

Thanks,

tglx

#27 lore raw
In reply to Paul E. McKenney

Paul,

Quoted 14 lines · Paul E. McKenney
Quoted 2 lines · On Thu, Aug 06, 2020 at 01:45:45PM +0200, [email protected] wrote:

The safety thing is concerned with RT tasks. It doesn't pretend to help with runnaway IRQs, never has, never will.

Getting into the time machine back to the 1990s...

DYNIX/ptx had a discretionary mechanism to deal with excessive interrupts. There was a function that long-running interrupt handlers were supposed to call periodically that would return false if the system felt that the CPU had done enough interrupts for the time being. In that case, the interrupt handler was supposed to schedule itself for a later time, but leave the interrupt unacknowledged in order to prevent retriggering in the meantime.

Quoted 8 lines

Of course, this mechanism would be rather less helpful in Linux.

For one, Linux has way more device drivers and way more oddball devices. In contrast, the few devices that DYNIX/ptx supported were carefully selected, and the selection criteria included being able to put up with this sort of thing. Also, the fact that there was but a handful of device drivers meant that changes like this could be more easily propagated through all drivers.

We could do that completely at the core interrupt handling level.

Quoted 4 lines

Also, Linux supports way more workloads. In contrast, DYNIX/ptx could pick a small percentage of each CPU that would be permitted to be used by hardware interrupt handlers. As in there are probably Linux workloads that run >90% of some poor CPU within hardware interrupt handlers.

Yet another tunable. /me runs

#28 lore raw
Quoted 19 lines · Thomas Gleixner

Paul,

Quoted 16 lines · Paul E. McKenney
Quoted 2 lines · On Thu, Aug 06, 2020 at 01:45:45PM +0200, [email protected] wrote:

The safety thing is concerned with RT tasks. It doesn't pretend to help with runnaway IRQs, never has, never will.

Getting into the time machine back to the 1990s...

DYNIX/ptx had a discretionary mechanism to deal with excessive interrupts. There was a function that long-running interrupt handlers were supposed to call periodically that would return false if the system felt that the CPU had done enough interrupts for the time being. In that case, the interrupt handler was supposed to schedule itself for a later time, but leave the interrupt unacknowledged in order to prevent retriggering in the meantime.

Of course, this mechanism would be rather less helpful in Linux.

Quoted 8 lines
Quoted 6 lines

For one, Linux has way more device drivers and way more oddball devices. In contrast, the few devices that DYNIX/ptx supported were carefully selected, and the selection criteria included being able to put up with this sort of thing. Also, the fact that there was but a handful of device drivers meant that changes like this could be more easily propagated through all drivers.

We could do that completely at the core interrupt handling level.

Ah, true enough if the various NAPI-like devices give up the CPU from time to time. Which they might well do for all I know.

Quoted 6 lines
Quoted 4 lines

Also, Linux supports way more workloads. In contrast, DYNIX/ptx could pick a small percentage of each CPU that would be permitted to be used by hardware interrupt handlers. As in there are probably Linux workloads that run >90% of some poor CPU within hardware interrupt handlers.

Yet another tunable. /me runs

;-) ;-) ;-)

If there are workloads that would like to be able to keep one or more CPUs completely busy handling interrupts, it should be possible to create something that is used sort of like cond_resched() to keep RCU, the scheduler, and the various watchdogs and lockup detectors at bay.

For example, RCU could supply a function that checked to make sure that it was in an interrupt from idle, and if so report a quiescent state for that CPU. So if the CPU was idle and there wasn't anything pending for it, that CPU could safely stay in a hardirq handler indefinitely. I suppose that the function should return an indication in cases such as interrupt from non-idle.

Sort of like NO_HZ_FULL, but for hardirq handlers, and also allowing those handlers to use RCU read-side critical sections.

Or we could do what all the cool kids do these days, namely just apply machine learning, thus automatically self-tuning in real time.

/me runs...

Thanx, Paul