rcu_sched detected expedited stalls in amdgpu after suspend
Hi,
Since Linux 5.19-ish, I consistently get these types of errors when resuming from S3:
[15652.909157] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 7 jiffies s: 9981 root: 0x800/.
[15652.909162] rcu: blocking rcu_node structures (internal RCU debug):
[15652.909163] Task dump for CPU 11:
[15652.909164] task:kworker/u24:65 state:R running task stack: 0 pid:210218 ppid: 2 flags:0x00004008
[15652.909167] Workqueue: events_unbound async_run_entry_fn
[15652.909172] Call Trace:
[15652.909173] <TASK>
[15652.909174] ? atom_get_src_int+0x38e/0x680
[15652.909179] ? atom_op_test+0x67/0x190
[15652.909181] ? amdgpu_atom_execute_table_locked+0x19a/0x300
[15652.909184] ? atom_op_calltable+0xb1/0x110
[15652.909186] ? amdgpu_atom_execute_table_locked+0x19a/0x300
[15652.909189] ? atom_op_calltable+0xb1/0x110
[15652.909191] ? amdgpu_atom_execute_table_locked+0x19a/0x300
[15652.909193] ? __switch_to+0x137/0x440
[15652.909195] ? amdgpu_atom_asic_init+0xe0/0x100
[15652.909198] ? pci_bus_read_config_dword+0x36/0x50
[15652.909201] ? amdgpu_device_resume+0x10b/0x3e0
[15652.909203] ? amdgpu_pmops_resume+0x32/0x60
[15652.909204] ? pci_pm_suspend+0x2b0/0x2b0
[15652.909206] ? dpm_run_callback+0x35/0x1f0
[15652.909209] ? device_resume+0x1ca/0x220
[15652.909211] ? async_resume+0x19/0xe0
[15652.909213] ? async_run_entry_fn+0x33/0x120
[15652.909215] ? process_one_work+0x1d6/0x350
[15652.909218] ? worker_thread+0x24d/0x480
[15652.909220] ? kthread+0x137/0x150
[15652.909221] ? worker_clr_flags+0x40/0x40
[15652.909224] ? kthread_blkcg+0x30/0x30
[15652.909226] ? ret_from_fork+0x22/0x30
[15652.909227] </TASK>
[15653.015808] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 7 jiffies s: 9985 root: 0x800/.
[15653.015812] rcu: blocking rcu_node structures (internal RCU debug):
[15653.015813] Task dump for CPU 11:
[15653.015813] task:kworker/u24:65 state:R running task stack: 0 pid:210218 ppid: 2 flags:0x00004008
[15653.015816] Workqueue: events_unbound async_run_entry_fn
[15653.015820] Call Trace:
[15653.015820] <TASK>
[15653.015821] ? amdgpu_cgs_read_register+0x10/0x10
[15653.015825] ? smu7_copy_bytes_to_smc+0xd4/0x200
[15653.015828] ? polaris10_program_memory_timing_parameters+0x195/0x1b0
[15653.015831] ? sysvec_apic_timer_interrupt+0xa/0x80
[15653.015834] ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
[15653.015836] ? amdgpu_cgs_destroy_device+0x10/0x10
[15653.015839] ? sysvec_apic_timer_interrupt+0xa/0x80
[15653.015841] ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
[15653.015843] ? amdgpu_cgs_destroy_device+0x10/0x10
[15653.015846] ? amdgpu_device_rreg+0x8f/0xd0
[15653.015847] ? phm_wait_for_register_unequal+0x99/0xd0
[15653.015850] ? smu7_send_msg_to_smc+0x95/0x130
[15653.015853] ? smum_send_msg_to_smc+0x5d/0xa0
[15653.015854] ? amdgpu_cgs_read_ind_register+0xa0/0xa0
[15653.015857] ? smu7_enable_dpm_tasks+0x241f/0x28c0
[15653.015859] ? hwmgr_resume+0x31/0x70
[15653.015861] ? amdgpu_device_resume+0x1fa/0x3e0
[15653.015863] ? amdgpu_pmops_resume+0x32/0x60
[15653.015864] ? pci_pm_suspend+0x2b0/0x2b0
[15653.015866] ? dpm_run_callback+0x35/0x1f0
[15653.015868] ? device_resume+0x1ca/0x220
[15653.015870] ? async_resume+0x19/0xe0
[15653.015872] ? async_run_entry_fn+0x33/0x120
[15653.015874] ? process_one_work+0x1d6/0x350
[15653.015877] ? worker_thread+0x24d/0x480
[15653.015878] ? kthread+0x137/0x150
[15653.015880] ? worker_clr_flags+0x40/0x40
[15653.015882] ? kthread_blkcg+0x30/0x30
[15653.015884] ? ret_from_fork+0x22/0x30
[15653.015886] </TASK>I have not noticed any resulting problems. I am reporting this in the hope that it is easy to fix the issue and remove the error messages which may obscure some future problem.
Thanks,
Alex.
Paul E. McKenney
maintainerQuoted 74 lines · On Mon, Jun 27, 2022 at 03:22:24PM -0400, Alex Xu (Hello71) wrote:
Hi,
Since Linux 5.19-ish, I consistently get these types of errors when resuming from S3:
[15652.909157] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 7 jiffies s: 9981 root: 0x800/. [15652.909162] rcu: blocking rcu_node structures (internal RCU debug): [15652.909163] Task dump for CPU 11: [15652.909164] task:kworker/u24:65 state:R running task stack: 0 pid:210218 ppid: 2 flags:0x00004008 [15652.909167] Workqueue: events_unbound async_run_entry_fn [15652.909172] Call Trace: [15652.909173] <TASK> [15652.909174] ? atom_get_src_int+0x38e/0x680 [15652.909179] ? atom_op_test+0x67/0x190 [15652.909181] ? amdgpu_atom_execute_table_locked+0x19a/0x300 [15652.909184] ? atom_op_calltable+0xb1/0x110 [15652.909186] ? amdgpu_atom_execute_table_locked+0x19a/0x300 [15652.909189] ? atom_op_calltable+0xb1/0x110 [15652.909191] ? amdgpu_atom_execute_table_locked+0x19a/0x300 [15652.909193] ? __switch_to+0x137/0x440 [15652.909195] ? amdgpu_atom_asic_init+0xe0/0x100 [15652.909198] ? pci_bus_read_config_dword+0x36/0x50 [15652.909201] ? amdgpu_device_resume+0x10b/0x3e0 [15652.909203] ? amdgpu_pmops_resume+0x32/0x60 [15652.909204] ? pci_pm_suspend+0x2b0/0x2b0 [15652.909206] ? dpm_run_callback+0x35/0x1f0 [15652.909209] ? device_resume+0x1ca/0x220 [15652.909211] ? async_resume+0x19/0xe0 [15652.909213] ? async_run_entry_fn+0x33/0x120 [15652.909215] ? process_one_work+0x1d6/0x350 [15652.909218] ? worker_thread+0x24d/0x480 [15652.909220] ? kthread+0x137/0x150 [15652.909221] ? worker_clr_flags+0x40/0x40 [15652.909224] ? kthread_blkcg+0x30/0x30 [15652.909226] ? ret_from_fork+0x22/0x30 [15652.909227] </TASK> [15653.015808] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 7 jiffies s: 9985 root: 0x800/. [15653.015812] rcu: blocking rcu_node structures (internal RCU debug): [15653.015813] Task dump for CPU 11: [15653.015813] task:kworker/u24:65 state:R running task stack: 0 pid:210218 ppid: 2 flags:0x00004008 [15653.015816] Workqueue: events_unbound async_run_entry_fn [15653.015820] Call Trace: [15653.015820] <TASK> [15653.015821] ? amdgpu_cgs_read_register+0x10/0x10 [15653.015825] ? smu7_copy_bytes_to_smc+0xd4/0x200 [15653.015828] ? polaris10_program_memory_timing_parameters+0x195/0x1b0 [15653.015831] ? sysvec_apic_timer_interrupt+0xa/0x80 [15653.015834] ? asm_sysvec_apic_timer_interrupt+0x1b/0x20 [15653.015836] ? amdgpu_cgs_destroy_device+0x10/0x10 [15653.015839] ? sysvec_apic_timer_interrupt+0xa/0x80 [15653.015841] ? asm_sysvec_apic_timer_interrupt+0x1b/0x20 [15653.015843] ? amdgpu_cgs_destroy_device+0x10/0x10 [15653.015846] ? amdgpu_device_rreg+0x8f/0xd0 [15653.015847] ? phm_wait_for_register_unequal+0x99/0xd0 [15653.015850] ? smu7_send_msg_to_smc+0x95/0x130 [15653.015853] ? smum_send_msg_to_smc+0x5d/0xa0 [15653.015854] ? amdgpu_cgs_read_ind_register+0xa0/0xa0 [15653.015857] ? smu7_enable_dpm_tasks+0x241f/0x28c0 [15653.015859] ? hwmgr_resume+0x31/0x70 [15653.015861] ? amdgpu_device_resume+0x1fa/0x3e0 [15653.015863] ? amdgpu_pmops_resume+0x32/0x60 [15653.015864] ? pci_pm_suspend+0x2b0/0x2b0 [15653.015866] ? dpm_run_callback+0x35/0x1f0 [15653.015868] ? device_resume+0x1ca/0x220 [15653.015870] ? async_resume+0x19/0xe0 [15653.015872] ? async_run_entry_fn+0x33/0x120 [15653.015874] ? process_one_work+0x1d6/0x350 [15653.015877] ? worker_thread+0x24d/0x480 [15653.015878] ? kthread+0x137/0x150 [15653.015880] ? worker_clr_flags+0x40/0x40 [15653.015882] ? kthread_blkcg+0x30/0x30 [15653.015884] ? ret_from_fork+0x22/0x30 [15653.015886] </TASK>
Quoted 3 lines
I have not noticed any resulting problems. I am reporting this in the hope that it is easy to fix the issue and remove the error messages which may obscure some future problem.
The usual way this happens is for a task to be spinning. In this case, that might be due to excessive lock contention on async_lock within the async_run_entry_fn() function. Or perhaps the problem is in one of the functions preceded by "?" above.
One way to debug this is to place trace_printk()s or similar in the functions called out above to track it down.
I bet that the reason this showed up in v5.19 is this guy:
28b3ae426598 ("rcu: Introduce CONFIG_RCU_EXP_CPU_STALL_TIMEOUT")
So do you have CONFIG_RCU_EXP_CPU_STALL_TIMEOUT set to some small number of milliseconds? If so, you can override this by adjusting the CONFIG_RCU_EXP_CPU_STALL_TIMEOUT Kconfig option or by booting with a longer timeout via the rcupdate.rcu_exp_cpu_stall_timeout= kernel boot parameter.
But if you are running on an Android platform, Uladzislau will be interested in addressing the underlying issue, so I have added him on CC.
Thanx, Paul
CONFIG_ANDROID (was: rcu_sched detected expedited stalls in amdgpu after suspend)
In reply to a message not in the archiveExcerpts from Paul E. McKenney's message of June 28, 2022 12:12 am:
Quoted 22 lines · On Mon, Jun 27, 2022 at 09:50:53PM -0400, Alex Xu (Hello71) wrote:
Quoted 11 lines
Ah, I see. I have selected the default value for
CONFIG_RCU_EXP_CPU_STALL_TIMEOUT, but that is 20 if ANDROID. I am not using Android; I'm not sure there exist Android devices with AMD GPUs. However, I have setCONFIG_ANDROID=yin order to use ANDROID_BINDER_IPC=m for emulation.In general, I think
CONFIG_ANDROIDis not a reliable method for detecting if the kernel is for an Android device; for example, Fedora setsCONFIG_ANDROID, but (AFAIK) its kernel is not intended for use with Android userspace.Quoted 5 lines
On the other hand, it's not clear to me why the value 20 should be for Android only anyways. If, as you say in
https://lore.kernel.org/lkml/20220216195508.GM4285@paulmck-ThinkPad-P17-Gen-1/,
it is related to the size of the system, perhaps some other heuristic would be more appropriate.It is related to the fact that quite a few Android guys want these 20-millisecond short-timeout expedited RCU CPU stall warnings, but no one else does. Not yet anyway.
Quoted 7 lines
And let's face it, the intent and purpose of
CONFIG_ANDROID=yis extremely straightforward and unmistakeable. So perhaps people not running Android devices but wanting a little bit of the Android functionality should do something other than settingCONFIG_ANDROID=yin their .config files. Me, I am surprised that it took this long for something like this to bite you.But just out of curiosity, what would you suggest instead?
Both Debian and Fedora set CONFIG_ANDROID, specifically for binder. If major distro vendors are consistently making this "mistake", then perhaps the problem is elsewhere.
In my own opinion, assuming that binderfs means Android vendor is not a good assumption. The ANDROID help says:
Quoted 1 line
Enable support for various drivers needed on the Android platform
It doesn't say "Enable only if building an Android device", or "Enable only if you are Google". Isn't the traditional Linux philosophy a collection of pieces to be assembled, without gratuitous hidden dependencies? For example, [0] removes the unnecessary Android dependency, it doesn't block the whole thing with "depends on ANDROID".
It seems to me that the proper way to set some configuration for Android kernels is or should be to ask the Android kernel config maintainers, not to set it based on an upstream kernel option. There is, after all, no CONFIG_FEDORA or CONFIG_UBUNTU or CONFIG_HANNAH_MONTANA.
WireGuard and random also use CONFIG_ANDROID in a similar "proxy" way as rcu, there to see if suspends are "frequent". This seems dubious for the same reasons.
I wonder if it might be time to retire CONFIG_ANDROID: the only remaining driver covered is binder, which originates from Android but is no longer used exclusively on Android systems. Like ufs-qcom, binder is no longer used exclusively on Android devices; it is also used for Android device emulators, which might be used on Android-like mobile devices, or might not.
My understanding is that both Android and upstream kernel developers intend to add no more Android-specific drivers, so binder should be the only one covered for the foreseeable future.
Quoted 1 line
For that matter, why the private reply?
Mail client issues, not intentional. Lists re-added, plus Android, WireGuard, and random.
Thanks,
Alex.
Re: CONFIG_ANDROID (was: rcu_sched detected expedited stalls in amdgpu after suspend)
Hi Alex,
Quoted 3 lines · On Tue, Jun 28, 2022 at 11:02:40AM -0400, Alex Xu (Hello71) wrote:
WireGuard and random also use
CONFIG_ANDROIDin a similar "proxy" way as rcu, there to see if suspends are "frequent". This seems dubious for the same reasons.
I'd be happy to take a patch in WireGuard and random.c to get rid of the CONFIG_ANDROID usage, if you can conduct an analysis and conclude this won't break anything inadvertently.
Jason
Paul E. McKenney
maintainerRe: CONFIG_ANDROID (was: rcu_sched detected expedited stalls in amdgpu after suspend)
In reply to Alex Xu (Hello71)Quoted 62 lines · On Tue, Jun 28, 2022 at 11:02:40AM -0400, Alex Xu (Hello71) wrote:
Excerpts from Paul E. McKenney's message of June 28, 2022 12:12 am:
Quoted 22 lines · On Mon, Jun 27, 2022 at 09:50:53PM -0400, Alex Xu (Hello71) wrote:
Quoted 11 lines
Ah, I see. I have selected the default value for
CONFIG_RCU_EXP_CPU_STALL_TIMEOUT, but that is 20 if ANDROID. I am not using Android; I'm not sure there exist Android devices with AMD GPUs. However, I have setCONFIG_ANDROID=yin order to use ANDROID_BINDER_IPC=m for emulation.In general, I think
CONFIG_ANDROIDis not a reliable method for detecting if the kernel is for an Android device; for example, Fedora setsCONFIG_ANDROID, but (AFAIK) its kernel is not intended for use with Android userspace.Quoted 5 lines
On the other hand, it's not clear to me why the value 20 should be for Android only anyways. If, as you say in
https://lore.kernel.org/lkml/20220216195508.GM4285@paulmck-ThinkPad-P17-Gen-1/,
it is related to the size of the system, perhaps some other heuristic would be more appropriate.It is related to the fact that quite a few Android guys want these 20-millisecond short-timeout expedited RCU CPU stall warnings, but no one else does. Not yet anyway.
Quoted 7 lines
And let's face it, the intent and purpose of
CONFIG_ANDROID=yis extremely straightforward and unmistakeable. So perhaps people not running Android devices but wanting a little bit of the Android functionality should do something other than settingCONFIG_ANDROID=yin their .config files. Me, I am surprised that it took this long for something like this to bite you.But just out of curiosity, what would you suggest instead?
Both Debian and Fedora set
CONFIG_ANDROID, specifically for binder. If major distro vendors are consistently making this "mistake", then perhaps the problem is elsewhere.In my own opinion, assuming that binderfs means Android vendor is not a good assumption. The ANDROID help says:
Quoted 1 line
Enable support for various drivers needed on the Android platform
It doesn't say "Enable only if building an Android device", or "Enable only if you are Google". Isn't the traditional Linux philosophy a collection of pieces to be assembled, without gratuitous hidden dependencies? For example, [0] removes the unnecessary Android dependency, it doesn't block the whole thing with "depends on ANDROID".
It seems to me that the proper way to set some configuration for Android kernels is or should be to ask the Android kernel config maintainers, not to set it based on an upstream kernel option. There is, after all, no
CONFIG_FEDORAorCONFIG_UBUNTUorCONFIG_HANNAH_MONTANA.WireGuard and random also use
CONFIG_ANDROIDin a similar "proxy" way as rcu, there to see if suspends are "frequent". This seems dubious for the same reasons.I wonder if it might be time to retire
CONFIG_ANDROID: the only remaining driver covered is binder, which originates from Android but is no longer used exclusively on Android systems. Like ufs-qcom, binder is no longer used exclusively on Android devices; it is also used for Android device emulators, which might be used on Android-like mobile devices, or might not.
Quoted 3 lines
My understanding is that both Android and upstream kernel developers intend to add no more Android-specific drivers, so binder should be the only one covered for the foreseeable future.
Thank you for the perspective, but you never did suggest an alternative.
So here is is what I suggest given the current setup:
config RCU_EXP_CPU_STALL_TIMEOUT
int "Expedited RCU CPU stall timeout in milliseconds" depends on RCU_STALL_COMMON
range 0 21000
default 20 if ANDROID
default 0 if !ANDROID
help
If a given expedited RCU grace period extends more than the specified number of milliseconds, a CPU stall warning is printed. If the RCU grace period persists, additional CPU stall warnings are printed at more widely spaced intervals. A value of zero says to use the RCU_CPU_STALL_TIMEOUT value converted from seconds to milliseconds.
The default, and only the default, is controlled by ANDROID.
All you need to do to get the previous behavior is to add something like this to your defconfig file:
CONFIG_RCU_EXP_CPU_STALL_TIMEOUT=21000
Any reason why this will not work for you?
Quoted 4 lines
Quoted 1 line
For that matter, why the private reply?
Mail client issues, not intentional. Lists re-added, plus Android, WireGuard, and random.
Thank you!
Thanx, Paul
Quoted 4 lines
Thanks,
Alex.
Re: CONFIG_ANDROID (was: rcu_sched detected expedited stalls in amdgpu after suspend)
Excerpts from Paul E. McKenney's message of June 28, 2022 2:54 pm:
Quoted 6 lines
All you need to do to get the previous behavior is to add something like this to your defconfig file:
CONFIG_RCU_EXP_CPU_STALL_TIMEOUT=21000Any reason why this will not work for you?
As far as I know, I do not require any particular RCU debugging features intended for developers; as an individual user and distro maintainer, I would like to select the option corresponding to "emit errors for unexpected conditions which should be reported upstream", not "emit debugging information for development purposes".
Therefore, I think 0 is a suitable setting for me and most ordinary (not tightly controlled) distributions. My concern is that other users and distro maintainers will also have confusion about what value to set and whether the warnings are important, since the help text does not say anything about Android, and "make oldconfig" does not indicate that the default value is different for Android.
My suggestion is that the default be set to 0, and if a non-zero value is appropriate for Android, that should be communicated to the Android developers, not made conditional on CONFIG_ANDROID.
Thanks,
Alex.
Uladzislau Rezki
maintainerRe: CONFIG_ANDROID (was: rcu_sched detected expedited stalls in amdgpu after suspend)
Quoted 6 lines
Excerpts from Paul E. McKenney's message of June 28, 2022 2:54 pm:
Quoted 5 lines
All you need to do to get the previous behavior is to add something like this to your defconfig file:
CONFIG_RCU_EXP_CPU_STALL_TIMEOUT=21000
Quoted 8 lines
Quoted 1 line
Any reason why this will not work for you?
As far as I know, I do not require any particular RCU debugging features intended for developers; as an individual user and distro maintainer, I would like to select the option corresponding to "emit errors for unexpected conditions which should be reported upstream", not "emit debugging information for development purposes".
Sorry but we need to apply some assumption, i.e. to me the CONFIG_ANDROID indicates that a kernel runs on the Android wise device. When you enable this option on you specific box it is supposed that some Android related code are activated also on your device which may lead to some side effect.
Quoted 8 lines
Therefore, I think 0 is a suitable setting for me and most ordinary (not tightly controlled) distributions. My concern is that other users and distro maintainers will also have confusion about what value to set and whether the warnings are important, since the help text does not say anything about Android, and "make oldconfig" does not indicate that the default value is different for Android.
<snip>
kernel/rcu/Kconfig.debug +2 -1
@@ -94,7 +94,8 @@ config RCU_EXP_CPU_STALL_TIMEOUT If the RCU grace period persists, additional CPU stall warnings are printed at more widely spaced intervals. A value of zero says to use the RCU_CPU_STALL_TIMEOUT value converted from - seconds to milliseconds. + seconds to milliseconds. If CONFIG_ANDROID is set for non-Android + platform and you unsure, set the RCU_EXP_CPU_STALL_TIMEOUT to zero. config RCU_TRACE bool "Enable tracing for RCU"
<snip>
Will it work for you?
--
Uladzislau Rezki
Re: CONFIG_ANDROID (was: rcu_sched detected expedited stalls in amdgpu after suspend)
Hi guys,
Am 28.06.22 um 22:11 schrieb Uladzislau Rezki:
Quoted 7 lines
Quoted 7 lines
Excerpts from Paul E. McKenney's message of June 28, 2022 2:54 pm:
Quoted 6 lines
All you need to do to get the previous behavior is to add something like this to your defconfig file:
CONFIG_RCU_EXP_CPU_STALL_TIMEOUT=21000Any reason why this will not work for you?
sorry for jumping in so later, I was on vacation for a week.
Well when any RCU period is longer than 20ms and amdgpu in the backtrace my educated guess is that we messed up some timeout waiting for the hw.
We usually do wait a few us, but it can be that somebody is waiting for ms instead.
So there are some todos here as far as I can see and It would be helpful to get a cleaner backtrace if possible.
Regards,
Christian.
Uladzislau Rezki
maintainerRe: CONFIG_ANDROID (was: rcu_sched detected expedited stalls in amdgpu after suspend)
Hello.
Quoted 13 lines · Christian König
Hi guys,
Am 28.06.22 um 22:11 schrieb Uladzislau Rezki:
Quoted 7 lines
Quoted 7 lines
Excerpts from Paul E. McKenney's message of June 28, 2022 2:54 pm:
Quoted 6 lines
All you need to do to get the previous behavior is to add something like this to your defconfig file:
CONFIG_RCU_EXP_CPU_STALL_TIMEOUT=21000Any reason why this will not work for you?
sorry for jumping in so later, I was on vacation for a week.
Quoted 9 lines
Well when any RCU period is longer than 20ms and amdgpu in the backtrace my educated guess is that we messed up some timeout waiting for the hw.
We usually do wait a few us, but it can be that somebody is waiting for ms instead.
So there are some todos here as far as I can see and It would be helpful to get a cleaner backtrace if possible.
Actually CONFIG_ANDROID looks like is going to be removed, so the CONFIG_RCU_EXP_CPU_STALL_TIMEOUT will not have any dependencies on the CONFIG_ANDROID anymore:
https://lkml.org/lkml/2022/6/29/756
--
Uladzislau Rezki
Paul E. McKenney
maintainerRe: CONFIG_ANDROID (was: rcu_sched detected expedited stalls in amdgpu after suspend)
Quoted 22 lines · Uladzislau Rezki
Hello.
Quoted 19 lines · Christian König
Hi guys,
Am 28.06.22 um 22:11 schrieb Uladzislau Rezki:
Quoted 7 lines
Quoted 7 lines
Excerpts from Paul E. McKenney's message of June 28, 2022 2:54 pm:
Quoted 6 lines
All you need to do to get the previous behavior is to add something like this to your defconfig file:
CONFIG_RCU_EXP_CPU_STALL_TIMEOUT=21000Any reason why this will not work for you?
sorry for jumping in so later, I was on vacation for a week.
Well when any RCU period is longer than 20ms and amdgpu in the backtrace my educated guess is that we messed up some timeout waiting for the hw.
We usually do wait a few us, but it can be that somebody is waiting for ms instead.
Quoted 7 lines
Quoted 3 lines
So there are some todos here as far as I can see and It would be helpful to get a cleaner backtrace if possible.
Actually
CONFIG_ANDROIDlooks like is going to be removed, so theCONFIG_RCU_EXP_CPU_STALL_TIMEOUTwill not have any dependencies on theCONFIG_ANDROIDanymore:
But you can set the RCU_EXP_CPU_STALL_TIMEOUT Kconfig option, if you wish. Setting this option to 20 will get you the behavior previously obtained by setting the now-defunct ANDROID Kconfig option.
Thanx, Paul
Uladzislau Rezki
maintainerRe: CONFIG_ANDROID (was: rcu_sched detected expedited stalls in amdgpu after suspend)
Quoted 26 lines · Paul E. McKenney
Quoted 25 lines · Uladzislau Rezki
Hello.
Quoted 22 lines · Christian König
Hi guys,
Am 28.06.22 um 22:11 schrieb Uladzislau Rezki:
Quoted 7 lines
Quoted 7 lines
Excerpts from Paul E. McKenney's message of June 28, 2022 2:54 pm:
Quoted 6 lines
All you need to do to get the previous behavior is to add something like this to your defconfig file:
CONFIG_RCU_EXP_CPU_STALL_TIMEOUT=21000Any reason why this will not work for you?
sorry for jumping in so later, I was on vacation for a week.
Well when any RCU period is longer than 20ms and amdgpu in the backtrace my educated guess is that we messed up some timeout waiting for the hw.
We usually do wait a few us, but it can be that somebody is waiting for ms instead.
So there are some todos here as far as I can see and It would be helpful to get a cleaner backtrace if possible.
Quoted 9 lines
Quoted 4 lines
Actually
CONFIG_ANDROIDlooks like is going to be removed, so theCONFIG_RCU_EXP_CPU_STALL_TIMEOUTwill not have any dependencies on theCONFIG_ANDROIDanymore:But you can set the
RCU_EXP_CPU_STALL_TIMEOUTKconfigoption, if you wish. Setting this option to 20 will get you the behavior previously obtained by setting the now-defunct ANDROIDKconfigoption.
Right. Or over boot parameter. So for us it is not a big issue :)
--
Uladzislau Rezki
Paul E. McKenney
maintainerRe: CONFIG_ANDROID (was: rcu_sched detected expedited stalls in amdgpu after suspend)
Quoted 30 lines · Uladzislau Rezki
Quoted 29 lines · Paul E. McKenney
Quoted 28 lines · Uladzislau Rezki
Hello.
Quoted 13 lines · Christian König
Hi guys,
Am 28.06.22 um 22:11 schrieb Uladzislau Rezki:
Quoted 7 lines
Quoted 7 lines
Excerpts from Paul E. McKenney's message of June 28, 2022 2:54 pm:
Quoted 6 lines
All you need to do to get the previous behavior is to add something like this to your defconfig file:
CONFIG_RCU_EXP_CPU_STALL_TIMEOUT=21000Any reason why this will not work for you?
sorry for jumping in so later, I was on vacation for a week.
Quoted 9 lines
Well when any RCU period is longer than 20ms and amdgpu in the backtrace my educated guess is that we messed up some timeout waiting for the hw.
We usually do wait a few us, but it can be that somebody is waiting for ms instead.
So there are some todos here as far as I can see and It would be helpful to get a cleaner backtrace if possible.
Actually
CONFIG_ANDROIDlooks like is going to be removed, so theCONFIG_RCU_EXP_CPU_STALL_TIMEOUTwill not have any dependencies on theCONFIG_ANDROIDanymore:
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
But you can set the
RCU_EXP_CPU_STALL_TIMEOUTKconfigoption, if you wish. Setting this option to 20 will get you the behavior previously obtained by setting the now-defunct ANDROIDKconfigoption.Right. Or over boot parameter. So for us it is not a big issue :)
Specifically rcupdate.rcu_exp_cpu_stall_timeout, for those just now tuning in. ;-)
Thanx, Paul
Re: CONFIG_ANDROID (was: rcu_sched detected expedited stalls in amdgpu after suspend)
Am 06.07.22 um 22:42 schrieb Paul E. McKenney:
Quoted 30 lines · Uladzislau Rezki
Quoted 29 lines · Paul E. McKenney
Quoted 28 lines · Uladzislau Rezki
Quoted 27 lines
Hello.
Quoted 12 lines · Christian König
Hi guys,
Am 28.06.22 um 22:11 schrieb Uladzislau Rezki:
Quoted 7 lines
Quoted 7 lines
Excerpts from Paul E. McKenney's message of June 28, 2022 2:54 pm:
Quoted 6 lines
All you need to do to get the previous behavior is to add something like this to your defconfig file:
CONFIG_RCU_EXP_CPU_STALL_TIMEOUT=21000Any reason why this will not work for you?
sorry for jumping in so later, I was on vacation for a week.
Quoted 9 lines
Well when any RCU period is longer than 20ms and amdgpu in the backtrace my educated guess is that we messed up some timeout waiting for the hw.
We usually do wait a few us, but it can be that somebody is waiting for ms instead.
So there are some todos here as far as I can see and It would be helpful to get a cleaner backtrace if possible.
Actually
CONFIG_ANDROIDlooks like is going to be removed, so theCONFIG_RCU_EXP_CPU_STALL_TIMEOUTwill not have any dependencies on theCONFIG_ANDROIDanymore:
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
But you can set the
RCU_EXP_CPU_STALL_TIMEOUTKconfigoption, if you wish. Setting this option to 20 will get you the behavior previously obtained by setting the now-defunct ANDROIDKconfigoption.Right. Or over boot parameter. So for us it is not a big issue :)
Specifically rcupdate.rcu_exp_cpu_stall_timeout, for those just now tuning in. ;-)
I was just about to write a response asking for that :)
Thanks, I will suggest to our QA to add this parameter while doing some tests.
Regards,
Christian.
Quoted 2 lines
Thanx, Paul
Paul E. McKenney
maintainerRe: CONFIG_ANDROID (was: rcu_sched detected expedited stalls in amdgpu after suspend)
Quoted 36 lines · Christian König
Am 06.07.22 um 22:42 schrieb Paul E. McKenney:
Quoted 35 lines · Uladzislau Rezki
Quoted 34 lines · Paul E. McKenney
Quoted 33 lines · Uladzislau Rezki
Quoted 21 lines
Hello.
Quoted 18 lines · Christian König
Hi guys,
Am 28.06.22 um 22:11 schrieb Uladzislau Rezki:
Quoted 7 lines
Quoted 7 lines
Excerpts from Paul E. McKenney's message of June 28, 2022 2:54 pm:
Quoted 6 lines
All you need to do to get the previous behavior is to add something like this to your defconfig file:
CONFIG_RCU_EXP_CPU_STALL_TIMEOUT=21000Any reason why this will not work for you?
sorry for jumping in so later, I was on vacation for a week.
Well when any RCU period is longer than 20ms and amdgpu in the backtrace my educated guess is that we messed up some timeout waiting for the hw.
We usually do wait a few us, but it can be that somebody is waiting for ms instead.
Quoted 7 lines
Quoted 3 lines
So there are some todos here as far as I can see and It would be helpful to get a cleaner backtrace if possible.
Actually
CONFIG_ANDROIDlooks like is going to be removed, so theCONFIG_RCU_EXP_CPU_STALL_TIMEOUTwill not have any dependencies on theCONFIG_ANDROIDanymore:But you can set the
RCU_EXP_CPU_STALL_TIMEOUTKconfigoption, if you wish. Setting this option to 20 will get you the behavior previously obtained by setting the now-defunct ANDROIDKconfigoption.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Right. Or over boot parameter. So for us it is not a big issue :)
Specifically rcupdate.rcu_exp_cpu_stall_timeout, for those just now tuning in. ;-)
I was just about to write a response asking for that :)
Thanks, I will suggest to our QA to add this parameter while doing some tests.
Very good! Please let me know how it goes.
Thanx, Paul