x86/resctrl: fix an imbalance in domain_remove_cpu
Subsystems (2)
PATCH 1/1 x86/resctrl: fix an imbalance in domain_remove_cpu
domain_add_cpu() calls domain_setup_mon_state() only when r->mon_capable is true where it will initialize d->mbm_over. However, domain_remove_cpu() calls cancel_delayed_work(&d->mbm_over) without checking r->mon_capable. Hence, it triggers a debugobjects warning when offlining CPUs because those timer debugobjects are never initialized.
ODEBUG: assert_init not available (active state 0) object type:timer_list hint: 0x0
WARNING: CPU: 143 PID: 789 at lib/debugobjects.c:484 debug_print_object+0xfe/0x140 Hardware name: HP Synergy 680 Gen9/Synergy 680 Gen9 Compute Module, BIOS I40 05/23/2018 RIP: 0010:debug_print_object+0xfe/0x140 Call Trace: debug_object_assert_init+0x1f5/0x240 del_timer+0x6f/0xf0 try_to_grab_pending+0x42/0x3c0 cancel_delayed_work+0x7d/0x150 resctrl_offline_cpu+0x3c0/0x520 cpuhp_invoke_callback+0x197/0x1120 cpuhp_thread_fun+0x252/0x2f0 smpboot_thread_fn+0x255/0x440 kthread+0x1e6/0x210 ret_from_fork+0x3a/0x50
--- arch/x86/kernel/cpu/resctrl/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/core.c +1 -1
@@ -618,7 +618,7 @@ static void domain_remove_cpu(int cpu, struct rdt_resource *r)if (static_branch_unlikely (&rdt_mon_enable_key))rmdir_mondata_subdir_allrdtgrp (r, d->id);list_del (&d->list); -if (is_mbm_enabled ()) +if (r->mon_capable &&is_mbm_enabled ())cancel_delayed_work (&d->mbm_over);if (is_llc_occupancy_enabled () &&has_busy_rmid (r, d)) {/*
Signature · 1 line
2.21.0 (Apple Git-122.2)
Hi Qian,
Quoted 7 lines · Qian Cai
domain_add_cpu()callsdomain_setup_mon_state()only whenr->mon_capableis true where it will initialized->mbm_over. However,domain_remove_cpu()callscancel_delayed_work(&d->mbm_over)without checkingr->mon_capable. Hence, it triggers a debugobjects warning when offlining CPUs because those timer debugobjects are never initialized.
Could you elaborate a little more on the failure symptom?
If I understand correctly, the error you described was due to r->mon_capable set to false while is_mbm_enabled() returns true? Which means on this platform rdt_mon_features is non zero?
And in get_rdt_mon_resources() it will invoke rdt_get_mon_l3_config(), however the only possible failure to do not set r->mon_capable is that it
failed in dom_data_init() due to kcalloc() failure? Then the logic inget_rdt_resources() is that it will ignore the return error if rdt allocate feature is supported on this platform? If this is the case, the r->mon_capable is not an indicator for whether the overflow thread has been created, right? Can we simply remove the check of r->mon_capable in domain_add_cpu() and invoke domain_setup_mon_state() directly?
Quoted 30 lines
ODEBUG:
assert_initnot available (active state 0) object type:timer_listhint:0x0WARNING: CPU: 143 PID: 789 at lib/debugobjects.c:484 debug_print_object+0xfe/0x140 Hardware name: HP Synergy 680 Gen9/Synergy 680 Gen9 Compute Module, BIOS I40 05/23/2018 RIP: 0010:debug_print_object+0xfe/0x140 Call Trace: debug_object_assert_init+0x1f5/0x240 del_timer+0x6f/0xf0 try_to_grab_pending+0x42/0x3c0 cancel_delayed_work+0x7d/0x150 resctrl_offline_cpu+0x3c0/0x520 cpuhp_invoke_callback+0x197/0x1120 cpuhp_thread_fun+0x252/0x2f0 smpboot_thread_fn+0x255/0x440 kthread+0x1e6/0x210 ret_from_fork+0x3a/0x50Fixes e33026831bdb ("x86/intel_rdt/mbm: Handle counter overflow")Signed-off-by Qian Cai <[email protected]>--- arch/x86/kernel/cpu/resctrl/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kernel/cpu/resctrl/core.c +0
-0
arch/x86/kernel/cpu/resctrl/core.c static void domain_remove_cpu(int cpu, struct rdt_resource *r) · line 622
Quoted 6 lines
arch/x86/kernel/cpu/resctrl/core.c +1
-1@@ -618,7 +618,7 @@ static void domain_remove_cpu(int cpu, struct rdt_resource *r)if (static_branch_unlikely (&rdt_mon_enable_key))rmdir_mondata_subdir_allrdtgrp (r, d->id);list_del (&d->list); -if (is_mbm_enabled ()) +if (r->mon_capable &&is_mbm_enabled ())cancel_delayed_work (&d->mbm_over);
Humm, it looks like there are two places within this function invoked cancel_delayed_work(&d->mbm_over),
why not adding the check for both of them?
thanks,
Y
Quoted 7 lines · Ryan Chen
Could you elaborate a little more on the failure symptom? If I understand correctly, the error you described was due to
r->mon_capableset to false whileis_mbm_enabled()returns true?
Yes.
Quoted 1 line
Which means on this platform
rdt_mon_featuresis non zero?
No idea. I did add some debug code that indicated resctrl_online_cpu() found 3 resources in for_each_capable_rdt_resource(r). Only the first one set r->mon_capable.
Quoted 3 lines
And in
get_rdt_mon_resources()it will invokerdt_get_mon_l3_config(), however the only possible failure to do not setr->mon_capableis that it failed indom_data_init()due tokcalloc()failure? Then the logic in
Very likely. Should be easy to confirm.
Quoted 2 lines
get_rdt_resources()is that it will ignore the return error if rdt allocate feature is supported on this platform? If
Yes.
Quoted 2 lines
this is the case, the
r->mon_capable
is not an indicator for whether the overflow thread has been created, right?
I am not sure about that.
Quoted 2 lines
Can we simply remove the check of
r->mon_capableindomain_add_cpu()and invokedomain_setup_mon_state()directly?
That should work too, but it is so perfect align with the r->alloc_capable check above, so I am not sure it is a good idea to break it.
Quoted 3 lines
Humm, it looks like there are two places within this function invoked
cancel_delayed_work(&d->mbm_over),
why not adding the check for both of them?
Because I am not sure about the second one. It was never executed due to “cpu != d->mbm_work_cpu“ even after offlining all CPUs except cpu 0 and never cause anything wrong yet, so I could not test it yet, but I can see why it might need a similar check too if d->mbm_work_cpu is non-zero and could trigger the same imbalance.
Quoted 14 lines · Ryan Chen
Hi Qian,
Quoted 7 lines · Qian Cai
domain_add_cpu()callsdomain_setup_mon_state()only whenr->mon_capableis true where it will initialized->mbm_over. However,domain_remove_cpu()callscancel_delayed_work(&d->mbm_over)without checkingr->mon_capable. Hence, it triggers a debugobjects warning when offlining CPUs because those timer debugobjects are never initialized.
Quoted 12 lines
Could you elaborate a little more on the failure symptom?
If I understand correctly, the error you described was due tor->mon_capableset to false whileis_mbm_enabled()returns true? Which means on this platformrdt_mon_featuresis non zero?
And inget_rdt_mon_resources()it will invokerdt_get_mon_l3_config(), however the only possible failure to do not setr->mon_capableis that it failed indom_data_init()due tokcalloc()failure? Then the logic inget_rdt_resources()is that it will ignore the return error if rdt allocate feature is supported on this platform? If this is the case, ther->mon_capableis not an indicator for whether the overflow thread has been created, right? Can we simply remove the check ofr->mon_capableindomain_add_cpu()and invokedomain_setup_mon_state()directly?
Actually,
domain_add_cpu r->name = L3, r->alloc_capable = 1, r->mon_capable = 1 domain_add_cpu r->name = L3DATA, r->alloc_capable = 1, r->mon_capable = 0 domain_add_cpu r->name = L3CODE, r->alloc_capable = 1, r->mon_capable = 0
rdt_get_mon_l3_config() will only set r->mon_capable = 1 for L3.
Quoted 25 lines
Quoted 25 lines
ODEBUG:
assert_initnot available (active state 0) object type:timer_listhint:0x0WARNING: CPU: 143 PID: 789 at lib/debugobjects.c:484 debug_print_object+0xfe/0x140 Hardware name: HP Synergy 680 Gen9/Synergy 680 Gen9 Compute Module, BIOS I40 05/23/2018 RIP: 0010:debug_print_object+0xfe/0x140 Call Trace: debug_object_assert_init+0x1f5/0x240 del_timer+0x6f/0xf0 try_to_grab_pending+0x42/0x3c0 cancel_delayed_work+0x7d/0x150 resctrl_offline_cpu+0x3c0/0x520 cpuhp_invoke_callback+0x197/0x1120 cpuhp_thread_fun+0x252/0x2f0 smpboot_thread_fn+0x255/0x440 kthread+0x1e6/0x210 ret_from_fork+0x3a/0x50Fixes e33026831bdb ("x86/intel_rdt/mbm: Handle counter overflow")Signed-off-by Qian Cai <[email protected]>---
arch/x86/kernel/cpu/resctrl/core.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 14 lines
arch/x86/kernel/cpu/resctrl/core.c static void domain_remove_cpu(int cpu, struct rdt_resource *r) · line 622
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/core.c +1
-1@@ -618,7 +618,7 @@ static void domain_remove_cpu(int cpu, struct rdt_resource *r)if (static_branch_unlikely (&rdt_mon_enable_key))rmdir_mondata_subdir_allrdtgrp (r, d->id);list_del (&d->list); -if (is_mbm_enabled ()) +if (r->mon_capable &&is_mbm_enabled ())cancel_delayed_work (&d->mbm_over);Humm, it looks like there are two places within this function invoked
cancel_delayed_work(&d->mbm_over),
why not adding the check for both of them?
Here it only check L3, so it will skip correctly for L3DATA and L3CODE to not call cancel_delayed_work(). Recalled the above that only L3 will have r->capable set.
if (r == &rdt_resources_all[RDT_RESOURCE_L3]) {
if (is_mbm_enabled() && cpu == d->mbm_work_cpu) {
cancel_delayed_work(&d->mbm_over);Hence, r->mon_capable check seems redundant here.
Reinette Chatre
maintainerHi Qian,
Quoted 77 lines · Qian Cai
Quoted 12 lines · Ryan Chen
Hi Qian,
Quoted 7 lines · Qian Cai
domain_add_cpu()callsdomain_setup_mon_state()only whenr->mon_capableis true where it will initialized->mbm_over. However,domain_remove_cpu()callscancel_delayed_work(&d->mbm_over)without checkingr->mon_capable. Hence, it triggers a debugobjects warning when offlining CPUs because those timer debugobjects are never initialized.Quoted 12 lines
Could you elaborate a little more on the failure symptom?
If I understand correctly, the error you described was due tor->mon_capableset to false whileis_mbm_enabled()returns true? Which means on this platformrdt_mon_featuresis non zero?
And inget_rdt_mon_resources()it will invokerdt_get_mon_l3_config(), however the only possible failure to do not setr->mon_capableis that it failed indom_data_init()due tokcalloc()failure? Then the logic inget_rdt_resources()is that it will ignore the return error if rdt allocate feature is supported on this platform? If this is the case, ther->mon_capableis not an indicator for whether the overflow thread has been created, right? Can we simply remove the check ofr->mon_capableindomain_add_cpu()and invokedomain_setup_mon_state()directly?Actually,
domain_add_cpur->name= L3,r->alloc_capable= 1,r->mon_capable= 1domain_add_cpur->name= L3DATA,r->alloc_capable= 1,r->mon_capable= 0domain_add_cpur->name= L3CODE,r->alloc_capable= 1,r->mon_capable= 0
rdt_get_mon_l3_config()will only setr->mon_capable= 1 for L3.Quoted 25 lines
Quoted 25 lines
ODEBUG:
assert_initnot available (active state 0) object type:timer_listhint:0x0WARNING: CPU: 143 PID: 789 at lib/debugobjects.c:484 debug_print_object+0xfe/0x140 Hardware name: HP Synergy 680 Gen9/Synergy 680 Gen9 Compute Module, BIOS I40 05/23/2018 RIP: 0010:debug_print_object+0xfe/0x140 Call Trace: debug_object_assert_init+0x1f5/0x240 del_timer+0x6f/0xf0 try_to_grab_pending+0x42/0x3c0 cancel_delayed_work+0x7d/0x150 resctrl_offline_cpu+0x3c0/0x520 cpuhp_invoke_callback+0x197/0x1120 cpuhp_thread_fun+0x252/0x2f0 smpboot_thread_fn+0x255/0x440 kthread+0x1e6/0x210 ret_from_fork+0x3a/0x50Fixes e33026831bdb ("x86/intel_rdt/mbm: Handle counter overflow")Signed-off-by Qian Cai <[email protected]>---
arch/x86/kernel/cpu/resctrl/core.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)Quoted 14 lines
arch/x86/kernel/cpu/resctrl/core.c static void domain_remove_cpu(int cpu, struct rdt_resource *r) · line 622
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/core.c +1
-1@@ -618,7 +618,7 @@ static void domain_remove_cpu(int cpu, struct rdt_resource *r)if (static_branch_unlikely (&rdt_mon_enable_key))rmdir_mondata_subdir_allrdtgrp (r, d->id);list_del (&d->list); -if (is_mbm_enabled ()) +if (r->mon_capable &&is_mbm_enabled ())cancel_delayed_work (&d->mbm_over);Humm, it looks like there are two places within this function invoked
cancel_delayed_work(&d->mbm_over),
why not adding the check for both of them?Here it only check L3, so it will skip correctly for L3DATA and L3CODE to not call
cancel_delayed_work(). Recalled the above that only L3 will haver->capableset.
Quoted 6 lines
if (r == &rdt_resources_all[RDT_RESOURCE_L3]) { if (is_mbm_enabled() && cpu == d->mbm_work_cpu) { cancel_delayed_work(&d->mbm_over);Hence,
r->mon_capablecheck seems redundant here.
Thank you very much for catching this. Your change looks like the right thing to do. As Ryan pointed out this is not obvious at first and looking back at this commit at a later time may benefit from some more details. For example, how about:
"A system that supports resource monitoring may have multiple resources while not all of these resources are capable of monitoring. Monitoring related state is initialized only for resources that are capable of monitoring and correspondingly this state should subsequently only be removed from these resources that are capable of monitoring.
domain_add_cpu() calls domain_setup_mon_state() only when r->mon_capable is true where it will initialize d->mbm_over. However, domain_remove_cpu() calls cancel_delayed_work(&d->mbm_over) without checking r->mon_capable resulting in an attempt to cancel d->mbm_over on all resources, even those that never initialized d->mbm_over because they are not capable of monitoring. Hence, it triggers a debugobjects warning when offlining CPUs because those timer debugobjects are never initialized.
ODEBUG:..."
Reinette
Quoted 20 lines · Reinette Chatre
"A system that supports resource monitoring may have multiple resources while not all of these resources are capable of monitoring. Monitoring related state is initialized only for resources that are capable of monitoring and correspondingly this state should subsequently only be removed from these resources that are capable of monitoring.
domain_add_cpu()callsdomain_setup_mon_state()only whenr->mon_capableis true where it will initialized->mbm_over. However,domain_remove_cpu()callscancel_delayed_work(&d->mbm_over)without checkingr->mon_capableresulting in an attempt to canceld->mbm_overon all resources, even those that never initializedd->mbm_overbecause they are not capable of monitoring. Hence, it triggers a debugobjects warning when offlining CPUs because those timer debugobjects are never initialized.
Quoted 1 line
ODEBUG:..."
Looks better to me. Do you want me to send a v2 for it or you could update it for merging?
Reinette Chatre
maintainerHi Qian,
Quoted 20 lines · Qian Cai
Quoted 20 lines · Reinette Chatre
"A system that supports resource monitoring may have multiple resources while not all of these resources are capable of monitoring. Monitoring related state is initialized only for resources that are capable of monitoring and correspondingly this state should subsequently only be removed from these resources that are capable of monitoring.
domain_add_cpu()callsdomain_setup_mon_state()only whenr->mon_capableis true where it will initialized->mbm_over. However,domain_remove_cpu()callscancel_delayed_work(&d->mbm_over)without checkingr->mon_capableresulting in an attempt to canceld->mbm_overon all resources, even those that never initializedd->mbm_overbecause they are not capable of monitoring. Hence, it triggers a debugobjects warning when offlining CPUs because those timer debugobjects are never initialized.
Quoted 4 lines
Quoted 1 line
ODEBUG:..."
Looks better to me. Do you want me to send a v2 for it or you could update it for merging?
Could you please send v2? I am not the one that provides final approval for inclusion nor the one that will take care of merging afterwards.
Thank you very much
Reinette
Quoted 11 lines · Qian Cai
Here it only check L3, so it will skip correctly for L3DATA and L3CODE to not call
cancel_delayed_work(). Recalled the above that only L3 will haver->capableset.if (r == &rdt_resources_all[RDT_RESOURCE_L3]) { if (is_mbm_enabled() && cpu == d->mbm_work_cpu) { cancel_delayed_work(&d->mbm_over);Hence,
r->mon_capablecheck seems redundant here.
I see. Thanks for explaining.
Signature · 2 lines
thanks, Ryan