x86/resctrl: avoid divide by 0 num_rmid
Subsystems (2)
PATCH 1/1 x86/resctrl: avoid divide by 0 num_rmid
x86_cache_max_rmid's default is -1. If the hardware or VM doesn't set the right cpuid bits, num_rmid can be 0.
---
I ran into this on a VM on granite rapids. I guess the VMM told the kernel it was a GNR, but didn't set all the cache/rsctl bits.
arch/x86/kernel/cpu/resctrl/monitor.c | 6 ++++++ 1 file changed, 6 insertions(+)
arch/x86/kernel/cpu/resctrl/monitor.c +6 -0
@@ -359,6 +359,12 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)else if (mbm_offset >MBM_CNTR_WIDTH_OFFSET_MAX )pr_warn ("Ignoring impossible MBM counter offset\n" ); +if (r->num_rmid <1 ) { +pr_warn ("Invalid num_rmid %d, cach_max_rmid was %d\n" , + r->num_rmid, boot_cpu_data.x86_cache_max_rmid); + r->num_rmid =1 ; + } +/* * A reasonable upper limit on the max threshold is the number * of lines tagged per RMID if all RMIDs have the same number of
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
Reinette Chatre
maintainerHi Barret,
Quoted 9 lines · Barret Rhoden
x86_cache_max_rmid's default is -1. If the hardware or VM doesn't set the right cpuid bits,
num_rmidcan be 0.Signed-off-by Barret Rhoden <[email protected]>---
I ran into this on a VM on granite rapids. I guess the VMM told the kernel it was a GNR, but didn't set all the cache/rsctl bits.
The -1 default of x86_cache_max_rmid is assigned if the hardware does not support *any* L3 monitoring. Specifically:
resctrl_cpu_detect():
if (!cpu_has(c, X86_FEATURE_CQM_LLC)) {
c->x86_cache_max_rmid = -1;
...
}The function modified by this patch, rdt_get_mon_l3_config() only runs if the hardware supports one or more of the L3 monitoring sub-features (X86_FEATURE_CQM_OCCUP_LLC, X86_FEATURE_CQM_MBM_TOTAL, or X86_FEATURE_CQM_MBM_LOCAL) that depend on X86_FEATURE_CQM_LLC per cpuid_deps[].
I tried to reproduce the issue on real hardware by using clearcpuid to disable X86_FEATURE_CQM_LLC and the CPUID dependencies did the right thing by automatically disabling X86_FEATURE_CQM_OCCUP_LLC, X86_FEATURE_CQM_MBM_TOTAL, X86_FEATURE_CQM_MBM_LOCAL, not running rdt_get_mon_l3_config() at all, and not even attempt to enumerate any of the L3 monitoring details.
What are the symptoms when you encounter this issue?
Would it be possible to send me the CPUID flags of leaf 7, subleaf 0 as well as all sub-leaves of leaf 0xF?
Could you please also elaborate what the impact of this issue is? Is this a VM that has been released with many users impacted or something encountered during development of this VM?
Quoted 8 lines
arch/x86/kernel/cpu/resctrl/monitor.c | 6 ++++++ 1 file changed, 6 insertions(+)arch/x86/kernel/cpu/resctrl/monitor.c +0
-0
arch/x86/kernel/cpu/resctrl/monitor.c int __init rdt_get_mon_l3_config(struct rdt_resource *r) · line 365
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/monitor.c +4
-0@@ -359,6 +359,12 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)else if (mbm_offset >MBM_CNTR_WIDTH_OFFSET_MAX )pr_warn ("Ignoring impossible MBM counter offset\n" ); +if (r->num_rmid <1 ) { +pr_warn ("Invalid num_rmid %d, cach_max_rmid was %d\n" , + r->num_rmid, boot_cpu_data.x86_cache_max_rmid); + r->num_rmid =1 ;
I do not think enumeration of this feature should proceed/succeed if there clearly is a configuration issue.
arch/x86/kernel/cpu/resctrl/monitor.c int __init rdt_get_mon_l3_config(struct rdt_resource *r) · line 370
Quoted 5 lines
arch/x86/kernel/cpu/resctrl/monitor.c +2
-0+ } +/* * A reasonable upper limit on the max threshold is the number * of lines tagged per RMID if all RMIDs have the same number of
Reinette
Quoted 27 lines · Reinette Chatre
Hi Barret,
Quoted 9 lines · Barret Rhoden
x86_cache_max_rmid's default is -1. If the hardware or VM doesn't set the right cpuid bits,
num_rmidcan be 0.Signed-off-by Barret Rhoden <[email protected]>---
I ran into this on a VM on granite rapids. I guess the VMM told the kernel it was a GNR, but didn't set all the cache/rsctl bits.The -1 default of
x86_cache_max_rmidis assigned if the hardware does not support *any* L3 monitoring. Specifically:
resctrl_cpu_detect():if (!cpu_has(c, X86_FEATURE_CQM_LLC)) { c->x86_cache_max_rmid = -1; ... }The function modified by this patch,
rdt_get_mon_l3_config()only runs if the hardware supports one or more of the L3 monitoring sub-features (X86_FEATURE_CQM_OCCUP_LLC,X86_FEATURE_CQM_MBM_TOTAL, orX86_FEATURE_CQM_MBM_LOCAL) that depend onX86_FEATURE_CQM_LLCper cpuid_deps[].
Quoted 7 lines
I tried to reproduce the issue on real hardware by using clearcpuid to disable
X86_FEATURE_CQM_LLCand the CPUID dependencies did the right thing by automatically disablingX86_FEATURE_CQM_OCCUP_LLC,X86_FEATURE_CQM_MBM_TOTAL,X86_FEATURE_CQM_MBM_LOCAL, not runningrdt_get_mon_l3_config()at all, and not even attempt to enumerate any of the L3 monitoring details.What are the symptoms when you encounter this issue?
Linux crashes during boot with a divide error, and the splat backtrace is in rdt_get_mon_l3_config().
Quoted 2 lines
Would it be possible to send me the CPUID flags of leaf 7, subleaf 0 as well as all sub-leaves of leaf
0xF?
# ./cpuid 0x7 0 CPUID for Leaf 0x00000007, Sublevel 0x00000000: eax: 00000002 ebx: f1bf2ffb ecx: 1b415f7e edx: bc814410
# ./cpuid 0x7 1 CPUID for Leaf 0x00000007, Sublevel 0x00000001: eax: 00201c30 ebx: 00000000 ecx: 00000000 edx: 00084000
# ./cpuid 0x7 2 CPUID for Leaf 0x00000007, Sublevel 0x00000002: eax: 00000000 ebx: 00000000 ecx: 00000000 edx: 0000003f
Quoted 3 lines
Could you please also elaborate what the impact of this issue is? Is this a VM that has been released with many users impacted or something encountered during development of this VM?
This is with cloud-hypervisor. We do have a couple of local patches for running on machines with more than 256 cpus. I didn't see anything in our changes related to cpuid 0x7, but maybe it's on our end.
But I imagine the problem isn't widespread and could be considered developmental.
I'll keep poking on my end - maybe I had some other cruft in my system (in the kernel build or in cloud_hypervisor).
Thanks,
Barret
Reinette Chatre
maintainerHi Barret,
Quoted 38 lines · Barret Rhoden
Quoted 27 lines · Reinette Chatre
Hi Barret,
Quoted 9 lines · Barret Rhoden
x86_cache_max_rmid's default is -1. If the hardware or VM doesn't set the right cpuid bits,
num_rmidcan be 0.Signed-off-by Barret Rhoden <[email protected]>---
I ran into this on a VM on granite rapids. I guess the VMM told the kernel it was a GNR, but didn't set all the cache/rsctl bits.The -1 default of
x86_cache_max_rmidis assigned if the hardware does not support *any* L3 monitoring. Specifically:
resctrl_cpu_detect():
if (!cpu_has(c,X86_FEATURE_CQM_LLC)) {
c->x86_cache_max_rmid = -1;
...
}The function modified by this patch,
rdt_get_mon_l3_config()only runs if the hardware supports one or more of the L3 monitoring sub-features (X86_FEATURE_CQM_OCCUP_LLC,X86_FEATURE_CQM_MBM_TOTAL, orX86_FEATURE_CQM_MBM_LOCAL) that depend onX86_FEATURE_CQM_LLCper cpuid_deps[].Quoted 7 lines
I tried to reproduce the issue on real hardware by using clearcpuid to disable
X86_FEATURE_CQM_LLCand the CPUID dependencies did the right thing by automatically disablingX86_FEATURE_CQM_OCCUP_LLC,X86_FEATURE_CQM_MBM_TOTAL,X86_FEATURE_CQM_MBM_LOCAL, not runningrdt_get_mon_l3_config()at all, and not even attempt to enumerate any of the L3 monitoring details.What are the symptoms when you encounter this issue?
Linux crashes during boot with a divide error, and the splat backtrace is in
rdt_get_mon_l3_config().
Quoted 7 lines
Quoted 2 lines
Would it be possible to send me the CPUID flags of leaf 7, subleaf 0 as well as all sub-leaves of leaf
0xF?# ./cpuid 0x7 0 CPUID for Leaf 0x00000007, Sublevel 0x00000000: eax: 00000002 ebx: f1bf2ffb
Looks like X86_FEATURE_CQM_LLC is off.
Quoted 10 lines
ecx:
1b415f7e
edx:bc814410# ./cpuid 0x7 1 CPUID for Leaf 0x00000007, Sublevel 0x00000001: eax: 00201c30 ebx: 00000000 ecx: 00000000 edx: 00084000
Quoted 7 lines
# ./cpuid 0x7 2 CPUID for Leaf 0x00000007, Sublevel 0x00000002: eax: 00000000 ebx: 00000000 ecx: 00000000 edx: 0000003f
Thank you. Could you please also pass on all sub-leaves of leaf 0xF?
Thank you very much.
Reinette
Reinette Chatre
maintainerHi Barret,
Quoted 67 lines · Reinette Chatre
Quoted 38 lines · Barret Rhoden
Quoted 27 lines · Reinette Chatre
Hi Barret,
Quoted 9 lines · Barret Rhoden
x86_cache_max_rmid's default is -1. If the hardware or VM doesn't set the right cpuid bits,
num_rmidcan be 0.Signed-off-by Barret Rhoden <[email protected]>---
I ran into this on a VM on granite rapids. I guess the VMM told the kernel it was a GNR, but didn't set all the cache/rsctl bits.The -1 default of
x86_cache_max_rmidis assigned if the hardware does not support *any* L3 monitoring. Specifically:
resctrl_cpu_detect():
if (!cpu_has(c,X86_FEATURE_CQM_LLC)) {
c->x86_cache_max_rmid = -1;
...
}The function modified by this patch,
rdt_get_mon_l3_config()only runs if the hardware supports one or more of the L3 monitoring sub-features (X86_FEATURE_CQM_OCCUP_LLC,X86_FEATURE_CQM_MBM_TOTAL, orX86_FEATURE_CQM_MBM_LOCAL) that depend onX86_FEATURE_CQM_LLCper cpuid_deps[].Quoted 7 lines
I tried to reproduce the issue on real hardware by using clearcpuid to disable
X86_FEATURE_CQM_LLCand the CPUID dependencies did the right thing by automatically disablingX86_FEATURE_CQM_OCCUP_LLC,X86_FEATURE_CQM_MBM_TOTAL,X86_FEATURE_CQM_MBM_LOCAL, not runningrdt_get_mon_l3_config()at all, and not even attempt to enumerate any of the L3 monitoring details.What are the symptoms when you encounter this issue?
Linux crashes during boot with a divide error, and the splat backtrace is in
rdt_get_mon_l3_config().Quoted 7 lines
Quoted 2 lines
Would it be possible to send me the CPUID flags of leaf 7, subleaf 0 as well as all sub-leaves of leaf
0xF?# ./cpuid 0x7 0 CPUID for Leaf 0x00000007, Sublevel 0x00000000: eax: 00000002 ebx: f1bf2ffbLooks like
X86_FEATURE_CQM_LLCis off.Quoted 10 lines
ecx:
1b415f7e
edx:bc814410# ./cpuid 0x7 1 CPUID for Leaf 0x00000007, Sublevel 0x00000001: eax: 00201c30 ebx: 00000000 ecx: 00000000 edx: 00084000Quoted 7 lines
# ./cpuid 0x7 2 CPUID for Leaf 0x00000007, Sublevel 0x00000002: eax: 00000000 ebx: 00000000 ecx: 00000000 edx: 0000003f
Quoted 2 lines
Thank you. Could you please also pass on all sub-leaves of leaf
0xF?
One more thing, could you please check the kernel logs for any other issues when that VM boots up?
I would like to check if it contains a warning with text like:
"x86 CPU feature dependency check failure"
Reinette