perf/x86/intel/uncore: Querying number of CHAs from CAPID6 register
Subsystems (2)
PATCH v2 1/1 perf/x86/intel/uncore: Querying number of CHAs from CAPID6 register
From: Kan Liang <[email protected]>
The number of CHAs is miscalculated on multi PCI domain systems on Skylake server.
(From Kroening, Gary:
For systems with a single PCI segment, it is sufficient to look for the bus number to change in order to determine that all of the CHa's have been counted for a single socket.
However, for multi PCI segment systems, each socket is given a new segment and the bus number does NOT change. So looking only for the bus number to change ends up counting all of the CHa's on all sockets in the system. This leads to writing CPU MSRs beyond a valid range and causes an error in ivbep_uncore_msr_init_box().)
To determine the number of CHAs, it should read bits 27:0 in the CAPID6 register located at Device 30, Function 3, Offset 0x9C. These 28 bits form a bit vector of available LLC slices and the CHAs that manage those slices.
support")
---
Changes since V1:
- add missed pci_dev_put()
- Drop ugly casting by using hweight32()
- Add comments for macros.
arch/x86/events/intel/uncore_snbep.c | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-)
arch/x86/events/intel/uncore_snbep.c +17 -14
@@ -3562,24 +3562,27 @@ static struct intel_uncore_type *skx_msr_uncores[] = { NULL, }; +/* +* To determine the number of CHAs, it should read bits 27:0 in the CAPID6 +* register which located at Device 30, Function 3, Offset 0x9C. PCI ID 0x2083. +*/ +#define SKX_CAPID6 0x9c +#define SKX_CHA_BIT_MASK GENMASK(27, 0) +static int skx_count_chabox (void ) { -struct pci_dev *chabox_dev =NULL ; -int bus, count =0 ; +struct pci_dev *dev =NULL ; +u32 val =0 ; -while (1 ) { - chabox_dev =pci_get_device (PCI_VENDOR_ID_INTEL ,0x208d , chabox_dev); -if (!chabox_dev) -break ; -if (count ==0 ) - bus = chabox_dev->bus->number; -if (bus != chabox_dev->bus->number) -break ; - count++; - } + dev =pci_get_device (PCI_VENDOR_ID_INTEL ,0x2083 , dev); +if (!dev) + goto out; -pci_dev_put (chabox_dev); -return count; +pci_read_config_dword (dev,SKX_CAPID6 , &val); + val &=SKX_CHA_BIT_MASK ; +out: +pci_dev_put (dev); +return hweight32 (val); }void skx_uncore_cpu_init (void )
Signature · 1 line
2.7.4
I've tested this patch on the same set of hubless (single-segment) and scalable (segment-per-socket) configurations as for Kan's version 1.
As far as we can tell this will also work for Cascade Lake, but will need revisiting for Ice Lake.
Thanks.
Gary
Quoted 98 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Tuesday, March 13, 2018 1:52 PM
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: Kroening, Gary; Travis, Mike; Banman, Andrew; Sivanich, Dimitri; Anderson, Russ; [email protected]; [email protected]; Kan Liang
Subject: [PATCH V2] perf/x86/intel/uncore: Querying number of CHAs from CAPID6 registerFrom: Kan Liang <[email protected]>
The number of CHAs is miscalculated on multi PCI domain systems on Skylake server.
(From Kroening, Gary:
For systems with a single PCI segment, it is sufficient to look for the bus number to change in order to determine that all of the CHa's have been counted for a single socket.
However, for multi PCI segment systems, each socket is given a new segment and the bus number does NOT change. So looking only for the bus number to change ends up counting all of the CHa's on all sockets in the system. This leads to writing CPU MSRs beyond a valid range and causes an error inivbep_uncore_msr_init_box().)To determine the number of CHAs, it should read bits 27:0 in the CAPID6 register located at Device 30, Function 3, Offset
0x9C. These 28 bits form a bit vector of available LLC slices and the CHAs that manage those slices.Fixes cd34cd97b7b4 ("perf/x86/intel/uncore: Add Skylake server uncoresupport")
Reported-by Kroening, Gary <[email protected]>Signed-off-by Kan Liang <[email protected]>---
Changes since V1:
- add missedpci_dev_put()
- Drop ugly casting by usinghweight32()
- Add comments for macros.arch/x86/events/intel/uncore_snbep.c | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-)arch/x86/events/intel/uncore_snbep.c +2
-0@@ -3562,24 +3562,27 @@ static struct intel_uncore_type *skx_msr_uncores[] = {NULL , }; +/* +* To determine the number of CHAs, it should read bits 27:0 in the CAPID6
+ * register which located at Device 30, Function 3, Offset0x9C. PCI ID0x2083.
+ */arch/x86/events/intel/uncore_snbep.c +5
-4+#define SKX_CAPID6 0x9c +#define SKX_CHA_BIT_MASK GENMASK(27, 0) +static int skx_count_chabox (void ) { -struct pci_dev *chabox_dev =NULL ; -int bus, count =0 ; +struct pci_dev *dev =NULL ; +u32 val =0 ; -while (1 ) { - chabox_dev = pci_get_device(PCI_VENDOR_ID_INTEL, 0x208d,
chabox_dev);arch/x86/events/intel/uncore_snbep.c +8
-11-if (!chabox_dev) -break ; -if (count ==0 ) - bus = chabox_dev->bus->number; -if (bus != chabox_dev->bus->number) -break ; - count++; - } + dev =pci_get_device (PCI_VENDOR_ID_INTEL ,0x2083 , dev); +if (!dev) + goto out; -pci_dev_put (chabox_dev); -return count; +pci_read_config_dword (dev,SKX_CAPID6 , &val); + val &=SKX_CHA_BIT_MASK ; +out: +pci_dev_put (dev); +return hweight32 (val); }void skx_uncore_cpu_init (void ) --2.7.4
Quoted 3 lines · Gary
I've tested this patch on the same set of hubless (single-segment) and scalable (segment-per-socket) configurations as for Kan's version 1.
As far as we can tell this will also work for Cascade Lake, but will need revisiting for Ice Lake.
For Ice lake, there will be another method to query the number of CHAs. This patch is only for Skylake server.
Thanks,
Kan
Quoted 103 lines
Thanks.
GaryQuoted 98 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Tuesday, March 13, 2018 1:52 PM
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: Kroening, Gary; Travis, Mike; Banman, Andrew; Sivanich, Dimitri; Anderson, Russ; [email protected]; [email protected]; Kan Liang
Subject: [PATCH V2] perf/x86/intel/uncore: Querying number of CHAs from CAPID6 registerFrom: Kan Liang <[email protected]>
The number of CHAs is miscalculated on multi PCI domain systems on Skylake server.
(From Kroening, Gary:
For systems with a single PCI segment, it is sufficient to look for the bus number to change in order to determine that all of the CHa's have been counted for a single socket.
However, for multi PCI segment systems, each socket is given a new segment and the bus number does NOT change. So looking only for the bus number to change ends up counting all of the CHa's on all sockets in the system. This leads to writing CPU MSRs beyond a valid range and causes an error inivbep_uncore_msr_init_box().)To determine the number of CHAs, it should read bits 27:0 in the CAPID6 register located at Device 30, Function 3, Offset
0x9C. These 28 bits form a bit vector of available LLC slices and the CHAs that manage those slices.Fixes cd34cd97b7b4 ("perf/x86/intel/uncore: Add Skylake server uncoresupport")
Reported-by Kroening, Gary <[email protected]>Signed-off-by Kan Liang <[email protected]>---
Changes since V1:
- add missedpci_dev_put()
- Drop ugly casting by usinghweight32()
- Add comments for macros.arch/x86/events/intel/uncore_snbep.c | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-)arch/x86/events/intel/uncore_snbep.c +2
-0@@ -3562,24 +3562,27 @@ static struct intel_uncore_type *skx_msr_uncores[] = {NULL , }; +/* +* To determine the number of CHAs, it should read bits 27:0 in the CAPID6
+ * register which located at Device 30, Function 3, Offset0x9C. PCI ID0x2083.
+ */arch/x86/events/intel/uncore_snbep.c +5
-4+#define SKX_CAPID6 0x9c +#define SKX_CHA_BIT_MASK GENMASK(27, 0) +static int skx_count_chabox (void ) { -struct pci_dev *chabox_dev =NULL ; -int bus, count =0 ; +struct pci_dev *dev =NULL ; +u32 val =0 ; -while (1 ) { - chabox_dev = pci_get_device(PCI_VENDOR_ID_INTEL, 0x208d,
chabox_dev);arch/x86/events/intel/uncore_snbep.c +8
-11-if (!chabox_dev) -break ; -if (count ==0 ) - bus = chabox_dev->bus->number; -if (bus != chabox_dev->bus->number) -break ; - count++; - } + dev =pci_get_device (PCI_VENDOR_ID_INTEL ,0x2083 , dev); +if (!dev) + goto out; -pci_dev_put (chabox_dev); -return count; +pci_read_config_dword (dev,SKX_CAPID6 , &val); + val &=SKX_CHA_BIT_MASK ; +out: +pci_dev_put (dev); +return hweight32 (val); }void skx_uncore_cpu_init (void ) --2.7.4
Quoted 16 lines · On Tue, Mar 13, 2018 at 8:51 PM, <[email protected]> wrote:
From: Kan Liang <[email protected]>
The number of CHAs is miscalculated on multi PCI domain systems on Skylake server.
(From Kroening, Gary:
For systems with a single PCI segment, it is sufficient to look for the bus number to change in order to determine that all of the CHa's have been counted for a single socket.
However, for multi PCI segment systems, each socket is given a new segment and the bus number does NOT change. So looking only for the bus number to change ends up counting all of the CHa's on all sockets in the system. This leads to writing CPU MSRs beyond a valid range and causes an error inivbep_uncore_msr_init_box().)
Quoted 5 lines
To determine the number of CHAs, it should read bits 27:0 in the CAPID6 register located at Device 30, Function 3, Offset
0x9C. These 28 bits form a bit vector of available LLC slices and the CHAs that manage those slices.
FWIW,
Quoted 63 lines
Fixes cd34cd97b7b4 ("perf/x86/intel/uncore: Add Skylake server uncoresupport")
Reported-by Kroening, Gary <[email protected]>Signed-off-by Kan Liang <[email protected]>---
Changes since V1:
- add missedpci_dev_put()
- Drop ugly casting by usinghweight32()
- Add comments for macros.arch/x86/events/intel/uncore_snbep.c | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-)arch/x86/events/intel/uncore_snbep.c +17
-15@@ -3562,24 +3562,27 @@ static struct intel_uncore_type *skx_msr_uncores[] = { NULL, }; +/* +* To determine the number of CHAs, it should read bits 27:0 in the CAPID6 +* register which located at Device 30, Function 3, Offset 0x9C. PCI ID 0x2083. +*/ +#define SKX_CAPID6 0x9c +#define SKX_CHA_BIT_MASK GENMASK(27, 0) +static int skx_count_chabox (void ) { -struct pci_dev *chabox_dev =NULL ; -int bus, count =0 ; +struct pci_dev *dev =NULL ; +u32 val =0 ; -while (1 ) { - chabox_dev =pci_get_device (PCI_VENDOR_ID_INTEL ,0x208d , chabox_dev); -if (!chabox_dev) -break ; -if (count ==0 ) - bus = chabox_dev->bus->number; -if (bus != chabox_dev->bus->number) -break ; - count++; - } + dev =pci_get_device (PCI_VENDOR_ID_INTEL ,0x2083 , dev); +if (!dev) + goto out; -pci_dev_put (chabox_dev); -return count; +pci_read_config_dword (dev,SKX_CAPID6 , &val); + val &=SKX_CHA_BIT_MASK ; +out: +pci_dev_put (dev); +return hweight32 (val); }void skx_uncore_cpu_init (void ) --2.7.4
Signature · 2 lines
With Best Regards, Andy Shevchenko
tip-bot for Kan Liang
merge bot[tip:perf/urgent] perf/x86/intel/uncore: Fix multi-domain PCI CHA enumeration bug on Skylake servers
In reply to [email protected]perf/x86/intel/uncore: Fix multi-domain PCI CHA enumeration bug on Skylake servers
Commit-ID: 320b0651f32b830add6497fcdcfdcb6ae8c7b8a0 Gitweb: https://git.kernel.org/tip/320b0651f32b830add6497fcdcfdcb6ae8c7b8a0 Author: Kan Liang <[email protected]> AuthorDate: Tue, 13 Mar 2018 11:51:34 -0700 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 20 Mar 2018 08:58:47 +0100
perf/x86/intel/uncore: Fix multi-domain PCI CHA enumeration bug on Skylake servers
The number of CHAs is miscalculated on multi-domain PCI Skylake server systems, resulting in an uncore driver initialization error.
Gary Kroening explains:
"For systems with a single PCI segment, it is sufficient to look for the
bus number to change in order to determine that all of the CHa's have been counted for a single socket.
However, for multi PCI segment systems, each socket is given a new segment and the bus number does NOT change. So looking only for the bus number to change ends up counting all of the CHa's on all sockets in the system. This leads to writing CPU MSRs beyond a valid range and causes an error in ivbep_uncore_msr_init_box()."
To fix this bug, query the number of CHAs from the CAPID6 register:
it should read bits 27:0 in the CAPID6 register located at Device 30, Function 3, Offset 0x9C. These 28 bits form a bit vector of available LLC slices and the CHAs that manage those slices.
--- arch/x86/events/intel/uncore_snbep.c | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-)
arch/x86/events/intel/uncore_snbep.c +17 -14
@@ -3563,24 +3563,27 @@ static struct intel_uncore_type *skx_msr_uncores[] = { NULL, }; +/* +* To determine the number of CHAs, it should read bits 27:0 in the CAPID6 +* register which located at Device 30, Function 3, Offset 0x9C. PCI ID 0x2083. +*/ +#define SKX_CAPID6 0x9c +#define SKX_CHA_BIT_MASK GENMASK(27, 0) +static int skx_count_chabox (void ) { -struct pci_dev *chabox_dev =NULL ; -int bus, count =0 ; +struct pci_dev *dev =NULL ; +u32 val =0 ; -while (1 ) { - chabox_dev =pci_get_device (PCI_VENDOR_ID_INTEL ,0x208d , chabox_dev); -if (!chabox_dev) -break ; -if (count ==0 ) - bus = chabox_dev->bus->number; -if (bus != chabox_dev->bus->number) -break ; - count++; - } + dev =pci_get_device (PCI_VENDOR_ID_INTEL ,0x2083 , dev); +if (!dev) + goto out; -pci_dev_put (chabox_dev); -return count; +pci_read_config_dword (dev,SKX_CAPID6 , &val); + val &=SKX_CHA_BIT_MASK ; +out: +pci_dev_put (dev); +return hweight32 (val); }void skx_uncore_cpu_init (void )