Skip to contentSkip to conversation

x86/resctrl: Work around MBM errata

PATCH Applied Fenghua Yu 3 participants 7 messages Oct 14, 2020 – Oct 27, 2020

Subsystems (3)
lore
Share & tools
lore

Fenghua Yu

author
#0 lore raw

v3 cover x86/resctrl: Work around MBM errata

Intel Memory Bandwidth Monitoring (MBM) counters may report system memory bandwidth incorrectly on some Intel processors. The errata are reported in erratum SKX99 [1], erratum BDF102 [2] and RDT reference manual [3].

The errata are worked around using a correction factor table. Since the correction factor table is not publicly documented anywhere, it's documented in Documentation/x86/resctrl.rst along with the errata.

1. Erratum SKX99 in Intel Xeon Processor Scalable Family Specification Update:
http://web.archive.org/web/20200716124958/https://www.intel.com/content/www/us/en/processors/xeon/scalable/xeon-scalable-spec-update.html
2. Erratum BDF102 in Intel Xeon E5-2600 v4 Processor Product Family Specification Update:
http://web.archive.org/web/20191125200531/https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e5-v4-spec-update.pdf
3. The errata in Intel Resource Director Technology (Intel RDT) on 2nd Generation Intel Xeon Scalable Processors Reference Manual:
https://software.intel.com/content/www/us/en/develop/articles/intel-resource-director-technology-rdt-reference-manual.html

Change Log:
v3:
- Remove unnecessary conf.py change in patch 1 (Randy).

v2:
- Document the errata and the correction table in resctrl.rst (Boris).
- Address various comments on patch 2 (Boris).
- Change the documentation URLs to stable archive.org (Tony).

Fenghua Yu (2):
Documentation: x86: Rename resctrl_ui.rst and add two errata to the
file
x86/resctrl: Correct MBM total and local values

 Documentation/x86/index.rst                   |  2 +-
 .../x86/{resctrl_ui.rst => resctrl.rst}       | 82 +++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/core.c            |  4 +
 arch/x86/kernel/cpu/resctrl/internal.h        |  1 +
 arch/x86/kernel/cpu/resctrl/monitor.c         | 82 ++++++++++++++++++-
 5 files changed, 168 insertions(+), 3 deletions(-)
 rename Documentation/x86/{resctrl_ui.rst => resctrl.rst} (92%)
Signature · 1 line
2.28.0
author
#1 lore raw

PATCH v3 1/2 Documentation: x86: Rename resctrl_ui.rst and add two errata to the file

Intel Memory Bandwidth Monitoring (MBM) counters may report system memory bandwidth incorrectly on some Intel processors. The errata are reported in erratum SKX99 [1], erratum BDF102 [2] and RDT reference manual [3].

To work around the errata, MBM total and local readings are corrected using a correction factor table.

Since the correction factor table is not publicly documented anywhere, the table and the errata are documented in Documentation/x86/resctrl.rst for future reference. The resctrl.rst file is renamed from Documentation/x86/resctrl_ui.rst because the file won't contain user interface only anymore.

1. Erratum SKX99 in Intel Xeon Processor Scalable Family Specification Update:
http://web.archive.org/web/20200716124958/https://www.intel.com/content/www/us/en/processors/xeon/scalable/xeon-scalable-spec-update.html
2. Erratum BDF102 in Intel Xeon E5-2600 v4 Processor Product Family Specification Update:
http://web.archive.org/web/20191125200531/https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e5-v4-spec-update.pdf
3. The errata in Intel Resource Director Technology (Intel RDT) on 2nd Generation Intel Xeon Scalable Processors Reference Manual:
https://software.intel.com/content/www/us/en/develop/articles/intel-resource-director-technology-rdt-reference-manual.html

---
Change Log:
v3:
- Remove unnecessary conf.py change in patch 1 (Randy).

v2:
- Document the correction factor table and errata in resctrl.rst (Boris).
- Change the documentation URLs to stable archive.org (Tony).

 Documentation/x86/index.rst                   |  2 +-
 .../x86/{resctrl_ui.rst => resctrl.rst}       | 82 +++++++++++++++++++
 2 files changed, 83 insertions(+), 1 deletion(-)
 rename Documentation/x86/{resctrl_ui.rst => resctrl.rst} (92%)
Documentation/x86/index.rst +1 -1
index 265d9e9a093b..49d2fd9f0e5b 100644
@@ -25,7 +25,7 @@ x86-specific Documentation
    pti
    mds
    microcode
-   resctrl_ui
+   resctrl
    tsx_async_abort
    usb-legacy-support
    i386/index
Documentation/x86/resctrl.rst renamed from Documentation/x86/resctrl_ui.rst +82 -0

85 lines not on this page · raw message

Signature · 1 line
2.28.0
merge bot Applied
#2 lore raw

[tip: x86/cache] Documentation/x86: Rename resctrl_ui.rst and add two errata to the file

The following commit has been merged into the x86/cache branch of tip:

The following commit has been merged into the x86/cache branch of tip:

Commit-ID:     d1b22e36e3188f47dbf8aaef54d9e79df8e91f4c
Gitweb:        https://git.kernel.org/tip/d1b22e36e3188f47dbf8aaef54d9e79df8e91f4c
Author:        Fenghua Yu <[email protected]>
AuthorDate:    Wed, 14 Oct 2020 00:49:26 
Committer:     Borislav Petkov <[email protected]>
CommitterDate: Tue, 27 Oct 2020 16:47:00 +01:00

Documentation/x86: Rename resctrl_ui.rst and add two errata to the file

Intel Memory Bandwidth Monitoring (MBM) counters may report system memory bandwidth incorrectly on some Intel processors. This is reported in documented in erratum SKX99, erratum BDF102 and in the RDT reference manual, see Documentation/x86/index.rst.

To work around the errata, MBM total and local readings are corrected using a correction factor table.

Since the correction factor table is not publicly documented anywhere, document the table and the errata in Documentation/x86/resctrl.rst for future reference.

[ bp: Move web links to the doc, massage. ]

---
 Documentation/x86/index.rst      |    2 +-
 Documentation/x86/resctrl.rst    | 1304 +++++++++++++++++++++++++++++-
 Documentation/x86/resctrl_ui.rst | 1211 +---------------------------
 3 files changed, 1305 insertions(+), 1212 deletions(-)
 create mode 100644 Documentation/x86/resctrl.rst
 delete mode 100644 Documentation/x86/resctrl_ui.rst
Documentation/x86/index.rst +1 -1
index b224d12..9c6ebf3 100644
@@ -27,7 +27,7 @@ x86-specific Documentation
    pti
    mds
    microcode
-   resctrl_ui
+   resctrl
    tsx_async_abort
    usb-legacy-support
    i386/index
Documentation/x86/resctrl.rst new file +1304 -0

1,304 lines not on this page · raw message

Documentation/x86/resctrl_ui.rst deleted +0 -1211

1,211 lines not on this page · raw message

author
#3 lore raw

PATCH v3 2/2 x86/resctrl: Correct MBM total and local values

In reply to Fenghua Yu

Intel Memory Bandwidth Monitoring (MBM) counters may report system memory bandwidth incorrectly on some Intel processors. The errata SKX99 for Skylake server, BDF102 for Broadwell server, and the correction factor table are documented in Documentation/x86/resctrl.rst.

Intel MBM counters track metrics according to the assigned Resource Monitor ID (RMID) for that logical core. The IA32_QM_CTR register (MSR 0xC8E), used to report these metrics, may report incorrect system bandwidth for certain RMID values.

Due to the errata, system memory bandwidth may not match what is reported.

To work around the errata, MBM total and local readings are corrected using the correction factor table. If rmid > rmid threshold, MBM total and local values should be multiplied by the correction factor.

---
Change Log:
v2:
- Add "const" for mbm_cf_table[] (Boris).
- Add verbs to corrected_mbm_counter() and intel_rdt_mbm_quirk() (Boris).
- Describe the errata in the commit message (Boris).
- Fix checkpatch.pl warning on "/* FALLTHROUGH */" (Boris).

 arch/x86/kernel/cpu/resctrl/core.c     |  4 ++
 arch/x86/kernel/cpu/resctrl/internal.h |  1 +
 arch/x86/kernel/cpu/resctrl/monitor.c  | 82 +++++++++++++++++++++++++-
 3 files changed, 85 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +4 -0
index 9e1712e8aef7..b3bdc477599e 100644
@@ -893,6 +893,10 @@ static __init void __check_quirks_intel(void)
 			set_rdt_options("!cmt,!mbmtotal,!mbmlocal,!l3cat");
 		else
 			set_rdt_options("!l3cat");
+		fallthrough;
+	case INTEL_FAM6_BROADWELL_X:
+		intel_rdt_mbm_apply_quirk();
+		break;
 	}
 }
 
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
index 80fa997fae60..6cb068fcf501 100644
@@ -616,6 +616,7 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 void mbm_setup_overflow_handler(struct rdt_domain *dom,
 				unsigned long delay_ms);
 void mbm_handle_overflow(struct work_struct *work);
+void __init intel_rdt_mbm_apply_quirk(void);
 bool is_mba_sc(struct rdt_resource *r);
 void setup_default_ctrlval(struct rdt_resource *r, u32 *dc, u32 *dm);
 u32 delay_bw_map(unsigned long bw, struct rdt_resource *r);
arch/x86/kernel/cpu/resctrl/monitor.c +80 -2

103 lines not on this page · raw message

Signature · 1 line
2.28.0
maintainer
#4 lore raw

Re: [PATCH v3 2/2] x86/resctrl: Correct MBM total and local values

arch/x86/kernel/cpu/resctrl/monitor.c unsigned int rdt_mon_features; · line 87

Quoted 4 lines · Fenghua Yu
arch/x86/kernel/cpu/resctrl/monitor.c +4 -0
+static const struct mbm_correction_factor_table {
+	u32 rmidthreshold;
+	u64 cf;
+} mbm_cf_table[] = {

That thing wants to be __initdata, AFAICT, since the only function touching it is __init.

Made it so.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author
#5 lore raw

Re: [PATCH v3 2/2] x86/resctrl: Correct MBM total and local values

Hi, Boris,

Quoted 10 lines · Borislav Petkov

arch/x86/kernel/cpu/resctrl/monitor.c unsigned int rdt_mon_features; · line 87

Quoted 4 lines · Fenghua Yu
arch/x86/kernel/cpu/resctrl/monitor.c +4 -0
+static const struct mbm_correction_factor_table {
+	u32 rmidthreshold;
+	u64 cf;
+} mbm_cf_table[] = {

That thing wants to be __initdata, AFAICT, since the only function touching it is __init.

Made it so.

You are right. The correction value is read from this table only once in __init. So this table should be __initdata.

Thank you very much for your review!

-Fenghua

merge bot Applied
#6 lore raw

[tip: x86/cache] x86/resctrl: Correct MBM total and local values

In reply to Fenghua Yu
The following commit has been merged into the x86/cache branch of tip:

The following commit has been merged into the x86/cache branch of tip:

Commit-ID:     4868a61d498af3627c3b571aa48107a845001e51
Gitweb:        https://git.kernel.org/tip/4868a61d498af3627c3b571aa48107a845001e51
Author:        Fenghua Yu <[email protected]>
AuthorDate:    Wed, 14 Oct 2020 00:49:27 
Committer:     Borislav Petkov <[email protected]>
CommitterDate: Tue, 27 Oct 2020 18:57:22 +01:00

x86/resctrl: Correct MBM total and local values

Intel Memory Bandwidth Monitoring (MBM) counters may report system memory bandwidth incorrectly on some Intel processors. The errata SKX99 for Skylake server, BDF102 for Broadwell server, and the correction factor table are documented in Documentation/x86/resctrl.rst.

Intel MBM counters track metrics according to the assigned Resource Monitor ID (RMID) for that logical core. The IA32_QM_CTR register (MSR 0xC8E) used to report these metrics, may report incorrect system bandwidth for certain RMID values.

Due to the errata, system memory bandwidth may not match what is reported.

To work around the errata, correct MBM total and local readings using a correction factor table. If rmid > rmid threshold, MBM total and local values should be multiplied by the correction factor.

[ bp: Mark mbm_cf_table[] __initdata. ]

---
 arch/x86/kernel/cpu/resctrl/core.c     |  4 +-
 arch/x86/kernel/cpu/resctrl/internal.h |  1 +-
 arch/x86/kernel/cpu/resctrl/monitor.c  | 82 ++++++++++++++++++++++++-
 3 files changed, 85 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +4 -0
index e5f4ee8..1a681fd 100644
@@ -893,6 +893,10 @@ static __init void __check_quirks_intel(void)
 			set_rdt_options("!cmt,!mbmtotal,!mbmlocal,!l3cat");
 		else
 			set_rdt_options("!l3cat");
+		fallthrough;
+	case INTEL_FAM6_BROADWELL_X:
+		intel_rdt_mbm_apply_quirk();
+		break;
 	}
 }
 
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
index 80fa997..6cb068f 100644
@@ -616,6 +616,7 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 void mbm_setup_overflow_handler(struct rdt_domain *dom,
 				unsigned long delay_ms);
 void mbm_handle_overflow(struct work_struct *work);
+void __init intel_rdt_mbm_apply_quirk(void);
 bool is_mba_sc(struct rdt_resource *r);
 void setup_default_ctrlval(struct rdt_resource *r, u32 *dc, u32 *dm);
 u32 delay_bw_map(unsigned long bw, struct rdt_resource *r);
arch/x86/kernel/cpu/resctrl/monitor.c +80 -2

103 lines not on this page · raw message