ACPI: APEI: GHES: Improve ghes_notify_nmi() status check
Subsystems (3)
Tony Luck
reviewerPATCH 1/1 ACPI: APEI: GHES: Improve ghes_notify_nmi() status check
ghes_notify_nmi() is called for every NMI and must check whether the NMI was generated because an error was signalled by platform firmware.
This check is very expensive as for each registered GHES NMI source it reads from the acpi generic address attached to this error source to get the physical address of the acpi_hest_generic_status block. It then checks the "block_status" to see if an error was logged.
The ACPI/APEI code must create virtual mappings for each of those physical addresses, and tear them down afterwards. On an Icelake system this takes around 15,000 TSC cycles. Enough to disturb efforts to profile system performance.
If that were not bad enough, there are some atomic accesses in the code path that will cause cache line bounces between CPUs. A problem that gets worse as the core count increases.
But BIOS changes neither the acpi generic address nor the physical address of the acpi_hest_generic_status block. So this walk can be done once when the NMI is registered to save the virtual address (unmapping if the NMI is ever unregistered). The "block_status" can be checked directly in the NMI handler. This can be done without any atomic accesses.
Resulting time to check that there is not an error record is around 900 cycles.
---
N.B. I only talked to an Intel BIOS expert about this. GHES code is shared by other architectures, so it would be wise to get confirmation on whether this assumption applies to all, or is Intel (or X86) specific.
--- include/acpi/ghes.h | 1 + drivers/acpi/apei/ghes.c | 39 ++++++++++++++++++++++++++++++++++++--- 2 files changed, 37 insertions(+), 3 deletions(-)
include/acpi/ghes.h +1 -0
@@ -29,6 +29,7 @@ struct ghes { };struct device *dev;struct list_head elist; +void __iomem *error_status_vaddr; };struct ghes_estatus_node {
drivers/acpi/apei/ghes.c +36 -3
@@ -1425,7 +1425,21 @@ static LIST_HEAD(ghes_nmi);static int ghes_notify_nmi (unsigned int cmd,struct pt_regs *regs) {static DEFINE_RAW_SPINLOCK (ghes_notify_lock_nmi); +bool active_error = false;int ret =NMI_DONE ; +struct ghes *ghes; + +rcu_read_lock (); +list_for_each_entry_rcu (ghes, &ghes_nmi, list) { +if (ghes->error_status_vaddr &&readl (ghes->error_status_vaddr)) { + active_error = true; +break ; + } + } +rcu_read_unlock (); + +if (!active_error) +return ret;if (!atomic_add_unless (&ghes_in_nmi,1 ,1 ))return ret;
@@ -1439,13 +1453,26 @@ static int ghes_notify_nmi(unsigned int cmd, struct pt_regs *regs)return ret; } -static void ghes_nmi_add (struct ghes *ghes) +static int ghes_nmi_add (struct ghes *ghes) { +struct acpi_hest_generic *g = ghes->generic; +u64 paddr; +int rc; + + rc =apei_read (&paddr, &g->error_status_address); +if (rc) +return rc; + ghes->error_status_vaddr =acpi_os_ioremap (paddr,sizeof (ghes->estatus->block_status)); +if (!ghes->error_status_vaddr) +return AE_BAD_ADDRESS ; +mutex_lock (&ghes_list_mutex);if (list_empty (&ghes_nmi))register_nmi_handler (NMI_LOCAL , ghes_notify_nmi,0 ,"ghes" );list_add_rcu (&ghes->list, &ghes_nmi);mutex_unlock (&ghes_list_mutex); + +return 0 ; }static void ghes_nmi_remove (struct ghes *ghes)
@@ -1455,6 +1482,10 @@ static void ghes_nmi_remove(struct ghes *ghes)if (list_empty (&ghes_nmi))unregister_nmi_handler (NMI_LOCAL ,"ghes" );mutex_unlock (&ghes_list_mutex); + +if (ghes->error_status_vaddr) +iounmap (ghes->error_status_vaddr); +/* * To synchronize with NMI handler, ghes can only be * freed after NMI handler finishes.
@@ -1462,7 +1493,7 @@ static void ghes_nmi_remove(struct ghes *ghes) synchronize_rcu(); }#else /* CONFIG_HAVE_ACPI_APEI_NMI */ -static inline void ghes_nmi_add (struct ghes *ghes) { } +static inline int ghes_nmi_add (struct ghes *ghes) {return -EINVAL ; }static inline void ghes_nmi_remove (struct ghes *ghes) { }#endif /* CONFIG_HAVE_ACPI_APEI_NMI */
@@ -1630,7 +1661,9 @@ static int ghes_probe(struct platform_device *ghes_dev)ghes_sea_add (ghes);break ;case ACPI_HEST_NOTIFY_NMI : -ghes_nmi_add (ghes); + rc =ghes_nmi_add (ghes); +if (rc) + goto err;break ;case ACPI_HEST_NOTIFY_SOFTWARE_DELEGATED : rc =apei_sdei_register_ghes (ghes);
Signature · 1 line
2.51.0
Quoted 24 lines · Tony Luck
ghes_notify_nmi()is called for every NMI and must check whether the NMI was generated because an error was signalled by platform firmware.This check is very expensive as for each registered GHES NMI source it reads from the acpi generic address attached to this error source to get the physical address of the
acpi_hest_generic_statusblock. It then checks the "block_status" to see if an error was logged.The ACPI/APEI code must create virtual mappings for each of those physical addresses, and tear them down afterwards. On an Icelake system this takes around 15,000 TSC cycles. Enough to disturb efforts to profile system performance.
If that were not bad enough, there are some atomic accesses in the code path that will cause cache line bounces between CPUs. A problem that gets worse as the core count increases.
But BIOS changes neither the acpi generic address nor the physical address of the
acpi_hest_generic_statusblock. So this walk can be done once when the NMI is registered to save the virtual address (unmapping if the NMI is ever unregistered). The "block_status" can be checked directly in the NMI handler. This can be done without any atomic accesses.Resulting time to check that there is not an error record is around 900 cycles.
Quoted 7 lines
Reported-by Andi Kleen <[email protected]>Signed-off-by Tony Luck <[email protected]>---
N.B. I only talked to an Intel BIOS expert about this. GHES code is shared by other architectures, so it would be wise to get confirmation on whether this assumption applies to all, or is Intel (or X86) specific.
I think that is how the ACPI spec describes it.
https://uefi.org/specs/ACPI/6.5/18_Platform_Error_Interfaces.html?highlight=hest#error-source-discovery
The HEST and other tables are fixed at init time. There's an ACPI notify event for if/when a device method needs to be re-evaluted, but I don't think anything in APEI expects that.
Thanks,
Yazen
Luck, Tony
reviewerQuoted 11 lines · Yazen Ghannam
Quoted 3 lines · Tony Luck
N.B. I only talked to an Intel BIOS expert about this. GHES code is shared by other architectures, so it would be wise to get confirmation on whether this assumption applies to all, or is Intel (or X86) specific.
I think that is how the ACPI spec describes it.
https://uefi.org/specs/ACPI/6.5/18_Platform_Error_Interfaces.html?highlight=hest#error-source-discoveryThe HEST and other tables are fixed at init time. There's an ACPI notify event for if/when a device method needs to be re-evaluted, but I don't think anything in APEI expects that.
Yazen,
Thanks for looking.
James Morse pointed me to the fact that HAVE_ACPI_APEI_NMI is only selected by arch/x86/Kconfig. So scope is limited to x86.
That "18.3 Error Source Discovery" section of the ACPI spec feels pretty vague. The "During initialization, OSPM examines the tables" bit is comforting, but I was worried that although the tables are static (so the generic ACPI address from the NMI error source shouldn't change) there is an extra level of indirection where that points to a location that holds the address of the error status block. Some might not consider that as part of the "tables".
Further pondering has me 99% convinced that there would be no reason for firmware to update that pointer (though it has led me to wonder why that indirection exists).
Rafael: I think this patch is good to go.
-Tony
Shuai Xue
reviewerQuoted 11 lines · Tony Luck
ghes_notify_nmi()is called for every NMI and must check whether the NMI was generated because an error was signalled by platform firmware.This check is very expensive as for each registered GHES NMI source it reads from the acpi generic address attached to this error source to get the physical address of the
acpi_hest_generic_statusblock. It then checks the "block_status" to see if an error was logged.The ACPI/APEI code must create virtual mappings for each of those physical addresses, and tear them down afterwards. On an Icelake system this takes around 15,000 TSC cycles. Enough to disturb efforts to profile system performance.
Hi, Tony
Interesting.
If I understand correctly, you mean ghes_peek_estatus() and ghes_clear_estatus().
I conducted performance testing on our system (ARM v8) and found the following average costs:
- ghes_peek_estatus(): 8,138.3 ns (21,160 cycles)
- ghes_clear_estatus(): 2,038.3 ns (5,300 cycles)
Quoted 4 lines
If that were not bad enough, there are some atomic accesses in the code path that will cause cache line bounces between CPUs. A problem that gets worse as the core count increases.
Could you elaborate on which specific atomic accesses you're referring to?
Quoted 9 lines
But BIOS changes neither the acpi generic address nor the physical address of the
acpi_hest_generic_statusblock. So this walk can be done once when the NMI is registered to save the virtual address (unmapping if the NMI is ever unregistered). The "block_status" can be checked directly in the NMI handler. This can be done without any atomic accesses.Resulting time to check that there is not an error record is around 900 cycles.
Quoted 7 lines
Reported-by Andi Kleen <[email protected]>Signed-off-by Tony Luck <[email protected]>---
N.B. I only talked to an Intel BIOS expert about this. GHES code is shared by other architectures, so it would be wise to get confirmation on whether this assumption applies to all, or is Intel (or X86) specific.
The assumption is "BIOS changes neither the acpi generic address nor the physical address of the acpi_hest_generic_status block."?
I've consulted with our BIOS experts from both ARM and RISC-V platform teams, and they confirmed that error status blocks are reserved at boot time and remain unchanged during runtime.
Quoted 30 lines
--- include/acpi/ghes.h | 1 + drivers/acpi/apei/ghes.c | 39 ++++++++++++++++++++++++++++++++++++--- 2 files changed, 37 insertions(+), 3 deletions(-)include/acpi/ghes.h +1
-0@@ -29,6 +29,7 @@ struct ghes { };struct device *dev;struct list_head elist; +void __iomem *error_status_vaddr; };struct ghes_estatus_node {drivers/acpi/apei/ghes.c +4
-0@@ -1425,7 +1425,21 @@ static LIST_HEAD(ghes_nmi);static int ghes_notify_nmi (unsigned int cmd,struct pt_regs *regs) {static DEFINE_RAW_SPINLOCK (ghes_notify_lock_nmi); +bool active_error = false;int ret =NMI_DONE ; +struct ghes *ghes; + +rcu_read_lock ();
drivers/acpi/apei/ghes.c static LIST_HEAD(ghes_nmi); · line 1442
Quoted 10 lines
drivers/acpi/apei/ghes.c +10
-0+list_for_each_entry_rcu (ghes, &ghes_nmi, list) { +if (ghes->error_status_vaddr &&readl (ghes->error_status_vaddr)) { + active_error = true; +break ; + } + } +rcu_read_unlock (); + +if (!active_error) +return ret;
Shoud we put active_error into struct ghes? If we know it is active, we do not need to call __ghes_peek_estatus() to estatus->block_status().
Thanks.
Shuai
Luck, Tony
reviewerQuoted 19 lines · Shuai Xue
Quoted 11 lines · Tony Luck
ghes_notify_nmi()is called for every NMI and must check whether the NMI was generated because an error was signalled by platform firmware.This check is very expensive as for each registered GHES NMI source it reads from the acpi generic address attached to this error source to get the physical address of the
acpi_hest_generic_statusblock. It then checks the "block_status" to see if an error was logged.The ACPI/APEI code must create virtual mappings for each of those physical addresses, and tear them down afterwards. On an Icelake system this takes around 15,000 TSC cycles. Enough to disturb efforts to profile system performance.
Hi, Tony
Interesting.
Quoted 8 lines
If I understand correctly, you mean
ghes_peek_estatus()andghes_clear_estatus().I conducted performance testing on our system (ARM v8) and found the following average costs:
-
ghes_peek_estatus(): 8,138.3 ns (21,160 cycles)
-ghes_clear_estatus(): 2,038.3 ns (5,300 cycles)
ARM doesn't use the NMI path (HAVE_ACPI_APEI_NMI is only set on X86). But maybe you are looking at ghes_notify_sea() which seems similar?
Quoted 7 lines
Quoted 5 lines
If that were not bad enough, there are some atomic accesses in the code path that will cause cache line bounces between CPUs. A problem that gets worse as the core count increases.
Could you elaborate on which specific atomic accesses you're referring to?
ghes_notify_nmi() starts with code to ensure only one CPU executes the GHES NMI path.
if (!atomic_add_unless(&ghes_in_nmi, 1, 1)) return ret;
Looks like an optimization to avoid having a bunch of CPUs queue up waiting for this spinllock:
raw_spin_lock(&ghes_notify_lock_nmi);
when the first one to get it will find and handle the logged error.
Quoted 18 lines
Quoted 10 lines
But BIOS changes neither the acpi generic address nor the physical address of the
acpi_hest_generic_statusblock. So this walk can be done once when the NMI is registered to save the virtual address (unmapping if the NMI is ever unregistered). The "block_status" can be checked directly in the NMI handler. This can be done without any atomic accesses.Resulting time to check that there is not an error record is around 900 cycles.
Quoted 7 lines
Reported-by Andi Kleen <[email protected]>Signed-off-by Tony Luck <[email protected]>---
N.B. I only talked to an Intel BIOS expert about this. GHES code is shared by other architectures, so it would be wise to get confirmation on whether this assumption applies to all, or is Intel (or X86) specific.
Quoted 6 lines
The assumption is "BIOS changes neither the acpi generic address nor the physical address of the
acpi_hest_generic_statusblock."?I've consulted with our BIOS experts from both ARM and RISC-V platform teams, and they confirmed that error status blocks are reserved at boot time and remain unchanged during runtime.
Thanks. Good to have this confirmation.
Quoted 34 lines
Quoted 34 lines
--- include/acpi/ghes.h | 1 + drivers/acpi/apei/ghes.c | 39 ++++++++++++++++++++++++++++++++++++--- 2 files changed, 37 insertions(+), 3 deletions(-)include/acpi/ghes.h +1
-0@@ -29,6 +29,7 @@ struct ghes { };struct device *dev;struct list_head elist; +void __iomem *error_status_vaddr; };struct ghes_estatus_node {drivers/acpi/apei/ghes.c +9
-0@@ -1425,7 +1425,21 @@ static LIST_HEAD(ghes_nmi);static int ghes_notify_nmi (unsigned int cmd,struct pt_regs *regs) {static DEFINE_RAW_SPINLOCK (ghes_notify_lock_nmi); +bool active_error = false;int ret =NMI_DONE ; +struct ghes *ghes; + +rcu_read_lock (); +list_for_each_entry_rcu (ghes, &ghes_nmi, list) { +if (ghes->error_status_vaddr &&readl (ghes->error_status_vaddr)) { + active_error = true; +break ; + }
Quoted 8 lines
drivers/acpi/apei/ghes.c static LIST_HEAD(ghes_nmi); · line 1442
Quoted 5 lines
drivers/acpi/apei/ghes.c +5
-0+ } +rcu_read_unlock (); + +if (!active_error) +return ret;Shoud we put
active_errorinto struct ghes? If we know it is active, we do not need to call__ghes_peek_estatus()toestatus->block_status().
That might be a useful addition. I was primarily concerned in fixing the "no erroor" case that happens at a very high rate while profiling the system with "perf". But skipping (or just removing? __ghes_peek_estatus()) if you have already confirmed that there is a logged error would be good.
If you can use the same technique for ghes_notify_sea() then it would be sensible to move the code I added to ghes_nmi_add() to ghes_new() to save the virtual address for every type of GHES notification.
Quoted 3 lines
Thanks.
Shuai
-Tony
Shuai Xue
reviewerQuoted 23 lines · Tony
Quoted 22 lines · Shuai Xue
Quoted 11 lines · Tony Luck
ghes_notify_nmi()is called for every NMI and must check whether the NMI was generated because an error was signalled by platform firmware.This check is very expensive as for each registered GHES NMI source it reads from the acpi generic address attached to this error source to get the physical address of the
acpi_hest_generic_statusblock. It then checks the "block_status" to see if an error was logged.The ACPI/APEI code must create virtual mappings for each of those physical addresses, and tear them down afterwards. On an Icelake system this takes around 15,000 TSC cycles. Enough to disturb efforts to profile system performance.
Hi, Tony
Interesting.
If I understand correctly, you mean
ghes_peek_estatus()andghes_clear_estatus().
Quoted 8 lines
Quoted 5 lines
I conducted performance testing on our system (ARM v8) and found the following average costs:
-
ghes_peek_estatus(): 8,138.3 ns (21,160 cycles)
-ghes_clear_estatus(): 2,038.3 ns (5,300 cycles)ARM doesn't use the NMI path (
HAVE_ACPI_APEI_NMIis only set on X86). But maybe you are looking atghes_notify_sea()which seems similar?
Yes. It is measured in ghes_notify_sea().
Quoted 14 lines
Quoted 7 lines
Quoted 5 lines
If that were not bad enough, there are some atomic accesses in the code path that will cause cache line bounces between CPUs. A problem that gets worse as the core count increases.
Could you elaborate on which specific atomic accesses you're referring to?
ghes_notify_nmi()starts with code to ensure only one CPU executes the GHES NMI path.if (!atomic_add_unless(&ghes_in_nmi, 1, 1)) return ret;
Quoted 6 lines
Looks like an optimization to avoid having a bunch of CPUs queue up waiting for this spinllock:
raw_spin_lock(&ghes_notify_lock_nmi);when the first one to get it will find and handle the logged error.
If an NMI issued, at last one status block is active. I don't see how the code path is different.
Quoted 70 lines
Quoted 18 lines
Quoted 10 lines
But BIOS changes neither the acpi generic address nor the physical address of the
acpi_hest_generic_statusblock. So this walk can be done once when the NMI is registered to save the virtual address (unmapping if the NMI is ever unregistered). The "block_status" can be checked directly in the NMI handler. This can be done without any atomic accesses.Resulting time to check that there is not an error record is around 900 cycles.
Quoted 7 lines
Reported-by Andi Kleen <[email protected]>Signed-off-by Tony Luck <[email protected]>---
N.B. I only talked to an Intel BIOS expert about this. GHES code is shared by other architectures, so it would be wise to get confirmation on whether this assumption applies to all, or is Intel (or X86) specific.Quoted 6 lines
The assumption is "BIOS changes neither the acpi generic address nor the physical address of the
acpi_hest_generic_statusblock."?I've consulted with our BIOS experts from both ARM and RISC-V platform teams, and they confirmed that error status blocks are reserved at boot time and remain unchanged during runtime.
Thanks. Good to have this confirmation.
Quoted 34 lines
Quoted 34 lines
--- include/acpi/ghes.h | 1 + drivers/acpi/apei/ghes.c | 39 ++++++++++++++++++++++++++++++++++++--- 2 files changed, 37 insertions(+), 3 deletions(-)include/acpi/ghes.h +1
-0@@ -29,6 +29,7 @@ struct ghes { };struct device *dev;struct list_head elist; +void __iomem *error_status_vaddr; };struct ghes_estatus_node {drivers/acpi/apei/ghes.c +9
-0@@ -1425,7 +1425,21 @@ static LIST_HEAD(ghes_nmi);static int ghes_notify_nmi (unsigned int cmd,struct pt_regs *regs) {static DEFINE_RAW_SPINLOCK (ghes_notify_lock_nmi); +bool active_error = false;int ret =NMI_DONE ; +struct ghes *ghes; + +rcu_read_lock (); +list_for_each_entry_rcu (ghes, &ghes_nmi, list) { +if (ghes->error_status_vaddr &&readl (ghes->error_status_vaddr)) { + active_error = true; +break ; + }Quoted 8 lines
drivers/acpi/apei/ghes.c static LIST_HEAD(ghes_nmi); · line 1442
Quoted 5 lines
drivers/acpi/apei/ghes.c +5
-0+ } +rcu_read_unlock (); + +if (!active_error) +return ret;Shoud we put
active_errorinto struct ghes? If we know it is active, we do not need to call__ghes_peek_estatus()toestatus->block_status().
Quoted 3 lines
That might be a useful addition. I was primarily concerned in fixing the "no erroor" case that happens at a very high rate while profiling the system with "perf".
Do you mean you see "NMI received for unknown reason" when profiling with "perf"? And the unknown error is handled by ghes_notify_nmi().
I see some unknown NMI in production in Intel platform, but I did not figure out how it happend. Can you help to explain it?
Quoted 7 lines
But skipping (or just removing?
__ghes_peek_estatus()) if you have already confirmed that there is a logged error would be good.If you can use the same technique for
ghes_notify_sea()then it would be sensible to move the code I added toghes_nmi_add()toghes_new()to save the virtual address for every type of GHES notification.
Sure, I'd like to add it for ghes_notify_sea().
Thanks.
Shuai
Luck, Tony
reviewerQuoted 54 lines · Shuai Xue
Quoted 23 lines · Tony
Quoted 22 lines · Shuai Xue
Quoted 11 lines · Tony Luck
ghes_notify_nmi()is called for every NMI and must check whether the NMI was generated because an error was signalled by platform firmware.This check is very expensive as for each registered GHES NMI source it reads from the acpi generic address attached to this error source to get the physical address of the
acpi_hest_generic_statusblock. It then checks the "block_status" to see if an error was logged.The ACPI/APEI code must create virtual mappings for each of those physical addresses, and tear them down afterwards. On an Icelake system this takes around 15,000 TSC cycles. Enough to disturb efforts to profile system performance.
Hi, Tony
Interesting.
If I understand correctly, you mean
ghes_peek_estatus()andghes_clear_estatus().Quoted 8 lines
Quoted 5 lines
I conducted performance testing on our system (ARM v8) and found the following average costs:
-
ghes_peek_estatus(): 8,138.3 ns (21,160 cycles)
-ghes_clear_estatus(): 2,038.3 ns (5,300 cycles)ARM doesn't use the NMI path (
HAVE_ACPI_APEI_NMIis only set on X86). But maybe you are looking atghes_notify_sea()which seems similar?Yes. It is measured in
ghes_notify_sea().Quoted 17 lines
Quoted 7 lines
Quoted 5 lines
If that were not bad enough, there are some atomic accesses in the code path that will cause cache line bounces between CPUs. A problem that gets worse as the core count increases.
Could you elaborate on which specific atomic accesses you're referring to?
ghes_notify_nmi()starts with code to ensure only one CPU executes the GHES NMI path.if (!atomic_add_unless(&ghes_in_nmi, 1, 1)) return ret;Looks like an optimization to avoid having a bunch of CPUs queue up waiting for this spinllock:
Quoted 6 lines
Quoted 3 lines
raw_spin_lock(&ghes_notify_lock_nmi);when the first one to get it will find and handle the logged error.
If an NMI issued, at last one status block is active. I don't see how the code path is different.
On X86 NMI doesn't include a vector. There is just one interrupt entry point for any NMI.
Linux deals with this with each subsystem that may expect an NMI registering on an NMI notify chain. When an NMI happens the core code calls each subsystem function to ask:
Was this NMI for you?
Was this NMI for you?
Was this NMI for you?
Was this NMI for you?
Was this NMI for you?
Was this NMI for you?
Was this NMI for you?
Was this NMI for you?
This is a problem when profiling the system with "perf" as there are thousands of NMIs, and each one calls ghes_notify_nmi() which will look to see if an error was logged, and (after all the ACPI/APEI mapping physical addresses, reading, and unmapping) will return NMI_DONE to indicate that no error was found, so this NMI came from some other source.
Quoted 71 lines
Quoted 71 lines
Quoted 19 lines
Quoted 11 lines
But BIOS changes neither the acpi generic address nor the physical address of the
acpi_hest_generic_statusblock. So this walk can be done once when the NMI is registered to save the virtual address (unmapping if the NMI is ever unregistered). The "block_status" can be checked directly in the NMI handler. This can be done without any atomic accesses.Resulting time to check that there is not an error record is around 900 cycles.
Quoted 7 lines
Reported-by Andi Kleen <[email protected]>Signed-off-by Tony Luck <[email protected]>---
N.B. I only talked to an Intel BIOS expert about this. GHES code is shared by other architectures, so it would be wise to get confirmation on whether this assumption applies to all, or is Intel (or X86) specific.Quoted 6 lines
The assumption is "BIOS changes neither the acpi generic address nor the physical address of the
acpi_hest_generic_statusblock."?I've consulted with our BIOS experts from both ARM and RISC-V platform teams, and they confirmed that error status blocks are reserved at boot time and remain unchanged during runtime.
Thanks. Good to have this confirmation.
Quoted 42 lines
Quoted 29 lines
--- include/acpi/ghes.h | 1 + drivers/acpi/apei/ghes.c | 39 ++++++++++++++++++++++++++++++++++++--- 2 files changed, 37 insertions(+), 3 deletions(-)include/acpi/ghes.h +1
-0@@ -29,6 +29,7 @@ struct ghes { };struct device *dev;struct list_head elist; +void __iomem *error_status_vaddr; };struct ghes_estatus_node {drivers/acpi/apei/ghes.c +4
-0@@ -1425,7 +1425,21 @@ static LIST_HEAD(ghes_nmi);static int ghes_notify_nmi (unsigned int cmd,struct pt_regs *regs) {static DEFINE_RAW_SPINLOCK (ghes_notify_lock_nmi); +bool active_error = false;int ret =NMI_DONE ; +struct ghes *ghes; + +rcu_read_lock ();drivers/acpi/apei/ghes.c static LIST_HEAD(ghes_nmi); · line 1442
Quoted 10 lines
drivers/acpi/apei/ghes.c +10
-0+list_for_each_entry_rcu (ghes, &ghes_nmi, list) { +if (ghes->error_status_vaddr &&readl (ghes->error_status_vaddr)) { + active_error = true; +break ; + } + } +rcu_read_unlock (); + +if (!active_error) +return ret;Shoud we put
active_errorinto struct ghes? If we know it is active, we do not need to call__ghes_peek_estatus()toestatus->block_status().
Quoted 6 lines
Quoted 3 lines
That might be a useful addition. I was primarily concerned in fixing the "no erroor" case that happens at a very high rate while profiling the system with "perf".
Do you mean you see "NMI received for unknown reason" when profiling with "perf"? And the unknown error is handled by
ghes_notify_nmi().
This message is printed when every handler on the NMI chain says "not my interrupt" with the NMI_DONE return value.
Quoted 8 lines
I see some unknown NMI in production in Intel platform, but I did not figure out how it happend. Can you help to explain it?
Quoted 4 lines
But skipping (or just removing?
__ghes_peek_estatus()) if you have already confirmed that there is a logged error would be good.
Quoted 5 lines
Quoted 3 lines
If you can use the same technique for
ghes_notify_sea()then it would be sensible to move the code I added toghes_nmi_add()toghes_new()to save the virtual address for every type of GHES notification.Sure, I'd like to add it for
ghes_notify_sea().
Great. Take my patch and fix it up to cover ghes_notify_sea() as well. I don't have a system that I can test that on. When you are done I can retest the ghes_notify_nmi() to check that it still works.
Quoted 3 lines
Thanks.
Shuai
-Tony
Quoted 1 line
Shuai Xue
reviewerQuoted 78 lines · Tony
Quoted 54 lines · Shuai Xue
Quoted 23 lines · Tony
Quoted 22 lines · Shuai Xue
Quoted 11 lines · Tony Luck
ghes_notify_nmi()is called for every NMI and must check whether the NMI was generated because an error was signalled by platform firmware.This check is very expensive as for each registered GHES NMI source it reads from the acpi generic address attached to this error source to get the physical address of the
acpi_hest_generic_statusblock. It then checks the "block_status" to see if an error was logged.The ACPI/APEI code must create virtual mappings for each of those physical addresses, and tear them down afterwards. On an Icelake system this takes around 15,000 TSC cycles. Enough to disturb efforts to profile system performance.
Hi, Tony
Interesting.
If I understand correctly, you mean
ghes_peek_estatus()andghes_clear_estatus().Quoted 8 lines
Quoted 5 lines
I conducted performance testing on our system (ARM v8) and found the following average costs:
-
ghes_peek_estatus(): 8,138.3 ns (21,160 cycles)
-ghes_clear_estatus(): 2,038.3 ns (5,300 cycles)ARM doesn't use the NMI path (
HAVE_ACPI_APEI_NMIis only set on X86). But maybe you are looking atghes_notify_sea()which seems similar?Yes. It is measured in
ghes_notify_sea().Quoted 17 lines
Quoted 7 lines
Quoted 5 lines
If that were not bad enough, there are some atomic accesses in the code path that will cause cache line bounces between CPUs. A problem that gets worse as the core count increases.
Could you elaborate on which specific atomic accesses you're referring to?
ghes_notify_nmi()starts with code to ensure only one CPU executes the GHES NMI path.if (!atomic_add_unless(&ghes_in_nmi, 1, 1)) return ret;Looks like an optimization to avoid having a bunch of CPUs queue up waiting for this spinllock:
Quoted 6 lines
Quoted 3 lines
raw_spin_lock(&ghes_notify_lock_nmi);when the first one to get it will find and handle the logged error.
If an NMI issued, at last one status block is active. I don't see how the code path is different.
On X86 NMI doesn't include a vector. There is just one interrupt entry point for any NMI.
Linux deals with this with each subsystem that may expect an NMI registering on an NMI notify chain. When an NMI happens the core code calls each subsystem function to ask:
Was this NMI for you?
Was this NMI for you?
Was this NMI for you?
Was this NMI for you?
Was this NMI for you?
Was this NMI for you?
Was this NMI for you?
Was this NMI for you?
Quoted 6 lines
This is a problem when profiling the system with "perf" as there are thousands of NMIs, and each one calls
ghes_notify_nmi()which will look to see if an error was logged, and (after all the ACPI/APEI mapping physical addresses, reading, and unmapping) will returnNMI_DONEto indicate that no error was found, so this NMI came from some other source.
I see, thank you the datails explaination.
Quoted 75 lines
Quoted 75 lines
Quoted 74 lines
Quoted 19 lines
Quoted 11 lines
But BIOS changes neither the acpi generic address nor the physical address of the
acpi_hest_generic_statusblock. So this walk can be done once when the NMI is registered to save the virtual address (unmapping if the NMI is ever unregistered). The "block_status" can be checked directly in the NMI handler. This can be done without any atomic accesses.Resulting time to check that there is not an error record is around 900 cycles.
Quoted 7 lines
Reported-by Andi Kleen <[email protected]>Signed-off-by Tony Luck <[email protected]>---
N.B. I only talked to an Intel BIOS expert about this. GHES code is shared by other architectures, so it would be wise to get confirmation on whether this assumption applies to all, or is Intel (or X86) specific.Quoted 6 lines
The assumption is "BIOS changes neither the acpi generic address nor the physical address of the
acpi_hest_generic_statusblock."?I've consulted with our BIOS experts from both ARM and RISC-V platform teams, and they confirmed that error status blocks are reserved at boot time and remain unchanged during runtime.
Thanks. Good to have this confirmation.
Quoted 34 lines
Quoted 34 lines
--- include/acpi/ghes.h | 1 + drivers/acpi/apei/ghes.c | 39 ++++++++++++++++++++++++++++++++++++--- 2 files changed, 37 insertions(+), 3 deletions(-)include/acpi/ghes.h +1
-0@@ -29,6 +29,7 @@ struct ghes { };struct device *dev;struct list_head elist; +void __iomem *error_status_vaddr; };struct ghes_estatus_node {drivers/acpi/apei/ghes.c +9
-0@@ -1425,7 +1425,21 @@ static LIST_HEAD(ghes_nmi);static int ghes_notify_nmi (unsigned int cmd,struct pt_regs *regs) {static DEFINE_RAW_SPINLOCK (ghes_notify_lock_nmi); +bool active_error = false;int ret =NMI_DONE ; +struct ghes *ghes; + +rcu_read_lock (); +list_for_each_entry_rcu (ghes, &ghes_nmi, list) { +if (ghes->error_status_vaddr &&readl (ghes->error_status_vaddr)) { + active_error = true; +break ; + }Quoted 8 lines
drivers/acpi/apei/ghes.c static LIST_HEAD(ghes_nmi); · line 1442
Quoted 5 lines
drivers/acpi/apei/ghes.c +5
-0+ } +rcu_read_unlock (); + +if (!active_error) +return ret;Shoud we put
active_errorinto struct ghes? If we know it is active, we do not need to call__ghes_peek_estatus()toestatus->block_status().That might be a useful addition. I was primarily concerned in fixing the "no erroor" case that happens at a very high rate while profiling the system with "perf".
Quoted 5 lines
Quoted 2 lines
Do you mean you see "NMI received for unknown reason" when profiling with "perf"? And the unknown error is handled by
ghes_notify_nmi().This message is printed when every handler on the NMI chain says "not my interrupt" with the
NMI_DONEreturn value.
$sudo perf script
swapper 0 [148] 1117148.086594: probe:ghes_notify_nmi: (ffffffffb3867210)
ffffffffb3867211 ghes_notify_nmi+0x1 (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb303c366 nmi_handle.part.0+0x56 (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb3dc7e0a default_do_nmi+0x11a (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb3dc7f40 exc_nmi+0x100 (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb3e0169a end_repeat_nmi+0x16 (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb3732915 _find_next_and_bit+0x65 (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb3177780 update_sg_lb_stats+0xc0 (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb317d5dc update_sd_lb_stats.constprop.0+0xcc (/lib/modules/6.6.102+/build/vmlinu>
ffffffffb317d7d1 find_busiest_group+0x41 (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb317dc0a load_balance+0x12a (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb317efd4 rebalance_domains+0x2b4 (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb31235d2 handle_softirqs+0xd2 (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb31238b3 __irq_exit_rcu+0xa3 (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb3dca462 sysvec_apic_timer_interrupt+0x72 (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb3e00e86 asm_sysvec_apic_timer_interrupt+0x16 (/lib/modules/6.6.102+/build/vmlin>
ffffffffb3dcbbee cpuidle_enter_state+0xbe (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb3adc6b9 cpuidle_enter+0x29 (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb31901ea cpuidle_idle_call+0xfa (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb31902db do_idle+0x7b (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb3190536 cpu_startup_entry+0x26 (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb30775d5 start_secondary+0x115 (/lib/modules/6.6.102+/build/vmlinux)
ffffffffb30001e7 secondary_startup_64_no_verify+0x182 (/lib/modules/6.6.102+/build/vmlin>Yes, I even find that ghes_notify_nmi() is called when no perf is used! (:
Quoted 12 lines
Quoted 12 lines
I see some unknown NMI in production in Intel platform, but I did not figure out how it happend. Can you help to explain it?
Quoted 7 lines
But skipping (or just removing?
__ghes_peek_estatus()) if you have already confirmed that there is a logged error would be good.If you can use the same technique for
ghes_notify_sea()then it would be sensible to move the code I added toghes_nmi_add()toghes_new()to save the virtual address for every type of GHES notification.
Quoted 5 lines
Quoted 1 line
Sure, I'd like to add it for
ghes_notify_sea().Great. Take my patch and fix it up to cover
ghes_notify_sea()as well. I don't have a system that I can test that on. When you are done I can retest theghes_notify_nmi()to check that it still works.
Got it.
Thanks.
Shuai