Skip to contentSkip to conversation

ACPI/APEI: Limit printable size of BERT table data

PATCH In review Darren Hart 3 participants 6 messages Mar 8, 2022 – Jun 29, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ACPI/APEI: Limit printable size of BERT table data

Platforms with large BERT table data can trigger soft lockup errors while attempting to print the entire BERT table data to the console at boot:

  watchdog: BUG: soft lockup - CPU#160 stuck for 23s! [swapper/0:1]

Observed on Ampere Altra systems with a single BERT record of ~250KB.

The original bert driver appears to have assumed relatively small table data. Since it is impractical to reassemble large table data from interwoven console messages, and the table data is available in

/sys/firmware/acpi/tables/data/BERT

limit the size for tables printed to the console to 1024 (for no reason other than it seemed like a good place to kick off the discussion, would appreciate feedback from existing users in terms of what size would maintain their current usage model).

Alternatively, we could make printing a CONFIG option, use the bert_disable boot arg (or something similar), or use a debug log level. However, all those solutions require extra steps or change the existing behavior for small table data. Limiting the size preserves existing behavior on existing platforms with small table data, and eliminates the soft lockups for platforms with large table data, while still making it available.

Cc "Rafael J. Wysocki" <[email protected]>
Cc Len Brown <[email protected]>
Cc James Morse <[email protected]>
Cc Tony Luck <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Doug Rady <[email protected]>
---
 drivers/acpi/apei/bert.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/acpi/apei/bert.c +6 -2
index 19e50fcbf4d6..ad8ab3f12cf3 100644
@@ -29,6 +29,7 @@
 
 #undef pr_fmt
 #define pr_fmt(fmt) "BERT: " fmt
+#define ACPI_BERT_PRINT_MAX_LEN 1024
 
 static int bert_disable;
 
@@ -58,8 +59,11 @@ static void __init bert_print_all(struct acpi_bert_region *region,
 		}
 
 		pr_info_once("Error records from previous boot:\n");
-
-		cper_estatus_print(KERN_INFO HW_ERR, estatus);
+		if (region_len < ACPI_BERT_PRINT_MAX_LEN)
+			cper_estatus_print(KERN_INFO HW_ERR, estatus);
+		else
+			pr_info_once("Max print length exceeded, table data is available at:\n"
+				     "/sys/firmware/acpi/tables/data/BERT");
 
 		/*
 		 * Because the boot error source is "one-time polled" type,
Signature · 1 line
2.31.1
#1 lore raw
Quoted 28 lines · Darren Hart

Platforms with large BERT table data can trigger soft lockup errors while attempting to print the entire BERT table data to the console at boot:

  watchdog: BUG: soft lockup - CPU#160 stuck for 23s! [swapper/0:1]

Observed on Ampere Altra systems with a single BERT record of ~250KB.

The original bert driver appears to have assumed relatively small table data. Since it is impractical to reassemble large table data from interwoven console messages, and the table data is available in

/sys/firmware/acpi/tables/data/BERT

limit the size for tables printed to the console to 1024 (for no reason other than it seemed like a good place to kick off the discussion, would appreciate feedback from existing users in terms of what size would maintain their current usage model).

Alternatively, we could make printing a CONFIG option, use the bert_disable boot arg (or something similar), or use a debug log level. However, all those solutions require extra steps or change the existing behavior for small table data. Limiting the size preserves existing behavior on existing platforms with small table data, and eliminates the soft lockups for platforms with large table data, while still making it available.

Quoted 7 lines
Cc "Rafael J. Wysocki" <[email protected]>
Cc Len Brown <[email protected]>
Cc James Morse <[email protected]>
Cc Tony Luck <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Doug Rady <[email protected]>

Not that I have a particularly strong opinion here, but this looks reasonable to me, so I've queued it up for 5.18.

APEI reviewers, please chime in if you disagree with the above.

Quoted 33 lines
---
 drivers/acpi/apei/bert.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/acpi/apei/bert.c +6 -3
index 19e50fcbf4d6..ad8ab3f12cf3 100644
@@ -29,6 +29,7 @@
 
 #undef pr_fmt
 #define pr_fmt(fmt) "BERT: " fmt
+#define ACPI_BERT_PRINT_MAX_LEN 1024
 
 static int bert_disable;
 
@@ -58,8 +59,11 @@ static void __init bert_print_all(struct acpi_bert_region *region,
                }
 
                pr_info_once("Error records from previous boot:\n");
-
-               cper_estatus_print(KERN_INFO HW_ERR, estatus);
+               if (region_len < ACPI_BERT_PRINT_MAX_LEN)
+                       cper_estatus_print(KERN_INFO HW_ERR, estatus);
+               else
+                       pr_info_once("Max print length exceeded, table data is available at:\n"
+                                    "/sys/firmware/acpi/tables/data/BERT");
 
                /*
                 * Because the boot error source is "one-time polled" type,
--

2.31.1

reviewer
#2 lore raw
Quoted 6 lines · Rafael J. Wysocki

On Tue, Mar 8, 2022 at 7:51 PM Darren Hart

Not that I have a particularly strong opinion here, but this looks reasonable to me, so I've queued it up for 5.18.

APEI reviewers, please chime in if you disagree with the above.

It looked reasonable to me when I skimmed it in March. But the reality check now needs cashing because some validation team here is complaining that they don't see any errors printed from their BERT tests. :-(

So I looked again. This test inside the loop seems bogus:

	if (region_len < ACPI_BERT_PRINT_MAX_LEN) {

because "region_len" isn't updated inside the loop. If it is too big then it will prevent Linux from printing any/all of the records in the BERT table (and the test could have been done before the loop).

Maybe below patch is better? It avoids printing individual CPER records that are too large (checking estatus_len instead of region_len).

I also added a limit to how many records to print (I randomly picked "5" as the limit ... the specific failing test only want to print one).

-Tony

[I will write up a proper commit message and add a Signed-off-by if this looks to be a reasonable direction]

drivers/acpi/apei/bert.c +11 -6
index 598fd19b65fa..4e894a728c02 100644
@@ -29,6 +29,8 @@
 
 #undef pr_fmt
 #define pr_fmt(fmt) "BERT: " fmt
+
+#define ACPI_BERT_PRINT_MAX_RECORDS 5
 #define ACPI_BERT_PRINT_MAX_LEN 1024
 
 static int bert_disable;
@@ -39,6 +41,7 @@ static void __init bert_print_all(struct acpi_bert_region *region,
 	struct acpi_hest_generic_status *estatus =
 		(struct acpi_hest_generic_status *)region;
 	int remain = region_len;
+	int ncper = 0, skipped = 0;
 	u32 estatus_len;
 
 	while (remain >= sizeof(struct acpi_bert_region)) {
@@ -46,24 +49,23 @@ static void __init bert_print_all(struct acpi_bert_region *region,
 		if (remain < estatus_len) {
 			pr_err(FW_BUG "Truncated status block (length: %u).\n",
 			       estatus_len);
-			return;
+			break;
 		}
 
 		/* No more error records. */
 		if (!estatus->block_status)
-			return;
+			break;
 
 		if (cper_estatus_check(estatus)) {
 			pr_err(FW_BUG "Invalid error record.\n");
-			return;
+			break;
 		}
 
 		pr_info_once("Error records from previous boot:\n");
-		if (region_len < ACPI_BERT_PRINT_MAX_LEN)
+		if (ncper++ < ACPI_BERT_PRINT_MAX_RECORDS && estatus_len < ACPI_BERT_PRINT_MAX_LEN)
 			cper_estatus_print(KERN_INFO HW_ERR, estatus);
 		else
-			pr_info_once("Max print length exceeded, table data is available at:\n"
-				     "/sys/firmware/acpi/tables/data/BERT");
+			skipped++;
 
 		/*
 		 * Because the boot error source is "one-time polled" type,
@@ -75,6 +77,9 @@ static void __init bert_print_all(struct acpi_bert_region *region,
 		estatus = (void *)estatus + estatus_len;
 		remain -= estatus_len;
 	}
+	if (skipped)
+		pr_info("Skipped %d error records, full table data is available at:\n"
+			"/sys/firmware/acpi/tables/data/BERT", skipped);
 }
 
 static int __init setup_bert_disable(char *str)
reviewer
#3 lore raw

PATCH 1/1 ACPI/APEI: Better fix to avoid spamming the console with old error logs

The fix in commit 3f8dec116210 ("ACPI/APEI: Limit printable size of BERT table data") does not work as intended on systems where the BIOS has a fixed size block of memory for the BERT table, relying on s/w to quit when it finds a record with estatus->block_status == 0. On these systems all errors are suppressed because the check:

if (region_len < ACPI_BERT_PRINT_MAX_LEN)

always fails.

New scheme skips individual CPER records that are too large, and also limits the total number of records that will be printed to 5.

Fixes 3f8dec116210 ("ACPI/APEI: Limit printable size of BERT table data")

---

Now in PATCH format with a real commit comment. This version fixes the issues seen by Intel's validation team.

 drivers/acpi/apei/bert.c | 31 +++++++++++++++++++++++--------
 1 file changed, 23 insertions(+), 8 deletions(-)
drivers/acpi/apei/bert.c +23 -8
index 598fd19b65fa..45973aa6e06d 100644
@@ -29,16 +29,26 @@
 
 #undef pr_fmt
 #define pr_fmt(fmt) "BERT: " fmt
+
+#define ACPI_BERT_PRINT_MAX_RECORDS 5
 #define ACPI_BERT_PRINT_MAX_LEN 1024
 
 static int bert_disable;
 
+/*
+ * Print "all" the error records in the BERT table, but avoid huge spam to
+ * the console if the BIOS included oversize records, or too many records.
+ * Skipping some records here does not lose anything because the full
+ * data is available to user tools in:
+ *	/sys/firmware/acpi/tables/data/BERT
+ */
 static void __init bert_print_all(struct acpi_bert_region *region,
 				  unsigned int region_len)
 {
 	struct acpi_hest_generic_status *estatus =
 		(struct acpi_hest_generic_status *)region;
 	int remain = region_len;
+	int printed = 0, skipped = 0;
 	u32 estatus_len;
 
 	while (remain >= sizeof(struct acpi_bert_region)) {
@@ -46,24 +56,26 @@ static void __init bert_print_all(struct acpi_bert_region *region,
 		if (remain < estatus_len) {
 			pr_err(FW_BUG "Truncated status block (length: %u).\n",
 			       estatus_len);
-			return;
+			break;
 		}
 
 		/* No more error records. */
 		if (!estatus->block_status)
-			return;
+			break;
 
 		if (cper_estatus_check(estatus)) {
 			pr_err(FW_BUG "Invalid error record.\n");
-			return;
+			break;
 		}
 
-		pr_info_once("Error records from previous boot:\n");
-		if (region_len < ACPI_BERT_PRINT_MAX_LEN)
+		if (estatus_len < ACPI_BERT_PRINT_MAX_LEN &&
+		    printed < ACPI_BERT_PRINT_MAX_RECORDS) {
+			pr_info_once("Error records from previous boot:\n");
 			cper_estatus_print(KERN_INFO HW_ERR, estatus);
-		else
-			pr_info_once("Max print length exceeded, table data is available at:\n"
-				     "/sys/firmware/acpi/tables/data/BERT");
+			printed++;
+		} else {
+			skipped++;
+		}
 
 		/*
 		 * Because the boot error source is "one-time polled" type,
@@ -75,6 +87,9 @@ static void __init bert_print_all(struct acpi_bert_region *region,
 		estatus = (void *)estatus + estatus_len;
 		remain -= estatus_len;
 	}
+
+	if (skipped)
+		pr_info(HW_ERR "Skipped %d error records\n", skipped);
 }
 
 static int __init setup_bert_disable(char *str)
Signature · 1 line
2.35.3
#4 lore raw

Re: [PATCH] ACPI/APEI: Better fix to avoid spamming the console with old error logs

Quoted 6 lines · Tony Luck

The fix in commit 3f8dec116210 ("ACPI/APEI: Limit printable size of BERT table data") does not work as intended on systems where the BIOS has a fixed size block of memory for the BERT table, relying on s/w to quit when it finds a record with estatus->block_status == 0. On these systems all errors are suppressed because the check:

Quoted 6 lines

if (region_len < ACPI_BERT_PRINT_MAX_LEN)

always fails.

New scheme skips individual CPER records that are too large, and also limits the total number of records that will be printed to 5.

Apologies for the delay.

This seems like a reasonable approach. Working to confirm new behavior on Ampere Altra systems (specifically how region_len and estatus_len are related).

Signature · 2 lines
Darren Hart
Ampere Computing / OS and Kernel
maintainer Applied
#5 lore raw

Re: [PATCH] ACPI/APEI: Better fix to avoid spamming the console with old error logs

In reply to Tony Luck
Quoted 92 lines · Tony Luck

The fix in commit 3f8dec116210 ("ACPI/APEI: Limit printable size of BERT table data") does not work as intended on systems where the BIOS has a fixed size block of memory for the BERT table, relying on s/w to quit when it finds a record with estatus->block_status == 0. On these systems all errors are suppressed because the check:

if (region_len < ACPI_BERT_PRINT_MAX_LEN)

always fails.

New scheme skips individual CPER records that are too large, and also limits the total number of records that will be printed to 5.

Fixes 3f8dec116210 ("ACPI/APEI: Limit printable size of BERT table data")

---

Now in PATCH format with a real commit comment. This version fixes the issues seen by Intel's validation team.

 drivers/acpi/apei/bert.c | 31 +++++++++++++++++++++++--------
 1 file changed, 23 insertions(+), 8 deletions(-)
drivers/acpi/apei/bert.c +20 -8
index 598fd19b65fa..45973aa6e06d 100644
@@ -29,16 +29,26 @@
 
 #undef pr_fmt
 #define pr_fmt(fmt) "BERT: " fmt
+
+#define ACPI_BERT_PRINT_MAX_RECORDS 5
 #define ACPI_BERT_PRINT_MAX_LEN 1024
 
 static int bert_disable;
 
+/*
+ * Print "all" the error records in the BERT table, but avoid huge spam to
+ * the console if the BIOS included oversize records, or too many records.
+ * Skipping some records here does not lose anything because the full
+ * data is available to user tools in:
+ *     /sys/firmware/acpi/tables/data/BERT
+ */
 static void __init bert_print_all(struct acpi_bert_region *region,
                                  unsigned int region_len)
 {
        struct acpi_hest_generic_status *estatus =
                (struct acpi_hest_generic_status *)region;
        int remain = region_len;
+       int printed = 0, skipped = 0;
        u32 estatus_len;
 
        while (remain >= sizeof(struct acpi_bert_region)) {
@@ -46,24 +56,26 @@ static void __init bert_print_all(struct acpi_bert_region *region,
                if (remain < estatus_len) {
                        pr_err(FW_BUG "Truncated status block (length: %u).\n",
                               estatus_len);
-                       return;
+                       break;
                }
 
                /* No more error records. */
                if (!estatus->block_status)
-                       return;
+                       break;
 
                if (cper_estatus_check(estatus)) {
                        pr_err(FW_BUG "Invalid error record.\n");
-                       return;
+                       break;
                }
 
-               pr_info_once("Error records from previous boot:\n");
-               if (region_len < ACPI_BERT_PRINT_MAX_LEN)
+               if (estatus_len < ACPI_BERT_PRINT_MAX_LEN &&
+                   printed < ACPI_BERT_PRINT_MAX_RECORDS) {
+                       pr_info_once("Error records from previous boot:\n");
                        cper_estatus_print(KERN_INFO HW_ERR, estatus);
-               else
-                       pr_info_once("Max print length exceeded, table data is available at:\n"
-                                    "/sys/firmware/acpi/tables/data/BERT");
+                       printed++;
+               } else {
+                       skipped++;
+               }
 
                /*
                 * Because the boot error source is "one-time polled" type,

drivers/acpi/apei/bert.c static void __init bert_print_all(struct acpi_bert_region *region,

Quoted 10 lines
drivers/acpi/apei/bert.c +3 -1
@@ -75,6 +87,9 @@ static void __init bert_print_all(struct acpi_bert_region *region,
                estatus = (void *)estatus + estatus_len;
                remain -= estatus_len;
        }
+
+       if (skipped)
+               pr_info(HW_ERR "Skipped %d error records\n", skipped);
 }
 
 static int __init setup_bert_disable(char *str)
--

Applied as 5.20 material, thanks!