x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.
PATCH v2 1/1 x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.
Linux has logical cpu offline capability. That can be triggered by:
# echo 0 > /sys/devices/system/cpu/cpuX/online
In Intel Architecture, MCE's are broadcasted to all CPUs in the system.
This includes the CPUs marked offline by Linux. Unless the CPU's were removed via an ACPI notification, in which case the cpu's are removed from the cpu_present_map.
This patch ensures offline CPU's don't participate in MCE rendezvous, but simply perform clearing some status bits to ensure a second MCE wont cause automatic shutdown.
Without the patch, mce_start will increment mce_callin, but mce_start would only wait for all online_cpus. So offline cpu's should avoid participating in the rendezvous process.
--- arch/x86/kernel/cpu/mcheck/mce.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
arch/x86/kernel/cpu/mcheck/mce.c +13 -0
@@ -998,7 +998,20 @@ void do_machine_check(struct pt_regs *regs, long error_code)u64 recover_paddr = ~0ull ;int flags =MF_ACTION_REQUIRED ;int lmce =0 ; +unsigned int cpu =smp_processor_id (); + +/* +* if this cpu is offline, just bail out. +*/ +if (cpu_is_offline (cpu)) { +u64 mcgstatus; + mcgstatus =mce_rdmsrl (MSR_IA32_MCG_STATUS ); +if (mcgstatus &MCG_STATUS_RIPV ) { +mce_wrmsrl (MSR_IA32_MCG_STATUS ,0 ); +return ; + } + }ist_enter (regs);this_cpu_inc (mce_exception_count);
Signature · 1 line
2.4.3
Quoted 18 lines · Ashok Raj
Linux has logical cpu offline capability. That can be triggered by:
# echo 0 > /sys/devices/system/cpu/cpuX/onlineIn Intel Architecture, MCE's are broadcasted to all CPUs in the system.
This includes the CPUs marked offline by Linux. Unless the CPU's were removed via an ACPI notification, in which case the cpu's are removed from the
cpu_present_map.This patch ensures offline CPU's don't participate in MCE rendezvous, but simply perform clearing some status bits to ensure a second MCE wont cause automatic shutdown.
Without the patch,
mce_startwill incrementmce_callin, butmce_startwould only wait for allonline_cpus. So offline cpu's should avoid participating in the rendezvous process.
Quoted 6 lines
Reviewed-by Tony Luck <[email protected]>Signed-off-by Ashok Raj <[email protected]>--- arch/x86/kernel/cpu/mcheck/mce.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
Tested on a box here, massaged commit message and queued for 4.4, thanks.
---
From: Ashok Raj <[email protected]>
Date: Thu, 3 Dec 2015 19:16:10 -0500
Subject: [PATCH] x86/mce: Ensure offline CPUs don't participate in rendezvous
process
Intel's MCA implementation broadcasts MCEs to all CPUs on the node. This poses a problem for offlined CPUs which cannot participate in the rendezvous process:
Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler Kernel Offset: disabled Rebooting in 100 seconds..
More specifically, Linux does a soft offline of a CPU when writing a 0 to /sys/devices/system/cpu/cpuX/online, which doesn't prevent the #MC exception from being broadcasted to that CPU.
Ensure that offline CPUs don't participate in the MCE rendezvous and clear the RIP valid status bit so that a second MCE won't cause a shutdown.
Without the patch, mce_start() will increment mce_callin and wait for all CPUs. Offlined CPUs should avoid participating in the rendezvous process altogether.
--- arch/x86/kernel/cpu/mcheck/mce.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
arch/x86/kernel/cpu/mcheck/mce.c +11 -0
@@ -1002,6 +1002,17 @@ void do_machine_check(struct pt_regs *regs, long error_code)int flags =MF_ACTION_REQUIRED ;int lmce =0 ; +/* If this CPU is offline, just bail out. */ +if (cpu_is_offline (smp_processor_id ())) { +u64 mcgstatus; + + mcgstatus =mce_rdmsrl (MSR_IA32_MCG_STATUS ); +if (mcgstatus &MCG_STATUS_RIPV ) { +mce_wrmsrl (MSR_IA32_MCG_STATUS ,0 ); +return ; + } + } +ist_enter (regs);this_cpu_inc (mce_exception_count);
Signature · 7 lines
2.3.5
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Quoted 1 line
Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler
Is that what we printed in this case? ... boy is that a misleading message ... we got *extra* cpus (the offline ones), not "Not all".
Good job we have a fix :-)
-Tony ���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m����������?�I�
Borislav Petkov
maintainerQuoted 6 lines · Tony
Quoted 1 line
Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handlerIs that what we printed in this case? ... boy is that a misleading message ... we got *extra* cpus (the offline ones), not "Not all".
Good job we have a fix :-)
Well, we still have that printk string in there.
And that is incorrect too, because the MCE (at least the one I'm injecting) gets broadcasted to the CPUs on the *node* and not to the whole system.
If we had to be precise, text should say "Not all CPUs which the MCE was broadcasted to entered the exception handler..."
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Quoted 3 lines
And that is incorrect too, because the MCE (at least the one I'm injecting) gets broadcasted to the CPUs on the *node* and not to the whole system.
Which system? What kind of machine check? On Intel we expect machine checks to be broadcast to all logical cpus on all nodes (unless local machine check is enabled, in which case SRAR style machine checks go only to the logical cpu that hit the error).
The code is written to that expectation ... and we don't report things as well if something else happens (like too many or too few cpus showing up).
-Tony ���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m����������?�I�
Borislav Petkov
maintainerQuoted 10 lines · Tony
Quoted 3 lines
And that is incorrect too, because the MCE (at least the one I'm injecting) gets broadcasted to the CPUs on the *node* and not to the whole system.
Which system? What kind of machine check? On Intel we expect machine checks to be broadcast to all logical cpus on all nodes (unless local machine check is enabled, in which case SRAR style machine checks go only to the logical cpu that hit the error).
The code is written to that expectation ... and we don't report things as well if something else happens (like too many or too few cpus showing up).
Box logs below.
BIOS is doing funny cores enumeration:
node #0, CPUs 0-7
node #1, CPUs 8-15
node #2, CPUs 16-23
node #3, CPUs 24-31
and then starts from node 0 again:
.... node #0, CPUs: #32 #33 #34 #35 #36 #37 #38 #39 .... node #1, CPUs: #40 #41 #42 #43 #44 #45 #46 #47 .... node #2, CPUs: #48 #49 #50 #51 #52 #53 #54 #55 .... node #3, CPUs: #56 #57 #58 #59 #60 #61 #62 #63
So I went and offlined cores 5 and 34 which are on node 0.
Why node 0? Well, when I inject error type 0x10 which is
0x00000010 Memory Uncorrectable non-fatal
it generates an MCE only on the node 0 cores. For that log see the end of this mail. The gist of it is that the CPUs on which #MC gets raised are the cores on node 0, i.e., 0-7 and 32-39.
Cores 5 and 34 are gone, of course.
I mean, even if the #MC gets raised only on the node, the fix still works.
$ grep -Ei "hardware.*CPU" /tmp/mce | sed 's/^.*CPU//' | sort -n 0: Machine Check Exception: 5 Bank 5: be00000000010090 1: Machine Check Exception: 5 Bank 5: be00000000010090 2: Machine Check Exception: 5 Bank 5: be00000000010090 3: Machine Check Exception: 5 Bank 5: be00000000010090 4: Machine Check Exception: 5 Bank 5: be00000000010090 6: Machine Check Exception: 5 Bank 5: be00000000010090 7: Machine Check Exception: 5 Bank 5: be00000000010090 32: Machine Check Exception: 5 Bank 5: be00000000010090 33: Machine Check Exception: 5 Bank 5: be00000000010090 35: Machine Check Exception: 5 Bank 5: be00000000010090 36: Machine Check Exception: 5 Bank 5: be00000000010090 37: Machine Check Exception: 5 Bank 5: be00000000010090 38: Machine Check Exception: 5 Bank 5: be00000000010090 39: Machine Check Exception: 5 Bank 5: be00000000010090
[ 0.859060] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-4650 0 @ 2.70GHz (family: 0x6, model: 0x2d, stepping: 0x7 ... [ 0.981593] x86: Booting SMP configuration: [ 0.991092] .... node #0, CPUs: #1 [ 1.013485] microcode: CPU1 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.034219] #2 [ 1.049577] microcode: CPU2 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.070309] #3 [ 1.085865] microcode: CPU3 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.106618] #4 [ 1.121978] microcode: CPU4 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.142720] #5 [ 1.158079] microcode: CPU5 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.178833] #6 [ 1.194191] microcode: CPU6 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.214914] #7 [ 1.230471] microcode: CPU7 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.251309] [ 1.254854] .... node #1, CPUs: #8 [ 1.275173] microcode: CPU8 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.390509] #9 [ 1.406859] microcode: CPU9 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.427735] #10 [ 1.444303] microcode: CPU10 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.465343] #11 [ 1.481718] microcode: CPU11 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.502779] #12 [ 1.519156] microcode: CPU12 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.540171] #13 [ 1.556536] microcode: CPU13 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.577587] #14 [ 1.594127] microcode: CPU14 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.615131] #15 [ 1.631471] microcode: CPU15 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.652590] [ 1.656132] .... node #2, CPUs: #16 [ 1.676518] microcode: CPU16 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.791812] #17 [ 1.808189] microcode: CPU17 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.829292] #18 [ 1.845868] microcode: CPU18 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.866925] #19 [ 1.883311] microcode: CPU19 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.904386] #20 [ 1.920765] microcode: CPU20 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.941810] #21 [ 1.958169] microcode: CPU21 microcode updated early to revision 0x710, date = 2013-06-17 [ 1.979242] #22 [ 1.995787] microcode: CPU22 microcode updated early to revision 0x710, date = 2013-06-17 [ 2.016842] #23 [ 2.033182] microcode: CPU23 microcode updated early to revision 0x710, date = 2013-06-17 [ 2.054314] [ 2.057854] .... node #3, CPUs: #24 [ 2.078330] microcode: CPU24 microcode updated early to revision 0x710, date = 2013-06-17 [ 2.193513] #25 [ 2.209874] microcode: CPU25 microcode updated early to revision 0x710, date = 2013-06-17 [ 2.230996] #26 [ 2.247563] microcode: CPU26 microcode updated early to revision 0x710, date = 2013-06-17 [ 2.268627] #27 [ 2.284998] microcode: CPU27 microcode updated early to revision 0x710, date = 2013-06-17 [ 2.306061] #28 [ 2.322437] microcode: CPU28 microcode updated early to revision 0x710, date = 2013-06-17 [ 2.343433] #29 [ 2.359780] microcode: CPU29 microcode updated early to revision 0x710, date = 2013-06-17 [ 2.380855] #30 [ 2.397397] microcode: CPU30 microcode updated early to revision 0x710, date = 2013-06-17 [ 2.418432] #31 [ 2.434759] microcode: CPU31 microcode updated early to revision 0x710, date = 2013-06-17 [ 2.455792] [ 2.459336] .... node #0, CPUs: #32 #33 #34 #35 #36 #37 #38 #39 [ 2.583817] .... node #1, CPUs: #40 #41 #42 #43 #44 #45 #46 #47 [ 2.710873] .... node #2, CPUs: #48 #49 #50 #51 #52 #53 #54 #55 [ 2.838069] .... node #3, CPUs: #56 #57 #58 #59 #60 #61 #62 #63 [ 2.964288] x86: Booted up 4 nodes, 64 CPUs [ 2.974471] smpboot: Total of 64 processors activated (344907.86 BogoMIPS)
[ 5290.635126] Broke affinity for irq 82
[ 5290.643222] Broke affinity for irq 111
[ 5290.651507] Broke affinity for irq 125
[ 5290.664107] smpboot: CPU 5 is now offline
[ 5298.371336] Broke affinity for irq 31
[ 5298.379528] Broke affinity for irq 82
[ 5298.387627] Broke affinity for irq 103
[ 5298.395908] Broke affinity for irq 110
[ 5298.404187] Broke affinity for irq 111
[ 5298.412450] Broke affinity for irq 112
[ 5298.420733] Broke affinity for irq 118
[ 5298.429017] Broke affinity for irq 124
[ 5298.437295] Broke affinity for irq 125
[ 5298.445584] Broke affinity for irq 127
[ 5298.453880] Broke affinity for irq 137
[ 5298.466543] smpboot: CPU 34 is now offline
[ 5302.187338] EINJ: Error INJection is initialized.
[ 5318.897170] Disabling lock debugging due to kernel taint
[ 5318.910775] mce: [Hardware Error]: CPU 37: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5318.931171] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5318.951567] mce: [Hardware Error]: TSC bab9f2d8a4e00 ADDR bb68ec00 MISC 20403ebe86
[ 5318.969835] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC b microcode 710
[ 5318.990959] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5319.003825] EDAC sbridge MC0: CPU 37: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.023215] EDAC sbridge MC0: TSC bab9f2d8a4e00
[ 5319.033036] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86
[ 5319.050338] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC b
[ 5319.069542] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset
:0xc00 grain:32 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5319.122943] mce: [Hardware Error]: CPU 3: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.143355] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5319.163846] mce: [Hardware Error]: TSC bab9f2d8a51c1 ADDR bb68ec00 MISC 20403ebe86
[ 5319.182249] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 6 microcode 710
[ 5319.203539] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5319.216586] EDAC sbridge MC0: CPU 3: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.235994] EDAC sbridge MC0: TSC bab9f2d8a51c1
[ 5319.245814] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86
[ 5319.263348] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 6
[ 5319.283041] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset
:0xc00 grain:32 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5319.337311] mce: [Hardware Error]: CPU 2: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.357960] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8159a4d0> {mutex_lock+0x10/0x27}
[ 5319.378519] mce: [Hardware Error]: TSC bab9f2d8a3feb ADDR bb68ec00 MISC 20403ebe86
[ 5319.397151] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 4 microcode 710
[ 5319.418650] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5319.431902] EDAC sbridge MC0: CPU 2: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.451491] EDAC sbridge MC0: TSC bab9f2d8a3feb
[ 5319.461311] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86
[ 5319.479022] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 4
[ 5319.499014] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset
:0xc00 grain:32 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5319.553209] mce: [Hardware Error]: CPU 6: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.574029] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5319.594953] mce: [Hardware Error]: TSC bab9f2d8a87ea ADDR bb68ec00 MISC 20403ebe86
[ 5319.613756] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC c microcode 710
[ 5319.635431] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5319.648873] EDAC sbridge MC0: CPU 6: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.668661] EDAC sbridge MC0: TSC bab9f2d8a87ea
[ 5319.678483] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86
[ 5319.696422] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC c
[ 5319.716789] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset
:0xc00 grain:32 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5319.771531] mce: [Hardware Error]: CPU 38: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.792743] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5319.813836] mce: [Hardware Error]: TSC bab9f2d8a87ce ADDR bb68ec00 MISC 20403ebe86
[ 5319.832819] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC d microcode 710
[ 5319.854654] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5319.868243] EDAC sbridge MC0: CPU 38: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5319.888366] EDAC sbridge MC0: TSC bab9f2d8a87ce
[ 5319.898186] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86
[ 5319.916192] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC d
[ 5319.936752] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5319.991752] mce: [Hardware Error]: CPU 35: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.013034] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5320.034166] mce: [Hardware Error]: TSC bab9f2d8a59dd ADDR bb68ec00 MISC 20403ebe86
[ 5320.053149] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 7 microcode 710
[ 5320.074972] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5320.088567] EDAC sbridge MC0: CPU 35: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.108688] EDAC sbridge MC0: TSC bab9f2d8a59dd
[ 5320.118511] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86
[ 5320.136527] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 7
[ 5320.157079] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5320.212025] mce: [Hardware Error]: CPU 39: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.233316] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5320.254462] mce: [Hardware Error]: TSC bab9f2d8a4f5c ADDR bb68ec00 MISC 20403ebe86
[ 5320.273455] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC f microcode 710
[ 5320.295303] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5320.308905] EDAC sbridge MC0: CPU 39: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.329026] EDAC sbridge MC0: TSC bab9f2d8a4f5c
[ 5320.338847] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86
[ 5320.356858] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC f
[ 5320.377433] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5320.432474] mce: [Hardware Error]: CPU 7: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.453569] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5320.474703] mce: [Hardware Error]: TSC bab9f2d8a4d60 ADDR bb68ec00 MISC 20403ebe86
[ 5320.493689] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC e microcode 710
[ 5320.515532] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5320.529139] EDAC sbridge MC0: CPU 7: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.549050] EDAC sbridge MC0: TSC bab9f2d8a4d60
[ 5320.558870] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86
[ 5320.576890] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC e
[ 5320.597478] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5320.652525] mce: [Hardware Error]: CPU 36: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.673804] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5320.694918] mce: [Hardware Error]: TSC bab9f2d8a5823 ADDR bb68ec00 MISC 20403ebe86
[ 5320.713916] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 9 microcode 710
[ 5320.735759] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5320.749347] EDAC sbridge MC0: CPU 36: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.769452] EDAC sbridge MC0: TSC bab9f2d8a5823
[ 5320.779273] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86
[ 5320.797296] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 9
[ 5320.817877] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5320.872972] mce: [Hardware Error]: CPU 33: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.894249] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5320.915390] mce: [Hardware Error]: TSC bab9f2d8a5326 ADDR bb68ec00 MISC 20403ebe86
[ 5320.934374] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 3 microcode 710
[ 5320.956222] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5320.969807] EDAC sbridge MC0: CPU 33: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5320.989913] EDAC sbridge MC0: TSC bab9f2d8a5326
[ 5320.999734] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86
[ 5321.017750] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 3
[ 5321.038284] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5321.093686] mce: [Hardware Error]: CPU 1: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5321.114770] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5321.135925] mce: [Hardware Error]: TSC bab9f2d8a5562 ADDR bb68ec00 MISC 20403ebe86
[ 5321.154918] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 2 microcode 710
[ 5321.176765] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5321.190369] EDAC sbridge MC0: CPU 1: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5321.210303] EDAC sbridge MC0: TSC bab9f2d8a5562
[ 5321.220123] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86
[ 5321.238146] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 2
[ 5321.258723] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5321.303358] mce: [Hardware Error]: CPU 4: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5321.324279] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5321.345397] mce: [Hardware Error]: TSC bab9f2d8a572f ADDR bb68ec00 MISC 20403ebe86
[ 5321.364380] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 8 microcode 710
[ 5321.386184] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5321.399729] EDAC sbridge MC0: CPU 4: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5321.419624] EDAC sbridge MC0: TSC bab9f2d8a572f
[ 5321.429445] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86
[ 5321.447454] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 8
[ 5321.467989] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5321.511475] mce: [Hardware Error]: CPU 32: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5321.532587] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5321.553689] mce: [Hardware Error]: TSC bab9f2d8a50f4 ADDR bb68ec00 MISC 20403ebe86
[ 5321.572681] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 1 microcode 710
[ 5321.594500] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5321.608057] EDAC sbridge MC0: CPU 32: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5321.628161] EDAC sbridge MC0: TSC bab9f2d8a50f4
[ 5321.637982] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86
[ 5321.655998] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 1
[ 5321.676524] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5321.720020] mce: [Hardware Error]: CPU 0: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5321.740939] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 5321.762058] mce: [Hardware Error]: TSC bab9f2d8a5034 ADDR bb68ec00 MISC 20403ebe86
[ 5321.781022] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 0 microcode 710
[ 5321.802837] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[ 5321.816395] EDAC sbridge MC0: CPU 0: Machine Check Exception: 5 Bank 5: be00000000010090
[ 5321.836300] EDAC sbridge MC0: TSC bab9f2d8a5034
[ 5321.846121] EDAC sbridge MC0: ADDR bb68ec00 EDAC sbridge MC0: MISC 20403ebe86
[ 5321.864127] EDAC sbridge MC0: PROCESSOR 0:206d7 TIME 1449517966 SOCKET 0 APIC 0
[ 5321.884647] EDAC MC0: 0 UE memory read error on CPU_SrcID#0_Ha#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0xbb68e offset:0xc00 grain:32 - area:DRAM err_code:0001:0090 socket:0 ha:0 channel_mask:1 rank:0)
[ 5321.928136] mce: [Hardware Error]: Machine check: Processor context corrupt
[ 5321.945589] Kernel panic - not syncing: Fatal machine check
[ 5321.985122] Kernel Offset: disabled
[ 5322.008492] Rebooting in 100 seconds..
[ 5421.226077] ACPI MEMORY or I/O RESET_REG.Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Quoted 2 lines · Borislav Petkov
Box logs below.
Do you have the dmidecode strings to find which platform this is?
Quoted 21 lines
BIOS is doing funny cores enumeration:
node #0, CPUs 0-7
node #1, CPUs 8-15
node #2, CPUs 16-23
node #3, CPUs 24-31and then starts from node 0 again:
.... node #0, CPUs: #32 #33 #34 #35 #36 #37 #38 #39 .... node #1, CPUs: #40 #41 #42 #43 #44 #45 #46 #47 .... node #2, CPUs: #48 #49 #50 #51 #52 #53 #54 #55 .... node #3, CPUs: #56 #57 #58 #59 #60 #61 #62 #63So I went and offlined cores 5 and 34 which are on node 0.
Why node 0? Well, when I inject error type
0x10which is
0x00000010Memory Uncorrectable non-fatal
Quoted 8 lines
it generates an MCE only on the node 0 cores. For that log see the end of this mail. The gist of it is that the CPUs on which #MC gets raised are the cores on node 0, i.e., 0-7 and 32-39.
Cores 5 and 34 are gone, of course.
I mean, even if the #MC gets raised only on the node, the fix still works.
Not sure how the fix works.. since we excluded only the ones offline. So unless all online cpu's check in, the code should give you the old behavior.
What does cat /proc/interrupts | grep MCE
In a system broadcasting, all cpu counts should be the same. Since we didn't increment the offline stats, if you were to bring the cpu up, it should be one less than other cpus...
Cheers,
Ashok
Borislav Petkov
maintainerQuoted 5 lines · Ashok
Quoted 2 lines · Borislav Petkov
Box logs below.
Do you have the dmidecode strings to find which platform this is?
Is this enough or you want complete dmidecode dump?
DMI: Intel Corporation LH Pass/S4600LH...., BIOS SE5C600.86B.99.99.2050.043020121425 04/30/2012
Quoted 3 lines
Not sure how the fix works.. since we excluded only the ones offline. So unless all online cpu's check in, the code should give you the old behavior.
Did you miss my statement in my previous mail where I said that the MCE is being raised only on the cores of node 0?
Quoted 1 line
What does cat
/proc/interrupts| grep MCE
Can't. Shell on the box is dead after the injection.
Quoted 3 lines
In a system broadcasting, all cpu counts should be the same. Since we didn't increment the offline stats, if you were to bring the cpu up, it should be one less than other cpus...
See the logs at the end of my previous email. #MC gets raised - or at least output from mce_panic() comes out only - on the cores of node 0.
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Quoted 4 lines · Borislav Petkov
Did you miss my statement in my previous mail where I said that the MCE is being raised only on the cores of node 0?
That's right.. but i think if MCE is only given to node0, then the system would panic eveytime with or without the patch. which is why i got confused.
I somehow misunderstood that with this patch the system didn't panic.
Cheers,
Ashok
Borislav Petkov
maintainerQuoted 10 lines · Ashok
Quoted 4 lines · Borislav Petkov
Did you miss my statement in my previous mail where I said that the MCE is being raised only on the cores of node 0?
That's right.. but i think if MCE is only given to node0, then the system would panic eveytime with or without the patch. which is why i got confused.
I somehow misunderstood that with this patch the system didn't panic.
No, the system did panic in both times. The "strange" observation is that the MCE gets reported only on the cores on node 0. Or at least only the printks from mce_panic() on the cores on node0 reach the serial console.
If we really broadcast only on node0, then that would be a problem if the corrupted data leaves the node and manages to corrupt storage when written out on some of the other nodes. I'm not sure if the kernel panicking the whole system is on time and there's not a small window between the detection and the panicking, in which the corruption might happen.
If so, this'd defeat the purpose of MCE broadcasting but I'm just hypothesizing here.
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Quoted 4 lines
No, the system did panic in both times. The "strange" observation is that the MCE gets reported only on the cores on node 0. Or at least only the printks from
mce_panic()on the cores on node0 reach the serial console.
You only see messages and logs from node0, because the cpus there are the only ones that see any errors logged in their banks.
The cpus on node 1, 2, 3 scan all banks and find nothing, so say nothing. There are no system-wide banks ... just core-wide (in recent generations banks 0-3) and socket-wide (banks >=4). But don't code those numbers into any generic code ... we will change them sooner or later.
-Tony ���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m����������?�I�
Borislav Petkov
maintainerQuoted 9 lines · Tony
Quoted 4 lines
No, the system did panic in both times. The "strange" observation is that the MCE gets reported only on the cores on node 0. Or at least only the printks from
mce_panic()on the cores on node0 reach the serial console.You only see messages and logs from node0, because the cpus there are the only ones that see any errors logged in their banks.
The cpus on node 1, 2, 3 scan all banks and find nothing, so say nothing.
Right, sure, of course. Doh!
Confirmation:
[ 183.840517] mce: do_machine_check: CPU: 30
[ 183.840531] mce: do_machine_check: CPU: 27
[ 183.840536] mce: do_machine_check: CPU: 29
[ 183.840541] mce: do_machine_check: CPU: 56
[ 183.840546] mce: do_machine_check: CPU: 28
[ 183.840548] mce: do_machine_check: CPU: 60
[ 183.840550] mce: do_machine_check: CPU: 24
[ 183.840557] mce: do_machine_check: CPU: 12
[ 183.840561] mce: do_machine_check: CPU: 45
[ 183.840565] mce: do_machine_check: CPU: 59
[ 183.840569] mce: do_machine_check: CPU: 57
[ 183.840572] mce: do_machine_check: CPU: 61
[ 183.840584] mce: do_machine_check: CPU: 0
[ 183.840587] mce: do_machine_check: CPU: 32
[ 183.840593] mce: do_machine_check: CPU: 63
[ 183.840596] mce: do_machine_check: CPU: 31
[ 183.840602] mce: do_machine_check: CPU: 42
[ 183.840606] mce: do_machine_check: CPU: 11
[ 183.840611] mce: do_machine_check: CPU: 41
[ 183.840613] mce: do_machine_check: CPU: 9
[ 183.840617] mce: do_machine_check: CPU: 62
[ 183.840619] mce: do_machine_check: CPU: 25
[ 183.840624] mce: do_machine_check: CPU: 58
[ 183.840627] mce: do_machine_check: CPU: 26
[ 183.840633] mce: do_machine_check: CPU: 5
[ 183.840638] mce: do_machine_check: CPU: 1
[ 183.840642] mce: do_machine_check: CPU: 37
[ 183.840648] mce: do_machine_check: CPU: 15
[ 183.840650] mce: do_machine_check: CPU: 47
[ 183.840653] mce: do_machine_check: CPU: 44
[ 183.840657] mce: do_machine_check: CPU: 14
[ 183.840659] mce: do_machine_check: CPU: 46
[ 183.840666] mce: do_machine_check: CPU: 52
[ 183.840670] mce: do_machine_check: CPU: 50
[ 183.840675] mce: do_machine_check: CPU: 48
[ 183.840677] mce: do_machine_check: CPU: 16
[ 183.840682] mce: do_machine_check: CPU: 54
[ 183.840686] mce: do_machine_check: CPU: 18
[ 183.840692] mce: do_machine_check: CPU: 40
[ 183.840695] mce: do_machine_check: CPU: 8
[ 183.840701] mce: do_machine_check: CPU: 2
[ 183.840705] mce: do_machine_check: CPU: 20
[ 183.840710] mce: do_machine_check: CPU: 13
[ 183.840712] mce: do_machine_check: CPU: 43
[ 183.840716] mce: do_machine_check: CPU: 10
[ 183.840722] mce: do_machine_check: CPU: 3
[ 183.840724] mce: do_machine_check: CPU: 35
[ 183.840727] mce: do_machine_check: CPU: 33
[ 183.840730] mce: do_machine_check: CPU: 34
[ 183.840734] mce: do_machine_check: CPU: 6
[ 183.840738] mce: do_machine_check: CPU: 38
[ 183.840743] mce: do_machine_check: CPU: 53
[ 183.840745] mce: do_machine_check: CPU: 21
[ 183.840750] mce: do_machine_check: CPU: 23
[ 183.840752] mce: do_machine_check: CPU: 55
[ 183.840755] mce: do_machine_check: CPU: 22
[ 183.840759] mce: do_machine_check: CPU: 49
[ 183.840761] mce: do_machine_check: CPU: 17
[ 183.840767] mce: do_machine_check: CPU: 19
[ 183.840770] mce: do_machine_check: CPU: 51
[ 183.840776] mce: do_machine_check: CPU: 39
[ 183.840778] mce: do_machine_check: CPU: 7
[ 183.840784] mce: do_machine_check: CPU: 36
[ 183.840786] mce: do_machine_check: CPU: 4
[ 184.485104] Disabling lock debugging due to kernel taint
[ 184.498006] mce: [Hardware Error]: CPU 32: Machine Check Exception: 5 Bank 5: be00000000010090
[ 184.498023] mce: [Hardware Error]: Machine check events logged
[ 184.531428] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffff8135de7f> {intel_idle+0xbf/0x130}
[ 184.551126] mce: [Hardware Error]: TSC c760ad064ccce ADDR bb68ec00 MISC 421c8c86
[ 184.568358] mce: [Hardware Error]: PROCESSOR 0:206d7 TIME 1449600598 SOCKET 0 APIC 1 microcode 710
[ 184.588862] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR...
mce: [Hardware Error]: CPU 0: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 1: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 2: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 32: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 33: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 34: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 35: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 36: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 37: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 38: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 39: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 3: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 4: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 5: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 6: Machine Check Exception: 5 Bank 5: be00000000010090
mce: [Hardware Error]: CPU 7: Machine Check Exception: 5 Bank 5: be00000000010090
CPUs:
[ 1.103200] x86: Booting SMP configuration: [ 1.112441] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 [ 1.227835] .... node #1, CPUs: #8 #9 #10 #11 #12 #13 #14 #15 [ 1.451861] .... node #2, CPUs: #16 #17 #18 #19 #20 #21 #22 #23 [ 1.674819] .... node #3, CPUs: #24 #25 #26 #27 #28 #29 #30 #31 [ 1.899011] .... node #0, CPUs: #32 #33 #34 #35 #36 #37 #38 #39 [ 2.026616] .... node #1, CPUs: #40 #41 #42 #43 #44 #45 #46 #47 [ 2.152645] .... node #2, CPUs: #48 #49 #50 #51 #52 #53 #54 #55 [ 2.276782] .... node #3, CPUs: #56 #57 #58 #59 #60 #61 #62 #63 [ 2.402263] x86: Booted up 4 nodes, 64 CPUs
Ok, all clear.
Thanks!
Signature · 4 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.Quoted 7 lines · Borislav Petkov
BIOS is doing funny cores enumeration:
node #0, CPUs 0-7
node #1, CPUs 8-15
node #2, CPUs 16-23
node #3, CPUs 24-31
Quoted 6 lines
and then starts from node 0 again:
.... node #0, CPUs: #32 #33 #34 #35 #36 #37 #38 #39 .... node #1, CPUs: #40 #41 #42 #43 #44 #45 #46 #47 .... node #2, CPUs: #48 #49 #50 #51 #52 #53 #54 #55 .... node #3, CPUs: #56 #57 #58 #59 #60 #61 #62 #63
That's normal. BIOS writers are encouraged to list all the hyperthread 0 cpus from each core, and then add the hyperthread 1 cpus later in the table. That way an OS that boots less than all the cpus will get the maximum number of real cores into play.
Quoted 5 lines
0x00000010Memory Uncorrectable non-fatalit generates an MCE only on the node 0 cores. For that log see the end of this mail. The gist of it is that the CPUs on which #MC gets raised are the cores on node 0, i.e., 0-7 and 32-39.
I think all the threads on all the sockets must have shown up in the machine check handler ... but only the ones on socket0 printed anything (they can all see the error in bank5 which is shared across the socket ... but cpus 8-15 etc. will see no errors in any banks ... so will be silent.)
-Tony