skd: disable discard support
PATCH 1/1 skd: disable discard support
The skd driver has never handled discards reliably.
The kernel will BUG as a result of issuing discards to the skd device. Disable the skd driver's discard support until it is proven reliable.
The device-mapper-test-suite test that exposed this bug just issues a discard that covers a portion of the skd device that was previously written through a dm-thin device. The discard spans the entire 1GB thin device (logical sector 0 through 2097152).
dmtest run --profile stec --suite thin-provisioning -n /discard_fully_provisioned_device/
associated device-mapper-test-suite ruby test code follows:
def test_discard_fully_provisioned_device
with_standard_pool(@size) do |pool|
with_new_thins(pool, @volume_size, 0, 1) do |thin, thin2|
wipe_device(thin)
wipe_device(thin2)
assert_used_blocks(pool, 2 * @blocks_per_dev) thin.discard(0, @volume_size) assert_used_blocks(pool, @blocks_per_dev)
end
end
...
--- drivers/block/skd_main.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
drivers/block/skd_main.c +3 -0
@@ -4441,12 +4441,15 @@ static int skd_cons_disk(struct skd_device *skdev)/* set sysfs ptimal_io_size to 8K */ blk_queue_io_opt (q,8192 ); +#if 0 +/* FIXME: Disable discard support until it no longer BUGs */ /* DISCARD Flag initialization. */ q->limits.discard_granularity =8192 ; q->limits.discard_alignment =0 ; q->limits.max_discard_sectors =UINT_MAX >>9 ; q->limits.discard_zeroes_data =1 ;queue_flag_set_unlocked (QUEUE_FLAG_DISCARD , q); +#endif queue_flag_set_unlocked (QUEUE_FLAG_NONROT , q);spin_lock_irqsave (&skdev->lock, flags);
Quoted 4 lines · Mike Snitzer
The skd driver has never handled discards reliably.
The kernel will BUG as a result of issuing discards to the skd device. Disable the skd driver's discard support until it is proven reliable.
Here is the first BUG I recently saw:
------------[ cut here ]------------ Uhhuh. NMI received for unknown reason 21 on CPU 0.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue
kernel BUG at include/linux/scatterlist.h:65!
invalid opcode: 0000 [#1] SMP
Modules linked in: dm_thin_pool(O) dm_bio_prison(O) dm_persistent_data(O) dm_bufio(O) dm_mod(O) libcrc32c ebtable_nat ebtables xt_CHECKSUM iptable_mangle bridge autofs4 target_core_iblock target_core_file target_core_pscsi target_core_mod configfs bnx2fc fcoe libfcoe libfc 8021q scsi_transport_fc garp scsi_tgt stp llc sunrpc cpufreq_ondemand ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan vhost tun kvm_intel kvm iTCO_wdt iTCO_vendor_support microcode i2c_i801 lpc_ich mfd_core igb i2c_algo_bit i2c_core i7core_edac edac_core ixgbe dca ptp pps_core mdio ses enclosure sg acpi_cpufreq ext4 jbd2 mbcache sr_mod cdrom pata_acpi ata_generic ata_piix skd sd_mod crc_t10dif crct10dif_common megaraid_sas [last unloaded: dm_mod] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G W O 3.14.0-rc1.snitm+ #4 Hardware name: FUJITSU PRIMERGY RX300 S6 /D2619, BIOS 6.00 Rev. 1.10.2619.N1 05/24/2011 task: ffff8803329aef20 ti: ffff8803329b0000 task.ti: ffff8803329b0000 RIP: 0010:[<ffffffff8125a481>] [<ffffffff8125a481>] blk_rq_map_sg+0x241/0x3f0 RSP: 0018:ffff88033fca3c38 EFLAGS: 00010002 RAX: ffffea000b2984f0 RBX: 0000000000010000 RCX: ffff8803286f6020 RDX: ffffea000b2984f0 RSI: 0000000000000000 RDI: ffff8803286f6000 RBP: ffff88033fca3cc8 R08: ffff8803290233c0 R09: 53538ec752528dc6 R10: ffff88032826f8e0 R11: 00000000555590c9 R12: 0000000000000000 R13: 0000000000000000 R14: 0000000000010000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88033fca0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000003bfd2f5170 CR3: 0000000001a0b000 CR4: 00000000000007e0 Stack: 0000000000000002 0000000000000000 ffff88033fca3cf8 ffffffff8112f67e ffff88033ffd7d80 ffff88032826f8e0 ffff8802555590c9 53538ec752528dc6 ffff8802b66bbb28 ffff8803286f6000 ffff8803290233c0 0000000101fd8b08 Call Trace: <IRQ> [<ffffffff8112f67e>] ? __alloc_pages_nodemask+0x12e/0x250 [<ffffffffa0071146>] skd_preop_sg_list+0x46/0x270 [skd] [<ffffffff811703f2>] ? alloc_pages_current+0xb2/0x170 [<ffffffffa0072997>] skd_request_fn+0x287/0x900 [skd] [<ffffffffa007551e>] ? skd_isr_completion_posted+0x1ee/0x5d0 [skd] [<ffffffffa0076233>] skd_isr+0x1a3/0x280 [skd] [<ffffffff810a73ed>] handle_irq_event_percpu+0x6d/0x200 [<ffffffff810a75c2>] handle_irq_event+0x42/0x70 [<ffffffff810aad19>] handle_edge_irq+0x69/0x120 [<ffffffff81005aec>] handle_irq+0x5c/0x150 [<ffffffff815471f2>] ? __atomic_notifier_call_chain+0x12/0x20 [<ffffffff81547216>] ? atomic_notifier_call_chain+0x16/0x20 [<ffffffff8154da1e>] do_IRQ+0x5e/0x110 [<ffffffff8154376a>] common_interrupt+0x6a/0x6a <EOI> [<ffffffff8144d6c3>] ? cpuidle_enter_state+0x53/0xd0 [<ffffffff8144d6bf>] ? cpuidle_enter_state+0x4f/0xd0 [<ffffffff8144d887>] cpuidle_idle_call+0xc7/0x160 [<ffffffff8100cf5e>] arch_cpu_idle+0xe/0x30 [<ffffffff810a696a>] cpu_idle_loop+0x9a/0x240 [<ffffffff810b9e64>] ? clockevents_register_device+0xc4/0x130 [<ffffffff810a6b33>] cpu_startup_entry+0x23/0x30 [<ffffffff81032d5a>] start_secondary+0x7a/0x80 Code: 41 5f c9 c3 66 0f 1f 44 00 00 44 29 f3 44 89 f2 44 89 de 4c 89 c8 eb 93 66 90 48 8b 4d b8 41 f6 c1 03 48 8b 01 0f 84 03 ff ff ff <0f> 0b eb fe 0f 1f 00 48 8b 45 c0 4c 8b 85 78 ff ff ff 48 8b b0 RIP [<ffffffff8125a481>] blk_rq_map_sg+0x241/0x3f0 RSP <ffff88033fca3c38> ---[ end trace 61da6cb864bf7eb8 ]--- Kernel panic - not syncing: Fatal exception in interrupt
Shutting down cpus with NMI
Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff)
Quoted 9 lines · Mike Snitzer
Quoted 4 lines · Mike Snitzer
The skd driver has never handled discards reliably.
The kernel will BUG as a result of issuing discards to the skd device. Disable the skd driver's discard support until it is proven reliable.
Here is the first BUG I recently saw:
And a 2nd:
Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 10 CPU: 10 PID: 0 Comm: swapper/10 Tainted: G W O 3.14.0-rc1.snitm+ #4 Hardware name: FUJITSU PRIMERGY RX300 S6 /D2619, BIOS 6.00 Rev. 1.10.2619.N1 05/24/2011 0000000000000000 ffff88033fd47bb8 ffffffff8153f180 000000000000fffa ffffffff817d8778 ffff88033fd47c38 ffffffff8153ef0d 0000000000000010 ffff88033fd47c48 ffff88033fd47be8 0000000000000000 0000000000000000 Call Trace: <NMI> [<ffffffff8153f180>] dump_stack+0x49/0x61 [<ffffffff8153ef0d>] panic+0xbb/0x1d5 [<ffffffff810e8761>] watchdog_overflow_callback+0xb1/0xc0 [<ffffffff8111e9b8>] __perf_event_overflow+0x98/0x220 [<ffffffff8111f2a4>] perf_event_overflow+0x14/0x20 [<ffffffff8102012e>] intel_pmu_handle_irq+0x1de/0x3c0 [<ffffffff8115f931>] ? unmap_kernel_range_noflush+0x11/0x20 [<ffffffff8131a5c5>] ? ghes_copy_tofrom_phys+0xe5/0x200 [<ffffffff81544e84>] perf_event_nmi_handler+0x34/0x60 [<ffffffff8154464a>] nmi_handle+0x8a/0x170 [<ffffffff81544848>] default_do_nmi+0x68/0x210 [<ffffffff81544a80>] do_nmi+0x90/0xe0 [<ffffffff81543ca7>] end_repeat_nmi+0x1e/0x2e [<ffffffffa06ef7a0>] ? skd_timer_tick_not_online+0x330/0x330 [skd] [<ffffffff815432a1>] ? _raw_spin_lock_irqsave+0x21/0x30 [<ffffffff815432a1>] ? _raw_spin_lock_irqsave+0x21/0x30 [<ffffffff815432a1>] ? _raw_spin_lock_irqsave+0x21/0x30 <<EOE>> <IRQ> [<ffffffffa06ef7d9>] skd_timer_tick+0x39/0x1e0 [skd] [<ffffffff81069480>] ? __queue_work+0x360/0x360 [<ffffffffa06ef7a0>] ? skd_timer_tick_not_online+0x330/0x330 [skd] [<ffffffff8105a318>] call_timer_fn+0x48/0x120 [<ffffffff8105aef5>] run_timer_softirq+0x225/0x290 [<ffffffffa06ef7a0>] ? skd_timer_tick_not_online+0x330/0x330 [skd] [<ffffffff8105365c>] __do_softirq+0xfc/0x2b0 [<ffffffff810bc09f>] ? tick_do_update_jiffies64+0x9f/0xd0 [<ffffffff8105390d>] irq_exit+0xbd/0xd0 [<ffffffff8154dbea>] smp_apic_timer_interrupt+0x4a/0x5a [<ffffffff8154c8ca>] apic_timer_interrupt+0x6a/0x70 <EOI> [<ffffffff8144d710>] ? cpuidle_enter_state+0xa0/0xd0 [<ffffffff8144d6cb>] ? cpuidle_enter_state+0x5b/0xd0 [<ffffffff8144d887>] cpuidle_idle_call+0xc7/0x160 [<ffffffff8100cf5e>] arch_cpu_idle+0xe/0x30 [<ffffffff810a696a>] cpu_idle_loop+0x9a/0x240 [<ffffffff810b9e64>] ? clockevents_register_device+0xc4/0x130 [<ffffffff810a6b33>] cpu_startup_entry+0x23/0x30 [<ffffffff81032d5a>] start_secondary+0x7a/0x80 Shutting down cpus with NMI Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff) ------------[ cut here ]------------ WARNING: CPU: 10 PID: 0 at arch/x86/kernel/smp.c:124 native_smp_send_reschedule+0x5f/0x70() Modules linked in: skd(O) dm_thin_pool(O) dm_bio_prison(O) dm_persistent_data(O) dm_bufio(O) libcrc32c ebtable_nat ebtables xt_CHECKSUM iptable_mangle bridge autofs4 target_core_iblock target_core_file target_core_pscsi target_core_mod configfs bnx2fc fcoe libfcoe 8021q libfc garp stp scsi_transport_fc llc scsi_tgt sunrpc cpufreq_ondemand ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan vhost tun kvm_intel kvm iTCO_wdt iTCO_vendor_support microcode i2c_i801 lpc_ich mfd_core igb i2c_algo_bit i2c_core i7core_edac edac_core ixgbe dca ptp pps_core mdio ses enclosure sg acpi_cpufreq ext4 jbd2 mbcache sr_mod cdrom pata_acpi ata_generic ata_piix sd_mod crc_t10dif crct10dif_common dm_mirror dm_region_hash dm_log dm_mod megaraid_sas [last unloaded: skd] CPU: 10 PID: 0 Comm: swapper/10 Tainted: G W O 3.14.0-rc1.snitm+ #4 Hardware name: FUJITSU PRIMERGY RX300 S6 /D2619, BIOS 6.00 Rev. 1.10.2619.N1 05/24/2011 000000000000007c ffff88033fd478c0 ffffffff8153f180 000000000000007c 0000000000000000 ffff88033fd47900 ffffffff8104e9bc ffff88033fd52c40 ffff88033fc52c40 0000000000000002 ffff88033fd52c40 ffff8803329be250 Call Trace: <NMI> [<ffffffff8153f180>] dump_stack+0x49/0x61 [<ffffffff8104e9bc>] warn_slowpath_common+0x8c/0xc0 [<ffffffff8104ea0a>] warn_slowpath_null+0x1a/0x20 [<ffffffff8103141f>] native_smp_send_reschedule+0x5f/0x70 [<ffffffff81087e3e>] trigger_load_balance+0x15e/0x200 [<ffffffff8107ccf7>] scheduler_tick+0xa7/0xe0 [<ffffffff8105a031>] update_process_times+0x61/0x80 [<ffffffff8131863c>] ? apei_exec_write_register_value+0x1c/0x20 [<ffffffff810bbfb9>] tick_sched_handle+0x39/0x80 [<ffffffff810bc1e4>] tick_sched_timer+0x54/0x90 [<ffffffff810743be>] __run_hrtimer+0x7e/0x1c0 [<ffffffff810bc190>] ? tick_nohz_handler+0xc0/0xc0 [<ffffffff810747ae>] hrtimer_interrupt+0x10e/0x260 [<ffffffff8103489b>] local_apic_timer_interrupt+0x3b/0x60 [<ffffffff8154dbe5>] smp_apic_timer_interrupt+0x45/0x5a [<ffffffff8154c8ca>] apic_timer_interrupt+0x6a/0x70 [<ffffffff8153efe4>] ? panic+0x192/0x1d5 [<ffffffff8153ef42>] ? panic+0xf0/0x1d5 [<ffffffff810e8761>] watchdog_overflow_callback+0xb1/0xc0 [<ffffffff8111e9b8>] __perf_event_overflow+0x98/0x220 [<ffffffff8111f2a4>] perf_event_overflow+0x14/0x20 [<ffffffff8102012e>] intel_pmu_handle_irq+0x1de/0x3c0 [<ffffffff8115f931>] ? unmap_kernel_range_noflush+0x11/0x20 [<ffffffff8131a5c5>] ? ghes_copy_tofrom_phys+0xe5/0x200 [<ffffffff81544e84>] perf_event_nmi_handler+0x34/0x60 [<ffffffff8154464a>] nmi_handle+0x8a/0x170 [<ffffffff81544848>] default_do_nmi+0x68/0x210 [<ffffffff81544a80>] do_nmi+0x90/0xe0 [<ffffffff81543ca7>] end_repeat_nmi+0x1e/0x2e [<ffffffffa06ef7a0>] ? skd_timer_tick_not_online+0x330/0x330 [skd] [<ffffffff815432a1>] ? _raw_spin_lock_irqsave+0x21/0x30 [<ffffffff815432a1>] ? _raw_spin_lock_irqsave+0x21/0x30 [<ffffffff815432a1>] ? _raw_spin_lock_irqsave+0x21/0x30 <<EOE>> <IRQ> [<ffffffffa06ef7d9>] skd_timer_tick+0x39/0x1e0 [skd] [<ffffffff81069480>] ? __queue_work+0x360/0x360 [<ffffffffa06ef7a0>] ? skd_timer_tick_not_online+0x330/0x330 [skd] [<ffffffff8105a318>] call_timer_fn+0x48/0x120 [<ffffffff8105aef5>] run_timer_softirq+0x225/0x290 [<ffffffffa06ef7a0>] ? skd_timer_tick_not_online+0x330/0x330 [skd] [<ffffffff8105365c>] __do_softirq+0xfc/0x2b0 [<ffffffff810bc09f>] ? tick_do_update_jiffies64+0x9f/0xd0 [<ffffffff8105390d>] irq_exit+0xbd/0xd0 [<ffffffff8154dbea>] smp_apic_timer_interrupt+0x4a/0x5a [<ffffffff8154c8ca>] apic_timer_interrupt+0x6a/0x70 <EOI> [<ffffffff8144d710>] ? cpuidle_enter_state+0xa0/0xd0 [<ffffffff8144d6cb>] ? cpuidle_enter_state+0x5b/0xd0 [<ffffffff8144d887>] cpuidle_idle_call+0xc7/0x160 [<ffffffff8100cf5e>] arch_cpu_idle+0xe/0x30 [<ffffffff810a696a>] cpu_idle_loop+0x9a/0x240 [<ffffffff810b9e64>] ? clockevents_register_device+0xc4/0x130 [<ffffffff810a6b33>] cpu_startup_entry+0x23/0x30 [<ffffffff81032d5a>] start_secondary+0x7a/0x80 ---[ end trace 72a22a0dddd989d3 ]---
Quoted 11 lines · Mike Snitzer
The skd driver has never handled discards reliably.
The kernel will BUG as a result of issuing discards to the skd device. Disable the skd driver's discard support until it is proven reliable.
The device-mapper-test-suite test that exposed this bug just issues a discard that covers a portion of the skd device that was previously written through a dm-thin device. The discard spans the entire 1GB thin device (logical sector 0 through 2097152).
dmtest run --profile stec --suite thin-provisioning -n /discard_fully_provisioned_device/
I retested after applying these linux-block.git commits ontop of 3.14-rc1:
5cb8850c9c4a block: Explicitly handle discard/write same segments 8423ae3d7a3c block: Fix cloning of discard/write same bios
And got this:
request botched: dev skd0: type=1, flags=12248081
sector 8390784, nr/cnr 0/128
bio ffff88033169cba0, biotail ffff88032e42bb60, buffer (null), len 0
------------[ cut here ]------------ kernel BUG at block/blk-core.c:2693! invalid opcode: 0000 [#1] SMP Modules linked in: dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio libcrc32c ebtable_nat ebtables xt_CHECKSUM iptable_mangle bridge autofs4 target_core_iblock t
arget_core_file target_core_pscsi target_core_mod configfs bnx2fc fcoe libfcoe 8021q libfc garp stp llc scsi_transport_fc scsi_tgt sunrpc cpufreq_ondemand ipt_REJECT n f_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables bnx2i cnic uio i pv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan vhost tun kvm_int el kvm iTCO_wdt iTCO_vendor_support microcode i2c_i801 lpc_ich mfd_core igb i2c_algo_bit i2c_core i7core_edac edac_core ixgbe dca ptp pps_core mdio ses enclosure sg ac pi_cpufreq dm_mod ext4 jbd2 mbcache sr_mod cdrom pata_acpi ata_generic ata_piix skd sd_mod crc_t10dif crct10dif_common megaraid_sas
CPU: 2 PID: 0 Comm: swapper/2 Tainted: G W 3.14.0-rc1.snitm+ #5 Hardware name: FUJITSU PRIMERGY RX300 S6 /D2619, BIOS 6.00 Rev. 1.10.2619.N1 05/24/2011 task: ffff88033299e150 ti: ffff8803329a4000 task.ti: ffff8803329a4000 RIP: 0010:[<ffffffff81252f1a>] [<ffffffff81252f1a>] __blk_end_request_all+0x2a/0x40 RSP: 0018:ffff88033fc43cf8 EFLAGS: 00010002 RAX: 0000000000000001 RBX: ffff88032e636ac8 RCX: 0000000000000006 RDX: 0000000000000001 RSI: ffff88033169cba0 RDI: ffff88032ec755c0 RBP: ffff88033fc43cf8 R08: 0000000000000002 R09: 0000000000000000 R10: 00000000000006f3 R11: 0000000000000001 R12: 0000000000000000 R13: ffff88033195faa8 R14: ffff8800ba396000 R15: 0000000000000001 FS: 0000000000000000(0000) GS:ffff88033fc40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000003bfea13000 CR3: 000000032fbdc000 CR4: 00000000000007e0 Stack: ffff88033fc43d58 ffffffffa0037b85 ffff88033fc43d48 ffffffff8129ca09 ffff88033fc43d28 ffff88032e636ac8 ffff8800ba396000 ffff88032e650080 ffff8800ba396000 ffff88032e650080 ffff88032e636ac8 0000000000003c17 Call Trace: <IRQ> [<ffffffffa0037b85>] skd_end_request+0x55/0x160 [skd] [<ffffffff8129ca09>] ? swiotlb_unmap_sg_attrs+0x69/0x80 [<ffffffffa003c513>] skd_isr_completion_posted+0x1e3/0x5d0 [skd] [<ffffffff810930a3>] ? __wake_up+0x53/0x70 [<ffffffffa003d1b2>] skd_isr+0x122/0x280 [skd] [<ffffffff810a73ed>] handle_irq_event_percpu+0x6d/0x200 [<ffffffff810a75c2>] handle_irq_event+0x42/0x70 [<ffffffff810aad19>] handle_edge_irq+0x69/0x120 [<ffffffff81005aec>] handle_irq+0x5c/0x150 [<ffffffff815470f2>] ? __atomic_notifier_call_chain+0x12/0x20 [<ffffffff81547116>] ? atomic_notifier_call_chain+0x16/0x20 [<ffffffff8154d91e>] do_IRQ+0x5e/0x110 [<ffffffff8154366a>] common_interrupt+0x6a/0x6a <EOI> [<ffffffff8144d5e3>] ? cpuidle_enter_state+0x53/0xd0 [<ffffffff8144d5df>] ? cpuidle_enter_state+0x4f/0xd0 [<ffffffff8144d7a7>] cpuidle_idle_call+0xc7/0x160 [<ffffffff8100cf5e>] arch_cpu_idle+0xe/0x30 [<ffffffff810a696a>] cpu_idle_loop+0x9a/0x240 [<ffffffff810b9e64>] ? clockevents_register_device+0xc4/0x130 [<ffffffff810a6b33>] cpu_startup_entry+0x23/0x30 [<ffffffff81032d5a>] start_secondary+0x7a/0x80 Code: 00 55 48 89 e5 66 66 66 66 90 48 8b 87 78 01 00 00 48 85 c0 75 10 31 c9 8b 57 64 e8 91 ff ff ff 84 c0 75 07 c9 c3 8b 48 64 eb ed <0f> 0b 0f 1f 40 00 eb fa 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 RIP [<ffffffff81252f1a>] __blk_end_request_all+0x2a/0x40 RSP <ffff88033fc43cf8> ---[ end trace 494de22d0f0be0f8 ]---
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.394 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.402 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.405 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.410 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.414 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.417 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.421 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.424 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.428 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.431 msecs
Kernel panic - not syncing: Fatal exception in interrupt
Shutting down cpus with NMI
Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff)