Skip to contentSkip to conversation

BUG: KASAN: use-after-free in __list_add_valid+0x81/0xa0 (5.11-rc4)

DISCUSSION Mikhail Gavrilov 2 participants 11 messages Feb 12, 2021 – Mar 16, 2021 1 message not in the archive Memory management

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 4 lines · Hillf Danton

BTW better run the reproducer again with KASAN enabled.

It happened today again with kernel 5.11 rc7 (e0756cfc7d7c) Why not try your patch?

list_del corruption, ffffdef70143e848->next is LIST_POISON1 (dead000000000100)

------------[ cut here ]------------
kernel BUG at lib/list_debug.c:45!
invalid opcode: 0000 [#1] SMP NOPTI
CPU: 13 PID: 263 Comm: kswapd0 Tainted: G        W        ---------
---  5.11.0-0.rc7.20210210gite0756cfc7d7c.150.fc35.x86_64 #1
Hardware name: System manufacturer System Product Name/ROG STRIX
X570-I GAMING, BIOS 3402 01/13/2021
RIP: 0010:__list_del_entry_valid.cold+0xf/0x47
Code: fe ff 0f 0b 48 89 d1 4c 89 c6 4c 89 ca 48 c7 c7 e0 26 64 9e e8
1b 12 fe ff 0f 0b 48 89 fe 48 c7 c7 70 27 64 9e e8 0a 12 fe ff <0f> 0b
48 c7 c7 20 28 64 9e e8 fc 11 fe ff 0f 0b 48 89 f2 48 89 fe
RSP: 0018:ffff9f2180863908 EFLAGS: 00010286
RAX: 000000000000004e RBX: ffff8f74d0fa1000 RCX: 0000000000000000
RDX: ffff8f7c885e9f60 RSI: ffff8f7c885db2a0 RDI: ffff8f7c885db2a0
RBP: ffff8f74d0fa1000 R08: 0000000000000000 R09: ffff9f2180863748
R10: ffff9f2180863740 R11: 0000000000000000 R12: ffff8f758edd8e00
R13: 0000000000012800 R14: ffffdef70143e840 R15: ffff8f758edd8e08
FS:  0000000000000000(0000) GS:ffff8f7c88400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000037203acb1000 CR3: 00000001d5c28000 CR4: 0000000000350ee0
Call Trace:
 z3fold_zpool_malloc+0x3e3/0x780
 ? _raw_spin_unlock+0x1f/0x30
 zswap_frontswap_store+0x43e/0x890
 __frontswap_store+0xc8/0x170
 swap_writepage+0x39/0x70
 pageout+0x125/0x540
 shrink_page_list+0x1329/0x1bc0
 shrink_inactive_list+0x12a/0x440
 shrink_lruvec+0x4a9/0x6d0
 ? super_cache_count+0x79/0xf0
 shrink_node+0x2d1/0x700
 balance_pgdat+0x2f5/0x650
 kswapd+0x21d/0x4d0
 ? do_wait_intr_irq+0xd0/0xd0
 ? balance_pgdat+0x650/0x650
 kthread+0x13a/0x150
 ? __kthread_bind_mask+0x60/0x60
 ret_from_fork+0x22/0x30
Modules linked in: tun snd_seq_dummy snd_hrtimer uinput rfcomm

nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter cmac bnep zstd sunrpc vfat fat hid_logitech_hidpp hid_logitech_dj snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi snd_hda_intel mt76x2u snd_intel_dspcfg soundwire_intel mt76x2_common mt76x02_usb soundwire_generic_allocation mt76_usb intel_rapl_msr iwlmvm snd_soc_core snd_usb_audio intel_rapl_common mt76x02_lib mt76 snd_compress snd_pcm_dmaengine snd_usbmidi_lib soundwire_cadence snd_rawmidi mac80211 snd_hda_codec joydev snd_hda_core uvcvideo ac97_bus snd_hwdep btusb snd_seq
videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_seq_device
btrtl edac_mce_amd btbcm iwlwifi snd_pcm videobuf2_common btintel kvm_amd eeepc_wmi snd_timer bluetooth kvm videodev asus_wmi snd ecdh_generic sparse_keymap irqbypass xpad mc libarc4 sp5100_tco rapl ff_memless cfg80211 wmi_bmof ecc video pcspkr soundcore k10temp i2c_piix4 rfkill acpi_cpufreq binfmt_misc ip_tables amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel cec drm ghash_clmulni_intel igb ccp nvme dca nvme_core i2c_algo_bit wmi pinctrl_amd fuse

---[ end trace a0c35e2a81af0791 ]---
RIP: 0010:__list_del_entry_valid.cold+0xf/0x47
Code: fe ff 0f 0b 48 89 d1 4c 89 c6 4c 89 ca 48 c7 c7 e0 26 64 9e e8
1b 12 fe ff 0f 0b 48 89 fe 48 c7 c7 70 27 64 9e e8 0a 12 fe ff <0f> 0b
48 c7 c7 20 28 64 9e e8 fc 11 fe ff 0f 0b 48 89 f2 48 89 fe
RSP: 0018:ffff9f2180863908 EFLAGS: 00010286
RAX: 000000000000004e RBX: ffff8f74d0fa1000 RCX: 0000000000000000
RDX: ffff8f7c885e9f60 RSI: ffff8f7c885db2a0 RDI: ffff8f7c885db2a0
RBP: ffff8f74d0fa1000 R08: 0000000000000000 R09: ffff9f2180863748
R10: ffff9f2180863740 R11: 0000000000000000 R12: ffff8f758edd8e00
R13: 0000000000012800 R14: ffffdef70143e840 R15: ffff8f758edd8e08
FS:  0000000000000000(0000) GS:ffff8f7c88400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000037203acb1000 CR3: 00000001d5c28000 CR4: 0000000000350ee0
note: kswapd0[263] exited with preempt_count 2

full kernel log: https://pastebin.com/FL1fZLJ0

Signature · 2 lines
Best Regards,
Mike Gavrilov.
#2 lore raw
Quoted 6 lines · On Fri, 12 Feb 2021 18:28:12 +0500 Mikhail Gavrilov wrote:
Quoted 3 lines · Hillf Danton

BTW better run the reproducer again with KASAN enabled.

It happened today again with kernel 5.11 rc7 (e0756cfc7d7c)

Thanks again.

Quoted 1 line

Why not try your patch?

Simply because it was half baked - I was not convinced it was a fix instead of papering over anything.

Quoted 16 lines

list_del corruption, ffffdef70143e848->next is LIST_POISON1 (dead000000000100)

------------[ cut here ]------------
kernel BUG at lib/list_debug.c:45!
invalid opcode: 0000 [#1] SMP NOPTI
CPU: 13 PID: 263 Comm: kswapd0 Tainted: G        W        ---------
---  5.11.0-0.rc7.20210210gite0756cfc7d7c.150.fc35.x86_64 #1
Hardware name: System manufacturer System Product Name/ROG STRIX
X570-I GAMING, BIOS 3402 01/13/2021
RIP: 0010:__list_del_entry_valid.cold+0xf/0x47
Code: fe ff 0f 0b 48 89 d1 4c 89 c6 4c 89 ca 48 c7 c7 e0 26 64 9e e8
1b 12 fe ff 0f 0b 48 89 fe 48 c7 c7 70 27 64 9e e8 0a 12 fe ff <0f> 0b
48 c7 c7 20 28 64 9e e8 fc 11 fe ff 0f 0b 48 89 f2 48 89 fe
RSP: 0018:ffff9f2180863908 EFLAGS: 00010286
RAX: 000000000000004e RBX: ffff8f74d0fa1000 RCX: 0000000000000000
RDX: ffff8f7c885e9f60 RSI: ffff8f7c885db2a0 RDI: ffff8f7c885db2a0
Quoted 8 lines
RBP: ffff8f74d0fa1000 R08: 0000000000000000 R09: ffff9f2180863748
R10: ffff9f2180863740 R11: 0000000000000000 R12: ffff8f758edd8e00
R13: 0000000000012800 R14: ffffdef70143e840 R15: ffff8f758edd8e08
FS:  0000000000000000(0000) GS:ffff8f7c88400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000037203acb1000 CR3: 00000001d5c28000 CR4: 0000000000350ee0
Call Trace:
 z3fold_zpool_malloc+0x3e3/0x780

There is a race producing the list corruption above.

Quoted 59 lines
 ? _raw_spin_unlock+0x1f/0x30
 zswap_frontswap_store+0x43e/0x890
 __frontswap_store+0xc8/0x170
 swap_writepage+0x39/0x70
 pageout+0x125/0x540
 shrink_page_list+0x1329/0x1bc0
 shrink_inactive_list+0x12a/0x440
 shrink_lruvec+0x4a9/0x6d0
 ? super_cache_count+0x79/0xf0
 shrink_node+0x2d1/0x700
 balance_pgdat+0x2f5/0x650
 kswapd+0x21d/0x4d0
 ? do_wait_intr_irq+0xd0/0xd0
 ? balance_pgdat+0x650/0x650
 kthread+0x13a/0x150
 ? __kthread_bind_mask+0x60/0x60
 ret_from_fork+0x22/0x30
Modules linked in: tun snd_seq_dummy snd_hrtimer uinput rfcomm

nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter cmac bnep zstd sunrpc vfat fat hid_logitech_hidpp hid_logitech_dj snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi snd_hda_intel mt76x2u snd_intel_dspcfg soundwire_intel mt76x2_common mt76x02_usb soundwire_generic_allocation mt76_usb intel_rapl_msr iwlmvm snd_soc_core snd_usb_audio intel_rapl_common mt76x02_lib mt76 snd_compress snd_pcm_dmaengine snd_usbmidi_lib soundwire_cadence snd_rawmidi mac80211 snd_hda_codec joydev snd_hda_core uvcvideo ac97_bus snd_hwdep btusb snd_seq
videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_seq_device
btrtl edac_mce_amd btbcm iwlwifi snd_pcm videobuf2_common btintel kvm_amd eeepc_wmi snd_timer bluetooth kvm videodev asus_wmi snd ecdh_generic sparse_keymap irqbypass xpad mc libarc4 sp5100_tco rapl ff_memless cfg80211 wmi_bmof ecc video pcspkr soundcore k10temp i2c_piix4 rfkill acpi_cpufreq binfmt_misc ip_tables amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel cec drm ghash_clmulni_intel igb ccp nvme dca nvme_core i2c_algo_bit wmi pinctrl_amd fuse

---[ end trace a0c35e2a81af0791 ]---
RIP: 0010:__list_del_entry_valid.cold+0xf/0x47
Code: fe ff 0f 0b 48 89 d1 4c 89 c6 4c 89 ca 48 c7 c7 e0 26 64 9e e8
1b 12 fe ff 0f 0b 48 89 fe 48 c7 c7 70 27 64 9e e8 0a 12 fe ff <0f> 0b
48 c7 c7 20 28 64 9e e8 fc 11 fe ff 0f 0b 48 89 f2 48 89 fe
RSP: 0018:ffff9f2180863908 EFLAGS: 00010286
RAX: 000000000000004e RBX: ffff8f74d0fa1000 RCX: 0000000000000000
RDX: ffff8f7c885e9f60 RSI: ffff8f7c885db2a0 RDI: ffff8f7c885db2a0
RBP: ffff8f74d0fa1000 R08: 0000000000000000 R09: ffff9f2180863748
R10: ffff9f2180863740 R11: 0000000000000000 R12: ffff8f758edd8e00
R13: 0000000000012800 R14: ffffdef70143e840 R15: ffff8f758edd8e08
FS:  0000000000000000(0000) GS:ffff8f7c88400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000037203acb1000 CR3: 00000001d5c28000 CR4: 0000000000350ee0
note: kswapd0[263] exited with preempt_count 2
Quoted 5 lines

full kernel log: https://pastebin.com/FL1fZLJ0

Signature · 2 lines
Best Regards,
Mike Gavrilov.

The comment below shows a race instance, though I failed to put things together to see how within two hours. Cut it and see what will come up.

mm/z3fold.c +4 -1
@@ -1129,19 +1129,22 @@ retry:
 	page = NULL;
 	if (can_sleep) {
 		spin_lock(&pool->stale_lock);
+		spin_lock(&pool->lock);
 		zhdr = list_first_entry_or_null(&pool->stale,
 						struct z3fold_header, buddy);
 		/*
-		 * Before allocating a page, let's see if we can take one from
+		 * Before allocating a page, lets see if we can take one from
 		 * the stale pages list. cancel_work_sync() can sleep so we
 		 * limit this case to the contexts where we can sleep
 		 */
 		if (zhdr) {
 			list_del(&zhdr->buddy);
+			spin_unlock(&pool->lock);
 			spin_unlock(&pool->stale_lock);
 			cancel_work_sync(&zhdr->work);
 			page = virt_to_page(zhdr);
 		} else {
+			spin_unlock(&pool->lock);
 			spin_unlock(&pool->stale_lock);
 		}
 	}
#3 lore raw
Quoted 20 lines · Hillf Danton

The comment below shows a race instance, though I failed to put things together to see how within two hours. Cut it and see what will come up.

mm/z3fold.c +2 -1
@@ -1129,19 +1129,22 @@ retry:
        page = NULL;
        if (can_sleep) {
                spin_lock(&pool->stale_lock);
+               spin_lock(&pool->lock);
                zhdr = list_first_entry_or_null(&pool->stale,
                                                struct z3fold_header, buddy);
                /*
-                * Before allocating a page, let's see if we can take one from
+                * Before allocating a page, lets see if we can take one from
                 * the stale pages list. cancel_work_sync() can sleep so we
                 * limit this case to the contexts where we can sleep
                 */
                if (zhdr) {

mm/z3fold.c retry: · line 1150

Quoted 10 lines
mm/z3fold.c +2 -0
                        list_del(&zhdr->buddy);
+                       spin_unlock(&pool->lock);
                        spin_unlock(&pool->stale_lock);
                        cancel_work_sync(&zhdr->work);
                        page = virt_to_page(zhdr);
                } else {
+                       spin_unlock(&pool->lock);
                        spin_unlock(&pool->stale_lock);
                }
        }

Hi,
It happened again with the patch above.
Is anything cleared up now?

[32451.229358] list_add corruption. next->prev should be prev
(ffffd08fbc661cd0), but was ffffffffa7643650. (next=ffff9e4d2848f1c0).
[32451.229395] ------------[ cut here ]------------
[32451.229398] kernel BUG at lib/list_debug.c:23!
[32451.229408] invalid opcode: 0000 [#1] SMP NOPTI
[32451.229414] CPU: 4 PID: 80665 Comm: kworker/u64:0 Tainted: G
W        --------- ---  5.11.0-155.fc35.x86_64+debug #1
[32451.229420] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[32451.229424] Workqueue: zswap3 compact_page_work
[32451.229433] RIP: 0010:__list_add_valid.cold+0xf/0x3f
[32451.229439] Code: 48 c7 c6 24 26 64 a8 48 89 ef 49 c7 c7 ea ff ff
ff e8 e8 71 01 00 e9 fa 10 9e ff 4c 89 c1 48 c7 c7 f0 26 64 a8 e8 50
12 fe ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 a0 27 64 a8 e8 39
12 fe
[32451.229444] RSP: 0018:ffffb08fd553fde0 EFLAGS: 00010286
[32451.229449] RAX: 0000000000000075 RBX: ffffe16d871550c0 RCX: 0000000000000000
[32451.229453] RDX: ffff9e53c73e9f60 RSI: ffff9e53c73db2a0 RDI: ffff9e53c73db2a0
[32451.229457] RBP: ffffd08fbc661cd0 R08: 0000000000000000 R09: ffffb08fd553fc20
[32451.229460] R10: ffffb08fd553fc18 R11: 0000000000000000 R12: ffff9e4ce4e29008
[32451.229464] R13: ffff9e4d85543010 R14: ffff9e4d2848f1c0 R15: ffff9e4d85543000
[32451.229468] FS:  0000000000000000(0000) GS:ffff9e53c7200000(0000)
knlGS:0000000000000000
[32451.229472] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[32451.229476] CR2: 00000fb21455dfe8 CR3: 0000000142968000 CR4: 0000000000350ee0
[32451.229480] Call Trace:
[32451.229485]  do_compact_page+0x28d/0xb60
[32451.229492]  ? debug_object_deactivate+0x55/0x140
[32451.229499]  ? lock_release+0x1e9/0x400
[32451.229505]  ? lock_release+0x1e9/0x400
[32451.229511]  process_one_work+0x2b0/0x5e0
[32451.229519]  worker_thread+0x55/0x3c0
[32451.229524]  ? process_one_work+0x5e0/0x5e0
[32451.229531]  kthread+0x13a/0x150
[32451.229540]  ? __kthread_bind_mask+0x60/0x60
[32451.229548]  ret_from_fork+0x22/0x30
[32451.229558] Modules linked in: snd_seq_dummy snd_hrtimer tun uinput

nls_utf8 isofs rfcomm netconsole nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd sunrpc xpad ff_memless vfat fat hid_logitech_hidpp hid_logitech_dj joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi uvcvideo videobuf2_vmalloc snd_hda_intel videobuf2_memops videobuf2_v4l2 snd_intel_dspcfg mt76x2u soundwire_intel mt76x2_common videobuf2_common mt76x02_usb soundwire_generic_allocation mt76_usb intel_rapl_msr intel_rapl_common iwlmvm snd_soc_core snd_usb_audio videodev snd_compress mt76x02_lib btusb snd_usbmidi_lib snd_pcm_dmaengine mt76 soundwire_cadence btrtl snd_rawmidi btbcm snd_hda_codec mc btintel edac_mce_amd mac80211 kvm_amd snd_hda_core uas bluetooth ac97_bus

[32451.229628]  iwlwifi snd_hwdep usb_storage kvm snd_seq libarc4

snd_seq_device ecdh_generic cfg80211 ecc irqbypass snd_pcm rapl eeepc_wmi asus_wmi sparse_keymap snd_timer rfkill snd video sp5100_tco wmi_bmof i2c_piix4 k10temp soundcore acpi_cpufreq binfmt_misc ip_tables amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched crct10dif_pclmul crc32_pclmul drm_kms_helper crc32c_intel cec drm igb ghash_clmulni_intel nvme ccp dca nvme_core i2c_algo_bit wmi pinctrl_amd fuse

[32451.229696] ---[ end trace 80d86d6942435514 ]---
[32451.229701] RIP: 0010:__list_add_valid.cold+0xf/0x3f
[32451.229706] Code: 48 c7 c6 24 26 64 a8 48 89 ef 49 c7 c7 ea ff ff
ff e8 e8 71 01 00 e9 fa 10 9e ff 4c 89 c1 48 c7 c7 f0 26 64 a8 e8 50
12 fe ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 a0 27 64 a8 e8 39
12 fe
[32451.229710] RSP: 0018:ffffb08fd553fde0 EFLAGS: 00010286
[32451.229715] RAX: 0000000000000075 RBX: ffffe16d871550c0 RCX: 0000000000000000
[32451.229721] RDX: ffff9e53c73e9f60 RSI: ffff9e53c73db2a0 RDI: ffff9e53c73db2a0
[32451.229725] RBP: ffffd08fbc661cd0 R08: 0000000000000000 R09: ffffb08fd553fc20
[32451.229729] R10: ffffb08fd553fc18 R11: 0000000000000000 R12: ffff9e4ce4e29008
[32451.229732] R13: ffff9e4d85543010 R14: ffff9e4d2848f1c0 R15: ffff9e4d85543000
[32451.229736] FS:  0000000000000000(0000) GS:ffff9e53c7200000(0000)
knlGS:0000000000000000
[32451.229740] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[32451.229744] CR2: 00000fb21455dfe8 CR3: 0000000142968000 CR4: 0000000000350ee0
[32451.229748] note: kworker/u64:0[80665] exited with preempt_count 2
[32476.846645] watchdog: BUG: soft lockup - CPU#0 stuck for 22s!
[vivaldi-bin:6991]
[32476.846658] Modules linked in: snd_seq_dummy snd_hrtimer tun uinput

nls_utf8 isofs rfcomm netconsole nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd sunrpc xpad ff_memless vfat fat hid_logitech_hidpp hid_logitech_dj joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi uvcvideo videobuf2_vmalloc snd_hda_intel videobuf2_memops videobuf2_v4l2 snd_intel_dspcfg mt76x2u soundwire_intel mt76x2_common videobuf2_common mt76x02_usb soundwire_generic_allocation mt76_usb intel_rapl_msr intel_rapl_common iwlmvm snd_soc_core snd_usb_audio videodev snd_compress mt76x02_lib btusb snd_usbmidi_lib snd_pcm_dmaengine mt76 soundwire_cadence btrtl snd_rawmidi btbcm snd_hda_codec mc btintel edac_mce_amd mac80211 kvm_amd snd_hda_core uas bluetooth ac97_bus

[32476.846704]  iwlwifi snd_hwdep usb_storage kvm snd_seq libarc4

snd_seq_device ecdh_generic cfg80211 ecc irqbypass snd_pcm rapl eeepc_wmi asus_wmi sparse_keymap snd_timer rfkill snd video sp5100_tco wmi_bmof i2c_piix4 k10temp soundcore acpi_cpufreq binfmt_misc ip_tables amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched crct10dif_pclmul crc32_pclmul drm_kms_helper crc32c_intel cec drm igb ghash_clmulni_intel nvme ccp dca nvme_core i2c_algo_bit wmi pinctrl_amd fuse

[32476.846874] irq event stamp: 0
[32476.846877] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[32476.846883] hardirqs last disabled at (0): [<ffffffffa70ddccb>]
copy_process+0x8fb/0x1de0
[32476.846889] softirqs last  enabled at (0): [<ffffffffa70ddccb>]
copy_process+0x8fb/0x1de0
[32476.846892] softirqs last disabled at (0): [<0000000000000000>] 0x0
[32476.846896] CPU: 0 PID: 6991 Comm: vivaldi-bin Tainted: G      D W
      --------- ---  5.11.0-155.fc35.x86_64+debug #1
[32476.846900] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[32476.846904] RIP: 0010:native_queued_spin_lock_slowpath+0x1ce/0x200
[32476.846909] Code: c1 ef 12 83 e0 03 83 ef 01 48 c1 e0 05 48 63 ff
48 05 80 f1 1e 00 48 03 04 fd 00 39 6e a8 48 89 08 8b 41 08 85 c0 75
09 f3 90 <8b> 41 08 85 c0 74 f7 48 8b 39 48 85 ff 0f 84 56 ff ff ff 0f
0d 0f
[32476.846913] RSP: 0000:ffffb08fd2937c10 EFLAGS: 00000246
[32476.846917] RAX: 0000000000000000 RBX: ffffe16d9e844240 RCX: ffff9e53c6bef180
[32476.846920] RDX: ffff9e4cc11a3d28 RSI: 0000000000040000 RDI: 000000000000000d
[32476.846923] RBP: ffff9e4cc11a3d28 R08: 0000000000040000 R09: 0000000000000000
[32476.846926] R10: 0000000000000001 R11: 0000000000000000 R12: ffff9e4cc11a3d40
[32476.846929] R13: ffff9e4cc11a3d28 R14: ffff9e4cc11a3d20 R15: 0000000000ddd8d8
[32476.846932] FS:  00007f4b852a0300(0000) GS:ffff9e53c6a00000(0000)
knlGS:0000000000000000
[32476.846935] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[32476.846939] CR2: 00003d0509d83000 CR3: 000000017733a000 CR4: 0000000000350ef0
[32476.846942] Call Trace:
[32476.846946]  do_raw_spin_lock+0x94/0xa0
[32476.846951]  _raw_spin_lock+0x63/0x80
[32476.846955]  zswap_frontswap_load+0x2f/0x2f0
[32476.846960]  ? psi_group_change+0x27d/0x290
[32476.846965]  __frontswap_load+0xc3/0x160
[32476.846969]  swap_readpage+0x1ca/0x3a0
[32476.846974]  swapin_readahead+0x2ee/0x4e0
[32476.846979]  do_swap_page+0x4a4/0x900
[32476.846983]  ? lock_release+0x1e9/0x400
[32476.846987]  ? trace_hardirqs_on+0x1b/0xe0
[32476.846992]  handle_mm_fault+0xe7d/0x19d0
[32476.846997]  do_user_addr_fault+0x1c7/0x4c0
[32476.847003]  exc_page_fault+0x67/0x2a0
[32476.847007]  ? asm_exc_page_fault+0x8/0x30
[32476.847011]  asm_exc_page_fault+0x1e/0x30
[32476.847015] RIP: 0033:0x55a5d9c33379
[32476.847018] Code: 00 00 4d 89 75 00 4c 89 f0 48 25 00 00 fc ff 48
8b 40 08 41 c7 46 03 03 00 00 00 49 8b 4d 00 44 89 61 07 49 8b 5d 00
4d 8b 37 <44> 89 73 0b a9 00 00 04 00 75 1a 83 e0 18 48 85 c0 74 12 49
8b 45
[32476.847022] RSP: 002b:00007fff34882340 EFLAGS: 00010206
[32476.847025] RAX: 0000000000000012 RBX: 00003d0509d82ff5 RCX: 00003d0509d82ff5
[32476.847028] RDX: 000055a5dbb578bb RSI: 0000000000000001 RDI: 0000000000000000
[32476.847031] RBP: 00007fff34882370 R08: 0000000000000000 R09: 0000000000000000
[32476.847034] R10: 00003d0500000000 R11: ffffffff00000000 R12: 0000000000000023
[32476.847037] R13: 0000376895df40a0 R14: 00003d0509d82f7d R15: 0000376895df4080
[32476.849645] watchdog: BUG: soft lockup - CPU#1 stuck for 22s!
[Chrome_ChildIOT:5472]
[32476.849652] Modules linked in: snd_seq_dummy snd_hrtimer tun uinput

nls_utf8 isofs rfcomm netconsole nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd sunrpc xpad ff_memless vfat fat hid_logitech_hidpp hid_logitech_dj joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi uvcvideo videobuf2_vmalloc snd_hda_intel videobuf2_memops videobuf2_v4l2 snd_intel_dspcfg mt76x2u soundwire_intel mt76x2_common videobuf2_common mt76x02_usb soundwire_generic_allocation mt76_usb intel_rapl_msr intel_rapl_common iwlmvm snd_soc_core snd_usb_audio videodev snd_compress mt76x02_lib btusb snd_usbmidi_lib snd_pcm_dmaengine mt76 soundwire_cadence btrtl snd_rawmidi btbcm snd_hda_codec mc btintel edac_mce_amd mac80211 kvm_amd snd_hda_core uas bluetooth ac97_bus

[32476.849687]  iwlwifi snd_hwdep usb_storage kvm snd_seq libarc4

snd_seq_device ecdh_generic cfg80211 ecc irqbypass snd_pcm rapl eeepc_wmi asus_wmi sparse_keymap snd_timer rfkill snd video sp5100_tco wmi_bmof i2c_piix4 k10temp soundcore acpi_cpufreq binfmt_misc ip_tables amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched crct10dif_pclmul crc32_pclmul drm_kms_helper crc32c_intel cec drm igb ghash_clmulni_intel nvme ccp dca nvme_core i2c_algo_bit wmi pinctrl_amd fuse

[32476.849713] irq event stamp: 0
[32476.849715] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[32476.849719] hardirqs last disabled at (0): [<ffffffffa70ddccb>]
copy_process+0x8fb/0x1de0
[32476.849723] softirqs last  enabled at (0): [<ffffffffa70ddccb>]
copy_process+0x8fb/0x1de0
[32476.849726] softirqs last disabled at (0): [<0000000000000000>] 0x0
[32476.849728] CPU: 1 PID: 5472 Comm: Chrome_ChildIOT Tainted: G
D W    L   --------- ---  5.11.0-155.fc35.x86_64+debug #1
[32476.849732] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[32476.849734] RIP: 0010:native_queued_spin_lock_slowpath+0x1ce/0x200
[32476.849738] Code: c1 ef 12 83 e0 03 83 ef 01 48 c1 e0 05 48 63 ff
48 05 80 f1 1e 00 48 03 04 fd 00 39 6e a8 48 89 08 8b 41 08 85 c0 75
09 f3 90 <8b> 41 08 85 c0 74 f7 48 8b 39 48 85 ff 0f 84 56 ff ff ff 0f
0d 0f
[32476.849741] RSP: 0000:ffffb08fc6c4bc10 EFLAGS: 00000246
[32476.849744] RAX: 0000000000000000 RBX: ffffe16d96c11140 RCX: ffff9e53c6def180
[32476.849746] RDX: ffff9e4cc11a3d28 RSI: 0000000000080000 RDI: 0000000000000016
[32476.849749] RBP: ffff9e4cc11a3d28 R08: 0000000000080000 R09: 0000000000000000
[32476.849751] R10: 0000000000000001 R11: 0000000000000000 R12: ffff9e4cc11a3d40
[32476.849753] R13: ffff9e4cc11a3d28 R14: ffff9e4cc11a3d20 R15: 0000000000f89940
[32476.849756] FS:  00007f9a02233640(0000) GS:ffff9e53c6c00000(0000)
knlGS:0000000000000000
[32476.849758] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[32476.849761] CR2: 00002312fd44ecd8 CR3: 00000001618f6000 CR4: 0000000000350ee0
[32476.849763] Call Trace:
[32476.849766]  do_raw_spin_lock+0x94/0xa0
[32476.849769]  _raw_spin_lock+0x63/0x80
[32476.849772]  zswap_frontswap_load+0x2f/0x2f0
[32476.849775]  ? psi_group_change+0x27d/0x290
[32476.849779]  __frontswap_load+0xc3/0x160
[32476.849782]  swap_readpage+0x1ca/0x3a0
[32476.849786]  swapin_readahead+0x450/0x4e0
[32476.849789]  ? lock_release+0x1e9/0x400
[32476.849793]  do_swap_page+0x4a4/0x900
[32476.849796]  ? lock_release+0x1e9/0x400
[32476.849799]  ? trace_hardirqs_on+0x1b/0xe0
[32476.849802]  handle_mm_fault+0xe7d/0x19d0
[32476.849807]  do_user_addr_fault+0x1c7/0x4c0
[32476.849810]  exc_page_fault+0x67/0x2a0
[32476.849813]  ? asm_exc_page_fault+0x8/0x30
[32476.849816]  asm_exc_page_fault+0x1e/0x30
[32476.849819] RIP: 0033:0x555d3e644fe2
[32476.849822] Code: c3 cc cc cc cc cc cc cc 55 48 89 e5 41 57 41 56
53 48 83 ec 68 49 89 fe 4c 8b 3f 48 8b 05 76 4d bd 08 49 8b 1f 48 31
c3 74 67 <48> 33 43 08 49 39 c7 74 4e c7 45 b8 04 00 00 00 c7 45 c8 04
00 00
[32476.849824] RSP: 002b:00007f9a02231ab0 EFLAGS: 00010202
[32476.849827] RAX: fffffffd55160cdb RBX: 00002312fd44ecd0 RCX: 0000000000000005
[32476.850461] RDX: 0000000000000000 RSI: 0000000000000005 RDI: 00002312fcd3f2a0
[32476.850463] RBP: 00007f9a02231b30 R08: 00002312fcfa4003 R09: 00007ffd204a88d0
[32476.850465] R10: 0000000000000000 R11: 0000000000000246 R12: 0000555d3e492590
[32476.850467] R13: 0000555d3e4cd840 R14: 00002312fcd3f2a0 R15: 00002312fd2481e0
[32476.850644] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [brave:5451]
[32476.850652] Modules linked in: snd_seq_dummy snd_hrtimer tun uinput

nls_utf8 isofs rfcomm netconsole nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd sunrpc xpad ff_memless vfat fat hid_logitech_hidpp hid_logitech_dj joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi uvcvideo videobuf2_vmalloc snd_hda_intel videobuf2_memops videobuf2_v4l2 snd_intel_dspcfg mt76x2u soundwire_intel mt76x2_common videobuf2_common mt76x02_usb soundwire_generic_allocation mt76_usb intel_rapl_msr intel_rapl_common iwlmvm snd_soc_core snd_usb_audio videodev snd_compress mt76x02_lib btusb snd_usbmidi_lib snd_pcm_dmaengine mt76 soundwire_cadence btrtl snd_rawmidi btbcm snd_hda_codec mc btintel edac_mce_amd mac80211 kvm_amd snd_hda_core uas bluetooth ac97_bus

[32476.850687]  iwlwifi snd_hwdep usb_storage kvm snd_seq libarc4

snd_seq_device ecdh_generic cfg80211 ecc irqbypass snd_pcm rapl eeepc_wmi asus_wmi sparse_keymap snd_timer rfkill snd video sp5100_tco wmi_bmof i2c_piix4 k10temp soundcore acpi_cpufreq binfmt_misc ip_tables amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched crct10dif_pclmul crc32_pclmul drm_kms_helper crc32c_intel cec drm igb ghash_clmulni_intel nvme ccp dca nvme_core i2c_algo_bit wmi pinctrl_amd fuse

[32476.850714] irq event stamp: 0
[32476.850716] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[32476.850719] hardirqs last disabled at (0): [<ffffffffa70ddccb>]
copy_process+0x8fb/0x1de0
[32476.850723] softirqs last  enabled at (0): [<ffffffffa70ddccb>]
copy_process+0x8fb/0x1de0
[32476.850726] softirqs last disabled at (0): [<0000000000000000>] 0x0

Full kernel log is here: https://pastebin.com/4SbhNp7V

Signature · 2 lines
Best Regards,
Mike Gavrilov.
#4 lore raw
Quoted 27 lines · On Sun, 28 Feb 2021 18:22:21 +0500 Mikhail Gavrilov wrote:
Quoted 26 lines · Hillf Danton

The comment below shows a race instance, though I failed to put things together to see how within two hours. Cut it and see what will come up.

mm/z3fold.c +3 -1
@@ -1129,19 +1129,22 @@ retry:
        page = NULL;
        if (can_sleep) {
                spin_lock(&pool->stale_lock);
+               spin_lock(&pool->lock);
                zhdr = list_first_entry_or_null(&pool->stale,
                                                struct z3fold_header, buddy);
                /*
-                * Before allocating a page, let's see if we can take one from
+                * Before allocating a page, lets see if we can take one from
                 * the stale pages list. cancel_work_sync() can sleep so we
                 * limit this case to the contexts where we can sleep
                 */
                if (zhdr) {
                        list_del(&zhdr->buddy);
+                       spin_unlock(&pool->lock);
                        spin_unlock(&pool->stale_lock);
                        cancel_work_sync(&zhdr->work);
                        page = virt_to_page(zhdr);
                } else {
Quoted 8 lines

mm/z3fold.c retry: · line 1150

Quoted 4 lines
mm/z3fold.c +1 -0
+                       spin_unlock(&pool->lock);
                        spin_unlock(&pool->stale_lock);
                }
        }

Hi,
It happened again with the patch above.

Thanks again.

Quoted 1 line

Is anything cleared up now?

See below.

Quoted 62 lines
[32451.229358] list_add corruption. next->prev should be prev
(ffffd08fbc661cd0), but was ffffffffa7643650. (next=ffff9e4d2848f1c0).
[32451.229395] ------------[ cut here ]------------
[32451.229398] kernel BUG at lib/list_debug.c:23!
[32451.229408] invalid opcode: 0000 [#1] SMP NOPTI
[32451.229414] CPU: 4 PID: 80665 Comm: kworker/u64:0 Tainted: G
W        --------- ---  5.11.0-155.fc35.x86_64+debug #1
[32451.229420] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[32451.229424] Workqueue: zswap3 compact_page_work
[32451.229433] RIP: 0010:__list_add_valid.cold+0xf/0x3f
[32451.229439] Code: 48 c7 c6 24 26 64 a8 48 89 ef 49 c7 c7 ea ff ff
ff e8 e8 71 01 00 e9 fa 10 9e ff 4c 89 c1 48 c7 c7 f0 26 64 a8 e8 50
12 fe ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 a0 27 64 a8 e8 39
12 fe
[32451.229444] RSP: 0018:ffffb08fd553fde0 EFLAGS: 00010286
[32451.229449] RAX: 0000000000000075 RBX: ffffe16d871550c0 RCX: 0000000000000000
[32451.229453] RDX: ffff9e53c73e9f60 RSI: ffff9e53c73db2a0 RDI: ffff9e53c73db2a0
[32451.229457] RBP: ffffd08fbc661cd0 R08: 0000000000000000 R09: ffffb08fd553fc20
[32451.229460] R10: ffffb08fd553fc18 R11: 0000000000000000 R12: ffff9e4ce4e29008
[32451.229464] R13: ffff9e4d85543010 R14: ffff9e4d2848f1c0 R15: ffff9e4d85543000
[32451.229468] FS:  0000000000000000(0000) GS:ffff9e53c7200000(0000)
knlGS:0000000000000000
[32451.229472] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[32451.229476] CR2: 00000fb21455dfe8 CR3: 0000000142968000 CR4: 0000000000350ee0
[32451.229480] Call Trace:
[32451.229485]  do_compact_page+0x28d/0xb60
[32451.229492]  ? debug_object_deactivate+0x55/0x140
[32451.229499]  ? lock_release+0x1e9/0x400
[32451.229505]  ? lock_release+0x1e9/0x400
[32451.229511]  process_one_work+0x2b0/0x5e0
[32451.229519]  worker_thread+0x55/0x3c0
[32451.229524]  ? process_one_work+0x5e0/0x5e0
[32451.229531]  kthread+0x13a/0x150
[32451.229540]  ? __kthread_bind_mask+0x60/0x60
[32451.229548]  ret_from_fork+0x22/0x30
[32451.229558] Modules linked in: snd_seq_dummy snd_hrtimer tun uinput

nls_utf8 isofs rfcomm netconsole nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd sunrpc xpad ff_memless vfat fat hid_logitech_hidpp hid_logitech_dj joydev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi uvcvideo videobuf2_vmalloc snd_hda_intel videobuf2_memops videobuf2_v4l2 snd_intel_dspcfg mt76x2u soundwire_intel mt76x2_common videobuf2_common mt76x02_usb soundwire_generic_allocation mt76_usb intel_rapl_msr intel_rapl_common iwlmvm snd_soc_core snd_usb_audio videodev snd_compress mt76x02_lib btusb snd_usbmidi_lib snd_pcm_dmaengine mt76 soundwire_cadence btrtl snd_rawmidi btbcm snd_hda_codec mc btintel edac_mce_amd mac80211 kvm_amd snd_hda_core uas bluetooth ac97_bus

[32451.229628]  iwlwifi snd_hwdep usb_storage kvm snd_seq libarc4

snd_seq_device ecdh_generic cfg80211 ecc irqbypass snd_pcm rapl eeepc_wmi asus_wmi sparse_keymap snd_timer rfkill snd video sp5100_tco wmi_bmof i2c_piix4 k10temp soundcore acpi_cpufreq binfmt_misc ip_tables amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched crct10dif_pclmul crc32_pclmul drm_kms_helper crc32c_intel cec drm igb ghash_clmulni_intel nvme ccp dca nvme_core i2c_algo_bit wmi pinctrl_amd fuse

[32451.229696] ---[ end trace 80d86d6942435514 ]---

[...]

Quoted 5 lines

Full kernel log is here: https://pastebin.com/4SbhNp7V

Signature · 2 lines
Best Regards,
Mike Gavrilov.

What we learn from your reports is

1/ in z3fold_free(), kref_put() creates the ground zero for the race cases reported,

2/ the stale_lock in combination with lock makes things more complicated than thought.

Instead of dropping something in the zero spot, the fix below goes the road mentioned before in this mail thread - add another list_head in parallel to the buddy and s/buddy/stale_node/ under every case of stale_lock.

mm/z3fold.c +7 -5
@@ -127,6 +127,7 @@ struct z3fold_header {
 	unsigned short first_num:2;
 	unsigned short mapped_count:2;
 	unsigned short foreign_handles:2;
+	struct list_head stale_node;
 };
 
 /**
@@ -429,6 +430,7 @@ static struct z3fold_header *init_z3fold
 	zhdr->slots = slots;
 	zhdr->pool = pool;
 	INIT_LIST_HEAD(&zhdr->buddy);
+	INIT_LIST_HEAD(&zhdr->stale_node);
 	INIT_WORK(&zhdr->work, compact_page_work);
 	return zhdr;
 }
@@ -556,7 +558,7 @@ static void __release_z3fold_page(struct
 		z3fold_page_unlock(zhdr);
 
 	spin_lock(&pool->stale_lock);
-	list_add(&zhdr->buddy, &pool->stale);
+	list_add(&zhdr->stale_node, &pool->stale);
 	queue_work(pool->release_wq, &pool->work);
 	spin_unlock(&pool->stale_lock);
 }
@@ -598,10 +600,10 @@ static void free_pages_work(struct work_
 	spin_lock(&pool->stale_lock);
 	while (!list_empty(&pool->stale)) {
 		struct z3fold_header *zhdr = list_first_entry(&pool->stale,
-						struct z3fold_header, buddy);
+						struct z3fold_header, stale_node);
 		struct page *page = virt_to_page(zhdr);
 
-		list_del(&zhdr->buddy);
+		list_del(&zhdr->stale_node);
 		if (WARN_ON(!test_bit(PAGE_STALE, &page->private)))
 			continue;
 		spin_unlock(&pool->stale_lock);
@@ -1140,14 +1142,14 @@ retry:
 	if (can_sleep) {
 		spin_lock(&pool->stale_lock);
 		zhdr = list_first_entry_or_null(&pool->stale,
-						struct z3fold_header, buddy);
+						struct z3fold_header, stale_node);
 		/*
 		 * Before allocating a page, let's see if we can take one from
 		 * the stale pages list. cancel_work_sync() can sleep so we
 		 * limit this case to the contexts where we can sleep
 		 */
 		if (zhdr) {
-			list_del(&zhdr->buddy);
+			list_del(&zhdr->stale_node);
 			spin_unlock(&pool->stale_lock);
 			cancel_work_sync(&zhdr->work);
 			page = virt_to_page(zhdr);

--

#5 lore raw
Quoted 62 lines · Hillf Danton

What we learn from your reports is

1/ in z3fold_free(), kref_put() creates the ground zero for the race cases reported,

2/ the stale_lock in combination with lock makes things more complicated than thought.

Instead of dropping something in the zero spot, the fix below goes the road mentioned before in this mail thread - add another list_head in parallel to the buddy and s/buddy/stale_node/ under every case of stale_lock.

mm/z3fold.c +6 -4
@@ -127,6 +127,7 @@ struct z3fold_header {
        unsigned short first_num:2;
        unsigned short mapped_count:2;
        unsigned short foreign_handles:2;
+       struct list_head stale_node;
 };
 
 /**
@@ -429,6 +430,7 @@ static struct z3fold_header *init_z3fold
        zhdr->slots = slots;
        zhdr->pool = pool;
        INIT_LIST_HEAD(&zhdr->buddy);
+       INIT_LIST_HEAD(&zhdr->stale_node);
        INIT_WORK(&zhdr->work, compact_page_work);
        return zhdr;
 }
@@ -556,7 +558,7 @@ static void __release_z3fold_page(struct
                z3fold_page_unlock(zhdr);
 
        spin_lock(&pool->stale_lock);
-       list_add(&zhdr->buddy, &pool->stale);
+       list_add(&zhdr->stale_node, &pool->stale);
        queue_work(pool->release_wq, &pool->work);
        spin_unlock(&pool->stale_lock);
 }
@@ -598,10 +600,10 @@ static void free_pages_work(struct work_
        spin_lock(&pool->stale_lock);
        while (!list_empty(&pool->stale)) {
                struct z3fold_header *zhdr = list_first_entry(&pool->stale,
-                                               struct z3fold_header, buddy);
+                                               struct z3fold_header, stale_node);
                struct page *page = virt_to_page(zhdr);
 
-               list_del(&zhdr->buddy);
+               list_del(&zhdr->stale_node);
                if (WARN_ON(!test_bit(PAGE_STALE, &page->private)))
                        continue;
                spin_unlock(&pool->stale_lock);
@@ -1140,14 +1142,14 @@ retry:
        if (can_sleep) {
                spin_lock(&pool->stale_lock);
                zhdr = list_first_entry_or_null(&pool->stale,
-                                               struct z3fold_header, buddy);
+                                               struct z3fold_header, stale_node);
                /*
                 * Before allocating a page, let's see if we can take one from

mm/z3fold.c retry:

Quoted 10 lines
mm/z3fold.c +1 -2
                 * the stale pages list. cancel_work_sync() can sleep so we
                 * limit this case to the contexts where we can sleep
                 */
                if (zhdr) {
-                       list_del(&zhdr->buddy);
+                       list_del(&zhdr->stale_node);
                        spin_unlock(&pool->stale_lock);
                        cancel_work_sync(&zhdr->work);
                        page = virt_to_page(zhdr);
--

The computer with patch above worked for a record time (3 days) without freezing.
https://postimg.cc/VShF5cJN

But after 3 days hangs with follow trace:

[263314.718807] general protection fault, probably for non-canonical
address 0x72c1224000000000: 0000 [#1] SMP NOPTI
[263314.718828] CPU: 3 PID: 476750 Comm: Chrome_IOThread Tainted: G
    W        --------- ---  5.12.0-0.rc1.162.fc35.x86_64+debug #1
[263314.718831] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[263314.718835] RIP: 0010:__list_add_valid+0x3/0x40
[263314.718841] Code: e9 5d ff ff ff b8 f4 ff ff ff e9 53 ff ff ff 48
c7 00 00 00 00 00 c7 40 08 00 00 00 00 e9 6b ff ff ff cc cc cc cc cc
49 89 d0 <48> 8b 52 08 48 39 f2 0f 85 03 cf 62 00 4c 8b 0a 4d 39 c1 0f
85 1f
[263314.718845] RSP: 0018:ffffae4345b5fac0 EFLAGS: 00010282
[263314.718849] RAX: 00000000000003c0 RBX: ffffed09c02ade80 RCX:
0000000000000000
[263314.718851] RDX: 72c1224000000000 RSI: ffffce433c462004 RDI:
ffff9d414ab7a000
[263314.718853] RBP: ffffce433c462004 R08: 72c1224000000000 R09:
0000000000000000
[263314.718856] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff9d4253053008
[263314.718858] R13: ffff9d414ab7a010 R14: 72c1224000000000 R15:
ffff9d414ab7a000
[263314.718860] FS:  00007f8ef636f640(0000) GS:ffff9d4947000000(0000)
knlGS:0000000000000000
[263314.718863] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[263314.718865] CR2: 000055634cbef560 CR3: 00000002dbdbc000 CR4:
0000000000350ee0
[263314.718867] Call Trace:
[263314.718875]  do_compact_page+0x28d/0xb60
[263314.718884]  ? z3fold_zpool_free+0x3a8/0x590
[263314.718888]  zswap_free_entry+0x43/0x70
[263314.718892]  zswap_frontswap_invalidate_page+0x8c/0x90
[263314.718895]  __frontswap_invalidate_page+0x5d/0x90
[263314.718898]  swap_range_free+0xcd/0xf0
[263314.718901]  swapcache_free_entries+0x128/0x1a0
[263314.718904]  free_swap_slot+0xbb/0xd0
[263314.718907]  __swap_entry_free+0x7a/0xa0
[263314.718910]  free_swap_and_cache+0x35/0x80
[263314.718913]  shmem_undo_range+0x188/0x7e0
[263314.718919]  ? ldsem_down_read+0x1f/0x40
[263314.718925]  shmem_evict_inode+0xe6/0x290
[263314.718928]  ? lock_release+0x1ef/0x410
[263314.718932]  ? var_wake_function+0x20/0x20
[263314.718936]  evict+0xcf/0x1d0
[263314.718940]  __dentry_kill+0xe8/0x190
[263314.718943]  ? dput+0x20/0x480
[263314.718946]  dput+0x2b8/0x480
[263314.718949]  __fput+0x102/0x260
[263314.718952]  task_work_run+0x5c/0xa0
[263314.718957]  exit_to_user_mode_prepare+0x232/0x240
[263314.718960]  syscall_exit_to_user_mode+0x27/0x70
[263314.718964]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[263314.718967] RIP: 0033:0x7f8f0b15d16b
[263314.718972] Code: 8b 15 09 7d 0c 00 f7 d8 64 89 02 48 c7 c0 ff ff
ff ff eb 89 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 0b 00 00
00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d5 7c 0c 00 f7 d8 64 89
01 48
[263314.718974] RSP: 002b:00007f8ef636d308 EFLAGS: 00000246 ORIG_RAX:
000000000000000b
[263314.718977] RAX: 0000000000000000 RBX: 00003e1813862928 RCX:
00007f8f0b15d16b
[263314.718979] RDX: 0000000000000000 RSI: 0000000000a4e000 RDI:
00007f8e5b43e000
[263314.718981] RBP: 00007f8ef636d320 R08: 0000000000000000 R09:
0000000000000000
[263314.718983] R10: 0000000000000000 R11: 0000000000000246 R12:
00007f8e5b43e000
[263314.718985] R13: 00003e18138628e0 R14: 00007f8ef636d330 R15:
00007f8ef636d330
[263314.718989] Modules linked in: nft_fib_inet nft_fib_ipv4

nft_fib_ipv6 nft_fib tun snd_seq_dummy snd_hrtimer uinput nls_utf8 isofs rfcomm netconsole nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd sunrpc vfat fat hid_logitech_hidpp hid_logitech_dj snd_hda_codec_realtek snd_hda_codec_generic mt76x2u mt76x2_common ledtrig_audio mt76x02_usb mt76_usb snd_hda_codec_hdmi mt76x02_lib intel_rapl_msr intel_rapl_common iwlmvm mt76 snd_hda_intel snd_intel_dspcfg soundwire_intel soundwire_generic_allocation snd_soc_core edac_mce_amd joydev mac80211 snd_compress snd_pcm_dmaengine snd_usb_audio soundwire_cadence kvm_amd snd_hda_codec btusb btrtl btbcm btintel snd_hda_core snd_usbmidi_lib uvcvideo ac97_bus snd_seq iwlwifi kvm snd_rawmidi bluetooth snd_hwdep videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common

[263314.719032]  eeepc_wmi snd_seq_device asus_wmi snd_pcm videodev

sparse_keymap xpad ecdh_generic mc libarc4 irqbypass wmi_bmof ff_memless cfg80211 video ecc snd_timer rapl sp5100_tco snd i2c_piix4 k10temp soundcore rfkill acpi_cpufreq binfmt_misc uas usb_storage amdgpu drm_ttm_helper ttm iommu_v2 crct10dif_pclmul gpu_sched crc32_pclmul crc32c_intel drm_kms_helper cec drm ghash_clmulni_intel ccp igb nvme dca nvme_core i2c_algo_bit wmi pinctrl_amd fuse [last
unloaded: ip_tables]

[263314.719079] ---[ end trace ba885cda1af90fb7 ]---
[263314.719081] RIP: 0010:__list_add_valid+0x3/0x40
[263314.719084] Code: e9 5d ff ff ff b8 f4 ff ff ff e9 53 ff ff ff 48
c7 00 00 00 00 00 c7 40 08 00 00 00 00 e9 6b ff ff ff cc cc cc cc cc
49 89 d0 <48> 8b 52 08 48 39 f2 0f 85 03 cf 62 00 4c 8b 0a 4d 39 c1 0f
85 1f
[263314.719086] RSP: 0018:ffffae4345b5fac0 EFLAGS: 00010282
[263314.719089] RAX: 00000000000003c0 RBX: ffffed09c02ade80 RCX:
0000000000000000
[263314.719091] RDX: 72c1224000000000 RSI: ffffce433c462004 RDI:
ffff9d414ab7a000
[263314.719093] RBP: ffffce433c462004 R08: 72c1224000000000 R09:
0000000000000000
[263314.719095] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff9d4253053008
[263314.719097] R13: ffff9d414ab7a010 R14: 72c1224000000000 R15:
ffff9d414ab7a000
[263314.719099] FS:  00007f8ef636f640(0000) GS:ffff9d4947000000(0000)
knlGS:0000000000000000
[263314.719101] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[263314.719104] CR2: 000055634cbef560 CR3: 00000002dbdbc000 CR4:
0000000000350ee0
[263314.719106] note: Chrome_IOThread[476750] exited with preempt_count 5
[263341.868981] watchdog: BUG: soft lockup - CPU#0 stuck for 23s!
[ThreadPoolForeg:513140]
[263341.868991] Modules linked in: nft_fib_inet nft_fib_ipv4

nft_fib_ipv6 nft_fib tun snd_seq_dummy snd_hrtimer uinput nls_utf8 isofs rfcomm netconsole nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd sunrpc vfat fat hid_logitech_hidpp hid_logitech_dj snd_hda_codec_realtek snd_hda_codec_generic mt76x2u mt76x2_common ledtrig_audio mt76x02_usb mt76_usb snd_hda_codec_hdmi mt76x02_lib intel_rapl_msr intel_rapl_common iwlmvm mt76 snd_hda_intel snd_intel_dspcfg soundwire_intel soundwire_generic_allocation snd_soc_core edac_mce_amd joydev mac80211 snd_compress snd_pcm_dmaengine snd_usb_audio soundwire_cadence kvm_amd snd_hda_codec btusb btrtl btbcm btintel snd_hda_core snd_usbmidi_lib uvcvideo ac97_bus snd_seq iwlwifi kvm snd_rawmidi bluetooth snd_hwdep videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common

[263341.869025]  eeepc_wmi snd_seq_device asus_wmi snd_pcm videodev

sparse_keymap xpad ecdh_generic mc libarc4 irqbypass wmi_bmof ff_memless cfg80211 video ecc snd_timer rapl sp5100_tco snd i2c_piix4 k10temp soundcore rfkill acpi_cpufreq binfmt_misc uas usb_storage amdgpu drm_ttm_helper ttm iommu_v2 crct10dif_pclmul gpu_sched crc32_pclmul crc32c_intel drm_kms_helper cec drm ghash_clmulni_intel ccp igb nvme dca nvme_core i2c_algo_bit wmi pinctrl_amd fuse [last
unloaded: ip_tables]

[263341.869052] irq event stamp: 0
[263341.869054] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[263341.869057] hardirqs last disabled at (0): [<ffffffffa80dd962>]
copy_process+0x902/0x1df0
[263341.869061] softirqs last  enabled at (0): [<ffffffffa80dd962>]
copy_process+0x902/0x1df0
[263341.869064] softirqs last disabled at (0): [<0000000000000000>] 0x0
[263341.869067] CPU: 0 PID: 513140 Comm: ThreadPoolForeg Tainted: G
  D W        --------- ---  5.12.0-0.rc1.162.fc35.x86_64+debug #1
[263341.869070] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[263341.869073] RIP: 0010:native_queued_spin_lock_slowpath+0x1ce/0x200
[263341.869076] Code: c1 ef 12 83 e0 03 83 ef 01 48 c1 e0 05 48 63 ff
48 05 00 f3 1e 00 48 03 04 fd 20 d9 6e a9 48 89 08 8b 41 08 85 c0 75
09 f3 90 <8b> 41 08 85 c0 74 f7 48 8b 39 48 85 ff 0f 84 56 ff ff ff 0f
0d 0f
[263341.869079] RSP: 0000:ffffae435505bbf0 EFLAGS: 00000246
[263341.869082] RAX: 0000000000000000 RBX: 0000000000e6c4f7 RCX:
ffff9d4946bef300
[263341.869084] RDX: ffff9d424197e4e8 RSI: 0000000000040000 RDI:
0000000000000014
[263341.869087] RBP: ffff9d424197e4e8 R08: 0000000000040000 R09:
0000000000000000
[263341.869089] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff9d424197e500
[263341.869091] R13: ffff9d424197e4e8 R14: ffffffffa9e87020 R15:
ffff9d424197e4e0
[263341.869094] FS:  00007ff2de694640(0000) GS:ffff9d4946a00000(0000)
knlGS:0000000000000000
[263341.869096] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[263341.869099] CR2: 00000070bfa4d3c0 CR3: 00000002f2878000 CR4:
0000000000350ef0
[263341.869101] Call Trace:
[263341.869104]  do_raw_spin_lock+0x94/0xa0
[263341.869107]  _raw_spin_lock+0x63/0x80
[263341.869111]  zswap_frontswap_load+0x30/0x2f0
[263341.869115]  ? trace_hardirqs_on+0x1b/0xe0
[263341.869120]  __frontswap_load+0xc3/0x160
[263341.869123]  swap_readpage+0x25b/0x440
[263341.869127]  swapin_readahead+0x450/0x4e0
[263341.869130]  ? lock_release+0x1ef/0x410
[263341.869134]  do_swap_page+0x4a4/0x900
[263341.869137]  __handle_mm_fault+0xbd6/0x1610
[263341.869140]  ? lock_acquire+0x177/0x3a0
[263341.869145]  handle_mm_fault+0xa2/0x270
[263341.869148]  do_user_addr_fault+0x1ea/0x6b0
[263341.869152]  exc_page_fault+0x67/0x2a0
[263341.869155]  ? asm_exc_page_fault+0x8/0x30
[263341.869158]  asm_exc_page_fault+0x1e/0x30
[263341.869161] RIP: 0033:0x55e1b76f7713
[263341.869164] Code: 8b 9f c8 00 00 00 4d 8b b7 d0 00 00 00 4c 39 f3
74 5a 4c 8d 25 8e 18 3c 05 4c 8d 2d aa 59 e6 fb 0f 1f 80 00 00 00 00
48 8b 3b <48> 8b 07 48 89 c1 4c 29 e1 48 c1 c9 03 48 81 f9 9f 00 00 00
0f 87
[263341.869166] RSP: 002b:00007ff2de693430 EFLAGS: 00010287
[263341.869169] RAX: 0000000000000000 RBX: 00007ff2de693568 RCX:
00000070c00c7be0
[263341.869171] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
00000070bfa4d3c0
[263341.869174] RBP: 00007ff2de693480 R08: 0000000000000000 R09:
00000000000000ca
[263341.869176] R10: 00007fff78981080 R11: 00007fff78981090 R12:
000055e1bcab8f90
[263341.869178] R13: 000055e1b355d0b3 R14: 00007ff2de693578 R15:
00007ff2de693500
[263341.870981] watchdog: BUG: soft lockup - CPU#1 stuck for 23s!
[steamwebhelper:3496089]
[263341.870987] Modules linked in: nft_fib_inet nft_fib_ipv4

nft_fib_ipv6 nft_fib tun snd_seq_dummy snd_hrtimer uinput nls_utf8 isofs rfcomm netconsole nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd sunrpc vfat fat hid_logitech_hidpp hid_logitech_dj snd_hda_codec_realtek snd_hda_codec_generic mt76x2u mt76x2_common ledtrig_audio mt76x02_usb mt76_usb snd_hda_codec_hdmi mt76x02_lib intel_rapl_msr intel_rapl_common iwlmvm mt76 snd_hda_intel snd_intel_dspcfg soundwire_intel soundwire_generic_allocation snd_soc_core edac_mce_amd joydev mac80211 snd_compress snd_pcm_dmaengine snd_usb_audio soundwire_cadence kvm_amd snd_hda_codec btusb btrtl btbcm btintel snd_hda_core snd_usbmidi_lib uvcvideo ac97_bus snd_seq iwlwifi kvm snd_rawmidi bluetooth snd_hwdep videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common

[263341.871021]  eeepc_wmi snd_seq_device asus_wmi snd_pcm videodev

sparse_keymap xpad ecdh_generic mc libarc4 irqbypass wmi_bmof ff_memless cfg80211 video ecc snd_timer rapl sp5100_tco snd i2c_piix4 k10temp soundcore rfkill acpi_cpufreq binfmt_misc uas usb_storage amdgpu drm_ttm_helper ttm iommu_v2 crct10dif_pclmul gpu_sched crc32_pclmul crc32c_intel drm_kms_helper cec drm ghash_clmulni_intel ccp igb nvme dca nvme_core i2c_algo_bit wmi pinctrl_amd fuse [last
unloaded: ip_tables]

[263341.871048] irq event stamp: 0
[263341.871050] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[263341.871054] hardirqs last disabled at (0): [<ffffffffa80dd962>]
copy_process+0x902/0x1df0
[263341.871058] softirqs last  enabled at (0): [<ffffffffa80dd962>]
copy_process+0x902/0x1df0
[263341.871061] softirqs last disabled at (0): [<0000000000000000>] 0x0
[263341.871064] CPU: 1 PID: 3496089 Comm: steamwebhelper Tainted: G
  D W    L   --------- ---  5.12.0-0.rc1.162.fc35.x86_64+debug #1
[263341.871067] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[263341.871069] RIP: 0010:native_queued_spin_lock_slowpath+0x137/0x200
[263341.871073] Code: e0 a9 1d 00 eb cb 41 83 c0 01 c1 e6 10 41 c1 e0
12 44 09 c6 89 f0 c1 e8 10 66 87 42 02 89 c7 c1 e7 10 75 73 31 ff eb
02 f3 90 <8b> 02 66 85 c0 75 f7 41 89 c0 66 45 31 c0 44 39 c6 0f 84 9b
00 00
[263341.871076] RSP: 0018:ffffae43549eb5f8 EFLAGS: 00000202
[263341.871078] RAX: 00000000000c0101 RBX: ffff9d414bc00000 RCX:
ffff9d4946def300
[263341.871081] RDX: ffff9d4253053008 RSI: 0000000000080000 RDI:
0000000000000000
[263341.871083] RBP: ffff9d4253053008 R08: 0000000000080000 R09:
0000000000000000
[263341.871085] R10: 0000000000000000 R11: 0000000000000001 R12:
ffff9d4253053020
[263341.871088] R13: ffff9d414bc00010 R14: 0000000000000000 R15:
ffffed09c02f0000
[263341.871090] FS:  00007f725d561d40(0000) GS:ffff9d4946c00000(0000)
knlGS:0000000000000000
[263341.871093] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[263341.871095] CR2: 0000366b51ea6fe8 CR3: 00000003865f2000 CR4:
0000000000350ee0
[263341.871098] Call Trace:
[263341.871101]  do_raw_spin_lock+0x94/0xa0
[263341.871104]  _raw_spin_lock+0x63/0x80
[263341.871107]  z3fold_page_isolate+0xbd/0x1b0
[263341.871112]  isolate_movable_page+0x94/0x180
[263341.871115]  isolate_migratepages_block+0x5db/0x1120
[263341.871120]  ? lock_release+0x1ef/0x410
[263341.871124]  compact_zone+0x5a4/0xfd0
[263341.871129]  compact_zone_order+0xaa/0xf0
[263341.871134]  try_to_compact_pages+0x111/0x3b0
[263341.871138]  __alloc_pages_direct_compact+0x79/0x210
[263341.871142]  __alloc_pages_slowpath.constprop.0+0x1d0/0xf90
[263341.871147]  ? __alloc_pages_nodemask+0x2e3/0x400
[263341.871151]  ? lock_release+0x1ef/0x410
[263341.871154]  __alloc_pages_nodemask+0x37d/0x400
[263341.871159]  ttm_pool_alloc+0x2a3/0x630 [ttm]
[263341.871167]  ttm_tt_populate+0x37/0xe0 [ttm]
[263341.871172]  ttm_bo_handle_move_mem+0x13a/0x170 [ttm]
[263341.871179]  ttm_bo_validate+0x15f/0x1b0 [ttm]
[263341.871184]  ? lock_release+0x1ef/0x410
[263341.871189]  ttm_bo_init_reserved+0x2f7/0x3e0 [ttm]
[263341.871195]  amdgpu_bo_do_create+0x1a8/0x630 [amdgpu]
[263341.871312]  ? amdgpu_bo_subtract_pin_size+0x50/0x50 [amdgpu]
[263341.871422]  amdgpu_bo_create+0x30/0x2e0 [amdgpu]
[263341.871531]  ? lock_acquire+0x177/0x3a0
[263341.871535]  ? trace_hardirqs_on+0x1b/0xe0
[263341.871539]  ? _raw_spin_unlock_irqrestore+0x37/0x40
[263341.871543]  ? lock_release+0x1ef/0x410
[263341.871547]  amdgpu_gem_create_ioctl+0x10e/0x370 [amdgpu]
[263341.871664]  ? amdgpu_gem_force_release+0x130/0x130 [amdgpu]
[263341.871774]  drm_ioctl_kernel+0x89/0xe0 [drm]
[263341.871797]  drm_ioctl+0x20f/0x3c0 [drm]
[263341.871816]  ? amdgpu_gem_force_release+0x130/0x130 [amdgpu]
[263341.871927]  ? selinux_file_ioctl+0x147/0x200
[263341.871931]  ? lock_acquired+0x200/0x390
[263341.871934]  ? lock_release+0x1ef/0x410
[263341.871937]  ? trace_hardirqs_on+0x1b/0xe0
[263341.871940]  ? _raw_spin_unlock_irqrestore+0x37/0x40
[263341.871944]  amdgpu_drm_ioctl+0x49/0x80 [amdgpu]
[263341.872053]  __x64_sys_ioctl+0x82/0xb0
[263341.872058]  do_syscall_64+0x33/0x40
[263341.872061]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[263341.872065] RIP: 0033:0x7f72610b22bb
[263341.872068] Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff ff ff 5b 5d
4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00
00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 85 bb 0c 00 f7 d8 64 89
01 48
[263341.872071] RSP: 002b:00007ffcd94a01f8 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[263341.872074] RAX: ffffffffffffffda RBX: 00007ffcd94a0250 RCX:
00007f72610b22bb
[263341.872076] RDX: 00007ffcd94a0250 RSI: 00000000c0206440 RDI:
0000000000000016
[263341.872078] RBP: 00000000c0206440 R08: 0000000000000009 R09:
00000000000000b8
[263341.872081] R10: 00007ffcd9568080 R11: 0000000000000246 R12:
000008c86f1ae3c0
[263341.872083] R13: 0000000000000016 R14: 0000000000200000 R15:
00000000019c6000
[263341.872983] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kswapd0:288]
[263341.872991] Modules linked in: nft_fib_inet nft_fib_ipv4

nft_fib_ipv6 nft_fib tun snd_seq_dummy snd_hrtimer uinput nls_utf8 isofs rfcomm netconsole nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd sunrpc vfat fat hid_logitech_hidpp hid_logitech_dj snd_hda_codec_realtek snd_hda_codec_generic mt76x2u mt76x2_common ledtrig_audio mt76x02_usb mt76_usb snd_hda_codec_hdmi mt76x02_lib intel_rapl_msr intel_rapl_common iwlmvm mt76 snd_hda_intel snd_intel_dspcfg soundwire_intel soundwire_generic_allocation snd_soc_core edac_mce_amd joydev mac80211 snd_compress snd_pcm_dmaengine snd_usb_audio soundwire_cadence kvm_amd snd_hda_codec btusb btrtl btbcm btintel snd_hda_core snd_usbmidi_lib uvcvideo ac97_bus snd_seq iwlwifi kvm snd_rawmidi bluetooth snd_hwdep videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common

[263341.873025]  eeepc_wmi snd_seq_device asus_wmi snd_pcm videodev

sparse_keymap xpad ecdh_generic mc libarc4 irqbypass wmi_bmof ff_memless cfg80211 video ecc snd_timer rapl sp5100_tco snd i2c_piix4 k10temp soundcore rfkill acpi_cpufreq binfmt_misc uas usb_storage amdgpu drm_ttm_helper ttm iommu_v2 crct10dif_pclmul gpu_sched crc32_pclmul crc32c_intel drm_kms_helper cec drm ghash_clmulni_intel ccp igb nvme dca nvme_core i2c_algo_bit wmi pinctrl_amd fuse [last
unloaded: ip_tables]

[263341.873052] irq event stamp: 36
[263341.873054] hardirqs last  enabled at (35): [<ffffffffa8d61117>]
_raw_spin_unlock_irqrestore+0x37/0x40
[263341.873059] hardirqs last disabled at (36): [<ffffffffa8d5a8a9>]
__schedule+0x6e9/0xb20
[263341.873063] softirqs last  enabled at (0): [<ffffffffa80dd962>]
copy_process+0x902/0x1df0
[263341.873066] softirqs last disabled at (0): [<0000000000000000>] 0x0
[263341.873069] CPU: 2 PID: 288 Comm: kswapd0 Tainted: G      D W    L
  --------- ---  5.12.0-0.rc1.162.fc35.x86_64+debug #1
[263341.873073] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[263341.873075] RIP: 0010:native_queued_spin_lock_slowpath+0x1ce/0x200
[263341.873079] Code: c1 ef 12 83 e0 03 83 ef 01 48 c1 e0 05 48 63 ff
48 05 00 f3 1e 00 48 03 04 fd 20 d9 6e a9 48 89 08 8b 41 08 85 c0 75
09 f3 90 <8b> 41 08 85 c0 74 f7 48 8b 39 48 85 ff 0f 84 56 ff ff ff 0f
0d 0f
[263341.873082] RSP: 0018:ffffae4340943898 EFLAGS: 00000246
[263341.873085] RAX: 0000000000000000 RBX: ffff9d4253053000 RCX:
ffff9d4946fef300
[263341.873087] RDX: ffff9d4253053008 RSI: 00000000000c0000 RDI:
0000000000000001
[263341.873090] RBP: ffff9d4253053008 R08: 00000000000c0000 R09:
0000000000000000
[263341.873092] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff9d4253053020
[263341.873094] R13: 0000000000000003 R14: 0000000000000003 R15:
ffff9d41760b2000
[263341.873097] FS:  0000000000000000(0000) GS:ffff9d4946e00000(0000)
knlGS:0000000000000000
[263341.873099] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[263341.873102] CR2: 0000021509b13000 CR3: 00000004130c8000 CR4:
0000000000350ee0
[263341.873104] Call Trace:
[263341.873107]  do_raw_spin_lock+0x94/0xa0
[263341.873110]  _raw_spin_lock+0x63/0x80
[263341.873114]  __z3fold_alloc+0x78/0x3d0
[263341.873118]  z3fold_zpool_malloc+0x4a5/0x7c0
[263341.873121]  ? _raw_spin_unlock+0x1f/0x30
[263341.873125]  zswap_frontswap_store+0x43e/0x890
[263341.873130]  __frontswap_store+0xc8/0x170
[263341.873134]  swap_writepage+0x39/0x70
[263341.873137]  pageout+0x125/0x540
[263341.873142]  shrink_page_list+0x131b/0x1bb0
[263341.873147]  shrink_inactive_list+0x12a/0x440
[263341.873152]  shrink_lruvec+0x4aa/0x6d0
[263341.873158]  shrink_node+0x2d1/0x700
[263341.873163]  balance_pgdat+0x2f5/0x650
[263341.873169]  kswapd+0x21d/0x4d0
[263341.873172]  ? do_wait_intr_irq+0xd0/0xd0
[263341.873176]  ? balance_pgdat+0x650/0x650
[263341.873179]  kthread+0x13a/0x150
[263341.873183]  ? __kthread_bind_mask+0x60/0x60
[263341.873187]  ret_from_fork+0x22/0x30

It related?

Full kernel log is here: https://pastebin.com/x0KbXN9L

Signature · 2 lines
Best Regards,
Mike Gavrilov.
#6 lore raw
Quoted 79 lines · On Fri, 5 Mar 2021 14:33:14 +0500 Mikhail Gavrilov wrote:
Quoted 62 lines · Hillf Danton

What we learn from your reports is

1/ in z3fold_free(), kref_put() creates the ground zero for the race cases reported,

2/ the stale_lock in combination with lock makes things more complicated than thought.

Instead of dropping something in the zero spot, the fix below goes the road mentioned before in this mail thread - add another list_head in parallel to the buddy and s/buddy/stale_node/ under every case of stale_lock.

mm/z3fold.c +6 -4
@@ -127,6 +127,7 @@ struct z3fold_header {
        unsigned short first_num:2;
        unsigned short mapped_count:2;
        unsigned short foreign_handles:2;
+       struct list_head stale_node;
 };
 
 /**
@@ -429,6 +430,7 @@ static struct z3fold_header *init_z3fold
        zhdr->slots = slots;
        zhdr->pool = pool;
        INIT_LIST_HEAD(&zhdr->buddy);
+       INIT_LIST_HEAD(&zhdr->stale_node);
        INIT_WORK(&zhdr->work, compact_page_work);
        return zhdr;
 }
@@ -556,7 +558,7 @@ static void __release_z3fold_page(struct
                z3fold_page_unlock(zhdr);
 
        spin_lock(&pool->stale_lock);
-       list_add(&zhdr->buddy, &pool->stale);
+       list_add(&zhdr->stale_node, &pool->stale);
        queue_work(pool->release_wq, &pool->work);
        spin_unlock(&pool->stale_lock);
 }
@@ -598,10 +600,10 @@ static void free_pages_work(struct work_
        spin_lock(&pool->stale_lock);
        while (!list_empty(&pool->stale)) {
                struct z3fold_header *zhdr = list_first_entry(&pool->stale,
-                                               struct z3fold_header, buddy);
+                                               struct z3fold_header, stale_node);
                struct page *page = virt_to_page(zhdr);
 
-               list_del(&zhdr->buddy);
+               list_del(&zhdr->stale_node);
                if (WARN_ON(!test_bit(PAGE_STALE, &page->private)))
                        continue;
                spin_unlock(&pool->stale_lock);
@@ -1140,14 +1142,14 @@ retry:
        if (can_sleep) {
                spin_lock(&pool->stale_lock);
                zhdr = list_first_entry_or_null(&pool->stale,
-                                               struct z3fold_header, buddy);
+                                               struct z3fold_header, stale_node);
                /*
                 * Before allocating a page, let's see if we can take one from

mm/z3fold.c retry:

Quoted 10 lines
mm/z3fold.c +1 -2
                 * the stale pages list. cancel_work_sync() can sleep so we
                 * limit this case to the contexts where we can sleep
                 */
                if (zhdr) {
-                       list_del(&zhdr->buddy);
+                       list_del(&zhdr->stale_node);
                        spin_unlock(&pool->stale_lock);
                        cancel_work_sync(&zhdr->work);
                        page = virt_to_page(zhdr);
--

The computer with patch above worked for a record time (3 days) without freezing.
https://postimg.cc/VShF5cJN

Quoted 1 line

But after 3 days hangs with follow trace:

Thanks again for your report.

Quoted 23 lines
[263314.718807] general protection fault, probably for non-canonical
address 0x72c1224000000000: 0000 [#1] SMP NOPTI
[263314.718828] CPU: 3 PID: 476750 Comm: Chrome_IOThread Tainted: G
    W        --------- ---  5.12.0-0.rc1.162.fc35.x86_64+debug #1
[263314.718831] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[263314.718835] RIP: 0010:__list_add_valid+0x3/0x40
[263314.718841] Code: e9 5d ff ff ff b8 f4 ff ff ff e9 53 ff ff ff 48
c7 00 00 00 00 00 c7 40 08 00 00 00 00 e9 6b ff ff ff cc cc cc cc cc
49 89 d0 <48> 8b 52 08 48 39 f2 0f 85 03 cf 62 00 4c 8b 0a 4d 39 c1 0f
85 1f
[263314.718845] RSP: 0018:ffffae4345b5fac0 EFLAGS: 00010282
[263314.718849] RAX: 00000000000003c0 RBX: ffffed09c02ade80 RCX:
0000000000000000
[263314.718851] RDX: 72c1224000000000 RSI: ffffce433c462004 RDI:
ffff9d414ab7a000
[263314.718853] RBP: ffffce433c462004 R08: 72c1224000000000 R09:
0000000000000000
[263314.718856] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff9d4253053008
[263314.718858] R13: ffff9d414ab7a010 R14: 72c1224000000000 R15:

ffff9d414ab7a000

Quoted 8 lines
[263314.718860] FS:  00007f8ef636f640(0000) GS:ffff9d4947000000(0000)
knlGS:0000000000000000
[263314.718863] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[263314.718865] CR2: 000055634cbef560 CR3: 00000002dbdbc000 CR4:
0000000000350ee0
[263314.718867] Call Trace:
[263314.718875]  do_compact_page+0x28d/0xb60
[263314.718884]  ? z3fold_zpool_free+0x3a8/0x590

One part of the race is the free path on CPU#3.

Quoted 22 lines
[263314.718888]  zswap_free_entry+0x43/0x70
[263314.718892]  zswap_frontswap_invalidate_page+0x8c/0x90
[263314.718895]  __frontswap_invalidate_page+0x5d/0x90
[263314.718898]  swap_range_free+0xcd/0xf0
[263314.718901]  swapcache_free_entries+0x128/0x1a0
[263314.718904]  free_swap_slot+0xbb/0xd0
[263314.718907]  __swap_entry_free+0x7a/0xa0
[263314.718910]  free_swap_and_cache+0x35/0x80
[263314.718913]  shmem_undo_range+0x188/0x7e0
[263314.718919]  ? ldsem_down_read+0x1f/0x40
[263314.718925]  shmem_evict_inode+0xe6/0x290
[263314.718928]  ? lock_release+0x1ef/0x410
[263314.718932]  ? var_wake_function+0x20/0x20
[263314.718936]  evict+0xcf/0x1d0
[263314.718940]  __dentry_kill+0xe8/0x190
[263314.718943]  ? dput+0x20/0x480
[263314.718946]  dput+0x2b8/0x480
[263314.718949]  __fput+0x102/0x260
[263314.718952]  task_work_run+0x5c/0xa0
[263314.718957]  exit_to_user_mode_prepare+0x232/0x240
[263314.718960]  syscall_exit_to_user_mode+0x27/0x70
[263314.718964]  entry_SYSCALL_64_after_hwframe+0x44/0xae

[...]

Quoted 2 lines
[263341.868981] watchdog: BUG: soft lockup - CPU#0 stuck for 23s!

[ThreadPoolForeg:513140]

[...]

Quoted 28 lines
[263341.869052] irq event stamp: 0
[263341.869054] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[263341.869057] hardirqs last disabled at (0): [<ffffffffa80dd962>]
copy_process+0x902/0x1df0
[263341.869061] softirqs last  enabled at (0): [<ffffffffa80dd962>]
copy_process+0x902/0x1df0
[263341.869064] softirqs last disabled at (0): [<0000000000000000>] 0x0
[263341.869067] CPU: 0 PID: 513140 Comm: ThreadPoolForeg Tainted: G
  D W        --------- ---  5.12.0-0.rc1.162.fc35.x86_64+debug #1
[263341.869070] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[263341.869073] RIP: 0010:native_queued_spin_lock_slowpath+0x1ce/0x200
[263341.869076] Code: c1 ef 12 83 e0 03 83 ef 01 48 c1 e0 05 48 63 ff
48 05 00 f3 1e 00 48 03 04 fd 20 d9 6e a9 48 89 08 8b 41 08 85 c0 75
09 f3 90 <8b> 41 08 85 c0 74 f7 48 8b 39 48 85 ff 0f 84 56 ff ff ff 0f
0d 0f
[263341.869079] RSP: 0000:ffffae435505bbf0 EFLAGS: 00000246
[263341.869082] RAX: 0000000000000000 RBX: 0000000000e6c4f7 RCX:
ffff9d4946bef300
[263341.869084] RDX: ffff9d424197e4e8 RSI: 0000000000040000 RDI:
0000000000000014
[263341.869087] RBP: ffff9d424197e4e8 R08: 0000000000040000 R09:
0000000000000000
[263341.869089] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff9d424197e500
[263341.869091] R13: ffff9d424197e4e8 R14: ffffffffa9e87020 R15:
ffff9d424197e4e0
[263341.869094] FS:  00007ff2de694640(0000) GS:ffff9d4946a00000(0000)
Quoted 8 lines

knlGS:0000000000000000

[263341.869096] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[263341.869099] CR2: 00000070bfa4d3c0 CR3: 00000002f2878000 CR4:
0000000000350ef0
[263341.869101] Call Trace:
[263341.869104]  do_raw_spin_lock+0x94/0xa0
[263341.869107]  _raw_spin_lock+0x63/0x80
[263341.869111]  zswap_frontswap_load+0x30/0x2f0

A foot from z3fold on CPU#0.

Quoted 59 lines
[263341.869115]  ? trace_hardirqs_on+0x1b/0xe0
[263341.869120]  __frontswap_load+0xc3/0x160
[263341.869123]  swap_readpage+0x25b/0x440
[263341.869127]  swapin_readahead+0x450/0x4e0
[263341.869130]  ? lock_release+0x1ef/0x410
[263341.869134]  do_swap_page+0x4a4/0x900
[263341.869137]  __handle_mm_fault+0xbd6/0x1610
[263341.869140]  ? lock_acquire+0x177/0x3a0
[263341.869145]  handle_mm_fault+0xa2/0x270
[263341.869148]  do_user_addr_fault+0x1ea/0x6b0
[263341.869152]  exc_page_fault+0x67/0x2a0
[263341.869155]  ? asm_exc_page_fault+0x8/0x30
[263341.869158]  asm_exc_page_fault+0x1e/0x30
[263341.869161] RIP: 0033:0x55e1b76f7713
[263341.869164] Code: 8b 9f c8 00 00 00 4d 8b b7 d0 00 00 00 4c 39 f3
74 5a 4c 8d 25 8e 18 3c 05 4c 8d 2d aa 59 e6 fb 0f 1f 80 00 00 00 00
48 8b 3b <48> 8b 07 48 89 c1 4c 29 e1 48 c1 c9 03 48 81 f9 9f 00 00 00
0f 87
[263341.869166] RSP: 002b:00007ff2de693430 EFLAGS: 00010287
[263341.869169] RAX: 0000000000000000 RBX: 00007ff2de693568 RCX:
00000070c00c7be0
[263341.869171] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
00000070bfa4d3c0
[263341.869174] RBP: 00007ff2de693480 R08: 0000000000000000 R09:
00000000000000ca
[263341.869176] R10: 00007fff78981080 R11: 00007fff78981090 R12:
000055e1bcab8f90
[263341.869178] R13: 000055e1b355d0b3 R14: 00007ff2de693578 R15:
00007ff2de693500
[263341.870981] watchdog: BUG: soft lockup - CPU#1 stuck for 23s!
[steamwebhelper:3496089]
[263341.871048] irq event stamp: 0
[263341.871050] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[263341.871054] hardirqs last disabled at (0): [<ffffffffa80dd962>]
copy_process+0x902/0x1df0
[263341.871058] softirqs last  enabled at (0): [<ffffffffa80dd962>]
copy_process+0x902/0x1df0
[263341.871061] softirqs last disabled at (0): [<0000000000000000>] 0x0
[263341.871064] CPU: 1 PID: 3496089 Comm: steamwebhelper Tainted: G
  D W    L   --------- ---  5.12.0-0.rc1.162.fc35.x86_64+debug #1
[263341.871067] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[263341.871069] RIP: 0010:native_queued_spin_lock_slowpath+0x137/0x200
[263341.871073] Code: e0 a9 1d 00 eb cb 41 83 c0 01 c1 e6 10 41 c1 e0
12 44 09 c6 89 f0 c1 e8 10 66 87 42 02 89 c7 c1 e7 10 75 73 31 ff eb
02 f3 90 <8b> 02 66 85 c0 75 f7 41 89 c0 66 45 31 c0 44 39 c6 0f 84 9b
00 00
[263341.871076] RSP: 0018:ffffae43549eb5f8 EFLAGS: 00000202
[263341.871078] RAX: 00000000000c0101 RBX: ffff9d414bc00000 RCX:
ffff9d4946def300
[263341.871081] RDX: ffff9d4253053008 RSI: 0000000000080000 RDI:
0000000000000000
[263341.871083] RBP: ffff9d4253053008 R08: 0000000000080000 R09:
0000000000000000
[263341.871085] R10: 0000000000000000 R11: 0000000000000001 R12:
ffff9d4253053020
[263341.871088] R13: ffff9d414bc00010 R14: 0000000000000000 R15:
ffffed09c02f0000
[263341.871090] FS:  00007f725d561d40(0000) GS:ffff9d4946c00000(0000)
Quoted 8 lines

knlGS:0000000000000000

[263341.871093] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[263341.871095] CR2: 0000366b51ea6fe8 CR3: 00000003865f2000 CR4:
0000000000350ee0
[263341.871098] Call Trace:
[263341.871101]  do_raw_spin_lock+0x94/0xa0
[263341.871104]  _raw_spin_lock+0x63/0x80
[263341.871107]  z3fold_page_isolate+0xbd/0x1b0

The isolate path on CPU#1.

Quoted 85 lines
[263341.871112]  isolate_movable_page+0x94/0x180
[263341.871115]  isolate_migratepages_block+0x5db/0x1120
[263341.871120]  ? lock_release+0x1ef/0x410
[263341.871124]  compact_zone+0x5a4/0xfd0
[263341.871129]  compact_zone_order+0xaa/0xf0
[263341.871134]  try_to_compact_pages+0x111/0x3b0
[263341.871138]  __alloc_pages_direct_compact+0x79/0x210
[263341.871142]  __alloc_pages_slowpath.constprop.0+0x1d0/0xf90
[263341.871147]  ? __alloc_pages_nodemask+0x2e3/0x400
[263341.871151]  ? lock_release+0x1ef/0x410
[263341.871154]  __alloc_pages_nodemask+0x37d/0x400
[263341.871159]  ttm_pool_alloc+0x2a3/0x630 [ttm]
[263341.871167]  ttm_tt_populate+0x37/0xe0 [ttm]
[263341.871172]  ttm_bo_handle_move_mem+0x13a/0x170 [ttm]
[263341.871179]  ttm_bo_validate+0x15f/0x1b0 [ttm]
[263341.871184]  ? lock_release+0x1ef/0x410
[263341.871189]  ttm_bo_init_reserved+0x2f7/0x3e0 [ttm]
[263341.871195]  amdgpu_bo_do_create+0x1a8/0x630 [amdgpu]
[263341.871312]  ? amdgpu_bo_subtract_pin_size+0x50/0x50 [amdgpu]
[263341.871422]  amdgpu_bo_create+0x30/0x2e0 [amdgpu]
[263341.871531]  ? lock_acquire+0x177/0x3a0
[263341.871535]  ? trace_hardirqs_on+0x1b/0xe0
[263341.871539]  ? _raw_spin_unlock_irqrestore+0x37/0x40
[263341.871543]  ? lock_release+0x1ef/0x410
[263341.871547]  amdgpu_gem_create_ioctl+0x10e/0x370 [amdgpu]
[263341.871664]  ? amdgpu_gem_force_release+0x130/0x130 [amdgpu]
[263341.871774]  drm_ioctl_kernel+0x89/0xe0 [drm]
[263341.871797]  drm_ioctl+0x20f/0x3c0 [drm]
[263341.871816]  ? amdgpu_gem_force_release+0x130/0x130 [amdgpu]
[263341.871927]  ? selinux_file_ioctl+0x147/0x200
[263341.871931]  ? lock_acquired+0x200/0x390
[263341.871934]  ? lock_release+0x1ef/0x410
[263341.871937]  ? trace_hardirqs_on+0x1b/0xe0
[263341.871940]  ? _raw_spin_unlock_irqrestore+0x37/0x40
[263341.871944]  amdgpu_drm_ioctl+0x49/0x80 [amdgpu]
[263341.872053]  __x64_sys_ioctl+0x82/0xb0
[263341.872058]  do_syscall_64+0x33/0x40
[263341.872061]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[263341.872065] RIP: 0033:0x7f72610b22bb
[263341.872068] Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff ff ff 5b 5d
4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00
00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 85 bb 0c 00 f7 d8 64 89
01 48
[263341.872071] RSP: 002b:00007ffcd94a01f8 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[263341.872074] RAX: ffffffffffffffda RBX: 00007ffcd94a0250 RCX:
00007f72610b22bb
[263341.872076] RDX: 00007ffcd94a0250 RSI: 00000000c0206440 RDI:
0000000000000016
[263341.872078] RBP: 00000000c0206440 R08: 0000000000000009 R09:
00000000000000b8
[263341.872081] R10: 00007ffcd9568080 R11: 0000000000000246 R12:
000008c86f1ae3c0
[263341.872083] R13: 0000000000000016 R14: 0000000000200000 R15:
00000000019c6000
[263341.872983] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kswapd0:288]
[263341.873052] irq event stamp: 36
[263341.873054] hardirqs last  enabled at (35): [<ffffffffa8d61117>]
_raw_spin_unlock_irqrestore+0x37/0x40
[263341.873059] hardirqs last disabled at (36): [<ffffffffa8d5a8a9>]
__schedule+0x6e9/0xb20
[263341.873063] softirqs last  enabled at (0): [<ffffffffa80dd962>]
copy_process+0x902/0x1df0
[263341.873066] softirqs last disabled at (0): [<0000000000000000>] 0x0
[263341.873069] CPU: 2 PID: 288 Comm: kswapd0 Tainted: G      D W    L
  --------- ---  5.12.0-0.rc1.162.fc35.x86_64+debug #1
[263341.873073] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[263341.873075] RIP: 0010:native_queued_spin_lock_slowpath+0x1ce/0x200
[263341.873079] Code: c1 ef 12 83 e0 03 83 ef 01 48 c1 e0 05 48 63 ff
48 05 00 f3 1e 00 48 03 04 fd 20 d9 6e a9 48 89 08 8b 41 08 85 c0 75
09 f3 90 <8b> 41 08 85 c0 74 f7 48 8b 39 48 85 ff 0f 84 56 ff ff ff 0f
0d 0f
[263341.873082] RSP: 0018:ffffae4340943898 EFLAGS: 00000246
[263341.873085] RAX: 0000000000000000 RBX: ffff9d4253053000 RCX:
ffff9d4946fef300
[263341.873087] RDX: ffff9d4253053008 RSI: 00000000000c0000 RDI:
0000000000000001
[263341.873090] RBP: ffff9d4253053008 R08: 00000000000c0000 R09:
0000000000000000
[263341.873092] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff9d4253053020
[263341.873094] R13: 0000000000000003 R14: 0000000000000003 R15:
ffff9d41760b2000
[263341.873097] FS:  0000000000000000(0000) GS:ffff9d4946e00000(0000)
Quoted 8 lines

knlGS:0000000000000000

[263341.873099] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[263341.873102] CR2: 0000021509b13000 CR3: 00000004130c8000 CR4:
0000000000350ee0
[263341.873104] Call Trace:
[263341.873107]  do_raw_spin_lock+0x94/0xa0
[263341.873110]  _raw_spin_lock+0x63/0x80
[263341.873114]  __z3fold_alloc+0x78/0x3d0

The alloc path on CPU#2.

Quoted 19 lines
[263341.873118]  z3fold_zpool_malloc+0x4a5/0x7c0
[263341.873121]  ? _raw_spin_unlock+0x1f/0x30
[263341.873125]  zswap_frontswap_store+0x43e/0x890
[263341.873130]  __frontswap_store+0xc8/0x170
[263341.873134]  swap_writepage+0x39/0x70
[263341.873137]  pageout+0x125/0x540
[263341.873142]  shrink_page_list+0x131b/0x1bb0
[263341.873147]  shrink_inactive_list+0x12a/0x440
[263341.873152]  shrink_lruvec+0x4aa/0x6d0
[263341.873158]  shrink_node+0x2d1/0x700
[263341.873163]  balance_pgdat+0x2f5/0x650
[263341.873169]  kswapd+0x21d/0x4d0
[263341.873172]  ? do_wait_intr_irq+0xd0/0xd0
[263341.873176]  ? balance_pgdat+0x650/0x650
[263341.873179]  kthread+0x13a/0x150
[263341.873183]  ? __kthread_bind_mask+0x60/0x60
[263341.873187]  ret_from_fork+0x22/0x30
Quoted 1 line

It related?

Yes, it is the same race as we saw before. But after cutting the race between poo->stale_lock and pool->lock with the patch above, the race between the free path and isolate/putback path came up.

Quoted 8 lines

Full kernel log is here: https://pastebin.com/x0KbXN9L

Signature · 2 lines
Best Regards,
Mike Gavrilov.

Try the diff below in combination with the patch above

mm/z3fold.c +2 -0
@@ -1676,8 +1676,10 @@ static void z3fold_page_putback(struct p
 	pool = zhdr_to_pool(zhdr);
 
 	z3fold_page_lock(zhdr);
+	spin_lock(&pool->lock);
 	if (!list_empty(&zhdr->buddy))
 		list_del_init(&zhdr->buddy);
+	spin_unlock(&pool->lock);
 	INIT_LIST_HEAD(&page->lru);
 	if (kref_put(&zhdr->refcount, release_z3fold_page_locked)) {
 		atomic64_dec(&pool->pages_nr);
#7 lore raw
Quoted 10 lines · Hillf Danton

Yes, it is the same race as we saw before. But after cutting the race between poo->stale_lock and pool->lock with the patch above, the race between the free path and isolate/putback path came up.

Try the diff below in combination with the patch above

mm/z3fold.c static void z3fold_page_putback(struct p · line 1685

Quoted 10 lines
mm/z3fold.c +2 -0
@@ -1676,8 +1676,10 @@ static void z3fold_page_putback(struct p
        pool = zhdr_to_pool(zhdr);
 
        z3fold_page_lock(zhdr);
+       spin_lock(&pool->lock);
        if (!list_empty(&zhdr->buddy))
                list_del_init(&zhdr->buddy);
+       spin_unlock(&pool->lock);
        INIT_LIST_HEAD(&page->lru);
        if (kref_put(&zhdr->refcount, release_z3fold_page_locked)) {
                atomic64_dec(&pool->pages_nr);

Unfortunately even with combination of two latest patches computer hanged again after two days uptime.

[185000.747401] list_add corruption. next->prev should be prev
(ffffe0c1bea61f40), but was 0000000000000000. (next=ffff9bb90b444000).
[185000.747438] ------------[ cut here ]------------
[185000.747441] kernel BUG at lib/list_debug.c:23!
[185000.747449] invalid opcode: 0000 [#1] SMP NOPTI
[185000.747454] CPU: 22 PID: 1588003 Comm: Web Content Tainted: G
  W        --------- ---
5.12.0-0.rc1.20210305git280d542f6ffa.164.fc35.x86_64 #1
[185000.747458] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[185000.747462] RIP: 0010:__list_add_valid.cold+0xf/0x3f
[185000.747469] Code: 48 c7 c6 7c a9 64 84 48 89 ef 49 c7 c7 ea ff ff
ff e8 9d 81 01 00 e9 5f ee 9c ff 4c 89 c1 48 c7 c7 48 aa 64 84 e8 74
fd fd ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 f8 aa 64 84 e8 5d
fd fd
[185000.747472] RSP: 0000:ffffc0c1c61cfc10 EFLAGS: 00010286
[185000.747476] RAX: 0000000000000075 RBX: fffffa0d596b0a40 RCX:
0000000000000000
[185000.747479] RDX: ffff9bbc097e97a0 RSI: ffff9bbc097daae0 RDI:
ffff9bbc097daae0
[185000.747482] RBP: ffffe0c1bea61f40 R08: 0000000000000000 R09:
ffffc0c1c61cfa58
[185000.747485] R10: ffffc0c1c61cfa50 R11: 0000000000000000 R12:
ffff9bb537b4f008
[185000.747488] R13: ffff9bba5ac29010 R14: ffff9bb90b444000 R15:
ffff9bba5ac29000
[185000.747491] FS:  00007f198ea257c0(0000) GS:ffff9bbc09600000(0000)
knlGS:0000000000000000
[185000.747495] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[185000.747498] CR2: 00000fe3aa523500 CR3: 000000012f870000 CR4:
0000000000350ee0
[185000.747501] Call Trace:
[185000.747504]  do_compact_page+0x28d/0xb60
[185000.747509]  ? _raw_spin_unlock+0x1f/0x30
[185000.747514]  ? z3fold_zpool_free+0x3a8/0x590
[185000.747518]  zswap_free_entry+0x43/0x70
[185000.747523]  zswap_frontswap_invalidate_page+0x8c/0x90
[185000.747527]  __frontswap_invalidate_page+0x5d/0x90
[185000.747531]  swap_range_free+0xcd/0xf0
[185000.747535]  swapcache_free_entries+0x128/0x1a0
[185000.747539]  free_swap_slot+0xbb/0xd0
[185000.747543]  __swap_entry_free+0x7a/0xa0
[185000.747547]  do_swap_page+0x393/0x900
[185000.747551]  __handle_mm_fault+0xbd6/0x1610
[185000.747557]  handle_mm_fault+0xa2/0x270
[185000.747561]  do_user_addr_fault+0x1ea/0x6b0
[185000.747566]  exc_page_fault+0x67/0x2a0
[185000.747570]  ? asm_exc_page_fault+0x8/0x30
[185000.747574]  asm_exc_page_fault+0x1e/0x30
[185000.747578] RIP: 0033:0x7f198eb8be30
[185000.747582] Code: 9d 48 81 fa 80 00 00 00 77 19 c5 fe 7f 07 c5 fe
7f 47 20 c5 fe 7f 44 17 e0 c5 fe 7f 44 17 c0 c5 f8 77 c3 48 8d 8f 80
00 00 00 <c5> fe 7f 07 48 83 e1 80 c5 fe 7f 44 17 e0 c5 fe 7f 47 20 c5
fe 7f
[185000.747585] RSP: 002b:00007ffea7e406e8 EFLAGS: 00010202
[185000.747589] RAX: 00000fe3aa523500 RBX: 00007ffea7e40738 RCX:
00000fe3aa523580
[185000.747592] RDX: 0000000000004000 RSI: 00000000000000fa RDI:
00000fe3aa523500
[185000.747594] RBP: 0000600000000000 R08: 00007f195295a000 R09:
ffffffffffffffff
[185000.747597] R10: 0000556a267402c8 R11: 0000000000000206 R12:
00007f195295a800
[185000.747600] R13: fffffc0000000000 R14: 00007ffea7e40738 R15:
00007f195295a7f0
[185000.747605] Modules linked in: crypto_user tun snd_seq_dummy

snd_hrtimer uinput nls_utf8 isofs rfcomm netconsole nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd sunrpc vfat fat hid_logitech_hidpp usblp hid_logitech_dj snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi intel_rapl_msr mt76x2u joydev intel_rapl_common snd_hda_intel mt76x2_common iwlmvm snd_intel_dspcfg mt76x02_usb uvcvideo snd_intel_sdw_acpi snd_usb_audio mt76_usb snd_hda_codec edac_mce_amd videobuf2_vmalloc mt76x02_lib videobuf2_memops videobuf2_v4l2 snd_hda_core mt76 videobuf2_common snd_usbmidi_lib kvm_amd btusb snd_hwdep snd_rawmidi videodev btrtl mac80211 kvm snd_seq btbcm btintel snd_seq_device irqbypass libarc4 eeepc_wmi xpad mc bluetooth rapl ff_memless snd_pcm

[185000.747647]  iwlwifi asus_wmi sparse_keymap video snd_timer

ecdh_generic ecc wmi_bmof pcspkr snd cfg80211 soundcore sp5100_tco k10temp i2c_piix4 rfkill acpi_cpufreq binfmt_misc ip_tables uas usb_storage amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched crct10dif_pclmul drm_kms_helper crc32_pclmul crc32c_intel cec drm ghash_clmulni_intel ccp igb nvme dca nvme_core i2c_algo_bit wmi pinctrl_amd fuse

[185000.747878] ---[ end trace df51d3d2498d767d ]---
[185000.747882] RIP: 0010:__list_add_valid.cold+0xf/0x3f
[185000.747886] Code: 48 c7 c6 7c a9 64 84 48 89 ef 49 c7 c7 ea ff ff
ff e8 9d 81 01 00 e9 5f ee 9c ff 4c 89 c1 48 c7 c7 48 aa 64 84 e8 74
fd fd ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 f8 aa 64 84 e8 5d
fd fd
[185000.747889] RSP: 0000:ffffc0c1c61cfc10 EFLAGS: 00010286
[185000.747893] RAX: 0000000000000075 RBX: fffffa0d596b0a40 RCX:
0000000000000000
[185000.747895] RDX: ffff9bbc097e97a0 RSI: ffff9bbc097daae0 RDI:
ffff9bbc097daae0
[185000.747898] RBP: ffffe0c1bea61f40 R08: 0000000000000000 R09:
ffffc0c1c61cfa58
[185000.747901] R10: ffffc0c1c61cfa50 R11: 0000000000000000 R12:
ffff9bb537b4f008
[185000.747904] R13: ffff9bba5ac29010 R14: ffff9bb90b444000 R15:
ffff9bba5ac29000
[185000.747907] FS:  00007f198ea257c0(0000) GS:ffff9bbc09600000(0000)
knlGS:0000000000000000
[185000.747910] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[185000.747913] CR2: 00000fe3aa523500 CR3: 000000012f870000 CR4:
0000000000350ee0
[185000.747916] note: Web Content[1588003] exited with preempt_count 6
[185026.580248] watchdog: BUG: soft lockup - CPU#0 stuck for 22s!
[Chrome_ChildIOT:1951362]
[185026.580262] Modules linked in: crypto_user tun snd_seq_dummy

snd_hrtimer uinput nls_utf8 isofs rfcomm netconsole nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd sunrpc vfat fat hid_logitech_hidpp usblp hid_logitech_dj snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi intel_rapl_msr mt76x2u joydev intel_rapl_common snd_hda_intel mt76x2_common iwlmvm snd_intel_dspcfg mt76x02_usb uvcvideo snd_intel_sdw_acpi snd_usb_audio mt76_usb snd_hda_codec edac_mce_amd videobuf2_vmalloc mt76x02_lib videobuf2_memops videobuf2_v4l2 snd_hda_core mt76 videobuf2_common snd_usbmidi_lib kvm_amd btusb snd_hwdep snd_rawmidi videodev btrtl mac80211 kvm snd_seq btbcm btintel snd_seq_device irqbypass libarc4 eeepc_wmi xpad mc bluetooth rapl ff_memless snd_pcm

[185026.580306]  iwlwifi asus_wmi sparse_keymap video snd_timer

ecdh_generic ecc wmi_bmof pcspkr snd cfg80211 soundcore sp5100_tco k10temp i2c_piix4 rfkill acpi_cpufreq binfmt_misc ip_tables uas usb_storage amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched crct10dif_pclmul drm_kms_helper crc32_pclmul crc32c_intel cec drm ghash_clmulni_intel ccp igb nvme dca nvme_core i2c_algo_bit wmi pinctrl_amd fuse

[185026.580334] irq event stamp: 0
[185026.580337] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[185026.580342] hardirqs last disabled at (0): [<ffffffff830dd962>]
copy_process+0x902/0x1df0
[185026.580349] softirqs last  enabled at (0): [<ffffffff830dd962>]
copy_process+0x902/0x1df0
[185026.580353] softirqs last disabled at (0): [<0000000000000000>] 0x0
[185026.580357] CPU: 0 PID: 1951362 Comm: Chrome_ChildIOT Tainted: G
   D W        --------- ---
5.12.0-0.rc1.20210305git280d542f6ffa.164.fc35.x86_64 #1
[185026.580362] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[185026.580365] RIP: 0010:native_queued_spin_lock_slowpath+0x1ce/0x200
[185026.580370] Code: c1 ef 12 83 e0 03 83 ef 01 48 c1 e0 05 48 63 ff
48 05 00 f3 1e 00 48 03 04 fd 20 d9 6e 84 48 89 08 8b 41 08 85 c0 75
09 f3 90 <8b> 41 08 85 c0 74 f7 48 8b 39 48 85 ff 0f 84 56 ff ff ff 0f
0d 0f
[185026.580374] RSP: 0000:ffffc0c1d596fbf0 EFLAGS: 00000246
[185026.580379] RAX: 0000000000000000 RBX: 0000000000f10bea RCX:
ffff9bbc06bef300
[185026.580382] RDX: ffff9bb500b9fb48 RSI: 0000000000040000 RDI:
0000000000000013
[185026.580385] RBP: ffff9bb500b9fb48 R08: 0000000000040000 R09:
0000000000000000
[185026.580388] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff9bb500b9fb60
[185026.580391] R13: ffff9bb500b9fb48 R14: ffffffff84e87020 R15:
ffff9bb500b9fb40
[185026.580394] FS:  00007f2e2ffe6640(0000) GS:ffff9bbc06a00000(0000)
knlGS:0000000000000000
[185026.580398] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[185026.580401] CR2: 00002a8e6ef59188 CR3: 0000000306082000 CR4:
0000000000350ef0
[185026.580405] Call Trace:
[185026.580408]  do_raw_spin_lock+0x94/0xa0
[185026.580665]  _raw_spin_lock+0x63/0x80
[185026.580670]  zswap_frontswap_load+0x30/0x2f0
[185026.580676]  ? trace_hardirqs_on+0x1b/0xe0
[185026.580681]  __frontswap_load+0xc3/0x160
[185026.580685]  swap_readpage+0x257/0x430
[185026.580689]  swapin_readahead+0x450/0x4e0
[185026.580693]  ? lock_release+0x1ef/0x410
[185026.580698]  do_swap_page+0x4a4/0x900
[185026.580703]  __handle_mm_fault+0xbd6/0x1610
[185026.580795]  handle_mm_fault+0xa2/0x270
[185026.580799]  do_user_addr_fault+0x1ea/0x6b0
[185026.580804]  exc_page_fault+0x67/0x2a0
[185026.580808]  ? asm_exc_page_fault+0x8/0x30
[185026.580889]  asm_exc_page_fault+0x1e/0x30
[185026.580893] RIP: 0033:0x55d9d6466038
[185026.580897] Code: cc cc 55 48 89 e5 48 89 7f 10 48 89 7f 18 5d c3
cc cc 55 48 89 e5 48 8b 47 10 48 8b 4f 18 48 89 41 10 48 8b 47 10 48
8b 4f 18 <48> 89 48 18 0f 57 c0 0f 11 47 10 5d c3 cc cc cc cc cc cc cc
cc cc
[185026.580900] RSP: 002b:00007f2e2ffe46b0 EFLAGS: 00010246
[185026.580968] RAX: 00002a8e6ef59170 RBX: 00002a8e6ef40420 RCX:
00002a8e6ef4e370
[185026.580972] RDX: 00002a8e6d6715e0 RSI: 00002a8e6dcb02e0 RDI:
00002a8e6ef40420
[185026.580974] RBP: 00007f2e2ffe46b0 R08: 0000000000000000 R09:
00007fff260af5d0
[185026.581039] R10: 0000000000000000 R11: 0000000000000246 R12:
0000000000000020
[185026.581042] R13: 00002a8e6dcb02e0 R14: 000055d9deb9b1e0 R15:

00002a8e6dcb02e0

Full kernel log is here: https://pastebin.com/WmBLJ3MR

Signature · 2 lines
Best Regards,
Mike Gavrilov.
#8 lore raw
Quoted 13 lines · On Mon, 8 Mar 2021 20:42:42 +0500 Mikhail Gavrilov wrote:
Quoted 12 lines · Hillf Danton

Yes, it is the same race as we saw before. But after cutting the race between poo->stale_lock and pool->lock with the patch above, the race between the free path and isolate/putback path came up.

Try the diff below in combination with the patch above

mm/z3fold.c +0 -0
@@ -1676,8 +1676,10 @@ static void z3fold_page_putback(struct p
        pool = zhdr_to_pool(zhdr);
 
Quoted 11 lines

mm/z3fold.c static void z3fold_page_putback(struct p · line 1685

Quoted 8 lines
mm/z3fold.c +2 -0
        z3fold_page_lock(zhdr);
+       spin_lock(&pool->lock);
        if (!list_empty(&zhdr->buddy))
                list_del_init(&zhdr->buddy);
+       spin_unlock(&pool->lock);
        INIT_LIST_HEAD(&page->lru);
        if (kref_put(&zhdr->refcount, release_z3fold_page_locked)) {
                atomic64_dec(&pool->pages_nr);

Unfortunately even with combination of two latest patches computer hanged again after two days uptime.

Thanks again for your report.

Quoted 3 lines
[185000.747401] list_add corruption. next->prev should be prev

(ffffe0c1bea61f40), but was 0000000000000000. (next=ffff9bb90b444000).

At the first glance, the zero pointer goes out of the box of race because

1/ the Call Trace shows it is the free path (of the supposed race victim),

2/ on the race winner side however either list_del or list_del_init
would not leave a null pointer behind - the list_add captured in this report is under pool->lock.

Quoted 206 lines
[185000.747438] ------------[ cut here ]------------
[185000.747441] kernel BUG at lib/list_debug.c:23!
[185000.747449] invalid opcode: 0000 [#1] SMP NOPTI
[185000.747454] CPU: 22 PID: 1588003 Comm: Web Content Tainted: G
  W        --------- ---
5.12.0-0.rc1.20210305git280d542f6ffa.164.fc35.x86_64 #1
[185000.747458] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[185000.747462] RIP: 0010:__list_add_valid.cold+0xf/0x3f
[185000.747469] Code: 48 c7 c6 7c a9 64 84 48 89 ef 49 c7 c7 ea ff ff
ff e8 9d 81 01 00 e9 5f ee 9c ff 4c 89 c1 48 c7 c7 48 aa 64 84 e8 74
fd fd ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 f8 aa 64 84 e8 5d
fd fd
[185000.747472] RSP: 0000:ffffc0c1c61cfc10 EFLAGS: 00010286
[185000.747476] RAX: 0000000000000075 RBX: fffffa0d596b0a40 RCX:
0000000000000000
[185000.747479] RDX: ffff9bbc097e97a0 RSI: ffff9bbc097daae0 RDI:
ffff9bbc097daae0
[185000.747482] RBP: ffffe0c1bea61f40 R08: 0000000000000000 R09:
ffffc0c1c61cfa58
[185000.747485] R10: ffffc0c1c61cfa50 R11: 0000000000000000 R12:
ffff9bb537b4f008
[185000.747488] R13: ffff9bba5ac29010 R14: ffff9bb90b444000 R15:
ffff9bba5ac29000
[185000.747491] FS:  00007f198ea257c0(0000) GS:ffff9bbc09600000(0000)
knlGS:0000000000000000
[185000.747495] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[185000.747498] CR2: 00000fe3aa523500 CR3: 000000012f870000 CR4:
0000000000350ee0
[185000.747501] Call Trace:
[185000.747504]  do_compact_page+0x28d/0xb60
[185000.747509]  ? _raw_spin_unlock+0x1f/0x30
[185000.747514]  ? z3fold_zpool_free+0x3a8/0x590
[185000.747518]  zswap_free_entry+0x43/0x70
[185000.747523]  zswap_frontswap_invalidate_page+0x8c/0x90
[185000.747527]  __frontswap_invalidate_page+0x5d/0x90
[185000.747531]  swap_range_free+0xcd/0xf0
[185000.747535]  swapcache_free_entries+0x128/0x1a0
[185000.747539]  free_swap_slot+0xbb/0xd0
[185000.747543]  __swap_entry_free+0x7a/0xa0
[185000.747547]  do_swap_page+0x393/0x900
[185000.747551]  __handle_mm_fault+0xbd6/0x1610
[185000.747557]  handle_mm_fault+0xa2/0x270
[185000.747561]  do_user_addr_fault+0x1ea/0x6b0
[185000.747566]  exc_page_fault+0x67/0x2a0
[185000.747570]  ? asm_exc_page_fault+0x8/0x30
[185000.747574]  asm_exc_page_fault+0x1e/0x30
[185000.747578] RIP: 0033:0x7f198eb8be30
[185000.747582] Code: 9d 48 81 fa 80 00 00 00 77 19 c5 fe 7f 07 c5 fe
7f 47 20 c5 fe 7f 44 17 e0 c5 fe 7f 44 17 c0 c5 f8 77 c3 48 8d 8f 80
00 00 00 <c5> fe 7f 07 48 83 e1 80 c5 fe 7f 44 17 e0 c5 fe 7f 47 20 c5
fe 7f
[185000.747585] RSP: 002b:00007ffea7e406e8 EFLAGS: 00010202
[185000.747589] RAX: 00000fe3aa523500 RBX: 00007ffea7e40738 RCX:
00000fe3aa523580
[185000.747592] RDX: 0000000000004000 RSI: 00000000000000fa RDI:
00000fe3aa523500
[185000.747594] RBP: 0000600000000000 R08: 00007f195295a000 R09:
ffffffffffffffff
[185000.747597] R10: 0000556a267402c8 R11: 0000000000000206 R12:
00007f195295a800
[185000.747600] R13: fffffc0000000000 R14: 00007ffea7e40738 R15:
00007f195295a7f0
[185000.747605] Modules linked in: crypto_user tun snd_seq_dummy

snd_hrtimer uinput nls_utf8 isofs rfcomm netconsole nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd sunrpc vfat fat hid_logitech_hidpp usblp hid_logitech_dj snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi intel_rapl_msr mt76x2u joydev intel_rapl_common snd_hda_intel mt76x2_common iwlmvm snd_intel_dspcfg mt76x02_usb uvcvideo snd_intel_sdw_acpi snd_usb_audio mt76_usb snd_hda_codec edac_mce_amd videobuf2_vmalloc mt76x02_lib videobuf2_memops videobuf2_v4l2 snd_hda_core mt76 videobuf2_common snd_usbmidi_lib kvm_amd btusb snd_hwdep snd_rawmidi videodev btrtl mac80211 kvm snd_seq btbcm btintel snd_seq_device irqbypass libarc4 eeepc_wmi xpad mc bluetooth rapl ff_memless snd_pcm

[185000.747647]  iwlwifi asus_wmi sparse_keymap video snd_timer

ecdh_generic ecc wmi_bmof pcspkr snd cfg80211 soundcore sp5100_tco k10temp i2c_piix4 rfkill acpi_cpufreq binfmt_misc ip_tables uas usb_storage amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched crct10dif_pclmul drm_kms_helper crc32_pclmul crc32c_intel cec drm ghash_clmulni_intel ccp igb nvme dca nvme_core i2c_algo_bit wmi pinctrl_amd fuse

[185000.747878] ---[ end trace df51d3d2498d767d ]---
[185000.747882] RIP: 0010:__list_add_valid.cold+0xf/0x3f
[185000.747886] Code: 48 c7 c6 7c a9 64 84 48 89 ef 49 c7 c7 ea ff ff
ff e8 9d 81 01 00 e9 5f ee 9c ff 4c 89 c1 48 c7 c7 48 aa 64 84 e8 74
fd fd ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 f8 aa 64 84 e8 5d
fd fd
[185000.747889] RSP: 0000:ffffc0c1c61cfc10 EFLAGS: 00010286
[185000.747893] RAX: 0000000000000075 RBX: fffffa0d596b0a40 RCX:
0000000000000000
[185000.747895] RDX: ffff9bbc097e97a0 RSI: ffff9bbc097daae0 RDI:
ffff9bbc097daae0
[185000.747898] RBP: ffffe0c1bea61f40 R08: 0000000000000000 R09:
ffffc0c1c61cfa58
[185000.747901] R10: ffffc0c1c61cfa50 R11: 0000000000000000 R12:
ffff9bb537b4f008
[185000.747904] R13: ffff9bba5ac29010 R14: ffff9bb90b444000 R15:
ffff9bba5ac29000
[185000.747907] FS:  00007f198ea257c0(0000) GS:ffff9bbc09600000(0000)
knlGS:0000000000000000
[185000.747910] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[185000.747913] CR2: 00000fe3aa523500 CR3: 000000012f870000 CR4:
0000000000350ee0
[185000.747916] note: Web Content[1588003] exited with preempt_count 6
[185026.580248] watchdog: BUG: soft lockup - CPU#0 stuck for 22s!
[Chrome_ChildIOT:1951362]
[185026.580262] Modules linked in: crypto_user tun snd_seq_dummy

snd_hrtimer uinput nls_utf8 isofs rfcomm netconsole nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd sunrpc vfat fat hid_logitech_hidpp usblp hid_logitech_dj snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi intel_rapl_msr mt76x2u joydev intel_rapl_common snd_hda_intel mt76x2_common iwlmvm snd_intel_dspcfg mt76x02_usb uvcvideo snd_intel_sdw_acpi snd_usb_audio mt76_usb snd_hda_codec edac_mce_amd videobuf2_vmalloc mt76x02_lib videobuf2_memops videobuf2_v4l2 snd_hda_core mt76 videobuf2_common snd_usbmidi_lib kvm_amd btusb snd_hwdep snd_rawmidi videodev btrtl mac80211 kvm snd_seq btbcm btintel snd_seq_device irqbypass libarc4 eeepc_wmi xpad mc bluetooth rapl ff_memless snd_pcm

[185026.580306]  iwlwifi asus_wmi sparse_keymap video snd_timer

ecdh_generic ecc wmi_bmof pcspkr snd cfg80211 soundcore sp5100_tco k10temp i2c_piix4 rfkill acpi_cpufreq binfmt_misc ip_tables uas usb_storage amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched crct10dif_pclmul drm_kms_helper crc32_pclmul crc32c_intel cec drm ghash_clmulni_intel ccp igb nvme dca nvme_core i2c_algo_bit wmi pinctrl_amd fuse

[185026.580334] irq event stamp: 0
[185026.580337] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[185026.580342] hardirqs last disabled at (0): [<ffffffff830dd962>]
copy_process+0x902/0x1df0
[185026.580349] softirqs last  enabled at (0): [<ffffffff830dd962>]
copy_process+0x902/0x1df0
[185026.580353] softirqs last disabled at (0): [<0000000000000000>] 0x0
[185026.580357] CPU: 0 PID: 1951362 Comm: Chrome_ChildIOT Tainted: G
   D W        --------- ---
5.12.0-0.rc1.20210305git280d542f6ffa.164.fc35.x86_64 #1
[185026.580362] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[185026.580365] RIP: 0010:native_queued_spin_lock_slowpath+0x1ce/0x200
[185026.580370] Code: c1 ef 12 83 e0 03 83 ef 01 48 c1 e0 05 48 63 ff
48 05 00 f3 1e 00 48 03 04 fd 20 d9 6e 84 48 89 08 8b 41 08 85 c0 75
09 f3 90 <8b> 41 08 85 c0 74 f7 48 8b 39 48 85 ff 0f 84 56 ff ff ff 0f
0d 0f
[185026.580374] RSP: 0000:ffffc0c1d596fbf0 EFLAGS: 00000246
[185026.580379] RAX: 0000000000000000 RBX: 0000000000f10bea RCX:
ffff9bbc06bef300
[185026.580382] RDX: ffff9bb500b9fb48 RSI: 0000000000040000 RDI:
0000000000000013
[185026.580385] RBP: ffff9bb500b9fb48 R08: 0000000000040000 R09:
0000000000000000
[185026.580388] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff9bb500b9fb60
[185026.580391] R13: ffff9bb500b9fb48 R14: ffffffff84e87020 R15:
ffff9bb500b9fb40
[185026.580394] FS:  00007f2e2ffe6640(0000) GS:ffff9bbc06a00000(0000)
knlGS:0000000000000000
[185026.580398] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[185026.580401] CR2: 00002a8e6ef59188 CR3: 0000000306082000 CR4:
0000000000350ef0
[185026.580405] Call Trace:
[185026.580408]  do_raw_spin_lock+0x94/0xa0
[185026.580665]  _raw_spin_lock+0x63/0x80
[185026.580670]  zswap_frontswap_load+0x30/0x2f0
[185026.580676]  ? trace_hardirqs_on+0x1b/0xe0
[185026.580681]  __frontswap_load+0xc3/0x160
[185026.580685]  swap_readpage+0x257/0x430
[185026.580689]  swapin_readahead+0x450/0x4e0
[185026.580693]  ? lock_release+0x1ef/0x410
[185026.580698]  do_swap_page+0x4a4/0x900
[185026.580703]  __handle_mm_fault+0xbd6/0x1610
[185026.580795]  handle_mm_fault+0xa2/0x270
[185026.580799]  do_user_addr_fault+0x1ea/0x6b0
[185026.580804]  exc_page_fault+0x67/0x2a0
[185026.580808]  ? asm_exc_page_fault+0x8/0x30
[185026.580889]  asm_exc_page_fault+0x1e/0x30
[185026.580893] RIP: 0033:0x55d9d6466038
[185026.580897] Code: cc cc 55 48 89 e5 48 89 7f 10 48 89 7f 18 5d c3
cc cc 55 48 89 e5 48 8b 47 10 48 8b 4f 18 48 89 41 10 48 8b 47 10 48
8b 4f 18 <48> 89 48 18 0f 57 c0 0f 11 47 10 5d c3 cc cc cc cc cc cc cc
cc cc
[185026.580900] RSP: 002b:00007f2e2ffe46b0 EFLAGS: 00010246
[185026.580968] RAX: 00002a8e6ef59170 RBX: 00002a8e6ef40420 RCX:
00002a8e6ef4e370
[185026.580972] RDX: 00002a8e6d6715e0 RSI: 00002a8e6dcb02e0 RDI:
00002a8e6ef40420
[185026.580974] RBP: 00007f2e2ffe46b0 R08: 0000000000000000 R09:
00007fff260af5d0
[185026.581039] R10: 0000000000000000 R11: 0000000000000246 R12:
0000000000000020
[185026.581042] R13: 00002a8e6dcb02e0 R14: 000055d9deb9b1e0 R15:

00002a8e6dcb02e0

Full kernel log is here: https://pastebin.com/WmBLJ3MR

Signature · 2 lines
Best Regards,
Mike Gavrilov.
#9 lore raw
Quoted 7 lines · Hillf Danton

At the first glance, the zero pointer goes out of the box of race because

1/ the Call Trace shows it is the free path (of the supposed race victim),

2/ on the race winner side however either list_del or list_del_init
would not leave a null pointer behind - the list_add captured in this report is under pool->lock.

No more ideas how to fix it?

Kernel panics continue happens again and again with you patches and

recent commits.

[102491.134247] ------------[ cut here ]------------
[102491.134248] list_add corruption. next->prev should be prev
(ffffcc447ea60c78), but was ffffffff8a64ec20. (next=ffff8adc731d3f40).
[102491.134266] ODEBUG: free active (active state 0) object type:
work_struct hint: compact_page_work+0x0/0x10
[102491.134294] ------------[ cut here ]------------
[102491.134295] kernel BUG at lib/list_debug.c:23!
[102491.134299] invalid opcode: 0000 [#1] SMP NOPTI
[102491.134301] CPU: 22 PID: 863413 Comm: kworker/u64:0 Tainted: G
   W        --------- ---
5.12.0-0.rc2.20210310git05a59d79793d.168.fc35.x86_64 #1
[102491.134303] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[102491.134305] Workqueue: zswap3 compact_page_work
[102491.134309] RIP: 0010:__list_add_valid.cold+0xf/0x3f
[102491.134312] Code: 48 c7 c6 34 a9 64 8b 48 89 ef 49 c7 c7 ea ff ff
ff e8 91 81 01 00 e9 ac f5 9c ff 4c 89 c1 48 c7 c7 00 aa 64 8b e8 67
fd fd ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 b0 aa 64 8b e8 50
fd fd
[102491.134313] RSP: 0018:ffffac4494cabde0 EFLAGS: 00010282
[102491.134315] RAX: 0000000000000075 RBX: ffffe6ed86ac8580 RCX:
0000000000000000
[102491.134316] RDX: ffff8ae3497e97a0 RSI: ffff8ae3497daae0 RDI:
ffff8ae3497daae0
[102491.134316] RBP: ffffcc447ea60c78 R08: 0000000000000000 R09:
0000000000000000
[102491.134317] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff8adc4e317a08
[102491.134318] R13: ffff8adceb216010 R14: ffff8adc731d3f40 R15:
ffff8adceb216000
[102491.134319] FS:  0000000000000000(0000) GS:ffff8ae349600000(0000)
knlGS:0000000000000000
[102491.134320] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[102491.134320] CR2: 0000376f1d60afe8 CR3: 00000003b1c28000 CR4:
0000000000350ee0
[102491.134321] Call Trace:
[102491.134324]  do_compact_page+0x28d/0xb60
[102491.134326]  ? debug_object_deactivate+0x55/0x140
[102491.134329]  ? lock_release+0x1ef/0x410
[102491.134331]  ? lock_release+0x1ef/0x410
[102491.134333]  process_one_work+0x2b0/0x5e0
[102491.134337]  worker_thread+0x55/0x3c0
[102491.134339]  ? process_one_work+0x5e0/0x5e0
[102491.134340]  kthread+0x13a/0x150
[102491.134342]  ? __kthread_bind_mask+0x60/0x60
[102491.134345]  ret_from_fork+0x22/0x30
[102491.134349] Modules linked in: snd_seq_dummy snd_hrtimer nls_utf8
isofs uas usb_storage tun uinput rfcomm netconsole nft_objref
nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet
nft_fib_ipv4 nft_fib_ipv6 nft_fib
[102491.134375] WARNING: CPU: 18 PID: 182 at lib/debugobjects.c:505
debug_print_object+0x6e/0x90
[102491.134380]  nft_reject_inet nf_reject_ipv4
[102491.134383] Modules linked in:
[102491.134385]  nf_reject_ipv6 nft_reject
[102491.134388]  snd_seq_dummy
[102491.134390]  nft_ct
[102491.134393]  snd_hrtimer
[102491.134395]  nft_chain_nat nf_nat
[102491.134398]  nls_utf8
[102491.134400]  nf_conntrack nf_defrag_ipv6
[102491.134403]  isofs
[102491.134405]  nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep

zstd sunrpc vfat fat hid_logitech_hidpp hid_logitech_dj intel_rapl_msr intel_rapl_common uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_hda_codec_realtek videobuf2_common mt76x2u mt76x2_common snd_hda_codec_generic mt76x02_usb iwlmvm videodev edac_mce_amd snd_usb_audio ledtrig_audio snd_hda_codec_hdmi mt76_usb mt76x02_lib snd_usbmidi_lib kvm_amd snd_hda_intel mt76 snd_rawmidi snd_intel_dspcfg mc joydev snd_intel_sdw_acpi mac80211 kvm snd_hda_codec snd_hda_core iwlwifi btusb snd_hwdep btrtl snd_seq btbcm btintel eeepc_wmi asus_wmi snd_seq_device irqbypass xpad sparse_keymap bluetooth libarc4 rapl snd_pcm video ff_memless wmi_bmof ecdh_generic ecc cfg80211 pcspkr snd_timer k10temp snd sp5100_tco i2c_piix4 soundcore rfkill acpi_cpufreq binfmt_misc ip_tables amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel cec igb drm nvme ghash_clmulni_intel dca ccp

[102491.134476]  nvme_core i2c_algo_bit wmi pinctrl_amd fuse
[102491.134484] ---[ end trace 562b0b01453e6613 ]---
[102491.134505] RIP: 0010:__list_add_valid.cold+0xf/0x3f
[102491.134509] Code: 48 c7 c6 34 a9 64 8b 48 89 ef 49 c7 c7 ea ff ff
ff e8 91 81 01 00 e9 ac f5 9c ff 4c 89 c1 48 c7 c7 00 aa 64 8b e8 67
fd fd ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 b0 aa 64 8b e8 50
fd fd
[102491.134511] RSP: 0018:ffffac4494cabde0 EFLAGS: 00010282
[102491.134514] RAX: 0000000000000075 RBX: ffffe6ed86ac8580 RCX:
0000000000000000
[102491.134992]  uas usb_storage tun uinput rfcomm netconsole
nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet
nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4
nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat
[102491.135036] RDX: ffff8ae3497e97a0 RSI: ffff8ae3497daae0 RDI:
ffff8ae3497daae0
[102491.135039] RBP: ffffcc447ea60c78 R08: 0000000000000000 R09:
0000000000000000
[102491.135040] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff8adc4e317a08
[102491.135041] R13: ffff8adceb216010 R14: ffff8adc731d3f40 R15:
ffff8adceb216000
[102491.135042] FS:  0000000000000000(0000) GS:ffff8ae349600000(0000)
knlGS:0000000000000000
[102491.135047]  nf_conntrack nf_defrag_ipv6
[102491.135051] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[102491.135054]  nf_defrag_ipv4
[102491.135056] CR2: 0000376f1d60afe8 CR3: 00000003b1c28000 CR4:
0000000000350ee0
[102491.135059]  ip_set
[102491.135061] note: kworker/u64:0[863413] exited with preempt_count 2
[102491.135064]  nf_tables nfnetlink cmac bnep zstd sunrpc vfat fat

hid_logitech_hidpp hid_logitech_dj intel_rapl_msr intel_rapl_common uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_hda_codec_realtek videobuf2_common mt76x2u mt76x2_common snd_hda_codec_generic mt76x02_usb iwlmvm videodev edac_mce_amd snd_usb_audio ledtrig_audio snd_hda_codec_hdmi mt76_usb mt76x02_lib snd_usbmidi_lib kvm_amd snd_hda_intel mt76 snd_rawmidi snd_intel_dspcfg mc joydev snd_intel_sdw_acpi mac80211 kvm snd_hda_codec snd_hda_core iwlwifi btusb snd_hwdep btrtl snd_seq btbcm btintel eeepc_wmi asus_wmi snd_seq_device irqbypass xpad sparse_keymap bluetooth libarc4 rapl snd_pcm video ff_memless wmi_bmof ecdh_generic ecc cfg80211 pcspkr snd_timer k10temp snd sp5100_tco i2c_piix4 soundcore rfkill acpi_cpufreq binfmt_misc ip_tables amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel cec igb drm nvme ghash_clmulni_intel dca ccp nvme_core i2c_algo_bit wmi

[102491.135357]  pinctrl_amd fuse
[102491.135366] CPU: 18 PID: 182 Comm: kcompactd0 Tainted: G      D W
      --------- ---
5.12.0-0.rc2.20210310git05a59d79793d.168.fc35.x86_64 #1
[102491.135369] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[102491.135372] RIP: 0010:debug_print_object+0x6e/0x90
[102491.135403] Code: 49 89 c1 8b 43 10 83 c2 01 8b 4b 14 48 c7 c7 e8
ab 64 8b 89 15 a7 0b 37 03 4c 8b 45 00 48 8b 14 c5 a0 80 2a 8b e8 50
4d 60 00 <0f> 0b 83 05 25 0e 99 01 01 48 83 c4 08 5b 5d c3 83 05 17 0e
99 01
[102491.135406] RSP: 0018:ffffac448080bb78 EFLAGS: 00010296
[102491.135409] RAX: 000000000000005e RBX: ffff8add6de5c7a8 RCX:
0000000000000027
[102491.135412] RDX: ffff8ae348fdaae8 RSI: 0000000000000001 RDI:
ffff8ae348fdaae0
[102491.135415] RBP: ffffffff8b221320 R08: 0000000000000000 R09:
0000000000000000
[102491.135417] R10: 0000000000000000 R11: 0000000000000000 R12:
dead000000000122
[102491.135419] R13: dead000000000100 R14: ffffffff8b221320 R15:
0000000000000005
[102491.135421] FS:  0000000000000000(0000) GS:ffff8ae348e00000(0000)
knlGS:0000000000000000
[102491.135441] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[102491.135443] CR2: 00007f41b9ee7000 CR3: 00000003b1c28000 CR4:
0000000000350ee0
[102491.135446] Call Trace:
[102491.135451]  debug_check_no_obj_freed+0x1db/0x220
[102491.135455]  free_pcp_prepare+0x132/0x270
[102491.135459]  free_unref_page+0x18/0xd0
[102491.135463]  migrate_pages+0x8b9/0x1200
[102491.135467]  ? isolate_freepages_block+0x4a0/0x4a0
[102491.135471]  ? split_map_pages+0x160/0x160
[102491.135490]  compact_zone+0x680/0xfd0
[102491.135493]  ? __free_object+0x2b9/0x300
[102491.135496]  ? lock_release+0x1ef/0x410
[102491.135500]  proactive_compact_node+0x78/0xb0
[102491.135505]  kcompactd+0x38a/0x440
[102491.135509]  ? do_wait_intr_irq+0xd0/0xd0
[102491.135512]  ? kcompactd_do_work+0x3a0/0x3a0
[102491.135515]  kthread+0x13a/0x150
[102491.135520]  ? __kthread_bind_mask+0x60/0x60
[102491.135533]  ret_from_fork+0x22/0x30
[102491.135539] irq event stamp: 220
[102491.135541] hardirqs last  enabled at (219): [<ffffffff8ad62217>]
_raw_spin_unlock_irqrestore+0x37/0x40
[102491.135545] hardirqs last disabled at (220): [<ffffffff8ad5b9a9>]
__schedule+0x6e9/0xb20
[102491.135548] softirqs last  enabled at (0): [<ffffffff8a0dd990>]
copy_process+0x910/0x1e00
[102491.135552] softirqs last disabled at (0): [<0000000000000000>] 0x0
[102491.135555] ---[ end trace 562b0b01453e6614 ]---
[102494.954915] iwlwifi 0000:04:00.0: Error sending SCAN_REQ_UMAC:
time out after 2000ms.
[102494.954950] iwlwifi 0000:04:00.0: Current CMD queue read_ptr 93 write_ptr 94
[102494.956242] iwlwifi 0000:04:00.0: Start IWL Error Log Dump:
[102494.956245] iwlwifi 0000:04:00.0: Status: 0x00000040, count: 6
[102494.956248] iwlwifi 0000:04:00.0: Loaded firmware version:
59.601f3a66.0 cc-a0-59.ucode
[102494.956251] iwlwifi 0000:04:00.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN
[102494.956255] iwlwifi 0000:04:00.0: 0x00A022F0 | trm_hw_status0
[102494.956257] iwlwifi 0000:04:00.0: 0x00000000 | trm_hw_status1
[102494.956260] iwlwifi 0000:04:00.0: 0x004FAA36 | branchlink2
[102494.956262] iwlwifi 0000:04:00.0: 0x000145FA | interruptlink1
[102494.956265] iwlwifi 0000:04:00.0: 0x000145FA | interruptlink2
[102494.956268] iwlwifi 0000:04:00.0: 0x004F51B0 | data1
[102494.956270] iwlwifi 0000:04:00.0: 0x01000000 | data2
[102494.956272] iwlwifi 0000:04:00.0: 0x00000000 | data3
[102494.956275] iwlwifi 0000:04:00.0: 0x00000000 | beacon time
[102494.956277] iwlwifi 0000:04:00.0: 0xD78321C6 | tsf low
[102494.956279] iwlwifi 0000:04:00.0: 0x00000017 | tsf hi
[102494.956282] iwlwifi 0000:04:00.0: 0x00000000 | time gp1
[102494.956284] iwlwifi 0000:04:00.0: 0xD783784B | time gp2
[102494.956286] iwlwifi 0000:04:00.0: 0x00000001 | uCode revision type
[102494.956289] iwlwifi 0000:04:00.0: 0x0000003B | uCode version major
[102494.956291] iwlwifi 0000:04:00.0: 0x601F3A66 | uCode version minor
[102494.956294] iwlwifi 0000:04:00.0: 0x00000340 | hw version
[102494.956296] iwlwifi 0000:04:00.0: 0x00C89000 | board version
[102494.956299] iwlwifi 0000:04:00.0: 0x807DFD04 | hcmd
[102494.956302] iwlwifi 0000:04:00.0: 0x00020000 | isr0
[102494.956304] iwlwifi 0000:04:00.0: 0x01000000 | isr1
[102494.956306] iwlwifi 0000:04:00.0: 0x08F04002 | isr2
[102494.956309] iwlwifi 0000:04:00.0: 0x04C3000C | isr3
[102494.956312] iwlwifi 0000:04:00.0: 0x00000000 | isr4
[102494.956315] iwlwifi 0000:04:00.0: 0x005C019C | last cmd Id
[102494.956318] iwlwifi 0000:04:00.0: 0x004F51B0 | wait_event
[102494.956320] iwlwifi 0000:04:00.0: 0x00004B99 | l2p_control
[102494.956322] iwlwifi 0000:04:00.0: 0x00000000 | l2p_duration
[102494.956325] iwlwifi 0000:04:00.0: 0x00000003 | l2p_mhvalid
[102494.956327] iwlwifi 0000:04:00.0: 0x00000000 | l2p_addr_match
[102494.956329] iwlwifi 0000:04:00.0: 0x0000000B | lmpm_pmg_sel
[102494.956332] iwlwifi 0000:04:00.0: 0x00000000 | timestamp
[102494.956334] iwlwifi 0000:04:00.0: 0x000080EC | flow_handler
[102494.956380] iwlwifi 0000:04:00.0: Start IWL Error Log Dump:
[102494.956382] iwlwifi 0000:04:00.0: Status: 0x00000040, count: 7
[102494.956385] iwlwifi 0000:04:00.0: 0x20000066 | NMI_INTERRUPT_HOST
[102494.956387] iwlwifi 0000:04:00.0: 0x00000000 | umac branchlink1
[102494.956390] iwlwifi 0000:04:00.0: 0x804568FC | umac branchlink2
[102494.956392] iwlwifi 0000:04:00.0: 0xC0084F3C | umac interruptlink1
[102494.956395] iwlwifi 0000:04:00.0: 0x80477750 | umac interruptlink2
[102494.956397] iwlwifi 0000:04:00.0: 0x01000000 | umac data1
[102494.956399] iwlwifi 0000:04:00.0: 0x80477750 | umac data2
[102494.956401] iwlwifi 0000:04:00.0: 0x00000000 | umac data3
[102494.956404] iwlwifi 0000:04:00.0: 0x0000003B | umac major
[102494.956406] iwlwifi 0000:04:00.0: 0x601F3A66 | umac minor
[102494.956408] iwlwifi 0000:04:00.0: 0xD7837848 | frame pointer
[102494.956411] iwlwifi 0000:04:00.0: 0xC0885F30 | stack pointer
[102494.956413] iwlwifi 0000:04:00.0: 0x005D010D | last host cmd
[102494.956415] iwlwifi 0000:04:00.0: 0x00000000 | isr status reg
[102494.956430] iwlwifi 0000:04:00.0: IML/ROM dump:
[102494.956432] iwlwifi 0000:04:00.0: 0x00000003 | IML/ROM error/state
[102494.956446] iwlwifi 0000:04:00.0: 0x00005590 | IML/ROM data1
[102494.956460] iwlwifi 0000:04:00.0: 0x00000080 | IML/ROM WFPM_AUTH_KEY_0
[102494.956470] iwlwifi 0000:04:00.0: Fseq Registers:
[102494.956475] iwlwifi 0000:04:00.0: 0x60000000 | FSEQ_ERROR_CODE
[102494.956480] iwlwifi 0000:04:00.0: 0x80290021 | FSEQ_TOP_INIT_VERSION
[102494.956486] iwlwifi 0000:04:00.0: 0x00050008 | FSEQ_CNVIO_INIT_VERSION
[102494.956491] iwlwifi 0000:04:00.0: 0x0000A503 | FSEQ_OTP_VERSION
[102494.956496] iwlwifi 0000:04:00.0: 0x80000003 | FSEQ_TOP_CONTENT_VERSION
[102494.956502] iwlwifi 0000:04:00.0: 0x4552414E | FSEQ_ALIVE_TOKEN
[102494.956507] iwlwifi 0000:04:00.0: 0x00100530 | FSEQ_CNVI_ID
[102494.956512] iwlwifi 0000:04:00.0: 0x00000532 | FSEQ_CNVR_ID
[102494.956518] iwlwifi 0000:04:00.0: 0x00100530 | CNVI_AUX_MISC_CHIP
[102494.956525] iwlwifi 0000:04:00.0: 0x00000532 | CNVR_AUX_MISC_CHIP
[102494.956532] iwlwifi 0000:04:00.0: 0x05B0905B |
CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
[102494.956540] iwlwifi 0000:04:00.0: 0x0000025B |
CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR
[102494.956762] iwlwifi 0000:04:00.0: WRT: Collecting data: ini trigger 4 fired.
[102494.956789] ieee80211 phy0: Hardware restart was requested
[102494.956816] iwlwifi 0000:04:00.0: Scan failed! ret -110
[102494.956925] ------------[ cut here ]------------
[102494.956928] WARNING: CPU: 30 PID: 930660 at
net/mac80211/scan.c:411 __ieee80211_scan_completed+0x2bb/0x520
[mac80211]
[102494.956962] Modules linked in: snd_seq_dummy snd_hrtimer nls_utf8

isofs uas usb_storage tun uinput rfcomm netconsole nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd sunrpc vfat fat hid_logitech_hidpp hid_logitech_dj intel_rapl_msr intel_rapl_common uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_hda_codec_realtek videobuf2_common mt76x2u mt76x2_common snd_hda_codec_generic mt76x02_usb iwlmvm videodev edac_mce_amd snd_usb_audio ledtrig_audio snd_hda_codec_hdmi mt76_usb mt76x02_lib snd_usbmidi_lib kvm_amd snd_hda_intel mt76 snd_rawmidi snd_intel_dspcfg mc joydev snd_intel_sdw_acpi mac80211 kvm snd_hda_codec snd_hda_core iwlwifi btusb snd_hwdep btrtl snd_seq btbcm btintel eeepc_wmi asus_wmi snd_seq_device irqbypass xpad sparse_keymap bluetooth libarc4

[102494.957007]  rapl snd_pcm video ff_memless wmi_bmof ecdh_generic

ecc cfg80211 pcspkr snd_timer k10temp snd sp5100_tco i2c_piix4 soundcore rfkill acpi_cpufreq binfmt_misc ip_tables amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel cec igb drm nvme ghash_clmulni_intel dca ccp nvme_core i2c_algo_bit wmi pinctrl_amd fuse

[102494.957036] CPU: 30 PID: 930660 Comm: kworker/u64:2 Tainted: G
 D W        --------- ---
5.12.0-0.rc2.20210310git05a59d79793d.168.fc35.x86_64 #1
[102494.957039] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[102494.957042] Workqueue: phy0 ieee80211_scan_work [mac80211]
[102494.957073] RIP: 0010:__ieee80211_scan_completed+0x2bb/0x520 [mac80211]
[102494.957826] Code: ca 0f 82 7d 01 00 00 48 89 ef e8 80 2f 00 00 e9
72 fe ff ff 0f 0b 48 83 bd d8 1c 00 00 00 41 be 01 00 00 00 0f 85 9e
fd ff ff <0f> 0b 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 8b 85 a0 1c 00 00
e9 69
[102494.957830] RSP: 0018:ffffac4495033db0 EFLAGS: 00010246
[102494.957834] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
0000000000000000
[102494.957836] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
ffff8adc770f8e00
[102494.957839] RBP: ffff8adc770f8e00 R08: 0000000000000001 R09:
ffffffffc1395e40
[102494.957842] R10: ffffac4495033de8 R11: 0000000000000000 R12:
0000000000000001
[102494.957844] R13: 0000000000000000 R14: 0000000000000001 R15:
ffff8adc770f8e00
[102494.957847] FS:  0000000000000000(0000) GS:ffff8ae34a600000(0000)
knlGS:0000000000000000
[102494.957850] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[102494.957853] CR2: 00003de20ab47000 CR3: 00000001346e0000 CR4:
0000000000350ee0
[102494.957856] Call Trace:
[102494.957862]  ieee80211_scan_work+0x15c/0x860 [mac80211]
[102494.957893]  ? debug_object_deactivate+0x55/0x140
[102494.957899]  ? lock_release+0x1ef/0x410
[102494.957913]  ? lock_release+0x1ef/0x410
[102494.957917]  process_one_work+0x2b0/0x5e0
[102494.957923]  worker_thread+0x55/0x3c0
[102494.957926]  ? process_one_work+0x5e0/0x5e0
[102494.957930]  kthread+0x13a/0x150
[102494.957934]  ? __kthread_bind_mask+0x60/0x60
[102494.957939]  ret_from_fork+0x22/0x30
[102494.957945] irq event stamp: 0
[102494.957948] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[102494.957951] hardirqs last disabled at (0): [<ffffffff8a0dd990>]
copy_process+0x910/0x1e00
[102494.957956] softirqs last  enabled at (0): [<ffffffff8a0dd990>]
copy_process+0x910/0x1e00
[102494.957959] softirqs last disabled at (0): [<0000000000000000>] 0x0
[102494.957962] ---[ end trace 562b0b01453e6615 ]---

Full kernel log is here: https://pastebin.com/A7dwr8ZV

--
Best Regards,
Mike Gavrilov.

#10 lore raw
Quoted 5 lines · On Tue, 16 Mar 2021 00:21:05 +0500 Mikhail Gavrilov wrote:
Quoted 4 lines · Hillf Danton

At the first glance, the zero pointer goes out of the box of race because

1/ the Call Trace shows it is the free path (of the supposed race victim),

Quoted 8 lines
Quoted 3 lines

2/ on the race winner side however either list_del or list_del_init
would not leave a null pointer behind - the list_add captured in this report is under pool->lock.

No more ideas how to fix it?

Kernel panics continue happens again and again with you patches and

recent commits.

Thanks for your report.

Quoted 4 lines
[102491.134247] ------------[ cut here ]------------
[102491.134248] list_add corruption. next->prev should be prev

(ffffcc447ea60c78), but was ffffffff8a64ec20. (next=ffff8adc731d3f40).

The same race as we saw over a couple of weeks.

Quoted 3 lines
[102491.134266] ODEBUG: free active (active state 0) object type:
work_struct hint: compact_page_work+0x0/0x10
[102491.134294] ------------[ cut here ]------------

This is a new one.

Quoted 4 lines
[102491.134295] kernel BUG at lib/list_debug.c:23!
[102491.134299] invalid opcode: 0000 [#1] SMP NOPTI
[102491.134301] CPU: 22 PID: 863413 Comm: kworker/u64:0 Tainted: G
   W        --------- ---

The victim was running on CPU22.

Quoted 118 lines

5.12.0-0.rc2.20210310git05a59d79793d.168.fc35.x86_64 #1

[102491.134303] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[102491.134305] Workqueue: zswap3 compact_page_work
[102491.134309] RIP: 0010:__list_add_valid.cold+0xf/0x3f
[102491.134312] Code: 48 c7 c6 34 a9 64 8b 48 89 ef 49 c7 c7 ea ff ff
ff e8 91 81 01 00 e9 ac f5 9c ff 4c 89 c1 48 c7 c7 00 aa 64 8b e8 67
fd fd ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 b0 aa 64 8b e8 50
fd fd
[102491.134313] RSP: 0018:ffffac4494cabde0 EFLAGS: 00010282
[102491.134315] RAX: 0000000000000075 RBX: ffffe6ed86ac8580 RCX:
0000000000000000
[102491.134316] RDX: ffff8ae3497e97a0 RSI: ffff8ae3497daae0 RDI:
ffff8ae3497daae0
[102491.134316] RBP: ffffcc447ea60c78 R08: 0000000000000000 R09:
0000000000000000
[102491.134317] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff8adc4e317a08
[102491.134318] R13: ffff8adceb216010 R14: ffff8adc731d3f40 R15:
ffff8adceb216000
[102491.134319] FS:  0000000000000000(0000) GS:ffff8ae349600000(0000)
knlGS:0000000000000000
[102491.134320] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[102491.134320] CR2: 0000376f1d60afe8 CR3: 00000003b1c28000 CR4:
0000000000350ee0
[102491.134321] Call Trace:
[102491.134324]  do_compact_page+0x28d/0xb60
[102491.134326]  ? debug_object_deactivate+0x55/0x140
[102491.134329]  ? lock_release+0x1ef/0x410
[102491.134331]  ? lock_release+0x1ef/0x410
[102491.134333]  process_one_work+0x2b0/0x5e0
[102491.134337]  worker_thread+0x55/0x3c0
[102491.134339]  ? process_one_work+0x5e0/0x5e0
[102491.134340]  kthread+0x13a/0x150
[102491.134342]  ? __kthread_bind_mask+0x60/0x60
[102491.134345]  ret_from_fork+0x22/0x30
[102491.134349] Modules linked in: snd_seq_dummy snd_hrtimer nls_utf8
isofs uas usb_storage tun uinput rfcomm netconsole nft_objref
nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet
nft_fib_ipv4 nft_fib_ipv6 nft_fib
[102491.134375] WARNING: CPU: 18 PID: 182 at lib/debugobjects.c:505
debug_print_object+0x6e/0x90
[102491.134380]  nft_reject_inet nf_reject_ipv4
[102491.134383] Modules linked in:
[102491.134385]  nf_reject_ipv6 nft_reject
[102491.134388]  snd_seq_dummy
[102491.134390]  nft_ct
[102491.134393]  snd_hrtimer
[102491.134395]  nft_chain_nat nf_nat
[102491.134398]  nls_utf8
[102491.134400]  nf_conntrack nf_defrag_ipv6
[102491.134403]  isofs
[102491.134405]  nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep

zstd sunrpc vfat fat hid_logitech_hidpp hid_logitech_dj intel_rapl_msr intel_rapl_common uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_hda_codec_realtek videobuf2_common mt76x2u mt76x2_common snd_hda_codec_generic mt76x02_usb iwlmvm videodev edac_mce_amd snd_usb_audio ledtrig_audio snd_hda_codec_hdmi mt76_usb mt76x02_lib snd_usbmidi_lib kvm_amd snd_hda_intel mt76 snd_rawmidi snd_intel_dspcfg mc joydev snd_intel_sdw_acpi mac80211 kvm snd_hda_codec snd_hda_core iwlwifi btusb snd_hwdep btrtl snd_seq btbcm btintel eeepc_wmi asus_wmi snd_seq_device irqbypass xpad sparse_keymap bluetooth libarc4 rapl snd_pcm video ff_memless wmi_bmof ecdh_generic ecc cfg80211 pcspkr snd_timer k10temp snd sp5100_tco i2c_piix4 soundcore rfkill acpi_cpufreq binfmt_misc ip_tables amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel cec igb drm nvme ghash_clmulni_intel dca ccp

[102491.134476]  nvme_core i2c_algo_bit wmi pinctrl_amd fuse
[102491.134484] ---[ end trace 562b0b01453e6613 ]---
[102491.134505] RIP: 0010:__list_add_valid.cold+0xf/0x3f
[102491.134509] Code: 48 c7 c6 34 a9 64 8b 48 89 ef 49 c7 c7 ea ff ff
ff e8 91 81 01 00 e9 ac f5 9c ff 4c 89 c1 48 c7 c7 00 aa 64 8b e8 67
fd fd ff <0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 b0 aa 64 8b e8 50
fd fd
[102491.134511] RSP: 0018:ffffac4494cabde0 EFLAGS: 00010282
[102491.134514] RAX: 0000000000000075 RBX: ffffe6ed86ac8580 RCX:
0000000000000000
[102491.134992]  uas usb_storage tun uinput rfcomm netconsole
nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet
nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4
nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat
[102491.135036] RDX: ffff8ae3497e97a0 RSI: ffff8ae3497daae0 RDI:
ffff8ae3497daae0
[102491.135039] RBP: ffffcc447ea60c78 R08: 0000000000000000 R09:
0000000000000000
[102491.135040] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff8adc4e317a08
[102491.135041] R13: ffff8adceb216010 R14: ffff8adc731d3f40 R15:
ffff8adceb216000
[102491.135042] FS:  0000000000000000(0000) GS:ffff8ae349600000(0000)
knlGS:0000000000000000
[102491.135047]  nf_conntrack nf_defrag_ipv6
[102491.135051] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[102491.135054]  nf_defrag_ipv4
[102491.135056] CR2: 0000376f1d60afe8 CR3: 00000003b1c28000 CR4:
0000000000350ee0
[102491.135059]  ip_set
[102491.135061] note: kworker/u64:0[863413] exited with preempt_count 2
[102491.135064]  nf_tables nfnetlink cmac bnep zstd sunrpc vfat fat

hid_logitech_hidpp hid_logitech_dj intel_rapl_msr intel_rapl_common uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_hda_codec_realtek videobuf2_common mt76x2u mt76x2_common snd_hda_codec_generic mt76x02_usb iwlmvm videodev edac_mce_amd snd_usb_audio ledtrig_audio snd_hda_codec_hdmi mt76_usb mt76x02_lib snd_usbmidi_lib kvm_amd snd_hda_intel mt76 snd_rawmidi snd_intel_dspcfg mc joydev snd_intel_sdw_acpi mac80211 kvm snd_hda_codec snd_hda_core iwlwifi btusb snd_hwdep btrtl snd_seq btbcm btintel eeepc_wmi asus_wmi snd_seq_device irqbypass xpad sparse_keymap bluetooth libarc4 rapl snd_pcm video ff_memless wmi_bmof ecdh_generic ecc cfg80211 pcspkr snd_timer k10temp snd sp5100_tco i2c_piix4 soundcore rfkill acpi_cpufreq binfmt_misc ip_tables amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel cec igb drm nvme ghash_clmulni_intel dca ccp nvme_core i2c_algo_bit wmi

Quoted 3 lines
[102491.135357]  pinctrl_amd fuse
[102491.135366] CPU: 18 PID: 182 Comm: kcompactd0 Tainted: G      D W
      --------- ---

Was the culprit running on CPU 18?

Quoted 205 lines

5.12.0-0.rc2.20210310git05a59d79793d.168.fc35.x86_64 #1

[102491.135369] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[102491.135372] RIP: 0010:debug_print_object+0x6e/0x90
[102491.135403] Code: 49 89 c1 8b 43 10 83 c2 01 8b 4b 14 48 c7 c7 e8
ab 64 8b 89 15 a7 0b 37 03 4c 8b 45 00 48 8b 14 c5 a0 80 2a 8b e8 50
4d 60 00 <0f> 0b 83 05 25 0e 99 01 01 48 83 c4 08 5b 5d c3 83 05 17 0e
99 01
[102491.135406] RSP: 0018:ffffac448080bb78 EFLAGS: 00010296
[102491.135409] RAX: 000000000000005e RBX: ffff8add6de5c7a8 RCX:
0000000000000027
[102491.135412] RDX: ffff8ae348fdaae8 RSI: 0000000000000001 RDI:
ffff8ae348fdaae0
[102491.135415] RBP: ffffffff8b221320 R08: 0000000000000000 R09:
0000000000000000
[102491.135417] R10: 0000000000000000 R11: 0000000000000000 R12:
dead000000000122
[102491.135419] R13: dead000000000100 R14: ffffffff8b221320 R15:
0000000000000005
[102491.135421] FS:  0000000000000000(0000) GS:ffff8ae348e00000(0000)
knlGS:0000000000000000
[102491.135441] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[102491.135443] CR2: 00007f41b9ee7000 CR3: 00000003b1c28000 CR4:
0000000000350ee0
[102491.135446] Call Trace:
[102491.135451]  debug_check_no_obj_freed+0x1db/0x220
[102491.135455]  free_pcp_prepare+0x132/0x270
[102491.135459]  free_unref_page+0x18/0xd0
[102491.135463]  migrate_pages+0x8b9/0x1200
[102491.135467]  ? isolate_freepages_block+0x4a0/0x4a0
[102491.135471]  ? split_map_pages+0x160/0x160
[102491.135490]  compact_zone+0x680/0xfd0
[102491.135493]  ? __free_object+0x2b9/0x300
[102491.135496]  ? lock_release+0x1ef/0x410
[102491.135500]  proactive_compact_node+0x78/0xb0
[102491.135505]  kcompactd+0x38a/0x440
[102491.135509]  ? do_wait_intr_irq+0xd0/0xd0
[102491.135512]  ? kcompactd_do_work+0x3a0/0x3a0
[102491.135515]  kthread+0x13a/0x150
[102491.135520]  ? __kthread_bind_mask+0x60/0x60
[102491.135533]  ret_from_fork+0x22/0x30
[102491.135539] irq event stamp: 220
[102491.135541] hardirqs last  enabled at (219): [<ffffffff8ad62217>]
_raw_spin_unlock_irqrestore+0x37/0x40
[102491.135545] hardirqs last disabled at (220): [<ffffffff8ad5b9a9>]
__schedule+0x6e9/0xb20
[102491.135548] softirqs last  enabled at (0): [<ffffffff8a0dd990>]
copy_process+0x910/0x1e00
[102491.135552] softirqs last disabled at (0): [<0000000000000000>] 0x0
[102491.135555] ---[ end trace 562b0b01453e6614 ]---
[102494.954915] iwlwifi 0000:04:00.0: Error sending SCAN_REQ_UMAC:
time out after 2000ms.
[102494.954950] iwlwifi 0000:04:00.0: Current CMD queue read_ptr 93 write_ptr 94
[102494.956242] iwlwifi 0000:04:00.0: Start IWL Error Log Dump:
[102494.956245] iwlwifi 0000:04:00.0: Status: 0x00000040, count: 6
[102494.956248] iwlwifi 0000:04:00.0: Loaded firmware version:
59.601f3a66.0 cc-a0-59.ucode
[102494.956251] iwlwifi 0000:04:00.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN
[102494.956255] iwlwifi 0000:04:00.0: 0x00A022F0 | trm_hw_status0
[102494.956257] iwlwifi 0000:04:00.0: 0x00000000 | trm_hw_status1
[102494.956260] iwlwifi 0000:04:00.0: 0x004FAA36 | branchlink2
[102494.956262] iwlwifi 0000:04:00.0: 0x000145FA | interruptlink1
[102494.956265] iwlwifi 0000:04:00.0: 0x000145FA | interruptlink2
[102494.956268] iwlwifi 0000:04:00.0: 0x004F51B0 | data1
[102494.956270] iwlwifi 0000:04:00.0: 0x01000000 | data2
[102494.956272] iwlwifi 0000:04:00.0: 0x00000000 | data3
[102494.956275] iwlwifi 0000:04:00.0: 0x00000000 | beacon time
[102494.956277] iwlwifi 0000:04:00.0: 0xD78321C6 | tsf low
[102494.956279] iwlwifi 0000:04:00.0: 0x00000017 | tsf hi
[102494.956282] iwlwifi 0000:04:00.0: 0x00000000 | time gp1
[102494.956284] iwlwifi 0000:04:00.0: 0xD783784B | time gp2
[102494.956286] iwlwifi 0000:04:00.0: 0x00000001 | uCode revision type
[102494.956289] iwlwifi 0000:04:00.0: 0x0000003B | uCode version major
[102494.956291] iwlwifi 0000:04:00.0: 0x601F3A66 | uCode version minor
[102494.956294] iwlwifi 0000:04:00.0: 0x00000340 | hw version
[102494.956296] iwlwifi 0000:04:00.0: 0x00C89000 | board version
[102494.956299] iwlwifi 0000:04:00.0: 0x807DFD04 | hcmd
[102494.956302] iwlwifi 0000:04:00.0: 0x00020000 | isr0
[102494.956304] iwlwifi 0000:04:00.0: 0x01000000 | isr1
[102494.956306] iwlwifi 0000:04:00.0: 0x08F04002 | isr2
[102494.956309] iwlwifi 0000:04:00.0: 0x04C3000C | isr3
[102494.956312] iwlwifi 0000:04:00.0: 0x00000000 | isr4
[102494.956315] iwlwifi 0000:04:00.0: 0x005C019C | last cmd Id
[102494.956318] iwlwifi 0000:04:00.0: 0x004F51B0 | wait_event
[102494.956320] iwlwifi 0000:04:00.0: 0x00004B99 | l2p_control
[102494.956322] iwlwifi 0000:04:00.0: 0x00000000 | l2p_duration
[102494.956325] iwlwifi 0000:04:00.0: 0x00000003 | l2p_mhvalid
[102494.956327] iwlwifi 0000:04:00.0: 0x00000000 | l2p_addr_match
[102494.956329] iwlwifi 0000:04:00.0: 0x0000000B | lmpm_pmg_sel
[102494.956332] iwlwifi 0000:04:00.0: 0x00000000 | timestamp
[102494.956334] iwlwifi 0000:04:00.0: 0x000080EC | flow_handler
[102494.956380] iwlwifi 0000:04:00.0: Start IWL Error Log Dump:
[102494.956382] iwlwifi 0000:04:00.0: Status: 0x00000040, count: 7
[102494.956385] iwlwifi 0000:04:00.0: 0x20000066 | NMI_INTERRUPT_HOST
[102494.956387] iwlwifi 0000:04:00.0: 0x00000000 | umac branchlink1
[102494.956390] iwlwifi 0000:04:00.0: 0x804568FC | umac branchlink2
[102494.956392] iwlwifi 0000:04:00.0: 0xC0084F3C | umac interruptlink1
[102494.956395] iwlwifi 0000:04:00.0: 0x80477750 | umac interruptlink2
[102494.956397] iwlwifi 0000:04:00.0: 0x01000000 | umac data1
[102494.956399] iwlwifi 0000:04:00.0: 0x80477750 | umac data2
[102494.956401] iwlwifi 0000:04:00.0: 0x00000000 | umac data3
[102494.956404] iwlwifi 0000:04:00.0: 0x0000003B | umac major
[102494.956406] iwlwifi 0000:04:00.0: 0x601F3A66 | umac minor
[102494.956408] iwlwifi 0000:04:00.0: 0xD7837848 | frame pointer
[102494.956411] iwlwifi 0000:04:00.0: 0xC0885F30 | stack pointer
[102494.956413] iwlwifi 0000:04:00.0: 0x005D010D | last host cmd
[102494.956415] iwlwifi 0000:04:00.0: 0x00000000 | isr status reg
[102494.956430] iwlwifi 0000:04:00.0: IML/ROM dump:
[102494.956432] iwlwifi 0000:04:00.0: 0x00000003 | IML/ROM error/state
[102494.956446] iwlwifi 0000:04:00.0: 0x00005590 | IML/ROM data1
[102494.956460] iwlwifi 0000:04:00.0: 0x00000080 | IML/ROM WFPM_AUTH_KEY_0
[102494.956470] iwlwifi 0000:04:00.0: Fseq Registers:
[102494.956475] iwlwifi 0000:04:00.0: 0x60000000 | FSEQ_ERROR_CODE
[102494.956480] iwlwifi 0000:04:00.0: 0x80290021 | FSEQ_TOP_INIT_VERSION
[102494.956486] iwlwifi 0000:04:00.0: 0x00050008 | FSEQ_CNVIO_INIT_VERSION
[102494.956491] iwlwifi 0000:04:00.0: 0x0000A503 | FSEQ_OTP_VERSION
[102494.956496] iwlwifi 0000:04:00.0: 0x80000003 | FSEQ_TOP_CONTENT_VERSION
[102494.956502] iwlwifi 0000:04:00.0: 0x4552414E | FSEQ_ALIVE_TOKEN
[102494.956507] iwlwifi 0000:04:00.0: 0x00100530 | FSEQ_CNVI_ID
[102494.956512] iwlwifi 0000:04:00.0: 0x00000532 | FSEQ_CNVR_ID
[102494.956518] iwlwifi 0000:04:00.0: 0x00100530 | CNVI_AUX_MISC_CHIP
[102494.956525] iwlwifi 0000:04:00.0: 0x00000532 | CNVR_AUX_MISC_CHIP
[102494.956532] iwlwifi 0000:04:00.0: 0x05B0905B |
CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
[102494.956540] iwlwifi 0000:04:00.0: 0x0000025B |
CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR
[102494.956762] iwlwifi 0000:04:00.0: WRT: Collecting data: ini trigger 4 fired.
[102494.956789] ieee80211 phy0: Hardware restart was requested
[102494.956816] iwlwifi 0000:04:00.0: Scan failed! ret -110
[102494.956925] ------------[ cut here ]------------
[102494.956928] WARNING: CPU: 30 PID: 930660 at
net/mac80211/scan.c:411 __ieee80211_scan_completed+0x2bb/0x520
[mac80211]
[102494.956962] Modules linked in: snd_seq_dummy snd_hrtimer nls_utf8

isofs uas usb_storage tun uinput rfcomm netconsole nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd sunrpc vfat fat hid_logitech_hidpp hid_logitech_dj intel_rapl_msr intel_rapl_common uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_hda_codec_realtek videobuf2_common mt76x2u mt76x2_common snd_hda_codec_generic mt76x02_usb iwlmvm videodev edac_mce_amd snd_usb_audio ledtrig_audio snd_hda_codec_hdmi mt76_usb mt76x02_lib snd_usbmidi_lib kvm_amd snd_hda_intel mt76 snd_rawmidi snd_intel_dspcfg mc joydev snd_intel_sdw_acpi mac80211 kvm snd_hda_codec snd_hda_core iwlwifi btusb snd_hwdep btrtl snd_seq btbcm btintel eeepc_wmi asus_wmi snd_seq_device irqbypass xpad sparse_keymap bluetooth libarc4

[102494.957007]  rapl snd_pcm video ff_memless wmi_bmof ecdh_generic

ecc cfg80211 pcspkr snd_timer k10temp snd sp5100_tco i2c_piix4 soundcore rfkill acpi_cpufreq binfmt_misc ip_tables amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel cec igb drm nvme ghash_clmulni_intel dca ccp nvme_core i2c_algo_bit wmi pinctrl_amd fuse

[102494.957036] CPU: 30 PID: 930660 Comm: kworker/u64:2 Tainted: G
 D W        --------- ---
5.12.0-0.rc2.20210310git05a59d79793d.168.fc35.x86_64 #1
[102494.957039] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 3402 01/13/2021
[102494.957042] Workqueue: phy0 ieee80211_scan_work [mac80211]
[102494.957073] RIP: 0010:__ieee80211_scan_completed+0x2bb/0x520 [mac80211]
[102494.957826] Code: ca 0f 82 7d 01 00 00 48 89 ef e8 80 2f 00 00 e9
72 fe ff ff 0f 0b 48 83 bd d8 1c 00 00 00 41 be 01 00 00 00 0f 85 9e
fd ff ff <0f> 0b 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 8b 85 a0 1c 00 00
e9 69
[102494.957830] RSP: 0018:ffffac4495033db0 EFLAGS: 00010246
[102494.957834] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
0000000000000000
[102494.957836] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
ffff8adc770f8e00
[102494.957839] RBP: ffff8adc770f8e00 R08: 0000000000000001 R09:
ffffffffc1395e40
[102494.957842] R10: ffffac4495033de8 R11: 0000000000000000 R12:
0000000000000001
[102494.957844] R13: 0000000000000000 R14: 0000000000000001 R15:
ffff8adc770f8e00
[102494.957847] FS:  0000000000000000(0000) GS:ffff8ae34a600000(0000)
knlGS:0000000000000000
[102494.957850] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[102494.957853] CR2: 00003de20ab47000 CR3: 00000001346e0000 CR4:
0000000000350ee0
[102494.957856] Call Trace:
[102494.957862]  ieee80211_scan_work+0x15c/0x860 [mac80211]
[102494.957893]  ? debug_object_deactivate+0x55/0x140
[102494.957899]  ? lock_release+0x1ef/0x410
[102494.957913]  ? lock_release+0x1ef/0x410
[102494.957917]  process_one_work+0x2b0/0x5e0
[102494.957923]  worker_thread+0x55/0x3c0
[102494.957926]  ? process_one_work+0x5e0/0x5e0
[102494.957930]  kthread+0x13a/0x150
[102494.957934]  ? __kthread_bind_mask+0x60/0x60
[102494.957939]  ret_from_fork+0x22/0x30
[102494.957945] irq event stamp: 0
[102494.957948] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[102494.957951] hardirqs last disabled at (0): [<ffffffff8a0dd990>]
copy_process+0x910/0x1e00
[102494.957956] softirqs last  enabled at (0): [<ffffffff8a0dd990>]
copy_process+0x910/0x1e00
[102494.957959] softirqs last disabled at (0): [<0000000000000000>] 0x0
[102494.957962] ---[ end trace 562b0b01453e6615 ]---
Quoted 6 lines

Full kernel log is here: https://pastebin.com/A7dwr8ZV

--
Best Regards,
Mike Gavrilov.

Lets see the race between the work and compact/migrate pathes.

	work				migrate
	----				-------
					VM_BUG_ON_PAGE(!test_bit(PAGE_CLAIMED,
							&page->private), page);

					zhdr = page_address(page);
					if (!z3fold_page_trylock(zhdr))
						return -EAGAIN;

					if (work_pending(&zhdr->work)) {
						z3fold_page_unlock(zhdr);
						return -EAGAIN;
					}
			*&*-->		page->private = 0;
					z3fold_page_unlock(zhdr);

					page_mapcount_reset(page);
					clear_bit(PAGE_CLAIMED, &page->private);
					put_page(page);
					return 0;

	z3fold_page_lock(zhdr);

	if (test_bit(PAGE_STALE, &page->private) ||
	    test_and_set_bit(PAGE_CLAIMED, &page->private)) {
		z3fold_page_unlock(zhdr);
		return;
	}
	...
	z3fold_page_unlock(zhdr);

====
1/ no chance for race at the first glance because 1) PAGE_CLAIMED is checked on both sides and 2) both take z3fold_page_lock(zhdr).

2/ it is bad to reset page->private because it goes odd with the clearing of PAGE_CLAIMED before put_page().

3/ the trigger of the second warning indicates it is necessary to wait for the work to be done before put_page().

That said, the quick fix is to cancel work if the first hunk below wont survive your test. Note it wont make sense without the previous diffs.

mm/z3fold.c +1 -1
@@ -1623,7 +1623,6 @@ static int z3fold_page_migrate(struct ad
 	new_zhdr = page_address(newpage);
 	memcpy(new_zhdr, zhdr, PAGE_SIZE);
 	newpage->private = page->private;
-	page->private = 0;
 	z3fold_page_unlock(zhdr);
 	spin_lock_init(&new_zhdr->page_lock);
 	INIT_WORK(&new_zhdr->work, compact_page_work);
@@ -1654,6 +1653,7 @@ static int z3fold_page_migrate(struct ad
 
 	queue_work_on(new_zhdr->cpu, pool->compact_wq, &new_zhdr->work);
 
+	cancel_work_sync(&zhdr->work);
 	page_mapcount_reset(page);
 	clear_bit(PAGE_CLAIMED, &page->private);
 	put_page(page);
#11 lore raw
In reply to Hillf Danton
Quoted 9 lines · Hillf Danton

At the first glance, the zero pointer goes out of the box of race because

1/ the Call Trace shows it is the free path (of the supposed race victim),

2/ on the race winner side however either list_del or list_del_init
would not leave a null pointer behind - the list_add captured in this report is under pool->lock.

No more ideas how to fix it?

Kernel panics continue happens again and again with you patches and recent

commits.

[102491.134247] ------------[ cut here ]------------
[102491.134248] list_add corruption. next->prev should be prev
(ffffcc447ea60c78), but was ffffffff8a64ec20. (next=ffff8adc731d3f40).
[102491.134266] ODEBUG: free active (active state 0) object type:
work_struct hint: compact_page_work+0x0/0x10
[102491.134294] ------------[ cut here ]------------
[102491.134295] kernel BUG at lib/list_debug.c:23!
[102491.134299] invalid opcode: 0000 [#1] SMP NOPTI
[102491.134301] CPU: 22 PID: 863413 Comm: kworker/u64:0 Tainted: G
W        --------- ---
5.12.0-0.rc2.20210310git05a59d79793d.168.fc35.x86_64 #1
[102491.134303] Hardware name: System manufacturer System Product Name/ROG
STRIX X570-I GAMING, BIOS 3402 01/13/2021
[102491.134305] Workqueue: zswap3 compact_page_work
[102491.134309] RIP: 0010:__list_add_valid.cold+0xf/0x3f
[102491.134312] Code: 48 c7 c6 34 a9 64 8b 48 89 ef 49 c7 c7 ea ff ff ff e8
91 81 01 00 e9 ac f5 9c ff 4c 89 c1 48 c7 c7 00 aa 64 8b e8 67 fd fd ff
<0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 b0 aa 64 8b e8 50 fd fd
[102491.134313] RSP: 0018:ffffac4494cabde0 EFLAGS: 00010282
[102491.134315] RAX: 0000000000000075 RBX: ffffe6ed86ac8580 RCX:
0000000000000000
[102491.134316] RDX: ffff8ae3497e97a0 RSI: ffff8ae3497daae0 RDI:
ffff8ae3497daae0
[102491.134316] RBP: ffffcc447ea60c78 R08: 0000000000000000 R09:
0000000000000000
[102491.134317] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff8adc4e317a08
[102491.134318] R13: ffff8adceb216010 R14: ffff8adc731d3f40 R15:
ffff8adceb216000
[102491.134319] FS:  0000000000000000(0000) GS:ffff8ae349600000(0000)
knlGS:0000000000000000
[102491.134320] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[102491.134320] CR2: 0000376f1d60afe8 CR3: 00000003b1c28000 CR4:
0000000000350ee0
[102491.134321] Call Trace:
[102491.134324]  do_compact_page+0x28d/0xb60
[102491.134326]  ? debug_object_deactivate+0x55/0x140
[102491.134329]  ? lock_release+0x1ef/0x410
[102491.134331]  ? lock_release+0x1ef/0x410
[102491.134333]  process_one_work+0x2b0/0x5e0
[102491.134337]  worker_thread+0x55/0x3c0
[102491.134339]  ? process_one_work+0x5e0/0x5e0
[102491.134340]  kthread+0x13a/0x150
[102491.134342]  ? __kthread_bind_mask+0x60/0x60
[102491.134345]  ret_from_fork+0x22/0x30
[102491.134349] Modules linked in: snd_seq_dummy snd_hrtimer nls_utf8 isofs
uas usb_storage tun uinput rfcomm netconsole nft_objref
nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4
nft_fib_ipv6 nft_fib
[102491.134375] WARNING: CPU: 18 PID: 182 at lib/debugobjects.c:505
debug_print_object+0x6e/0x90
[102491.134380]  nft_reject_inet nf_reject_ipv4
[102491.134383] Modules linked in:
[102491.134385]  nf_reject_ipv6 nft_reject
[102491.134388]  snd_seq_dummy
[102491.134390]  nft_ct
[102491.134393]  snd_hrtimer
[102491.134395]  nft_chain_nat nf_nat
[102491.134398]  nls_utf8
[102491.134400]  nf_conntrack nf_defrag_ipv6
[102491.134403]  isofs
[102491.134405]  nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd

sunrpc vfat fat hid_logitech_hidpp hid_logitech_dj intel_rapl_msr intel_rapl_common uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_hda_codec_realtek videobuf2_common mt76x2u mt76x2_common snd_hda_codec_generic mt76x02_usb iwlmvm videodev edac_mce_amd snd_usb_audio ledtrig_audio snd_hda_codec_hdmi mt76_usb mt76x02_lib snd_usbmidi_lib kvm_amd snd_hda_intel mt76 snd_rawmidi snd_intel_dspcfg mc joydev snd_intel_sdw_acpi mac80211 kvm snd_hda_codec snd_hda_core iwlwifi btusb snd_hwdep btrtl snd_seq btbcm btintel eeepc_wmi asus_wmi snd_seq_device irqbypass xpad sparse_keymap bluetooth libarc4 rapl snd_pcm video ff_memless wmi_bmof ecdh_generic ecc cfg80211 pcspkr snd_timer k10temp snd sp5100_tco i2c_piix4 soundcore rfkill acpi_cpufreq binfmt_misc ip_tables amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel cec igb drm nvme ghash_clmulni_intel dca ccp

[102491.134476]  nvme_core i2c_algo_bit wmi pinctrl_amd fuse
[102491.134484] ---[ end trace 562b0b01453e6613 ]---
[102491.134505] RIP: 0010:__list_add_valid.cold+0xf/0x3f
[102491.134509] Code: 48 c7 c6 34 a9 64 8b 48 89 ef 49 c7 c7 ea ff ff ff e8
91 81 01 00 e9 ac f5 9c ff 4c 89 c1 48 c7 c7 00 aa 64 8b e8 67 fd fd ff
<0f> 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 b0 aa 64 8b e8 50 fd fd
[102491.134511] RSP: 0018:ffffac4494cabde0 EFLAGS: 00010282
[102491.134514] RAX: 0000000000000075 RBX: ffffe6ed86ac8580 RCX:
0000000000000000
[102491.134992]  uas usb_storage tun uinput rfcomm netconsole nft_objref
nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4
nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6
nft_reject nft_ct nft_chain_nat nf_nat
[102491.135036] RDX: ffff8ae3497e97a0 RSI: ffff8ae3497daae0 RDI:
ffff8ae3497daae0
[102491.135039] RBP: ffffcc447ea60c78 R08: 0000000000000000 R09:
0000000000000000
[102491.135040] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff8adc4e317a08
[102491.135041] R13: ffff8adceb216010 R14: ffff8adc731d3f40 R15:
ffff8adceb216000
[102491.135042] FS:  0000000000000000(0000) GS:ffff8ae349600000(0000)
knlGS:0000000000000000
[102491.135047]  nf_conntrack nf_defrag_ipv6
[102491.135051] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[102491.135054]  nf_defrag_ipv4
[102491.135056] CR2: 0000376f1d60afe8 CR3: 00000003b1c28000 CR4:
0000000000350ee0
[102491.135059]  ip_set
[102491.135061] note: kworker/u64:0[863413] exited with preempt_count 2
[102491.135064]  nf_tables nfnetlink cmac bnep zstd sunrpc vfat fat

hid_logitech_hidpp hid_logitech_dj intel_rapl_msr intel_rapl_common uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_hda_codec_realtek videobuf2_common mt76x2u mt76x2_common snd_hda_codec_generic mt76x02_usb iwlmvm videodev edac_mce_amd snd_usb_audio ledtrig_audio snd_hda_codec_hdmi mt76_usb mt76x02_lib snd_usbmidi_lib kvm_amd snd_hda_intel mt76 snd_rawmidi snd_intel_dspcfg mc joydev snd_intel_sdw_acpi mac80211 kvm snd_hda_codec snd_hda_core iwlwifi btusb snd_hwdep btrtl snd_seq btbcm btintel eeepc_wmi asus_wmi snd_seq_device irqbypass xpad sparse_keymap bluetooth libarc4 rapl snd_pcm video ff_memless wmi_bmof ecdh_generic ecc cfg80211 pcspkr snd_timer k10temp snd sp5100_tco i2c_piix4 soundcore rfkill acpi_cpufreq binfmt_misc ip_tables amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel cec igb drm nvme ghash_clmulni_intel dca ccp nvme_core i2c_algo_bit wmi

[102491.135357]  pinctrl_amd fuse
[102491.135366] CPU: 18 PID: 182 Comm: kcompactd0 Tainted: G      D W
  --------- ---  5.12.0-0.rc2.20210310git05a59d79793d.168.fc35.x86_64 #1
[102491.135369] Hardware name: System manufacturer System Product Name/ROG
STRIX X570-I GAMING, BIOS 3402 01/13/2021
[102491.135372] RIP: 0010:debug_print_object+0x6e/0x90
[102491.135403] Code: 49 89 c1 8b 43 10 83 c2 01 8b 4b 14 48 c7 c7 e8 ab 64
8b 89 15 a7 0b 37 03 4c 8b 45 00 48 8b 14 c5 a0 80 2a 8b e8 50 4d 60 00
<0f> 0b 83 05 25 0e 99 01 01 48 83 c4 08 5b 5d c3 83 05 17 0e 99 01
[102491.135406] RSP: 0018:ffffac448080bb78 EFLAGS: 00010296
[102491.135409] RAX: 000000000000005e RBX: ffff8add6de5c7a8 RCX:
0000000000000027
[102491.135412] RDX: ffff8ae348fdaae8 RSI: 0000000000000001 RDI:
ffff8ae348fdaae0
[102491.135415] RBP: ffffffff8b221320 R08: 0000000000000000 R09:
0000000000000000
[102491.135417] R10: 0000000000000000 R11: 0000000000000000 R12:
dead000000000122
[102491.135419] R13: dead000000000100 R14: ffffffff8b221320 R15:
0000000000000005
[102491.135421] FS:  0000000000000000(0000) GS:ffff8ae348e00000(0000)
knlGS:0000000000000000
[102491.135441] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[102491.135443] CR2: 00007f41b9ee7000 CR3: 00000003b1c28000 CR4:
0000000000350ee0
[102491.135446] Call Trace:
[102491.135451]  debug_check_no_obj_freed+0x1db/0x220
[102491.135455]  free_pcp_prepare+0x132/0x270
[102491.135459]  free_unref_page+0x18/0xd0
[102491.135463]  migrate_pages+0x8b9/0x1200
[102491.135467]  ? isolate_freepages_block+0x4a0/0x4a0
[102491.135471]  ? split_map_pages+0x160/0x160
[102491.135490]  compact_zone+0x680/0xfd0
[102491.135493]  ? __free_object+0x2b9/0x300
[102491.135496]  ? lock_release+0x1ef/0x410
[102491.135500]  proactive_compact_node+0x78/0xb0
[102491.135505]  kcompactd+0x38a/0x440
[102491.135509]  ? do_wait_intr_irq+0xd0/0xd0
[102491.135512]  ? kcompactd_do_work+0x3a0/0x3a0
[102491.135515]  kthread+0x13a/0x150
[102491.135520]  ? __kthread_bind_mask+0x60/0x60
[102491.135533]  ret_from_fork+0x22/0x30
[102491.135539] irq event stamp: 220
[102491.135541] hardirqs last  enabled at (219): [<ffffffff8ad62217>]
_raw_spin_unlock_irqrestore+0x37/0x40
[102491.135545] hardirqs last disabled at (220): [<ffffffff8ad5b9a9>]
__schedule+0x6e9/0xb20
[102491.135548] softirqs last  enabled at (0): [<ffffffff8a0dd990>]
copy_process+0x910/0x1e00
[102491.135552] softirqs last disabled at (0): [<0000000000000000>] 0x0
[102491.135555] ---[ end trace 562b0b01453e6614 ]---
[102494.954915] iwlwifi 0000:04:00.0: Error sending SCAN_REQ_UMAC: time out
after 2000ms.
[102494.954950] iwlwifi 0000:04:00.0: Current CMD queue read_ptr 93
write_ptr 94
[102494.956242] iwlwifi 0000:04:00.0: Start IWL Error Log Dump:
[102494.956245] iwlwifi 0000:04:00.0: Status: 0x00000040, count: 6
[102494.956248] iwlwifi 0000:04:00.0: Loaded firmware version:
59.601f3a66.0 cc-a0-59.ucode
[102494.956251] iwlwifi 0000:04:00.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN
[102494.956255] iwlwifi 0000:04:00.0: 0x00A022F0 | trm_hw_status0
[102494.956257] iwlwifi 0000:04:00.0: 0x00000000 | trm_hw_status1
[102494.956260] iwlwifi 0000:04:00.0: 0x004FAA36 | branchlink2
[102494.956262] iwlwifi 0000:04:00.0: 0x000145FA | interruptlink1
[102494.956265] iwlwifi 0000:04:00.0: 0x000145FA | interruptlink2
[102494.956268] iwlwifi 0000:04:00.0: 0x004F51B0 | data1
[102494.956270] iwlwifi 0000:04:00.0: 0x01000000 | data2
[102494.956272] iwlwifi 0000:04:00.0: 0x00000000 | data3
[102494.956275] iwlwifi 0000:04:00.0: 0x00000000 | beacon time
[102494.956277] iwlwifi 0000:04:00.0: 0xD78321C6 | tsf low
[102494.956279] iwlwifi 0000:04:00.0: 0x00000017 | tsf hi
[102494.956282] iwlwifi 0000:04:00.0: 0x00000000 | time gp1
[102494.956284] iwlwifi 0000:04:00.0: 0xD783784B | time gp2
[102494.956286] iwlwifi 0000:04:00.0: 0x00000001 | uCode revision type
[102494.956289] iwlwifi 0000:04:00.0: 0x0000003B | uCode version major
[102494.956291] iwlwifi 0000:04:00.0: 0x601F3A66 | uCode version minor
[102494.956294] iwlwifi 0000:04:00.0: 0x00000340 | hw version
[102494.956296] iwlwifi 0000:04:00.0: 0x00C89000 | board version
[102494.956299] iwlwifi 0000:04:00.0: 0x807DFD04 | hcmd
[102494.956302] iwlwifi 0000:04:00.0: 0x00020000 | isr0
[102494.956304] iwlwifi 0000:04:00.0: 0x01000000 | isr1
[102494.956306] iwlwifi 0000:04:00.0: 0x08F04002 | isr2
[102494.956309] iwlwifi 0000:04:00.0: 0x04C3000C | isr3
[102494.956312] iwlwifi 0000:04:00.0: 0x00000000 | isr4
[102494.956315] iwlwifi 0000:04:00.0: 0x005C019C | last cmd Id
[102494.956318] iwlwifi 0000:04:00.0: 0x004F51B0 | wait_event
[102494.956320] iwlwifi 0000:04:00.0: 0x00004B99 | l2p_control
[102494.956322] iwlwifi 0000:04:00.0: 0x00000000 | l2p_duration
[102494.956325] iwlwifi 0000:04:00.0: 0x00000003 | l2p_mhvalid
[102494.956327] iwlwifi 0000:04:00.0: 0x00000000 | l2p_addr_match
[102494.956329] iwlwifi 0000:04:00.0: 0x0000000B | lmpm_pmg_sel
[102494.956332] iwlwifi 0000:04:00.0: 0x00000000 | timestamp
[102494.956334] iwlwifi 0000:04:00.0: 0x000080EC | flow_handler
[102494.956380] iwlwifi 0000:04:00.0: Start IWL Error Log Dump:
[102494.956382] iwlwifi 0000:04:00.0: Status: 0x00000040, count: 7
[102494.956385] iwlwifi 0000:04:00.0: 0x20000066 | NMI_INTERRUPT_HOST
[102494.956387] iwlwifi 0000:04:00.0: 0x00000000 | umac branchlink1
[102494.956390] iwlwifi 0000:04:00.0: 0x804568FC | umac branchlink2
[102494.956392] iwlwifi 0000:04:00.0: 0xC0084F3C | umac interruptlink1
[102494.956395] iwlwifi 0000:04:00.0: 0x80477750 | umac interruptlink2
[102494.956397] iwlwifi 0000:04:00.0: 0x01000000 | umac data1
[102494.956399] iwlwifi 0000:04:00.0: 0x80477750 | umac data2
[102494.956401] iwlwifi 0000:04:00.0: 0x00000000 | umac data3
[102494.956404] iwlwifi 0000:04:00.0: 0x0000003B | umac major
[102494.956406] iwlwifi 0000:04:00.0: 0x601F3A66 | umac minor
[102494.956408] iwlwifi 0000:04:00.0: 0xD7837848 | frame pointer
[102494.956411] iwlwifi 0000:04:00.0: 0xC0885F30 | stack pointer
[102494.956413] iwlwifi 0000:04:00.0: 0x005D010D | last host cmd
[102494.956415] iwlwifi 0000:04:00.0: 0x00000000 | isr status reg
[102494.956430] iwlwifi 0000:04:00.0: IML/ROM dump:
[102494.956432] iwlwifi 0000:04:00.0: 0x00000003 | IML/ROM error/state
[102494.956446] iwlwifi 0000:04:00.0: 0x00005590 | IML/ROM data1
[102494.956460] iwlwifi 0000:04:00.0: 0x00000080 | IML/ROM WFPM_AUTH_KEY_0
[102494.956470] iwlwifi 0000:04:00.0: Fseq Registers:
[102494.956475] iwlwifi 0000:04:00.0: 0x60000000 | FSEQ_ERROR_CODE
[102494.956480] iwlwifi 0000:04:00.0: 0x80290021 | FSEQ_TOP_INIT_VERSION
[102494.956486] iwlwifi 0000:04:00.0: 0x00050008 | FSEQ_CNVIO_INIT_VERSION
[102494.956491] iwlwifi 0000:04:00.0: 0x0000A503 | FSEQ_OTP_VERSION
[102494.956496] iwlwifi 0000:04:00.0: 0x80000003 | FSEQ_TOP_CONTENT_VERSION
[102494.956502] iwlwifi 0000:04:00.0: 0x4552414E | FSEQ_ALIVE_TOKEN
[102494.956507] iwlwifi 0000:04:00.0: 0x00100530 | FSEQ_CNVI_ID
[102494.956512] iwlwifi 0000:04:00.0: 0x00000532 | FSEQ_CNVR_ID
[102494.956518] iwlwifi 0000:04:00.0: 0x00100530 | CNVI_AUX_MISC_CHIP
[102494.956525] iwlwifi 0000:04:00.0: 0x00000532 | CNVR_AUX_MISC_CHIP
[102494.956532] iwlwifi 0000:04:00.0: 0x05B0905B |
CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
[102494.956540] iwlwifi 0000:04:00.0: 0x0000025B |
CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR
[102494.956762] iwlwifi 0000:04:00.0: WRT: Collecting data: ini trigger 4
fired.
[102494.956789] ieee80211 phy0: Hardware restart was requested
[102494.956816] iwlwifi 0000:04:00.0: Scan failed! ret -110
[102494.956925] ------------[ cut here ]------------
[102494.956928] WARNING: CPU: 30 PID: 930660 at net/mac80211/scan.c:411
__ieee80211_scan_completed+0x2bb/0x520 [mac80211]
[102494.956962] Modules linked in: snd_seq_dummy snd_hrtimer nls_utf8 isofs

uas usb_storage tun uinput rfcomm netconsole nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink cmac bnep zstd sunrpc vfat fat hid_logitech_hidpp hid_logitech_dj intel_rapl_msr intel_rapl_common uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_hda_codec_realtek videobuf2_common mt76x2u mt76x2_common snd_hda_codec_generic mt76x02_usb iwlmvm videodev edac_mce_amd snd_usb_audio ledtrig_audio snd_hda_codec_hdmi mt76_usb mt76x02_lib snd_usbmidi_lib kvm_amd snd_hda_intel mt76 snd_rawmidi snd_intel_dspcfg mc joydev snd_intel_sdw_acpi mac80211 kvm snd_hda_codec snd_hda_core iwlwifi btusb snd_hwdep btrtl snd_seq btbcm btintel eeepc_wmi asus_wmi snd_seq_device irqbypass xpad sparse_keymap bluetooth libarc4

[102494.957007]  rapl snd_pcm video ff_memless wmi_bmof ecdh_generic ecc

cfg80211 pcspkr snd_timer k10temp snd sp5100_tco i2c_piix4 soundcore rfkill acpi_cpufreq binfmt_misc ip_tables amdgpu drm_ttm_helper ttm iommu_v2 gpu_sched drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel cec igb drm nvme ghash_clmulni_intel dca ccp nvme_core i2c_algo_bit wmi pinctrl_amd fuse

[102494.957036] CPU: 30 PID: 930660 Comm: kworker/u64:2 Tainted: G      D
W        --------- ---
5.12.0-0.rc2.20210310git05a59d79793d.168.fc35.x86_64 #1
[102494.957039] Hardware name: System manufacturer System Product Name/ROG
STRIX X570-I GAMING, BIOS 3402 01/13/2021
[102494.957042] Workqueue: phy0 ieee80211_scan_work [mac80211]
[102494.957073] RIP: 0010:__ieee80211_scan_completed+0x2bb/0x520 [mac80211]
[102494.957826] Code: ca 0f 82 7d 01 00 00 48 89 ef e8 80 2f 00 00 e9 72 fe
ff ff 0f 0b 48 83 bd d8 1c 00 00 00 41 be 01 00 00 00 0f 85 9e fd ff ff
<0f> 0b 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 8b 85 a0 1c 00 00 e9 69
[102494.957830] RSP: 0018:ffffac4495033db0 EFLAGS: 00010246
[102494.957834] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
0000000000000000
[102494.957836] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
ffff8adc770f8e00
[102494.957839] RBP: ffff8adc770f8e00 R08: 0000000000000001 R09:
ffffffffc1395e40
[102494.957842] R10: ffffac4495033de8 R11: 0000000000000000 R12:
0000000000000001
[102494.957844] R13: 0000000000000000 R14: 0000000000000001 R15:
ffff8adc770f8e00
[102494.957847] FS:  0000000000000000(0000) GS:ffff8ae34a600000(0000)
knlGS:0000000000000000
[102494.957850] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[102494.957853] CR2: 00003de20ab47000 CR3: 00000001346e0000 CR4:
0000000000350ee0
[102494.957856] Call Trace:
[102494.957862]  ieee80211_scan_work+0x15c/0x860 [mac80211]
[102494.957893]  ? debug_object_deactivate+0x55/0x140
[102494.957899]  ? lock_release+0x1ef/0x410
[102494.957913]  ? lock_release+0x1ef/0x410
[102494.957917]  process_one_work+0x2b0/0x5e0
[102494.957923]  worker_thread+0x55/0x3c0
[102494.957926]  ? process_one_work+0x5e0/0x5e0
[102494.957930]  kthread+0x13a/0x150
[102494.957934]  ? __kthread_bind_mask+0x60/0x60
[102494.957939]  ret_from_fork+0x22/0x30
[102494.957945] irq event stamp: 0
[102494.957948] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[102494.957951] hardirqs last disabled at (0): [<ffffffff8a0dd990>]
copy_process+0x910/0x1e00
[102494.957956] softirqs last  enabled at (0): [<ffffffff8a0dd990>]
copy_process+0x910/0x1e00
[102494.957959] softirqs last disabled at (0): [<0000000000000000>] 0x0
[102494.957962] ---[ end trace 562b0b01453e6615 ]---

Full kernel log is here: https://pastebin.com/A7dwr8ZV

Signature · 2 lines
Best Regards,
Mike Gavrilov.