fstests btrfs/042 triggle 'qgroup reserved space leaked'
Hi,
fstests btrfs/042 triggle 'qgroup reserved space leaked'
kernel source: btrfs misc-next
kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...
dmesg output:
[15788.980873] run fstests btrfs/042 at 2022-09-24 00:40:24 [15803.880347] BTRFS info (device sdb1): using crc32c (crc32c-intel) checksum algorithm [15803.897721] BTRFS info (device sdb1): using free space tree [15803.932525] BTRFS info (device sdb1): enabling ssd optimizations [15818.525145] BTRFS: device fsid b255009c-2a39-49ed-b230-b4e26befd321 devid 1 transid 6 /dev/sdb2 scanned by systemd-udevd (310493) [15818.791882] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm [15818.808805] BTRFS info (device sdb2): using free space tree [15818.837770] BTRFS info (device sdb2): enabling ssd optimizations [15818.847176] BTRFS info (device sdb2): checking UUID tree [15818.911997] BTRFS info (device sdb2): qgroup scan completed (inconsistency flag cleared) [15838.397073] BTRFS warning (device sdb2): qgroup 1/1 has unreleased space, type 0 rsv 12288 [15838.406954] BTRFS warning (device sdb2): qgroup 0/260 has unreleased space, type 0 rsv 4096 [15838.416728] BTRFS warning (device sdb2): qgroup 0/259 has unreleased space, type 0 rsv 4096 [15838.426511] BTRFS warning (device sdb2): qgroup 0/257 has unreleased space, type 0 rsv 4096 [15838.436351] ------------[ cut here ]------------ [15838.442380] WARNING: CPU: 0 PID: 310592 at fs/btrfs/disk-io.c:4668 close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs
fs/btrfs/disk-io.c:4668
if (btrfs_check_quota_leak(fs_info)) {
L4668 WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));
btrfs_err(fs_info, "qgroup reserved space leaked");
}[15838.452948] Modules linked in: ext4 mbcache jbd2 loop rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill ib_core sunrpc dm_multipath amdgpu iommu_v2 gpu_sched drm_buddy intel_rapl_msr intel_rapl_common btrfs sb_edac snd_hda_codec_realtek x86_pkg_temp_thermal snd_hda_codec_generic radeon intel_powerclamp snd_hda_codec_hdmi ledtrig_audio coretemp blake2b_generic snd_hda_intel xor dcdbas kvm_intel mei_wdt raid6_pq snd_intel_dspcfg i2c_algo_bit iTCO_wdt iTCO_vendor_support zstd_compress dell_smm_hwmon snd_intel_sdw_acpi drm_display_helper kvm snd_hda_codec cec snd_hda_core drm_ttm_helper irqbypass snd_hwdep rapl ttm intel_cstate snd_seq dm_mod snd_seq_device drm_kms_helper intel_uncore snd_pcm pcspkr syscopyarea mei_me snd_timer sysfillrect i2c_i801 snd sysimgblt i2c_smbus lpc_ich mei fb_sys_fops soundcore fuse drm xfs sd_mod t10_pi sr_mod cdrom sg crct10dif_pclmul crc32_pclmul bnx2x crc32c_intel ahci libahci ghash_clmulni_intel mdio libata mpt3sas e10 00e [15838.453270] raid_class scsi_transport_sas wmi i2c_dev ipmi_devintf ipmi_msghandler [15838.551107] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 [15838.560649] pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 [15838.659255] pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 [15838.779406] CPU: 0 PID: 310592 Comm: umount Not tainted 6.0.0-7.0.debug.el7.x86_64 #1 [15838.789287] Hardware name: Dell Inc. Precision T7610/0NK70N, BIOS A18 09/11/2019 [15838.798748] RIP: 0010:close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs [15838.805988] Code: c7 00 9b 2f c3 e8 1c e7 ff ff 48 8b 3c 24 be 08 00 00 00 e8 2f 78 70 db f0 41 80 4f 10 02 4c 89 ff e8 31 46 f4 ff 84 c0 74 11 <0f> 0b 48 c7 c6 60 9b 2f c3 4c 89 ff e8 b0 8c ff ff 4c 89 ff 49 8d All code ======== 0: c7 00 9b 2f c3 e8 movl $0xe8c32f9b,(%rax) 6: 1c e7 sbb $0xe7,%al 8: ff (bad) 9: ff 48 8b decl -0x75(%rax) c: 3c 24 cmp $0x24,%al e: be 08 00 00 00 mov $0x8,%esi 13: e8 2f 78 70 db callq 0xffffffffdb707847 18: f0 41 80 4f 10 02 lock orb $0x2,0x10(%r15) 1e: 4c 89 ff mov %r15,%rdi 21: e8 31 46 f4 ff callq 0xfffffffffff44657 26: 84 c0 test %al,%al 28: 74 11 je 0x3b 2a:* 0f 0b ud2 <-- trapping instruction 2c: 48 c7 c6 60 9b 2f c3 mov $0xffffffffc32f9b60,%rsi 33: 4c 89 ff mov %r15,%rdi 36: e8 b0 8c ff ff callq 0xffffffffffff8ceb 3b: 4c 89 ff mov %r15,%rdi
3e: 49 rex.WB
3f: 8d .byte 0x8d
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: 48 c7 c6 60 9b 2f c3 mov $0xffffffffc32f9b60,%rsi 9: 4c 89 ff mov %r15,%rdi c: e8 b0 8c ff ff callq 0xffffffffffff8cc1 11: 4c 89 ff mov %r15,%rdi 14: 49 rex.WB 15: 8d .byte 0x8d [15838.829023] RSP: 0018:ffff88810b7bfb98 EFLAGS: 00010202 [15838.836422] RAX: 0000000000000001 RBX: ffff88828ee54d58 RCX: 0000000000000000 [15838.845734] RDX: 1ffff11064bd1ad3 RSI: 0000000000000008 RDI: ffff888325e8d6a0 [15838.855015] RBP: ffff88828ee54fd0 R08: ffffed13f39c7a21 R09: 0000000000000000 [15838.864347] R10: ffffed1418ee9840 R11: ffff88a0c774c200 R12: ffff88824b4487b0 [15838.873690] R13: ffff88828ee55130 R14: ffff88819ea76ec0 R15: ffff88828ee54000 [15838.883005] FS: 00007f41e8aaa500(0000) GS:ffff889f9ce00000(0000) knlGS:0000000000000000 [15838.893294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15838.901224] CR2: 00007fa141b622d8 CR3: 00000001e2a34003 CR4: 00000000001706f0 [15838.910553] Call Trace: [15838.915219] <TASK> [15838.919527] ? fsnotify_destroy_marks (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/mark.c:839) [15838.926367] ? btrfs_cleanup_one_transaction.cold.75 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/perf_event.h:1189 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:720) btrfs [15838.935255] ? fsnotify_sb_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/atomic64_64.h:22 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-long.h:29 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-instrumented.h:1266 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:95) [15838.941822] ? __fsnotify_vfsmount_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:91) [15838.948905] ? evict_inodes (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713) [15838.954918] ? dispose_list (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713) [15838.960947] ? btrfs_sync_fs (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:1570) btrfs [15838.967745] generic_shutdown_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:491) [15838.974445] kill_anon_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1072 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1086) [15838.980345] btrfs_kill_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:2551) btrfs [15838.987130] deactivate_locked_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:332) [15838.993736] cleanup_mnt (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/namespace.c:1187) [15838.999417] task_work_run (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/task_work.c:177 (discriminator 1)) [15839.005185] exit_to_user_mode_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/resume_user_mode.h:49 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:169 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:201) [15839.012112] syscall_exit_to_user_mode (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:128 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:296) [15839.018875] do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87) [15839.024600] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271) [15839.031881] ? do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87) [15839.037753] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271) [15839.045010] entry_SYSCALL_64_after_hwframe (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/entry_64.S:120) [15839.052194] RIP: 0033:0x7f41e8953a6b [15839.057913] Code: 0f 1e fa 48 89 fe 31 ff e9 72 08 00 00 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 89 63 0a 00 f7 d8 All code ======== 0: 0f 1e fa nop %edx 3: 48 89 fe mov %rdi,%rsi 6: 31 ff xor %edi,%edi 8: e9 72 08 00 00 jmpq 0x87f d: 66 90 xchg %ax,%ax f: f3 0f 1e fa endbr64 13: 31 f6 xor %esi,%esi 15: e9 05 00 00 00 jmpq 0x1f 1a: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 1f: f3 0f 1e fa endbr64 23: b8 a6 00 00 00 mov $0xa6,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 05 ja 0x37 32: c3 retq 33: 0f 1f 40 00 nopl 0x0(%rax)
37: 48 8b 15 89 63 0a 00 mov 0xa6389(%rip),%rdx # 0xa63c7
3e: f7 d8 neg %eax
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 05 ja 0xd 8: c3 retq 9: 0f 1f 40 00 nopl 0x0(%rax) d: 48 8b 15 89 63 0a 00 mov 0xa6389(%rip),%rdx # 0xa639d 14: f7 d8 neg %eax [15839.081093] RSP: 002b:00007fff313c23d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [15839.090894] RAX: 0000000000000000 RBX: 000055d74a755540 RCX: 00007f41e8953a6b [15839.100242] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000055d74a75a550 [15839.109546] RBP: 000055d74a755310 R08: 0000000000000000 R09: 000055d74a754010 [15839.118783] R10: 00007f41e89faaa0 R11: 0000000000000246 R12: 0000000000000000 [15839.128016] R13: 000055d74a75a550 R14: 000055d74a755420 R15: 000055d74a755310 [15839.137246] </TASK> [15839.141444] irq event stamp: 36915 [15839.146842] hardirqs last enabled at (36929): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:45 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:80 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:138 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:264 (discriminator 1)) [15839.157472] hardirqs last disabled at (36942): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:262 (discriminator 1)) [15839.168059] softirqs last enabled at (34740): __do_softirq (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/preempt.h:27 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:415 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:600) [15839.178441] softirqs last disabled at (34701): irq_exit_rcu (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:445 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:650 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:662) [15839.188788] ---[ end trace 0000000000000000 ]--- [15839.195355] BTRFS error (device sdb2): qgroup reserved space leaked [15839.544913] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm [15839.563333] BTRFS info (device sdb2): using free space tree [15839.597640] BTRFS info (device sdb2): enabling ssd optimizations
Best Regards
Wang Yugui ([email protected])
2022/09/24
Quoted 5 lines · Wang Yugui
Hi,
fstests btrfs/042 triggle 'qgroup reserved space leaked'
kernel source: btrfs misc-next
Which commit HEAD?
As I can not reproduce using a somewhat older misc-next.
The HEAD I'm on is 2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8 ("btrfs: stop tracking failed reads in the I/O tree").
If it's a regression it can be much easier to pin down.
Quoted 3 lines
kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...
And any reproducibility? 16 runs no reproduce.
Thanks,
Qu
Quoted 153 lines
dmesg output:
[15788.980873] run fstests btrfs/042 at 2022-09-24 00:40:24 [15803.880347] BTRFS info (device sdb1): using crc32c (crc32c-intel) checksum algorithm [15803.897721] BTRFS info (device sdb1): using free space tree [15803.932525] BTRFS info (device sdb1): enabling ssd optimizations [15818.525145] BTRFS: device fsid b255009c-2a39-49ed-b230-b4e26befd321 devid 1 transid 6 /dev/sdb2 scanned by systemd-udevd (310493) [15818.791882] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm [15818.808805] BTRFS info (device sdb2): using free space tree [15818.837770] BTRFS info (device sdb2): enabling ssd optimizations [15818.847176] BTRFS info (device sdb2): checking UUID tree [15818.911997] BTRFS info (device sdb2): qgroup scan completed (inconsistency flag cleared) [15838.397073] BTRFS warning (device sdb2): qgroup 1/1 has unreleased space, type 0 rsv 12288 [15838.406954] BTRFS warning (device sdb2): qgroup 0/260 has unreleased space, type 0 rsv 4096 [15838.416728] BTRFS warning (device sdb2): qgroup 0/259 has unreleased space, type 0 rsv 4096 [15838.426511] BTRFS warning (device sdb2): qgroup 0/257 has unreleased space, type 0 rsv 4096 [15838.436351] ------------[ cut here ]------------ [15838.442380] WARNING: CPU: 0 PID: 310592 at fs/btrfs/disk-io.c:4668 close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs
fs/btrfs/disk-io.c:4668if (btrfs_check_quota_leak(fs_info)) { L4668 WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG)); btrfs_err(fs_info, "qgroup reserved space leaked"); }[15838.452948] Modules linked in: ext4 mbcache jbd2 loop rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill ib_core sunrpc dm_multipath amdgpu iommu_v2 gpu_sched drm_buddy intel_rapl_msr intel_rapl_common btrfs sb_edac snd_hda_codec_realtek x86_pkg_temp_thermal snd_hda_codec_generic radeon intel_powerclamp snd_hda_codec_hdmi ledtrig_audio coretemp blake2b_generic snd_hda_intel xor dcdbas kvm_intel mei_wdt raid6_pq snd_intel_dspcfg i2c_algo_bit iTCO_wdt iTCO_vendor_support zstd_compress dell_smm_hwmon snd_intel_sdw_acpi drm_display_helper kvm snd_hda_codec cec snd_hda_core drm_ttm_helper irqbypass snd_hwdep rapl ttm intel_cstate snd_seq dm_mod snd_seq_device drm_kms_helper intel_uncore snd_pcm pcspkr syscopyarea mei_me snd_timer sysfillrect i2c_i801 snd sysimgblt i2c_smbus lpc_ich mei fb_sys_fops soundcore fuse drm xfs sd_mod t10_pi sr_mod cdrom sg crct10dif_pclmul crc32_pclmul bnx2x crc32c_intel ahci libahci ghash_clmulni_intel mdio libata mpt3sas e10 00e [15838.453270] raid_class scsi_transport_sas wmi i2c_dev ipmi_devintf ipmi_msghandler [15838.551107] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 [15838.560649] pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 [15838.659255] pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 [15838.779406] CPU: 0 PID: 310592 Comm: umount Not tainted 6.0.0-7.0.debug.el7.x86_64 #1 [15838.789287] Hardware name: Dell Inc. Precision T7610/0NK70N, BIOS A18 09/11/2019 [15838.798748] RIP: 0010:close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs [15838.805988] Code: c7 00 9b 2f c3 e8 1c e7 ff ff 48 8b 3c 24 be 08 00 00 00 e8 2f 78 70 db f0 41 80 4f 10 02 4c 89 ff e8 31 46 f4 ff 84 c0 74 11 <0f> 0b 48 c7 c6 60 9b 2f c3 4c 89 ff e8 b0 8c ff ff 4c 89 ff 49 8d All code ======== 0: c7 00 9b 2f c3 e8 movl $0xe8c32f9b,(%rax) 6: 1c e7 sbb $0xe7,%al 8: ff (bad) 9: ff 48 8b decl -0x75(%rax) c: 3c 24 cmp $0x24,%al e: be 08 00 00 00 mov $0x8,%esi 13: e8 2f 78 70 db callq 0xffffffffdb707847 18: f0 41 80 4f 10 02 lock orb $0x2,0x10(%r15) 1e: 4c 89 ff mov %r15,%rdi 21: e8 31 46 f4 ff callq 0xfffffffffff44657 26: 84 c0 test %al,%al 28: 74 11 je 0x3b 2a:* 0f 0b ud2 <-- trapping instruction 2c: 48 c7 c6 60 9b 2f c3 mov $0xffffffffc32f9b60,%rsi 33: 4c 89 ff mov %r15,%rdi 36: e8 b0 8c ff ff callq 0xffffffffffff8ceb 3b: 4c 89 ff mov %r15,%rdi3e: 49 rex.WB
3f: 8d .byte0x8dCode starting with the faulting instruction
===========================================0: 0f 0b ud2 2: 48 c7 c6 60 9b 2f c3 mov $0xffffffffc32f9b60,%rsi 9: 4c 89 ff mov %r15,%rdi c: e8 b0 8c ff ff callq 0xffffffffffff8cc1 11: 4c 89 ff mov %r15,%rdi 14: 49 rex.WB 15: 8d .byte 0x8d [15838.829023] RSP: 0018:ffff88810b7bfb98 EFLAGS: 00010202 [15838.836422] RAX: 0000000000000001 RBX: ffff88828ee54d58 RCX: 0000000000000000 [15838.845734] RDX: 1ffff11064bd1ad3 RSI: 0000000000000008 RDI: ffff888325e8d6a0 [15838.855015] RBP: ffff88828ee54fd0 R08: ffffed13f39c7a21 R09: 0000000000000000 [15838.864347] R10: ffffed1418ee9840 R11: ffff88a0c774c200 R12: ffff88824b4487b0 [15838.873690] R13: ffff88828ee55130 R14: ffff88819ea76ec0 R15: ffff88828ee54000 [15838.883005] FS: 00007f41e8aaa500(0000) GS:ffff889f9ce00000(0000) knlGS:0000000000000000 [15838.893294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15838.901224] CR2: 00007fa141b622d8 CR3: 00000001e2a34003 CR4: 00000000001706f0 [15838.910553] Call Trace: [15838.915219] <TASK> [15838.919527] ? fsnotify_destroy_marks (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/mark.c:839) [15838.926367] ? btrfs_cleanup_one_transaction.cold.75 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/perf_event.h:1189 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:720) btrfs [15838.935255] ? fsnotify_sb_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/atomic64_64.h:22 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-long.h:29 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-instrumented.h:1266 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:95) [15838.941822] ? __fsnotify_vfsmount_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:91) [15838.948905] ? evict_inodes (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713) [15838.954918] ? dispose_list (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713) [15838.960947] ? btrfs_sync_fs (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:1570) btrfs [15838.967745] generic_shutdown_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:491) [15838.974445] kill_anon_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1072 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1086) [15838.980345] btrfs_kill_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:2551) btrfs [15838.987130] deactivate_locked_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:332) [15838.993736] cleanup_mnt (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/namespace.c:1187) [15838.999417] task_work_run (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/task_work.c:177 (discriminator 1)) [15839.005185] exit_to_user_mode_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/resume_user_mode.h:49 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:169 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:201) [15839.012112] syscall_exit_to_user_mode (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:128 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:296) [15839.018875] do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87) [15839.024600] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271) [15839.031881] ? do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87) [15839.037753] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271) [15839.045010] entry_SYSCALL_64_after_hwframe (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/entry_64.S:120) [15839.052194] RIP: 0033:0x7f41e8953a6b [15839.057913] Code: 0f 1e fa 48 89 fe 31 ff e9 72 08 00 00 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 89 63 0a 00 f7 d8 All code ======== 0: 0f 1e fa nop %edx 3: 48 89 fe mov %rdi,%rsi 6: 31 ff xor %edi,%edi 8: e9 72 08 00 00 jmpq 0x87f d: 66 90 xchg %ax,%ax f: f3 0f 1e fa endbr64 13: 31 f6 xor %esi,%esi 15: e9 05 00 00 00 jmpq 0x1f 1a: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 1f: f3 0f 1e fa endbr64 23: b8 a6 00 00 00 mov $0xa6,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 05 ja 0x37 32: c3 retq 33: 0f 1f 40 00 nopl 0x0(%rax)37: 48 8b 15 89 63 0a 00 mov 0xa6389(%rip),%rdx #
0xa63c7
3e: f7 d8 neg %eaxCode starting with the faulting instruction
===========================================0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 05 ja 0xd 8: c3 retq 9: 0f 1f 40 00 nopl 0x0(%rax) d: 48 8b 15 89 63 0a 00 mov 0xa6389(%rip),%rdx # 0xa639d 14: f7 d8 neg %eax [15839.081093] RSP: 002b:00007fff313c23d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [15839.090894] RAX: 0000000000000000 RBX: 000055d74a755540 RCX: 00007f41e8953a6b [15839.100242] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000055d74a75a550 [15839.109546] RBP: 000055d74a755310 R08: 0000000000000000 R09: 000055d74a754010 [15839.118783] R10: 00007f41e89faaa0 R11: 0000000000000246 R12: 0000000000000000 [15839.128016] R13: 000055d74a75a550 R14: 000055d74a755420 R15: 000055d74a755310 [15839.137246] </TASK> [15839.141444] irq event stamp: 36915 [15839.146842] hardirqs last enabled at (36929): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:45 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:80 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:138 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:264 (discriminator 1)) [15839.157472] hardirqs last disabled at (36942): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:262 (discriminator 1)) [15839.168059] softirqs last enabled at (34740): __do_softirq (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/preempt.h:27 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:415 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:600) [15839.178441] softirqs last disabled at (34701): irq_exit_rcu (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:445 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:650 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:662) [15839.188788] ---[ end trace 0000000000000000 ]--- [15839.195355] BTRFS error (device sdb2): qgroup reserved space leaked [15839.544913] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm [15839.563333] BTRFS info (device sdb2): using free space tree [15839.597640] BTRFS info (device sdb2): enabling ssd optimizationsBest Regards
Wang Yugui ([email protected])
2022/09/24
Quoted 16 lines · Qu Wenruo
Quoted 5 lines · Wang Yugui
Hi,
fstests btrfs/042 triggle 'qgroup reserved space leaked'
kernel source: btrfs misc-next
Which commit HEAD?
As I can not reproduce using a somewhat older misc-next.
The HEAD I'm on is
2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8("btrfs:
stop tracking failed reads in the I/O tree").
Quoted 7 lines
If it's a regression it can be much easier to pin down.
Quoted 3 lines
kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...And any reproducibility? 16 runs no reproduce.
Furthermore, I checked the result from Josef's nightly testing farm:
No such report on btrfs/042.
Any reproducibility or extra patches on your branch?
Thanks,
Qu
Quoted 264 lines
Thanks,
QuQuoted 261 lines
dmesg output:
[15788.980873] run fstests btrfs/042 at 2022-09-24 00:40:24 [15803.880347] BTRFS info (device sdb1): using crc32c (crc32c-intel) checksum algorithm [15803.897721] BTRFS info (device sdb1): using free space tree [15803.932525] BTRFS info (device sdb1): enabling ssd optimizations [15818.525145] BTRFS: device fsid b255009c-2a39-49ed-b230-b4e26befd321 devid 1 transid 6 /dev/sdb2 scanned by systemd-udevd (310493) [15818.791882] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm [15818.808805] BTRFS info (device sdb2): using free space tree [15818.837770] BTRFS info (device sdb2): enabling ssd optimizations [15818.847176] BTRFS info (device sdb2): checking UUID tree [15818.911997] BTRFS info (device sdb2): qgroup scan completed (inconsistency flag cleared) [15838.397073] BTRFS warning (device sdb2): qgroup 1/1 has unreleased space, type 0 rsv 12288 [15838.406954] BTRFS warning (device sdb2): qgroup 0/260 has unreleased space, type 0 rsv 4096 [15838.416728] BTRFS warning (device sdb2): qgroup 0/259 has unreleased space, type 0 rsv 4096 [15838.426511] BTRFS warning (device sdb2): qgroup 0/257 has unreleased space, type 0 rsv 4096 [15838.436351] ------------[ cut here ]------------ [15838.442380] WARNING: CPU: 0 PID: 310592 at fs/btrfs/disk-io.c:4668
close_ctree(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749(discriminator 14)) btrfs
fs/btrfs/disk-io.c:4668
if (btrfs_check_quota_leak(fs_info)) {
L4668 WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));btrfs_err(fs_info, "qgroup reserved space leaked"); }[15838.452948] Modules linked in: ext4 mbcache jbd2 loop
rpcsec_gss_krb5auth_rpcgssnfsv4dns_resolvernfs lockd grace fscache netfs rfkillib_coresunrpcdm_multipathamdgpuiommu_v2gpu_scheddrm_buddyintel_rapl_msrintel_rapl_commonbtrfssb_edacsnd_hda_codec_realtekx86_pkg_temp_thermalsnd_hda_codec_genericradeonintel_powerclampsnd_hda_codec_hdmiledtrig_audiocoretempblake2b_genericsnd_hda_intelxor dcdbaskvm_intelmei_wdtraid6_pqsnd_intel_dspcfgi2c_algo_bitiTCO_wdtiTCO_vendor_supportzstd_compressdell_smm_hwmonsnd_intel_sdw_acpidrm_display_helperkvmsnd_hda_codeccecsnd_hda_coredrm_ttm_helperirqbypasssnd_hwdeprapl ttmintel_cstatesnd_seqdm_modsnd_seq_devicedrm_kms_helperintel_uncoresnd_pcmpcspkr syscopyareamei_mesnd_timersysfillrecti2c_i801snd sysimgblti2c_smbuslpc_ichmeifb_sys_fopssoundcore fuse drm xfssd_modt10_pisr_modcdrom sgcrct10dif_pclmulcrc32_pclmulbnx2xcrc32c_intelahci libahcighash_clmulni_intelmdio libata mpt3sas e10
00e[15838.453270] raid_class scsi_transport_sas wmi i2c_dev ipmi_devintf ipmi_msghandler [15838.551107] Unloaded tainted modules: acpi_cpufreq():1acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1
[15838.560649] pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1
[15838.659255] pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 [15838.779406] CPU: 0 PID: 310592 Comm: umount Not tainted 6.0.0-7.0.debug.el7.x86_64 #1 [15838.789287] Hardware name: Dell Inc. Precision T7610/0NK70N, BIOS A18 09/11/2019 [15838.798748] RIP: 0010:close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs [15838.805988] Code: c7 00 9b 2f c3 e8 1c e7 ff ff 48 8b 3c 24 be 0800 00 00 e8 2f 78 70 db f0 41 80 4f 10 02 4c 89 ff e8 31 46 f4 ff 84 c0 74 11 <0f> 0b 48 c7 c6 60 9b 2f c3 4c 89 ff e8 b0 8c ff ff 4c 89 ff 49 8d
All code
========
0: c7 00 9b 2f c3 e8 movl $0xe8c32f9b,(%rax) 6: 1c e7 sbb $0xe7,%al
8: ff (bad)
9: ff 48 8b decl -0x75(%rax)
c: 3c 24 cmp $0x24,%al
e: be 08 00 00 00 mov $0x8,%esi
13: e8 2f 78 70 db callq0xffffffffdb70784718: f0 41 80 4f 10 02 lock orb $0x2,0x10(%r15) 1e: 4c 89 ff mov %r15,%rdi
21: e8 31 46 f4 ff callq0xfffffffffff4465726: 84 c0 test %al,%al
28: 74 11 je0x3b
2a:* 0f 0b ud2 <-- trapping instruction
2c: 48 c7 c6 60 9b 2f c3 mov $0xffffffffc32f9b60,%rsi 33: 4c 89 ff mov %r15,%rdi
36: e8 b0 8c ff ff callq0xffffffffffff8ceb3b: 4c 89 ff mov %r15,%rdi
3e: 49 rex.WB
3f: 8d .byte0x8dCode starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: 48 c7 c6 60 9b 2f c3 mov $0xffffffffc32f9b60,%rsi 9: 4c 89 ff mov %r15,%rdi
c: e8 b0 8c ff ff callq0xffffffffffff8cc111: 4c 89 ff mov %r15,%rdi
14: 49 rex.WB
15: 8d .byte0x8d[15838.829023] RSP: 0018:ffff88810b7bfb98 EFLAGS: 00010202 [15838.836422] RAX: 0000000000000001 RBX: ffff88828ee54d58 RCX: 0000000000000000 [15838.845734] RDX: 1ffff11064bd1ad3 RSI: 0000000000000008 RDI: ffff888325e8d6a0 [15838.855015] RBP: ffff88828ee54fd0 R08: ffffed13f39c7a21 R09: 0000000000000000 [15838.864347] R10: ffffed1418ee9840 R11: ffff88a0c774c200 R12: ffff88824b4487b0 [15838.873690] R13: ffff88828ee55130 R14: ffff88819ea76ec0 R15: ffff88828ee54000 [15838.883005] FS: 00007f41e8aaa500(0000) GS:ffff889f9ce00000(0000) knlGS:0000000000000000 [15838.893294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15838.901224] CR2: 00007fa141b622d8 CR3: 00000001e2a34003 CR4: 00000000001706f0 [15838.910553] Call Trace: [15838.915219] <TASK> [15838.919527] ? fsnotify_destroy_marks (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/mark.c:839) [15838.926367] ? btrfs_cleanup_one_transaction.cold.75 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/perf_event.h:1189 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:720) btrfs [15838.935255] ? fsnotify_sb_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/atomic64_64.h:22 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-long.h:29 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-instrumented.h:1266 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:95) [15838.941822] ? __fsnotify_vfsmount_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:91) [15838.948905] ? evict_inodes (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713) [15838.954918] ? dispose_list (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713) [15838.960947] ? btrfs_sync_fs (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:1570) btrfs [15838.967745] generic_shutdown_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:491) [15838.974445] kill_anon_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1072 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1086) [15838.980345] btrfs_kill_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:2551) btrfs [15838.987130] deactivate_locked_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:332) [15838.993736] cleanup_mnt (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/namespace.c:1187) [15838.999417] task_work_run (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/task_work.c:177 (discriminator 1)) [15839.005185] exit_to_user_mode_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/resume_user_mode.h:49 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:169 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:201) [15839.012112] syscall_exit_to_user_mode (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:128 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:296) [15839.018875] do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87) [15839.024600] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271) [15839.031881] ? do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87) [15839.037753] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271) [15839.045010] entry_SYSCALL_64_after_hwframe (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/entry_64.S:120) [15839.052194] RIP: 0033:0x7f41e8953a6b [15839.057913] Code: 0f 1e fa 48 89 fe 31 ff e9 72 08 00 00 66 90 f30f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 89 63 0a 00 f7 d8
All code
========
0: 0f 1e fa nop %edx
3: 48 89 fe mov %rdi,%rsi
6: 31 ff xor %edi,%edi
8: e9 72 08 00 00 jmpq0x87f
d: 66 90 xchg %ax,%ax
f: f3 0f 1e fa endbr64
13: 31 f6 xor %esi,%esi
15: e9 05 00 00 00 jmpq0x1f
1a: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 1f: f3 0f 1e fa endbr64
23: b8 a6 00 00 00 mov $0xa6,%eax
28: 0f 05 syscall
2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 05 ja0x37
32: c3 retq
33: 0f 1f 40 00 nopl 0x0(%rax)
37: 48 8b 15 89 63 0a 00 mov 0xa6389(%rip),%rdx #0xa63c7
3e: f7 d8 neg %eaxCode starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 05 ja0xd
8: c3 retq
9: 0f 1f 40 00 nopl 0x0(%rax)
d: 48 8b 15 89 63 0a 00 mov 0xa6389(%rip),%rdx #0xa639d
14: f7 d8 neg %eax[15839.081093] RSP: 002b:00007fff313c23d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [15839.090894] RAX: 0000000000000000 RBX: 000055d74a755540 RCX: 00007f41e8953a6b [15839.100242] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000055d74a75a550 [15839.109546] RBP: 000055d74a755310 R08: 0000000000000000 R09: 000055d74a754010 [15839.118783] R10: 00007f41e89faaa0 R11: 0000000000000246 R12: 0000000000000000 [15839.128016] R13: 000055d74a75a550 R14: 000055d74a755420 R15: 000055d74a755310 [15839.137246] </TASK> [15839.141444] irq event stamp: 36915 [15839.146842] hardirqs last enabled at (36929): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:45 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:80 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:138 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:264 (discriminator 1)) [15839.157472] hardirqs last disabled at (36942): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:262 (discriminator 1)) [15839.168059] softirqs last enabled at (34740): __do_softirq (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/preempt.h:27 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:415 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:600) [15839.178441] softirqs last disabled at (34701): irq_exit_rcu (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:445 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:650 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:662) [15839.188788] ---[ end trace 0000000000000000 ]--- [15839.195355] BTRFS error (device sdb2): qgroup reserved space leaked [15839.544913] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm [15839.563333] BTRFS info (device sdb2): using free space tree [15839.597640] BTRFS info (device sdb2): enabling ssd optimizationsBest Regards
Wang Yugui ([email protected])
2022/09/24
Hi,
Quoted 14 lines · Wang Yugui
Quoted 5 lines
Hi,
fstests btrfs/042 triggle 'qgroup reserved space leaked'
kernel source: btrfs misc-next
Which commit HEAD?
As I can not reproduce using a somewhat older misc-next.
The HEAD I'm on is
2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8("btrfs: stop tracking failed reads in the I/O tree").
Quoted 7 lines
If it's a regression it can be much easier to pin down.
Quoted 3 lines
kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...And any reproducibility? 16 runs no reproduce.
btrfs source version: misc-next: bf940dd88f48,
plus some minor local patch(no qgroup related)
kernel: 6.0-rc6
reproduce rate:
1) 100%(3/3) when local debug config **1
2) 0% (0/3) when local release config
**1:local debug config, about 100x slow than release config
a) memory debug CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/...
b) lockdep debug
CONFIG_PROVE_LOCKING/...
c) btrfs debugCONFIG_BTRFS_FS_CHECK_INTEGRITY=y CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y CONFIG_BTRFS_DEBUG=yCONFIG_BTRFS_ASSERT=yCONFIG_BTRFS_FS_REF_VERIFY=y
From source:fs/btrfs/disk-io.c:4668
if (btrfs_check_quota_leak(fs_info)) {
L4668 WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));
btrfs_err(fs_info, "qgroup reserved space leaked");
}This problem will triggle fstests btrfs/042 to failure only when CONFIG_BTRFS_DEBUG=y ?
maybe related issue:
when lockdep debug is enabled, the following issue become very easy to reproduce too.
https://lore.kernel.org/linux-nfs/[email protected]/
so there maybe some lockdep debug related , but not btrfs related problem in kernel 6.0.
more test(remove some minor local patch(no qgroup related)) will be done, and then I will report the result.
Best Regards
Wang Yugui ([email protected])
2022/09/24
Quoted 156 lines
Thanks,
QuQuoted 153 lines
dmesg output:
[15788.980873] run fstests btrfs/042 at 2022-09-24 00:40:24 [15803.880347] BTRFS info (device sdb1): using crc32c (crc32c-intel) checksum algorithm [15803.897721] BTRFS info (device sdb1): using free space tree [15803.932525] BTRFS info (device sdb1): enabling ssd optimizations [15818.525145] BTRFS: device fsid b255009c-2a39-49ed-b230-b4e26befd321 devid 1 transid 6 /dev/sdb2 scanned by systemd-udevd (310493) [15818.791882] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm [15818.808805] BTRFS info (device sdb2): using free space tree [15818.837770] BTRFS info (device sdb2): enabling ssd optimizations [15818.847176] BTRFS info (device sdb2): checking UUID tree [15818.911997] BTRFS info (device sdb2): qgroup scan completed (inconsistency flag cleared) [15838.397073] BTRFS warning (device sdb2): qgroup 1/1 has unreleased space, type 0 rsv 12288 [15838.406954] BTRFS warning (device sdb2): qgroup 0/260 has unreleased space, type 0 rsv 4096 [15838.416728] BTRFS warning (device sdb2): qgroup 0/259 has unreleased space, type 0 rsv 4096 [15838.426511] BTRFS warning (device sdb2): qgroup 0/257 has unreleased space, type 0 rsv 4096 [15838.436351] ------------[ cut here ]------------ [15838.442380] WARNING: CPU: 0 PID: 310592 at fs/btrfs/disk-io.c:4668 close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs
fs/btrfs/disk-io.c:4668if (btrfs_check_quota_leak(fs_info)) { L4668 WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG)); btrfs_err(fs_info, "qgroup reserved space leaked"); }[15838.452948] Modules linked in: ext4 mbcache jbd2 loop rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill ib_core sunrpc dm_multipath amdgpu iommu_v2 gpu_sched drm_buddy intel_rapl_msr intel_rapl_common btrfs sb_edac snd_hda_codec_realtek x86_pkg_temp_thermal snd_hda_codec_generic radeon intel_powerclamp snd_hda_codec_hdmi ledtrig_audio coretemp blake2b_generic snd_hda_intel xor dcdbas kvm_intel mei_wdt raid6_pq snd_intel_dspcfg i2c_algo_bit iTCO_wdt iTCO_vendor_support zstd_compress dell_smm_hwmon snd_intel_sdw_acpi drm_display_helper kvm snd_hda_codec cec snd_hda_core drm_ttm_helper irqbypass snd_hwdep rapl ttm intel_cstate snd_seq dm_mod snd_seq_device drm_kms_helper intel_uncore snd_pcm pcspkr syscopyarea mei_me snd_timer sysfillrect i2c_i801 snd sysimgblt i2c_smbus lpc_ich mei fb_sys_fops soundcore fuse drm xfs sd_mod t10_pi sr_mod cdrom sg crct10dif_pclmul crc32_pclmul bnx2x crc32c_intel ahci libahci ghash_clmulni_intel mdio libata mpt3sas e10 00e [15838.453270] raid_class scsi_transport_sas wmi i2c_dev ipmi_devintf ipmi_msghandler [15838.551107] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 [15838.560649] pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 [15838.659255] pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 [15838.779406] CPU: 0 PID: 310592 Comm: umount Not tainted 6.0.0-7.0.debug.el7.x86_64 #1 [15838.789287] Hardware name: Dell Inc. Precision T7610/0NK70N, BIOS A18 09/11/2019 [15838.798748] RIP: 0010:close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs [15838.805988] Code: c7 00 9b 2f c3 e8 1c e7 ff ff 48 8b 3c 24 be 08 00 00 00 e8 2f 78 70 db f0 41 80 4f 10 02 4c 89 ff e8 31 46 f4 ff 84 c0 74 11 <0f> 0b 48 c7 c6 60 9b 2f c3 4c 89 ff e8 b0 8c ff ff 4c 89 ff 49 8d All code ======== 0: c7 00 9b 2f c3 e8 movl $0xe8c32f9b,(%rax) 6: 1c e7 sbb $0xe7,%al 8: ff (bad) 9: ff 48 8b decl -0x75(%rax) c: 3c 24 cmp $0x24,%al e: be 08 00 00 00 mov $0x8,%esi 13: e8 2f 78 70 db callq 0xffffffffdb707847 18: f0 41 80 4f 10 02 lock orb $0x2,0x10(%r15) 1e: 4c 89 ff mov %r15,%rdi 21: e8 31 46 f4 ff callq 0xfffffffffff44657 26: 84 c0 test %al,%al 28: 74 11 je 0x3b 2a:* 0f 0b ud2 <-- trapping instruction 2c: 48 c7 c6 60 9b 2f c3 mov $0xffffffffc32f9b60,%rsi 33: 4c 89 ff mov %r15,%rdi 36: e8 b0 8c ff ff callq 0xffffffffffff8ceb 3b: 4c 89 ff mov %r15,%rdi3e: 49 rex.WB
3f: 8d .byte0x8dCode starting with the faulting instruction
===========================================0: 0f 0b ud2 2: 48 c7 c6 60 9b 2f c3 mov $0xffffffffc32f9b60,%rsi 9: 4c 89 ff mov %r15,%rdi c: e8 b0 8c ff ff callq 0xffffffffffff8cc1 11: 4c 89 ff mov %r15,%rdi 14: 49 rex.WB 15: 8d .byte 0x8d [15838.829023] RSP: 0018:ffff88810b7bfb98 EFLAGS: 00010202 [15838.836422] RAX: 0000000000000001 RBX: ffff88828ee54d58 RCX: 0000000000000000 [15838.845734] RDX: 1ffff11064bd1ad3 RSI: 0000000000000008 RDI: ffff888325e8d6a0 [15838.855015] RBP: ffff88828ee54fd0 R08: ffffed13f39c7a21 R09: 0000000000000000 [15838.864347] R10: ffffed1418ee9840 R11: ffff88a0c774c200 R12: ffff88824b4487b0 [15838.873690] R13: ffff88828ee55130 R14: ffff88819ea76ec0 R15: ffff88828ee54000 [15838.883005] FS: 00007f41e8aaa500(0000) GS:ffff889f9ce00000(0000) knlGS:0000000000000000 [15838.893294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15838.901224] CR2: 00007fa141b622d8 CR3: 00000001e2a34003 CR4: 00000000001706f0 [15838.910553] Call Trace: [15838.915219] <TASK> [15838.919527] ? fsnotify_destroy_marks (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/mark.c:839) [15838.926367] ? btrfs_cleanup_one_transaction.cold.75 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/perf_event.h:1189 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:720) btrfs [15838.935255] ? fsnotify_sb_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/atomic64_64.h:22 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-long.h:29 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-instrumented.h:1266 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:95) [15838.941822] ? __fsnotify_vfsmount_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:91) [15838.948905] ? evict_inodes (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713) [15838.954918] ? dispose_list (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713) [15838.960947] ? btrfs_sync_fs (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:1570) btrfs [15838.967745] generic_shutdown_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:491) [15838.974445] kill_anon_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1072 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1086) [15838.980345] btrfs_kill_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:2551) btrfs [15838.987130] deactivate_locked_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:332) [15838.993736] cleanup_mnt (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/namespace.c:1187) [15838.999417] task_work_run (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/task_work.c:177 (discriminator 1)) [15839.005185] exit_to_user_mode_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/resume_user_mode.h:49 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:169 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:201) [15839.012112] syscall_exit_to_user_mode (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:128 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:296) [15839.018875] do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87) [15839.024600] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271) [15839.031881] ? do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87) [15839.037753] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271) [15839.045010] entry_SYSCALL_64_after_hwframe (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/entry_64.S:120) [15839.052194] RIP: 0033:0x7f41e8953a6b [15839.057913] Code: 0f 1e fa 48 89 fe 31 ff e9 72 08 00 00 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 89 63 0a 00 f7 d8 All code ======== 0: 0f 1e fa nop %edx 3: 48 89 fe mov %rdi,%rsi 6: 31 ff xor %edi,%edi 8: e9 72 08 00 00 jmpq 0x87f d: 66 90 xchg %ax,%ax f: f3 0f 1e fa endbr64 13: 31 f6 xor %esi,%esi 15: e9 05 00 00 00 jmpq 0x1f 1a: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 1f: f3 0f 1e fa endbr64 23: b8 a6 00 00 00 mov $0xa6,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 05 ja 0x37 32: c3 retq 33: 0f 1f 40 00 nopl 0x0(%rax)37: 48 8b 15 89 63 0a 00 mov 0xa6389(%rip),%rdx #
0xa63c7
3e: f7 d8 neg %eaxCode starting with the faulting instruction
===========================================0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 05 ja 0xd 8: c3 retq 9: 0f 1f 40 00 nopl 0x0(%rax) d: 48 8b 15 89 63 0a 00 mov 0xa6389(%rip),%rdx # 0xa639d 14: f7 d8 neg %eax [15839.081093] RSP: 002b:00007fff313c23d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [15839.090894] RAX: 0000000000000000 RBX: 000055d74a755540 RCX: 00007f41e8953a6b [15839.100242] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000055d74a75a550 [15839.109546] RBP: 000055d74a755310 R08: 0000000000000000 R09: 000055d74a754010 [15839.118783] R10: 00007f41e89faaa0 R11: 0000000000000246 R12: 0000000000000000 [15839.128016] R13: 000055d74a75a550 R14: 000055d74a755420 R15: 000055d74a755310 [15839.137246] </TASK> [15839.141444] irq event stamp: 36915 [15839.146842] hardirqs last enabled at (36929): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:45 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:80 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:138 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:264 (discriminator 1)) [15839.157472] hardirqs last disabled at (36942): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:262 (discriminator 1)) [15839.168059] softirqs last enabled at (34740): __do_softirq (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/preempt.h:27 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:415 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:600) [15839.178441] softirqs last disabled at (34701): irq_exit_rcu (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:445 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:650 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:662) [15839.188788] ---[ end trace 0000000000000000 ]--- [15839.195355] BTRFS error (device sdb2): qgroup reserved space leaked [15839.544913] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm [15839.563333] BTRFS info (device sdb2): using free space tree [15839.597640] BTRFS info (device sdb2): enabling ssd optimizationsBest Regards
Wang Yugui ([email protected])
2022/09/24
Quoted 32 lines · Wang Yugui
Hi,
Quoted 14 lines · Wang Yugui
Quoted 5 lines
Hi,
fstests btrfs/042 triggle 'qgroup reserved space leaked'
kernel source: btrfs misc-next
Which commit HEAD?
As I can not reproduce using a somewhat older misc-next.
The HEAD I'm on is
2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8("btrfs: stop tracking failed reads in the I/O tree").Quoted 7 lines
If it's a regression it can be much easier to pin down.
Quoted 3 lines
kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...And any reproducibility? 16 runs no reproduce.
btrfs source version: misc-next:
bf940dd88f48,
plus some minor local patch(no qgroup related)
kernel: 6.0-rc6reproduce rate:
1) 100%(3/3) when local debug config **1
2) 0% (0/3) when local release config
Quoted 11 lines
**1:local debug config, about 100x slow than release config
a) memory debug CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/...
b) lockdep debug
CONFIG_PROVE_LOCKING/...
c) btrfs debugCONFIG_BTRFS_FS_CHECK_INTEGRITY=yCONFIG_BTRFS_FS_RUN_SANITY_TESTS=yCONFIG_BTRFS_DEBUG=yCONFIG_BTRFS_ASSERT=yCONFIG_BTRFS_FS_REF_VERIFY=y
I always run with all btrfs features enabled.
So is the lockdep.
KASAN is known to be slow, thus that is only enabled when there is suspision on memory corruption caused by some wild pointer.
Quoted 21 lines
From source:
fs/btrfs/disk-io.c:4668if (btrfs_check_quota_leak(fs_info)) { L4668 WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG)); btrfs_err(fs_info, "qgroup reserved space leaked"); }This problem will triggle fstests btrfs/042 to failure only when
CONFIG_BTRFS_DEBUG=y?maybe related issue:
when lockdep debug is enabled, the following issue become very easy to reproduce too.
https://lore.kernel.org/linux-nfs/[email protected]/
so there maybe some lockdep debug related , but not btrfs related problem in kernel 6.0.
Quoted 2 lines
more test(remove some minor local patch(no qgroup related)) will be done, and then I will report the result.
Better to provide the patches, as I just finished a 16 runs of btrfs/042, no reproduce.
Thus I'm starting to suspect the off-tree patches.
Thanks,
Qu
Quoted 163 lines
Best Regards
Wang Yugui ([email protected])
2022/09/24Quoted 156 lines
Thanks,
QuQuoted 153 lines
dmesg output:
[15788.980873] run fstests btrfs/042 at 2022-09-24 00:40:24 [15803.880347] BTRFS info (device sdb1): using crc32c (crc32c-intel) checksum algorithm [15803.897721] BTRFS info (device sdb1): using free space tree [15803.932525] BTRFS info (device sdb1): enabling ssd optimizations [15818.525145] BTRFS: device fsid b255009c-2a39-49ed-b230-b4e26befd321 devid 1 transid 6 /dev/sdb2 scanned by systemd-udevd (310493) [15818.791882] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm [15818.808805] BTRFS info (device sdb2): using free space tree [15818.837770] BTRFS info (device sdb2): enabling ssd optimizations [15818.847176] BTRFS info (device sdb2): checking UUID tree [15818.911997] BTRFS info (device sdb2): qgroup scan completed (inconsistency flag cleared) [15838.397073] BTRFS warning (device sdb2): qgroup 1/1 has unreleased space, type 0 rsv 12288 [15838.406954] BTRFS warning (device sdb2): qgroup 0/260 has unreleased space, type 0 rsv 4096 [15838.416728] BTRFS warning (device sdb2): qgroup 0/259 has unreleased space, type 0 rsv 4096 [15838.426511] BTRFS warning (device sdb2): qgroup 0/257 has unreleased space, type 0 rsv 4096 [15838.436351] ------------[ cut here ]------------ [15838.442380] WARNING: CPU: 0 PID: 310592 at fs/btrfs/disk-io.c:4668 close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs
fs/btrfs/disk-io.c:4668if (btrfs_check_quota_leak(fs_info)) { L4668 WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG)); btrfs_err(fs_info, "qgroup reserved space leaked"); }[15838.452948] Modules linked in: ext4 mbcache jbd2 loop rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill ib_core sunrpc dm_multipath amdgpu iommu_v2 gpu_sched drm_buddy intel_rapl_msr intel_rapl_common btrfs sb_edac snd_hda_codec_realtek x86_pkg_temp_thermal snd_hda_codec_generic radeon intel_powerclamp snd_hda_codec_hdmi ledtrig_audio coretemp blake2b_generic snd_hda_intel xor dcdbas kvm_intel mei_wdt raid6_pq snd_intel_dspcfg i2c_algo_bit iTCO_wdt iTCO_vendor_support zstd_compress dell_smm_hwmon snd_intel_sdw_acpi drm_display_helper kvm snd_hda_codec cec snd_hda_core drm_ttm_helper irqbypass snd_hwdep rapl ttm intel_cstate snd_seq dm_mod snd_seq_device drm_kms_helper intel_uncore snd_pcm pcspkr syscopyarea mei_me snd_timer sysfillrect i2c_i801 snd sysimgblt i2c_smbus lpc_ich mei fb_sys_fops soundcore fuse drm xfs sd_mod t10_pi sr_mod cdrom sg crct10dif_pclmul crc32_pclmul bnx2x crc32c_intel ahci libahci ghash_clmulni_intel mdio libata mpt3sas e10 00e [15838.453270] raid_class scsi_transport_sas wmi i2c_dev ipmi_devintf ipmi_msghandler [15838.551107] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 [15838.560649] pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 [15838.659255] pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 [15838.779406] CPU: 0 PID: 310592 Comm: umount Not tainted 6.0.0-7.0.debug.el7.x86_64 #1 [15838.789287] Hardware name: Dell Inc. Precision T7610/0NK70N, BIOS A18 09/11/2019 [15838.798748] RIP: 0010:close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs [15838.805988] Code: c7 00 9b 2f c3 e8 1c e7 ff ff 48 8b 3c 24 be 08 00 00 00 e8 2f 78 70 db f0 41 80 4f 10 02 4c 89 ff e8 31 46 f4 ff 84 c0 74 11 <0f> 0b 48 c7 c6 60 9b 2f c3 4c 89 ff e8 b0 8c ff ff 4c 89 ff 49 8d All code ======== 0: c7 00 9b 2f c3 e8 movl $0xe8c32f9b,(%rax) 6: 1c e7 sbb $0xe7,%al 8: ff (bad) 9: ff 48 8b decl -0x75(%rax) c: 3c 24 cmp $0x24,%al e: be 08 00 00 00 mov $0x8,%esi 13: e8 2f 78 70 db callq 0xffffffffdb707847 18: f0 41 80 4f 10 02 lock orb $0x2,0x10(%r15) 1e: 4c 89 ff mov %r15,%rdi 21: e8 31 46 f4 ff callq 0xfffffffffff44657 26: 84 c0 test %al,%al 28: 74 11 je 0x3b 2a:* 0f 0b ud2 <-- trapping instruction 2c: 48 c7 c6 60 9b 2f c3 mov $0xffffffffc32f9b60,%rsi 33: 4c 89 ff mov %r15,%rdi 36: e8 b0 8c ff ff callq 0xffffffffffff8ceb 3b: 4c 89 ff mov %r15,%rdi 3e: 49 rex.WB 3f: 8d .byte 0x8dCode starting with the faulting instruction
===========================================0: 0f 0b ud2 2: 48 c7 c6 60 9b 2f c3 mov $0xffffffffc32f9b60,%rsi 9: 4c 89 ff mov %r15,%rdi c: e8 b0 8c ff ff callq 0xffffffffffff8cc1 11: 4c 89 ff mov %r15,%rdi 14: 49 rex.WB 15: 8d .byte 0x8d [15838.829023] RSP: 0018:ffff88810b7bfb98 EFLAGS: 00010202 [15838.836422] RAX: 0000000000000001 RBX: ffff88828ee54d58 RCX: 0000000000000000 [15838.845734] RDX: 1ffff11064bd1ad3 RSI: 0000000000000008 RDI: ffff888325e8d6a0 [15838.855015] RBP: ffff88828ee54fd0 R08: ffffed13f39c7a21 R09: 0000000000000000 [15838.864347] R10: ffffed1418ee9840 R11: ffff88a0c774c200 R12: ffff88824b4487b0 [15838.873690] R13: ffff88828ee55130 R14: ffff88819ea76ec0 R15: ffff88828ee54000 [15838.883005] FS: 00007f41e8aaa500(0000) GS:ffff889f9ce00000(0000) knlGS:0000000000000000 [15838.893294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15838.901224] CR2: 00007fa141b622d8 CR3: 00000001e2a34003 CR4: 00000000001706f0 [15838.910553] Call Trace: [15838.915219] <TASK> [15838.919527] ? fsnotify_destroy_marks (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/mark.c:839) [15838.926367] ? btrfs_cleanup_one_transaction.cold.75 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/perf_event.h:1189 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:720) btrfs [15838.935255] ? fsnotify_sb_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/atomic64_64.h:22 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-long.h:29 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-instrumented.h:1266 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:95) [15838.941822] ? __fsnotify_vfsmount_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:91) [15838.948905] ? evict_inodes (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713) [15838.954918] ? dispose_list (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713) [15838.960947] ? btrfs_sync_fs (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:1570) btrfs [15838.967745] generic_shutdown_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:491) [15838.974445] kill_anon_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1072 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1086) [15838.980345] btrfs_kill_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:2551) btrfs [15838.987130] deactivate_locked_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:332) [15838.993736] cleanup_mnt (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/namespace.c:1187) [15838.999417] task_work_run (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/task_work.c:177 (discriminator 1)) [15839.005185] exit_to_user_mode_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/resume_user_mode.h:49 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:169 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:201) [15839.012112] syscall_exit_to_user_mode (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:128 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:296) [15839.018875] do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87) [15839.024600] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271) [15839.031881] ? do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87) [15839.037753] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271) [15839.045010] entry_SYSCALL_64_after_hwframe (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/entry_64.S:120) [15839.052194] RIP: 0033:0x7f41e8953a6b [15839.057913] Code: 0f 1e fa 48 89 fe 31 ff e9 72 08 00 00 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 89 63 0a 00 f7 d8 All code ======== 0: 0f 1e fa nop %edx 3: 48 89 fe mov %rdi,%rsi 6: 31 ff xor %edi,%edi 8: e9 72 08 00 00 jmpq 0x87f d: 66 90 xchg %ax,%ax f: f3 0f 1e fa endbr64 13: 31 f6 xor %esi,%esi 15: e9 05 00 00 00 jmpq 0x1f 1a: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 1f: f3 0f 1e fa endbr64 23: b8 a6 00 00 00 mov $0xa6,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 05 ja 0x37 32: c3 retq 33: 0f 1f 40 00 nopl 0x0(%rax) 37: 48 8b 15 89 63 0a 00 mov 0xa6389(%rip),%rdx # 0xa63c7 3e: f7 d8 neg %eaxCode starting with the faulting instruction
===========================================0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 05 ja 0xd 8: c3 retq 9: 0f 1f 40 00 nopl 0x0(%rax) d: 48 8b 15 89 63 0a 00 mov 0xa6389(%rip),%rdx # 0xa639d 14: f7 d8 neg %eax [15839.081093] RSP: 002b:00007fff313c23d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [15839.090894] RAX: 0000000000000000 RBX: 000055d74a755540 RCX: 00007f41e8953a6b [15839.100242] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000055d74a75a550 [15839.109546] RBP: 000055d74a755310 R08: 0000000000000000 R09: 000055d74a754010 [15839.118783] R10: 00007f41e89faaa0 R11: 0000000000000246 R12: 0000000000000000 [15839.128016] R13: 000055d74a75a550 R14: 000055d74a755420 R15: 000055d74a755310 [15839.137246] </TASK> [15839.141444] irq event stamp: 36915 [15839.146842] hardirqs last enabled at (36929): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:45 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:80 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:138 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:264 (discriminator 1)) [15839.157472] hardirqs last disabled at (36942): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:262 (discriminator 1)) [15839.168059] softirqs last enabled at (34740): __do_softirq (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/preempt.h:27 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:415 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:600) [15839.178441] softirqs last disabled at (34701): irq_exit_rcu (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:445 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:650 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:662) [15839.188788] ---[ end trace 0000000000000000 ]--- [15839.195355] BTRFS error (device sdb2): qgroup reserved space leaked [15839.544913] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm [15839.563333] BTRFS info (device sdb2): using free space tree [15839.597640] BTRFS info (device sdb2): enabling ssd optimizationsBest Regards
Wang Yugui ([email protected])
2022/09/24
Hi,
Quoted 72 lines · Wang Yugui
Quoted 32 lines
Hi,
Quoted 14 lines · Wang Yugui
Quoted 5 lines
Hi,
fstests btrfs/042 triggle 'qgroup reserved space leaked'
kernel source: btrfs misc-next
Which commit HEAD?
As I can not reproduce using a somewhat older misc-next.
The HEAD I'm on is
2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8("btrfs: stop tracking failed reads in the I/O tree").Quoted 7 lines
If it's a regression it can be much easier to pin down.
Quoted 3 lines
kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...And any reproducibility? 16 runs no reproduce.
btrfs source version: misc-next:
bf940dd88f48,
plus some minor local patch(no qgroup related)
kernel: 6.0-rc6reproduce rate:
1) 100%(3/3) when local debug config **1
2) 0% (0/3) when local release configQuoted 11 lines
**1:local debug config, about 100x slow than release config
a) memory debug CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/...
b) lockdep debug
CONFIG_PROVE_LOCKING/...
c) btrfs debugCONFIG_BTRFS_FS_CHECK_INTEGRITY=yCONFIG_BTRFS_FS_RUN_SANITY_TESTS=yCONFIG_BTRFS_DEBUG=yCONFIG_BTRFS_ASSERT=yCONFIG_BTRFS_FS_REF_VERIFY=yI always run with all btrfs features enabled.
So is the lockdep.
KASAN is known to be slow, thus that is only enabled when there is suspision on memory corruption caused by some wild pointer.
Quoted 21 lines
From source:
fs/btrfs/disk-io.c:4668if (btrfs_check_quota_leak(fs_info)) { L4668 WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG)); btrfs_err(fs_info, "qgroup reserved space leaked"); }This problem will triggle fstests btrfs/042 to failure only when
CONFIG_BTRFS_DEBUG=y?maybe related issue:
when lockdep debug is enabled, the following issue become very easy to reproduce too.
https://lore.kernel.org/linux-nfs/[email protected]/
so there maybe some lockdep debug related , but not btrfs related problem in kernel 6.0.
Quoted 6 lines
Quoted 2 lines
more test(remove some minor local patch(no qgroup related)) will be done, and then I will report the result.
Better to provide the patches, as I just finished a 16 runs of btrfs/042, no reproduce.
Thus I'm starting to suspect the off-tree patches.
This problem happen on linux 6.0-rc6+ (master a63f2e7cb110, without btrfs misc-next patch, without local off-tree patch)
so this problem is not related to the patches still in btrfs misc-next.
reproduce rate:
100%(3/3) when local debug config
and the whole config file is attached.
this problem does not happen for 5.15.y.
but yet no test result of 5.16.x/5.17.x/5.18.x/5.19.x
Best Regards
Wang Yugui ([email protected])
2022/09/24
Quoted 77 lines · Wang Yugui
Hi,
Quoted 75 lines · Wang Yugui
Quoted 32 lines
Hi,
Quoted 14 lines · Wang Yugui
Quoted 5 lines
Hi,
fstests btrfs/042 triggle 'qgroup reserved space leaked'
kernel source: btrfs misc-next
Which commit HEAD?
As I can not reproduce using a somewhat older misc-next.
The HEAD I'm on is
2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8("btrfs: stop tracking failed reads in the I/O tree").Quoted 7 lines
If it's a regression it can be much easier to pin down.
Quoted 3 lines
kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...And any reproducibility? 16 runs no reproduce.
btrfs source version: misc-next:
bf940dd88f48,
plus some minor local patch(no qgroup related)
kernel: 6.0-rc6reproduce rate:
1) 100%(3/3) when local debug config **1
2) 0% (0/3) when local release configQuoted 11 lines
**1:local debug config, about 100x slow than release config
a) memory debug CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/...
b) lockdep debug
CONFIG_PROVE_LOCKING/...
c) btrfs debugCONFIG_BTRFS_FS_CHECK_INTEGRITY=yCONFIG_BTRFS_FS_RUN_SANITY_TESTS=yCONFIG_BTRFS_DEBUG=yCONFIG_BTRFS_ASSERT=yCONFIG_BTRFS_FS_REF_VERIFY=yI always run with all btrfs features enabled.
So is the lockdep.
KASAN is known to be slow, thus that is only enabled when there is suspision on memory corruption caused by some wild pointer.
Quoted 23 lines
From source:
fs/btrfs/disk-io.c:4668if (btrfs_check_quota_leak(fs_info)) { L4668 WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG)); btrfs_err(fs_info, "qgroup reserved space leaked"); }This problem will triggle fstests btrfs/042 to failure only when
CONFIG_BTRFS_DEBUG=y?maybe related issue:
when lockdep debug is enabled, the following issue become very easy to reproduce too.
https://lore.kernel.org/linux-nfs/[email protected]/
so there maybe some lockdep debug related , but not btrfs related problem in kernel 6.0.more test(remove some minor local patch(no qgroup related)) will be done, and then I will report the result.
Quoted 6 lines
Quoted 3 lines
Better to provide the patches, as I just finished a 16 runs of btrfs/042, no reproduce.
Thus I'm starting to suspect the off-tree patches.
This problem happen on linux 6.0-rc6+ (master
a63f2e7cb110, without btrfs misc-next patch, without local off-tree patch)
Same base, still nope.
Quoted 6 lines
so this problem is not related to the patches still in btrfs misc-next.
reproduce rate:
100%(3/3) when local debug config
and the whole config file is attached.
I don't think the config makes much difference, as the main difference is in KASAN and KMEMLEAK, which should not impact the test result.
And are you running just that test, or with the full auto group?
Quoted 8 lines
this problem does not happen for 5.15.y.
but yet no test result of 5.16.x/5.17.x/5.18.x/5.19.xBest Regards
Wang Yugui ([email protected])
2022/09/24
Hi,
Quoted 95 lines · Wang Yugui
Quoted 77 lines
Hi,
Quoted 75 lines · Wang Yugui
Quoted 32 lines
Hi,
Quoted 14 lines · Wang Yugui
Quoted 5 lines
Hi,
fstests btrfs/042 triggle 'qgroup reserved space leaked'
kernel source: btrfs misc-next
Which commit HEAD?
As I can not reproduce using a somewhat older misc-next.
The HEAD I'm on is
2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8("btrfs: stop tracking failed reads in the I/O tree").Quoted 7 lines
If it's a regression it can be much easier to pin down.
Quoted 3 lines
kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...And any reproducibility? 16 runs no reproduce.
btrfs source version: misc-next:
bf940dd88f48,
plus some minor local patch(no qgroup related)
kernel: 6.0-rc6reproduce rate:
1) 100%(3/3) when local debug config **1
2) 0% (0/3) when local release configQuoted 11 lines
**1:local debug config, about 100x slow than release config
a) memory debug CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/...
b) lockdep debug
CONFIG_PROVE_LOCKING/...
c) btrfs debugCONFIG_BTRFS_FS_CHECK_INTEGRITY=yCONFIG_BTRFS_FS_RUN_SANITY_TESTS=yCONFIG_BTRFS_DEBUG=yCONFIG_BTRFS_ASSERT=yCONFIG_BTRFS_FS_REF_VERIFY=yI always run with all btrfs features enabled.
So is the lockdep.
KASAN is known to be slow, thus that is only enabled when there is suspision on memory corruption caused by some wild pointer.
Quoted 23 lines
From source:
fs/btrfs/disk-io.c:4668if (btrfs_check_quota_leak(fs_info)) { L4668 WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG)); btrfs_err(fs_info, "qgroup reserved space leaked"); }This problem will triggle fstests btrfs/042 to failure only when
CONFIG_BTRFS_DEBUG=y?maybe related issue:
when lockdep debug is enabled, the following issue become very easy to reproduce too.
https://lore.kernel.org/linux-nfs/[email protected]/
so there maybe some lockdep debug related , but not btrfs related problem in kernel 6.0.more test(remove some minor local patch(no qgroup related)) will be done, and then I will report the result.
Quoted 6 lines
Quoted 3 lines
Better to provide the patches, as I just finished a 16 runs of btrfs/042, no reproduce.
Thus I'm starting to suspect the off-tree patches.
This problem happen on linux 6.0-rc6+ (master
a63f2e7cb110, without btrfs misc-next patch, without local off-tree patch)Same base, still nope.
Quoted 6 lines
so this problem is not related to the patches still in btrfs misc-next.
reproduce rate:
100%(3/3) when local debug config
and the whole config file is attached.
Quoted 4 lines
I don't think the config makes much difference, as the main difference is in KASAN and KMEMLEAK, which should not impact the test result.
And are you running just that test, or with the full auto group?
For 6.0-rc6 with btrfs misc-next, I tried to run full auto group. btrfs/042 failed, others(btrfs/001 ~ btrfs/157) are OK, and then I rebooted the test machine.
for 6.0-rc6 without btrfs misc-next, I tested btrfs/042 and btrfs/001 on the same machine.
then I tested 6.0-rc6 without btrfs misc-next on another 2 servers.
reproduce rate:
server1 3/3
server2 2/3
server3 3/3
total rate: 8/9
all 3 servers are in good status(ECC memory status and SSD status).
Best Regards
Wang Yugui ([email protected])
2022/09/24
On 2022/9/24 14:41, Wang Yugui wrote:
[...]
Quoted 14 lines
Quoted 5 lines
I don't think the config makes much difference, as the main difference is in KASAN and KMEMLEAK, which should not impact the test result.
And are you running just that test, or with the full auto group?
For 6.0-rc6 with btrfs misc-next, I tried to run full auto group. btrfs/042 failed, others(btrfs/001 ~ btrfs/157) are OK, and then I rebooted the test machine.
for 6.0-rc6 without btrfs misc-next, I tested btrfs/042 and btrfs/001 on the same machine.
then I tested 6.0-rc6 without btrfs misc-next on another 2 servers.
Quoted 7 lines
reproduce rate:
server1 3/3
server2 2/3
server3 3/3
total rate: 8/9all 3 servers are in good status(ECC memory status and SSD status).
Considering no one can reproduce your failure so far, I strongly doubt if it's something else.
Please provide the following info:
- Full dmesg since bootup.
Better to reboot, then run btrfs/042, and take the dmesg.
- local.config file of xfstests
- result/btrfs/042* files of xfstests
Only need to run that exact test case, since you have a very high chance to reproduce.
- the full output of "./check btrfs/042"
Quoted 6 lines
Best Regards
Wang Yugui ([email protected])
2022/09/24
Hi,
Quoted 24 lines · Wang Yugui
[...]
Quoted 14 lines
Quoted 5 lines
I don't think the config makes much difference, as the main difference is in KASAN and KMEMLEAK, which should not impact the test result.
And are you running just that test, or with the full auto group?
For 6.0-rc6 with btrfs misc-next, I tried to run full auto group. btrfs/042 failed, others(btrfs/001 ~ btrfs/157) are OK, and then I rebooted the test machine.
for 6.0-rc6 without btrfs misc-next, I tested btrfs/042 and btrfs/001 on the same machine.
then I tested 6.0-rc6 without btrfs misc-next on another 2 servers.Quoted 7 lines
reproduce rate:
server1 3/3
server2 2/3
server3 3/3
total rate: 8/9all 3 servers are in good status(ECC memory status and SSD status).
Quoted 7 lines
Considering no one can reproduce your failure so far, I strongly doubt if it's something else.
Please provide the following info:
- Full dmesg since bootup.
Better to reboot, then run btrfs/042, and take the dmesg.
attachment file(dmesg)
Quoted 1 line
- local.config file of xfstests
attachment file(local.config)
Quoted 3 lines
- result/btrfs/042* files of xfstests
Only need to run that exact test case, since you have a very high chance to reproduce.
attachment file(042.dmesg/042.full)
Quoted 1 line
- the full output of "./check btrfs/042"
[root@T7610 xfstests]# ./check btrfs/042
FSTYP -- btrfs PLATFORM -- Linux/x86_64 T7610 6.0.0-7.1.debug.el7.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Sep 24 10:35:15 CST 2022 MKFS_OPTIONS -- /dev/sdb2
MOUNT_OPTIONS -- /dev/sdb2 /mnt/scratch
btrfs/042 1s ... _check_dmesg: something found in dmesg (see /usr/hpc-bio/xfstests/results//btrfs/042.dmesg)
Ran: btrfs/042
Failures: btrfs/042
Failed 1 of 1 tests
[root@T7610 xfstests]#
Best Regards
Wang Yugui ([email protected])
2022/09/24
Quoted 101 lines · Wang Yugui
Hi,
Quoted 99 lines · Wang Yugui
Quoted 78 lines
Hi,
Quoted 76 lines · Wang Yugui
Quoted 32 lines
Hi,
Quoted 14 lines · Wang Yugui
Quoted 5 lines
Hi,
fstests btrfs/042 triggle 'qgroup reserved space leaked'
kernel source: btrfs misc-next
Which commit HEAD?
As I can not reproduce using a somewhat older misc-next.
The HEAD I'm on is
2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8("btrfs: stop tracking failed reads in the I/O tree").Quoted 7 lines
If it's a regression it can be much easier to pin down.
Quoted 3 lines
kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...And any reproducibility? 16 runs no reproduce.
btrfs source version: misc-next:
bf940dd88f48,
plus some minor local patch(no qgroup related)
kernel: 6.0-rc6reproduce rate:
1) 100%(3/3) when local debug config **1
2) 0% (0/3) when local release configQuoted 11 lines
**1:local debug config, about 100x slow than release config
a) memory debug CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/...
b) lockdep debug
CONFIG_PROVE_LOCKING/...
c) btrfs debugCONFIG_BTRFS_FS_CHECK_INTEGRITY=yCONFIG_BTRFS_FS_RUN_SANITY_TESTS=yCONFIG_BTRFS_DEBUG=yCONFIG_BTRFS_ASSERT=yCONFIG_BTRFS_FS_REF_VERIFY=yI always run with all btrfs features enabled.
So is the lockdep.
KASAN is known to be slow, thus that is only enabled when there is suspision on memory corruption caused by some wild pointer.
Quoted 24 lines
From source:
fs/btrfs/disk-io.c:4668if (btrfs_check_quota_leak(fs_info)) { L4668 WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG)); btrfs_err(fs_info, "qgroup reserved space leaked"); }This problem will triggle fstests btrfs/042 to failure only when
CONFIG_BTRFS_DEBUG=y?maybe related issue:
when lockdep debug is enabled, the following issue become very easy to reproduce too.
https://lore.kernel.org/linux-nfs/[email protected]/
so there maybe some lockdep debug related , but not btrfs related problem in kernel 6.0.more test(remove some minor local patch(no qgroup related)) will be done, and then I will report the result.
Quoted 6 lines
Quoted 3 lines
Better to provide the patches, as I just finished a 16 runs of btrfs/042, no reproduce.
Thus I'm starting to suspect the off-tree patches.
This problem happen on linux 6.0-rc6+ (master
a63f2e7cb110, without btrfs misc-next patch, without local off-tree patch)Same base, still nope.
Quoted 6 lines
so this problem is not related to the patches still in btrfs misc-next.
reproduce rate:
100%(3/3) when local debug config
and the whole config file is attached.I don't think the config makes much difference, as the main difference is in KASAN and KMEMLEAK, which should not impact the test result.
Quoted 9 lines
Quoted 1 line
And are you running just that test, or with the full auto group?
For 6.0-rc6 with btrfs misc-next, I tried to run full auto group. btrfs/042 failed, others(btrfs/001 ~ btrfs/157) are OK, and then I rebooted the test machine.
for 6.0-rc6 without btrfs misc-next, I tested btrfs/042 and btrfs/001 on the same machine.
May I ask if "6.0-rc6 without btrfs misc-next" is 6.0-rc6 *without any other changes*? It looks so weird cause I can't reproduce the issue either.
--
Su
Quoted 13 lines
then I tested 6.0-rc6 without btrfs misc-next on another 2 servers.
reproduce rate:
server1 3/3
server2 2/3
server3 3/3
total rate: 8/9all 3 servers are in good status(ECC memory status and SSD status).
Best Regards
Wang Yugui ([email protected])
2022/09/24
Hi,
Quoted 126 lines · Wang Yugui
Quoted 124 lines
Hi,
Quoted 107 lines · Wang Yugui
Quoted 86 lines
Hi,
Quoted 84 lines · Wang Yugui
Quoted 34 lines
Hi,
Quoted 16 lines · Wang Yugui
Quoted 5 lines
Hi,
fstests btrfs/042 triggle 'qgroup reserved space leaked'
kernel source: btrfs misc-next
Which commit HEAD?
As I can not reproduce using a somewhat older misc-next.
The HEAD I'm on is
2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8("btrfs: stop tracking failed reads in the I/O tree").Quoted 7 lines
If it's a regression it can be much easier to pin down.
Quoted 3 lines
kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...And any reproducibility? 16 runs no reproduce.
btrfs source version: misc-next:
bf940dd88f48,
plus some minor local patch(no qgroup related)
kernel: 6.0-rc6reproduce rate:
1) 100%(3/3) when local debug config **1
2) 0% (0/3) when local release configQuoted 11 lines
**1:local debug config, about 100x slow than release config
a) memory debug CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/...
b) lockdep debug
CONFIG_PROVE_LOCKING/...
c) btrfs debugCONFIG_BTRFS_FS_CHECK_INTEGRITY=yCONFIG_BTRFS_FS_RUN_SANITY_TESTS=yCONFIG_BTRFS_DEBUG=yCONFIG_BTRFS_ASSERT=yCONFIG_BTRFS_FS_REF_VERIFY=yI always run with all btrfs features enabled.
So is the lockdep.
KASAN is known to be slow, thus that is only enabled when there is suspision on memory corruption caused by some wild pointer.
Quoted 28 lines
From source:
fs/btrfs/disk-io.c:4668if (btrfs_check_quota_leak(fs_info)) { L4668 WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG)); btrfs_err(fs_info, "qgroup reserved space leaked"); }This problem will triggle fstests btrfs/042 to failure only when
CONFIG_BTRFS_DEBUG=y?maybe related issue:
when lockdep debug is enabled, the following issue become very easy to
reproduce too.
https://lore.kernel.org/linux-nfs/[email protected]/
so there maybe some lockdep debug related , but not btrfs related
problem in kernel 6.0.more test(remove some minor local patch(no qgroup related)) will be done,
and then I will report the result.Quoted 8 lines
Quoted 4 lines
Better to provide the patches, as I just finished a 16 runs of btrfs/042, no reproduce.
Thus I'm starting to suspect the off-tree patches.
This problem happen on linux 6.0-rc6+ (master
a63f2e7cb110, without
btrfs misc-next patch, without local off-tree patch)Same base, still nope.
Quoted 7 lines
so this problem is not related to the patches still in btrfs misc-next.
reproduce rate:
100%(3/3) when local debug config
and the whole config file is attached.Quoted 7 lines
I don't think the config makes much difference, as the main difference
is in KASAN and KMEMLEAK, which should not impact the test result.And are you running just that test, or with the full auto group?
For 6.0-rc6 with btrfs misc-next, I tried to run full auto group.
btrfs/042 failed, others(btrfs/001 ~ btrfs/157) are OK, and then I
rebooted the test machine.
Quoted 6 lines
Quoted 4 lines
for 6.0-rc6 without btrfs misc-next, I tested btrfs/042 and btrfs/001 on
the same machine.May I ask if "6.0-rc6 without btrfs misc-next" is 6.0-rc6 *without any other changes*? It looks so weird cause I can't reproduce the issue either.
Yes. *without any other changes*
Best Regards
Wang Yugui ([email protected])
2022/09/24