Skip to contentSkip to conversation

fstests btrfs/042 triggle 'qgroup reserved space leaked'

DISCUSSION Wang Yugui 4 participants 12 messages Sep 23, 2022 – Sep 24, 2022 Btrfs file system

lore
Share & tools
lore

Wang Yugui

author
#0 lore raw

Hi,

fstests btrfs/042 triggle 'qgroup reserved space leaked'

kernel source: btrfs misc-next
kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...

dmesg output:

[15788.980873] run fstests btrfs/042 at 2022-09-24 00:40:24
[15803.880347] BTRFS info (device sdb1): using crc32c (crc32c-intel) checksum algorithm
[15803.897721] BTRFS info (device sdb1): using free space tree
[15803.932525] BTRFS info (device sdb1): enabling ssd optimizations
[15818.525145] BTRFS: device fsid b255009c-2a39-49ed-b230-b4e26befd321 devid 1 transid 6 /dev/sdb2 scanned by systemd-udevd (310493)
[15818.791882] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm
[15818.808805] BTRFS info (device sdb2): using free space tree
[15818.837770] BTRFS info (device sdb2): enabling ssd optimizations
[15818.847176] BTRFS info (device sdb2): checking UUID tree
[15818.911997] BTRFS info (device sdb2): qgroup scan completed (inconsistency flag cleared)
[15838.397073] BTRFS warning (device sdb2): qgroup 1/1 has unreleased space, type 0 rsv 12288
[15838.406954] BTRFS warning (device sdb2): qgroup 0/260 has unreleased space, type 0 rsv 4096
[15838.416728] BTRFS warning (device sdb2): qgroup 0/259 has unreleased space, type 0 rsv 4096
[15838.426511] BTRFS warning (device sdb2): qgroup 0/257 has unreleased space, type 0 rsv 4096
[15838.436351] ------------[ cut here ]------------
[15838.442380] WARNING: CPU: 0 PID: 310592 at fs/btrfs/disk-io.c:4668 close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs

fs/btrfs/disk-io.c:4668

    if (btrfs_check_quota_leak(fs_info)) {
L4668        WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));
        btrfs_err(fs_info, "qgroup reserved space leaked");
    }
[15838.452948] Modules linked in: ext4 mbcache jbd2 loop rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill ib_core sunrpc dm_multipath amdgpu iommu_v2 gpu_sched drm_buddy intel_rapl_msr intel_rapl_common btrfs sb_edac snd_hda_codec_realtek x86_pkg_temp_thermal snd_hda_codec_generic radeon intel_powerclamp snd_hda_codec_hdmi ledtrig_audio coretemp blake2b_generic snd_hda_intel xor dcdbas kvm_intel mei_wdt raid6_pq snd_intel_dspcfg i2c_algo_bit iTCO_wdt iTCO_vendor_support zstd_compress dell_smm_hwmon snd_intel_sdw_acpi drm_display_helper kvm snd_hda_codec cec snd_hda_core drm_ttm_helper irqbypass snd_hwdep rapl ttm intel_cstate snd_seq dm_mod snd_seq_device drm_kms_helper intel_uncore snd_pcm pcspkr syscopyarea mei_me snd_timer sysfillrect i2c_i801 snd sysimgblt i2c_smbus lpc_ich mei fb_sys_fops soundcore fuse drm xfs sd_mod t10_pi sr_mod cdrom sg crct10dif_pclmul crc32_pclmul bnx2x crc32c_intel ahci libahci ghash_clmulni_intel mdio libata mpt3sas e10
 00e
[15838.453270]  raid_class scsi_transport_sas wmi i2c_dev ipmi_devintf ipmi_msghandler
[15838.551107] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1
[15838.560649]  pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1
[15838.659255]  pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1
[15838.779406] CPU: 0 PID: 310592 Comm: umount Not tainted 6.0.0-7.0.debug.el7.x86_64 #1
[15838.789287] Hardware name: Dell Inc. Precision T7610/0NK70N, BIOS A18 09/11/2019
[15838.798748] RIP: 0010:close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs
[15838.805988] Code: c7 00 9b 2f c3 e8 1c e7 ff ff 48 8b 3c 24 be 08 00 00 00 e8 2f 78 70 db f0 41 80 4f 10 02 4c 89 ff e8 31 46 f4 ff 84 c0 74 11 <0f> 0b 48 c7 c6 60 9b 2f c3 4c 89 ff e8 b0 8c ff ff 4c 89 ff 49 8d
All code
========
   0:	c7 00 9b 2f c3 e8    	movl   $0xe8c32f9b,(%rax)
   6:	1c e7                	sbb    $0xe7,%al
   8:	ff                   	(bad)  
   9:	ff 48 8b             	decl   -0x75(%rax)
   c:	3c 24                	cmp    $0x24,%al
   e:	be 08 00 00 00       	mov    $0x8,%esi
  13:	e8 2f 78 70 db       	callq  0xffffffffdb707847
  18:	f0 41 80 4f 10 02    	lock orb $0x2,0x10(%r15)
  1e:	4c 89 ff             	mov    %r15,%rdi
  21:	e8 31 46 f4 ff       	callq  0xfffffffffff44657
  26:	84 c0                	test   %al,%al
  28:	74 11                	je     0x3b
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	48 c7 c6 60 9b 2f c3 	mov    $0xffffffffc32f9b60,%rsi
  33:	4c 89 ff             	mov    %r15,%rdi
  36:	e8 b0 8c ff ff       	callq  0xffffffffffff8ceb
  3b:	4c 89 ff             	mov    %r15,%rdi

3e: 49 rex.WB
3f: 8d .byte 0x8d

Code starting with the faulting instruction
===========================================
0: 0f 0b ud2

   2:	48 c7 c6 60 9b 2f c3 	mov    $0xffffffffc32f9b60,%rsi
   9:	4c 89 ff             	mov    %r15,%rdi
   c:	e8 b0 8c ff ff       	callq  0xffffffffffff8cc1
  11:	4c 89 ff             	mov    %r15,%rdi
  14:	49                   	rex.WB
  15:	8d                   	.byte 0x8d
[15838.829023] RSP: 0018:ffff88810b7bfb98 EFLAGS: 00010202
[15838.836422] RAX: 0000000000000001 RBX: ffff88828ee54d58 RCX: 0000000000000000
[15838.845734] RDX: 1ffff11064bd1ad3 RSI: 0000000000000008 RDI: ffff888325e8d6a0
[15838.855015] RBP: ffff88828ee54fd0 R08: ffffed13f39c7a21 R09: 0000000000000000
[15838.864347] R10: ffffed1418ee9840 R11: ffff88a0c774c200 R12: ffff88824b4487b0
[15838.873690] R13: ffff88828ee55130 R14: ffff88819ea76ec0 R15: ffff88828ee54000
[15838.883005] FS:  00007f41e8aaa500(0000) GS:ffff889f9ce00000(0000) knlGS:0000000000000000
[15838.893294] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15838.901224] CR2: 00007fa141b622d8 CR3: 00000001e2a34003 CR4: 00000000001706f0
[15838.910553] Call Trace:
[15838.915219]  <TASK>
[15838.919527] ? fsnotify_destroy_marks (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/mark.c:839) 
[15838.926367] ? btrfs_cleanup_one_transaction.cold.75 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/perf_event.h:1189 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:720) btrfs
[15838.935255] ? fsnotify_sb_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/atomic64_64.h:22 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-long.h:29 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-instrumented.h:1266 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:95) 
[15838.941822] ? __fsnotify_vfsmount_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:91) 
[15838.948905] ? evict_inodes (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713) 
[15838.954918] ? dispose_list (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713) 
[15838.960947] ? btrfs_sync_fs (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:1570) btrfs
[15838.967745] generic_shutdown_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:491) 
[15838.974445] kill_anon_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1072 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1086) 
[15838.980345] btrfs_kill_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:2551) btrfs
[15838.987130] deactivate_locked_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:332) 
[15838.993736] cleanup_mnt (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/namespace.c:1187) 
[15838.999417] task_work_run (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/task_work.c:177 (discriminator 1)) 
[15839.005185] exit_to_user_mode_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/resume_user_mode.h:49 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:169 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:201) 
[15839.012112] syscall_exit_to_user_mode (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:128 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:296) 
[15839.018875] do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87) 
[15839.024600] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271) 
[15839.031881] ? do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87) 
[15839.037753] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271) 
[15839.045010] entry_SYSCALL_64_after_hwframe (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/entry_64.S:120) 
[15839.052194] RIP: 0033:0x7f41e8953a6b
[15839.057913] Code: 0f 1e fa 48 89 fe 31 ff e9 72 08 00 00 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 89 63 0a 00 f7 d8
All code
========
   0:	0f 1e fa             	nop    %edx
   3:	48 89 fe             	mov    %rdi,%rsi
   6:	31 ff                	xor    %edi,%edi
   8:	e9 72 08 00 00       	jmpq   0x87f
   d:	66 90                	xchg   %ax,%ax
   f:	f3 0f 1e fa          	endbr64 
  13:	31 f6                	xor    %esi,%esi
  15:	e9 05 00 00 00       	jmpq   0x1f
  1a:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
  1f:	f3 0f 1e fa          	endbr64 
  23:	b8 a6 00 00 00       	mov    $0xa6,%eax
  28:	0f 05                	syscall 
  2a:*	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax		<-- trapping instruction
  30:	77 05                	ja     0x37
  32:	c3                   	retq   
  33:	0f 1f 40 00          	nopl   0x0(%rax)

37: 48 8b 15 89 63 0a 00 mov 0xa6389(%rip),%rdx # 0xa63c7
3e: f7 d8 neg %eax

Code starting with the faulting instruction
===========================================

   0:	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax
   6:	77 05                	ja     0xd
   8:	c3                   	retq   
   9:	0f 1f 40 00          	nopl   0x0(%rax)
   d:	48 8b 15 89 63 0a 00 	mov    0xa6389(%rip),%rdx        # 0xa639d
  14:	f7 d8                	neg    %eax
[15839.081093] RSP: 002b:00007fff313c23d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[15839.090894] RAX: 0000000000000000 RBX: 000055d74a755540 RCX: 00007f41e8953a6b
[15839.100242] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000055d74a75a550
[15839.109546] RBP: 000055d74a755310 R08: 0000000000000000 R09: 000055d74a754010
[15839.118783] R10: 00007f41e89faaa0 R11: 0000000000000246 R12: 0000000000000000
[15839.128016] R13: 000055d74a75a550 R14: 000055d74a755420 R15: 000055d74a755310
[15839.137246]  </TASK>
[15839.141444] irq event stamp: 36915
[15839.146842] hardirqs last enabled at (36929): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:45 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:80 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:138 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:264 (discriminator 1)) 
[15839.157472] hardirqs last disabled at (36942): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:262 (discriminator 1)) 
[15839.168059] softirqs last enabled at (34740): __do_softirq (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/preempt.h:27 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:415 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:600) 
[15839.178441] softirqs last disabled at (34701): irq_exit_rcu (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:445 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:650 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:662) 
[15839.188788] ---[ end trace 0000000000000000 ]---
[15839.195355] BTRFS error (device sdb2): qgroup reserved space leaked
[15839.544913] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm
[15839.563333] BTRFS info (device sdb2): using free space tree
[15839.597640] BTRFS info (device sdb2): enabling ssd optimizations

Best Regards
Wang Yugui ([email protected])
2022/09/24

#1 lore raw
Quoted 5 lines · Wang Yugui

Hi,

fstests btrfs/042 triggle 'qgroup reserved space leaked'

kernel source: btrfs misc-next

Which commit HEAD?

As I can not reproduce using a somewhat older misc-next.

The HEAD I'm on is 2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8 ("btrfs: stop tracking failed reads in the I/O tree").

If it's a regression it can be much easier to pin down.

Quoted 3 lines

kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...

And any reproducibility? 16 runs no reproduce.

Thanks,
Qu

Quoted 153 lines

dmesg output:

[15788.980873] run fstests btrfs/042 at 2022-09-24 00:40:24
[15803.880347] BTRFS info (device sdb1): using crc32c (crc32c-intel) checksum algorithm
[15803.897721] BTRFS info (device sdb1): using free space tree
[15803.932525] BTRFS info (device sdb1): enabling ssd optimizations
[15818.525145] BTRFS: device fsid b255009c-2a39-49ed-b230-b4e26befd321 devid 1 transid 6 /dev/sdb2 scanned by systemd-udevd (310493)
[15818.791882] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm
[15818.808805] BTRFS info (device sdb2): using free space tree
[15818.837770] BTRFS info (device sdb2): enabling ssd optimizations
[15818.847176] BTRFS info (device sdb2): checking UUID tree
[15818.911997] BTRFS info (device sdb2): qgroup scan completed (inconsistency flag cleared)
[15838.397073] BTRFS warning (device sdb2): qgroup 1/1 has unreleased space, type 0 rsv 12288
[15838.406954] BTRFS warning (device sdb2): qgroup 0/260 has unreleased space, type 0 rsv 4096
[15838.416728] BTRFS warning (device sdb2): qgroup 0/259 has unreleased space, type 0 rsv 4096
[15838.426511] BTRFS warning (device sdb2): qgroup 0/257 has unreleased space, type 0 rsv 4096
[15838.436351] ------------[ cut here ]------------
[15838.442380] WARNING: CPU: 0 PID: 310592 at fs/btrfs/disk-io.c:4668 close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs

fs/btrfs/disk-io.c:4668

     if (btrfs_check_quota_leak(fs_info)) {
L4668        WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));
         btrfs_err(fs_info, "qgroup reserved space leaked");
     }
[15838.452948] Modules linked in: ext4 mbcache jbd2 loop rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill ib_core sunrpc dm_multipath amdgpu iommu_v2 gpu_sched drm_buddy intel_rapl_msr intel_rapl_common btrfs sb_edac snd_hda_codec_realtek x86_pkg_temp_thermal snd_hda_codec_generic radeon intel_powerclamp snd_hda_codec_hdmi ledtrig_audio coretemp blake2b_generic snd_hda_intel xor dcdbas kvm_intel mei_wdt raid6_pq snd_intel_dspcfg i2c_algo_bit iTCO_wdt iTCO_vendor_support zstd_compress dell_smm_hwmon snd_intel_sdw_acpi drm_display_helper kvm snd_hda_codec cec snd_hda_core drm_ttm_helper irqbypass snd_hwdep rapl ttm intel_cstate snd_seq dm_mod snd_seq_device drm_kms_helper intel_uncore snd_pcm pcspkr syscopyarea mei_me snd_timer sysfillrect i2c_i801 snd sysimgblt i2c_smbus lpc_ich mei fb_sys_fops soundcore fuse drm xfs sd_mod t10_pi sr_mod cdrom sg crct10dif_pclmul crc32_pclmul bnx2x crc32c_intel ahci libahci ghash_clmulni_intel mdio libata mpt3sas e10
  00e
[15838.453270]  raid_class scsi_transport_sas wmi i2c_dev ipmi_devintf ipmi_msghandler
[15838.551107] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1
[15838.560649]  pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1
[15838.659255]  pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1
[15838.779406] CPU: 0 PID: 310592 Comm: umount Not tainted 6.0.0-7.0.debug.el7.x86_64 #1
[15838.789287] Hardware name: Dell Inc. Precision T7610/0NK70N, BIOS A18 09/11/2019
[15838.798748] RIP: 0010:close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs
[15838.805988] Code: c7 00 9b 2f c3 e8 1c e7 ff ff 48 8b 3c 24 be 08 00 00 00 e8 2f 78 70 db f0 41 80 4f 10 02 4c 89 ff e8 31 46 f4 ff 84 c0 74 11 <0f> 0b 48 c7 c6 60 9b 2f c3 4c 89 ff e8 b0 8c ff ff 4c 89 ff 49 8d
All code
========
    0:	c7 00 9b 2f c3 e8    	movl   $0xe8c32f9b,(%rax)
    6:	1c e7                	sbb    $0xe7,%al
    8:	ff                   	(bad)
    9:	ff 48 8b             	decl   -0x75(%rax)
    c:	3c 24                	cmp    $0x24,%al
    e:	be 08 00 00 00       	mov    $0x8,%esi
   13:	e8 2f 78 70 db       	callq  0xffffffffdb707847
   18:	f0 41 80 4f 10 02    	lock orb $0x2,0x10(%r15)
   1e:	4c 89 ff             	mov    %r15,%rdi
   21:	e8 31 46 f4 ff       	callq  0xfffffffffff44657
   26:	84 c0                	test   %al,%al
   28:	74 11                	je     0x3b
   2a:*	0f 0b                	ud2    		<-- trapping instruction
   2c:	48 c7 c6 60 9b 2f c3 	mov    $0xffffffffc32f9b60,%rsi
   33:	4c 89 ff             	mov    %r15,%rdi
   36:	e8 b0 8c ff ff       	callq  0xffffffffffff8ceb
   3b:	4c 89 ff             	mov    %r15,%rdi

3e: 49 rex.WB
3f: 8d .byte 0x8d

Code starting with the faulting instruction
===========================================

    0:	0f 0b                	ud2
    2:	48 c7 c6 60 9b 2f c3 	mov    $0xffffffffc32f9b60,%rsi
    9:	4c 89 ff             	mov    %r15,%rdi
    c:	e8 b0 8c ff ff       	callq  0xffffffffffff8cc1
   11:	4c 89 ff             	mov    %r15,%rdi
   14:	49                   	rex.WB
   15:	8d                   	.byte 0x8d
[15838.829023] RSP: 0018:ffff88810b7bfb98 EFLAGS: 00010202
[15838.836422] RAX: 0000000000000001 RBX: ffff88828ee54d58 RCX: 0000000000000000
[15838.845734] RDX: 1ffff11064bd1ad3 RSI: 0000000000000008 RDI: ffff888325e8d6a0
[15838.855015] RBP: ffff88828ee54fd0 R08: ffffed13f39c7a21 R09: 0000000000000000
[15838.864347] R10: ffffed1418ee9840 R11: ffff88a0c774c200 R12: ffff88824b4487b0
[15838.873690] R13: ffff88828ee55130 R14: ffff88819ea76ec0 R15: ffff88828ee54000
[15838.883005] FS:  00007f41e8aaa500(0000) GS:ffff889f9ce00000(0000) knlGS:0000000000000000
[15838.893294] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15838.901224] CR2: 00007fa141b622d8 CR3: 00000001e2a34003 CR4: 00000000001706f0
[15838.910553] Call Trace:
[15838.915219]  <TASK>
[15838.919527] ? fsnotify_destroy_marks (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/mark.c:839)
[15838.926367] ? btrfs_cleanup_one_transaction.cold.75 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/perf_event.h:1189 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:720) btrfs
[15838.935255] ? fsnotify_sb_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/atomic64_64.h:22 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-long.h:29 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-instrumented.h:1266 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:95)
[15838.941822] ? __fsnotify_vfsmount_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:91)
[15838.948905] ? evict_inodes (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713)
[15838.954918] ? dispose_list (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713)
[15838.960947] ? btrfs_sync_fs (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:1570) btrfs
[15838.967745] generic_shutdown_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:491)
[15838.974445] kill_anon_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1072 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1086)
[15838.980345] btrfs_kill_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:2551) btrfs
[15838.987130] deactivate_locked_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:332)
[15838.993736] cleanup_mnt (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/namespace.c:1187)
[15838.999417] task_work_run (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/task_work.c:177 (discriminator 1))
[15839.005185] exit_to_user_mode_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/resume_user_mode.h:49 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:169 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:201)
[15839.012112] syscall_exit_to_user_mode (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:128 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:296)
[15839.018875] do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87)
[15839.024600] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271)
[15839.031881] ? do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87)
[15839.037753] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271)
[15839.045010] entry_SYSCALL_64_after_hwframe (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/entry_64.S:120)
[15839.052194] RIP: 0033:0x7f41e8953a6b
[15839.057913] Code: 0f 1e fa 48 89 fe 31 ff e9 72 08 00 00 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 89 63 0a 00 f7 d8
All code
========
    0:	0f 1e fa             	nop    %edx
    3:	48 89 fe             	mov    %rdi,%rsi
    6:	31 ff                	xor    %edi,%edi
    8:	e9 72 08 00 00       	jmpq   0x87f
    d:	66 90                	xchg   %ax,%ax
    f:	f3 0f 1e fa          	endbr64
   13:	31 f6                	xor    %esi,%esi
   15:	e9 05 00 00 00       	jmpq   0x1f
   1a:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
   1f:	f3 0f 1e fa          	endbr64
   23:	b8 a6 00 00 00       	mov    $0xa6,%eax
   28:	0f 05                	syscall
   2a:*	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax		<-- trapping instruction
   30:	77 05                	ja     0x37
   32:	c3                   	retq
   33:	0f 1f 40 00          	nopl   0x0(%rax)

37: 48 8b 15 89 63 0a 00 mov 0xa6389(%rip),%rdx # 0xa63c7
3e: f7 d8 neg %eax

Code starting with the faulting instruction
===========================================

    0:	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax
    6:	77 05                	ja     0xd
    8:	c3                   	retq
    9:	0f 1f 40 00          	nopl   0x0(%rax)
    d:	48 8b 15 89 63 0a 00 	mov    0xa6389(%rip),%rdx        # 0xa639d
   14:	f7 d8                	neg    %eax
[15839.081093] RSP: 002b:00007fff313c23d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[15839.090894] RAX: 0000000000000000 RBX: 000055d74a755540 RCX: 00007f41e8953a6b
[15839.100242] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000055d74a75a550
[15839.109546] RBP: 000055d74a755310 R08: 0000000000000000 R09: 000055d74a754010
[15839.118783] R10: 00007f41e89faaa0 R11: 0000000000000246 R12: 0000000000000000
[15839.128016] R13: 000055d74a75a550 R14: 000055d74a755420 R15: 000055d74a755310
[15839.137246]  </TASK>
[15839.141444] irq event stamp: 36915
[15839.146842] hardirqs last enabled at (36929): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:45 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:80 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:138 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:264 (discriminator 1))
[15839.157472] hardirqs last disabled at (36942): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:262 (discriminator 1))
[15839.168059] softirqs last enabled at (34740): __do_softirq (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/preempt.h:27 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:415 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:600)
[15839.178441] softirqs last disabled at (34701): irq_exit_rcu (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:445 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:650 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:662)
[15839.188788] ---[ end trace 0000000000000000 ]---
[15839.195355] BTRFS error (device sdb2): qgroup reserved space leaked
[15839.544913] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm
[15839.563333] BTRFS info (device sdb2): using free space tree
[15839.597640] BTRFS info (device sdb2): enabling ssd optimizations

Best Regards
Wang Yugui ([email protected])
2022/09/24

#2 lore raw
Quoted 16 lines · Qu Wenruo
Quoted 5 lines · Wang Yugui

Hi,

fstests btrfs/042 triggle 'qgroup reserved space leaked'

kernel source: btrfs misc-next

Which commit HEAD?

As I can not reproduce using a somewhat older misc-next.

The HEAD I'm on is 2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8 ("btrfs:
stop tracking failed reads in the I/O tree").

Quoted 7 lines

If it's a regression it can be much easier to pin down.

Quoted 3 lines

kernel config:
    memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...

And any reproducibility? 16 runs no reproduce.

Furthermore, I checked the result from Josef's nightly testing farm:

http://toxicpanda.com/

No such report on btrfs/042.

Any reproducibility or extra patches on your branch?

Thanks,
Qu

Quoted 264 lines

Thanks,
Qu

Quoted 261 lines

dmesg output:

[15788.980873] run fstests btrfs/042 at 2022-09-24 00:40:24
[15803.880347] BTRFS info (device sdb1): using crc32c (crc32c-intel)
checksum algorithm
[15803.897721] BTRFS info (device sdb1): using free space tree
[15803.932525] BTRFS info (device sdb1): enabling ssd optimizations
[15818.525145] BTRFS: device fsid b255009c-2a39-49ed-b230-b4e26befd321
devid 1 transid 6 /dev/sdb2 scanned by systemd-udevd (310493)
[15818.791882] BTRFS info (device sdb2): using crc32c (crc32c-intel)
checksum algorithm
[15818.808805] BTRFS info (device sdb2): using free space tree
[15818.837770] BTRFS info (device sdb2): enabling ssd optimizations
[15818.847176] BTRFS info (device sdb2): checking UUID tree
[15818.911997] BTRFS info (device sdb2): qgroup scan completed
(inconsistency flag cleared)
[15838.397073] BTRFS warning (device sdb2): qgroup 1/1 has unreleased
space, type 0 rsv 12288
[15838.406954] BTRFS warning (device sdb2): qgroup 0/260 has
unreleased space, type 0 rsv 4096
[15838.416728] BTRFS warning (device sdb2): qgroup 0/259 has
unreleased space, type 0 rsv 4096
[15838.426511] BTRFS warning (device sdb2): qgroup 0/257 has
unreleased space, type 0 rsv 4096
[15838.436351] ------------[ cut here ]------------
[15838.442380] WARNING: CPU: 0 PID: 310592 at fs/btrfs/disk-io.c:4668

close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs

fs/btrfs/disk-io.c:4668
     if (btrfs_check_quota_leak(fs_info)) {
L4668        WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG)); btrfs_err(fs_info, "qgroup reserved space leaked"); }

[15838.452948] Modules linked in: ext4 mbcache jbd2 loop

rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill ib_core sunrpc dm_multipath amdgpu iommu_v2 gpu_sched drm_buddy intel_rapl_msr intel_rapl_common btrfs sb_edac snd_hda_codec_realtek x86_pkg_temp_thermal snd_hda_codec_generic radeon intel_powerclamp snd_hda_codec_hdmi ledtrig_audio coretemp blake2b_generic snd_hda_intel xor dcdbas kvm_intel mei_wdt raid6_pq snd_intel_dspcfg i2c_algo_bit iTCO_wdt iTCO_vendor_support zstd_compress dell_smm_hwmon snd_intel_sdw_acpi drm_display_helper kvm snd_hda_codec cec snd_hda_core drm_ttm_helper irqbypass snd_hwdep rapl ttm intel_cstate snd_seq dm_mod snd_seq_device drm_kms_helper intel_uncore snd_pcm pcspkr syscopyarea mei_me snd_timer sysfillrect i2c_i801 snd sysimgblt i2c_smbus lpc_ich mei fb_sys_fops soundcore fuse drm xfs sd_mod t10_pi sr_mod cdrom sg crct10dif_pclmul crc32_pclmul bnx2x crc32c_intel ahci libahci ghash_clmulni_intel mdio libata mpt3sas e10
  00e

[15838.453270]  raid_class scsi_transport_sas wmi i2c_dev ipmi_devintf
ipmi_msghandler
[15838.551107] Unloaded tainted modules: acpi_cpufreq():1

acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1

[15838.560649]  pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1

pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1

[15838.659255]  pcc_cpufreq():1 acpi_cpufreq():1 fjes():1
pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1
acpi_cpufreq():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1
[15838.779406] CPU: 0 PID: 310592 Comm: umount Not tainted
6.0.0-7.0.debug.el7.x86_64 #1
[15838.789287] Hardware name: Dell Inc. Precision T7610/0NK70N, BIOS
A18 09/11/2019
[15838.798748] RIP: 0010:close_ctree
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs
[15838.805988] Code: c7 00 9b 2f c3 e8 1c e7 ff ff 48 8b 3c 24 be 08

00 00 00 e8 2f 78 70 db f0 41 80 4f 10 02 4c 89 ff e8 31 46 f4 ff 84 c0 74 11 <0f> 0b 48 c7 c6 60 9b 2f c3 4c 89 ff e8 b0 8c ff ff 4c 89 ff 49 8d
All code
========
    0:    c7 00 9b 2f c3 e8        movl   $0xe8c32f9b,(%rax) 6:    1c e7                    sbb    $0xe7,%al
    8:    ff                       (bad)
    9:    ff 48 8b                 decl   -0x75(%rax)
    c:    3c 24                    cmp    $0x24,%al
    e:    be 08 00 00 00           mov    $0x8,%esi
   13:    e8 2f 78 70 db           callq  0xffffffffdb707847 18:    f0 41 80 4f 10 02        lock orb $0x2,0x10(%r15) 1e:    4c 89 ff                 mov    %r15,%rdi
   21:    e8 31 46 f4 ff           callq  0xfffffffffff44657 26:    84 c0                    test   %al,%al
   28:    74 11                    je     0x3b
   2a:*    0f 0b                    ud2            <-- trapping instruction
   2c:    48 c7 c6 60 9b 2f c3     mov    $0xffffffffc32f9b60,%rsi 33:    4c 89 ff                 mov    %r15,%rdi
   36:    e8 b0 8c ff ff           callq  0xffffffffffff8ceb 3b:    4c 89 ff                 mov    %r15,%rdi
   3e:    49                       rex.WB
   3f:    8d                       .byte 0x8d

Code starting with the faulting instruction
===========================================
    0:    0f 0b                    ud2
    2:    48 c7 c6 60 9b 2f c3     mov    $0xffffffffc32f9b60,%rsi 9:    4c 89 ff                 mov    %r15,%rdi
    c:    e8 b0 8c ff ff           callq  0xffffffffffff8cc1 11:    4c 89 ff                 mov    %r15,%rdi
   14:    49                       rex.WB
   15:    8d                       .byte 0x8d

[15838.829023] RSP: 0018:ffff88810b7bfb98 EFLAGS: 00010202
[15838.836422] RAX: 0000000000000001 RBX: ffff88828ee54d58 RCX:
0000000000000000
[15838.845734] RDX: 1ffff11064bd1ad3 RSI: 0000000000000008 RDI:
ffff888325e8d6a0
[15838.855015] RBP: ffff88828ee54fd0 R08: ffffed13f39c7a21 R09:
0000000000000000
[15838.864347] R10: ffffed1418ee9840 R11: ffff88a0c774c200 R12:
ffff88824b4487b0
[15838.873690] R13: ffff88828ee55130 R14: ffff88819ea76ec0 R15:
ffff88828ee54000
[15838.883005] FS:  00007f41e8aaa500(0000) GS:ffff889f9ce00000(0000)
knlGS:0000000000000000
[15838.893294] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15838.901224] CR2: 00007fa141b622d8 CR3: 00000001e2a34003 CR4:
00000000001706f0
[15838.910553] Call Trace:
[15838.915219]  <TASK>
[15838.919527] ? fsnotify_destroy_marks
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/mark.c:839)
[15838.926367] ? btrfs_cleanup_one_transaction.cold.75
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/perf_event.h:1189 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:720) btrfs
[15838.935255] ? fsnotify_sb_delete
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/atomic64_64.h:22 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-long.h:29 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-instrumented.h:1266 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:95)
[15838.941822] ? __fsnotify_vfsmount_delete
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:91)
[15838.948905] ? evict_inodes
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713)
[15838.954918] ? dispose_list
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713)
[15838.960947] ? btrfs_sync_fs
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:1570) btrfs
[15838.967745] generic_shutdown_super
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:491)
[15838.974445] kill_anon_super
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1072 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1086)
[15838.980345] btrfs_kill_super
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:2551) btrfs
[15838.987130] deactivate_locked_super
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:332)
[15838.993736] cleanup_mnt
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/namespace.c:1187)
[15838.999417] task_work_run
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/task_work.c:177 (discriminator 1))
[15839.005185] exit_to_user_mode_prepare
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/resume_user_mode.h:49 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:169 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:201)
[15839.012112] syscall_exit_to_user_mode
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:128 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:296)
[15839.018875] do_syscall_64
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87)
[15839.024600] ? lockdep_hardirqs_on_prepare
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271)
[15839.031881] ? do_syscall_64
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87)
[15839.037753] ? lockdep_hardirqs_on_prepare
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271)
[15839.045010] entry_SYSCALL_64_after_hwframe
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/entry_64.S:120)
[15839.052194] RIP: 0033:0x7f41e8953a6b
[15839.057913] Code: 0f 1e fa 48 89 fe 31 ff e9 72 08 00 00 66 90 f3

0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 89 63 0a 00 f7 d8
All code
========
    0:    0f 1e fa                 nop    %edx
    3:    48 89 fe                 mov    %rdi,%rsi
    6:    31 ff                    xor    %edi,%edi
    8:    e9 72 08 00 00           jmpq   0x87f
    d:    66 90                    xchg   %ax,%ax
    f:    f3 0f 1e fa              endbr64
   13:    31 f6                    xor    %esi,%esi
   15:    e9 05 00 00 00           jmpq   0x1f
   1a:    0f 1f 44 00 00           nopl   0x0(%rax,%rax,1) 1f:    f3 0f 1e fa              endbr64
   23:    b8 a6 00 00 00           mov    $0xa6,%eax
   28:    0f 05                    syscall
   2a:*    48 3d 00 f0 ff ff        cmp $0xfffffffffffff000,%rax        <-- trapping instruction 30:    77 05                    ja     0x37
   32:    c3                       retq
   33:    0f 1f 40 00              nopl   0x0(%rax)
   37:    48 8b 15 89 63 0a 00     mov    0xa6389(%rip),%rdx        # 0xa63c7
   3e:    f7 d8                    neg    %eax

Code starting with the faulting instruction
===========================================
    0:    48 3d 00 f0 ff ff        cmp    $0xfffffffffffff000,%rax 6:    77 05                    ja     0xd
    8:    c3                       retq
    9:    0f 1f 40 00              nopl   0x0(%rax)
    d:    48 8b 15 89 63 0a 00     mov    0xa6389(%rip),%rdx        # 0xa639d
   14:    f7 d8                    neg    %eax

[15839.081093] RSP: 002b:00007fff313c23d8 EFLAGS: 00000246 ORIG_RAX:
00000000000000a6
[15839.090894] RAX: 0000000000000000 RBX: 000055d74a755540 RCX:
00007f41e8953a6b
[15839.100242] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
000055d74a75a550
[15839.109546] RBP: 000055d74a755310 R08: 0000000000000000 R09:
000055d74a754010
[15839.118783] R10: 00007f41e89faaa0 R11: 0000000000000246 R12:
0000000000000000
[15839.128016] R13: 000055d74a75a550 R14: 000055d74a755420 R15:
000055d74a755310
[15839.137246]  </TASK>
[15839.141444] irq event stamp: 36915
[15839.146842] hardirqs last enabled at (36929): __up_console_sem
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:45 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:80 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:138 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:264 (discriminator 1))
[15839.157472] hardirqs last disabled at (36942): __up_console_sem
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:262 (discriminator 1))
[15839.168059] softirqs last enabled at (34740): __do_softirq
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/preempt.h:27 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:415 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:600)
[15839.178441] softirqs last disabled at (34701): irq_exit_rcu
(/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:445 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:650 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:662)
[15839.188788] ---[ end trace 0000000000000000 ]---
[15839.195355] BTRFS error (device sdb2): qgroup reserved space leaked
[15839.544913] BTRFS info (device sdb2): using crc32c (crc32c-intel)
checksum algorithm
[15839.563333] BTRFS info (device sdb2): using free space tree
[15839.597640] BTRFS info (device sdb2): enabling ssd optimizations

Best Regards
Wang Yugui ([email protected])
2022/09/24

author
#3 lore raw
In reply to Qu Wenruo

Hi,

Quoted 14 lines · Wang Yugui
Quoted 5 lines

Hi,

fstests btrfs/042 triggle 'qgroup reserved space leaked'

kernel source: btrfs misc-next

Which commit HEAD?

As I can not reproduce using a somewhat older misc-next.

The HEAD I'm on is 2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8 ("btrfs: stop tracking failed reads in the I/O tree").

Quoted 7 lines

If it's a regression it can be much easier to pin down.

Quoted 3 lines

kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...

And any reproducibility? 16 runs no reproduce.

btrfs source version: misc-next: bf940dd88f48,
plus some minor local patch(no qgroup related)
kernel: 6.0-rc6

reproduce rate:
1) 100%(3/3) when local debug config **1
2) 0% (0/3) when local release config

**1:local debug config, about 100x slow than release config
a) memory debug CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/...
b) lockdep debug
CONFIG_PROVE_LOCKING/...
c) btrfs debug
CONFIG_BTRFS_FS_CHECK_INTEGRITY=y CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y CONFIG_BTRFS_DEBUG=y
CONFIG_BTRFS_ASSERT=y
CONFIG_BTRFS_FS_REF_VERIFY=y

From source:
fs/btrfs/disk-io.c:4668

    if (btrfs_check_quota_leak(fs_info)) {
L4668        WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));
        btrfs_err(fs_info, "qgroup reserved space leaked");
    }

This problem will triggle fstests btrfs/042 to failure only when CONFIG_BTRFS_DEBUG=y ?

maybe related issue:
when lockdep debug is enabled, the following issue become very easy to reproduce too.
https://lore.kernel.org/linux-nfs/[email protected]/
so there maybe some lockdep debug related , but not btrfs related problem in kernel 6.0.

more test(remove some minor local patch(no qgroup related)) will be done, and then I will report the result.

Best Regards
Wang Yugui ([email protected])
2022/09/24

Quoted 156 lines

Thanks,
Qu

Quoted 153 lines

dmesg output:

[15788.980873] run fstests btrfs/042 at 2022-09-24 00:40:24
[15803.880347] BTRFS info (device sdb1): using crc32c (crc32c-intel) checksum algorithm
[15803.897721] BTRFS info (device sdb1): using free space tree
[15803.932525] BTRFS info (device sdb1): enabling ssd optimizations
[15818.525145] BTRFS: device fsid b255009c-2a39-49ed-b230-b4e26befd321 devid 1 transid 6 /dev/sdb2 scanned by systemd-udevd (310493)
[15818.791882] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm
[15818.808805] BTRFS info (device sdb2): using free space tree
[15818.837770] BTRFS info (device sdb2): enabling ssd optimizations
[15818.847176] BTRFS info (device sdb2): checking UUID tree
[15818.911997] BTRFS info (device sdb2): qgroup scan completed (inconsistency flag cleared)
[15838.397073] BTRFS warning (device sdb2): qgroup 1/1 has unreleased space, type 0 rsv 12288
[15838.406954] BTRFS warning (device sdb2): qgroup 0/260 has unreleased space, type 0 rsv 4096
[15838.416728] BTRFS warning (device sdb2): qgroup 0/259 has unreleased space, type 0 rsv 4096
[15838.426511] BTRFS warning (device sdb2): qgroup 0/257 has unreleased space, type 0 rsv 4096
[15838.436351] ------------[ cut here ]------------
[15838.442380] WARNING: CPU: 0 PID: 310592 at fs/btrfs/disk-io.c:4668 close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs

fs/btrfs/disk-io.c:4668

     if (btrfs_check_quota_leak(fs_info)) {
L4668        WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));
         btrfs_err(fs_info, "qgroup reserved space leaked");
     }
[15838.452948] Modules linked in: ext4 mbcache jbd2 loop rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill ib_core sunrpc dm_multipath amdgpu iommu_v2 gpu_sched drm_buddy intel_rapl_msr intel_rapl_common btrfs sb_edac snd_hda_codec_realtek x86_pkg_temp_thermal snd_hda_codec_generic radeon intel_powerclamp snd_hda_codec_hdmi ledtrig_audio coretemp blake2b_generic snd_hda_intel xor dcdbas kvm_intel mei_wdt raid6_pq snd_intel_dspcfg i2c_algo_bit iTCO_wdt iTCO_vendor_support zstd_compress dell_smm_hwmon snd_intel_sdw_acpi drm_display_helper kvm snd_hda_codec cec snd_hda_core drm_ttm_helper irqbypass snd_hwdep rapl ttm intel_cstate snd_seq dm_mod snd_seq_device drm_kms_helper intel_uncore snd_pcm pcspkr syscopyarea mei_me snd_timer sysfillrect i2c_i801 snd sysimgblt i2c_smbus lpc_ich mei fb_sys_fops soundcore fuse drm xfs sd_mod t10_pi sr_mod cdrom sg crct10dif_pclmul crc32_pclmul bnx2x crc32c_intel ahci libahci ghash_clmulni_intel mdio libata mpt3sas e10
  00e
[15838.453270]  raid_class scsi_transport_sas wmi i2c_dev ipmi_devintf ipmi_msghandler
[15838.551107] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1
[15838.560649]  pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1
[15838.659255]  pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1
[15838.779406] CPU: 0 PID: 310592 Comm: umount Not tainted 6.0.0-7.0.debug.el7.x86_64 #1
[15838.789287] Hardware name: Dell Inc. Precision T7610/0NK70N, BIOS A18 09/11/2019
[15838.798748] RIP: 0010:close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs
[15838.805988] Code: c7 00 9b 2f c3 e8 1c e7 ff ff 48 8b 3c 24 be 08 00 00 00 e8 2f 78 70 db f0 41 80 4f 10 02 4c 89 ff e8 31 46 f4 ff 84 c0 74 11 <0f> 0b 48 c7 c6 60 9b 2f c3 4c 89 ff e8 b0 8c ff ff 4c 89 ff 49 8d
All code
========
    0:	c7 00 9b 2f c3 e8    	movl   $0xe8c32f9b,(%rax)
    6:	1c e7                	sbb    $0xe7,%al
    8:	ff                   	(bad)
    9:	ff 48 8b             	decl   -0x75(%rax)
    c:	3c 24                	cmp    $0x24,%al
    e:	be 08 00 00 00       	mov    $0x8,%esi
   13:	e8 2f 78 70 db       	callq  0xffffffffdb707847
   18:	f0 41 80 4f 10 02    	lock orb $0x2,0x10(%r15)
   1e:	4c 89 ff             	mov    %r15,%rdi
   21:	e8 31 46 f4 ff       	callq  0xfffffffffff44657
   26:	84 c0                	test   %al,%al
   28:	74 11                	je     0x3b
   2a:*	0f 0b                	ud2    		<-- trapping instruction
   2c:	48 c7 c6 60 9b 2f c3 	mov    $0xffffffffc32f9b60,%rsi
   33:	4c 89 ff             	mov    %r15,%rdi
   36:	e8 b0 8c ff ff       	callq  0xffffffffffff8ceb
   3b:	4c 89 ff             	mov    %r15,%rdi

3e: 49 rex.WB
3f: 8d .byte 0x8d

Code starting with the faulting instruction
===========================================

    0:	0f 0b                	ud2
    2:	48 c7 c6 60 9b 2f c3 	mov    $0xffffffffc32f9b60,%rsi
    9:	4c 89 ff             	mov    %r15,%rdi
    c:	e8 b0 8c ff ff       	callq  0xffffffffffff8cc1
   11:	4c 89 ff             	mov    %r15,%rdi
   14:	49                   	rex.WB
   15:	8d                   	.byte 0x8d
[15838.829023] RSP: 0018:ffff88810b7bfb98 EFLAGS: 00010202
[15838.836422] RAX: 0000000000000001 RBX: ffff88828ee54d58 RCX: 0000000000000000
[15838.845734] RDX: 1ffff11064bd1ad3 RSI: 0000000000000008 RDI: ffff888325e8d6a0
[15838.855015] RBP: ffff88828ee54fd0 R08: ffffed13f39c7a21 R09: 0000000000000000
[15838.864347] R10: ffffed1418ee9840 R11: ffff88a0c774c200 R12: ffff88824b4487b0
[15838.873690] R13: ffff88828ee55130 R14: ffff88819ea76ec0 R15: ffff88828ee54000
[15838.883005] FS:  00007f41e8aaa500(0000) GS:ffff889f9ce00000(0000) knlGS:0000000000000000
[15838.893294] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15838.901224] CR2: 00007fa141b622d8 CR3: 00000001e2a34003 CR4: 00000000001706f0
[15838.910553] Call Trace:
[15838.915219]  <TASK>
[15838.919527] ? fsnotify_destroy_marks (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/mark.c:839)
[15838.926367] ? btrfs_cleanup_one_transaction.cold.75 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/perf_event.h:1189 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:720) btrfs
[15838.935255] ? fsnotify_sb_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/atomic64_64.h:22 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-long.h:29 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-instrumented.h:1266 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:95)
[15838.941822] ? __fsnotify_vfsmount_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:91)
[15838.948905] ? evict_inodes (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713)
[15838.954918] ? dispose_list (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713)
[15838.960947] ? btrfs_sync_fs (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:1570) btrfs
[15838.967745] generic_shutdown_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:491)
[15838.974445] kill_anon_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1072 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1086)
[15838.980345] btrfs_kill_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:2551) btrfs
[15838.987130] deactivate_locked_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:332)
[15838.993736] cleanup_mnt (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/namespace.c:1187)
[15838.999417] task_work_run (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/task_work.c:177 (discriminator 1))
[15839.005185] exit_to_user_mode_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/resume_user_mode.h:49 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:169 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:201)
[15839.012112] syscall_exit_to_user_mode (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:128 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:296)
[15839.018875] do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87)
[15839.024600] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271)
[15839.031881] ? do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87)
[15839.037753] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271)
[15839.045010] entry_SYSCALL_64_after_hwframe (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/entry_64.S:120)
[15839.052194] RIP: 0033:0x7f41e8953a6b
[15839.057913] Code: 0f 1e fa 48 89 fe 31 ff e9 72 08 00 00 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 89 63 0a 00 f7 d8
All code
========
    0:	0f 1e fa             	nop    %edx
    3:	48 89 fe             	mov    %rdi,%rsi
    6:	31 ff                	xor    %edi,%edi
    8:	e9 72 08 00 00       	jmpq   0x87f
    d:	66 90                	xchg   %ax,%ax
    f:	f3 0f 1e fa          	endbr64
   13:	31 f6                	xor    %esi,%esi
   15:	e9 05 00 00 00       	jmpq   0x1f
   1a:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
   1f:	f3 0f 1e fa          	endbr64
   23:	b8 a6 00 00 00       	mov    $0xa6,%eax
   28:	0f 05                	syscall
   2a:*	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax		<-- trapping instruction
   30:	77 05                	ja     0x37
   32:	c3                   	retq
   33:	0f 1f 40 00          	nopl   0x0(%rax)

37: 48 8b 15 89 63 0a 00 mov 0xa6389(%rip),%rdx # 0xa63c7
3e: f7 d8 neg %eax

Code starting with the faulting instruction
===========================================

    0:	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax
    6:	77 05                	ja     0xd
    8:	c3                   	retq
    9:	0f 1f 40 00          	nopl   0x0(%rax)
    d:	48 8b 15 89 63 0a 00 	mov    0xa6389(%rip),%rdx        # 0xa639d
   14:	f7 d8                	neg    %eax
[15839.081093] RSP: 002b:00007fff313c23d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[15839.090894] RAX: 0000000000000000 RBX: 000055d74a755540 RCX: 00007f41e8953a6b
[15839.100242] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000055d74a75a550
[15839.109546] RBP: 000055d74a755310 R08: 0000000000000000 R09: 000055d74a754010
[15839.118783] R10: 00007f41e89faaa0 R11: 0000000000000246 R12: 0000000000000000
[15839.128016] R13: 000055d74a75a550 R14: 000055d74a755420 R15: 000055d74a755310
[15839.137246]  </TASK>
[15839.141444] irq event stamp: 36915
[15839.146842] hardirqs last enabled at (36929): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:45 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:80 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:138 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:264 (discriminator 1))
[15839.157472] hardirqs last disabled at (36942): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:262 (discriminator 1))
[15839.168059] softirqs last enabled at (34740): __do_softirq (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/preempt.h:27 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:415 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:600)
[15839.178441] softirqs last disabled at (34701): irq_exit_rcu (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:445 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:650 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:662)
[15839.188788] ---[ end trace 0000000000000000 ]---
[15839.195355] BTRFS error (device sdb2): qgroup reserved space leaked
[15839.544913] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm
[15839.563333] BTRFS info (device sdb2): using free space tree
[15839.597640] BTRFS info (device sdb2): enabling ssd optimizations

Best Regards
Wang Yugui ([email protected])
2022/09/24

#4 lore raw
Quoted 32 lines · Wang Yugui

Hi,

Quoted 14 lines · Wang Yugui
Quoted 5 lines

Hi,

fstests btrfs/042 triggle 'qgroup reserved space leaked'

kernel source: btrfs misc-next

Which commit HEAD?

As I can not reproduce using a somewhat older misc-next.

The HEAD I'm on is 2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8 ("btrfs: stop tracking failed reads in the I/O tree").

Quoted 7 lines

If it's a regression it can be much easier to pin down.

Quoted 3 lines

kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...

And any reproducibility? 16 runs no reproduce.

btrfs source version: misc-next: bf940dd88f48,
plus some minor local patch(no qgroup related)
kernel: 6.0-rc6

reproduce rate:
1) 100%(3/3) when local debug config **1
2) 0% (0/3) when local release config

Quoted 11 lines

**1:local debug config, about 100x slow than release config
a) memory debug CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/...
b) lockdep debug
CONFIG_PROVE_LOCKING/...
c) btrfs debug
CONFIG_BTRFS_FS_CHECK_INTEGRITY=y CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y CONFIG_BTRFS_DEBUG=y
CONFIG_BTRFS_ASSERT=y
CONFIG_BTRFS_FS_REF_VERIFY=y

I always run with all btrfs features enabled.

So is the lockdep.

KASAN is known to be slow, thus that is only enabled when there is suspision on memory corruption caused by some wild pointer.

Quoted 21 lines

From source:
fs/btrfs/disk-io.c:4668

     if (btrfs_check_quota_leak(fs_info)) {
L4668        WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));
         btrfs_err(fs_info, "qgroup reserved space leaked");
     }

This problem will triggle fstests btrfs/042 to failure only when CONFIG_BTRFS_DEBUG=y ?

maybe related issue:
when lockdep debug is enabled, the following issue become very easy to reproduce too.
https://lore.kernel.org/linux-nfs/[email protected]/
so there maybe some lockdep debug related , but not btrfs related problem in kernel 6.0.

Quoted 2 lines

more test(remove some minor local patch(no qgroup related)) will be done, and then I will report the result.

Better to provide the patches, as I just finished a 16 runs of btrfs/042, no reproduce.

Thus I'm starting to suspect the off-tree patches.

Thanks,
Qu

Quoted 163 lines

Best Regards
Wang Yugui ([email protected])
2022/09/24

Quoted 156 lines

Thanks,
Qu

Quoted 153 lines

dmesg output:

[15788.980873] run fstests btrfs/042 at 2022-09-24 00:40:24
[15803.880347] BTRFS info (device sdb1): using crc32c (crc32c-intel) checksum algorithm
[15803.897721] BTRFS info (device sdb1): using free space tree
[15803.932525] BTRFS info (device sdb1): enabling ssd optimizations
[15818.525145] BTRFS: device fsid b255009c-2a39-49ed-b230-b4e26befd321 devid 1 transid 6 /dev/sdb2 scanned by systemd-udevd (310493)
[15818.791882] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm
[15818.808805] BTRFS info (device sdb2): using free space tree
[15818.837770] BTRFS info (device sdb2): enabling ssd optimizations
[15818.847176] BTRFS info (device sdb2): checking UUID tree
[15818.911997] BTRFS info (device sdb2): qgroup scan completed (inconsistency flag cleared)
[15838.397073] BTRFS warning (device sdb2): qgroup 1/1 has unreleased space, type 0 rsv 12288
[15838.406954] BTRFS warning (device sdb2): qgroup 0/260 has unreleased space, type 0 rsv 4096
[15838.416728] BTRFS warning (device sdb2): qgroup 0/259 has unreleased space, type 0 rsv 4096
[15838.426511] BTRFS warning (device sdb2): qgroup 0/257 has unreleased space, type 0 rsv 4096
[15838.436351] ------------[ cut here ]------------
[15838.442380] WARNING: CPU: 0 PID: 310592 at fs/btrfs/disk-io.c:4668 close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs

fs/btrfs/disk-io.c:4668

      if (btrfs_check_quota_leak(fs_info)) {
L4668        WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));
          btrfs_err(fs_info, "qgroup reserved space leaked");
      }
[15838.452948] Modules linked in: ext4 mbcache jbd2 loop rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill ib_core sunrpc dm_multipath amdgpu iommu_v2 gpu_sched drm_buddy intel_rapl_msr intel_rapl_common btrfs sb_edac snd_hda_codec_realtek x86_pkg_temp_thermal snd_hda_codec_generic radeon intel_powerclamp snd_hda_codec_hdmi ledtrig_audio coretemp blake2b_generic snd_hda_intel xor dcdbas kvm_intel mei_wdt raid6_pq snd_intel_dspcfg i2c_algo_bit iTCO_wdt iTCO_vendor_support zstd_compress dell_smm_hwmon snd_intel_sdw_acpi drm_display_helper kvm snd_hda_codec cec snd_hda_core drm_ttm_helper irqbypass snd_hwdep rapl ttm intel_cstate snd_seq dm_mod snd_seq_device drm_kms_helper intel_uncore snd_pcm pcspkr syscopyarea mei_me snd_timer sysfillrect i2c_i801 snd sysimgblt i2c_smbus lpc_ich mei fb_sys_fops soundcore fuse drm xfs sd_mod t10_pi sr_mod cdrom sg crct10dif_pclmul crc32_pclmul bnx2x crc32c_intel ahci libahci ghash_clmulni_intel mdio libata mpt3sas e10
   00e
[15838.453270]  raid_class scsi_transport_sas wmi i2c_dev ipmi_devintf ipmi_msghandler
[15838.551107] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1
[15838.560649]  pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 fjes():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1
[15838.659255]  pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1 fjes():1
[15838.779406] CPU: 0 PID: 310592 Comm: umount Not tainted 6.0.0-7.0.debug.el7.x86_64 #1
[15838.789287] Hardware name: Dell Inc. Precision T7610/0NK70N, BIOS A18 09/11/2019
[15838.798748] RIP: 0010:close_ctree (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:749 (discriminator 14)) btrfs
[15838.805988] Code: c7 00 9b 2f c3 e8 1c e7 ff ff 48 8b 3c 24 be 08 00 00 00 e8 2f 78 70 db f0 41 80 4f 10 02 4c 89 ff e8 31 46 f4 ff 84 c0 74 11 <0f> 0b 48 c7 c6 60 9b 2f c3 4c 89 ff e8 b0 8c ff ff 4c 89 ff 49 8d
All code
========
     0:	c7 00 9b 2f c3 e8    	movl   $0xe8c32f9b,(%rax)
     6:	1c e7                	sbb    $0xe7,%al
     8:	ff                   	(bad)
     9:	ff 48 8b             	decl   -0x75(%rax)
     c:	3c 24                	cmp    $0x24,%al
     e:	be 08 00 00 00       	mov    $0x8,%esi
    13:	e8 2f 78 70 db       	callq  0xffffffffdb707847
    18:	f0 41 80 4f 10 02    	lock orb $0x2,0x10(%r15)
    1e:	4c 89 ff             	mov    %r15,%rdi
    21:	e8 31 46 f4 ff       	callq  0xfffffffffff44657
    26:	84 c0                	test   %al,%al
    28:	74 11                	je     0x3b
    2a:*	0f 0b                	ud2    		<-- trapping instruction
    2c:	48 c7 c6 60 9b 2f c3 	mov    $0xffffffffc32f9b60,%rsi
    33:	4c 89 ff             	mov    %r15,%rdi
    36:	e8 b0 8c ff ff       	callq  0xffffffffffff8ceb
    3b:	4c 89 ff             	mov    %r15,%rdi
    3e:	49                   	rex.WB
    3f:	8d                   	.byte 0x8d

Code starting with the faulting instruction
===========================================

     0:	0f 0b                	ud2
     2:	48 c7 c6 60 9b 2f c3 	mov    $0xffffffffc32f9b60,%rsi
     9:	4c 89 ff             	mov    %r15,%rdi
     c:	e8 b0 8c ff ff       	callq  0xffffffffffff8cc1
    11:	4c 89 ff             	mov    %r15,%rdi
    14:	49                   	rex.WB
    15:	8d                   	.byte 0x8d
[15838.829023] RSP: 0018:ffff88810b7bfb98 EFLAGS: 00010202
[15838.836422] RAX: 0000000000000001 RBX: ffff88828ee54d58 RCX: 0000000000000000
[15838.845734] RDX: 1ffff11064bd1ad3 RSI: 0000000000000008 RDI: ffff888325e8d6a0
[15838.855015] RBP: ffff88828ee54fd0 R08: ffffed13f39c7a21 R09: 0000000000000000
[15838.864347] R10: ffffed1418ee9840 R11: ffff88a0c774c200 R12: ffff88824b4487b0
[15838.873690] R13: ffff88828ee55130 R14: ffff88819ea76ec0 R15: ffff88828ee54000
[15838.883005] FS:  00007f41e8aaa500(0000) GS:ffff889f9ce00000(0000) knlGS:0000000000000000
[15838.893294] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15838.901224] CR2: 00007fa141b622d8 CR3: 00000001e2a34003 CR4: 00000000001706f0
[15838.910553] Call Trace:
[15838.915219]  <TASK>
[15838.919527] ? fsnotify_destroy_marks (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/mark.c:839)
[15838.926367] ? btrfs_cleanup_one_transaction.cold.75 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/perf_event.h:1189 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/trace/events/btrfs.h:720) btrfs
[15838.935255] ? fsnotify_sb_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/atomic64_64.h:22 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-long.h:29 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/atomic/atomic-instrumented.h:1266 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:95)
[15838.941822] ? __fsnotify_vfsmount_delete (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/notify/fsnotify.c:91)
[15838.948905] ? evict_inodes (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713)
[15838.954918] ? dispose_list (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/inode.c:713)
[15838.960947] ? btrfs_sync_fs (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:1570) btrfs
[15838.967745] generic_shutdown_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:491)
[15838.974445] kill_anon_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1072 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:1086)
[15838.980345] btrfs_kill_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/btrfs/super.c:2551) btrfs
[15838.987130] deactivate_locked_super (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/super.c:332)
[15838.993736] cleanup_mnt (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/fs/namespace.c:1187)
[15838.999417] task_work_run (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/task_work.c:177 (discriminator 1))
[15839.005185] exit_to_user_mode_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/include/linux/resume_user_mode.h:49 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:169 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:201)
[15839.012112] syscall_exit_to_user_mode (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:128 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/entry/common.c:296)
[15839.018875] do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87)
[15839.024600] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271)
[15839.031881] ? do_syscall_64 (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/common.c:87)
[15839.037753] ? lockdep_hardirqs_on_prepare (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4260 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4319 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/locking/lockdep.c:4271)
[15839.045010] entry_SYSCALL_64_after_hwframe (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/entry/entry_64.S:120)
[15839.052194] RIP: 0033:0x7f41e8953a6b
[15839.057913] Code: 0f 1e fa 48 89 fe 31 ff e9 72 08 00 00 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 89 63 0a 00 f7 d8
All code
========
     0:	0f 1e fa             	nop    %edx
     3:	48 89 fe             	mov    %rdi,%rsi
     6:	31 ff                	xor    %edi,%edi
     8:	e9 72 08 00 00       	jmpq   0x87f
     d:	66 90                	xchg   %ax,%ax
     f:	f3 0f 1e fa          	endbr64
    13:	31 f6                	xor    %esi,%esi
    15:	e9 05 00 00 00       	jmpq   0x1f
    1a:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
    1f:	f3 0f 1e fa          	endbr64
    23:	b8 a6 00 00 00       	mov    $0xa6,%eax
    28:	0f 05                	syscall
    2a:*	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax		<-- trapping instruction
    30:	77 05                	ja     0x37
    32:	c3                   	retq
    33:	0f 1f 40 00          	nopl   0x0(%rax)
    37:	48 8b 15 89 63 0a 00 	mov    0xa6389(%rip),%rdx        # 0xa63c7
    3e:	f7 d8                	neg    %eax

Code starting with the faulting instruction
===========================================

     0:	48 3d 00 f0 ff ff    	cmp    $0xfffffffffffff000,%rax
     6:	77 05                	ja     0xd
     8:	c3                   	retq
     9:	0f 1f 40 00          	nopl   0x0(%rax)
     d:	48 8b 15 89 63 0a 00 	mov    0xa6389(%rip),%rdx        # 0xa639d
    14:	f7 d8                	neg    %eax
[15839.081093] RSP: 002b:00007fff313c23d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[15839.090894] RAX: 0000000000000000 RBX: 000055d74a755540 RCX: 00007f41e8953a6b
[15839.100242] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000055d74a75a550
[15839.109546] RBP: 000055d74a755310 R08: 0000000000000000 R09: 000055d74a754010
[15839.118783] R10: 00007f41e89faaa0 R11: 0000000000000246 R12: 0000000000000000
[15839.128016] R13: 000055d74a75a550 R14: 000055d74a755420 R15: 000055d74a755310
[15839.137246]  </TASK>
[15839.141444] irq event stamp: 36915
[15839.146842] hardirqs last enabled at (36929): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:45 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:80 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/irqflags.h:138 (discriminator 1) /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:264 (discriminator 1))
[15839.157472] hardirqs last disabled at (36942): __up_console_sem (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/printk/printk.c:262 (discriminator 1))
[15839.168059] softirqs last enabled at (34740): __do_softirq (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/arch/x86/include/asm/preempt.h:27 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:415 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:600)
[15839.178441] softirqs last disabled at (34701): irq_exit_rcu (/usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:445 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:650 /usr/src/debug/kernel-6.0-rc6/linux-6.0.0-7.0.debug.el7.x86_64/kernel/softirq.c:662)
[15839.188788] ---[ end trace 0000000000000000 ]---
[15839.195355] BTRFS error (device sdb2): qgroup reserved space leaked
[15839.544913] BTRFS info (device sdb2): using crc32c (crc32c-intel) checksum algorithm
[15839.563333] BTRFS info (device sdb2): using free space tree
[15839.597640] BTRFS info (device sdb2): enabling ssd optimizations

Best Regards
Wang Yugui ([email protected])
2022/09/24

author
#5 lore raw

Hi,

Quoted 72 lines · Wang Yugui
Quoted 32 lines

Hi,

Quoted 14 lines · Wang Yugui
Quoted 5 lines

Hi,

fstests btrfs/042 triggle 'qgroup reserved space leaked'

kernel source: btrfs misc-next

Which commit HEAD?

As I can not reproduce using a somewhat older misc-next.

The HEAD I'm on is 2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8 ("btrfs: stop tracking failed reads in the I/O tree").

Quoted 7 lines

If it's a regression it can be much easier to pin down.

Quoted 3 lines

kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...

And any reproducibility? 16 runs no reproduce.

btrfs source version: misc-next: bf940dd88f48,
plus some minor local patch(no qgroup related)
kernel: 6.0-rc6

reproduce rate:
1) 100%(3/3) when local debug config **1
2) 0% (0/3) when local release config

Quoted 11 lines

**1:local debug config, about 100x slow than release config
a) memory debug CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/...
b) lockdep debug
CONFIG_PROVE_LOCKING/...
c) btrfs debug
CONFIG_BTRFS_FS_CHECK_INTEGRITY=y CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y CONFIG_BTRFS_DEBUG=y
CONFIG_BTRFS_ASSERT=y
CONFIG_BTRFS_FS_REF_VERIFY=y

I always run with all btrfs features enabled.

So is the lockdep.

KASAN is known to be slow, thus that is only enabled when there is suspision on memory corruption caused by some wild pointer.

Quoted 21 lines

From source:
fs/btrfs/disk-io.c:4668

     if (btrfs_check_quota_leak(fs_info)) {
L4668        WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));
         btrfs_err(fs_info, "qgroup reserved space leaked");
     }

This problem will triggle fstests btrfs/042 to failure only when CONFIG_BTRFS_DEBUG=y ?

maybe related issue:
when lockdep debug is enabled, the following issue become very easy to reproduce too.
https://lore.kernel.org/linux-nfs/[email protected]/
so there maybe some lockdep debug related , but not btrfs related problem in kernel 6.0.

Quoted 6 lines
Quoted 2 lines

more test(remove some minor local patch(no qgroup related)) will be done, and then I will report the result.

Better to provide the patches, as I just finished a 16 runs of btrfs/042, no reproduce.

Thus I'm starting to suspect the off-tree patches.

This problem happen on linux 6.0-rc6+ (master a63f2e7cb110, without btrfs misc-next patch, without local off-tree patch)
so this problem is not related to the patches still in btrfs misc-next.

reproduce rate:
100%(3/3) when local debug config
and the whole config file is attached.

this problem does not happen for 5.15.y.
but yet no test result of 5.16.x/5.17.x/5.18.x/5.19.x

Best Regards
Wang Yugui ([email protected])
2022/09/24

#6 lore raw
Quoted 77 lines · Wang Yugui

Hi,

Quoted 75 lines · Wang Yugui
Quoted 32 lines

Hi,

Quoted 14 lines · Wang Yugui
Quoted 5 lines

Hi,

fstests btrfs/042 triggle 'qgroup reserved space leaked'

kernel source: btrfs misc-next

Which commit HEAD?

As I can not reproduce using a somewhat older misc-next.

The HEAD I'm on is 2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8 ("btrfs: stop tracking failed reads in the I/O tree").

Quoted 7 lines

If it's a regression it can be much easier to pin down.

Quoted 3 lines

kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...

And any reproducibility? 16 runs no reproduce.

btrfs source version: misc-next: bf940dd88f48,
plus some minor local patch(no qgroup related)
kernel: 6.0-rc6

reproduce rate:
1) 100%(3/3) when local debug config **1
2) 0% (0/3) when local release config

Quoted 11 lines

**1:local debug config, about 100x slow than release config
a) memory debug CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/...
b) lockdep debug
CONFIG_PROVE_LOCKING/...
c) btrfs debug
CONFIG_BTRFS_FS_CHECK_INTEGRITY=y CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y CONFIG_BTRFS_DEBUG=y
CONFIG_BTRFS_ASSERT=y
CONFIG_BTRFS_FS_REF_VERIFY=y

I always run with all btrfs features enabled.

So is the lockdep.

KASAN is known to be slow, thus that is only enabled when there is suspision on memory corruption caused by some wild pointer.

Quoted 23 lines

From source:
fs/btrfs/disk-io.c:4668

      if (btrfs_check_quota_leak(fs_info)) {
L4668        WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));
          btrfs_err(fs_info, "qgroup reserved space leaked");
      }

This problem will triggle fstests btrfs/042 to failure only when CONFIG_BTRFS_DEBUG=y ?

maybe related issue:
when lockdep debug is enabled, the following issue become very easy to reproduce too.
https://lore.kernel.org/linux-nfs/[email protected]/
so there maybe some lockdep debug related , but not btrfs related problem in kernel 6.0.

more test(remove some minor local patch(no qgroup related)) will be done, and then I will report the result.

Quoted 6 lines
Quoted 3 lines

Better to provide the patches, as I just finished a 16 runs of btrfs/042, no reproduce.

Thus I'm starting to suspect the off-tree patches.

This problem happen on linux 6.0-rc6+ (master a63f2e7cb110, without btrfs misc-next patch, without local off-tree patch)

Same base, still nope.

Quoted 6 lines

so this problem is not related to the patches still in btrfs misc-next.

reproduce rate:
100%(3/3) when local debug config
and the whole config file is attached.

I don't think the config makes much difference, as the main difference is in KASAN and KMEMLEAK, which should not impact the test result.

And are you running just that test, or with the full auto group?

Quoted 8 lines

this problem does not happen for 5.15.y.
but yet no test result of 5.16.x/5.17.x/5.18.x/5.19.x

Best Regards
Wang Yugui ([email protected])
2022/09/24

author
#7 lore raw

Hi,

Quoted 95 lines · Wang Yugui
Quoted 77 lines

Hi,

Quoted 75 lines · Wang Yugui
Quoted 32 lines

Hi,

Quoted 14 lines · Wang Yugui
Quoted 5 lines

Hi,

fstests btrfs/042 triggle 'qgroup reserved space leaked'

kernel source: btrfs misc-next

Which commit HEAD?

As I can not reproduce using a somewhat older misc-next.

The HEAD I'm on is 2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8 ("btrfs: stop tracking failed reads in the I/O tree").

Quoted 7 lines

If it's a regression it can be much easier to pin down.

Quoted 3 lines

kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...

And any reproducibility? 16 runs no reproduce.

btrfs source version: misc-next: bf940dd88f48,
plus some minor local patch(no qgroup related)
kernel: 6.0-rc6

reproduce rate:
1) 100%(3/3) when local debug config **1
2) 0% (0/3) when local release config

Quoted 11 lines

**1:local debug config, about 100x slow than release config
a) memory debug CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/...
b) lockdep debug
CONFIG_PROVE_LOCKING/...
c) btrfs debug
CONFIG_BTRFS_FS_CHECK_INTEGRITY=y CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y CONFIG_BTRFS_DEBUG=y
CONFIG_BTRFS_ASSERT=y
CONFIG_BTRFS_FS_REF_VERIFY=y

I always run with all btrfs features enabled.

So is the lockdep.

KASAN is known to be slow, thus that is only enabled when there is suspision on memory corruption caused by some wild pointer.

Quoted 23 lines

From source:
fs/btrfs/disk-io.c:4668

      if (btrfs_check_quota_leak(fs_info)) {
L4668        WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));
          btrfs_err(fs_info, "qgroup reserved space leaked");
      }

This problem will triggle fstests btrfs/042 to failure only when CONFIG_BTRFS_DEBUG=y ?

maybe related issue:
when lockdep debug is enabled, the following issue become very easy to reproduce too.
https://lore.kernel.org/linux-nfs/[email protected]/
so there maybe some lockdep debug related , but not btrfs related problem in kernel 6.0.

more test(remove some minor local patch(no qgroup related)) will be done, and then I will report the result.

Quoted 6 lines
Quoted 3 lines

Better to provide the patches, as I just finished a 16 runs of btrfs/042, no reproduce.

Thus I'm starting to suspect the off-tree patches.

This problem happen on linux 6.0-rc6+ (master a63f2e7cb110, without btrfs misc-next patch, without local off-tree patch)

Same base, still nope.

Quoted 6 lines

so this problem is not related to the patches still in btrfs misc-next.

reproduce rate:
100%(3/3) when local debug config
and the whole config file is attached.

Quoted 4 lines

I don't think the config makes much difference, as the main difference is in KASAN and KMEMLEAK, which should not impact the test result.

And are you running just that test, or with the full auto group?

For 6.0-rc6 with btrfs misc-next, I tried to run full auto group. btrfs/042 failed, others(btrfs/001 ~ btrfs/157) are OK, and then I rebooted the test machine.

for 6.0-rc6 without btrfs misc-next, I tested btrfs/042 and btrfs/001 on the same machine.
then I tested 6.0-rc6 without btrfs misc-next on another 2 servers.

reproduce rate:
server1 3/3
server2 2/3
server3 3/3
total rate: 8/9

all 3 servers are in good status(ECC memory status and SSD status).

Best Regards
Wang Yugui ([email protected])
2022/09/24

#8 lore raw

On 2022/9/24 14:41, Wang Yugui wrote:
[...]

Quoted 14 lines
Quoted 5 lines

I don't think the config makes much difference, as the main difference is in KASAN and KMEMLEAK, which should not impact the test result.

And are you running just that test, or with the full auto group?

For 6.0-rc6 with btrfs misc-next, I tried to run full auto group. btrfs/042 failed, others(btrfs/001 ~ btrfs/157) are OK, and then I rebooted the test machine.

for 6.0-rc6 without btrfs misc-next, I tested btrfs/042 and btrfs/001 on the same machine.
then I tested 6.0-rc6 without btrfs misc-next on another 2 servers.

Quoted 7 lines

reproduce rate:
server1 3/3
server2 2/3
server3 3/3
total rate: 8/9

all 3 servers are in good status(ECC memory status and SSD status).

Considering no one can reproduce your failure so far, I strongly doubt if it's something else.

Please provide the following info:

- Full dmesg since bootup.
Better to reboot, then run btrfs/042, and take the dmesg.

- local.config file of xfstests

- result/btrfs/042* files of xfstests
Only need to run that exact test case, since you have a very high chance to reproduce.

- the full output of "./check btrfs/042"

Quoted 6 lines

Best Regards
Wang Yugui ([email protected])
2022/09/24

author
#9 lore raw

Hi,

Quoted 24 lines · Wang Yugui

[...]

Quoted 14 lines
Quoted 5 lines

I don't think the config makes much difference, as the main difference is in KASAN and KMEMLEAK, which should not impact the test result.

And are you running just that test, or with the full auto group?

For 6.0-rc6 with btrfs misc-next, I tried to run full auto group. btrfs/042 failed, others(btrfs/001 ~ btrfs/157) are OK, and then I rebooted the test machine.

for 6.0-rc6 without btrfs misc-next, I tested btrfs/042 and btrfs/001 on the same machine.
then I tested 6.0-rc6 without btrfs misc-next on another 2 servers.

Quoted 7 lines

reproduce rate:
server1 3/3
server2 2/3
server3 3/3
total rate: 8/9

all 3 servers are in good status(ECC memory status and SSD status).

Quoted 7 lines

Considering no one can reproduce your failure so far, I strongly doubt if it's something else.

Please provide the following info:

- Full dmesg since bootup.
Better to reboot, then run btrfs/042, and take the dmesg.

attachment file(dmesg)

Quoted 1 line

- local.config file of xfstests

attachment file(local.config)

Quoted 3 lines

- result/btrfs/042* files of xfstests
Only need to run that exact test case, since you have a very high chance to reproduce.

attachment file(042.dmesg/042.full)

Quoted 1 line

- the full output of "./check btrfs/042"

[root@T7610 xfstests]# ./check btrfs/042

FSTYP         -- btrfs
PLATFORM      -- Linux/x86_64 T7610 6.0.0-7.1.debug.el7.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Sep 24 10:35:15 CST 2022
MKFS_OPTIONS  -- /dev/sdb2

MOUNT_OPTIONS -- /dev/sdb2 /mnt/scratch

btrfs/042 1s ... _check_dmesg: something found in dmesg (see /usr/hpc-bio/xfstests/results//btrfs/042.dmesg)

Ran: btrfs/042
Failures: btrfs/042
Failed 1 of 1 tests

[root@T7610 xfstests]#

Best Regards
Wang Yugui ([email protected])
2022/09/24

#10 lore raw
In reply to Wang Yugui
Quoted 101 lines · Wang Yugui

Hi,

Quoted 99 lines · Wang Yugui
Quoted 78 lines

Hi,

Quoted 76 lines · Wang Yugui
Quoted 32 lines

Hi,

Quoted 14 lines · Wang Yugui
Quoted 5 lines

Hi,

fstests btrfs/042 triggle 'qgroup reserved space leaked'

kernel source: btrfs misc-next

Which commit HEAD?

As I can not reproduce using a somewhat older misc-next.

The HEAD I'm on is 2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8 ("btrfs: stop tracking failed reads in the I/O tree").

Quoted 7 lines

If it's a regression it can be much easier to pin down.

Quoted 3 lines

kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...

And any reproducibility? 16 runs no reproduce.

btrfs source version: misc-next: bf940dd88f48,
plus some minor local patch(no qgroup related)
kernel: 6.0-rc6

reproduce rate:
1) 100%(3/3) when local debug config **1
2) 0% (0/3) when local release config

Quoted 11 lines

**1:local debug config, about 100x slow than release config
a) memory debug CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/...
b) lockdep debug
CONFIG_PROVE_LOCKING/...
c) btrfs debug
CONFIG_BTRFS_FS_CHECK_INTEGRITY=y CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y CONFIG_BTRFS_DEBUG=y
CONFIG_BTRFS_ASSERT=y
CONFIG_BTRFS_FS_REF_VERIFY=y

I always run with all btrfs features enabled.

So is the lockdep.

KASAN is known to be slow, thus that is only enabled when there is suspision on memory corruption caused by some wild pointer.

Quoted 24 lines

From source:
fs/btrfs/disk-io.c:4668

      if (btrfs_check_quota_leak(fs_info)) {
L4668        WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));
          btrfs_err(fs_info, "qgroup reserved space 
          leaked");
      }

This problem will triggle fstests btrfs/042 to failure only when CONFIG_BTRFS_DEBUG=y ?

maybe related issue:
when lockdep debug is enabled, the following issue become very easy to reproduce too.
https://lore.kernel.org/linux-nfs/[email protected]/
so there maybe some lockdep debug related , but not btrfs related problem in kernel 6.0.

more test(remove some minor local patch(no qgroup related)) will be done, and then I will report the result.

Quoted 6 lines
Quoted 3 lines

Better to provide the patches, as I just finished a 16 runs of btrfs/042, no reproduce.

Thus I'm starting to suspect the off-tree patches.

This problem happen on linux 6.0-rc6+ (master a63f2e7cb110, without btrfs misc-next patch, without local off-tree patch)

Same base, still nope.

Quoted 6 lines

so this problem is not related to the patches still in btrfs misc-next.

reproduce rate:
100%(3/3) when local debug config
and the whole config file is attached.

I don't think the config makes much difference, as the main difference is in KASAN and KMEMLEAK, which should not impact the test result.

Quoted 9 lines
Quoted 1 line

And are you running just that test, or with the full auto group?

For 6.0-rc6 with btrfs misc-next, I tried to run full auto group. btrfs/042 failed, others(btrfs/001 ~ btrfs/157) are OK, and then I rebooted the test machine.

for 6.0-rc6 without btrfs misc-next, I tested btrfs/042 and btrfs/001 on the same machine.

May I ask if "6.0-rc6 without btrfs misc-next" is 6.0-rc6 *without any other changes*? It looks so weird cause I can't reproduce the issue either.

--
Su

Quoted 13 lines

then I tested 6.0-rc6 without btrfs misc-next on another 2 servers.

reproduce rate:
server1 3/3
server2 2/3
server3 3/3
total rate: 8/9

all 3 servers are in good status(ECC memory status and SSD status).

Best Regards
Wang Yugui ([email protected])
2022/09/24

author
#11 lore raw

Hi,

Quoted 126 lines · Wang Yugui
Quoted 124 lines

Hi,

Quoted 107 lines · Wang Yugui
Quoted 86 lines

Hi,

Quoted 84 lines · Wang Yugui
Quoted 34 lines

Hi,

Quoted 16 lines · Wang Yugui
Quoted 5 lines

Hi,

fstests btrfs/042 triggle 'qgroup reserved space leaked'

kernel source: btrfs misc-next

Which commit HEAD?

As I can not reproduce using a somewhat older misc-next.

The HEAD I'm on is 2d1aef6504bf8bdd7b6ca9fa4c0c5ab32f4da2a8 ("btrfs: stop tracking failed reads in the I/O tree").

Quoted 7 lines

If it's a regression it can be much easier to pin down.

Quoted 3 lines

kernel config:
memory debug: CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/... lock debug: CONFIG_PROVE_LOCKING/...

And any reproducibility? 16 runs no reproduce.

btrfs source version: misc-next: bf940dd88f48,
plus some minor local patch(no qgroup related)
kernel: 6.0-rc6

reproduce rate:
1) 100%(3/3) when local debug config **1
2) 0% (0/3) when local release config

Quoted 11 lines

**1:local debug config, about 100x slow than release config
a) memory debug CONFIG_KASAN/CONFIG_DEBUG_KMEMLEAK/...
b) lockdep debug
CONFIG_PROVE_LOCKING/...
c) btrfs debug
CONFIG_BTRFS_FS_CHECK_INTEGRITY=y CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y CONFIG_BTRFS_DEBUG=y
CONFIG_BTRFS_ASSERT=y
CONFIG_BTRFS_FS_REF_VERIFY=y

I always run with all btrfs features enabled.

So is the lockdep.

KASAN is known to be slow, thus that is only enabled when there is suspision on memory corruption caused by some wild pointer.

Quoted 28 lines

From source:
fs/btrfs/disk-io.c:4668

      if (btrfs_check_quota_leak(fs_info)) {
L4668        WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));
          btrfs_err(fs_info, "qgroup reserved space
          leaked");
      }

This problem will triggle fstests btrfs/042 to failure only when
CONFIG_BTRFS_DEBUG=y ?

maybe related issue:
when lockdep debug is enabled, the following issue become very easy to
reproduce too.
https://lore.kernel.org/linux-nfs/[email protected]/
so there maybe some lockdep debug related , but not btrfs related
problem in kernel 6.0.

more test(remove some minor local patch(no qgroup related)) will be done,
and then I will report the result.

Quoted 8 lines
Quoted 4 lines

Better to provide the patches, as I just finished a 16 runs of btrfs/042, no reproduce.

Thus I'm starting to suspect the off-tree patches.

This problem happen on linux 6.0-rc6+ (master a63f2e7cb110, without
btrfs misc-next patch, without local off-tree patch)

Same base, still nope.

Quoted 7 lines

so this problem is not related to the patches still in btrfs misc-next.

reproduce rate:
100%(3/3) when local debug config
and the whole config file is attached.

Quoted 7 lines

I don't think the config makes much difference, as the main difference
is in KASAN and KMEMLEAK, which should not impact the test result.

And are you running just that test, or with the full auto group?

For 6.0-rc6 with btrfs misc-next, I tried to run full auto group.
btrfs/042 failed, others(btrfs/001 ~ btrfs/157) are OK, and then I
rebooted the test machine.

Quoted 6 lines
Quoted 4 lines

for 6.0-rc6 without btrfs misc-next, I tested btrfs/042 and btrfs/001 on
the same machine.

May I ask if "6.0-rc6 without btrfs misc-next" is 6.0-rc6 *without any other changes*? It looks so weird cause I can't reproduce the issue either.

Yes. *without any other changes*

Best Regards
Wang Yugui ([email protected])
2022/09/24