Skip to contentSkip to conversation

BUG: held lock freed!

DISCUSSION Thomas Sattler 2 participants 4 messages Jun 26, 2007 – Jun 27, 2007

lore
Share & tools
lore
#0 lore raw

Hi there ...

I removed xfs from my system. The first reboot after replacing xfs with ext3 brought be

Jun 26 08:43:17 pearl =========================
Jun 26 08:43:17 pearl [ BUG: held lock freed! ]
Jun 26 08:43:17 pearl -------------------------
Jun 26 08:43:17 pearl udevd/3064 is freeing memory c16fbe40-c16fbe7f,
with a lock still held there!
Jun 26 08:43:17 pearl (&sbinfo->stat_lock){--..}, at: [<c0157eb1>]
shmem_delete_inode+0xc1/0xda
Jun 26 08:43:17 pearl 1 lock held by udevd/3064:
Jun 26 08:43:17 pearl #0:  (&sbinfo->stat_lock){--..}, at: [<c0157eb1>]
shmem_delete_inode+0xc1/0xda
Jun 26 08:43:17 pearl
Jun 26 08:43:17 pearl stack backtrace:
Jun 26 08:43:17 pearl [<c0135331>] debug_check_no_locks_freed+0xe7/0x11a
Jun 26 08:43:17 pearl [<c0158649>] kfree+0x45/0x7f
Jun 26 08:43:17 pearl [<c016c3d2>] free_fdtable_rcu+0x3a/0x70
Jun 26 08:43:17 pearl [<c012a95f>] __rcu_process_callbacks+0xfd/0x165
Jun 26 08:43:17 pearl [<c012a9d6>] rcu_process_callbacks+0xf/0x1e
Jun 26 08:43:17 pearl [<c0120c69>] tasklet_action+0x3d/0x68
Jun 26 08:43:17 pearl [<c0120b9e>] __do_softirq+0x41/0x92
Jun 26 08:43:17 pearl [<c0120c16>] do_softirq+0x27/0x3d
Jun 26 08:43:17 pearl [<c0120fae>] irq_exit+0x35/0x64
Jun 26 08:43:17 pearl [<c0106108>] do_IRQ+0x7e/0x92
Jun 26 08:43:17 pearl [<c0104634>] common_interrupt+0x24/0x34
Jun 26 08:43:17 pearl [<c010463e>] common_interrupt+0x2e/0x34
Jun 26 08:43:17 pearl [<c0136513>] lock_acquire+0x68/0x6e
Jun 26 08:43:17 pearl [<c0157eb1>] shmem_delete_inode+0xc1/0xda
Jun 26 08:43:17 pearl [<c02e4daa>] _spin_lock+0x29/0x34
Jun 26 08:43:17 pearl [<c0157eb1>] shmem_delete_inode+0xc1/0xda
Jun 26 08:43:17 pearl [<c0157eb1>] shmem_delete_inode+0xc1/0xda
Jun 26 08:43:17 pearl [<c0157df0>] shmem_delete_inode+0x0/0xda
Jun 26 08:43:17 pearl [<c016b2a8>] generic_delete_inode+0x8c/0xf4
Jun 26 08:43:17 pearl [<c016aa33>] iput+0x60/0x62
Jun 26 08:43:17 pearl [<c01636d7>] do_unlinkat+0xbe/0x132
Jun 26 08:43:17 pearl [<c0103bfa>] sysenter_past_esp+0x8f/0x99
Jun 26 08:43:17 pearl [<c0135227>] trace_hardirqs_on+0x11e/0x141
Jun 26 08:43:17 pearl [<c0103bca>] sysenter_past_esp+0x5f/0x99
Jun 26 08:43:17 pearl =======================

But it only came once, several reboots after that were ok. I changed my kernel config today: e1000 is now "=y" (was "=m"), I removed PCMCIA as I do not use it and some other modules complained about it, and I added CONFIG_HIGHMEM4G=y (was CONFIG_NOHIGHMEM=y)

The running kernel is 2.6.22.5 +cfs +squashfs. My distribution is gentoo (x86), quite up to date, udev is 104-r12.

Please CC me as I'm not subscribed to the list.

Thomas

Signature · 1 line
keep mailinglists in english, feel free to send PM in german
#1 lore raw
Quoted 4 lines · Thomas Sattler

Hi there ...

I removed xfs from my system. The first reboot after replacing xfs with ext3 brought be

Perhaps this is a curse that falls on those who desert XFS ;)

Quoted 44 lines
Jun 26 08:43:17 pearl =========================
Jun 26 08:43:17 pearl [ BUG: held lock freed! ]
Jun 26 08:43:17 pearl -------------------------
Jun 26 08:43:17 pearl udevd/3064 is freeing memory c16fbe40-c16fbe7f,
with a lock still held there!
Jun 26 08:43:17 pearl (&sbinfo->stat_lock){--..}, at: [<c0157eb1>]
shmem_delete_inode+0xc1/0xda
Jun 26 08:43:17 pearl 1 lock held by udevd/3064:
Jun 26 08:43:17 pearl #0:  (&sbinfo->stat_lock){--..}, at: [<c0157eb1>]
shmem_delete_inode+0xc1/0xda
Jun 26 08:43:17 pearl
Jun 26 08:43:17 pearl stack backtrace:
Jun 26 08:43:17 pearl [<c0135331>] debug_check_no_locks_freed+0xe7/0x11a
Jun 26 08:43:17 pearl [<c0158649>] kfree+0x45/0x7f
Jun 26 08:43:17 pearl [<c016c3d2>] free_fdtable_rcu+0x3a/0x70
Jun 26 08:43:17 pearl [<c012a95f>] __rcu_process_callbacks+0xfd/0x165
Jun 26 08:43:17 pearl [<c012a9d6>] rcu_process_callbacks+0xf/0x1e
Jun 26 08:43:17 pearl [<c0120c69>] tasklet_action+0x3d/0x68
Jun 26 08:43:17 pearl [<c0120b9e>] __do_softirq+0x41/0x92
Jun 26 08:43:17 pearl [<c0120c16>] do_softirq+0x27/0x3d
Jun 26 08:43:17 pearl [<c0120fae>] irq_exit+0x35/0x64
Jun 26 08:43:17 pearl [<c0106108>] do_IRQ+0x7e/0x92
Jun 26 08:43:17 pearl [<c0104634>] common_interrupt+0x24/0x34
Jun 26 08:43:17 pearl [<c010463e>] common_interrupt+0x2e/0x34
Jun 26 08:43:17 pearl [<c0136513>] lock_acquire+0x68/0x6e
Jun 26 08:43:17 pearl [<c0157eb1>] shmem_delete_inode+0xc1/0xda
Jun 26 08:43:17 pearl [<c02e4daa>] _spin_lock+0x29/0x34
Jun 26 08:43:17 pearl [<c0157eb1>] shmem_delete_inode+0xc1/0xda
Jun 26 08:43:17 pearl [<c0157eb1>] shmem_delete_inode+0xc1/0xda
Jun 26 08:43:17 pearl [<c0157df0>] shmem_delete_inode+0x0/0xda
Jun 26 08:43:17 pearl [<c016b2a8>] generic_delete_inode+0x8c/0xf4
Jun 26 08:43:17 pearl [<c016aa33>] iput+0x60/0x62
Jun 26 08:43:17 pearl [<c01636d7>] do_unlinkat+0xbe/0x132
Jun 26 08:43:17 pearl [<c0103bfa>] sysenter_past_esp+0x8f/0x99
Jun 26 08:43:17 pearl [<c0135227>] trace_hardirqs_on+0x11e/0x141
Jun 26 08:43:17 pearl [<c0103bca>] sysenter_past_esp+0x5f/0x99
Jun 26 08:43:17 pearl =======================

But it only came once, several reboots after that were ok. I changed my kernel config today: e1000 is now "=y" (was "=m"), I removed PCMCIA as I do not use it and some other modules complained about it, and I added CONFIG_HIGHMEM4G=y (was CONFIG_NOHIGHMEM=y)

Quoted 4 lines

The running kernel is 2.6.22.5 +cfs +squashfs. My distribution is gentoo (x86), quite up to date, udev is 104-r12.

Please CC me as I'm not subscribed to the list.

Odd. I can't see any error at the shmem_delete_inode end nor at the free_fdtable_rcu end. It seems to be some kind of corruption, whereby free_fdtable_rcu is kfree'ing some memory (perhaps fdt->open_fds), but the address kfreed is that of the shmem_sb_info in which it has just acquired a spinlock at the top of the stack.

I've not found any kfreeing of uninitialized pointer in fs/file.c.

It could come about through a single-bit error, and I was going to suggest that you give memtest86+ a good run overnight. And still do suggest that, though we seem to have rather too much of a coincidence for it to be a likely explanation. But I've no other ideas, sorry.

Hugh

#2 lore raw
Quoted 4 lines
Quoted 2 lines

I removed xfs from my system. The first reboot after replacing xfs with ext3 brought be

Perhaps this is a curse that falls on those who desert XFS ;)

My laptop sometimes 'kicks' his keyboard. That means no key is working any more, mouse is ok and I can copy and paste single characters in xterms to enter some commands. Pasting 'reboot' in a root-xterm does not work.

AFAIR it always happens when a DVB-T recording ends. But, even on daily recordings, only about once a month. Ingo Molnar told me to activate CONFIG_PROVE_LOCKING. Since then I had

=============================================
[ INFO: possible recursive locking detected ]
2.6.21.5-cfs-v17 #5
---------------------------------------------
xauth/6510 is trying to acquire lock:
(&(&ip->i_lock)->mr_lock){----}, at: [<e16e0715>] xfs_ilock+0x47/0x67 [xfs]

but task is already holding lock:
(&(&ip->i_lock)->mr_lock){----}, at: [<e16e0715>] xfs_ilock+0x47/0x67 [xfs]

other info that might help us debug this:
2 locks held by xauth/6510:
#0: (&inode->i_mutex){--..}, at: [<c016477c>] open_namei+0xe2/0x555 #1: (&(&ip->i_lock)->mr_lock){----}, at: [<e16e0715>]

xfs_ilock+0x47/0x67 [xfs]

stack backtrace:
 [<c0136026>] __lock_acquire+0x11e/0xb23
 [<c0136dd9>] lock_acquire+0x56/0x6e
 [<e16e0715>] xfs_ilock+0x47/0x67 [xfs]
 [<c012fe10>] down_write+0x2e/0x46
 [<e16e0715>] xfs_ilock+0x47/0x67 [xfs]
 [<e16e0715>] xfs_ilock+0x47/0x67 [xfs]
 [<e16e0fa3>] xfs_iget_core+0x291/0x579 [xfs]
 [<e16e1312>] xfs_iget+0x87/0xfd [xfs]
 [<e16f7665>] xfs_trans_iget+0xe6/0x151 [xfs]
 [<e16e4a53>] xfs_ialloc+0xb2/0x479 [xfs]
 [<e16f7fda>] xfs_dir_ialloc+0x7b/0x29d [xfs]
 [<c012fe10>] down_write+0x2e/0x46
 [<e16fda9b>] xfs_create+0x31c/0x5d6 [xfs]
 [<e1707676>] xfs_vn_mknod+0x19b/0x2ce [xfs]
 [<c01621f6>] vfs_create+0xa5/0xeb
 [<c0164811>] open_namei+0x177/0x555
 [<c015a8ab>] get_unused_fd+0x1f/0xb4
 [<c015ab5c>] do_filp_open+0x25/0x39
 [<c02e57da>] _spin_unlock+0x14/0x1c
 [<c015a936>] get_unused_fd+0xaa/0xb4
 [<c015abb2>] do_sys_open+0x42/0xc3
 [<c015ac6c>] sys_open+0x1c/0x1e
 [<c0103bca>] sysenter_past_esp+0x5f/0x99
 =======================

or

=============================================
[ INFO: possible recursive locking detected ]
2.6.21.5-cfs-v17 #5
---------------------------------------------
dotlockfile/6467 is trying to acquire lock:
(&(&ip->i_lock)->mr_lock){----}, at: [<e15a6715>] xfs_ilock+0x47/0x67 [xfs]

but task is already holding lock:
(&(&ip->i_lock)->mr_lock){----}, at: [<e15a6715>] xfs_ilock+0x47/0x67 [xfs]

other info that might help us debug this:
2 locks held by dotlockfile/6467:
#0: (&inode->i_mutex){--..}, at: [<c016477c>] open_namei+0xe2/0x555 #1: (&(&ip->i_lock)->mr_lock){----}, at: [<e15a6715>] xfs_ilock+0x47/0x67

stack backtrace:
 [<c0136026>] __lock_acquire+0x11e/0xb23
 [<c0136dd9>] lock_acquire+0x56/0x6e
 [<e15a6715>] xfs_ilock+0x47/0x67 [xfs]
 [<c012fe10>] down_write+0x2e/0x46
 [<e15a6715>] xfs_ilock+0x47/0x67 [xfs]
 [<e15a6715>] xfs_ilock+0x47/0x67 [xfs]
 [<e15a6fa3>] xfs_iget_core+0x291/0x579 [xfs]
 [<e15a7312>] xfs_iget+0x87/0xfd [xfs]
 [<e15bd665>] xfs_trans_iget+0xe6/0x151 [xfs]
 [<e15aaa53>] xfs_ialloc+0xb2/0x479 [xfs]
 [<e15bdfda>] xfs_dir_ialloc+0x7b/0x29d [xfs]
 [<c012fe10>] down_write+0x2e/0x46
 [<e15c3a9b>] xfs_create+0x31c/0x5d6 [xfs]
 [<e15cd676>] xfs_vn_mknod+0x19b/0x2ce [xfs]
 [<c01621f6>] vfs_create+0xa5/0xeb
 [<c0164811>] open_namei+0x177/0x555
 [<c015a8ab>] get_unused_fd+0x1f/0xb4
 [<c015ab5c>] do_filp_open+0x25/0x39
 [<c02e57da>] _spin_unlock+0x14/0x1c
 [<c015a936>] get_unused_fd+0xaa/0xb4
 [<c015abb2>] do_sys_open+0x42/0xc3
 [<c015ac6c>] sys_open+0x1c/0x1e
 [<c0103bca>] sysenter_past_esp+0x5f/0x99
 =======================

in dmesg right after bootup. Ingo said that xfs used to have problems with lockdep, but that this doesn't mean there's anything wrong with XFS, but that lockdep turns itself off after it finds the first locking problem. So I formated the data partition as ext3, which gave me the formerly posted info.

Quoted 5 lines

Odd. I can't see any error at the shmem_delete_inode end nor at the free_fdtable_rcu end. It seems to be some kind of corruption, whereby free_fdtable_rcu is kfree'ing some memory (perhaps fdt->open_fds), but the address kfreed is that of the shmem_sb_info in which it has just acquired a spinlock at the top of the stack.

There was a typo in my mail, I'm running 2.6.21.5-cfs-v18, not 2.6.22, sorry for that.

Quoted 4 lines

It could come about through a single-bit error, and I was going to suggest that you give memtest86+ a good run overnight. And still do suggest that, though we seem to have rather too much of a coincidence for it to be a likely explanation. But I've no other ideas, sorry.

I'll run memtest86+ this night and post the results tomorrow.

Please remember to CC me as I'm not subscribed to the list.

Thomas

Signature · 1 line
keep mailinglists in english, feel free to send PM in german
#3 lore raw
Quoted 1 line

I'll run memtest86+ this night and post the results tomorrow.

Memtest86+ did not show any problems:

time 8h
pass 24
errors 0

Please remember to CC me as I'm not subscribed to the list.

Thomas

Signature · 1 line
keep mailinglists in english, feel free to send PM in german