Skip to contentSkip to conversation

rm Tainted warning after kernel update.

DISCUSSION Grant Keller 3 participants 7 messages Sep 14, 2015 – Sep 16, 2015

lore
Share & tools
lore
#0 lore raw

Hello,

I have a server running Scientific Linux 6.7, and since updating to kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing in our message logs:

Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------
Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758
d_delete+0x260/0x2c0() (Tainted: G        W  -- ------------   )
Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8
Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl

auth_rpcgss autofs4 lockd sunrpc p4_clockmod freq_table speedstep_lib nf_conntrack_ftp iptable_mangle xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 xfs exportfs ppdev parport_pc parport sg e1000e microcode serio_raw iTCO_wdt iTCO_vendor_support ixgbe ptp pps_core mdio i2c_i801 lpc_ich mfd_core i5000_edac edac_core i5k_amb ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif 3w_9xxx pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]

Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G       
W  -- ------------    2.6.32-573.3.1.el6.x86_64 #1
Sep 14 11:43:03 localhost kernel: Call Trace:
Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ?
warn_slowpath_common+0x91/0xe0
Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ?
warn_slowpath_null+0x1a/0x20
Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ?
d_delete+0x260/0x2c0
Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0
Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ?
do_rmdir+0x184/0x1f0
Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210
Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ?
audit_syscall_entry+0x1d7/0x200
Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ?
sys_unlinkat+0x2d/0x40
Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ?
system_call_fastpath+0x16/0x1b
Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]---

This happens when we are expiring older backups from the archives, so I have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 installed. Looking for advice on how to proceed.

Signature · 9 lines
Grant Keller
System Operations
707-237-2451
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#1 lore raw
Quoted 43 lines · Grant Keller

Hello,

I have a server running Scientific Linux 6.7, and since updating to kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing in our message logs:

Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------
Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758
d_delete+0x260/0x2c0() (Tainted: G        W  -- ------------   )
Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8
Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl

auth_rpcgss autofs4 lockd sunrpc p4_clockmod freq_table speedstep_lib nf_conntrack_ftp iptable_mangle xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 xfs exportfs ppdev parport_pc parport sg e1000e microcode serio_raw iTCO_wdt iTCO_vendor_support ixgbe ptp pps_core mdio i2c_i801 lpc_ich mfd_core i5000_edac edac_core i5k_amb ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif 3w_9xxx pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]

Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G       
W  -- ------------    2.6.32-573.3.1.el6.x86_64 #1
Sep 14 11:43:03 localhost kernel: Call Trace:
Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ?
warn_slowpath_common+0x91/0xe0
Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ?
warn_slowpath_null+0x1a/0x20
Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ?
d_delete+0x260/0x2c0
Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0
Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ?
do_rmdir+0x184/0x1f0
Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210
Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ?
audit_syscall_entry+0x1d7/0x200
Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ?
sys_unlinkat+0x2d/0x40
Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ?
system_call_fastpath+0x16/0x1b
Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]---
Quoted 4 lines

This happens when we are expiring older backups from the archives, so I have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 installed. Looking for advice on how to proceed.

This looks like something funky going on in the vfs. The warning is from unhash_offsprings() and it appears to be complaining about a refcount on a dentry that is a child of a directory being removed. It checks a refcount on a dentry in one loop and either drops it or moves it to another list for apparent deletion. The second iteration of the aforementioned list sees a refcount on an object that wasn't there before.

I suspect this means something is going from 0->1 unexpectedly, but I'm not familiar enough with that code to grok why that shouldn't happen and how it could without reproducing it and digging into it from there. Have you identified an explicit reproducer? I assume files are simply being removed with 'rm -rf' here..? If so, does anything else have access to this directory structure (e.g., separate commands, a running backup application?) at the the time of removal?

Also, what kernel were you running before this started to occur?

Brian

Quoted 10 lines
Signature · 9 lines
Grant Keller
System Operations
707-237-2451
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
Quoted 56 lines · Brian Foster
Quoted 43 lines · Grant Keller

Hello,

I have a server running Scientific Linux 6.7, and since updating to kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing in our message logs:

Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------
Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758
d_delete+0x260/0x2c0() (Tainted: G        W  -- ------------   )
Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8
Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl

auth_rpcgss autofs4 lockd sunrpc p4_clockmod freq_table speedstep_lib nf_conntrack_ftp iptable_mangle xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 xfs exportfs ppdev parport_pc parport sg e1000e microcode serio_raw iTCO_wdt iTCO_vendor_support ixgbe ptp pps_core mdio i2c_i801 lpc_ich mfd_core i5000_edac edac_core i5k_amb ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif 3w_9xxx pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]

Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G       
W  -- ------------    2.6.32-573.3.1.el6.x86_64 #1
Sep 14 11:43:03 localhost kernel: Call Trace:
Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ?
warn_slowpath_common+0x91/0xe0
Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ?
warn_slowpath_null+0x1a/0x20
Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ?
d_delete+0x260/0x2c0
Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0
Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ?
do_rmdir+0x184/0x1f0
Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210
Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ?
audit_syscall_entry+0x1d7/0x200
Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ?
sys_unlinkat+0x2d/0x40
Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ?
system_call_fastpath+0x16/0x1b
Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]---
Quoted 4 lines

This happens when we are expiring older backups from the archives, so I have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 installed. Looking for advice on how to proceed.

This looks like something funky going on in the vfs. The warning is from unhash_offsprings() and it appears to be complaining about a refcount on a dentry that is a child of a directory being removed. It checks a refcount on a dentry in one loop and either drops it or moves it to another list for apparent deletion. The second iteration of the aforementioned list sees a refcount on an object that wasn't there before.

Quoted 7 lines

I suspect this means something is going from 0->1 unexpectedly, but I'm not familiar enough with that code to grok why that shouldn't happen and how it could without reproducing it and digging into it from there. Have you identified an explicit reproducer? I assume files are simply being removed with 'rm -rf' here..? If so, does anything else have access to this directory structure (e.g., separate commands, a running backup application?) at the the time of removal.

There could be something else running, but I would have to investigate the next time this happens. The rm -rf is called by our backup program expiring older backups from the filesystem. The thing is, the expirations happen on a nightly basis, but we don't always see these warnings in the logs. On the nights we do, there are 1000+ warnings.

Quoted 2 lines

Also, what kernel were you running before this started to occur?

2.6.32-573.el6.x86_64 was the previous kernel.

Quoted 4 lines

Brian

Signature · 9 lines
Grant Keller
System Operations
707-237-2451
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw

cc Lukas

Quoted 64 lines · Grant Keller
Quoted 56 lines · Brian Foster
Quoted 43 lines · Grant Keller

Hello,

I have a server running Scientific Linux 6.7, and since updating to kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing in our message logs:

Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------
Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758
d_delete+0x260/0x2c0() (Tainted: G        W  -- ------------   )
Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8
Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl

auth_rpcgss autofs4 lockd sunrpc p4_clockmod freq_table speedstep_lib nf_conntrack_ftp iptable_mangle xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 xfs exportfs ppdev parport_pc parport sg e1000e microcode serio_raw iTCO_wdt iTCO_vendor_support ixgbe ptp pps_core mdio i2c_i801 lpc_ich mfd_core i5000_edac edac_core i5k_amb ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif 3w_9xxx pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]

Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G       
W  -- ------------    2.6.32-573.3.1.el6.x86_64 #1
Sep 14 11:43:03 localhost kernel: Call Trace:
Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ?
warn_slowpath_common+0x91/0xe0
Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ?
warn_slowpath_null+0x1a/0x20
Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ?
d_delete+0x260/0x2c0
Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0
Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ?
do_rmdir+0x184/0x1f0
Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210
Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ?
audit_syscall_entry+0x1d7/0x200
Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ?
sys_unlinkat+0x2d/0x40
Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ?
system_call_fastpath+0x16/0x1b
Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]---
Quoted 4 lines

This happens when we are expiring older backups from the archives, so I have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 installed. Looking for advice on how to proceed.

This looks like something funky going on in the vfs. The warning is from unhash_offsprings() and it appears to be complaining about a refcount on a dentry that is a child of a directory being removed. It checks a refcount on a dentry in one loop and either drops it or moves it to another list for apparent deletion. The second iteration of the aforementioned list sees a refcount on an object that wasn't there before.

Quoted 7 lines

I suspect this means something is going from 0->1 unexpectedly, but I'm not familiar enough with that code to grok why that shouldn't happen and how it could without reproducing it and digging into it from there. Have you identified an explicit reproducer? I assume files are simply being removed with 'rm -rf' here..? If so, does anything else have access to this directory structure (e.g., separate commands, a running backup application?) at the the time of removal.

Quoted 8 lines

There could be something else running, but I would have to investigate the next time this happens. The rm -rf is called by our backup program expiring older backups from the filesystem. The thing is, the expirations happen on a nightly basis, but we don't always see these warnings in the logs. On the nights we do, there are 1000+ warnings.

Quoted 2 lines

Also, what kernel were you running before this started to occur?

2.6.32-573.el6.x86_64 was the previous kernel.

Interesting... there are only a few fs changes between this kernel and the current. One of them is this:

959c503 [fs] vfs: Unhash and evict unused children dentries after rmdir

... which actually introduces the unhash_offsprings() thing. I've cc'd Lukas who is probably more familiar with this code.

FWIW, I suspect the more you can elaborate on what the backup application might be doing here (beyond just the rm -rf), the more likely this can be reproduced and resolved.

Brian

Quoted 11 lines
Quoted 4 lines

Brian

Signature · 4 lines
Grant Keller
System Operations
707-237-2451
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw
Quoted 85 lines · Brian Foster

Date: Wed, 16 Sep 2015 06:50:23 -0400
From: Brian Foster <[email protected]>
To: Grant Keller <[email protected]>
Cc: [email protected], [email protected]
Subject: Re: rm Tainted warning after kernel update.

cc Lukas

Quoted 64 lines · Grant Keller
Quoted 56 lines · Brian Foster
Quoted 43 lines · Grant Keller

Hello,

I have a server running Scientific Linux 6.7, and since updating to kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing in our message logs:

Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------
Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758
d_delete+0x260/0x2c0() (Tainted: G        W  -- ------------   )
Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8
Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl

auth_rpcgss autofs4 lockd sunrpc p4_clockmod freq_table speedstep_lib nf_conntrack_ftp iptable_mangle xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 xfs exportfs ppdev parport_pc parport sg e1000e microcode serio_raw iTCO_wdt iTCO_vendor_support ixgbe ptp pps_core mdio i2c_i801 lpc_ich mfd_core i5000_edac edac_core i5k_amb ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif 3w_9xxx pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]

Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G       
W  -- ------------    2.6.32-573.3.1.el6.x86_64 #1
Sep 14 11:43:03 localhost kernel: Call Trace:
Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ?
warn_slowpath_common+0x91/0xe0
Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ?
warn_slowpath_null+0x1a/0x20
Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ?
d_delete+0x260/0x2c0
Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0
Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ?
do_rmdir+0x184/0x1f0
Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210
Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ?
audit_syscall_entry+0x1d7/0x200
Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ?
sys_unlinkat+0x2d/0x40
Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ?
system_call_fastpath+0x16/0x1b
Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]---
Quoted 4 lines

This happens when we are expiring older backups from the archives, so I have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 installed. Looking for advice on how to proceed.

This looks like something funky going on in the vfs. The warning is from unhash_offsprings() and it appears to be complaining about a refcount on a dentry that is a child of a directory being removed. It checks a refcount on a dentry in one loop and either drops it or moves it to another list for apparent deletion. The second iteration of the aforementioned list sees a refcount on an object that wasn't there before.

Quoted 7 lines

I suspect this means something is going from 0->1 unexpectedly, but I'm not familiar enough with that code to grok why that shouldn't happen and how it could without reproducing it and digging into it from there. Have you identified an explicit reproducer? I assume files are simply being removed with 'rm -rf' here..? If so, does anything else have access to this directory structure (e.g., separate commands, a running backup application?) at the the time of removal.

Quoted 8 lines

There could be something else running, but I would have to investigate the next time this happens. The rm -rf is called by our backup program expiring older backups from the filesystem. The thing is, the expirations happen on a nightly basis, but we don't always see these warnings in the logs. On the nights we do, there are 1000+ warnings.

Quoted 2 lines

Also, what kernel were you running before this started to occur?

2.6.32-573.el6.x86_64 was the previous kernel.

Interesting... there are only a few fs changes between this kernel and the current. One of them is this:

Quoted 8 lines

959c503 [fs] vfs: Unhash and evict unused children dentries after rmdir

... which actually introduces the unhash_offsprings() thing. I've cc'd Lukas who is probably more familiar with this code.

FWIW, I suspect the more you can elaborate on what the backup application might be doing here (beyond just the rm -rf), the more likely this can be reproduced and resolved.

Hi the lockdep warning should be fixed in the recent rhel6 kernel. Sorry about that.

-Lukas

Quoted 15 lines

Brian

Quoted 11 lines
Quoted 4 lines

Brian

Signature · 4 lines
Grant Keller
System Operations
707-237-2451
[email protected]
Quoted 5 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw
Quoted 92 lines · Lukáš Czerner
Quoted 90 lines · Brian Foster

Date: Wed, 16 Sep 2015 06:50:23 -0400
From: Brian Foster <[email protected]>
To: Grant Keller <[email protected]>
Cc: [email protected], [email protected]
Subject: Re: rm Tainted warning after kernel update.

cc Lukas

Quoted 64 lines · Grant Keller
Quoted 56 lines · Brian Foster
Quoted 43 lines · Grant Keller

Hello,

I have a server running Scientific Linux 6.7, and since updating to kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing in our message logs:

Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------
Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758
d_delete+0x260/0x2c0() (Tainted: G        W  -- ------------   )
Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8
Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl

auth_rpcgss autofs4 lockd sunrpc p4_clockmod freq_table speedstep_lib nf_conntrack_ftp iptable_mangle xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 xfs exportfs ppdev parport_pc parport sg e1000e microcode serio_raw iTCO_wdt iTCO_vendor_support ixgbe ptp pps_core mdio i2c_i801 lpc_ich mfd_core i5000_edac edac_core i5k_amb ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif 3w_9xxx pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]

Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G       
W  -- ------------    2.6.32-573.3.1.el6.x86_64 #1
Sep 14 11:43:03 localhost kernel: Call Trace:
Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ?
warn_slowpath_common+0x91/0xe0
Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ?
warn_slowpath_null+0x1a/0x20
Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ?
d_delete+0x260/0x2c0
Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0
Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ?
do_rmdir+0x184/0x1f0
Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210
Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ?
audit_syscall_entry+0x1d7/0x200
Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ?
sys_unlinkat+0x2d/0x40
Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ?
system_call_fastpath+0x16/0x1b
Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]---
Quoted 4 lines

This happens when we are expiring older backups from the archives, so I have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 installed. Looking for advice on how to proceed.

This looks like something funky going on in the vfs. The warning is from unhash_offsprings() and it appears to be complaining about a refcount on a dentry that is a child of a directory being removed. It checks a refcount on a dentry in one loop and either drops it or moves it to another list for apparent deletion. The second iteration of the aforementioned list sees a refcount on an object that wasn't there before.

Quoted 7 lines

I suspect this means something is going from 0->1 unexpectedly, but I'm not familiar enough with that code to grok why that shouldn't happen and how it could without reproducing it and digging into it from there. Have you identified an explicit reproducer? I assume files are simply being removed with 'rm -rf' here..? If so, does anything else have access to this directory structure (e.g., separate commands, a running backup application?) at the the time of removal.

Quoted 8 lines

There could be something else running, but I would have to investigate the next time this happens. The rm -rf is called by our backup program expiring older backups from the filesystem. The thing is, the expirations happen on a nightly basis, but we don't always see these warnings in the logs. On the nights we do, there are 1000+ warnings.

Quoted 2 lines

Also, what kernel were you running before this started to occur?

2.6.32-573.el6.x86_64 was the previous kernel.

Interesting... there are only a few fs changes between this kernel and the current. One of them is this:

959c503 [fs] vfs: Unhash and evict unused children dentries after rmdir

... which actually introduces the unhash_offsprings() thing. I've cc'd Lukas who is probably more familiar with this code.

Quoted 7 lines
Quoted 3 lines

FWIW, I suspect the more you can elaborate on what the backup application might be doing here (beyond just the rm -rf), the more likely this can be reproduced and resolved.

Hi the lockdep warning should be fixed in the recent rhel6 kernel. Sorry about that.

My understanding is that this patch has been reverted...

Grant,

If a newer kernel is not yet available, you might want to revert to the previous version until one is and then go straight to that.

Lukas,

If the patch is indeed reverted, then clearly this problem will go away. That aside, where does lockdep come into play? Note the warning reported above:

	WARNING: at fs/dcache.c:758 d_delete+0x260/0x2c0()

... is an explicit WARN_ON() in the code added by this commit. Are we referring to the same issue here?

Brian

Quoted 23 lines

-Lukas

Quoted 15 lines

Brian

Quoted 11 lines
Quoted 4 lines

Brian

Signature · 4 lines
Grant Keller
System Operations
707-237-2451
[email protected]
Quoted 5 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Quoted 4 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw
Quoted 120 lines · Brian Foster

Date: Wed, 16 Sep 2015 07:42:31 -0400
From: Brian Foster <[email protected]>
To: Lukáš Czerner <[email protected]>
Cc: Grant Keller <[email protected]>, [email protected]
Subject: Re: rm Tainted warning after kernel update.

Quoted 92 lines · Lukáš Czerner
Quoted 90 lines · Brian Foster

Date: Wed, 16 Sep 2015 06:50:23 -0400
From: Brian Foster <[email protected]>
To: Grant Keller <[email protected]>
Cc: [email protected], [email protected]
Subject: Re: rm Tainted warning after kernel update.

cc Lukas

Quoted 64 lines · Grant Keller
Quoted 56 lines · Brian Foster
Quoted 43 lines · Grant Keller

Hello,

I have a server running Scientific Linux 6.7, and since updating to kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing in our message logs:

Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------
Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758
d_delete+0x260/0x2c0() (Tainted: G        W  -- ------------   )
Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8
Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl

auth_rpcgss autofs4 lockd sunrpc p4_clockmod freq_table speedstep_lib nf_conntrack_ftp iptable_mangle xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 xfs exportfs ppdev parport_pc parport sg e1000e microcode serio_raw iTCO_wdt iTCO_vendor_support ixgbe ptp pps_core mdio i2c_i801 lpc_ich mfd_core i5000_edac edac_core i5k_amb ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif 3w_9xxx pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]

Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G       
W  -- ------------    2.6.32-573.3.1.el6.x86_64 #1
Sep 14 11:43:03 localhost kernel: Call Trace:
Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ?
warn_slowpath_common+0x91/0xe0
Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ?
warn_slowpath_null+0x1a/0x20
Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ?
d_delete+0x260/0x2c0
Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0
Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ?
do_rmdir+0x184/0x1f0
Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210
Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ?
audit_syscall_entry+0x1d7/0x200
Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ?
sys_unlinkat+0x2d/0x40
Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ?
system_call_fastpath+0x16/0x1b
Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]---
Quoted 4 lines

This happens when we are expiring older backups from the archives, so I have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 installed. Looking for advice on how to proceed.

This looks like something funky going on in the vfs. The warning is from unhash_offsprings() and it appears to be complaining about a refcount on a dentry that is a child of a directory being removed. It checks a refcount on a dentry in one loop and either drops it or moves it to another list for apparent deletion. The second iteration of the aforementioned list sees a refcount on an object that wasn't there before.

Quoted 7 lines

I suspect this means something is going from 0->1 unexpectedly, but I'm not familiar enough with that code to grok why that shouldn't happen and how it could without reproducing it and digging into it from there. Have you identified an explicit reproducer? I assume files are simply being removed with 'rm -rf' here..? If so, does anything else have access to this directory structure (e.g., separate commands, a running backup application?) at the the time of removal.

Quoted 8 lines

There could be something else running, but I would have to investigate the next time this happens. The rm -rf is called by our backup program expiring older backups from the filesystem. The thing is, the expirations happen on a nightly basis, but we don't always see these warnings in the logs. On the nights we do, there are 1000+ warnings.

Quoted 2 lines

Also, what kernel were you running before this started to occur?

2.6.32-573.el6.x86_64 was the previous kernel.

Interesting... there are only a few fs changes between this kernel and the current. One of them is this:

959c503 [fs] vfs: Unhash and evict unused children dentries after rmdir

... which actually introduces the unhash_offsprings() thing. I've cc'd Lukas who is probably more familiar with this code.

Quoted 7 lines
Quoted 3 lines

FWIW, I suspect the more you can elaborate on what the backup application might be doing here (beyond just the rm -rf), the more likely this can be reproduced and resolved.

Hi the lockdep warning should be fixed in the recent rhel6 kernel. Sorry about that.

My understanding is that this patch has been reverted...

Grant,

If a newer kernel is not yet available, you might want to revert to the previous version until one is and then go straight to that.

Lukas,

If the patch is indeed reverted, then clearly this problem will go away. That aside, where does lockdep come into play? Note the warning reported above:

Quoted 6 lines
	WARNING: at fs/dcache.c:758 d_delete+0x260/0x2c0()

... is an explicit WARN_ON() in the code added by this commit. Are we referring to the same issue here?

Brian

Hi Brian,

seems like we're not referring to the same issue as I though we're talking about the lockdep problem, but apparently we're not. Can we get a RH bugzilla for this issue so it can be dealt with properly ?

Thanks!
-Lukas

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs