rm Tainted warning after kernel update.
Hello,
I have a server running Scientific Linux 6.7, and since updating to kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing in our message logs:
Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------ Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758 d_delete+0x260/0x2c0() (Tainted: G W -- ------------ ) Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8 Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl
auth_rpcgss autofs4 lockd sunrpc p4_clockmod freq_table speedstep_lib nf_conntrack_ftp iptable_mangle xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 xfs exportfs ppdev parport_pc parport sg e1000e microcode serio_raw iTCO_wdt iTCO_vendor_support ixgbe ptp pps_core mdio i2c_i801 lpc_ich mfd_core i5000_edac edac_core i5k_amb ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif 3w_9xxx pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]
Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G W -- ------------ 2.6.32-573.3.1.el6.x86_64 #1 Sep 14 11:43:03 localhost kernel: Call Trace: Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ? warn_slowpath_common+0x91/0xe0 Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ? warn_slowpath_null+0x1a/0x20 Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ? d_delete+0x260/0x2c0 Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0 Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ? do_rmdir+0x184/0x1f0 Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210 Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ? audit_syscall_entry+0x1d7/0x200 Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ? sys_unlinkat+0x2d/0x40 Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]---
This happens when we are expiring older backups from the archives, so I have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 installed. Looking for advice on how to proceed.
Signature · 9 lines
Grant Keller System Operations 707-237-2451 [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 43 lines · Grant Keller
Hello,
I have a server running Scientific Linux 6.7, and since updating to kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing in our message logs:
Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------ Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758 d_delete+0x260/0x2c0() (Tainted: G W -- ------------ ) Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8 Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl
auth_rpcgssautofs4 lockd sunrpcp4_clockmodfreq_tablespeedstep_libnf_conntrack_ftpiptable_manglext_commentnf_conntrack_ipv4nf_defrag_ipv4xt_multiportiptable_filterip_tablesip6t_REJECTnf_conntrack_ipv6nf_defrag_ipv6xt_statenf_conntrackip6table_filterip6_tablesib_ipoibrdma_ucmib_ucmib_uverbsib_umadrdma_cmib_cmiw_cmib_saib_madib_coreib_addripv6 xfs exportfs ppdevparport_pcparport sg e1000e microcodeserio_rawiTCO_wdtiTCO_vendor_supportixgbe ptppps_coremdioi2c_i801lpc_ichmfd_corei5000_edacedac_corei5k_ambioatdma dca shpchp ext4 jbd2 mbcachesd_modcrc_t10dif3w_9xxxpata_acpiata_genericata_piixradeon ttmdrm_kms_helperdrmi2c_algo_biti2c_coredm_mirrordm_region_hashdm_logdm_mod[last unloaded:ipmi_msghandler]Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G W -- ------------ 2.6.32-573.3.1.el6.x86_64 #1 Sep 14 11:43:03 localhost kernel: Call Trace: Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ? warn_slowpath_common+0x91/0xe0 Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ? warn_slowpath_null+0x1a/0x20 Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ? d_delete+0x260/0x2c0 Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0 Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ? do_rmdir+0x184/0x1f0 Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210 Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ? audit_syscall_entry+0x1d7/0x200 Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ? sys_unlinkat+0x2d/0x40 Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]---
Quoted 4 lines
This happens when we are expiring older backups from the archives, so I have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 installed. Looking for advice on how to proceed.
This looks like something funky going on in the vfs. The warning is from unhash_offsprings() and it appears to be complaining about a refcount on a dentry that is a child of a directory being removed. It checks a refcount on a dentry in one loop and either drops it or moves it to another list for apparent deletion. The second iteration of the aforementioned list sees a refcount on an object that wasn't there before.
I suspect this means something is going from 0->1 unexpectedly, but I'm not familiar enough with that code to grok why that shouldn't happen and how it could without reproducing it and digging into it from there. Have you identified an explicit reproducer? I assume files are simply being removed with 'rm -rf' here..? If so, does anything else have access to this directory structure (e.g., separate commands, a running backup application?) at the the time of removal?
Also, what kernel were you running before this started to occur?
Brian
Quoted 10 lines
Signature · 9 lines
Grant Keller System Operations 707-237-2451 [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 56 lines · Brian Foster
Quoted 43 lines · Grant Keller
Hello,
I have a server running Scientific Linux 6.7, and since updating to kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing in our message logs:
Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------ Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758 d_delete+0x260/0x2c0() (Tainted: G W -- ------------ ) Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8 Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl
auth_rpcgssautofs4 lockd sunrpcp4_clockmodfreq_tablespeedstep_libnf_conntrack_ftpiptable_manglext_commentnf_conntrack_ipv4nf_defrag_ipv4xt_multiportiptable_filterip_tablesip6t_REJECTnf_conntrack_ipv6nf_defrag_ipv6xt_statenf_conntrackip6table_filterip6_tablesib_ipoibrdma_ucmib_ucmib_uverbsib_umadrdma_cmib_cmiw_cmib_saib_madib_coreib_addripv6 xfs exportfs ppdevparport_pcparport sg e1000e microcodeserio_rawiTCO_wdtiTCO_vendor_supportixgbe ptppps_coremdioi2c_i801lpc_ichmfd_corei5000_edacedac_corei5k_ambioatdma dca shpchp ext4 jbd2 mbcachesd_modcrc_t10dif3w_9xxxpata_acpiata_genericata_piixradeon ttmdrm_kms_helperdrmi2c_algo_biti2c_coredm_mirrordm_region_hashdm_logdm_mod[last unloaded:ipmi_msghandler]Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G W -- ------------ 2.6.32-573.3.1.el6.x86_64 #1 Sep 14 11:43:03 localhost kernel: Call Trace: Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ? warn_slowpath_common+0x91/0xe0 Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ? warn_slowpath_null+0x1a/0x20 Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ? d_delete+0x260/0x2c0 Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0 Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ? do_rmdir+0x184/0x1f0 Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210 Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ? audit_syscall_entry+0x1d7/0x200 Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ? sys_unlinkat+0x2d/0x40 Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]---Quoted 4 lines
This happens when we are expiring older backups from the archives, so I have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 installed. Looking for advice on how to proceed.
This looks like something funky going on in the vfs. The warning is from
unhash_offsprings()and it appears to be complaining about a refcount on a dentry that is a child of a directory being removed. It checks a refcount on a dentry in one loop and either drops it or moves it to another list for apparent deletion. The second iteration of the aforementioned list sees a refcount on an object that wasn't there before.
Quoted 7 lines
I suspect this means something is going from 0->1 unexpectedly, but I'm not familiar enough with that code to grok why that shouldn't happen and how it could without reproducing it and digging into it from there. Have you identified an explicit reproducer? I assume files are simply being removed with 'rm -rf' here..? If so, does anything else have access to this directory structure (e.g., separate commands, a running backup application?) at the the time of removal.
There could be something else running, but I would have to investigate the next time this happens. The rm -rf is called by our backup program expiring older backups from the filesystem. The thing is, the expirations happen on a nightly basis, but we don't always see these warnings in the logs. On the nights we do, there are 1000+ warnings.
Quoted 2 lines
Also, what kernel were you running before this started to occur?
2.6.32-573.el6.x86_64 was the previous kernel.
Quoted 4 lines
Brian
Signature · 9 lines
Grant Keller System Operations 707-237-2451 [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
cc Lukas
Quoted 64 lines · Grant Keller
Quoted 56 lines · Brian Foster
Quoted 43 lines · Grant Keller
Hello,
I have a server running Scientific Linux 6.7, and since updating to kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing in our message logs:
Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------ Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758 d_delete+0x260/0x2c0() (Tainted: G W -- ------------ ) Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8 Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl
auth_rpcgssautofs4 lockd sunrpcp4_clockmodfreq_tablespeedstep_libnf_conntrack_ftpiptable_manglext_commentnf_conntrack_ipv4nf_defrag_ipv4xt_multiportiptable_filterip_tablesip6t_REJECTnf_conntrack_ipv6nf_defrag_ipv6xt_statenf_conntrackip6table_filterip6_tablesib_ipoibrdma_ucmib_ucmib_uverbsib_umadrdma_cmib_cmiw_cmib_saib_madib_coreib_addripv6 xfs exportfs ppdevparport_pcparport sg e1000e microcodeserio_rawiTCO_wdtiTCO_vendor_supportixgbe ptppps_coremdioi2c_i801lpc_ichmfd_corei5000_edacedac_corei5k_ambioatdma dca shpchp ext4 jbd2 mbcachesd_modcrc_t10dif3w_9xxxpata_acpiata_genericata_piixradeon ttmdrm_kms_helperdrmi2c_algo_biti2c_coredm_mirrordm_region_hashdm_logdm_mod[last unloaded:ipmi_msghandler]Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G W -- ------------ 2.6.32-573.3.1.el6.x86_64 #1 Sep 14 11:43:03 localhost kernel: Call Trace: Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ? warn_slowpath_common+0x91/0xe0 Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ? warn_slowpath_null+0x1a/0x20 Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ? d_delete+0x260/0x2c0 Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0 Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ? do_rmdir+0x184/0x1f0 Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210 Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ? audit_syscall_entry+0x1d7/0x200 Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ? sys_unlinkat+0x2d/0x40 Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]---Quoted 4 lines
This happens when we are expiring older backups from the archives, so I have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 installed. Looking for advice on how to proceed.
This looks like something funky going on in the vfs. The warning is from
unhash_offsprings()and it appears to be complaining about a refcount on a dentry that is a child of a directory being removed. It checks a refcount on a dentry in one loop and either drops it or moves it to another list for apparent deletion. The second iteration of the aforementioned list sees a refcount on an object that wasn't there before.Quoted 7 lines
I suspect this means something is going from 0->1 unexpectedly, but I'm not familiar enough with that code to grok why that shouldn't happen and how it could without reproducing it and digging into it from there. Have you identified an explicit reproducer? I assume files are simply being removed with 'rm -rf' here..? If so, does anything else have access to this directory structure (e.g., separate commands, a running backup application?) at the the time of removal.
Quoted 8 lines
There could be something else running, but I would have to investigate the next time this happens. The rm -rf is called by our backup program expiring older backups from the filesystem. The thing is, the expirations happen on a nightly basis, but we don't always see these warnings in the logs. On the nights we do, there are 1000+ warnings.
Quoted 2 lines
Also, what kernel were you running before this started to occur?
2.6.32-573.el6.x86_64 was the previous kernel.
Interesting... there are only a few fs changes between this kernel and the current. One of them is this:
959c503 [fs] vfs: Unhash and evict unused children dentries after rmdir
... which actually introduces the unhash_offsprings() thing. I've cc'd Lukas who is probably more familiar with this code.
FWIW, I suspect the more you can elaborate on what the backup application might be doing here (beyond just the rm -rf), the more likely this can be reproduced and resolved.
Brian
Quoted 11 lines
Quoted 4 lines
Brian
Signature · 4 lines
Grant Keller System Operations 707-237-2451 [email protected]
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 85 lines · Brian Foster
Date: Wed, 16 Sep 2015 06:50:23 -0400
From: Brian Foster <[email protected]>
To: Grant Keller <[email protected]>
Cc: [email protected], [email protected]
Subject: Re: rm Tainted warning after kernel update.cc Lukas
Quoted 64 lines · Grant Keller
Quoted 56 lines · Brian Foster
Quoted 43 lines · Grant Keller
Hello,
I have a server running Scientific Linux 6.7, and since updating to kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing in our message logs:
Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------ Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758 d_delete+0x260/0x2c0() (Tainted: G W -- ------------ ) Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8 Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl
auth_rpcgssautofs4 lockd sunrpcp4_clockmodfreq_tablespeedstep_libnf_conntrack_ftpiptable_manglext_commentnf_conntrack_ipv4nf_defrag_ipv4xt_multiportiptable_filterip_tablesip6t_REJECTnf_conntrack_ipv6nf_defrag_ipv6xt_statenf_conntrackip6table_filterip6_tablesib_ipoibrdma_ucmib_ucmib_uverbsib_umadrdma_cmib_cmiw_cmib_saib_madib_coreib_addripv6 xfs exportfs ppdevparport_pcparport sg e1000e microcodeserio_rawiTCO_wdtiTCO_vendor_supportixgbe ptppps_coremdioi2c_i801lpc_ichmfd_corei5000_edacedac_corei5k_ambioatdma dca shpchp ext4 jbd2 mbcachesd_modcrc_t10dif3w_9xxxpata_acpiata_genericata_piixradeon ttmdrm_kms_helperdrmi2c_algo_biti2c_coredm_mirrordm_region_hashdm_logdm_mod[last unloaded:ipmi_msghandler]Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G W -- ------------ 2.6.32-573.3.1.el6.x86_64 #1 Sep 14 11:43:03 localhost kernel: Call Trace: Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ? warn_slowpath_common+0x91/0xe0 Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ? warn_slowpath_null+0x1a/0x20 Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ? d_delete+0x260/0x2c0 Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0 Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ? do_rmdir+0x184/0x1f0 Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210 Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ? audit_syscall_entry+0x1d7/0x200 Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ? sys_unlinkat+0x2d/0x40 Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]---Quoted 4 lines
This happens when we are expiring older backups from the archives, so I have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 installed. Looking for advice on how to proceed.
This looks like something funky going on in the vfs. The warning is from
unhash_offsprings()and it appears to be complaining about a refcount on a dentry that is a child of a directory being removed. It checks a refcount on a dentry in one loop and either drops it or moves it to another list for apparent deletion. The second iteration of the aforementioned list sees a refcount on an object that wasn't there before.Quoted 7 lines
I suspect this means something is going from 0->1 unexpectedly, but I'm not familiar enough with that code to grok why that shouldn't happen and how it could without reproducing it and digging into it from there. Have you identified an explicit reproducer? I assume files are simply being removed with 'rm -rf' here..? If so, does anything else have access to this directory structure (e.g., separate commands, a running backup application?) at the the time of removal.
Quoted 8 lines
There could be something else running, but I would have to investigate the next time this happens. The rm -rf is called by our backup program expiring older backups from the filesystem. The thing is, the expirations happen on a nightly basis, but we don't always see these warnings in the logs. On the nights we do, there are 1000+ warnings.
Quoted 2 lines
Also, what kernel were you running before this started to occur?
2.6.32-573.el6.x86_64 was the previous kernel.
Interesting... there are only a few fs changes between this kernel and the current. One of them is this:
Quoted 8 lines
959c503 [fs] vfs: Unhash and evict unused children dentries after rmdir
... which actually introduces the
unhash_offsprings()thing. I've cc'd Lukas who is probably more familiar with this code.FWIW, I suspect the more you can elaborate on what the backup application might be doing here (beyond just the rm -rf), the more likely this can be reproduced and resolved.
Hi the lockdep warning should be fixed in the recent rhel6 kernel. Sorry about that.
-Lukas
Quoted 15 lines
Brian
Quoted 11 lines
Quoted 4 lines
Brian
Signature · 4 lines
Grant Keller System Operations 707-237-2451 [email protected]
Quoted 5 lines
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 92 lines · Lukáš Czerner
Quoted 90 lines · Brian Foster
Date: Wed, 16 Sep 2015 06:50:23 -0400
From: Brian Foster <[email protected]>
To: Grant Keller <[email protected]>
Cc: [email protected], [email protected]
Subject: Re: rm Tainted warning after kernel update.cc Lukas
Quoted 64 lines · Grant Keller
Quoted 56 lines · Brian Foster
Quoted 43 lines · Grant Keller
Hello,
I have a server running Scientific Linux 6.7, and since updating to kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing in our message logs:
Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------ Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758 d_delete+0x260/0x2c0() (Tainted: G W -- ------------ ) Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8 Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl
auth_rpcgssautofs4 lockd sunrpcp4_clockmodfreq_tablespeedstep_libnf_conntrack_ftpiptable_manglext_commentnf_conntrack_ipv4nf_defrag_ipv4xt_multiportiptable_filterip_tablesip6t_REJECTnf_conntrack_ipv6nf_defrag_ipv6xt_statenf_conntrackip6table_filterip6_tablesib_ipoibrdma_ucmib_ucmib_uverbsib_umadrdma_cmib_cmiw_cmib_saib_madib_coreib_addripv6 xfs exportfs ppdevparport_pcparport sg e1000e microcodeserio_rawiTCO_wdtiTCO_vendor_supportixgbe ptppps_coremdioi2c_i801lpc_ichmfd_corei5000_edacedac_corei5k_ambioatdma dca shpchp ext4 jbd2 mbcachesd_modcrc_t10dif3w_9xxxpata_acpiata_genericata_piixradeon ttmdrm_kms_helperdrmi2c_algo_biti2c_coredm_mirrordm_region_hashdm_logdm_mod[last unloaded:ipmi_msghandler]Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G W -- ------------ 2.6.32-573.3.1.el6.x86_64 #1 Sep 14 11:43:03 localhost kernel: Call Trace: Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ? warn_slowpath_common+0x91/0xe0 Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ? warn_slowpath_null+0x1a/0x20 Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ? d_delete+0x260/0x2c0 Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0 Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ? do_rmdir+0x184/0x1f0 Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210 Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ? audit_syscall_entry+0x1d7/0x200 Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ? sys_unlinkat+0x2d/0x40 Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]---Quoted 4 lines
This happens when we are expiring older backups from the archives, so I have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 installed. Looking for advice on how to proceed.
This looks like something funky going on in the vfs. The warning is from
unhash_offsprings()and it appears to be complaining about a refcount on a dentry that is a child of a directory being removed. It checks a refcount on a dentry in one loop and either drops it or moves it to another list for apparent deletion. The second iteration of the aforementioned list sees a refcount on an object that wasn't there before.Quoted 7 lines
I suspect this means something is going from 0->1 unexpectedly, but I'm not familiar enough with that code to grok why that shouldn't happen and how it could without reproducing it and digging into it from there. Have you identified an explicit reproducer? I assume files are simply being removed with 'rm -rf' here..? If so, does anything else have access to this directory structure (e.g., separate commands, a running backup application?) at the the time of removal.
Quoted 8 lines
There could be something else running, but I would have to investigate the next time this happens. The rm -rf is called by our backup program expiring older backups from the filesystem. The thing is, the expirations happen on a nightly basis, but we don't always see these warnings in the logs. On the nights we do, there are 1000+ warnings.
Quoted 2 lines
Also, what kernel were you running before this started to occur?
2.6.32-573.el6.x86_64 was the previous kernel.
Interesting... there are only a few fs changes between this kernel and the current. One of them is this:
959c503 [fs] vfs: Unhash and evict unused children dentries after rmdir
... which actually introduces the
unhash_offsprings()thing. I've cc'd Lukas who is probably more familiar with this code.
Quoted 7 lines
Quoted 3 lines
FWIW, I suspect the more you can elaborate on what the backup application might be doing here (beyond just the rm -rf), the more likely this can be reproduced and resolved.
Hi the lockdep warning should be fixed in the recent rhel6 kernel. Sorry about that.
My understanding is that this patch has been reverted...
Grant,
If a newer kernel is not yet available, you might want to revert to the previous version until one is and then go straight to that.
Lukas,
If the patch is indeed reverted, then clearly this problem will go away. That aside, where does lockdep come into play? Note the warning reported above:
WARNING: at fs/dcache.c:758 d_delete+0x260/0x2c0()
... is an explicit WARN_ON() in the code added by this commit. Are we referring to the same issue here?
Brian
Quoted 23 lines
-Lukas
Quoted 15 lines
Brian
Quoted 11 lines
Quoted 4 lines
Brian
Signature · 4 lines
Grant Keller System Operations 707-237-2451 [email protected]Quoted 5 lines
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 120 lines · Brian Foster
Date: Wed, 16 Sep 2015 07:42:31 -0400
From: Brian Foster <[email protected]>
To: Lukáš Czerner <[email protected]>
Cc: Grant Keller <[email protected]>, [email protected]
Subject: Re: rm Tainted warning after kernel update.Quoted 92 lines · Lukáš Czerner
Quoted 90 lines · Brian Foster
Date: Wed, 16 Sep 2015 06:50:23 -0400
From: Brian Foster <[email protected]>
To: Grant Keller <[email protected]>
Cc: [email protected], [email protected]
Subject: Re: rm Tainted warning after kernel update.cc Lukas
Quoted 64 lines · Grant Keller
Quoted 56 lines · Brian Foster
Quoted 43 lines · Grant Keller
Hello,
I have a server running Scientific Linux 6.7, and since updating to kernel 2.6.32-573.3.1.el6.x86_64 the following error has begun appearing in our message logs:
Sep 14 11:43:03 localhost kernel: ------------[ cut here ]------------ Sep 14 11:43:03 localhost kernel: WARNING: at fs/dcache.c:758 d_delete+0x260/0x2c0() (Tainted: G W -- ------------ ) Sep 14 11:43:03 localhost kernel: Hardware name: X7DB8 Sep 14 11:43:03 localhost kernel: Modules linked in: nfsd nfs_acl
auth_rpcgssautofs4 lockd sunrpcp4_clockmodfreq_tablespeedstep_libnf_conntrack_ftpiptable_manglext_commentnf_conntrack_ipv4nf_defrag_ipv4xt_multiportiptable_filterip_tablesip6t_REJECTnf_conntrack_ipv6nf_defrag_ipv6xt_statenf_conntrackip6table_filterip6_tablesib_ipoibrdma_ucmib_ucmib_uverbsib_umadrdma_cmib_cmiw_cmib_saib_madib_coreib_addripv6 xfs exportfs ppdevparport_pcparport sg e1000e microcodeserio_rawiTCO_wdtiTCO_vendor_supportixgbe ptppps_coremdioi2c_i801lpc_ichmfd_corei5000_edacedac_corei5k_ambioatdma dca shpchp ext4 jbd2 mbcachesd_modcrc_t10dif3w_9xxxpata_acpiata_genericata_piixradeon ttmdrm_kms_helperdrmi2c_algo_biti2c_coredm_mirrordm_region_hashdm_logdm_mod[last unloaded:ipmi_msghandler]Sep 14 11:43:03 localhost kernel: Pid: 15893, comm: rm Tainted: G W -- ------------ 2.6.32-573.3.1.el6.x86_64 #1 Sep 14 11:43:03 localhost kernel: Call Trace: Sep 14 11:43:03 localhost kernel: [<ffffffff81077491>] ? warn_slowpath_common+0x91/0xe0 Sep 14 11:43:03 localhost kernel: [<ffffffff810774fa>] ? warn_slowpath_null+0x1a/0x20 Sep 14 11:43:03 localhost kernel: [<ffffffff811ae660>] ? d_delete+0x260/0x2c0 Sep 14 11:43:03 localhost kernel: [<ffffffff811a0908>] ? vfs_rmdir+0xe8/0xf0 Sep 14 11:43:03 localhost kernel: [<ffffffff811a3b64>] ? do_rmdir+0x184/0x1f0 Sep 14 11:43:03 localhost kernel: [<ffffffff81193511>] ? __fput+0x1a1/0x210 Sep 14 11:43:03 localhost kernel: [<ffffffff810e8ab7>] ? audit_syscall_entry+0x1d7/0x200 Sep 14 11:43:03 localhost kernel: [<ffffffff811a3bfd>] ? sys_unlinkat+0x2d/0x40 Sep 14 11:43:03 localhost kernel: [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b Sep 14 11:43:03 localhost kernel: ---[ end trace 6080ec4a7ec5ec25 ]---Quoted 4 lines
This happens when we are expiring older backups from the archives, so I have quite a few of these. We have xfsprogs 3.1.1-16.el6.x86_64 installed. Looking for advice on how to proceed.
This looks like something funky going on in the vfs. The warning is from
unhash_offsprings()and it appears to be complaining about a refcount on a dentry that is a child of a directory being removed. It checks a refcount on a dentry in one loop and either drops it or moves it to another list for apparent deletion. The second iteration of the aforementioned list sees a refcount on an object that wasn't there before.Quoted 7 lines
I suspect this means something is going from 0->1 unexpectedly, but I'm not familiar enough with that code to grok why that shouldn't happen and how it could without reproducing it and digging into it from there. Have you identified an explicit reproducer? I assume files are simply being removed with 'rm -rf' here..? If so, does anything else have access to this directory structure (e.g., separate commands, a running backup application?) at the the time of removal.
Quoted 8 lines
There could be something else running, but I would have to investigate the next time this happens. The rm -rf is called by our backup program expiring older backups from the filesystem. The thing is, the expirations happen on a nightly basis, but we don't always see these warnings in the logs. On the nights we do, there are 1000+ warnings.
Quoted 2 lines
Also, what kernel were you running before this started to occur?
2.6.32-573.el6.x86_64 was the previous kernel.
Interesting... there are only a few fs changes between this kernel and the current. One of them is this:
959c503 [fs] vfs: Unhash and evict unused children dentries after rmdir
... which actually introduces the
unhash_offsprings()thing. I've cc'd Lukas who is probably more familiar with this code.Quoted 7 lines
Quoted 3 lines
FWIW, I suspect the more you can elaborate on what the backup application might be doing here (beyond just the rm -rf), the more likely this can be reproduced and resolved.
Hi the lockdep warning should be fixed in the recent rhel6 kernel. Sorry about that.
My understanding is that this patch has been reverted...
Grant,
If a newer kernel is not yet available, you might want to revert to the previous version until one is and then go straight to that.
Lukas,
If the patch is indeed reverted, then clearly this problem will go away. That aside, where does lockdep come into play? Note the warning reported above:
Quoted 6 lines
WARNING: at fs/dcache.c:758 d_delete+0x260/0x2c0()... is an explicit
WARN_ON()in the code added by this commit. Are we referring to the same issue here?Brian
Hi Brian,
seems like we're not referring to the same issue as I though we're talking about the lockdep problem, but apparently we're not. Can we get a RH bugzilla for this issue so it can be dealt with properly ?
Thanks!
-Lukas
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs