Skip to contentSkip to conversation

kernel crash with damaged XFS

DISCUSSION Michael Monnerie 3 participants 4 messages Nov 6, 2009 – Nov 7, 2009

lore
Share & tools
lore
#0 lore raw

I had that during the last days on my system. It kept on running despite this error. I xfs_repair'ed the fs again and now it's working. This is the same server I talked about with Eric last time, where he improved xfs_repair and that repaired my fs. I had to run xfs_repair several times again until all errors were solved, and now nothing is reported anymore.
The server had no crash or whatever since the last repair, so I wonder where these corruptions come from.

Nov  2 00:01:12 orion.i.zmi.at kernel: Filesystem "dm-0": corrupt inode 3858839593 ((a)extents = 7).  Unmount and run xfs_repair.
Nov  2 00:01:12 orion.i.zmi.at kernel: Pid: 18910, comm: find Tainted: G          2.6.27.29-0.1-xen #1
Nov  2 00:01:12 orion.i.zmi.at kernel:
Nov  2 00:01:12 orion.i.zmi.at kernel: Call Trace:
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff8020c597>] show_trace_log_lvl+0x41/0x58
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff80464a12>] dump_stack+0x69/0x6f
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa033cbdb>] xfs_iformat_extents+0xc9/0x1c4 [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa033d136>] xfs_iformat+0x2b0/0x3f7 [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa033d364>] xfs_iread+0xe7/0x1eb [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa0338910>] xfs_iget_core+0x3a5/0x63a [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa0338c87>] xfs_iget+0xe2/0x187 [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa0352912>] xfs_lookup+0x79/0xa5 [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa035b53f>] xfs_vn_lookup+0x3c/0x78 [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a603b>] real_lookup+0x7e/0x10f
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a612f>] do_lookup+0x63/0xb6
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a87aa>] __link_path_walk+0x9f4/0xe58
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a8dd9>] path_walk+0x5e/0xba
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a8f97>] do_path_lookup+0x162/0x1b9
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a9930>] user_path_at+0x48/0x79
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a1e85>] vfs_lstat_fd+0x15/0x41
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a1f88>] sys_newfstatat+0x22/0x43
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff8020b3b8>] system_call_fastpath+0x16/0x1b
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<00007f265cb0f4de>] 0x7f265cb0f4de

mfg zmi

Signature · 10 lines
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#1 lore raw
Quoted 10 lines · Michael Monnerie

I had that during the last days on my system. It kept on running despite this error. I xfs_repair'ed the fs again and now it's working. This is the same server I talked about with Eric last time, where he improved xfs_repair and that repaired my fs. I had to run xfs_repair several times again until all errors were solved, and now nothing is reported anymore.
The server had no crash or whatever since the last repair, so I wonder where these corruptions come from.

Nov  2 00:01:12 orion.i.zmi.at kernel: Filesystem "dm-0": corrupt inode 3858839593 ((a)extents = 7).  Unmount and run xfs_repair.

Just FWIW this is not a crash, it is XFS properly handling corrruption it encountered. The corruption could be due to a bug in XFS or other code, or a hardware problem. Does repair fix it this time?

- Eric

Quoted 25 lines
Nov  2 00:01:12 orion.i.zmi.at kernel: Pid: 18910, comm: find Tainted: G          2.6.27.29-0.1-xen #1
Nov  2 00:01:12 orion.i.zmi.at kernel:
Nov  2 00:01:12 orion.i.zmi.at kernel: Call Trace:
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff8020c597>] show_trace_log_lvl+0x41/0x58
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff80464a12>] dump_stack+0x69/0x6f
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa033cbdb>] xfs_iformat_extents+0xc9/0x1c4 [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa033d136>] xfs_iformat+0x2b0/0x3f7 [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa033d364>] xfs_iread+0xe7/0x1eb [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa0338910>] xfs_iget_core+0x3a5/0x63a [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa0338c87>] xfs_iget 
+0xe2/0x187 [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa0352912>] xfs_lookup+0x79/0xa5 [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffffa035b53f>] xfs_vn_lookup+0x3c/0x78 [xfs]
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a603b>] real_lookup+0x7e/0x10f
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a612f>] do_lookup+0x63/0xb6
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a87aa>] __link_path_walk+0x9f4/0xe58
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a8dd9>] path_walk+0x5e/0xba
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a8f97>] do_path_lookup+0x162/0x1b9
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a9930>] user_path_at+0x48/0x79
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a1e85>] vfs_lstat_fd+0x15/0x41
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff802a1f88>] sys_newfstatat+0x22/0x43
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<ffffffff8020b3b8>] system_call_fastpath+0x16/0x1b
Nov  2 00:01:12 orion.i.zmi.at kernel:  [<00007f265cb0f4de>] 0x7f265cb0f4de
Quoted 13 lines

mfg zmi

Signature · 10 lines
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
Quoted 10 lines · On Freitag 06 November 2009 Eric Sandeen wrote:
Quoted 10 lines

I had that during the last days on my system. It kept on running despite
this error. I xfs_repair'ed the fs again and now it's working. This is the same server I talked about with Eric last time, where he improved xfs_repair and that repaired my fs. I had to run xfs_repair several times again until all errors were solved, and now nothing is reported anymore.
The server had no crash or whatever since the last repair, so I wonder where these corruptions come from.

Quoted 7 lines
Quoted 2 lines

Nov  2 00:01:12 orion.i.zmi.at kernel: Filesystem "dm-0": corrupt inode 3858839593 ((a)extents = 7).  Unmount and run xfs_repair.

Just FWIW this is not a crash, it is XFS properly handling corrruption   it encountered.  The corruption could be due to a bug in XFS or other code, or a hardware problem.  Does repair fix it this time?

Yes Eric, that's what I wrote but I see it can be misinterpreted as belonging to the old case. I meant the actual problem when saying:

I had to run xfs_repair several times again until all errors were solved, and now nothing is reported anymore. The server had no crash or whatever since the last repair, so I wonder where these corruptions come from.

Seems to be time to make a memory check. I have no other idea where the problem could come from.

mfg zmi

Signature · 10 lines
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw
Quoted 27 lines · Michael Monnerie
Quoted 10 lines · On Freitag 06 November 2009 Eric Sandeen wrote:
Quoted 10 lines

I had that during the last days on my system. It kept on running despite
this error. I xfs_repair'ed the fs again and now it's working. This is the same server I talked about with Eric last time, where he improved xfs_repair and that repaired my fs. I had to run xfs_repair several times again until all errors were solved, and now nothing is reported anymore.
The server had no crash or whatever since the last repair, so I wonder where these corruptions come from.

Quoted 7 lines
Quoted 2 lines

Nov  2 00:01:12 orion.i.zmi.at kernel: Filesystem "dm-0": corrupt inode 3858839593 ((a)extents = 7).  Unmount and run xfs_repair.

Just FWIW this is not a crash, it is XFS properly handling corrruption   it encountered.  The corruption could be due to a bug in XFS or other code, or a hardware problem.  Does repair fix it this time?

Yes Eric, that's what I wrote but I see it can be misinterpreted as belonging to the old case. I meant the actual problem when saying:

I had to run xfs_repair several times again until all errors were solved, and now nothing is reported anymore. The server had no crash or whatever since the last repair, so I wonder where these corruptions come from.

Quoted 16 lines

Seems to be time to make a memory check. I have no other idea where the problem could come from.

mfg zmi

Signature · 10 lines
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

memtest is a good idea
also run a short & long test on each hdd and show smartctl -a of each device to see if any may be failing

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs