Skip to contentSkip to conversation

xfs_repair fails with corrupt dinode 17491441757, extent total = 1, nblocks = 0. This is a bug.

DISCUSSION Arkadiusz Miśkiewicz 2 participants 10 messages Oct 31, 2011 – Nov 6, 2011

lore
Share & tools
lore
#0 lore raw

xfs_repair version 3.1.6

disconnected inode 17491441754, moving to lost+found disconnected inode 17491441755, moving to lost+found disconnected inode 17491441756, moving to lost+found disconnected inode 17491441757, moving to lost+found
corrupt dinode 17491441757, extent total = 1, nblocks = 0. This is a bug. Please capture the filesystem metadata with xfs_metadump and report it to [email protected].
cache_node_purge: refcount was 1, not zero (node=0x21450c90)

fatal error -- 117 - couldn't iget disconnected inode

30GB metadump image, 6.1GB compressed of ~7TB real partition
http://ixion.pld-linux.org/~arekm/lv_storage1.metadump.xz

You need ~8-12GB of memory for xfs_repair on this.

I can also provide ssh access to the system with this image and all needed stuff, so you don't need to download it or waste own resources.

In meantime I'll probably make ugly hack by making "couldn't iget disconnected inode" non fatal, so repair will be able to finish.

Signature · 7 lines
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#1 lore raw
Quoted 14 lines · Arkadiusz Mi??kiewicz

xfs_repair version 3.1.6

disconnected inode 17491441754, moving to lost+found disconnected inode 17491441755, moving to lost+found disconnected inode 17491441756, moving to lost+found disconnected inode 17491441757, moving to lost+found
corrupt dinode 17491441757, extent total = 1, nblocks = 0. This is a bug. Please capture the filesystem metadata with xfs_metadump and report it to [email protected].
cache_node_purge: refcount was 1, not zero (node=0x21450c90)

fatal error -- 117 - couldn't iget disconnected inode

Quoted 7 lines

30GB metadump image, 6.1GB compressed of ~7TB real partition
http://ixion.pld-linux.org/~arekm/lv_storage1.metadump.xz

You need ~8-12GB of memory for xfs_repair on this.

I can also provide ssh access to the system with this image and all needed stuff, so you don't need to download it or waste own resources.

I think I understand the problem - we found a disconnected inode, which we try to move to lost + found. For some reason the inode is found to be incorrect by xfs_iformat, so iget bailds out.

The fix will be to do a pass over the the inodes we want to move to correct such inconsistencies and/or junk them. I'll try to prepare a fix as soon as I get some time, but I'm fairly busy at the moment.

Btw, what did you to to the fs? Having the total blocks out of sync with the numbers in the data and attribute forks seems like an extremly unusal error case.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
Quoted 26 lines · Christoph Hellwig
Quoted 13 lines · Arkadiusz Mi??kiewicz

xfs_repair version 3.1.6

disconnected inode 17491441754, moving to lost+found disconnected inode 17491441755, moving to lost+found disconnected inode 17491441756, moving to lost+found disconnected inode 17491441757, moving to lost+found corrupt dinode 17491441757, extent total = 1, nblocks = 0. This is a bug. Please capture the filesystem metadata with xfs_metadump and report it to [email protected].
cache_node_purge: refcount was 1, not zero (node=0x21450c90)

fatal error -- 117 - couldn't iget disconnected inode

Quoted 7 lines

30GB metadump image, 6.1GB compressed of ~7TB real partition
http://ixion.pld-linux.org/~arekm/lv_storage1.metadump.xz

You need ~8-12GB of memory for xfs_repair on this.

I can also provide ssh access to the system with this image and all needed stuff, so you don't need to download it or waste own resources.

I think I understand the problem - we found a disconnected inode, which we try to move to lost + found. For some reason the inode is found to be incorrect by xfs_iformat, so iget bailds out.

Quoted 7 lines

The fix will be to do a pass over the the inodes we want to move to correct such inconsistencies and/or junk them. I'll try to prepare a fix as soon as I get some time, but I'm fairly busy at the moment.

Btw, what did you to to the fs? Having the total blocks out of sync with the numbers in the data and attribute forks seems like an extremly unusal error case.

Well,

This serwer has 16 various SATA disk connected to art-of-crap controller - Promise SuperTrak EX16350.

The system exhibits funny issues with intel_idle driver
(https://lkml.org/lkml/2011/10/28/270).

It has only 8GB of ram which xfs_repair eats for breakfast causing watchdog to reboot machine while xfs_repair was in progress (would be nice if repair could estimate needed ram before it is too late).

All these issues combined caused few reboots in which some were in middle of xfs_repair. Most likely all that caused such corruption.

Signature · 7 lines
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw
Quoted 9 lines · Arkadiusz Mi??kiewicz

This serwer has 16 various SATA disk connected to art-of-crap controller - Promise SuperTrak EX16350.

The system exhibits funny issues with intel_idle driver
(https://lkml.org/lkml/2011/10/28/270).

It has only 8GB of ram which xfs_repair eats for breakfast causing watchdog to reboot machine while xfs_repair was in progress (would be nice if repair could estimate needed ram before it is too late).

Yes. So far most of the issues are with the internal buffercache and I suspect we could do better sizing decisions there. If you can provide some testing (xfs_repair -n should be enough) I'll happily send you some RFC patches as soon as I get time for it.

In the meantime is there any chance you could send the output of

xfs_repair -n -vv -m 1

for this filesystem?

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Arkadiusz Mi??kiewicz

This serwer has 16 various SATA disk connected to art-of-crap controller
- Promise SuperTrak EX16350.

The system exhibits funny issues with intel_idle driver
(https://lkml.org/lkml/2011/10/28/270).

Quoted 8 lines
Quoted 3 lines

It has only 8GB of ram which xfs_repair eats for breakfast causing watchdog to reboot machine while xfs_repair was in progress (would be nice if repair could estimate needed ram before it is too late).

Yes. So far most of the issues are with the internal buffercache and I suspect we could do better sizing decisions there. If you can provide some testing (xfs_repair -n should be enough) I'll happily send you some RFC patches as soon as I get time for it.

I can do some testing (on image though but that shouldn't matter).

Quoted 6 lines

In the meantime is there any chance you could send the output of

xfs_repair -n -vv -m 1

for this filesystem?

Will such repair done on metadumped & restored image be enough for you?

Signature · 7 lines
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw
Quoted 7 lines · Arkadiusz Mi??kiewicz
Quoted 5 lines

In the meantime is there any chance you could send the output of

xfs_repair -n -vv -m 1

for this filesystem?

Will such repair done on metadumped & restored image be enough for you?

Yes.

I also looked at the memory heuristics and found some fairly obvious flaws. I'll have some test patches for you ASAP.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw

Can you give this patch a quick try on the image in that constrained setup?

libxfs/linux.c +16 -2
Index: xfsprogs-dev/libxfs/linux.c
===================================================================
@@ -207,8 +207,13 @@ platform_nproc(void)
 	return sysconf(_SC_NPROCESSORS_ONLN);
 }
 
+/*
+ * Return the memory that we can use freely.
+ *
+ * The return value is in kilobytes.
+ */
 unsigned long
-platform_physmem(void)
+platform_freemem(void)
 {
 	struct sysinfo  si;
 
@@ -217,5 +222,14 @@ platform_physmem(void)
 			progname);
 		exit(1);
 	}
-	return (si.totalram >> 10) * si.mem_unit;	/* kilobytes */
+
+	/*
+	 * Assume we can use memory that is marked free.  This is a very
+	 * conservative approximation given that there might be a lot of
+	 * pagecache that is easily reclaimable, but the only way to figure
+	 * out pagecache size is by parsing /proc/meminfo, and the format
+	 * of that file keeps changing.  This approach is still better than
+	 * guessing based on si.totalram which might be highly overestimated.
+	 */
+	return (si.freeram >> 10) * si.mem_unit;
 }
include/libxfs.h +1 -1
Index: xfsprogs-dev/include/libxfs.h
===================================================================
@@ -475,7 +475,7 @@ enum ce { CE_DEBUG, CE_CONT, CE_NOTE, CE
 
 #define LIBXFS_BBTOOFF64(bbs)	(((xfs_off_t)(bbs)) << BBSHIFT)
 extern int		libxfs_nproc(void);
-extern unsigned long	libxfs_physmem(void);	/* in kilobytes */
+extern unsigned long	libxfs_freemem(void);	/* in kilobytes */
 
 #include <xfs/xfs_ialloc.h>
 #include <xfs/xfs_rtalloc.h>
libxfs/darwin.c +14 -5
Index: xfsprogs-dev/libxfs/darwin.c
===================================================================
@@ -128,18 +128,27 @@ platform_nproc(void)
 	return ncpu;
 }
 
+/*
+ * Return the memory that we can use freely.
+ *
+ * The return value is in kilobytes.
+ */
 unsigned long
-platform_physmem(void)
+platform_freemem(void)
 {
-	unsigned long	physmem;
-	size_t		len = sizeof(physmem);
+	unsigned long	freemem;
+	size_t		len = sizeof(freemem);
 	static int	mib[2] = {CTL_HW, HW_PHYSMEM};
 
-	if (sysctl(mib, 2, &physmem, &len, NULL, 0) < 0) {
+	if (sysctl(mib, 2, &freemem, &len, NULL, 0) < 0) {
 		fprintf(stderr, _("%s: can't determine memory size\n"),
 			progname);
 		exit(1);
 	}
-	return physmem >> 10;
+
+ 	/*
+	 * Assume we can use approximately 3/4 of the physical memory.
+	 */
+	return (freemem >> (10 + 2)) * 3;
 }
 
libxfs/freebsd.c +14 -5
Index: xfsprogs-dev/libxfs/freebsd.c
===================================================================
@@ -187,17 +187,26 @@ platform_nproc(void)
 	return ncpu;
 }
 
+/*
+ * Return the memory that we can use freely.
+ *
+ * The return value is in kilobytes.
+ */
 unsigned long
-platform_physmem(void)
+platform_freemem(void)
 {
-	unsigned long	physmem;
-	size_t		len = sizeof(physmem);
+	unsigned long	freemem;
+	size_t		len = sizeof(freemem);
 	static int	mib[2] = {CTL_HW, HW_PHYSMEM};
 
-	if (sysctl(mib, 2, &physmem, &len, NULL, 0) < 0) {
+	if (sysctl(mib, 2, &freemem, &len, NULL, 0) < 0) {
 		fprintf(stderr, _("%s: can't determine memory size\n"),
 			progname);
 		exit(1);
 	}
-	return physmem >> 10;
+
+ 	/*
+	 * Assume we can use approximately 3/4 of the physical memory.
+	 */
+	return (freemem >> (10 + 2)) * 3;
 }
libxfs/init.c +2 -2
Index: xfsprogs-dev/libxfs/init.c
===================================================================
@@ -862,7 +862,7 @@ libxfs_nproc(void)
 }
 
 unsigned long
-libxfs_physmem(void)
+libxfs_freemem(void)
 {
-	return platform_physmem();
+	return platform_freemem();
 }
libxfs/init.h +1 -1
Index: xfsprogs-dev/libxfs/init.h
===================================================================
@@ -32,7 +32,7 @@ extern char *platform_findblockpath (cha
 extern int platform_direct_blockdev (void);
 extern int platform_align_blockdev (void);
 extern int platform_nproc(void);
-extern unsigned long platform_physmem(void);	/* in kilobytes */
+extern unsigned long platform_freemem(void);	/* in kilobytes */
 extern int platform_has_uuid;
 
 #endif	/* LIBXFS_INIT_H */
libxfs/irix.c +12 -3
Index: xfsprogs-dev/libxfs/irix.c
===================================================================
@@ -97,8 +97,13 @@ platform_nproc(void)
 	return sysmp(MP_NPROCS);
 }
 
+/*
+ * Return the memory that we can use freely.
+ *
+ * The return value is in kilobytes.
+ */
 unsigned long
-platform_physmem(void)
+platform_freemem(void)
 {
 	struct rminfo ri;
 
@@ -107,5 +112,9 @@ platform_physmem(void)
 			progname);
 		exit(1);
 	}
-	return (ri.physmem >> 10) * getpagesize();	/* kilobytes */
-}
\ No newline at end of file
+
+	/*
+	 * Assume we can use all free memory.
+	 */
+	return (ri.freemem >> 10) * getpagesize();	/* kilobytes */
+}
repair/xfs_repair.c +5 -2
Index: xfsprogs-dev/repair/xfs_repair.c
===================================================================
@@ -631,8 +631,11 @@ main(int argc, char **argv)
 		mem_used = (mp->m_sb.sb_icount >> (10 - 2)) +
 					(mp->m_sb.sb_dblocks >> (10 + 1)) +
 					50000;	/* rough estimate of 50MB overhead */
-		max_mem = max_mem_specified ? max_mem_specified * 1024 :
-						libxfs_physmem() * 3 / 4;
+
+		if (max_mem_specified)
+			max_mem = max_mem_specified * 1024;
+		else
+			max_mem = libxfs_freemem();
 
 		if (getrlimit(RLIMIT_AS, &rlim) != -1 &&
 					rlim.rlim_cur != RLIM_INFINITY) {

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#7 lore raw
Quoted 2 lines · Christoph Hellwig

Can you give this patch a quick try on the image in that constrained setup?

Ate up all 8GB of ram and then was killed:

[217802.127030] Out of memory: Kill process 22898:#0 (xfs_repair) score 784 or 
sacrifice child
[217802.127033] Killed process 22898:#0 (xfs_repair) total-vm:7156064kB, anon-

rss:6652784kB, file-rss:620kB

Now the question is if 8GB should be enough for repairing 7TB filesystem?

Signature · 7 lines
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#8 lore raw
In reply to Christoph Hellwig
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines · Arkadiusz Mi??kiewicz
Quoted 4 lines

In the meantime is there any chance you could send the output of

xfs_repair -n -vv -m 1

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

for this filesystem?

Will such repair done on metadumped & restored image be enough for you?

Yes.

I also looked at the memory heuristics and found some fairly obvious flaws. I'll have some test patches for you ASAP.

# LC_ALL=C ./xfs_repair -vv -m 1 /dev/vg_sys/lv_storage1 Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 97495744, imem = 380842, dblock =
1884752896, dmem = 920289
Required memory for repair is greater that the maximum specified with the -m option. Please increase it to at least 1319.

Signature · 7 lines
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#9 lore raw
In reply to Arkadiusz Miśkiewicz
Quoted 11 lines · Arkadiusz Miśkiewicz

xfs_repair version 3.1.6

disconnected inode 17491441754, moving to lost+found disconnected inode 17491441755, moving to lost+found disconnected inode 17491441756, moving to lost+found disconnected inode 17491441757, moving to lost+found
corrupt dinode 17491441757, extent total = 1, nblocks = 0. This is a bug. Please capture the filesystem metadata with xfs_metadump and report it to [email protected].
cache_node_purge: refcount was 1, not zero (node=0x21450c90)

Quoted 4 lines

fatal error -- 117 - couldn't iget disconnected inode

In meantime I'll probably make ugly hack by making "couldn't iget disconnected inode" non fatal, so repair will be able to finish.

With this one repair finished and then while repairing for second time (this time without hack) I got glibc catching invalid free:

name create failed in ino 17873999459 (117), filesystem may be out of space
bad hash table for directory inode 17875023137 (brak wpisu danych): przebudowano rebuilding directory inode 17875023137
*** glibc detected *** /sbin/xfs_repair: free(): invalid next size (normal): 0x00007f9ffcdc4c00 *** ======= Backtrace: =========
/lib64/libc.so.6(+0x758b6)[0x7fa146a6b8b6] /lib64/libxfs.so.0(libxfs_getbufr+0x81)[0x7fa14738de81] /lib64/libxfs.so.0(cache_node_get+0xca)[0x7fa14738a39a] /lib64/libxfs.so.0(libxfs_getbuf+0x29)[0x7fa14738dfc9]
/sbin/xfs_repair[0x425cd0]
/sbin/xfs_repair[0x426aa6]
/lib64/libpthread.so.0(+0x7ed5)[0x7fa146d8ced5] /lib64/libc.so.6(clone+0x6d)[0x7fa146acfe5d]
======= Memory map: ========

00400000-0043e000 r-xp 00000000 08:12 125854083                          /sbin/xfs_repair
0043f000-00440000 r--p 0003e000 08:12 125854083                          /sbin/xfs_repair
00440000-00441000 rw-p 0003f000 08:12 125854083                          /sbin/xfs_repair
00441000-2aa06000 rw-p 00000000 00:00 0                                  [heap]

7f9f00000000-7f9f01123000 rw-p 00000000 00:00 0
7f9f01123000-7f9f04000000 ---p 00000000 00:00 0
7f9f077ff000-7f9f07800000 ---p 00000000 00:00 0
7f9f07800000-7f9f08000000 rw-p 00000000 00:00 0
[...]

Signature · 7 lines
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs