Skip to contentSkip to conversation

XFS: inode with st_mode == 0

DISCUSSION Jan Kasprzak 10 participants 29 messages Dec 9, 2004 – Jan 28, 2005

lore
Share & tools
lore
#0 lore raw

Hi all,

I have seen the strange problem on our NFS server: yesterday I have found an empty file owned by UID 0/GID 0 and st_mode == 0 in my home directory (ls -l said "?--------- 1 root root 0 <date> <filename>"). The <filename> was correct name of a temporary file used by one of my cron jobs (and the cron job was failing because it could not rewrite the file). It was not possible to write to this file, so I have renamed it
as "badfile" for further investigation (using mv(1) on the NFS server itself).

Today I did "ls -l badfile", and the file had correct owner/group and st_mode as expected from file created by my cron job:

# stat badfile
  File: `badfile'
  Size: 0               Blocks: 0          IO Block: 4096   Regular File
Device: fd00h/64768d    Inode: 356913      Links: 1
Access: (0644/-rw-r--r--)  Uid: (xxxx/     kas)   Gid: (yyyy/ student)
Access: 2004-12-07 21:17:01.000000000 +0100
Modify: 2004-12-07 21:17:01.000000000 +0100

We have seen empty files with uid==gid==0 and st_mode==0 few times before, at places where newly-created regular files were expected, but we have simply deleted them. This time I have renamed the file and looked at it after a day or so.

These files were created over NFS (our home directories are served by NFS server) from various NFS clients (this one was Solaris 8, at least one of the previous instances was created by an IRIX NFS client). The filesystem itself is XFS on a LVM2 volume. The server is HP DL-585 quad opteron, running RHEL3 plus our 2.6.10-rc2 kernel.

Maybe some data is flushed in an incorrect order?

-Yenya

Signature · 3 lines
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
Quoted 2 lines

Whatever the Java applications and desktop dances may lead to, Unix will <
still be pushing the packets around for a quite a while. --Rob Pike <

#1 lore raw
Quoted 9 lines · Jan Kasprzak

Hi all,

I have seen the strange problem on our NFS server: yesterday I have found an empty file owned by UID 0/GID 0 and st_mode == 0 in my home directory (ls -l said "?--------- 1 root root 0 <date> <filename>"). The <filename> was correct name of a temporary file used by one of my cron jobs (and the cron job was failing because it could not rewrite the file). It was not possible to write to this file, so I have renamed it
as "badfile" for further investigation (using mv(1) on the NFS server itself).

Known problem

http://lkml.org/lkml/2004/11/23/283

Seems there is no solution yet

http://lkml.org/lkml/2004/11/30/145

...

Quoted 2 lines

Maybe some data is flushed in an incorrect order?

Maybe :)

Signature · 1 line
 / jakob
#2 lore raw

Jakob Oestergaard wrote:
: > I have seen the strange problem on our NFS server: yesterday I have : > found an empty file owned by UID 0/GID 0 and st_mode == 0 in my home : > directory (ls -l said "?--------- 1 root root 0 <date> <filename>"). : > The <filename> was correct name of a temporary file used by one of my : > cron jobs (and the cron job was failing because it could not rewrite the file). : > It was not possible to write to this file, so I have renamed it
: > as "badfile" for further investigation (using mv(1) on the NFS server itself). :
: Known problem
:
: http://lkml.org/lkml/2004/11/23/283
:
: Seems there is no solution yet
:
: http://lkml.org/lkml/2004/11/30/145
:
Hmm, thanks for the info.

In the meantime, I was able to reproduce this by running the
following script for an hour or so on Solaris 8 NFS client:

#!/bin/bash
FILE="testfile"
set -e
while :
do

        rm -f "$FILE"
        sleep 1
        echo trubka > "$FILE"
        sleep 1
        cat "$FILE">/dev/null

done

After some time I've got this:

emptyfile.sh: testfile: Invalid argument

$ ls -l testfile
?---------   1 root     root           0 Dec  9 14:51 testfile

-Yenya

Signature · 3 lines
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
Quoted 2 lines

Whatever the Java applications and desktop dances may lead to, Unix will <
still be pushing the packets around for a quite a while. --Rob Pike <

#3 lore raw
In reply to Jakob Oestergaard

[sorry, last post got out to far too few people]

Quoted 11 lines · Jakob Oestergaard
Quoted 9 lines · Jan Kasprzak

Hi all,

I have seen the strange problem on our NFS server: yesterday I have found an empty file owned by UID 0/GID 0 and st_mode == 0 in my home directory (ls -l said "?--------- 1 root root 0 <date> <filename>"). The <filename> was correct name of a temporary file used by one of my cron jobs (and the cron job was failing because it could not rewrite the file). It was not possible to write to this file, so I have renamed it
as "badfile" for further investigation (using mv(1) on the NFS server itself).

Quoted 7 lines

Known problem

http://lkml.org/lkml/2004/11/23/283

Seems there is no solution yet

http://lkml.org/lkml/2004/11/30/145

If it's really st_mode I suspect it's a different problem. Can you retry with current oss.sgi.com CVS (or the patch below). Note that this patch breaks xfsdump unfortunately, we're looking into a fix.

Quoted 3 lines
Quoted 1 line

Maybe some data is flushed in an incorrect order?

Maybe :)

No, the problem I've fixed was related to XFS getting the inode version number wrong - or at least different than NFSD expects.

Index: fs/xfs/xfs_ialloc.c
===================================================================
RCS file: /cvs/linux-2.6-xfs/fs/xfs/xfs_ialloc.c,v retrieving revision 1.175

xfs/xfs_ialloc.c +5 -0
diff -u -p -r1.175 xfs_ialloc.c
@@ -270,6 +270,11 @@ xfs_ialloc_ag_alloc(
 	INT_SET(dic.di_magic, ARCH_CONVERT, XFS_DINODE_MAGIC);
 	INT_SET(dic.di_version, ARCH_CONVERT, version);
 
+	/*
+	 * Start at generation 1 because the NFS code uses 0 as wildcard.
+	 */
+	INT_SET(dic.di_gen, ARCH_CONVERT, 1);
+
 	for (j = 0; j < nbufs; j++) {
 		/*
 		 * Get the block.

Index: fs/xfs/xfs_inode.c
===================================================================
RCS file: /cvs/linux-2.6-xfs/fs/xfs/xfs_inode.c,v
retrieving revision 1.406

xfs/xfs_inode.c +18 -2
diff -u -p -r1.406 xfs_inode.c
@@ -1028,6 +1028,16 @@ xfs_iread(
 		ip->i_d.di_projid = 0;
 	}
 
+	/*
+	 * IRIX and older Linux Kernel initialized di_gen to zero when
+	 * creating new inodes, but the NFSD uses i_generation = 0 as
+	 * a wildcard.  We bump di_gen here to avoid that problem for new
+	 * exports, and filehandles created by an older kernel using
+	 * the same filesystem are still valid as the wildcard matches it.
+	 */
+	if (unlikely(ip->i_d.di_gen == 0))
+		ip->i_d.di_gen++;
+
 	ip->i_delayed_blks = 0;
 
 	/*
@@ -2370,11 +2380,17 @@ xfs_ifree(
 		XFS_IFORK_DSIZE(ip) / (uint)sizeof(xfs_bmbt_rec_t);
 	ip->i_d.di_format = XFS_DINODE_FMT_EXTENTS;
 	ip->i_d.di_aformat = XFS_DINODE_FMT_EXTENTS;
+
 	/*
 	 * Bump the generation count so no one will be confused
-	 * by reincarnations of this inode.
+	 * by reincarnations of this inode.  Note that we have to
+	 * skip 0 as that would confuse the NFS server, and we
+	 * have to do it here because the XFS inode might be
+	 * around for a while after deletion (unlike the normal
+	 * Linux inode semantics).
 	 */
-	ip->i_d.di_gen++;
+	if (++ip->i_d.di_gen == 0)
+		ip->i_d.di_gen++;
 	xfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
 
 	if (delete) {
#4 lore raw

On Thu, Dec 09, 2004 at 09:54:14PM +0000, Christoph Hellwig wrote:
...

Quoted 4 lines

If it's really st_mode I suspect it's a different problem. Can you retry with current oss.sgi.com CVS (or the patch below). Note that this patch breaks xfsdump unfortunately, we're looking into a fix.

I'm running plain 2.6.9 plus the patch you sent now.

So far, the server hasn't hosed itself (and it's been running for almost half an hour - woohoo)

Now I can check out CVS trees without getting files with weird ownership and undeletable directories (which as *extremely* common before - in fact, earlier today before I applied the patch, I was unable to complete a cvs checkout of a moderately large software project).

I guess this is a thumbs up for your patch so far :)

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Maybe some data is flushed in an incorrect order?

Maybe :)

No, the problem I've fixed was related to XFS getting the inode version number wrong - or at least different than NFSD expects.

I have seen problems that your patch probably won't fix (like undeletable directories randomly occuring on ext3 filesystems as well - seems that once the dcache gets confused, it *really* gets confused).

Let's see how the box fares with your patch - I'll report back on new bugs as I find them.

Signature · 1 line
 / jakob
#5 lore raw
In reply to Christoph Hellwig

Christoph Hellwig wrote:
: > On Thu, Dec 09, 2004 at 01:59:18PM +0100, Jan Kasprzak wrote:
: > > I have seen the strange problem on our NFS server: yesterday I have : > > found an empty file owned by UID 0/GID 0 and st_mode == 0 in my home : > > directory (ls -l said "?--------- 1 root root 0 <date> <filename>"). [...]
: If it's really st_mode I suspect it's a different problem. Can you retry : with current oss.sgi.com CVS (or the patch below). Note that this patch : breaks xfsdump unfortunately, we're looking into a fix.
:
: > > Maybe some data is flushed in an incorrect order?
: >
: > Maybe :)
:
: No, the problem I've fixed was related to XFS getting the inode version : number wrong - or at least different than NFSD expects.
:
We have applied these two patches to 2.6.10-rc2, but this
does not help. A few minutes ago I've got the "?----------" file again from my test script. This time it took >4 hours (it was about an hour or so without this patch).

-Yenya

Signature · 3 lines
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
Quoted 2 lines

Whatever the Java applications and desktop dances may lead to, Unix will <
still be pushing the packets around for a quite a while. --Rob Pike <

#6 lore raw

On Tue, Dec 21, 2004 at 07:43:04PM +0100, Jan Kasprzak wrote:
...

Quoted 8 lines

:
: No, the problem I've fixed was related to XFS getting the inode version : number wrong - or at least different than NFSD expects.
:
We have applied these two patches to 2.6.10-rc2, but this
does not help. A few minutes ago I've got the "?----------" file again from my test script. This time it took >4 hours (it was about an hour or so without this patch).

I run the patch on 2.6.9 - it solved the problem in the common case. Before the patch, I was unable to complete a "cvs checkout" of a moderately large tree - would end up with undeletable directories and lots of other weird things... After the patch, I can run cvs checkout.

However, we still see the problem - so the patch does not solve this completely, as you have observed as well.

Our most common situation is that a new file gets created as a symlink pointing to itself, instead of as a regular file.

It also happens regularly, that a command (be it cvs, etags, ld or something else) reports EINVAL when attempting to create/write a file.

So, status on my side is; things still suck, but they suck less than on vanilla 2.6.9

Signature · 1 line
 / jakob
#7 lore raw
Quoted 19 lines · Jakob Oestergaard

On Tue, Dec 21, 2004 at 07:43:04PM +0100, Jan Kasprzak wrote:
...

Quoted 8 lines

:
: No, the problem I've fixed was related to XFS getting the inode version : number wrong - or at least different than NFSD expects.
:
We have applied these two patches to 2.6.10-rc2, but this
does not help. A few minutes ago I've got the "?----------" file again from my test script. This time it took >4 hours (it was about an hour or so without this patch).

I run the patch on 2.6.9 - it solved the problem in the common case. Before the patch, I was unable to complete a "cvs checkout" of a moderately large tree - would end up with undeletable directories and lots of other weird things... After the patch, I can run cvs checkout.

However, we still see the problem - so the patch does not solve this completely, as you have observed as well.

Quoted 8 lines

Our most common situation is that a new file gets created as a symlink pointing to itself, instead of as a regular file.

It also happens regularly, that a command (be it cvs, etags, ld or something else) reports EINVAL when attempting to create/write a file.

So, status on my side is; things still suck, but they suck less than on vanilla 2.6.9

I have a better patch than the one I gave you (attached below). If you send me a mail with steps to reproduce your remaining problems I'll put this very high on my TODO list after christmas. Btw, any chance you could try XFS CVS (which is at 2.6.9) + the patch below instead of plain 2.6.9, there have been various other fixes in the last months.

Index: fs/xfs/xfs_vfsops.c
===================================================================
RCS file: /cvs/linux-2.6-xfs/fs/xfs/xfs_vfsops.c,v retrieving revision 1.459

xfs/xfs_vfsops.c +2 -2
diff -u -p -r1.459 xfs_vfsops.c
@@ -1581,7 +1581,7 @@ xfs_syncsub(
 }
 
 /*
- * xfs_vget - called by DMAPI to get vnode from file handle
+ * xfs_vget - called by DMAPI and NFSD to get vnode from file handle
  */
 STATIC int
 xfs_vget(
@@ -1623,7 +1623,7 @@ xfs_vget(
 		return XFS_ERROR(EIO);
 	}
 
-	if (ip->i_d.di_mode == 0 || (igen && (ip->i_d.di_gen != igen))) {
+	if (ip->i_d.di_mode == 0 || ip->i_d.di_gen != igen) {
 		xfs_iput_new(ip, XFS_ILOCK_SHARED);
 		*vpp = NULL;
 		return XFS_ERROR(ENOENT);

Index: fs/xfs/linux-2.6/xfs_super.c
===================================================================
RCS file: /cvs/linux-2.6-xfs/fs/xfs/linux-2.6/xfs_super.c,v retrieving revision 1.321

xfs/linux-2.6/xfs_super.c +34 -0
diff -u -p -r1.321 xfs_super.c
@@ -731,6 +731,39 @@ linvfs_get_dentry(
 	return result;
 }
 
+STATIC struct dentry *
+linvfs_decode_fh(
+	struct super_block	*sb,
+	__u32			*fh,
+	int			fh_len,
+	int			fileid_type,
+	int (*acceptable)(
+		void		*context,
+		struct dentry	*de),
+	void			*context)
+{
+	__u32 parent[2];
+	parent[0] = parent[1] = 0;
+	
+	if (fh_len < 2 || fileid_type > 2)
+		return NULL;
+	
+	if (fileid_type == 2 && fh_len > 2) {
+		if (fh_len == 3) {
+			printk(KERN_WARNING
+			       "XFS: detected filehandle without "
+			       "parent inode generation information.");
+			return ERR_PTR(-ESTALE);
+		}
+			
+		parent[0] = fh[2];
+		parent[1] = fh[3];
+	}
+	
+	return find_exported_dentry(sb, fh, parent, acceptable, context);
+
+}
+
 STATIC int
 linvfs_show_options(
 	struct seq_file		*m,
@@ -893,6 +926,7 @@ linvfs_get_sb(
 
 
 STATIC struct export_operations linvfs_export_ops = {
+	.decode_fh		= linvfs_decode_fh,
 	.get_parent		= linvfs_get_parent,
 	.get_dentry		= linvfs_get_dentry,
 };

Index: include/linux/fs.h
===================================================================
RCS file: /cvs/linux-2.6-xfs/include/linux/fs.h,v
retrieving revision 1.11

linux/fs.h +4 -0
diff -u -p -r1.11 fs.h
@@ -1115,6 +1115,10 @@ struct export_operations {
 
 };
 
+extern struct dentry *
+find_exported_dentry(struct super_block *sb, void *obj, void *parent,
+		     int (*acceptable)(void *context, struct dentry *de),
+		     void *context);
 
 struct file_system_type {
 	const char *name;
#8 lore raw

On Wed, Dec 22, 2004 at 06:23:44PM +0000, Christoph Hellwig wrote:
...

Quoted 5 lines

I have a better patch than the one I gave you (attached below). If you send me a mail with steps to reproduce your remaining problems I'll put this very high on my TODO list after christmas. Btw, any chance you could try XFS CVS (which is at 2.6.9) + the patch below instead of plain 2.6.9, there have been various other fixes in the last months.

Thank you Christoph,

I will see what I can do - my schedule is a little tight these days, but I'll try it out at the first given opportunity (which will definitely be after christmas but hopefully before new year).

Signature · 1 line
 / jakob
#9 lore raw
In reply to Christoph Hellwig

On Wed, Dec 22, 2004 at 06:23:44PM +0000, Christoph Hellwig wrote:
...

Quoted 5 lines

I have a better patch than the one I gave you (attached below). If you send me a mail with steps to reproduce your remaining problems I'll put this very high on my TODO list after christmas. Btw, any chance you could try XFS CVS (which is at 2.6.9) + the patch below instead of plain 2.6.9, there have been various other fixes in the last months.

I have been on XFS CVS + the patch you sent for five days now.

Summary:

XFS and related dcache problems (seen in ext3 too) with files that get
created as symlinks to themselves or undeletable directories, or just
get plain wrong permissions or ownership: These problems seem to have
gone away completely. Very very nice!

I still get NFS stale handle problems (the weird ones that can be
worked around at times by running a ls -l on the server-side).

I talked Anders ([email protected]) into testing the same kernel on
another system he's running, where his main prolem were the stale handles. He too now only sees the weird stale handle problems.

So, it looks like the current 2.6 status on file serving is:
*) Can be done with XFS+NFS+SMP iff running a SGI kernel *) Will see weird NFS stale handle problems no matter which kernel
is used

Which is a lot better than the previous "2.6 will eat your files and
make whatever remains owned by guest" situation :)

Good work!

(Any suggestions on the knfsd issue with stale handles?)

Signature · 1 line
 / jakob
#10 lore raw
Quoted 1 line · Jakob Oestergaard

(Any suggestions on the knfsd issue with stale handles?)

I'd suggest asking [email protected] for help, tell them your exact setup - OS rev on client/server, mount options /etc/exports options, etc..

I'm a bit lost in NFS deep magic land still :)

#11 lore raw
In reply to Christoph Hellwig
Quoted 6 lines · Christoph Hellwig
Quoted 6 lines
Quoted 6 lines

We have applied these two patches to 2.6.10-rc2, but this
does not help. A few minutes ago I've got the "?----------" file again from my test script. This time it took >4 hours (it was about an hour or so without this patch).

I'm seeing this problem occasionally too.

I'm running 2.6.10

The patch you provided below can be made to apply but I get:
LD .tmp_vmlinux1
fs/built-in.o(.text+0xa0ed3): In function `linvfs_decode_fh':
: undefined reference to `find_exported_dentry'

I assume that since you say:

Quoted 5 lines

Btw, any chance you could
try XFS CVS (which is at 2.6.9) + the patch below instead of plain 2.6.9, there have been various other fixes in the last months.

That not all the changes in XFS CVS have made it to 2.6.10?

Is there a 2.6.10 patch that I could apply? Or do you have any other suggestions.

David

Quoted 107 lines

Index: fs/xfs/xfs_vfsops.c
===================================================================
RCS file: /cvs/linux-2.6-xfs/fs/xfs/xfs_vfsops.c,v retrieving revision 1.459
diff -u -p -r1.459 xfs_vfsops.c

xfs/xfs_vfsops.c +1 -1
@@ -1581,7 +1581,7 @@ xfs_syncsub(
 }
 
 /*
- * xfs_vget - called by DMAPI to get vnode from file handle
+ * xfs_vget - called by DMAPI and NFSD to get vnode from file handle
 */

STATIC int
xfs_vget(

xfs/xfs_vfsops.c +1 -1
@@ -1623,7 +1623,7 @@ xfs_vget(
 		return XFS_ERROR(EIO);
 	}
 
-	if (ip->i_d.di_mode == 0 || (igen && (ip->i_d.di_gen != igen))) {
+	if (ip->i_d.di_mode == 0 || ip->i_d.di_gen != igen) {
 		xfs_iput_new(ip, XFS_ILOCK_SHARED);
 		*vpp = NULL;
 		return XFS_ERROR(ENOENT);

Index: fs/xfs/linux-2.6/xfs_super.c
===================================================================
RCS file: /cvs/linux-2.6-xfs/fs/xfs/linux-2.6/xfs_super.c,v retrieving revision 1.321
diff -u -p -r1.321 xfs_super.c

xfs/linux-2.6/xfs_super.c +33 -0
@@ -731,6 +731,39 @@ linvfs_get_dentry(
 	return result;
 }
 
+STATIC struct dentry *
+linvfs_decode_fh(
+	struct super_block	*sb,
+	__u32			*fh,
+	int			fh_len,
+	int			fileid_type,
+	int (*acceptable)(
+		void		*context,
+		struct dentry	*de),
+	void			*context)
+{
+	__u32 parent[2];
+	parent[0] = parent[1] = 0;
+	
+	if (fh_len < 2 || fileid_type > 2)
+		return NULL;
+	
+	if (fileid_type == 2 && fh_len > 2) {
+		if (fh_len == 3) {
+			printk(KERN_WARNING
+			       "XFS: detected filehandle without "
+			       "parent inode generation information.");
+			return ERR_PTR(-ESTALE);
+		}
+			
+		parent[0] = fh[2];
+		parent[1] = fh[3];
+	}
+	
+	return find_exported_dentry(sb, fh, parent, acceptable, context);
+
+}
+

STATIC int
linvfs_show_options(
struct seq_file *m,

xfs/linux-2.6/xfs_super.c +1 -0
@@ -893,6 +926,7 @@ linvfs_get_sb(
 
 
 STATIC struct export_operations linvfs_export_ops = {
+	.decode_fh		= linvfs_decode_fh,
 	.get_parent		= linvfs_get_parent,
 	.get_dentry		= linvfs_get_dentry,
 };

Index: include/linux/fs.h
===================================================================
RCS file: /cvs/linux-2.6-xfs/include/linux/fs.h,v
retrieving revision 1.11
diff -u -p -r1.11 fs.h

linux/fs.h +4 -1
@@ -1115,6 +1115,10 @@ struct export_operations {
 
 };
 
+extern struct dentry *
+find_exported_dentry(struct super_block *sb, void *obj, void *parent,
+		     int (*acceptable)(void *context, struct dentry *de),
+		     void *context);
 
 struct file_system_type {
 	const char *name;
-

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#12 lore raw
Quoted 1 line · David Greaves

...

Quoted 8 lines
Quoted 4 lines

try XFS CVS (which is at 2.6.9) + the patch below instead of plain 2.6.9, there have been various other fixes in the last months.

That not all the changes in XFS CVS have made it to 2.6.10?

Is there a 2.6.10 patch that I could apply? Or do you have any other suggestions.

AFAIK the best you can do is to get the most recent XFS kernel from SGI's CVS (this one is based on 2.6.10).

If you run that kernel, then most of the former problems will be gone; *) I only have one undeletable directory on my system - so it seems that
this error is no longer common ;)
*) 2.6.10 apparently fixes the knfsd stale handle problem
*) I no longer see the weird directory/file/??? mode problems or random ownership assignment

So apart from the general well known instability problems that will occur when you actually start *using* the system, there should be no
major problems running XFS+SMP+NFS on SGI's 2.6.10 based kernel ;)

Signature · 1 line
 / jakob
#13 lore raw
Quoted 5 lines · Jakob Oestergaard
Quoted 2 lines

Is there a 2.6.10 patch that I could apply? Or do you have any other suggestions.

AFAIK the best you can do is to get the most recent XFS kernel from SGI's CVS (this one is based on 2.6.10).

The -mm tree also has these fixes; we'll get them merged into mainline soon.

Quoted 3 lines

If you run that kernel, then most of the former problems will be gone; *) I only have one undeletable directory on my system - so it seems that
this error is no longer common ;)

You may need to run xfs_repair to clean that up..? Or does the problem persist after a repair?

cheers.

Signature · 1 line
Nathan
#14 lore raw

On Sat, Jan 15, 2005 at 01:09:08PM +1100, Nathan Scott wrote:
...

Quoted 5 lines
Quoted 2 lines

AFAIK the best you can do is to get the most recent XFS kernel from SGI's CVS (this one is based on 2.6.10).

The -mm tree also has these fixes; we'll get them merged into mainline soon.

Okeydokey - good

Quoted 7 lines
Quoted 4 lines

If you run that kernel, then most of the former problems will be gone; *) I only have one undeletable directory on my system - so it seems that
this error is no longer common ;)

You may need to run xfs_repair to clean that up..? Or does the problem persist after a repair?

I'm running Debian Woody - the xfs_check/xfs_repair there didn't seem to find anything last I tried. I have not re-checked for this last problem though.

I figured I might need to run the CVS version of xfs tools, and, well, me being busy and all, I thought I'd just leave the 'delete_me' directory hanging until some time I got more time on my hands ;)

Signature · 1 line
 / jakob
#15 lore raw
In reply to Jakob Oestergaard
Quoted 2 lines · Jakob Oestergaard

So apart from the general well known instability problems that will occur when you actually start *using* the system, there should be no

What known instabilities?

#16 lore raw
Quoted 5 lines · Christoph Hellwig
Quoted 2 lines · Jakob Oestergaard

So apart from the general well known instability problems that will occur when you actually start *using* the system, there should be no

What known instabilities?

Where should I begin? ;)

Most of the following have already been posted to LKML - primarily by Anders ([email protected]) - it seems that noone cares, but I'll repost a summary that Anders sent me below:

-------
Scenario 1: Mailservers:
2.6.10 (~24-40 hours uptime):
Running ext3 on mailqueue:

<SNIP>

Unable to handle kernel NULL pointer dereference at virtual address 00000004
printing eip:
c018a095
*pde = 00000000
Oops: 0002 [#1]
SMP
Modules linked in: nfs e1000 iptable_nat ipt_connlimit rtc
CPU:    2
EIP:    0060:[<c018a095>]    Not tainted
EFLAGS: 00010286   (2.6.8.1)

EIP is at journal_commit_transaction+0x535/0x10e5
eax: cac1e26c ebx: 00000000 ecx: f7cec400 edx: f7cec400
esi: f65f3000 edi: cac1e26c ebp: f65f3000 esp: f65f3dc0
ds: 007b es: 007b ss: 0068
Process kjournald (pid: 174, threadinfo=f65f3000 task=c2308b70)

Stack: f65f3e64 00000000 00000000 00000000 00000000 00000000 f7cec400 cda565fc
       0000149a 00000004 f65f3e48 c01132d8 00000002 c202ad20 00000001 f65f3e5c
       c202ad20 c202ad20 00000002 00000001 0000001e 01c1af60 f65f3e68 c0407dc0
Call Trace:
 [<c01132d8>] scheduler_tick+0x468/0x470
 [<c01127b5>] find_busiest_group+0x105/0x310
 [<c011db8e>] del_timer_sync+0x7e/0xa0
 [<c018cd4d>] kjournald+0xbd/0x230
 [<c0114b10>] autoremove_wake_function+0x0/0x40
 [<c0114b10>] autoremove_wake_function+0x0/0x40
 [<c0103f16>] ret_from_fork+0x6/0x14
 [<c018cc70>] commit_timeout+0x0/0x10
 [<c018cc90>] kjournald+0x0/0x230
 [<c01024bd>] kernel_thread_helper+0x5/0x18
Code: f0 ff 43 04 8b 03 83 e0 04 74 4c 8b 8c 24 b8 01 00 00 c6 81
 <2>SoftDog: Initiating system reboot

</SNIP>

-------
Scenario 2: Mailservers:
Running XFS on mailqueue:

<SNIP>
Filesystem "sdb1": xfs_trans_delete_ail: attempting to delete a log item that is not in the AIL
xfs_force_shutdown(sdb1,0x8) called from line 382 of file fs/xfs/xfs_trans_ail.c. Return address = 0xc0216a56
@Linux version 2.6.9 ([email protected]) (gcc version 2.96 20000731 (Red Hat Linux 7.3 2.96-113)) #1 SMP Tue Oct 19 16:04:55 CEST 2004 </SNIP>

=======
Resolution to the mailserver problem:
2.4.28 is perfectly stable on these machines.

-------
Scenario 3: Webservers:

2.6.10, 2.6.10-ac8 (~3-12 hours uptime):

    <SNIP>
    Unable to handle kernel paging request
    <2>SoftDog: Initiating system reboot.
    <SNIP>
    (No more...) :(

=======
Resolution to the webserver problem:
2.4.28/2.4.29-rc2 are stable here

-------
Scenario 4: Storageservers:
2.6.8.1:
Oopses after ~5-10 hours whith SMP on. - Cannot find the actual Oopses
anymore and 2.6.8+ havent been tested as we cannot afford anymore downtime on these servers.

=======
Resolution to the storage server problem:
2.6.8.1 UP is stable (but oopses regularly after memory allocation failures)

Hardware on all servers: IBM x335 and x345.

Mentioned errors seen on a total of 17 servers.

Signature · 1 line
 / jakob
#17 lore raw
Quoted 2 lines · Jakob Oestergaard

Where should I begin? ;)

Guess we've been struggeling with much of the same problems..

Quoted 3 lines

-------
Scenario 2: Mailservers:
Running XFS on mailqueue:

The 2.6.10-1.737_FC3 + 's/posix_lock_file/posix_lock_file_wait/' on fs/nfs/file.c seems stable on our mailserver running XFS on mail queue and spool (mbox). 4 days of uptime!

Quoted 5 lines

=======
Resolution to the storage server problem:
2.6.8.1 UP is stable (but oopses regularly after memory allocation failures)

My XFS-fileserver ran 2.6.9-rc3 stable since october 25. Got lots of "possible deadlock in kmem_alloc (mode:0xd0)" this weekend, so I upgraded to plain 2.6.10. Seems OK so far.

Quoted 2 lines

Hardware on all servers: IBM x335 and x345.

Mail servers: Dell 2650, IBM ServeRAID 6M, EXP400. File servers: IBM x330, qla2300, infortrend eonstor.

All running Whitebox/centos RHEL clone.

-jf

#18 lore raw

Hi,

Quoted 2 lines · Jan-Frode Myklebust

Guess we've been struggeling with much of the same problems..

Seems like it. :)

Quoted 7 lines
Quoted 3 lines

-------
Scenario 2: Mailservers:
Running XFS on mailqueue:

The 2.6.10-1.737_FC3 + 's/posix_lock_file/posix_lock_file_wait/' on fs/nfs/file.c seems stable on our mailserver running XFS on mail queue and spool (mbox). 4 days of uptime!

Yes - We had those errors to:

"Kernel panic - not syncing: Attempting to free lock with active block list"

- on 2.6.10 on the webservers, which was fixed with that particular patch. But this is a different error as our mailservers dont't act as NFS clients. All use local XFS.

Sad thing is that the mailservers crashes every 10-20 hours on 2.6.x, but I'm not able to reproduce it in a test environment, and at time of original post to LKML noone was able to do anything about it without a reproduceable testcase. :(

Quoted 9 lines
Quoted 4 lines

=======
Resolution to the storage server problem:
2.6.8.1 UP is stable (but oopses regularly after memory allocation failures)

My XFS-fileserver ran 2.6.9-rc3 stable since october 25. Got lots of "possible deadlock in kmem_alloc (mode:0xd0)" this weekend, so I upgraded to plain 2.6.10. Seems OK so far.

OK, as far as i remember, we had the same messages in the kernel log when running with SMP.

Signature · 9 lines
Med venlig hilsen - Best regards - Meilleures salutations

Anders Saaby
Systems Engineer
------------------------------------------------
Cohaesio A/S - Maglebjergvej 5D - DK-2800 Lyngby
Phone: +45 45 880 888 - Fax: +45 45 880 777
Mail: [email protected] - http://www.cohaesio.com
------------------------------------------------
#19 lore raw

journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

In reply to Jakob Oestergaard

For more of this look up subjects:
Bad things happening to journaled filesystem machines Oops in kjournald
and from author:
Anders Saaby

I also can't keep a recent 2.6 or 2.6*-ac* kernel up more than a few hours on a machine under real load. Perhaps us folks with the problem need to talk to the powers who be to come up with a strategy to make a report they can use. My guess is we're not sending something that can be used.

Signature · 1 line
jeffrey hundstad
Quoted 110 lines · Jakob Oestergaard
Quoted 10 lines · Christoph Hellwig
Quoted 4 lines · Jakob Oestergaard

So apart from the general well known instability problems that will occur when you actually start *using* the system, there should be no

What known instabilities?

Where should I begin? ;)

Most of the following have already been posted to LKML - primarily by Anders ([email protected]) - it seems that noone cares, but I'll repost a summary that Anders sent me below:

-------
Scenario 1: Mailservers:
2.6.10 (~24-40 hours uptime):
Running ext3 on mailqueue:

<SNIP>

Unable to handle kernel NULL pointer dereference at virtual address 00000004
printing eip:
c018a095
*pde = 00000000
Oops: 0002 [#1]
SMP
Modules linked in: nfs e1000 iptable_nat ipt_connlimit rtc
CPU:    2
EIP:    0060:[<c018a095>]    Not tainted
EFLAGS: 00010286   (2.6.8.1)

EIP is at journal_commit_transaction+0x535/0x10e5
eax: cac1e26c ebx: 00000000 ecx: f7cec400 edx: f7cec400
esi: f65f3000 edi: cac1e26c ebp: f65f3000 esp: f65f3dc0
ds: 007b es: 007b ss: 0068
Process kjournald (pid: 174, threadinfo=f65f3000 task=c2308b70)

Stack: f65f3e64 00000000 00000000 00000000 00000000 00000000 f7cec400 cda565fc
      0000149a 00000004 f65f3e48 c01132d8 00000002 c202ad20 00000001 f65f3e5c
      c202ad20 c202ad20 00000002 00000001 0000001e 01c1af60 f65f3e68 c0407dc0
Call Trace:
[<c01132d8>] scheduler_tick+0x468/0x470
[<c01127b5>] find_busiest_group+0x105/0x310
[<c011db8e>] del_timer_sync+0x7e/0xa0
[<c018cd4d>] kjournald+0xbd/0x230
[<c0114b10>] autoremove_wake_function+0x0/0x40
[<c0114b10>] autoremove_wake_function+0x0/0x40
[<c0103f16>] ret_from_fork+0x6/0x14
[<c018cc70>] commit_timeout+0x0/0x10
[<c018cc90>] kjournald+0x0/0x230
[<c01024bd>] kernel_thread_helper+0x5/0x18
Code: f0 ff 43 04 8b 03 83 e0 04 74 4c 8b 8c 24 b8 01 00 00 c6 81

<2>SoftDog: Initiating system reboot
</SNIP>

-------
Scenario 2: Mailservers:
Running XFS on mailqueue:

<SNIP>
Filesystem "sdb1": xfs_trans_delete_ail: attempting to delete a log item that is not in the AIL xfs_force_shutdown(sdb1,0x8) called from line 382 of file fs/xfs/xfs_trans_ail.c. Return address = 0xc0216a56 @Linux version 2.6.9 ([email protected]) (gcc version 2.96 20000731 (Red
Hat Linux 7.3 2.96-113)) #1 SMP Tue Oct 19 16:04:55 CEST 2004
</SNIP>

=======
Resolution to the mailserver problem:
2.4.28 is perfectly stable on these machines.

-------
Scenario 3: Webservers:

2.6.10, 2.6.10-ac8 (~3-12 hours uptime):

   <SNIP>
   Unable to handle kernel paging request
   <2>SoftDog: Initiating system reboot.
   <SNIP>
   (No more...) :(

=======
Resolution to the webserver problem:
2.4.28/2.4.29-rc2 are stable here

-------
Scenario 4: Storageservers:
2.6.8.1:
Oopses after ~5-10 hours whith SMP on. - Cannot find the actual Oopses
anymore and 2.6.8+ havent been tested as we cannot afford anymore downtime on these servers.

=======
Resolution to the storage server problem:
2.6.8.1 UP is stable (but oopses regularly after memory allocation failures)

Hardware on all servers: IBM x335 and x345.

Mentioned errors seen on a total of 17 servers.

#20 lore raw

Re: journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

Quoted 5 lines · Jeffrey Hundstad

I also can't keep a recent 2.6 or 2.6*-ac* kernel up more than a few
hours on a machine under real load. Perhaps us folks with the problem
need to talk to the powers who be to come up with a strategy to make a report they can use. My guess is we're not sending something that can be used.

I need a way to reproduce it. Preferably on a hardware configuration that is running 2.6.10-ac10 or later because of the bio and acpi fixes. I'm not interested in any report including binary drivers and to be honest the least complex configuration the better. I also care that the hardware passes memtest86+ !

I also don't care about XFS although Christoph may well do.

Alan

#21 lore raw

Re: journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

In reply to Jeffrey Hundstad
Quoted 8 lines · Jeffrey Hundstad

For more of this look up subjects:
Bad things happening to journaled filesystem machines Oops in kjournald
and from author:
Anders Saaby

I also can't keep a recent 2.6 or 2.6*-ac* kernel up more than a few hours on a machine under real load. Perhaps us folks with the problem need to talk to the powers who be to come up with a strategy to make a report they can use. My guess is we're not sending something that can be used.

I have found two server in my operation that seem to do quite well on linux-2.6.7. So I believe the brokenness is after this point and before linux-2.6.8.1.

...so far I'm not seeing problems after two days with linux-2.6.10-ac10. I'm still crossing my fingers and knocking on wood.

Signature · 1 line
jeffrey hundstad
#22 lore raw

Re: journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

In reply to Jeffrey Hundstad

Hi,

Quoted 3 lines · Jeffrey Hundstad

For more of this look up subjects:
Bad things happening to journaled filesystem machines Oops in kjournald

That seems to have been due to the xattr problems recently fixed in Linus's tree. The xattr race was allowing one process to delete an unshared xattr block while another was trying to share it, and the journaling code was getting upset when the second process then tried to commit the now-deleted block.

--Stephen

#23 lore raw

Re: journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

Quoted 11 lines · Stephen C. Tweedie

Hi,

Quoted 5 lines · Jeffrey Hundstad

For more of this look up subjects:
Bad things happening to journaled filesystem machines Oops in kjournald

Quoted 7 lines

That seems to have been due to the xattr problems recently fixed in Linus's tree. The xattr race was allowing one process to delete an unshared xattr block while another was trying to share it, and the journaling code was getting upset when the second process then tried to commit the now-deleted block.

Thanks for the update.

I wonder if there are several problems. Alan Cox claimed that there was a fix in linux-2.6.10-ac10 that might alleviate the problem.

On linux-2.6.10-ac10 I've got one machine that's been up for 6 days now that would never last more then 1 before. On the other hand I have one machine that did die after two days.

Does linux-2.6.11-rc2 have both the linux-2.6.10-ac10 fix and the xattr problem fixed? If so, I'll test there.

Signature · 1 line
Jeffrey Hundstad
#24 lore raw

Re: journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

Hi,

Quoted 5 lines · Jeffrey Hundstad
Quoted 2 lines
Quoted 2 lines

Bad things happening to journaled filesystem machines Oops in kjournald

I wonder if there are several problems. Alan Cox claimed that there was a fix in linux-2.6.10-ac10 that might alleviate the problem.

I'm not sure --- there are a couple of bio/bh-related fixes in that patch, but nothing against jbd/ext3 itself.

Quoted 2 lines

Does linux-2.6.11-rc2 have both the linux-2.6.10-ac10 fix and the xattr problem fixed?

Not sure about how much of -ac went in, but it has the xattr fix.

--Stephen

#25 lore raw

Re: journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

Quoted 22 lines · Stephen C. Tweedie

Hi,

Quoted 10 lines · Jeffrey Hundstad
Quoted 4 lines
Quoted 4 lines

Bad things happening to journaled filesystem machines Oops in kjournald

I wonder if there are several problems. Alan Cox claimed that there was a fix in linux-2.6.10-ac10 that might alleviate the problem.

I'm not sure --- there are a couple of bio/bh-related fixes in that patch, but nothing against jbd/ext3 itself.

Quoted 10 lines
Quoted 3 lines

Does linux-2.6.11-rc2 have both the linux-2.6.10-ac10 fix and the xattr problem fixed?

Not sure about how much of -ac went in, but it has the xattr fix.

--Stephen

I've had my machine that would crash daily if not hourly stay up for 10 days now. This is with the linux-2.6.10-ac10 kernel. I was wondering if anyone else is having similiar results.

#26 lore raw

Re: journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

Hi,

Quoted 7 lines · Jeffrey E. Hundstad
Quoted 4 lines
Quoted 2 lines

Does linux-2.6.11-rc2 have both the linux-2.6.10-ac10 fix and the xattr problem fixed?

Not sure about how much of -ac went in, but it has the xattr fix.

I've had my machine that would crash daily if not hourly stay up for 10 days now. This is with the linux-2.6.10-ac10 kernel.

Good to know. Are you using xattrs extensively (eg. for ACLs, SELinux or Samba 4)?

--Stephen

#27 lore raw

Re: journaled filesystems -- known instability; Was: XFS: inode with st_mode == 0

Quoted 18 lines · Stephen C. Tweedie

Hi,

Quoted 9 lines · Jeffrey E. Hundstad
Quoted 9 lines
Quoted 3 lines

Does linux-2.6.11-rc2 have both the linux-2.6.10-ac10 fix and the xattr problem fixed?

Not sure about how much of -ac went in, but it has the xattr fix.

Quoted 11 lines
Quoted 3 lines

I've had my machine that would crash daily if not hourly stay up for 10 days now. This is with the linux-2.6.10-ac10 kernel.

Good to know. Are you using xattrs extensively (eg. for ACLs, SELinux or Samba 4)?

--Stephen

On the machines that were having problems we really weren't using them for anything. I think I may have been running into the BIO problem that was fixed in 2.6.10-ac10.

#28 lore raw
In reply to Christoph Hellwig

Christoph Hellwig wrote:
: I have a better patch than the one I gave you (attached below). If you : send me a mail with steps to reproduce your remaining problems I'll put : this very high on my TODO list after christmas. Btw, any chance you could : try XFS CVS (which is at 2.6.9) + the patch below instead of plain 2.6.9, : there have been various other fixes in the last months.
:
Just FWIW, this patch (applied to 2.6.10) seems to fix the problem
for me. I was not able to reproduce it by running my test script for ~24 hours.

Thanks!

-Yenya

Signature · 3 lines
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
Quoted 2 lines

Whatever the Java applications and desktop dances may lead to, Unix will <
still be pushing the packets around for a quite a while. --Rob Pike <