nfsd-fh: found a name that I didn't expect

due to hardware problems I just started our fall back server and got these messages for 2 files:
nfsd-fh: found a name that I didn't expect: bin/uptime
nfsd-fh: found a name that I didn't expect: bin/uptime
nfsd: last server has exited
nfsd: unexporting all filesystems
nfsd-fh: found a name that I didn't expect: lib/libident.so.0
nfsd-fh: found a name that I didn't expect: lib/libident.so.0

Any ideas how we can prevent this in the future ?

Hello!

Quoted 8 lines · Bernd Schubert

due to hardware problems I just started our fall back server and got these messages for 2 files:
nfsd-fh: found a name that I didn't expect: bin/uptime
nfsd-fh: found a name that I didn't expect: bin/uptime
nfsd: last server has exited
nfsd: unexporting all filesystems
nfsd-fh: found a name that I didn't expect: lib/libident.so.0
nfsd-fh: found a name that I didn't expect: lib/libident.so.0

Hm. Does the message is visible each time you access those files?

Does reiserfsck have found anything?

Quoted 2 lines

Any ideas how we can prevent this in the future ?

We do not yet understand on how to reproduce that locally, because this message indicates pretty strange conditions.

Quoted 8 lines

Quoted 8 lines

due to hardware problems I just started our fall back server and got these messages for 2 files:
nfsd-fh: found a name that I didn't expect: bin/uptime
nfsd-fh: found a name that I didn't expect: bin/uptime
nfsd: last server has exited
nfsd: unexporting all filesystems
nfsd-fh: found a name that I didn't expect: lib/libident.so.0
nfsd-fh: found a name that I didn't expect: lib/libident.so.0

Quoted 1 line

Hm. Does the message is visible each time you access those files?

Yes, probably since uptime and libident.so.0 were called in and endless loop from the clients, the nfs-servers log was filled with those messages and I had to stop the nfsd and do the cp- and mv-procedure for those files.

Quoted 1 line

Does reiserfsck have found anything?

As I said, the backup script runs a reiserfsck itself and found no problems, I hope there is no corruption after a simple boot, but I will check this later on this day.

Quoted 2 lines

We do not yet understand on how to reproduce that locally, because this message indicates pretty strange conditions.

Is there anything I can do to debug this, when I observe this phenomena the next time (e.g. enabling nfs-debugging via proc, etc) ?

Quoted 5 lines

Quoted 1 line

Does reiserfsck have found anything?

As I said, the backup script runs a reiserfsck itself and found no problems, I hope there is no corruption after a simple boot, but I will check this later on this day.

Ok, let us know of the results.

Quoted 4 lines

Quoted 3 lines

We do not yet understand on how to reproduce that locally, because this message indicates pretty strange conditions.

Quoted 2 lines

Is there anything I can do to debug this, when I observe this phenomena the next time (e.g. enabling nfs-debugging via proc, etc) ?

Please try to reboot the server not changing fs, if the problem still persists, then take FS metadata snapshot (debugreiserfs -p /dev/device | bzip2 -9c >metadata.bz2, use recent reiserfsprogs please) and make it available for us to download, please.

Quoted 1 line

Ok, let us know of the results.

As I expected, 'reiserfsck --check' didn't find any problems.

As noted in my other e-email, /usr/sbin/logrotate was also affected. I don't like it to say, but after rebooting everything was fine, so those messages doesn't appear any longer.
Does this mean memory problems again ? We even have loaded the

ecc-kernel-module since last week, to be able to problems befor the filesystem is affected, but it didn't show any memory-problem on this machine.
Or could it be that we have 3GB RAM and the vanilla kernel is not suitable for this (I read something like this in the LKML, but can't remember the details).

Quoted 4 lines

As noted in my other e-email, /usr/sbin/logrotate was also affected. I don't like it to say, but after rebooting everything was fine, so those messages doesn't appear any longer.
Does this mean memory problems again ? We even have loaded the

Not likely.

The problems seems to have something to do with strange dentries appearing in dentry cache, also you have verified that this is not because of fs error.
What kernel are you running?

I think may be race with iget4 might cause this, but I am not sure. We have a patch for this iget4 race, and if you are willing to test it, I can send it to you.

Quoted 4 lines

The problems seems to have something to do with strange dentries appearing in dentry cache, also you have verified that this is not because of fs error.
What kernel are you running?

2.4.20 with the ptrace patch.

Quoted 3 lines

I think may be race with iget4 might cause this, but I am not sure. We have a patch for this iget4 race, and if you are willing to test it, I can send it to you.

Yes of course I am. I hope the other admins won't kill me, but I guess they know that I willing to do experiments from time to time ;-)
Could something worse happen when I try the patch, that could not be fixed by simple rebooting ?

The problems is to detect if this is really caused by this race. Since I still

Update: While I was writing and thinking about how to trigger it, I just again saw the problem for /usr/bin/uptime and another file and again a reboot could fix it for those files.

See the patch below.
It should not break anything.

Quoted 3 lines

The problems is to detect if this is really caused by this race. Since I still

Well, race is of type when several parallel processes try to access the file whose inode is not in memory.
This resembles what you describe with lots of clients accessing same file over NFS at the same time.

Hello!

Quoted 5 lines · Bernd Schubert

Quoted 4 lines

The problems seems to have something to do with strange dentries appearing in dentry cache, also you have verified that this is not because of fs error.
What kernel are you running?

2.4.20 with the ptrace patch.

Ok.

Quoted 7 lines

Quoted 3 lines

I think may be race with iget4 might cause this, but I am not sure. We have a patch for this iget4 race, and if you are willing to test it, I can send it to you.

Yes of course I am. I hope the other admins won't kill me, but I guess they know that I willing to do experiments from time to time ;-)
Could something worse happen when I try the patch, that could not be fixed by simple rebooting ?

See the patch below.
It should not break anything.

Quoted 9 lines

Quoted 7 lines

Quoted 3 lines

I think may be race with iget4 might cause this, but I am not sure. We have a patch for this iget4 race, and if you are willing to test it, I can send it to you.

Yes of course I am. I hope the other admins won't kill me, but I guess they know that I willing to do experiments from time to time ;-)
Could something worse happen when I try the patch, that could not be fixed by simple rebooting ?

See the patch below.
It should not break anything.

Well, we keep getting this error in massess too... and in our case it is very likely that clients access the same file from all the clients...

However I cannot quickly test that atm but if it is really safe I might test it next week.

Hello!

Quoted 7 lines · Soeren Sonnenburg

Quoted 7 lines

Quoted 7 lines

Quoted 3 lines

I think may be race with iget4 might cause this, but I am not sure. We have a patch for this iget4 race, and if you are willing to test it, I can send it to you.

Yes of course I am. I hope the other admins won't kill me, but I guess they know that I willing to do experiments from time to time ;-)
Could something worse happen when I try the patch, that could not be fixed by simple rebooting ?

Quoted 4 lines

Quoted 2 lines

See the patch below.
It should not break anything.

Well, we keep getting this error in massess too... and in our case it is very likely that clients access the same file from all the clients...

Are all of your NFS exports where these errors happens reiserfs?

Quoted 2 lines

However I cannot quickly test that atm but if it is really safe I might test it next week.

Well, there is a problem reported with that 2.4.20 backport, that I am going to look into. If you can try 2.4.21-pre6 with iget5_locked.diff (I just sent to the list in this same thread) that would be great.

Thank you.

Bye,
Oleg

Quoted 9 lines

Quoted 8 lines

Quoted 6 lines

Quoted 4 lines

Yes of course I am. I hope the other admins won't kill me, but I guess they know that I willing to do experiments from time to time ;-)
Could something worse happen when I try the patch, that could not be fixed by simple rebooting ?

See the patch below.
It should not break anything.

Well, we keep getting this error in massess too... and in our case it is very likely that clients access the same file from all the clients...

Are all of your NFS exports where these errors happens reiserfs?

This is the same machine I was telling you about on lkml ...

well, I can't do experimental testing as there are like 40 diskless machines using
that disk ... I could somewhen in the evening/night do some stress test but thats it...

See the patch below.
It should not break anything.

Quoted 1 line

EIP; c0149870 <iput+20/200> <=====

Quoted 2 lines

See the patch below.
It should not break anything.

I just tried to cause the problems again, but without any success. I simply don't get the messages today. I tried to call /usr/bin/uptime from all

Anyway I also tested the kernel-patch, but it causes problems with mounts from another nfs-server. Well, mounting works fine, but e.g. running 'ls {mountdir}' returns 'Unknown error 524' (or was it 542 I can't remember). So even my homedirectory was not available on the server when I tried the iget-patch.

Well on system-reboot there I also get an oops. I have attached the ksymoops output.

Anyway I also tested the kernel-patch, but it causes problems with mounts from another nfs-server. Well, mounting works fine, but e.g. running 'ls

Hello!

Quoted 2 lines · Bernd Schubert

Anyway I also tested the kernel-patch, but it causes problems with mounts from another nfs-server. Well, mounting works fine, but e.g. running 'ls

Ah, stupid me.
I have another patch that we created before iget5_locked backport. It's a bit of hack to beat the race, but it works (fixes the race). It does not touch any code outside of reiserfs.
So if you apply that to 2.4.20 and see if you still can see the NFS problem over time...
Patch is below.

Quoted 6 lines

I have another patch that we created before iget5_locked backport. It's a bit of hack to beat the race, but it works (fixes the race). It does not touch any code outside of reiserfs.
So if you apply that to 2.4.20 and see if you still can see the NFS problem over time...
Patch is below.

Err, sorry but I have to ask again:
This problem needs a patch on the server only - yes ?

Then I will at least patch+compile the kernel.

Quoted 2 lines

Err, sorry but I have to ask again:
This problem needs a patch on the server only - yes ?

Yes. Well, may be this race is not the cause of the problem you are observing, but prerequisites are similar, so we'd better make sure this is different problem.

Quoted 1 line

Then I will at least patch+compile the kernel.

Thank you.

Quoted 2 lines

Anyway I also tested the kernel-patch, but it causes problems with mounts from another nfs-server. Well, mounting works fine, but e.g. running 'ls

Ah, stupid me.
I have another patch that we created before iget5_locked backport. It's a bit of hack to beat the race, but it works (fixes the race). It does not touch any code outside of reiserfs.
So if you apply that to 2.4.20 and see if you still can see the NFS problem over time...
Patch is below.

Hello!

Quoted 1 line · Bernd Schubert

Anyway I also tested the kernel-patch, but it causes problems with mounts from another nfs-server. Well, mounting works fine, but e.g. running 'ls

Ah, stupid me.
I have another patch that we created before iget5_locked backport. It's a bit of hack to beat the race, but it works (fixes the race). It does not touch any code outside of reiserfs.
So if you apply that to 2.4.20 and see if you still can see the NFS problem over time...
Patch is below.

Bye,
Oleg

===== fs/reiserfs/inode.c 1.42 vs edited =====

fs/reiserfs/inode.c +24 -2

@@ -20,6 +20,10 @@
 static int reiserfs_get_block (struct inode * inode, long block,
 			       struct buffer_head * bh_result, int create);
 
+/* This spinlock guards inode pkey in private part of inode
+   against race between find_actor() vs reiserfs_read_inode2 */
+static spinlock_t keycopy_lock = SPIN_LOCK_UNLOCKED;
+
 void reiserfs_delete_inode (struct inode * inode)
 {
     int jbegin_count = JOURNAL_PER_BALANCE_CNT * 2; 

@@ -898,8 +902,9 @@
     bh = PATH_PLAST_BUFFER (path);
     ih = PATH_PITEM_HEAD (path);
 
-
+    spin_lock(&keycopy_lock);
     copy_key (INODE_PKEY (inode), &(ih->ih_key));
+    spin_unlock(&keycopy_lock);
     inode->i_blksize = PAGE_SIZE;
 
     INIT_LIST_HEAD(&inode->u.reiserfs_i.i_prealloc_list) ;

@@ -1220,10 +1225,27 @@
 				unsigned long inode_no, void *opaque )
 {
     struct reiserfs_iget4_args *args;
+    int retval;
 
     args = opaque;
+    /* We protect against possible parallel init_inode() on another CPU here. */
+    spin_lock(&keycopy_lock);
     /* args is already in CPU order */
-    return le32_to_cpu(INODE_PKEY(inode)->k_dir_id) == args -> objectid;
+    if (le32_to_cpu(INODE_PKEY(inode)->k_dir_id) == args -> objectid)
+	retval = 1;
+    else
+	/* If The key does not match, lets see if we are racing
+	   with another iget4, that already progressed so far
+	   to reiserfs_read_inode2() and was preempted in
+	   call to search_by_key(). The signs of that are:
+	     Inode is locked
+	     dirid and object id are zero (not yet initialized)*/
+	retval = (inode->i_state & I_LOCK) &&
+		 !INODE_PKEY(inode)->k_dir_id &&
+		 !INODE_PKEY(inode)->k_objectid;
+
+    spin_unlock(&keycopy_lock);
+    return retval;
 }
 
 struct inode * reiserfs_iget (struct super_block * s, const struct cpu_key * key)

Quoted 6 lines · Oleg Drokin

Hello!

Quoted 2 lines · Bernd Schubert

Anyway I also tested the kernel-patch, but it causes problems with mounts from another nfs-server. Well, mounting works fine, but e.g. running 'ls

Quoted 7 lines

Ah, stupid me.
I have another patch that we created before iget5_locked backport. It's a bit of hack to beat the race, but it works (fixes the race). It does not touch any code outside of reiserfs.
So if you apply that to 2.4.20 and see if you still can see the NFS problem over time...
Patch is below.

I just got an "nfsd-fh: found a name that I didn't expect" yesterday. I'm using a Red Hat 2.4.20 RPM with 2.4.20-pending+data-logging+quota. Should I apply just this patch or both this patch and the iget5_locked_2.4.20 patch?
Thanks,
John Dalbec

Quoted 59 lines

Bye,
Oleg

===== fs/reiserfs/inode.c 1.42 vs edited =====

fs/reiserfs/inode.c +23 -2

@@ -20,6 +20,10 @@
 static int reiserfs_get_block (struct inode * inode, long block,
 			       struct buffer_head * bh_result, int create);
 
+/* This spinlock guards inode pkey in private part of inode
+   against race between find_actor() vs reiserfs_read_inode2 */
+static spinlock_t keycopy_lock = SPIN_LOCK_UNLOCKED;
+
 void reiserfs_delete_inode (struct inode * inode)
 {
     int jbegin_count = JOURNAL_PER_BALANCE_CNT * 2;

@@ -898,8 +902,9 @@
     bh = PATH_PLAST_BUFFER (path);
     ih = PATH_PITEM_HEAD (path);
 
-
+    spin_lock(&keycopy_lock);
     copy_key (INODE_PKEY (inode), &(ih->ih_key));
+    spin_unlock(&keycopy_lock);
     inode->i_blksize = PAGE_SIZE;
 
     INIT_LIST_HEAD(&inode->u.reiserfs_i.i_prealloc_list) ;

@@ -1220,10 +1225,27 @@
 				unsigned long inode_no, void *opaque )
 {
     struct reiserfs_iget4_args *args;
+    int retval;
 
     args = opaque;
+    /* We protect against possible parallel init_inode() on another CPU
 here. */ +    spin_lock(&keycopy_lock);
     /* args is already in CPU order */
-    return le32_to_cpu(INODE_PKEY(inode)->k_dir_id) == args -> objectid;
+    if (le32_to_cpu(INODE_PKEY(inode)->k_dir_id) == args -> objectid)
+	retval = 1;
+    else
+	/* If The key does not match, lets see if we are racing
+	   with another iget4, that already progressed so far
+	   to reiserfs_read_inode2() and was preempted in
+	   call to search_by_key(). The signs of that are:
+	     Inode is locked
+	     dirid and object id are zero (not yet initialized)*/
+	retval = (inode->i_state & I_LOCK) &&
+		 !INODE_PKEY(inode)->k_dir_id &&
+		 !INODE_PKEY(inode)->k_objectid;
+
+    spin_unlock(&keycopy_lock);
+    return retval;
 }
 
 struct inode * reiserfs_iget (struct super_block * s, const struct

cpu_key * key)

I just got an "nfsd-fh: found a name that I didn't expect" yesterday. I'm using a Red Hat 2.4.20 RPM with 2.4.20-pending+data-logging+quota. Should I apply just this patch or both this patch and the iget5_locked_2.4.20 patch?

Quoted 57 lines

===== fs/reiserfs/inode.c 1.42 vs edited =====

fs/reiserfs/inode.c +8 -1

@@ -20,6 +20,10 @@
 static int reiserfs_get_block (struct inode * inode, long block,
 			       struct buffer_head * bh_result, int create);
 
+/* This spinlock guards inode pkey in private part of inode
+   against race between find_actor() vs reiserfs_read_inode2 */
+static spinlock_t keycopy_lock = SPIN_LOCK_UNLOCKED;
+
 void reiserfs_delete_inode (struct inode * inode)
 {
    int jbegin_count = JOURNAL_PER_BALANCE_CNT * 2; 

@@ -898,8 +902,9 @@
    bh = PATH_PLAST_BUFFER (path);
    ih = PATH_PITEM_HEAD (path);
 
-
+    spin_lock(&keycopy_lock);
    copy_key (INODE_PKEY (inode), &(ih->ih_key));
+    spin_unlock(&keycopy_lock);
    inode->i_blksize = PAGE_SIZE;
 
    INIT_LIST_HEAD(&inode->u.reiserfs_i.i_prealloc_list) ;

@@ -1220,10 +1225,27 @@
 				unsigned long inode_no, void *opaque )
 {
    struct reiserfs_iget4_args *args;
+    int retval;
 
    args = opaque;
+    /* We protect against possible parallel init_inode() on another CPU 

here. */

fs/reiserfs/inode.c +16 -1

+    spin_lock(&keycopy_lock);
    /* args is already in CPU order */
-    return le32_to_cpu(INODE_PKEY(inode)->k_dir_id) == args -> objectid;
+    if (le32_to_cpu(INODE_PKEY(inode)->k_dir_id) == args -> objectid)
+	retval = 1;
+    else
+	/* If The key does not match, lets see if we are racing
+	   with another iget4, that already progressed so far
+	   to reiserfs_read_inode2() and was preempted in
+	   call to search_by_key(). The signs of that are:
+	     Inode is locked
+	     dirid and object id are zero (not yet initialized)*/
+	retval = (inode->i_state & I_LOCK) &&
+		 !INODE_PKEY(inode)->k_dir_id &&
+		 !INODE_PKEY(inode)->k_objectid;
+
+    spin_unlock(&keycopy_lock);
+    return retval;

}

struct inode * reiserfs_iget (struct super_block * s, const struct cpu_key * key)

We do not yet understand on how to reproduce that locally, because this message indicates pretty strange conditions.

Hello!

Quoted 4 lines · Roland

i've read your thread very interessted, because i think i've a very similar problem (but until know, i didn't think about reiserfs as source): kernel 2.4.20 SMP, HighMem, HighMem IO (without ptrace patch) root filesystem ~70GB ext3 on Compaq CISS

Quoted 8 lines

/data filesystems ~1.1TB reiserfs (on top of lvm) on a second Compaq CISS
System: Compaq Dual PIII 2GB Ram
the system is used as nfs fileserver for diskless clients (about 30 right now) first we experienced high load situations during daily cron jobs, later we saw log messages very similar to what Bernd has (/usr/sbin/logrotate, /usr/lib/perl5/...) and last week we had a crash of the fileserver (no sysreq key's, console blanking didn't responded, ping to system was ok)

Was there anything in logs after reboot?

Quoted 3 lines

since that crash we don't have any 'nfsd-fh: found a name that I didn't expect:' messages; but we need to find out what was behind that crash... Any ideas?

If only you had related kernel oops/whatever was caused the crash (either on your HDD, or captured via serial console/network oops dumper/whatever), that would be much easier.

Quoted 3 lines

(it would be possible for me to provide a machine which is 100% identical to our production fileserver if this would help for tests, but i don't have ressources to put up more than one client for this testsystem...)

Since we do not know yet on how to produce those nfsd-fh stuff, there is no point in using your test server yet.