Skip to contentSkip to conversation

rdirplus broken? (EBUSY)

3.6-rc3 Simon Kirby 2 participants 7 messages Aug 27, 2012 – Aug 27, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hello!

Something seems broiken in 3.6-rc[123] which was fine in 3.5 and earlier. This is a 3.4.1 knfsd server with ext3 and XFS-based NFS exports:

/ 192.168.13.0/24(rw,no_root_squash,no_subtree_check,async)
/pics 192.168.13.0/24(rw,no_root_squash,no_subtree_check,async)
/raid 192.168.13.0/24(rw,no_root_squash,no_subtree_check,async)

and a 3.6-rc3 client with this in fstab:

flick:/ /flick nfs rw,vers=3
flick:/raid /flick/raid nfs rw,vers=3
flick:/pics /flick/pics nfs rw,vers=3

This seems to fail now as follows:

[sroot@oof:/]# mount flick
[sroot@oof:/]# mount flick/raid
[sroot@oof:/]# mount flick/pics
[sroot@oof:/]# ls -l flick
ls: cannot access flick/pics: Device or resource busy
ls: cannot access flick/raid: Device or resource busy
total 2180
drwxr-xr-x  45 root root    4096 Jun 18 14:19 ./
drwxr-xr-x  58 root root    4096 Jul  3 22:24 ../
...
??????????   ? ?    ?          ?            ? pics
??????????   ? ?    ?          ?            ? raid
...
[sroot@oof:/]# cd flick/pics
flick/pics: Device or resource busy.

These mount points are now stuck and cannot be unmounted until I reboot (umount -l fails with EBUSY).

If I mount with "nordirplus", I can't seem to get it to break. However, sometimes it will work regardless. I can bisect this if it would help..

Simon-

#1 lore raw
Quoted 33 lines · Simon Kirby

Hello!

Something seems broiken in 3.6-rc[123] which was fine in 3.5 and earlier. This is a 3.4.1 knfsd server with ext3 and XFS-based NFS exports:

/ 192.168.13.0/24(rw,no_root_squash,no_subtree_check,async)
/pics 192.168.13.0/24(rw,no_root_squash,no_subtree_check,async)
/raid 192.168.13.0/24(rw,no_root_squash,no_subtree_check,async)

and a 3.6-rc3 client with this in fstab:

flick:/ /flick nfs rw,vers=3
flick:/raid /flick/raid nfs rw,vers=3
flick:/pics /flick/pics nfs rw,vers=3

This seems to fail now as follows:

[sroot@oof:/]# mount flick
[sroot@oof:/]# mount flick/raid
[sroot@oof:/]# mount flick/pics
[sroot@oof:/]# ls -l flick
ls: cannot access flick/pics: Device or resource busy
ls: cannot access flick/raid: Device or resource busy
total 2180
drwxr-xr-x  45 root root    4096 Jun 18 14:19 ./
drwxr-xr-x  58 root root    4096 Jul  3 22:24 ../
...
??????????   ? ?    ?          ?            ? pics
??????????   ? ?    ?          ?            ? raid
...
[sroot@oof:/]# cd flick/pics
flick/pics: Device or resource busy.
Quoted 5 lines

These mount points are now stuck and cannot be unmounted until I reboot (umount -l fails with EBUSY).

If I mount with "nordirplus", I can't seem to get it to break. However, sometimes it will work regardless. I can bisect this if it would help..

This is still the case with 3.6-rc5. I hadn't noticed any problem since mounting with nordirplus, and it broke immediately after removing the option again. I will bisect.

Simon-

#2 lore raw
Quoted 38 lines · Simon Kirby
Quoted 36 lines · Simon Kirby

Hello!

Something seems broiken in 3.6-rc[123] which was fine in 3.5 and earlier. This is a 3.4.1 knfsd server with ext3 and XFS-based NFS exports:

/ 192.168.13.0/24(rw,no_root_squash,no_subtree_check,async)
/pics 192.168.13.0/24(rw,no_root_squash,no_subtree_check,async)
/raid 192.168.13.0/24(rw,no_root_squash,no_subtree_check,async)

and a 3.6-rc3 client with this in fstab:

flick:/ /flick nfs rw,vers=3
flick:/raid /flick/raid nfs rw,vers=3
flick:/pics /flick/pics nfs rw,vers=3

This seems to fail now as follows:

[sroot@oof:/]# mount flick
[sroot@oof:/]# mount flick/raid
[sroot@oof:/]# mount flick/pics
[sroot@oof:/]# ls -l flick
ls: cannot access flick/pics: Device or resource busy
ls: cannot access flick/raid: Device or resource busy
total 2180
drwxr-xr-x  45 root root    4096 Jun 18 14:19 ./
drwxr-xr-x  58 root root    4096 Jul  3 22:24 ../
...
??????????   ? ?    ?          ?            ? pics
??????????   ? ?    ?          ?            ? raid
...
[sroot@oof:/]# cd flick/pics
flick/pics: Device or resource busy.

These mount points are now stuck and cannot be unmounted until I reboot (umount -l fails with EBUSY).

Quoted 6 lines
Quoted 2 lines

If I mount with "nordirplus", I can't seem to get it to break. However, sometimes it will work regardless. I can bisect this if it would help..

This is still the case with 3.6-rc5. I hadn't noticed any problem since mounting with nordirplus, and it broke immediately after removing the option again. I will bisect.

The symptoms sound similar to
http://marc.info/?l=linux-fsdevel&m=134738157303017&w=2

Might be worth checking whether it's that patch?

--b.

#3 lore raw
Quoted 40 lines · J. Bruce Fields
Quoted 39 lines · Simon Kirby
Quoted 36 lines · Simon Kirby

Something seems broiken in 3.6-rc[123] which was fine in 3.5 and earlier. This is a 3.4.1 knfsd server with ext3 and XFS-based NFS exports:

/ 192.168.13.0/24(rw,no_root_squash,no_subtree_check,async)
/pics 192.168.13.0/24(rw,no_root_squash,no_subtree_check,async)
/raid 192.168.13.0/24(rw,no_root_squash,no_subtree_check,async)

and a 3.6-rc3 client with this in fstab:

flick:/ /flick nfs rw,vers=3
flick:/raid /flick/raid nfs rw,vers=3
flick:/pics /flick/pics nfs rw,vers=3

This seems to fail now as follows:

[sroot@oof:/]# mount flick
[sroot@oof:/]# mount flick/raid
[sroot@oof:/]# mount flick/pics
[sroot@oof:/]# ls -l flick
ls: cannot access flick/pics: Device or resource busy
ls: cannot access flick/raid: Device or resource busy
total 2180
drwxr-xr-x  45 root root    4096 Jun 18 14:19 ./
drwxr-xr-x  58 root root    4096 Jul  3 22:24 ../
...
??????????   ? ?    ?          ?            ? pics
??????????   ? ?    ?          ?            ? raid
...
[sroot@oof:/]# cd flick/pics
flick/pics: Device or resource busy.

These mount points are now stuck and cannot be unmounted until I reboot (umount -l fails with EBUSY).

If I mount with "nordirplus", I can't seem to get it to break. However, sometimes it will work regardless. I can bisect this if it would help..

Quoted 8 lines
Quoted 3 lines

This is still the case with 3.6-rc5. I hadn't noticed any problem since mounting with nordirplus, and it broke immediately after removing the option again. I will bisect.

The symptoms sound similar to
http://marc.info/?l=linux-fsdevel&m=134738157303017&w=2

Might be worth checking whether it's that patch?

Indeed! I tried this hack:

fs/dcache.c +4 -0
index 8086636..649a112 100644
@@ -2404,6 +2404,10 @@ out_unalias:
 	if (likely(!d_mountpoint(alias))) {
 		__d_move(alias, dentry);
 		ret = alias;
+	} else {
+		printk(KERN_WARNING "VFS: __d_move()ing a d_mountpoint(), uh oh\n");
+		__d_move(alias, dentry);
+		ret = alias;
 	}
 out_err:
 	spin_unlock(&inode->i_lock);

With this applied, "ls -l flick" prints:

[   77.217420] VFS: __d_move()ing a d_mountpoint(), uh oh
[   77.222390] VFS: __d_move()ing a d_mountpoint(), uh oh

...and "pics" and "raid" then work as they did before, or with "nordirplus" set. So, is something broken with nordirplus or the NFS layer, or should __d_unalias() really move a mountpoint? With nordirplus, it works without complaining about moving a mountpoint.

Simon-

#4 lore raw
Quoted 27 lines · Simon Kirby
Quoted 4 lines · J. Bruce Fields

The symptoms sound similar to
http://marc.info/?l=linux-fsdevel&m=134738157303017&w=2

Might be worth checking whether it's that patch?

Indeed! I tried this hack:

fs/dcache.c +4 -0
index 8086636..649a112 100644
@@ -2404,6 +2404,10 @@ out_unalias:
 	if (likely(!d_mountpoint(alias))) {
 		__d_move(alias, dentry);
 		ret = alias;
+	} else {
+		printk(KERN_WARNING "VFS: __d_move()ing a d_mountpoint(), uh oh\n");
+		__d_move(alias, dentry);
+		ret = alias;
 	}
 out_err:
 	spin_unlock(&inode->i_lock);

With this applied, "ls -l flick" prints:

Quoted 7 lines
[   77.217420] VFS: __d_move()ing a d_mountpoint(), uh oh
[   77.222390] VFS: __d_move()ing a d_mountpoint(), uh oh

...and "pics" and "raid" then work as they did before, or with "nordirplus" set. So, is something broken with nordirplus or the NFS layer, or should __d_unalias() really move a mountpoint? With nordirplus, it works without complaining about moving a mountpoint.

By the way, This seems fixed in 3.6-rc6, likely due to c3f52af3e03013db5237e339c817beaae5ec9e3a. Thanks!

Simon-

#5 lore raw
Quoted 32 lines · Simon Kirby
Quoted 30 lines · Simon Kirby
Quoted 4 lines · J. Bruce Fields

The symptoms sound similar to
http://marc.info/?l=linux-fsdevel&m=134738157303017&w=2

Might be worth checking whether it's that patch?

Indeed! I tried this hack:

fs/dcache.c +4 -0
index 8086636..649a112 100644
@@ -2404,6 +2404,10 @@ out_unalias:
 	if (likely(!d_mountpoint(alias))) {
 		__d_move(alias, dentry);
 		ret = alias;
+	} else {
+		printk(KERN_WARNING "VFS: __d_move()ing a d_mountpoint(), uh oh\n");
+		__d_move(alias, dentry);
+		ret = alias;
 	}
 out_err:
 	spin_unlock(&inode->i_lock);

With this applied, "ls -l flick" prints:

[   77.217420] VFS: __d_move()ing a d_mountpoint(), uh oh
[   77.222390] VFS: __d_move()ing a d_mountpoint(), uh oh
Quoted 7 lines
Quoted 4 lines

...and "pics" and "raid" then work as they did before, or with "nordirplus" set. So, is something broken with nordirplus or the NFS layer, or should __d_unalias() really move a mountpoint? With nordirplus, it works without complaining about moving a mountpoint.

By the way, This seems fixed in 3.6-rc6, likely due to c3f52af3e03013db5237e339c817beaae5ec9e3a. Thanks!

I confused myself with my own patch here. This still happens to me in release 3.6, making it not possible for me to use these NFS mounts unless I set "nordirplus" or apply my above call-__d_move-anyway patch.

I'm also getting file data corruption when mounted TCP, for some stupid reason, even with all TSO/GSO/GRO disabled, and this goes away with UDP. Reproducible on different client hardware, and on client kernels back to 2.6.32. Probably related to the 3.4.1 server. More debugging to do...

Simon-

#6 lore raw
Quoted 4 lines · Simon Kirby

I'm also getting file data corruption when mounted TCP, for some stupid reason, even with all TSO/GSO/GRO disabled, and this goes away with UDP. Reproducible on different client hardware, and on client kernels back to 2.6.32. Probably related to the 3.4.1 server. More debugging to do...

Is this still happening?

--b.