Skip to contentSkip to conversation

3.17.0+ files disappearing after playing old dos game on nfsroot laptop

DISCUSSION Hans de Bruin 2 participants 11 messages Oct 15, 2014 – May 1, 2015

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

After playing an old dos game i am missing files on my nfsroot installed laptop. Which one, wel /bin/ls is at least one of them. After a reboot there all back again. But not al is well. Some icons on my kde panel where gone. I have seen this twice in the last day's

Signature · 1 line
Hans
#1 lore raw
Quoted 5 lines · Hans de Bruin

After playing an old dos game i am missing files on my nfsroot installed laptop. Which one, wel /bin/ls is at least one of them. After a reboot there all back again. But not al is well. Some icons on my kde panel where gone. I have seen this twice in the last day's

The problem appears immediately after starting dosemu so this bisectable

Signature · 1 line
Hans
#2 lore raw
Quoted 9 lines · Hans de Bruin
Quoted 5 lines · Hans de Bruin

After playing an old dos game i am missing files on my nfsroot installed laptop. Which one, wel /bin/ls is at least one of them. After a reboot there all back again. But not al is well. Some icons on my kde panel where gone. I have seen this twice in the last day's

The problem appears immediately after starting dosemu so this bisectable

My bisect ended here:

commit 8ed936b5671bfb33d89bc60bdcc7cf0470ba52fe

Author: Eric W. Biederman <[email protected]>
Date:   Tue Oct 1 18:33:48 2013 -0700

vfs: Lazily remove mounts on unlinked files and directories.

I haven reverted it yet.

Eric,

Immediately after starting dosemu all files under mount point /usr are gone. When I play a old dos game for a while (binaries are under /home) even /bin/ls disappears. Al mounts are nfs, even /.

Could you look in to this and cc this to the relevant kernel mailinglist?

Signature · 1 line
Hans
#3 lore raw
Quoted 9 lines · Hans de Bruin
Quoted 5 lines · Hans de Bruin

After playing an old dos game i am missing files on my nfsroot installed laptop. Which one, wel /bin/ls is at least one of them. After a reboot there all back again. But not al is well. Some icons on my kde panel where gone. I have seen this twice in the last day's

The problem appears immediately after starting dosemu so this bisectable

My bisect ended here:

commit 8ed936b5671bfb33d89bc60bdcc7cf0470ba52fe

Author: Eric W. Biederman <[email protected]>
Date:   Tue Oct 1 18:33:48 2013 -0700

vfs: Lazily remove mounts on unlinked files and directories.

I haven reverted it yet.

Eric,

Immediately after starting dosemu all files under mount point /usr are gone. When I play a old dos game for a while (binaries are under /home) even /bin/ls disappears. Al mounts are nfs, even /.

Could you look in to this and cc this to the relevant kernel mailinglist?

Signature · 1 line
Hans
#4 lore raw
Quoted 26 lines · Hans de Bruin
Quoted 9 lines · Hans de Bruin
Quoted 5 lines · Hans de Bruin

After playing an old dos game i am missing files on my nfsroot installed laptop. Which one, wel /bin/ls is at least one of them. After a reboot there all back again. But not al is well. Some icons on my kde panel where gone. I have seen this twice in the last day's

The problem appears immediately after starting dosemu so this bisectable

My bisect ended here:

commit 8ed936b5671bfb33d89bc60bdcc7cf0470ba52fe

Author: Eric W. Biederman <[email protected]>
Date:   Tue Oct 1 18:33:48 2013 -0700

vfs: Lazily remove mounts on unlinked files and directories.

I haven reverted it yet.

Quoted 8 lines

Eric,

Immediately after starting dosemu all files under mount point /usr are gone. When I play a old dos game for a while (binaries are under /home) even /bin/ls disappears. Al mounts are nfs, even /.

Could you look in to this and cc this to the relevant kernel mailinglist?

At this point I don't know enough to reproduce this. What does /proc/mounts look like before you start dosemu?

My expectation is that you should only see this if the mount points are removed on the nfs server (which does not sound like it is the case).

Although a transient malfunction of the nfs server or misplaced call to check_submounts_and_drop could cause mounts to disappear as well. During testing autofs was observed to have an inappropriate call to d_invalidate and it is unlikely but possible something like that is going on with nfs as well.

Are your nfs mounts read-only or read-write?

What is your nfs-server and what is it exporting? Which distro are you running?
Which version of dosemu are you running?
How is dosemu configured to access files on your filesystem?

Eric

#5 lore raw
Quoted 34 lines · Eric W. Biederman
Quoted 32 lines · Hans de Bruin
Quoted 9 lines · Hans de Bruin
Quoted 5 lines · Hans de Bruin

After playing an old dos game i am missing files on my nfsroot installed laptop. Which one, wel /bin/ls is at least one of them. After a reboot there all back again. But not al is well. Some icons on my kde panel where gone. I have seen this twice in the last day's

The problem appears immediately after starting dosemu so this bisectable

My bisect ended here:

commit 8ed936b5671bfb33d89bc60bdcc7cf0470ba52fe

Author: Eric W. Biederman <[email protected]>
Date:   Tue Oct 1 18:33:48 2013 -0700

vfs: Lazily remove mounts on unlinked files and directories.

I haven reverted it yet.

Eric,

Immediately after starting dosemu all files under mount point /usr are gone. When I play a old dos game for a while (binaries are under /home) even /bin/ls disappears. Al mounts are nfs, even /.

Quoted 5 lines
Quoted 2 lines

Could you look in to this and cc this to the relevant kernel mailinglist?

At this point I don't know enough to reproduce this. What does /proc/mounts look like before you start dosemu?

bash-4.2$ cat /proc/mounts
rootfs / rootfs rw 0 0
10.10.0.1:/nfs/root/psion_14.1 / nfs rw,relatime,vers=3,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountproto=tcp,local_lock=all,addr=10.10.0.1 0 0 devtmpfs /dev devtmpfs rw,relatime,size=1031016k,nr_inodes=220978,mode=755 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
tmpfs /run tmpfs rw,relatime,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620 0 0
cgroup /sys/fs/cgroup cgroup rw,relatime,cpu 0 0
/dev/shm /tmp tmpfs rw,relatime,size=524288k 0 0
/dev/shm /dev/shm tmpfs rw,relatime,size=524288k 0 0
nfs:/nfs/usr/slackware-14.1/usr /usr nfs ro,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 0 0 nfs:/nfs/home /home nfs rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 0 0 nfs:/nfs/mp3 /mp3 nfs rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 0 0 nfs:/nfs/src /usr/src nfs rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 0 0 nfs:/nfs/video /video nfs rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 0 0

I now know why I do not see any file in /usr after running dosemu. The whole /usr mount disappears in /proc/mounts. When I remount it I have a usable laptop again. Running dosemu a second time does not remove the mount again. In the mean time I have seen /usr disappear after running other programs like xterm and firefox. But until now never after remouting it.

Quoted 3 lines

My expectation is that you should only see this if the mount points are removed on the nfs server (which does not sound like it is the case).

This is a at home environment with a nfs server in the meter cupboard. I have not changed the exports.

Quoted 3 lines

Although a transient malfunction of the nfs server or misplaced call to check_submounts_and_drop could cause mounts to disappear as well. During testing autofs was observed to have an inappropriate call

I am not using autofs

Quoted 4 lines

to d_invalidate and it is unlikely but possible something like that is going on with nfs as well.

Are your nfs mounts read-only or read-write?

/usr is mounted ro and exported ro

Quoted 3 lines

What is your nfs-server and what is it exporting?
Which distro are you running?

The nfs-server is slackware64 14.0 with a 3.4 kernel part from exports:
/nfs/usr -ro,async,no_subtree_check 10.10.0.0/16

The laptop is slackware(32 bit) 14.1 with yesterday's linus kernel

Quoted 1 line

Which version of dosemu are you running?

1.4.0.8

Quoted 1 line

How is dosemu configured to access files on your filesystem?

I do not understand what you mean by this. It uses ~/.dosemu/drive_c

Signature · 1 line
Hans
#6 lore raw
Quoted 1 line · Hans de Bruin

On 10/24/2014 08:18 PM, Eric W. Biederman wrote:

[...]

Quoted 32 lines
Quoted 2 lines

At this point I don't know enough to reproduce this. What does /proc/mounts look like before you start dosemu?

bash-4.2$ cat /proc/mounts
rootfs / rootfs rw 0 0
10.10.0.1:/nfs/root/psion_14.1 / nfs rw,relatime,vers=3,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountproto=tcp,local_lock=all,addr=10.10.0.1 0 0
devtmpfs /dev devtmpfs rw,relatime,size=1031016k,nr_inodes=220978,mode=755 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
tmpfs /run tmpfs rw,relatime,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620 0 0
cgroup /sys/fs/cgroup cgroup rw,relatime,cpu 0 0
/dev/shm /tmp tmpfs rw,relatime,size=524288k 0 0
/dev/shm /dev/shm tmpfs rw,relatime,size=524288k 0 0
nfs:/nfs/usr/slackware-14.1/usr /usr nfs ro,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 0 0
nfs:/nfs/home /home nfs rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 0 0
nfs:/nfs/mp3 /mp3 nfs rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 0 0
nfs:/nfs/src /usr/src nfs rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 0 0
nfs:/nfs/video /video nfs rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 0 0

Quoted 5 lines

I now know why I do not see any file in /usr after running dosemu. The whole /usr mount disappears in /proc/mounts. When I remount it I have a usable laptop again. Running dosemu a second time does not remove the mount again. In the mean time I have seen /usr disappear after running other programs like xterm and firefox. But until now never after remouting it.

That is interesting.

It sounds like occassionally your /home mount disappears as well.

Both of those would be consistent with my patch working doing what it was designed to do.

My guess is that your nfs mount on / is being weird, and causing valid directory dentries (AKA /usr and /home) to disappear temporariliy, and it looks like my change is magnifying that weirdness by causing the mounts on those directory entries to disappear.

What I don't understand is exactly why nfs is being weird that way yet.

I expect what needs to happen is to confirm that nfs directory entry revalidation is buggy, and at least for the short term re-add the nfs logic that will avoid dropping a dentry if it is a mount point, or path to a mount point, to avoid the nfs bugs.

Quoted 5 lines
Quoted 2 lines

My expectation is that you should only see this if the mount points are removed on the nfs server (which does not sound like it is the case).

This is a at home environment with a nfs server in the meter cupboard. I have not changed the exports.

On the nfs server doing "rmdir .../nfs/posion_14.1/home" or "rmdir .../nfs/posion_14.1/usr" should cause the behavior you are seeing.

As that you aren't doing that something is weird is going on.

Quoted 5 lines
Quoted 3 lines

Although a transient malfunction of the nfs server or misplaced call to check_submounts_and_drop could cause mounts to disappear as well. During testing autofs was observed to have an inappropriate call

I am not using autofs

I mentioned autofs because I think something similar is probably going on with nfs, as was observed with autofs.

Quoted 6 lines
Quoted 4 lines

to d_invalidate and it is unlikely but possible something like that is going on with nfs as well.

Are your nfs mounts read-only or read-write?

/usr is mounted ro and exported ro

/ is mounted read-write but that seems seems immaterial at the moment.

Quoted 7 lines
Quoted 2 lines

What is your nfs-server and what is it exporting?
Which distro are you running?

The nfs-server is slackware64 14.0 with a 3.4 kernel part from exports:
/nfs/usr -ro,async,no_subtree_check 10.10.0.0/16

Quoted 5 lines

The laptop is slackware(32 bit) 14.1 with yesterday's linus kernel

Quoted 1 line

Which version of dosemu are you running?

1.4.0.8

Thanks that information helps.

Eric

#7 lore raw
Quoted 38 lines · Eric W. Biederman
Quoted 1 line · Hans de Bruin

On 10/24/2014 08:18 PM, Eric W. Biederman wrote:

[...]

Quoted 32 lines
Quoted 2 lines

At this point I don't know enough to reproduce this. What does /proc/mounts look like before you start dosemu?

bash-4.2$ cat /proc/mounts
rootfs / rootfs rw 0 0
10.10.0.1:/nfs/root/psion_14.1 / nfs rw,relatime,vers=3,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountproto=tcp,local_lock=all,addr=10.10.0.1 0 0
devtmpfs /dev devtmpfs rw,relatime,size=1031016k,nr_inodes=220978,mode=755 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
tmpfs /run tmpfs rw,relatime,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620 0 0
cgroup /sys/fs/cgroup cgroup rw,relatime,cpu 0 0
/dev/shm /tmp tmpfs rw,relatime,size=524288k 0 0
/dev/shm /dev/shm tmpfs rw,relatime,size=524288k 0 0
nfs:/nfs/usr/slackware-14.1/usr /usr nfs ro,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 0 0
nfs:/nfs/home /home nfs rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 0 0
nfs:/nfs/mp3 /mp3 nfs rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 0 0
nfs:/nfs/src /usr/src nfs rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 0 0
nfs:/nfs/video /video nfs rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 0 0

Quoted 7 lines
Quoted 5 lines

I now know why I do not see any file in /usr after running dosemu. The whole /usr mount disappears in /proc/mounts. When I remount it I have a usable laptop again. Running dosemu a second time does not remove the mount again. In the mean time I have seen /usr disappear after running other programs like xterm and firefox. But until now never after remouting it.

That is interesting.

boot
run dosemu -> /usr unmounted
mount /usr
run dosemu -> /usr still mounted
sync; echo 3 > /proc/sys/vm/drop_caches
run dosemu -> /usr unmounted

Quoted 3 lines

It sounds like occassionally your /home mount disappears as well.

I have not seen that happening

...

Quoted 6 lines

I expect what needs to happen is to confirm that nfs directory entry revalidation is buggy, and at least for the short term re-add the nfs logic that will avoid dropping a dentry if it is a mount point, or path to a mount point, to avoid the nfs bugs.

ok, I will go in waiting mode.

Signature · 1 line
Hans
#8 lore raw
Quoted 7 lines · Hans de Bruin
Quoted 5 lines

I expect what needs to happen is to confirm that nfs directory entry revalidation is buggy, and at least for the short term re-add the nfs logic that will avoid dropping a dentry if it is a mount point, or path to a mount point, to avoid the nfs bugs.

ok, I will go in waiting mode.

I dropped the ball on this but it looks like someone else hit the problem and the following two commits fixed this issue:

Can you confirm that things are working again?

commit fa9233699cc1dc236f4cf42245d13e40966938c5

Author: Trond Myklebust <[email protected]>
Date:   Mon Feb 23 18:51:32 2015 -0500

NFS: Don't require a filehandle to refresh the inode in nfs_prime_dcache()

If the server does not return a valid set of attributes that we can use to either create a file or refresh the inode, then there is no value in calling nfs_prime_dcache().

However if we're just refreshing the inode using the attributes that the server returned, then it shouldn't matter whether or not we have a filehandle, as long as we check the fsid+fileid combination.

commit 6c441c254eea2354d686be7f5544bcd79fb6a61f

Author: Trond Myklebust <[email protected]>
Date:   Sun Feb 22 16:35:36 2015 -0500

NFS: Don't invalidate a submounted dentry in nfs_prime_dcache()

If we're traversing a directory which contains a submounted filesystem, or one that has a referral, the NFS server that is processing the READDIR request will often return information for the underlying (mounted-on) directory. It may, or may not, also return filehandle information.

If this happens, and the lookup in nfs_prime_dcache() returns the dentry for the submounted directory, the filehandle comparison will fail, and we call d_invalidate(). Post-commit 8ed936b5671bf ("vfs: Lazily remove mounts on unlinked files and directories."), this means the entire subtree is unmounted.

The following minimal patch addresses this problem by punting on the invalidation if there is a submount.

Kudos to Neil Brown <[email protected]> for having tracked down this issue (see link).

#9 lore raw
Quoted 8 lines · Eric W. Biederman
Quoted 6 lines · Hans de Bruin
Quoted 5 lines

I expect what needs to happen is to confirm that nfs directory entry revalidation is buggy, and at least for the short term re-add the nfs logic that will avoid dropping a dentry if it is a mount point, or path to a mount point, to avoid the nfs bugs.

Quoted 6 lines
Quoted 1 line

ok, I will go in waiting mode.

I dropped the ball on this but it looks like someone else hit the problem and the following two commits fixed this issue:

Can you confirm that things are working again?

I noticed the issue was gone in one of the previous releases. I was wondering whether you changed something or not. Sorry for not notifying you.

Signature · 1 line
Hans
#10 lore raw
Quoted 11 lines · Hans de Bruin
Quoted 10 lines · Eric W. Biederman
Quoted 7 lines · Hans de Bruin
Quoted 5 lines

I expect what needs to happen is to confirm that nfs directory entry revalidation is buggy, and at least for the short term re-add the nfs logic that will avoid dropping a dentry if it is a mount point, or path to a mount point, to avoid the nfs bugs.

ok, I will go in waiting mode.

Quoted 8 lines
Quoted 4 lines

I dropped the ball on this but it looks like someone else hit the problem and the following two commits fixed this issue:

Can you confirm that things are working again?

I noticed the issue was gone in one of the previous releases. I was wondering whether you changed something or not. Sorry for not notifying you.

No problem, and thank you for confirming the regression is fixed. One more thing I can now mark off my list.

Eric