v9fs does not honor open file handles on anonymous files
I have a small shell script:
#!/bin/bash
cat <<-EOF
EOF
Running this causes bash to fork via clone(), set fd=0 to point to an empty file in /tmp, unlink it and then execve cat. Specifically, something like this;
[pid 3699] open("/tmp/sh-thd-1388524249",
O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3
[pid 3699] open("/tmp/sh-thd-1388524249", O_RDONLY) = 4
[pid 3699] close(3) = 0
[pid 3699] unlink("/tmp/sh-thd-1388524249") = 0
[pid 3699] dup2(4, 0) = 0
[pid 3699] close(4) = 0
[pid 3699] execve("/bin/cat", ["cat"], [/* 22 vars */]) = 0One of the first things that cat does is fstat fd=0. This informs cat that the file is empty and it will quit. If /tmp is on other filesystems, cat will fstat fd=0, receive a return code of 0 from fstat, print nothing and then quit normally. If /tmp is on 9pfs, cat will fstat fd=0, receive ENOENT from fstat, print cat: -: No such file or directory to stderr and die.
It seems that v9fs_vfs_unlink() is killing the file while we still have open file handles. I have confirmed that this behavior occurs on Linux 3.13.0-rc6. This breaks several things when Gentoo is on a 9p rootfs (e.g. gcc-config, any emerge command that involves a C compiler, etcetera) inside QEMU. I have placed /tmp and /var/tmp/portage on a tmpfs as a hack-fix, but it would be better to get this fixed.
I doubt that I will write a patch to fix this. I am sending the details to the list so the 9p maintainers or any other interested individual can fix it.
Hi,
Richard Yao wrote on Tue, Dec 31, 2013 :
Quoted 24 lines
#!/bin/bash
cat <<-EOF
EOFRunning this causes bash to fork via
clone(), set fd=0 to point to an empty file in /tmp, unlink it and then execve cat. Specifically, something like this;[pid 3699] open("/tmp/sh-thd-1388524249",
O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3[pid 3699] open("/tmp/sh-thd-1388524249", O_RDONLY) = 4 [pid 3699] close(3) = 0 [pid 3699] unlink("/tmp/sh-thd-1388524249") = 0 [pid 3699] dup2(4, 0) = 0 [pid 3699] close(4) = 0 [pid 3699] execve("/bin/cat", ["cat"], [/* 22 vars */]) = 0It seems that
v9fs_vfs_unlink()is killing the file while we still have open file handles. I have confirmed that this behavior occurs on Linux 3.13.0-rc6. This breaks several things when Gentoo is on a 9p rootfs (e.g. gcc-config, any emerge command that involves a C compiler, etcetera) inside QEMU. I have placed /tmp and/var/tmp/portageon a tmpfs as a hack-fix, but it would be better to get this fixed.
Quoted 3 lines
I doubt that I will write a patch to fix this. I am sending the details to the list so the 9p maintainers or any other interested individual can fix it.
I'm not sure if it is the client's job to remember the file has been unlinked and only really unlink it after all the file handles are closed or if we should expect the server to do it.
It might not matter in the case of qemu acting as a 9p server, but there are a couple of network 9p2000.L servers out there (diod[1] and nfs-ganesha[2]), and if the file is open on one client and unlinked on another client.. How can the second client wait properly?
For what's it's worth, nfs-ganesha already behaves properly and this will work with /tmp being a 9P mount off it.
It might be worth looking into qemu's code and see if it wouldn't be easy to hold the unlink ? I've got to admit I honestly have no clue there.
(or at least send them a copy of your mail :))
[1] diod "Distributed I/O daemon" - http://code.google.com/p/diod/
[2] nfs-ganesha - https://github.com/nfs-ganesha/nfs-ganesha
Cheers,
Signature · 1 line
Dominique Martinet
Quoted 1 line · Dominique Martinet
Hi,
Thanks for the fast reply.
Quoted 25 lines
Richard Yao wrote on Tue, Dec 31, 2013 :
Quoted 24 lines
#!/bin/bash
cat <<-EOF
EOFRunning this causes bash to fork via
clone(), set fd=0 to point to an empty file in /tmp, unlink it and then execve cat. Specifically, something like this;[pid 3699] open("/tmp/sh-thd-1388524249",
O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3[pid 3699] open("/tmp/sh-thd-1388524249", O_RDONLY) = 4 [pid 3699] close(3) = 0 [pid 3699] unlink("/tmp/sh-thd-1388524249") = 0 [pid 3699] dup2(4, 0) = 0 [pid 3699] close(4) = 0 [pid 3699] execve("/bin/cat", ["cat"], [/* 22 vars */]) = 0It seems that
v9fs_vfs_unlink()is killing the file while we still have open file handles. I have confirmed that this behavior occurs on Linux 3.13.0-rc6. This breaks several things when Gentoo is on a 9p rootfs (e.g. gcc-config, any emerge command that involves a C compiler, etcetera) inside QEMU. I have placed /tmp and/var/tmp/portageon a tmpfs as a hack-fix, but it would be better to get this fixed.
Quoted 7 lines
Quoted 3 lines
I doubt that I will write a patch to fix this. I am sending the details to the list so the 9p maintainers or any other interested individual can fix it.
I'm not sure if it is the client's job to remember the file has been unlinked and only really unlink it after all the file handles are closed or if we should expect the server to do it.
The answer to this is largely philosophical in nature, but it is a good question to ask. Quite honestly, I have no preference, but I was not sure who was responsible. Having caught another bug that I thought was in QEMU that turned out to be in Linux, I decided to ping the Linux folks first this time.
Speaking of which, that patch did not get much attention because I did not send it to the right lists. I will resend it soon:
http://www.spinics.net/lists/linux-virtualization/msg21716.html
Quoted 5 lines
It might not matter in the case of qemu acting as a 9p server, but there are a couple of network 9p2000.L servers out there (diod[1] and nfs-ganesha[2]), and if the file is open on one client and unlinked on another client.. How can the second client wait properly?
Quoted 8 lines
For what's it's worth, nfs-ganesha already behaves properly and this will work with /tmp being a 9P mount off it.
It might be worth looking into qemu's code and see if it wouldn't be easy to hold the unlink ? I've got to admit I honestly have no clue there.
(or at least send them a copy of your mail :))
If nfs-ganesha handles this properly, then I am going to say this is QEMU's bug. I had CCed Aneesh, who I am told is responsible for the QEMU 9p code, so hopefully we will hear from him soon. Otherwise, I will resend to the QEMU list.
Quoted 7 lines
[1] diod "Distributed I/O daemon" - http://code.google.com/p/diod/
[2] nfs-ganesha - https://github.com/nfs-ganesha/nfs-ganeshaCheers,