permission denied with >= ~2.6.25
Subsystems (2)
Hello list,
recently I updated my client to a new vendor kernel(SLES11), but the problem is the same if I use a 2.6.33.3 as a client.
If I'm doing svn checkouts in an endless loop,
or running make -j X builds sometimes it happens
that I get a "Permision denied" error.
If I strace it, for svn, the following sequence appears:
5262 rename("Kernel.ERWQPsWCA5/.svn/tmp/entries", "Kernel.ERWQPsWCA5/.svn/entries") = 0
5262 lstat("Kernel.ERWQPsWCA5/.svn/entries", {st_mode=S_IFREG|0666, st_size=205, ...}) = 0
5262 chmod("Kernel.ERWQPsWCA5/.svn/entries", 0444) = 0 5262 open("Kernel.ERWQPsWCA5/.svn/format.tmp", O_RDWR|O_CREAT|O_EXCL, 0666) = 3
5262 write(3, "9\n", 2) = 2 5262 close(3) = 0
5262 rename("Kernel.ERWQPsWCA5/.svn/format.tmp", "Kernel.ERWQPsWCA5/.svn/format") = 0
5262 lstat("Kernel.ERWQPsWCA5/.svn/format", 0x7fff79949350) = -1 EACCES (Permission denied)
^^^^^^^^^^^^ Here
5262 write(2, "svn: Can't change perms of file "..., 83) = 83 5262 lstat("Kernel.ERWQPsWCA5/.svn/log", 0x7fff79949650) = -1 ENOENT (No such file or directory)
5262 unlink("Kernel.ERWQPsWCA5/.svn/lock") = 0
After a rename, the lstat call fails. I tried to isolate it, but without any luck so far.
I dont have the problem if I go back to my vendor kernel <= 2.6.22. I wonder if its more a NFS Server problem or a problem on the client side so I tried writing a C programm to trigger this behaviour. But without any success so far.
Maybe this problem is already known?
regards,
Martin
Quoted 9 lines · Martin Vogt
Hello list,
recently I updated my client to a new vendor kernel(SLES11), but the problem is the same if I use a 2.6.33.3 as a client.
If I'm doing svn checkouts in an endless loop,
or running make -j X builds sometimes it happens
that I get a "Permision denied" error.
Hello,
I tested 2.6.34-rc7 now and it has the problem too. Then I checked 2.6.24.7 which has the problem, but 2.6.23.1 looks fine.
Quoted 11 lines
svn: In directory 'Kernel.9TKDNMQTDm/arch/arm/mach-realview'
svn: Error processing command 'readonly' in 'Kernel.9TKDNMQTDm/arch/arm/mach-realview'
svn: Can't change perms of file 'Kernel.9TKDNMQTDm/arch/arm/mach-realview/.svn/text-base/Makefile.boot.svn-base':
Permission denied
Tue May 11 12:57:51 CEST 2010,1
runcounter: 1
vogt@pxe3[vogt_nfs]>uname -a
Linux pxe3.itwm.fhg.de 2.6.34-rc7-0.1-default #2 SMP Tue May 11 10:28:41 CEST 2010x86_64x86_64x86_64GNU/Linux
Usually the "permission denied" happens very early. In this case the first run.
regards,
Martin
Quoted 6 lines
Hello,
I tested 2.6.34-rc7 now and it has the problem too. Then I checked 2.6.24.7 which has the problem, but 2.6.23.1 looks fine.
Hello,
I searched a bit more in the git tree, when the "Permission denied" problems does occur first(for me).
I'm no expert in git. I used git bisect visualize to find a commit which should build and then tested if it works/works not.
The first commit which introduced the "Permission denied" problem, was:
1-run:
Quoted 5 lines
commit
77a55a1fe8f26f7d022986a599b68002e21d968a
NFS: Eliminatenfs_renew_times()
svn: Can't change perms of file 'Kernel-pxe3-1.h2cuuZBQlF/arch/h8300/platform/h8 s/.svn/log.1': Permission denied
When I go back one commit, the "Permission denied" disappears, but now I get an "Input/output error".
5-run:
Quoted 4 lines
commit
92f6c178250170222f6d80c8ae725400765aa7a4
"Don't callnfs_renew_times()innfs_dentry_iput()"
svn: Can't read file 'Kernel-pxe3-5.HQikkhUNHk/drivers/usb/storage/.svn/log': Input/output error
Thus it looks that the problem is now "I/O Error" instead of permission denied. And that I have to search when the I/O Error disappears?
Maybe someone has an idea what went wrong here?
All I can say that I have reproduced a similar svn problem with 2.6.33 as server/client and with 2.6.34-rc7 as client. And that on my system does not go away with sync, tcp, udp, noac, nosharecache.
But I cannot say that the problem in the 2.6.33 (server/client) combination is the same.
Because with this the svn error is "No such file or directory". The problem never existed on my 2.6.16 kernel from SLES10, but now every kernel >= 2.6.24 seems to be affected.
regards,
Martin
Quoted 7 lines · Martin Vogt
Quoted 6 lines
Hello,
I tested 2.6.34-rc7 now and it has the problem too. Then I checked 2.6.24.7 which has the problem, but 2.6.23.1 looks fine.
Hello,
2.6.23.1 down to 2.6.20.1 has the same problem and 2.6.16.<vendor> too.
But its seems that, the older the kernel gets, the longer it takes that the EPERM or EIO appears.
For example: 2.6.34-rc7 is really quick ;-)
But now I have no more ideas for debugging.
regards,
Martin
PS: here a two more straces:
1:
==
13037 open("Kernel-pxe2-1.JGD9NGERWJ/arch/ppc/syslib/pci_auto.c.tmp", O_RDWR|O_CREAT|O_EXCL, 0666) = 6
13037 read(5, "/*\n * arch/ppc/syslib/pci_auto.c"..., 16384) = 12365
13037 write(6, "/*\n * arch/ppc/syslib/pci_auto.c"..., 4096) = 4096
13037 write(6, ",\n\t\t\tcurrent_bus,\n\t\t\tpci_devfn,\n"..., 4096) = 4096 13037 write(6, "e behind a\n\t * P2P bridge in a s"..., 4096) = 4096
13037 read(5, "", 16384) = 0
13037 close(5) = 0
13037 write(6, "_devfn,\n\t\t\t\t\t\tPCI_LATENCY_TIMER,"..., 77) = 77
13037 close(6) = 0
13037 rename("Kernel-pxe2-1.JGD9NGERWJ/arch/ppc/syslib/pci_auto.c.tmp", "Kernel-pxe2-1.JGD9NGERWJ/arch/ppc/syslib/pci_auto.c") = 0
13037 rename("Kernel-pxe2-1.JGD9NGERWJ/arch/ppc/syslib/.svn/tmp/text-base/pci_auto.c.svn-base", "Kernel-pxe2-1.JGD9NGERWJ/arch/ppc/sysli
b/.svn/text-base/pci_auto.c.svn-base") = 0
13037 lstat("Kernel-pxe2-1.JGD9NGERWJ/arch/ppc/syslib/.svn/text-base/pci_auto.c.svn-base", {st_mode=S_IFREG|0666, st_size=12365, ...}) =
0
13037 chmod("Kernel-pxe2-1.JGD9NGERWJ/arch/ppc/syslib/.svn/text-base/pci_auto.c.svn-base", 0444) = -1 EACCES (Permission denied)
^^^^^^^^^^^^^^^^^
Here
13037 fcntl(3, F_GETFL) = 0x2 (flags O_RDWR)
13037 fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
13037 write(3, "( failure ( ( 155009 55:In direc"..., 423) = 423
2:
==
13048 open("Kernel-pxe2-1.5MzYLADYdV/arch/ppc/kernel/traps.c.tmp", O_RDWR|O_CREAT|O_EXCL, 0666) = 6
13048 read(5, "/*\n * arch/ppc/kernel/traps.c\n "..., 16384) = 16384
13048 write(6, "/*\n * arch/ppc/kernel/traps.c\n "..., 4096) = 4096
13048 write(6, "ble.\n * Note that the 601 only t"..., 4096) = 4096
13048 write(6, "CSR_IMPE)\n\t\t\tprintk(\"Machine Che"..., 4096) = 4096
13048 read(5, ", bug->file,\n\t\t bug->line "..., 16384) = 9123
13048 write(6, "\t\t0x7c0007fe\n\n#define INST_STRIN"..., 4096) = 4096
13048 write(6, ", bug->file,\n\t\t bug->line "..., 4096) = 4096
13048 write(6, "if (!user_mode(regs)) {\n\t\tdebugg"..., 4096) = 4096
13048 read(5, "", 16384) = 0
13048 close(5) = 0
13048 write(6, "UND)) {\n\t\tcode = FPE_FLTUND;\n\t\ts"..., 931) = 931
13048 close(6) = 0
13048 rename("Kernel-pxe2-1.5MzYLADYdV/arch/ppc/kernel/traps.c.tmp", "Kernel-pxe2-1.5MzYLADYdV/arch/ppc/kernel/traps.c") = 0
13048 rename("Kernel-pxe2-1.5MzYLADYdV/arch/ppc/kernel/.svn/tmp/text-base/traps.c.svn-base", "Kernel-pxe2-1.5MzYLADYdV/arch/ppc/kernel/.svn/text-base/traps.c.svn-base") = 0 13048 lstat("Kernel-pxe2-1.5MzYLADYdV/arch/ppc/kernel/.svn/text-base/traps.c.svn-base", 0x7fff57f65820) = -1 EACCES (Permission denied)
^^^^^^^^^^^^^^^
Here
13048 fcntl(3, F_GETFL) = 0x2 (flags O_RDWR)
13048 fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
13048 write(3, "( failure ( ( 155009 55:In direc"..., 420) = 420
A message that is not in the archive (sent off-list or not yet received): 20100516175213.GA9225-DA0fgfH9LvLH7LAFlePENYQuADTiUCJX@public.gmane.org
Hello,
today I tried 2.6.34 on server/client with standard hardware. After around the 69th checkout I had an "Invalid cross-device link" Error.
(Before that it was EIO/EPERM)
Test ran for around 9 hours before it failed.
Quoted 2 lines
svn: In directory 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc'
svn: Can't move
'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h.tmp'
Quoted 4 lines
to '
Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h': Invalid cross-device link
Mon May 17 23:46:32 CEST 2010,1
runcounter: 69
Martin
Quoted 8 lines · Martin Vogt
Hello,
today I tried 2.6.34 on server/client with standard hardware. After around the 69th checkout I had an "Invalid cross-device link" Error.
(Before that it was EIO/EPERM)
Quoted 10 lines
Test ran for around 9 hours before it failed.
Quoted 2 lines
svn: In directory 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc'
svn: Can't move'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h.tmp'
Quoted 4 lines
to '
Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h': Invalid cross-device link
Mon May 17 23:46:32 CEST 2010,1
runcounter: 69
Ok.
RTFM: no_subtree_check :-(
This option made the checkouts reliable.
Martin
Quoted 20 lines · Martin Vogt
Quoted 8 lines · Martin Vogt
Hello,
today I tried 2.6.34 on server/client with standard hardware. After around the 69th checkout I had an "Invalid cross-device link" Error.
(Before that it was EIO/EPERM)Quoted 10 lines
Test ran for around 9 hours before it failed.
Quoted 2 lines
svn: In directory 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc'
svn: Can't move'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h.tmp'
Quoted 4 lines
to '
Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h': Invalid cross-device link
Mon May 17 23:46:32 CEST 2010,1
runcounter: 69
Quoted 4 lines
Ok.
RTFM:no_subtree_check:-(This option made the checkouts reliable.
Erp. Should have thought of that. It's no longer the default in recent nfs-utils, for this sort of reason.
(But, note: for anyone exporting directories that aren't mountpoints, that may expose more of their filesystem than intended.)
--b.
Quoted 22 lines · J. Bruce Fields
Quoted 21 lines · Martin Vogt
Quoted 8 lines · Martin Vogt
Hello,
today I tried 2.6.34 on server/client with standard hardware. After around the 69th checkout I had an "Invalid cross-device link" Error.
(Before that it was EIO/EPERM)Quoted 9 lines
Test ran for around 9 hours before it failed.
Quoted 2 lines
svn: In directory 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc'
svn: Can't move'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h.tmp'
Quoted 4 lines
to '
Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h': Invalid cross-device link
Mon May 17 23:46:32 CEST 2010,1
runcounter: 69Ok.
RTFM:no_subtree_check:-(
Quoted 8 lines
Quoted 1 line
This option made the checkouts reliable.
Erp. Should have thought of that. It's no longer the default in recent nfs-utils, for this sort of reason.
(But, note: for anyone exporting directories that aren't mountpoints, that may expose more of their filesystem than intended.)
Hello,
the report for the latest kernel was done with "no_subtree_check"
-9 hours client/server both 2.6.34 before "Invalid cross-device link"
2.6.34 still shows some svn problem with "no_subtree_check" on. But only after 9 hours (vs 10 minutes, with subtree_check)
It was done with:
Quoted 1 line
log:Assuming default behaviour ('
no_subtree_check').
exports:
Quoted 2 lines
/var/tmp/nfs_export192.168.9.0/255.255.255.0(rw,async,wdelay,acl,no_root_squash,fsid=6666)
Using "no_subtree_check" made it only "much more" stable than before, but not "stable".
Server was 4 core (2 sockets) Xeon [email protected],
8GB mem.
Exported fs was ext3 from a Standard Sata drive,
and the client was a dual core E8400.
Mount client:
mount -o rw,nosuid,nodev,rsize=8192,wsize=8192,vers=3 -t nfs pxe2:/var/tmp/nfs_export /home/local
I'm attaching my test script. The script needs to be run two times in parallel on the machine. (In screen for example).
If someone wants to reproduce this.A single checkout on the machine would even takes longer to reproduce the problem.
I imported a normal kernel tree into a svn repository on a dedicated server only for this and started svnserve -d as normal user.
regards,
Martin
Quoted 28 lines · Martin Vogt
Quoted 27 lines · J. Bruce Fields
Quoted 14 lines · Martin Vogt
Quoted 13 lines · Martin Vogt
Hello,
today I tried 2.6.34 on server/client with standard hardware. After around the 69th checkout I had an "Invalid cross-device link" Error.
(Before that it was EIO/EPERM)Test ran for around 9 hours before it failed.
Quoted 2 lines
svn: In directory 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc'
svn: Can't move'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h.tmp'
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
to '
Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h': Invalid cross-device link
Mon May 17 23:46:32 CEST 2010,1
runcounter: 69Ok.
RTFM:no_subtree_check:-(This option made the checkouts reliable.
Erp. Should have thought of that. It's no longer the default in recent nfs-utils, for this sort of reason.
Quoted 7 lines
Quoted 3 lines
(But, note: for anyone exporting directories that aren't mountpoints, that may expose more of their filesystem than intended.)
Hello,
the report for the latest kernel was done with "
no_subtree_check"
OK, sorry, I don't know.
You've been varying both client and server, but from your initial reports it sounded like this was something that first started happening on a client upgrade?
Have you been able to keep the server fixed while doing a binary search through the client version for the first non-working client?
--b.
Quoted 40 lines
-9 hours client/server both 2.6.34 before "Invalid cross-device link"
2.6.34 still shows some svn problem with "
no_subtree_check" on. But only after 9 hours (vs 10 minutes, withsubtree_check)It was done with:
Quoted 1 line
log:Assuming default behaviour ('
no_subtree_check').exports:
Quoted 2 lines
/var/tmp/nfs_export192.168.9.0/255.255.255.0(rw,async,wdelay,acl,no_root_squash,fsid=6666)Using "
no_subtree_check" made it only "much more" stable than before, but not "stable".Server was 4 core (2 sockets) Xeon [email protected],
8GB mem.
Exported fs was ext3 from a Standard Sata drive,
and the client was a dual core E8400.Mount client:
mount -o rw,nosuid,nodev,rsize=8192,wsize=8192,vers=3 -t nfs pxe2:/var/tmp/nfs_export/home/localI'm attaching my test script. The script needs to be run two times in parallel on the machine. (In screen for example).
If someone wants to reproduce this.A single checkout on the machine would even takes longer to reproduce the problem.I imported a normal kernel tree into a svn repository on a dedicated server only for this and started svnserve -d as normal user.
regards,
Martin
Quoted 33 lines · J. Bruce Fields
Quoted 32 lines · Martin Vogt
Quoted 22 lines · J. Bruce Fields
Quoted 21 lines · Martin Vogt
Quoted 8 lines · Martin Vogt
Hello,
today I tried 2.6.34 on server/client with standard hardware. After around the 69th checkout I had an "Invalid cross-device link" Error.
(Before that it was EIO/EPERM)Quoted 9 lines
Test ran for around 9 hours before it failed.
Quoted 2 lines
svn: In directory 'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc'
svn: Can't move'Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h.tmp'
Quoted 4 lines
to '
Kernel-pxe3-69.EZvgHTmbzv/include/asm-sparc/resource.h': Invalid cross-device link
Mon May 17 23:46:32 CEST 2010,1
runcounter: 69Ok.
RTFM:no_subtree_check:-(Quoted 7 lines
Quoted 1 line
This option made the checkouts reliable.
Erp. Should have thought of that. It's no longer the default in recent nfs-utils, for this sort of reason.
(But, note: for anyone exporting directories that aren't mountpoints, that may expose more of their filesystem than intended.)
Hello,
Quoted 8 lines
Quoted 1 line
the report for the latest kernel was done with "
no_subtree_check"OK, sorry, I don't know.
You've been varying both client and server, but from your initial reports it sounded like this was something that first started happening on a client upgrade?
Yes. I tried to go back with the kernels on the client. (and keep the server fixed)
But up to 2.6.16 I had this svn error. Up to this time I didn't know that no_subtree_check would increase the stability that much.
All my tests were done with "subtree_check", except the tests with 2.6.34 kernel on server/client.
On this server no_subtree_check was the default because of newer nfs-utils.
With this kernel it took ~9hours until I got the "Invalid cross-device link".
Quoted 2 lines
Have you been able to keep the server fixed while doing a binary search through the client version for the first non-working client?
No, test were done with "subtree_check" and as first I thought that older clients are stable, but that's not true. It only takes more time until svn aborts.
But the abort after 9hours was only one tests. I will start the test again and see how long it takes until it aborts.
Martin