possible bug in nfs-kernel-server

Hi, I'm Omar Walid Llorente and I am a systems administrator at the Politechnical University of Madrid (UPM), Spain. I write you in the hope you can help us manage a problem that have discovered recently about our new datastore architecture in our teaching labs. We have created a gluster distributed volume that we reexport with NFS to our lab clients via intermediate servers.

First of all thanks for all your work and sorry if this isn't related with your package, but I think it has a good chance. I'll try explain myself as short as possible.

As introduced previously, we have a problem exporting with nfs-kernel-server-1.2.8-6 (ubuntu based) a directory previously mounted with gluster-3.7.4 via fuse mount.

The problem is quite simple to reproduce and always repeatable: if a file has read-only permissions for owner and user wants to copy it, permissions problem arises:

cdc@client:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod
444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied

(quoted diff) +0 -2

-r--r--r-- 1 cdc admincdc 7 nov  3  2015 444.txt
-r--r--r-- 1 cdc admincdc 0 nov  3  2015 kk.txt

cdc@client:~$

If the file permissions are not read-only, there is no problem:

cdc@client:~$ rm -f kk.txt 644.txt; echo "prueba" > 644.txt; chmod
644 644.txt; cp -p 644.txt kk.txt; ls -ld 644.txt kk.txt

(quoted diff) +0 -2

-rw-r--r-- 1 cdc admincdc 7 nov  3  2015 644.txt
-rw-r--r-- 1 cdc admincdc 7 nov  3  2015 kk.txt

cdc@client:~$

If we track it down with strace, the problem arises exactly when fsync() is called from cp.

Of course, if we try this combination of commands in other directories not mounted by nfs (local ones) or mounted with samba/cifs or even mounted with nfs-ganesha (both fuse mounted with gluster), this doen't happen. This problem doesn't happen either if the nfs-kernel-server exports a directory not mounted with fuse (any local one).

Please, tell me if this is the right place to post the probem and where is it if this is not. Let me know if we can help you any way to solve or test it (we've developed a small program in c that shows exactly the same behaviour).

LOGS ON SERVER SIDE (glusterfs mount logs):
[2015-11-20 10:51:53.872656] I [io-stats.c:1014:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2015-11-20 10:51:53.872692] I [io-stats.c:1019:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2015-11-20 10:51:53.872704] I [io-stats.c:1034:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2015-11-20 10:51:53.872714] I [io-stats.c:1046:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2015-11-20 10:51:53.874917] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-0: remote operation failed [Permission denied] [2015-11-20 10:51:53.874976] W [fuse-bridge.c:1230:fuse_err_cbk]

0-glusterfs-fuse: 63459954: REMOVEXATTR() /cdc/444.txt => -1 (Permission denied)
[2015-11-20 10:51:53.881389] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied] [2015-11-20 10:51:53.881434] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459961: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)

[2015-11-20 10:51:53.883072] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459964: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.883057] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied] [2015-11-20 10:51:53.884003] E [MSGID: 114031] [client-rpc-fops.c:466:client3_3_open_cbk] 0-home-lab-3-client-3:
remote operation failed. Path: /cdc/kk.txt (3175e0cd-8308-45b8-a4b0-699f6f8cf37f) [Permission denied] [2015-11-20 10:51:53.884056] W [fuse-bridge.c:969:fuse_fd_cbk] 0-glusterfs-fuse: 63459965: OPEN() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.885619] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied] [2015-11-20 10:51:53.885664] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459967: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.887908] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459971: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.887891] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied]

(NOTE: We have more gluster brick logs but we don't know if are relevant)

Signature · 13 lines

----------------------------------------------------------------
Centro de Cálculo         Depto. Ingeniería Sistemas Telemáticos
E-mail: [email protected]        Universidad Politécnica de Madrid
Fax:(+34) 913367333                 E.T.S. Ing. Telecomunicación
Tel:(+34) 915495700-Ext.3005                28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 33 lines · Omar Walid Llorente

Hi, I'm Omar Walid Llorente and I am a systems administrator at the Politechnical University of Madrid (UPM), Spain. I write you in the hope you can help us manage a problem that have discovered recently about our new datastore architecture in our teaching labs. We have created a gluster distributed volume that we reexport with NFS to our lab clients via intermediate servers.

First of all thanks for all your work and sorry if this isn't related with your package, but I think it has a good chance. I'll try explain myself as short as possible.

As introduced previously, we have a problem exporting with nfs-kernel-server-1.2.8-6 (ubuntu based) a directory previously mounted with gluster-3.7.4 via fuse mount.

The problem is quite simple to reproduce and always repeatable: if a file has read-only permissions for owner and user wants to copy it, permissions problem arises:

cdc@client:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod
444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied

(quoted diff) +0 -2

-r--r--r-- 1 cdc admincdc 7 nov  3  2015 444.txt
-r--r--r-- 1 cdc admincdc 0 nov  3  2015 kk.txt

cdc@client:~$

If the file permissions are not read-only, there is no problem:

cdc@client:~$ rm -f kk.txt 644.txt; echo "prueba" > 644.txt; chmod
644 644.txt; cp -p 644.txt kk.txt; ls -ld 644.txt kk.txt

(quoted diff) +0 -2

-rw-r--r-- 1 cdc admincdc 7 nov  3  2015 644.txt
-rw-r--r-- 1 cdc admincdc 7 nov  3  2015 kk.txt

cdc@client:~$

Quoted 2 lines

If we track it down with strace, the problem arises exactly when fsync() is called from cp.

So, cp is calling fsync on kk.txt before closing it?

Quoted 11 lines

Of course, if we try this combination of commands in other directories not mounted by nfs (local ones) or mounted with samba/cifs or even mounted with nfs-ganesha (both fuse mounted with gluster), this doen't happen. This problem doesn't happen either if the nfs-kernel-server exports a directory not mounted with fuse (any local one).

Please, tell me if this is the right place to post the probem and where is it if this is not. Let me know if we can help you any way to solve or test it (we've developed a small program in c that shows exactly the same behaviour).

It'd be interesting to see exactly which NFS operation is failing. You should be able to do this by running wireshark to sniff the NFS client/server traffic while running the reproducer, then reading through the traffic to find the failing operation.

I'm also curious:

Quoted 13 lines

LOGS ON SERVER SIDE (glusterfs mount logs):
[2015-11-20 10:51:53.872656] I [io-stats.c:1014:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2015-11-20 10:51:53.872692] I [io-stats.c:1019:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2015-11-20 10:51:53.872704] I [io-stats.c:1034:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2015-11-20 10:51:53.872714] I [io-stats.c:1046:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2015-11-20 10:51:53.874917] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-0: remote operation failed [Permission denied] [2015-11-20 10:51:53.874976] W [fuse-bridge.c:1230:fuse_err_cbk]

Quoted 8 lines

0-glusterfs-fuse: 63459954: REMOVEXATTR() /cdc/444.txt => -1 (Permission denied)
[2015-11-20 10:51:53.881389] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied] [2015-11-20 10:51:53.881434] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459961: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)

I wonder what this is about? Why would an fsync on the NFS client result in a REMOVEXATTR on the server side?

--b.

Quoted 42 lines

[2015-11-20 10:51:53.883072] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459964: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.883057] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied] [2015-11-20 10:51:53.884003] E [MSGID: 114031] [client-rpc-fops.c:466:client3_3_open_cbk] 0-home-lab-3-client-3:
remote operation failed. Path: /cdc/kk.txt (3175e0cd-8308-45b8-a4b0-699f6f8cf37f) [Permission denied] [2015-11-20 10:51:53.884056] W [fuse-bridge.c:969:fuse_fd_cbk] 0-glusterfs-fuse: 63459965: OPEN() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.885619] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied] [2015-11-20 10:51:53.885664] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459967: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.887908] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459971: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.887891] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied]

(NOTE: We have more gluster brick logs but we don't know if are relevant)

--
----------------------------------------------------------------
Centro de Cálculo Depto. Ingeniería Sistemas Telemáticos
E-mail: [email protected] Universidad Politécnica de Madrid
Fax:(+34) 913367333 E.T.S. Ing. Telecomunicación
Tel:(+34) 915495700-Ext.3005 28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 8 lines · On Fri, 20 Nov 2015 12:04:49 +0100 Omar Walid Llorente <[email protected]> wrote:

Hi, I'm Omar Walid Llorente and I am a systems administrator at the Politechnical University of Madrid (UPM), Spain. I write you in the hope you can help us manage a problem that have discovered recently about our new datastore architecture in our teaching labs. We have created a gluster distributed volume that we reexport with NFS to our lab clients via intermediate servers.

Quoted 8 lines

First of all thanks for all your work and sorry if this isn't related with your package, but I think it has a good chance. I'll try explain myself as short as possible.

As introduced previously, we have a problem exporting with nfs-kernel-server-1.2.8-6 (ubuntu based) a directory previously mounted with gluster-3.7.4 via fuse mount.

What's important here (for the nfs server) is the kernel version. What kernel version are you running on the server? Also, what NFS version is the client using? If you grab the mount's line out of /proc/mounts on the client then that would be helpful.

Also, does the NFS version matter here? If you're using NFSv4 then maybe try with NFSv3, or with v4 or so if you're already using v3?

Quoted 17 lines

The problem is quite simple to reproduce and always repeatable: if a file has read-only permissions for owner and user wants to copy it, permissions problem arises:

cdc@client:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied

(quoted diff) +0 -2

-r--r--r-- 1 cdc admincdc 7 nov  3  2015 444.txt
-r--r--r-- 1 cdc admincdc 0 nov  3  2015 kk.txt

cdc@client:~$

If the file permissions are not read-only, there is no problem:

cdc@client:~$ rm -f kk.txt 644.txt; echo "prueba" > 644.txt; chmod 644
644.txt; cp -p 644.txt kk.txt; ls -ld 644.txt kk.txt

(quoted diff) +0 -2

-rw-r--r-- 1 cdc admincdc 7 nov  3  2015 644.txt
-rw-r--r-- 1 cdc admincdc 7 nov  3  2015 kk.txt

cdc@client:~$

Quoted 9 lines

If we track it down with strace, the problem arises exactly when fsync() is called from cp.

Of course, if we try this combination of commands in other directories not mounted by nfs (local ones) or mounted with samba/cifs or even mounted with nfs-ganesha (both fuse mounted with gluster), this doen't happen. This problem doesn't happen either if the nfs-kernel-server exports a directory not mounted with fuse (any local one).

Ok, that's good info, but when dealing with a problem like this, it'd be best to get a capture of the network traffic between client and server while you're reproducing this. We can then look at it to figure out which RPC call is getting the actual error. That will help narrow down the problem a bit more.

You can do that with tcpdump. Something like this should do it:

    # tcpdump -i eth0 -w /tmp/nfs.pcap -s 512 port 2049

...reproduce the problem and then stop the capture. Then you can open /tmp/nfs.pcap with wireshark to analyze it (or send it to me and I'll take a look).

Quoted 68 lines

Please, tell me if this is the right place to post the probem and where is it if this is not. Let me know if we can help you any way to solve or test it (we've developed a small program in c that shows exactly the same behaviour).

Thanks again.

Omar

PS: Pointer to this email address came from:
http://wiki.linux-nfs.org/wiki/index.php/Reporting_bugs

ADDITIONAL INFO:

cdc@client:~$ uname -a
Linux l056 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:43:30 UTC
2015 i686 i686 i686 GNU/Linux
cdc@client:~$
cdc@client:~$ mount | grep home
cuentas02:/home-3/cdc on /home/cdc type nfs

(rw,noatime,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.15) cdc@client:~$

root@server-lab:~# uname -a
Linux cuentas02-lab.lab.dit.upm.es 3.13.0-63-generic #103-Ubuntu SMP
Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@server-lab:~#
root@server-lab:~# dpkg -l | grep nfs
ii  libnfsidmap2:amd64 0.25-5                           amd64
NFS idmapping library
ii  nfs-common 1:1.2.8-6ubuntu1.1               amd64        NFS
support files common to client and server
ii  nfs-kernel-server 1:1.2.8-6ubuntu1.1               amd64
support for NFS kernel server
root@server-lab:~#
root@server-lab:~# exportfs -v
/home-3

138.4.30.0/23(rw,async,wdelay,insecure,no_root_squash,no_subtree_check,fsid=3,sec=sys,rw,no_root_squash,no_all_squash) root@server-lab:~#

LOGS ON SERVER SIDE (glusterfs mount logs):
[2015-11-20 10:51:53.872656] I [io-stats.c:1014:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2015-11-20 10:51:53.872692] I [io-stats.c:1019:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2015-11-20 10:51:53.872704] I [io-stats.c:1034:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2015-11-20 10:51:53.872714] I [io-stats.c:1046:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2015-11-20 10:51:53.874917] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-0: remote operation failed [Permission denied] [2015-11-20 10:51:53.874976] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459954: REMOVEXATTR() /cdc/444.txt => -1 (Permission denied)
[2015-11-20 10:51:53.881389] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied] [2015-11-20 10:51:53.881434] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459961: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.883072] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459964: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.883057] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk]

Quoted 8 lines

0-home-lab-3-client-3: remote operation failed [Permission denied] [2015-11-20 10:51:53.884003] E [MSGID: 114031] [client-rpc-fops.c:466:client3_3_open_cbk] 0-home-lab-3-client-3:
remote operation failed. Path: /cdc/kk.txt (3175e0cd-8308-45b8-a4b0-699f6f8cf37f) [Permission denied] [2015-11-20 10:51:53.884056] W [fuse-bridge.c:969:fuse_fd_cbk] 0-glusterfs-fuse: 63459965: OPEN() /cdc/kk.txt => -1 (Permission denied)

The above message is interesting and might be related to the problem. That said, we generally set the NFSD_MAY_OWNER_OVERRIDE bit on opens of regular files, which allows the nfsd_permission check to pass regardless when the owner matches.

My guess would be that the dentry_open call in nfsd_open is failing here as the concept of "owner override" doesn't really get passed down to it. Still, it'd be good to confirm that...

Quoted 15 lines

[2015-11-20 10:51:53.885619] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied] [2015-11-20 10:51:53.885664] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459967: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.887908] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459971: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.887891] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied]

(NOTE: We have more gluster brick logs but we don't know if are relevant)

Hi, Jeff, thanks for the answer. I'm out of the office until next week, but when I come back, I'll try to do the tests and send you the info.

Thank you very much,

Omar

Quoted 171 lines · Jeff Layton

Quoted 8 lines · On Fri, 20 Nov 2015 12:04:49 +0100 Omar Walid Llorente <[email protected]> wrote:

Hi, I'm Omar Walid Llorente and I am a systems administrator at the Politechnical University of Madrid (UPM), Spain. I write you in the hope you can help us manage a problem that have discovered recently about our new datastore architecture in our teaching labs. We have created a gluster distributed volume that we reexport with NFS to our lab clients via intermediate servers.

Quoted 8 lines

First of all thanks for all your work and sorry if this isn't related with your package, but I think it has a good chance. I'll try explain myself as short as possible.

As introduced previously, we have a problem exporting with nfs-kernel-server-1.2.8-6 (ubuntu based) a directory previously mounted with gluster-3.7.4 via fuse mount.

What's important here (for the nfs server) is the kernel version. What kernel version are you running on the server? Also, what NFS version is the client using? If you grab the mount's line out of /proc/mounts on the client then that would be helpful.

Also, does the NFS version matter here? If you're using NFSv4 then maybe try with NFSv3, or with v4 or so if you're already using v3?

Quoted 17 lines

The problem is quite simple to reproduce and always repeatable: if a file has read-only permissions for owner and user wants to copy it, permissions problem arises:

cdc@client:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied

(quoted diff) +0 -2

-r--r--r-- 1 cdc admincdc 7 nov  3  2015 444.txt
-r--r--r-- 1 cdc admincdc 0 nov  3  2015 kk.txt

cdc@client:~$

If the file permissions are not read-only, there is no problem:

cdc@client:~$ rm -f kk.txt 644.txt; echo "prueba" > 644.txt; chmod 644
644.txt; cp -p 644.txt kk.txt; ls -ld 644.txt kk.txt

(quoted diff) +0 -2

-rw-r--r-- 1 cdc admincdc 7 nov  3  2015 644.txt
-rw-r--r-- 1 cdc admincdc 7 nov  3  2015 kk.txt

cdc@client:~$

Quoted 9 lines

If we track it down with strace, the problem arises exactly when fsync() is called from cp.

Of course, if we try this combination of commands in other directories not mounted by nfs (local ones) or mounted with samba/cifs or even mounted with nfs-ganesha (both fuse mounted with gluster), this doen't happen. This problem doesn't happen either if the nfs-kernel-server exports a directory not mounted with fuse (any local one).

Ok, that's good info, but when dealing with a problem like this, it'd be best to get a capture of the network traffic between client and server while you're reproducing this. We can then look at it to figure out which RPC call is getting the actual error. That will help narrow down the problem a bit more.

You can do that with tcpdump. Something like this should do it:

    # tcpdump -i eth0 -w /tmp/nfs.pcap -s 512 port 2049

...reproduce the problem and then stop the capture. Then you can open /tmp/nfs.pcap with wireshark to analyze it (or send it to me and I'll take a look).

Quoted 69 lines

Please, tell me if this is the right place to post the probem and where is it if this is not. Let me know if we can help you any way to solve or test it (we've developed a small program in c that shows exactly the same behaviour).

Thanks again.

Omar

PS: Pointer to this email address came from:
http://wiki.linux-nfs.org/wiki/index.php/Reporting_bugs

ADDITIONAL INFO:

cdc@client:~$ uname -a
Linux l056 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:43:30 UTC
2015 i686 i686 i686 GNU/Linux
cdc@client:~$
cdc@client:~$ mount | grep home
cuentas02:/home-3/cdc on /home/cdc type nfs

(rw,noatime,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.15) cdc@client:~$

root@server-lab:~# uname -a
Linux cuentas02-lab.lab.dit.upm.es 3.13.0-63-generic #103-Ubuntu SMP
Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@server-lab:~#
root@server-lab:~# dpkg -l | grep nfs
ii  libnfsidmap2:amd64 0.25-5                           amd64
NFS idmapping library
ii  nfs-common 1:1.2.8-6ubuntu1.1               amd64 NFS
support files common to client and server
ii  nfs-kernel-server 1:1.2.8-6ubuntu1.1               amd64
support for NFS kernel server
root@server-lab:~#
root@server-lab:~# exportfs -v
/home-3

138.4.30.0/23(rw,async,wdelay,insecure,no_root_squash,no_subtree_check,fsid=3,sec=sys,rw,no_root_squash,no_all_squash)

root@server-lab:~#

LOGS ON SERVER SIDE (glusterfs mount logs):
[2015-11-20 10:51:53.872656] I [io-stats.c:1014:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2015-11-20 10:51:53.872692] I [io-stats.c:1019:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2015-11-20 10:51:53.872704] I [io-stats.c:1034:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2015-11-20 10:51:53.872714] I [io-stats.c:1046:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2015-11-20 10:51:53.874917] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-0: remote operation failed [Permission denied] [2015-11-20 10:51:53.874976] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459954: REMOVEXATTR() /cdc/444.txt => -1 (Permission denied)
[2015-11-20 10:51:53.881389] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied] [2015-11-20 10:51:53.881434] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459961: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.883072] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459964: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.883057] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk]

Quoted 8 lines

0-home-lab-3-client-3: remote operation failed [Permission denied] [2015-11-20 10:51:53.884003] E [MSGID: 114031] [client-rpc-fops.c:466:client3_3_open_cbk] 0-home-lab-3-client-3:
remote operation failed. Path: /cdc/kk.txt (3175e0cd-8308-45b8-a4b0-699f6f8cf37f) [Permission denied] [2015-11-20 10:51:53.884056] W [fuse-bridge.c:969:fuse_fd_cbk] 0-glusterfs-fuse: 63459965: OPEN() /cdc/kk.txt => -1 (Permission denied)

The above message is interesting and might be related to the problem. That said, we generally set the NFSD_MAY_OWNER_OVERRIDE bit on opens of regular files, which allows the nfsd_permission check to pass regardless when the owner matches.

My guess would be that the dentry_open call in nfsd_open is failing here as the concept of "owner override" doesn't really get passed down to it. Still, it'd be good to confirm that...

Quoted 15 lines

[2015-11-20 10:51:53.885619] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied] [2015-11-20 10:51:53.885664] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459967: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.887908] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459971: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.887891] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied]

(NOTE: We have more gluster brick logs but we don't know if are relevant)

Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs packets (you can find them in attached file nfs-problem-nks.pcap.zip). Sorry for being so late.

What I did was the following:

1st) Create the RO file:

cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:

root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

3rd) Try to copy the RO file and get the error:

cdc@l056:~/prueba-git$ cp -p 444.txt kk.txt;
cp: failed to close ‘kk.txt’: Permission denied
cdc@l056:~/prueba-git$

4th) Close the capture:
^C26 packets captured
26 packets received by filter
0 packets dropped by kernel
root@l056:~#

Hope you can send us some clue about it. Do you need me to do any other test? Thanks in advance!

Omar

Quoted 186 lines · omar

Hi, Jeff, thanks for the answer. I'm out of the office until next week, but when I come back, I'll try to do the tests and send you the info.

Thank you very much,

Omar

Quoted 175 lines · Jeff Layton

Quoted 10 lines · On Fri, 20 Nov 2015 12:04:49 +0100 Omar Walid Llorente <[email protected]> wrote:

Hi, I'm Omar Walid Llorente and I am a systems administrator at the Politechnical University of Madrid (UPM), Spain. I write you in the hope
you can help us manage a problem that have discovered recently about our
new datastore architecture in our teaching labs. We have created a gluster distributed volume that we reexport with NFS to our lab clients via intermediate servers.

Quoted 8 lines

First of all thanks for all your work and sorry if this isn't related with your package, but I think it has a good chance. I'll try explain myself as short as possible.

As introduced previously, we have a problem exporting with nfs-kernel-server-1.2.8-6 (ubuntu based) a directory previously mounted with gluster-3.7.4 via fuse mount.

What's important here (for the nfs server) is the kernel version. What kernel version are you running on the server? Also, what NFS version is the client using? If you grab the mount's line out of /proc/mounts on the client then that would be helpful.

Also, does the NFS version matter here? If you're using NFSv4 then maybe try with NFSv3, or with v4 or so if you're already using v3?

Quoted 21 lines

The problem is quite simple to reproduce and always repeatable: if a file has read-only permissions for owner and user wants to copy it, permissions problem arises:

cdc@client:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied

(quoted diff) +0 -2

-r--r--r-- 1 cdc admincdc 7 nov  3  2015 444.txt
-r--r--r-- 1 cdc admincdc 0 nov  3  2015 kk.txt

cdc@client:~$

If the file permissions are not read-only, there is no problem:

cdc@client:~$ rm -f kk.txt 644.txt; echo "prueba" > 644.txt; chmod 644
644.txt; cp -p 644.txt kk.txt; ls -ld 644.txt kk.txt

(quoted diff) +0 -2

-rw-r--r-- 1 cdc admincdc 7 nov  3  2015 644.txt
-rw-r--r-- 1 cdc admincdc 7 nov  3  2015 kk.txt

cdc@client:~$

If we track it down with strace, the problem arises exactly when fsync()
is called from cp.

Quoted 6 lines

Of course, if we try this combination of commands in other directories not mounted by nfs (local ones) or mounted with samba/cifs or even mounted with nfs-ganesha (both fuse mounted with gluster), this doen't happen. This problem doesn't happen either if the nfs-kernel-server exports a directory not mounted with fuse (any local one).

Ok, that's good info, but when dealing with a problem like this, it'd be best to get a capture of the network traffic between client and server while you're reproducing this. We can then look at it to figure out which RPC call is getting the actual error. That will help narrow down the problem a bit more.

You can do that with tcpdump. Something like this should do it:

    # tcpdump -i eth0 -w /tmp/nfs.pcap -s 512 port 2049

...reproduce the problem and then stop the capture. Then you can open /tmp/nfs.pcap with wireshark to analyze it (or send it to me and I'll take a look).

Quoted 69 lines

Please, tell me if this is the right place to post the probem and where is it if this is not. Let me know if we can help you any way to solve or test it (we've developed a small program in c that shows exactly the same behaviour).

Thanks again.

Omar

PS: Pointer to this email address came from:
http://wiki.linux-nfs.org/wiki/index.php/Reporting_bugs

ADDITIONAL INFO:

cdc@client:~$ uname -a
Linux l056 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:43:30 UTC
2015 i686 i686 i686 GNU/Linux
cdc@client:~$
cdc@client:~$ mount | grep home
cuentas02:/home-3/cdc on /home/cdc type nfs

(rw,noatime,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.15) cdc@client:~$

root@server-lab:~# uname -a
Linux cuentas02-lab.lab.dit.upm.es 3.13.0-63-generic #103-Ubuntu SMP
Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@server-lab:~#
root@server-lab:~# dpkg -l | grep nfs
ii  libnfsidmap2:amd64 0.25-5                           amd64
NFS idmapping library
ii  nfs-common 1:1.2.8-6ubuntu1.1               amd64 NFS
support files common to client and server
ii  nfs-kernel-server 1:1.2.8-6ubuntu1.1               amd64
support for NFS kernel server
root@server-lab:~#
root@server-lab:~# exportfs -v
/home-3

138.4.30.0/23(rw,async,wdelay,insecure,no_root_squash,no_subtree_check,fsid=3,sec=sys,rw,no_root_squash,no_all_squash)

root@server-lab:~#

LOGS ON SERVER SIDE (glusterfs mount logs):
[2015-11-20 10:51:53.872656] I [io-stats.c:1014:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2015-11-20 10:51:53.872692] I [io-stats.c:1019:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2015-11-20 10:51:53.872704] I [io-stats.c:1034:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2015-11-20 10:51:53.872714] I [io-stats.c:1046:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2015-11-20 10:51:53.874917] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-0: remote operation failed [Permission denied] [2015-11-20 10:51:53.874976] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459954: REMOVEXATTR() /cdc/444.txt => -1 (Permission denied)
[2015-11-20 10:51:53.881389] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied] [2015-11-20 10:51:53.881434] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459961: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.883072] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459964: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.883057] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk]

Quoted 8 lines

0-home-lab-3-client-3: remote operation failed [Permission denied] [2015-11-20 10:51:53.884003] E [MSGID: 114031] [client-rpc-fops.c:466:client3_3_open_cbk] 0-home-lab-3-client-3:
remote operation failed. Path: /cdc/kk.txt (3175e0cd-8308-45b8-a4b0-699f6f8cf37f) [Permission denied] [2015-11-20 10:51:53.884056] W [fuse-bridge.c:969:fuse_fd_cbk] 0-glusterfs-fuse: 63459965: OPEN() /cdc/kk.txt => -1 (Permission denied)

The above message is interesting and might be related to the problem. That said, we generally set the NFSD_MAY_OWNER_OVERRIDE bit on opens of regular files, which allows the nfsd_permission check to pass regardless when the owner matches.

My guess would be that the dentry_open call in nfsd_open is failing here as the concept of "owner override" doesn't really get passed down to it. Still, it'd be good to confirm that...

Quoted 16 lines

[2015-11-20 10:51:53.885619] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied] [2015-11-20 10:51:53.885664] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459967: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.887908] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 63459971: REMOVEXATTR() /cdc/kk.txt => -1 (Permission denied)
[2015-11-20 10:51:53.887891] W [MSGID: 114031] [client-rpc-fops.c:1298:client3_3_removexattr_cbk] 0-home-lab-3-client-3: remote operation failed [Permission denied]

(NOTE: We have more gluster brick logs but we don't know if are relevant)

Quoted 7 lines · Omar Walid Llorente

Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs packets (you can find them in attached file nfs-problem-nks.pcap.zip). Sorry for being so late.

What I did was the following:

Quoted 9 lines

1st) Create the RO file:

cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:

root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It may be helpful to see what GlusterFS operations are being processed as part of NFS WRITE call (which has failed in this case).

Could you please try taking the packet trace on the machine where NFS server is running (without filtering out based on the port number).

Also I tried out the same test on Fedora22 machine, but haven't run into any issue. What are the fuse mount options you have used to mount gluster volume?

Quoted 21 lines · Soumya Koduri

Quoted 6 lines · Omar Walid Llorente

Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs packets (you can find them in attached file nfs-problem-nks.pcap.zip). Sorry for being so late.

What I did was the following:

Quoted 9 lines

1st) Create the RO file:

cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:

root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It may be helpful to see what GlusterFS operations are being processed as part of NFS WRITE call (which has failed in this case).

Quoted 7 lines

Could you please try taking the packet trace on the machine where NFS server is running (without filtering out based on the port number).

Also I tried out the same test on Fedora22 machine, but haven't run into any issue. What are the fuse mount options you have used to mount gluster volume?

Oh, I think this is a simple problem (but maybe hard to fix). The capture shows NFSv3 traffic like:

	CREATE -> OK
	SETATTR (mode set to 0400) -> OK
	WRITE -> NFS3ERR_ACCES

That write would succeed locally (because the mode doesn't matter to a local application that already holds the file open). It would fail over NFSv3, which doesn't know about the open--except that there's a hack for this case: NFSv3 servers allow IO operations to ignore the mode, if the operation comes from the owner of the file. NFSv3 clients are then careful to perform necessary access checks on open to ensure that this owner-override rule doesn't grant too many permissions.

That allows NFSv3 applications to see behavior that's mostly like a local filesystem, without opening much of a security hole (since the owner could always chmod anyway).

So, knfsd is making this special exception--but gluster (which I believe it's exporting in this case, via fuse?)--probably doesn't.... I'm not sure what you can do about that.

--b.

Thank you Bruce, others, for the responses. I send attached a complete capture of the issue, including the glusterfs transactions.

Hope this helps to clear where may it be...

Omar

Quoted 53 lines · J. Bruce Fields

Quoted 21 lines · Soumya Koduri

Quoted 6 lines · Omar Walid Llorente

Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs packets (you can find them in attached file nfs-problem-nks.pcap.zip). Sorry for being so late.

What I did was the following:

Quoted 9 lines

1st) Create the RO file:

cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:

root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It may be helpful to see what GlusterFS operations are being processed as part of NFS WRITE call (which has failed in this case).

Quoted 7 lines

Could you please try taking the packet trace on the machine where NFS server is running (without filtering out based on the port number).

Also I tried out the same test on Fedora22 machine, but haven't run into any issue. What are the fuse mount options you have used to mount gluster volume?

Oh, I think this is a simple problem (but maybe hard to fix). The capture shows NFSv3 traffic like:

    CREATE -> OK
    SETATTR (mode set to 0400) -> OK
    WRITE -> NFS3ERR_ACCES

That write would succeed locally (because the mode doesn't matter to a local application that already holds the file open). It would fail over NFSv3, which doesn't know about the open--except that there's a hack for this case: NFSv3 servers allow IO operations to ignore the mode, if the operation comes from the owner of the file. NFSv3 clients are then careful to perform necessary access checks on open to ensure that this owner-override rule doesn't grant too many permissions.

That allows NFSv3 applications to see behavior that's mostly like a local filesystem, without opening much of a security hole (since the owner could always chmod anyway).

So, knfsd is making this special exception--but gluster (which I believe it's exporting in this case, via fuse?)--probably doesn't.... I'm not sure what you can do about that.

--b.

As mentioned by Bruce, GlusterFS doesn't have owner-override rule except for setattr.

I did few experiments to check why this test case passes on plain glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.

NFS-Ganesha (for most of the FSALs) seem to be passing the actual request credentials to the back-end filesystem only for CREATE(-like) and UNLINK fops. For all the remaining fops, it does the access check at its end and then perform the operation with root credentials. That's the reason WRITE succeeded in your case as NFS-Ganesha (like kernel-NFS) skipped the access check if the request caller_uid proved to be the file's owner.

In case of native GlusterFS FUSE mount, there is no OPEN fop involved. WRITE is performed on the fd returned by CREATE. And strangely GlusterFS seem to be doing certain access checks only during OPEN but not for WRITE (this seems like a bug and probably needs to be fixed in Gluster).

Thanks,
Soumya

Quoted 63 lines · Omar Walid Llorente

Thank you Bruce, others, for the responses. I send attached a complete capture of the issue, including the glusterfs transactions.

Hope this helps to clear where may it be...

Omar

Quoted 53 lines · J. Bruce Fields

Quoted 21 lines · Soumya Koduri

Quoted 6 lines · Omar Walid Llorente

Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs packets (you can find them in attached file nfs-problem-nks.pcap.zip). Sorry for being so late.

What I did was the following:

Quoted 9 lines

1st) Create the RO file:

cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:

root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It may be helpful to see what GlusterFS operations are being processed as part of NFS WRITE call (which has failed in this case).

Quoted 7 lines

Could you please try taking the packet trace on the machine where NFS server is running (without filtering out based on the port number).

Also I tried out the same test on Fedora22 machine, but haven't run into any issue. What are the fuse mount options you have used to mount gluster volume?

Oh, I think this is a simple problem (but maybe hard to fix). The capture shows NFSv3 traffic like:

   CREATE -> OK
   SETATTR (mode set to 0400) -> OK
   WRITE -> NFS3ERR_ACCES

That write would succeed locally (because the mode doesn't matter to a local application that already holds the file open). It would fail over NFSv3, which doesn't know about the open--except that there's a hack for this case: NFSv3 servers allow IO operations to ignore the mode, if the operation comes from the owner of the file. NFSv3 clients are then careful to perform necessary access checks on open to ensure that this owner-override rule doesn't grant too many permissions.

That allows NFSv3 applications to see behavior that's mostly like a local filesystem, without opening much of a security hole (since the owner could always chmod anyway).

So, knfsd is making this special exception--but gluster (which I believe it's exporting in this case, via fuse?)--probably doesn't.... I'm not sure what you can do about that.

--b.

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

IIRC, permission checks are done in open(). write/read syscalls should NOT do much access checks (at least based on POSIX). This is why once an open is done, you remove permissions for that process, but it should still be able to read/write based on the open flags it did when it opened the file.

I don't know all the details of this defect, but gluster seems to be doing what it is supposed to do.

Regards, Malahal.

Quoted 92 lines · Soumya Koduri [[email protected]] wrote:

As mentioned by Bruce, GlusterFS doesn't have owner-override rule except for setattr.

I did few experiments to check why this test case passes on plain glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.

NFS-Ganesha (for most of the FSALs) seem to be passing the actual request credentials to the back-end filesystem only for CREATE(-like) and UNLINK fops. For all the remaining fops, it does the access check at its end and then perform the operation with root credentials. That's the reason WRITE succeeded in your case as NFS-Ganesha (like kernel-NFS) skipped the access check if the request caller_uid proved to be the file's owner.

In case of native GlusterFS FUSE mount, there is no OPEN fop involved. WRITE is performed on the fd returned by CREATE. And strangely GlusterFS seem to be doing certain access checks only during OPEN but not for WRITE (this seems like a bug and probably needs to be fixed in Gluster).

Thanks,
Soumya

Quoted 63 lines · Omar Walid Llorente

Thank you Bruce, others, for the responses. I send attached a complete capture of the issue, including the glusterfs transactions.

Hope this helps to clear where may it be...

Omar

Quoted 53 lines · J. Bruce Fields

Quoted 21 lines · Soumya Koduri

Quoted 6 lines · Omar Walid Llorente

Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs packets (you can find them in attached file nfs-problem-nks.pcap.zip). Sorry for being so late.

What I did was the following:

Quoted 9 lines

1st) Create the RO file:

cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:

root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It may be helpful to see what GlusterFS operations are being processed as part of NFS WRITE call (which has failed in this case).

Quoted 7 lines

Could you please try taking the packet trace on the machine where NFS server is running (without filtering out based on the port number).

Also I tried out the same test on Fedora22 machine, but haven't run into any issue. What are the fuse mount options you have used to mount gluster volume?

Oh, I think this is a simple problem (but maybe hard to fix). The capture shows NFSv3 traffic like:

    CREATE -> OK
    SETATTR (mode set to 0400) -> OK
    WRITE -> NFS3ERR_ACCES

That write would succeed locally (because the mode doesn't matter to a local application that already holds the file open). It would fail over NFSv3, which doesn't know about the open--except that there's a hack for this case: NFSv3 servers allow IO operations to ignore the mode, if the operation comes from the owner of the file. NFSv3 clients are then careful to perform necessary access checks on open to ensure that this owner-override rule doesn't grant too many permissions.

That allows NFSv3 applications to see behavior that's mostly like a local filesystem, without opening much of a security hole (since the owner could always chmod anyway).

So, knfsd is making this special exception--but gluster (which I believe it's exporting in this case, via fuse?)--probably doesn't.... I'm not sure what you can do about that.

--b.

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 6 lines · Malahal Naineni

IIRC, permission checks are done in open(). write/read syscalls should NOT do much access checks (at least based on POSIX). This is why once an open is done, you remove permissions for that process, but it should still be able to read/write based on the open flags it did when it opened the file.

Quoted 3 lines

I don't know all the details of this defect, but gluster seems to be doing what it is supposed to do.

Right. Thanks for the correction. I assumed the behavior should be same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But looks like (from 'man creat') the open() call that creates a read-only file may well return a read/write file descriptor, which is the reason the following WRITE can succeed.

Thanks,
Soumya

Quoted 96 lines

Regards, Malahal.

Quoted 92 lines · Soumya Koduri [[email protected]] wrote:

As mentioned by Bruce, GlusterFS doesn't have owner-override rule except for setattr.

I did few experiments to check why this test case passes on plain glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.

NFS-Ganesha (for most of the FSALs) seem to be passing the actual request credentials to the back-end filesystem only for CREATE(-like) and UNLINK fops. For all the remaining fops, it does the access check at its end and then perform the operation with root credentials. That's the reason WRITE succeeded in your case as NFS-Ganesha (like kernel-NFS) skipped the access check if the request caller_uid proved to be the file's owner.

In case of native GlusterFS FUSE mount, there is no OPEN fop involved. WRITE is performed on the fd returned by CREATE. And strangely GlusterFS seem to be doing certain access checks only during OPEN but not for WRITE (this seems like a bug and probably needs to be fixed in Gluster).

Thanks,
Soumya

Quoted 63 lines · Omar Walid Llorente

Thank you Bruce, others, for the responses. I send attached a complete capture of the issue, including the glusterfs transactions.

Hope this helps to clear where may it be...

Omar

Quoted 53 lines · J. Bruce Fields

Quoted 21 lines · Soumya Koduri

Quoted 6 lines · Omar Walid Llorente

Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs packets (you can find them in attached file nfs-problem-nks.pcap.zip). Sorry for being so late.

What I did was the following:

Quoted 9 lines

1st) Create the RO file:

cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:

root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It may be helpful to see what GlusterFS operations are being processed as part of NFS WRITE call (which has failed in this case).

Quoted 7 lines

Could you please try taking the packet trace on the machine where NFS server is running (without filtering out based on the port number).

Also I tried out the same test on Fedora22 machine, but haven't run into any issue. What are the fuse mount options you have used to mount gluster volume?

Oh, I think this is a simple problem (but maybe hard to fix). The capture shows NFSv3 traffic like:

   CREATE -> OK
   SETATTR (mode set to 0400) -> OK
   WRITE -> NFS3ERR_ACCES

That write would succeed locally (because the mode doesn't matter to a local application that already holds the file open). It would fail over NFSv3, which doesn't know about the open--except that there's a hack for this case: NFSv3 servers allow IO operations to ignore the mode, if the operation comes from the owner of the file. NFSv3 clients are then careful to perform necessary access checks on open to ensure that this owner-override rule doesn't grant too many permissions.

That allows NFSv3 applications to see behavior that's mostly like a local filesystem, without opening much of a security hole (since the owner could always chmod anyway).

So, knfsd is making this special exception--but gluster (which I believe it's exporting in this case, via fuse?)--probably doesn't.... I'm not sure what you can do about that.

--b.

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 9 lines · Soumya Koduri

Quoted 6 lines · Malahal Naineni

IIRC, permission checks are done in open(). write/read syscalls should NOT do much access checks (at least based on POSIX). This is why once an open is done, you remove permissions for that process, but it should still be able to read/write based on the open flags it did when it opened the file.

Quoted 8 lines

Quoted 3 lines

I don't know all the details of this defect, but gluster seems to be doing what it is supposed to do.

Right. Thanks for the correction. I assumed the behavior should be same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But looks like (from 'man creat') the open() call that creates a read-only file may well return a read/write file descriptor, which is the reason the following WRITE can succeed.

I forgot another complication, which is that knsfd actually does a temporary open before each read or write--I assume that's getting translated into fuse and gluster open operations?

In which case it might be worth experimenting with NFSv4 or with Jeff Layton's filehandle-caching patches. Neither's a real fix, but that could help confirm whether it's the temporary opens that are a problem.

--b.

Quoted 101 lines

Thanks,
Soumya

Quoted 96 lines

Regards, Malahal.

Quoted 92 lines · Soumya Koduri [[email protected]] wrote:

As mentioned by Bruce, GlusterFS doesn't have owner-override rule except for setattr.

I did few experiments to check why this test case passes on plain glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.

NFS-Ganesha (for most of the FSALs) seem to be passing the actual request credentials to the back-end filesystem only for CREATE(-like) and UNLINK fops. For all the remaining fops, it does the access check at its end and then perform the operation with root credentials. That's the reason WRITE succeeded in your case as NFS-Ganesha (like kernel-NFS) skipped the access check if the request caller_uid proved to be the file's owner.

In case of native GlusterFS FUSE mount, there is no OPEN fop involved. WRITE is performed on the fd returned by CREATE. And strangely GlusterFS seem to be doing certain access checks only during OPEN but not for WRITE (this seems like a bug and probably needs to be fixed in Gluster).

Thanks,
Soumya

Quoted 63 lines · Omar Walid Llorente

Thank you Bruce, others, for the responses. I send attached a complete capture of the issue, including the glusterfs transactions.

Hope this helps to clear where may it be...

Omar

Quoted 53 lines · J. Bruce Fields

Quoted 21 lines · Soumya Koduri

Quoted 6 lines · Omar Walid Llorente

Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs packets (you can find them in attached file nfs-problem-nks.pcap.zip). Sorry for being so late.

What I did was the following:

Quoted 9 lines

1st) Create the RO file:

cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:

root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It may be helpful to see what GlusterFS operations are being processed as part of NFS WRITE call (which has failed in this case).

Quoted 7 lines

Could you please try taking the packet trace on the machine where NFS server is running (without filtering out based on the port number).

Also I tried out the same test on Fedora22 machine, but haven't run into any issue. What are the fuse mount options you have used to mount gluster volume?

Oh, I think this is a simple problem (but maybe hard to fix). The capture shows NFSv3 traffic like:

    CREATE -> OK
    SETATTR (mode set to 0400) -> OK
    WRITE -> NFS3ERR_ACCES

That write would succeed locally (because the mode doesn't matter to a local application that already holds the file open). It would fail over NFSv3, which doesn't know about the open--except that there's a hack for this case: NFSv3 servers allow IO operations to ignore the mode, if the operation comes from the owner of the file. NFSv3 clients are then careful to perform necessary access checks on open to ensure that this owner-override rule doesn't grant too many permissions.

That allows NFSv3 applications to see behavior that's mostly like a local filesystem, without opening much of a security hole (since the owner could always chmod anyway).

So, knfsd is making this special exception--but gluster (which I believe it's exporting in this case, via fuse?)--probably doesn't.... I'm not sure what you can do about that.

--b.

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 19 lines · J. Bruce Fields

Quoted 17 lines · Soumya Koduri

Quoted 9 lines · Malahal Naineni

IIRC, permission checks are done in open(). write/read syscalls should NOT do much access checks (at least based on POSIX). This is why once an open is done, you remove permissions for that process, but it should still be able to read/write based on the open flags it did when it opened the file.

I don't know all the details of this defect, but gluster seems to be doing what it is supposed to do.

Right. Thanks for the correction. I assumed the behavior should be same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But looks like (from 'man creat') the open() call that creates a read-only file may well return a read/write file descriptor, which is the reason the following WRITE can succeed.

Quoted 4 lines

I forgot another complication, which is that knsfd actually does a temporary open before each read or write--I assume that's getting translated into fuse and gluster open operations?

yes. It is the OPEN done as part of NFS WRITE which fails with EACCESS error (with both NFSv3 and NFSv4 mounts).

63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS
164 V330 OPEN Call
65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS
164 [RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
116 V330 OPEN Reply (Call In 64)
67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
116 [RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call In 64)
68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
Reply (Call In 63) Error: NFS3ERR_ACCES

Thanks,
Soumya

Quoted 107 lines

In which case it might be worth experimenting with NFSv4 or with Jeff Layton's filehandle-caching patches. Neither's a real fix, but that could help confirm whether it's the temporary opens that are a problem.

--b.

Quoted 101 lines

Thanks,
Soumya

Quoted 96 lines

Regards, Malahal.

Quoted 92 lines · Soumya Koduri [[email protected]] wrote:

As mentioned by Bruce, GlusterFS doesn't have owner-override rule except for setattr.

I did few experiments to check why this test case passes on plain glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.

NFS-Ganesha (for most of the FSALs) seem to be passing the actual request credentials to the back-end filesystem only for CREATE(-like) and UNLINK fops. For all the remaining fops, it does the access check at its end and then perform the operation with root credentials. That's the reason WRITE succeeded in your case as NFS-Ganesha (like kernel-NFS) skipped the access check if the request caller_uid proved to be the file's owner.

In case of native GlusterFS FUSE mount, there is no OPEN fop involved. WRITE is performed on the fd returned by CREATE. And strangely GlusterFS seem to be doing certain access checks only during OPEN but not for WRITE (this seems like a bug and probably needs to be fixed in Gluster).

Thanks,
Soumya

Quoted 63 lines · Omar Walid Llorente

Thank you Bruce, others, for the responses. I send attached a complete capture of the issue, including the glusterfs transactions.

Hope this helps to clear where may it be...

Omar

Quoted 53 lines · J. Bruce Fields

Quoted 21 lines · Soumya Koduri

Quoted 6 lines · Omar Walid Llorente

Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs packets (you can find them in attached file nfs-problem-nks.pcap.zip). Sorry for being so late.

What I did was the following:

Quoted 9 lines

1st) Create the RO file:

cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:

root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It may be helpful to see what GlusterFS operations are being processed as part of NFS WRITE call (which has failed in this case).

Quoted 7 lines

Could you please try taking the packet trace on the machine where NFS server is running (without filtering out based on the port number).

Also I tried out the same test on Fedora22 machine, but haven't run into any issue. What are the fuse mount options you have used to mount gluster volume?

Oh, I think this is a simple problem (but maybe hard to fix). The capture shows NFSv3 traffic like:

   CREATE -> OK
   SETATTR (mode set to 0400) -> OK
   WRITE -> NFS3ERR_ACCES

That write would succeed locally (because the mode doesn't matter to a local application that already holds the file open). It would fail over NFSv3, which doesn't know about the open--except that there's a hack for this case: NFSv3 servers allow IO operations to ignore the mode, if the operation comes from the owner of the file. NFSv3 clients are then careful to perform necessary access checks on open to ensure that this owner-override rule doesn't grant too many permissions.

That allows NFSv3 applications to see behavior that's mostly like a local filesystem, without opening much of a security hole (since the owner could always chmod anyway).

So, knfsd is making this special exception--but gluster (which I believe it's exporting in this case, via fuse?)--probably doesn't.... I'm not sure what you can do about that.

--b.

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 26 lines · Soumya Koduri

Quoted 23 lines · J. Bruce Fields

Quoted 9 lines · Soumya Koduri

Quoted 6 lines · Malahal Naineni

IIRC, permission checks are done in open(). write/read syscalls should NOT do much access checks (at least based on POSIX). This is why once an open is done, you remove permissions for that process, but it should still be able to read/write based on the open flags it did when it opened the file.

Quoted 8 lines

Quoted 3 lines

I don't know all the details of this defect, but gluster seems to be doing what it is supposed to do.

Right. Thanks for the correction. I assumed the behavior should be same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But looks like (from 'man creat') the open() call that creates a read-only file may well return a read/write file descriptor, which is the reason the following WRITE can succeed.

I forgot another complication, which is that knsfd actually does a temporary open before each read or write--I assume that's getting translated into fuse and gluster open operations?

Quoted 2 lines

yes. It is the OPEN done as part of NFS WRITE which fails with EACCESS error (with both NFSv3 and NFSv4 mounts).

Makes sense for v3, but I wouldn't normally expect the extra temporary open on v4 WRITEs. Could you share any details?

--b.

Quoted 16 lines

63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS
164 V330 OPEN Call
65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS
164 [RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
116 V330 OPEN Reply (Call In 64)
67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
116 [RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call In 64)
68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
Reply (Call In 63) Error: NFS3ERR_ACCES

Quoted 110 lines

Thanks,
Soumya

Quoted 107 lines

In which case it might be worth experimenting with NFSv4 or with Jeff Layton's filehandle-caching patches. Neither's a real fix, but that could help confirm whether it's the temporary opens that are a problem.

--b.

Quoted 101 lines

Thanks,
Soumya

Quoted 96 lines

Regards, Malahal.

Quoted 92 lines · Soumya Koduri [[email protected]] wrote:

As mentioned by Bruce, GlusterFS doesn't have owner-override rule except for setattr.

I did few experiments to check why this test case passes on plain glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.

NFS-Ganesha (for most of the FSALs) seem to be passing the actual request credentials to the back-end filesystem only for CREATE(-like) and UNLINK fops. For all the remaining fops, it does the access check at its end and then perform the operation with root credentials. That's the reason WRITE succeeded in your case as NFS-Ganesha (like kernel-NFS) skipped the access check if the request caller_uid proved to be the file's owner.

In case of native GlusterFS FUSE mount, there is no OPEN fop involved. WRITE is performed on the fd returned by CREATE. And strangely GlusterFS seem to be doing certain access checks only during OPEN but not for WRITE (this seems like a bug and probably needs to be fixed in Gluster).

Thanks,
Soumya

Quoted 63 lines · Omar Walid Llorente

Thank you Bruce, others, for the responses. I send attached a complete capture of the issue, including the glusterfs transactions.

Hope this helps to clear where may it be...

Omar

Quoted 53 lines · J. Bruce Fields

Quoted 21 lines · Soumya Koduri

Quoted 6 lines · Omar Walid Llorente

Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs packets (you can find them in attached file nfs-problem-nks.pcap.zip). Sorry for being so late.

What I did was the following:

Quoted 9 lines

1st) Create the RO file:

cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:

root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It may be helpful to see what GlusterFS operations are being processed as part of NFS WRITE call (which has failed in this case).

Quoted 7 lines

Could you please try taking the packet trace on the machine where NFS server is running (without filtering out based on the port number).

Also I tried out the same test on Fedora22 machine, but haven't run into any issue. What are the fuse mount options you have used to mount gluster volume?

Oh, I think this is a simple problem (but maybe hard to fix). The capture shows NFSv3 traffic like:

    CREATE -> OK
    SETATTR (mode set to 0400) -> OK
    WRITE -> NFS3ERR_ACCES

That write would succeed locally (because the mode doesn't matter to a local application that already holds the file open). It would fail over NFSv3, which doesn't know about the open--except that there's a hack for this case: NFSv3 servers allow IO operations to ignore the mode, if the operation comes from the owner of the file. NFSv3 clients are then careful to perform necessary access checks on open to ensure that this owner-override rule doesn't grant too many permissions.

That allows NFSv3 applications to see behavior that's mostly like a local filesystem, without opening much of a security hole (since the owner could always chmod anyway).

So, knfsd is making this special exception--but gluster (which I believe it's exporting in this case, via fuse?)--probably doesn't.... I'm not sure what you can do about that.

--b.

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 27 lines · J. Bruce Fields

Quoted 26 lines · Soumya Koduri

Quoted 23 lines · J. Bruce Fields

Quoted 9 lines · Soumya Koduri

Quoted 6 lines · Malahal Naineni

IIRC, permission checks are done in open(). write/read syscalls should NOT do much access checks (at least based on POSIX). This is why once an open is done, you remove permissions for that process, but it should still be able to read/write based on the open flags it did when it opened the file.

Quoted 8 lines

Quoted 3 lines

I don't know all the details of this defect, but gluster seems to be doing what it is supposed to do.

Right. Thanks for the correction. I assumed the behavior should be same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But looks like (from 'man creat') the open() call that creates a read-only file may well return a read/write file descriptor, which is the reason the following WRITE can succeed.

I forgot another complication, which is that knsfd actually does a temporary open before each read or write--I assume that's getting translated into fuse and gluster open operations?

Quoted 6 lines

Quoted 2 lines

yes. It is the OPEN done as part of NFS WRITE which fails with EACCESS error (with both NFSv3 and NFSv4 mounts).

Makes sense for v3, but I wouldn't normally expect the extra temporary open on v4 WRITEs. Could you share any details?

I re-tried the test on v4 mount using Fedora23 machine, acting as both NFS server and client (Linux#4.2.3-300.fc23.x86_64). Please find the pkt trace attached.

56 07:23:25.567134 ::1 -> ::1 NFS 288 V4 Call
WRITE StateID: 0xf934 Offset: 0 Len: 7
57 07:23:25.567233 192.168.122.17 -> 192.168.122.202 GlusterFS 188
V330 GETXATTR Call
58 07:23:25.567732 192.168.122.202 -> 192.168.122.17 GlusterFS 112
V330 GETXATTR Reply (Call In 57)
59 07:23:25.567881 192.168.122.17 -> 192.168.122.202 GlusterFS 164
V330 OPEN Call

60 07:23:25.568354 192.168.122.202 -> 192.168.122.17 GlusterFS 116
V330 OPEN Reply (Call In 59)
61 07:23:25.568570 ::1 -> ::1 NFS 144 V4 Reply
(Call In 56) WRITE Status: NFS4ERR_ACCESS

Thanks,
Soumya

Quoted 133 lines

--b.

Quoted 16 lines

63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS
164 V330 OPEN Call
65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS
164 [RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
116 V330 OPEN Reply (Call In 64)
67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
116 [RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call In 64)
68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
Reply (Call In 63) Error: NFS3ERR_ACCES

Quoted 110 lines

Thanks,
Soumya

Quoted 107 lines

In which case it might be worth experimenting with NFSv4 or with Jeff Layton's filehandle-caching patches. Neither's a real fix, but that could help confirm whether it's the temporary opens that are a problem.

--b.

Quoted 101 lines

Thanks,
Soumya

Quoted 96 lines

Regards, Malahal.

Quoted 92 lines · Soumya Koduri [[email protected]] wrote:

As mentioned by Bruce, GlusterFS doesn't have owner-override rule except for setattr.

I did few experiments to check why this test case passes on plain glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.

NFS-Ganesha (for most of the FSALs) seem to be passing the actual request credentials to the back-end filesystem only for CREATE(-like) and UNLINK fops. For all the remaining fops, it does the access check at its end and then perform the operation with root credentials. That's the reason WRITE succeeded in your case as NFS-Ganesha (like kernel-NFS) skipped the access check if the request caller_uid proved to be the file's owner.

In case of native GlusterFS FUSE mount, there is no OPEN fop involved. WRITE is performed on the fd returned by CREATE. And strangely GlusterFS seem to be doing certain access checks only during OPEN but not for WRITE (this seems like a bug and probably needs to be fixed in Gluster).

Thanks,
Soumya

Quoted 63 lines · Omar Walid Llorente

Thank you Bruce, others, for the responses. I send attached a complete capture of the issue, including the glusterfs transactions.

Hope this helps to clear where may it be...

Omar

Quoted 53 lines · J. Bruce Fields

Quoted 21 lines · Soumya Koduri

Quoted 6 lines · Omar Walid Llorente

Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs packets (you can find them in attached file nfs-problem-nks.pcap.zip). Sorry for being so late.

What I did was the following:

Quoted 9 lines

1st) Create the RO file:

cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:

root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It may be helpful to see what GlusterFS operations are being processed as part of NFS WRITE call (which has failed in this case).

Quoted 7 lines

Could you please try taking the packet trace on the machine where NFS server is running (without filtering out based on the port number).

Also I tried out the same test on Fedora22 machine, but haven't run into any issue. What are the fuse mount options you have used to mount gluster volume?

Oh, I think this is a simple problem (but maybe hard to fix). The capture shows NFSv3 traffic like:

   CREATE -> OK
   SETATTR (mode set to 0400) -> OK
   WRITE -> NFS3ERR_ACCES

That write would succeed locally (because the mode doesn't matter to a local application that already holds the file open). It would fail over NFSv3, which doesn't know about the open--except that there's a hack for this case: NFSv3 servers allow IO operations to ignore the mode, if the operation comes from the owner of the file. NFSv3 clients are then careful to perform necessary access checks on open to ensure that this owner-override rule doesn't grant too many permissions.

That allows NFSv3 applications to see behavior that's mostly like a local filesystem, without opening much of a security hole (since the owner could always chmod anyway).

So, knfsd is making this special exception--but gluster (which I believe it's exporting in this case, via fuse?)--probably doesn't.... I'm not sure what you can do about that.

--b.

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 40 lines · Soumya Koduri

Quoted 27 lines · J. Bruce Fields

Quoted 26 lines · Soumya Koduri

Quoted 23 lines · J. Bruce Fields

Quoted 9 lines · Soumya Koduri

Quoted 6 lines · Malahal Naineni

IIRC, permission checks are done in open(). write/read syscalls should NOT do much access checks (at least based on POSIX). This is why once an open is done, you remove permissions for that process, but it should still be able to read/write based on the open flags it did when it opened the file.

Quoted 8 lines

Quoted 3 lines

I don't know all the details of this defect, but gluster seems to be doing what it is supposed to do.

Right. Thanks for the correction. I assumed the behavior should be same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But looks like (from 'man creat') the open() call that creates a read-only file may well return a read/write file descriptor, which is the reason the following WRITE can succeed.

I forgot another complication, which is that knsfd actually does a temporary open before each read or write--I assume that's getting translated into fuse and gluster open operations?

Quoted 6 lines

Quoted 2 lines

yes. It is the OPEN done as part of NFS WRITE which fails with EACCESS error (with both NFSv3 and NFSv4 mounts).

Makes sense for v3, but I wouldn't normally expect the extra temporary open on v4 WRITEs. Could you share any details?

I re-tried the test on v4 mount using Fedora23 machine, acting as both NFS server and client (Linux#4.2.3-300.fc23.x86_64). Please find the pkt trace attached.

Quoted 8 lines

56 07:23:25.567134 ::1 -> ::1 NFS 288 V4 Call
WRITE StateID: 0xf934 Offset: 0 Len: 7
57 07:23:25.567233 192.168.122.17 -> 192.168.122.202 GlusterFS 188
V330 GETXATTR Call
58 07:23:25.567732 192.168.122.202 -> 192.168.122.17 GlusterFS 112
V330 GETXATTR Reply (Call In 57)
59 07:23:25.567881 192.168.122.17 -> 192.168.122.202 GlusterFS 164
V330 OPEN Call

Remind me what kernel version your server is on?

--b.

Quoted 141 lines

60 07:23:25.568354 192.168.122.202 -> 192.168.122.17 GlusterFS 116
V330 OPEN Reply (Call In 59)
61 07:23:25.568570 ::1 -> ::1 NFS 144 V4 Reply
(Call In 56) WRITE Status: NFS4ERR_ACCESS

Thanks,
Soumya

Quoted 133 lines

--b.

Quoted 16 lines

63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS
164 V330 OPEN Call
65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS
164 [RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
116 V330 OPEN Reply (Call In 64)
67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
116 [RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call In 64)
68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
Reply (Call In 63) Error: NFS3ERR_ACCES

Quoted 110 lines

Thanks,
Soumya

Quoted 107 lines

In which case it might be worth experimenting with NFSv4 or with Jeff Layton's filehandle-caching patches. Neither's a real fix, but that could help confirm whether it's the temporary opens that are a problem.

--b.

Quoted 101 lines

Thanks,
Soumya

Quoted 96 lines

Regards, Malahal.

Quoted 92 lines · Soumya Koduri [[email protected]] wrote:

As mentioned by Bruce, GlusterFS doesn't have owner-override rule except for setattr.

I did few experiments to check why this test case passes on plain glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.

NFS-Ganesha (for most of the FSALs) seem to be passing the actual request credentials to the back-end filesystem only for CREATE(-like) and UNLINK fops. For all the remaining fops, it does the access check at its end and then perform the operation with root credentials. That's the reason WRITE succeeded in your case as NFS-Ganesha (like kernel-NFS) skipped the access check if the request caller_uid proved to be the file's owner.

In case of native GlusterFS FUSE mount, there is no OPEN fop involved. WRITE is performed on the fd returned by CREATE. And strangely GlusterFS seem to be doing certain access checks only during OPEN but not for WRITE (this seems like a bug and probably needs to be fixed in Gluster).

Thanks,
Soumya

Quoted 63 lines · Omar Walid Llorente

Thank you Bruce, others, for the responses. I send attached a complete capture of the issue, including the glusterfs transactions.

Hope this helps to clear where may it be...

Omar

Quoted 53 lines · J. Bruce Fields

Quoted 21 lines · Soumya Koduri

Quoted 6 lines · Omar Walid Llorente

Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs packets (you can find them in attached file nfs-problem-nks.pcap.zip). Sorry for being so late.

What I did was the following:

Quoted 9 lines

1st) Create the RO file:

cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:

root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It may be helpful to see what GlusterFS operations are being processed as part of NFS WRITE call (which has failed in this case).

Quoted 7 lines

Could you please try taking the packet trace on the machine where NFS server is running (without filtering out based on the port number).

Also I tried out the same test on Fedora22 machine, but haven't run into any issue. What are the fuse mount options you have used to mount gluster volume?

Oh, I think this is a simple problem (but maybe hard to fix). The capture shows NFSv3 traffic like:

    CREATE -> OK
    SETATTR (mode set to 0400) -> OK
    WRITE -> NFS3ERR_ACCES

That write would succeed locally (because the mode doesn't matter to a local application that already holds the file open). It would fail over NFSv3, which doesn't know about the open--except that there's a hack for this case: NFSv3 servers allow IO operations to ignore the mode, if the operation comes from the owner of the file. NFSv3 clients are then careful to perform necessary access checks on open to ensure that this owner-override rule doesn't grant too many permissions.

That allows NFSv3 applications to see behavior that's mostly like a local filesystem, without opening much of a security hole (since the owner could always chmod anyway).

So, knfsd is making this special exception--but gluster (which I believe it's exporting in this case, via fuse?)--probably doesn't.... I'm not sure what you can do about that.

--b.

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 50 lines · J. Bruce Fields

Quoted 48 lines · Soumya Koduri

Quoted 27 lines · J. Bruce Fields

Quoted 26 lines · Soumya Koduri

Quoted 23 lines · J. Bruce Fields

Quoted 9 lines · Soumya Koduri

Quoted 6 lines · Malahal Naineni

IIRC, permission checks are done in open(). write/read syscalls should NOT do much access checks (at least based on POSIX). This is why once an open is done, you remove permissions for that process, but it should still be able to read/write based on the open flags it did when it opened the file.

Quoted 8 lines

Quoted 3 lines

I don't know all the details of this defect, but gluster seems to be doing what it is supposed to do.

Right. Thanks for the correction. I assumed the behavior should be same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But looks like (from 'man creat') the open() call that creates a read-only file may well return a read/write file descriptor, which is the reason the following WRITE can succeed.

I forgot another complication, which is that knsfd actually does a temporary open before each read or write--I assume that's getting translated into fuse and gluster open operations?

Quoted 6 lines

Quoted 2 lines

yes. It is the OPEN done as part of NFS WRITE which fails with EACCESS error (with both NFSv3 and NFSv4 mounts).

Makes sense for v3, but I wouldn't normally expect the extra temporary open on v4 WRITEs. Could you share any details?

I re-tried the test on v4 mount using Fedora23 machine, acting as both NFS server and client (Linux#4.2.3-300.fc23.x86_64). Please find the pkt trace attached.

56 07:23:25.567134 ::1 -> ::1 NFS 288 V4 Call
WRITE StateID: 0xf934 Offset: 0 Len: 7
57 07:23:25.567233 192.168.122.17 -> 192.168.122.202 GlusterFS 188
V330 GETXATTR Call
58 07:23:25.567732 192.168.122.202 -> 192.168.122.17 GlusterFS 112
V330 GETXATTR Reply (Call In 57)
59 07:23:25.567881 192.168.122.17 -> 192.168.122.202 GlusterFS 164
V330 OPEN Call

Quoted 1 line

Remind me what kernel version your server is on?

NFS server is on fedora23 VM - Linux version 4.2.3-300.fc23.x86_64

Thanks,
Soumya

Quoted 147 lines

--b.

Quoted 141 lines

60 07:23:25.568354 192.168.122.202 -> 192.168.122.17 GlusterFS 116
V330 OPEN Reply (Call In 59)
61 07:23:25.568570 ::1 -> ::1 NFS 144 V4 Reply
(Call In 56) WRITE Status: NFS4ERR_ACCESS

Thanks,
Soumya

Quoted 133 lines

--b.

Quoted 16 lines

63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS
164 V330 OPEN Call
65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS
164 [RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
116 V330 OPEN Reply (Call In 64)
67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
116 [RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call In 64)
68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
Reply (Call In 63) Error: NFS3ERR_ACCES

Quoted 110 lines

Thanks,
Soumya

Quoted 107 lines

In which case it might be worth experimenting with NFSv4 or with Jeff Layton's filehandle-caching patches. Neither's a real fix, but that could help confirm whether it's the temporary opens that are a problem.

--b.

Quoted 101 lines

Thanks,
Soumya

Quoted 96 lines

Regards, Malahal.

Quoted 92 lines · Soumya Koduri [[email protected]] wrote:

As mentioned by Bruce, GlusterFS doesn't have owner-override rule except for setattr.

I did few experiments to check why this test case passes on plain glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.

NFS-Ganesha (for most of the FSALs) seem to be passing the actual request credentials to the back-end filesystem only for CREATE(-like) and UNLINK fops. For all the remaining fops, it does the access check at its end and then perform the operation with root credentials. That's the reason WRITE succeeded in your case as NFS-Ganesha (like kernel-NFS) skipped the access check if the request caller_uid proved to be the file's owner.

In case of native GlusterFS FUSE mount, there is no OPEN fop involved. WRITE is performed on the fd returned by CREATE. And strangely GlusterFS seem to be doing certain access checks only during OPEN but not for WRITE (this seems like a bug and probably needs to be fixed in Gluster).

Thanks,
Soumya

Quoted 63 lines · Omar Walid Llorente

Thank you Bruce, others, for the responses. I send attached a complete capture of the issue, including the glusterfs transactions.

Hope this helps to clear where may it be...

Omar

Quoted 53 lines · J. Bruce Fields

Quoted 21 lines · Soumya Koduri

Quoted 15 lines · Omar Walid Llorente

Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
packets (you can find them in attached file nfs-problem-nks.pcap.zip).
Sorry for being so late.

What I did was the following:

1st) Create the RO file:
cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:
root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It may be helpful to see what GlusterFS operations are being processed as part of NFS WRITE call (which has failed in this case).

Quoted 7 lines

Could you please try taking the packet trace on the machine where NFS server is running (without filtering out based on the port number).

Also I tried out the same test on Fedora22 machine, but haven't run into any issue. What are the fuse mount options you have used to mount gluster volume?

Oh, I think this is a simple problem (but maybe hard to fix). The capture shows NFSv3 traffic like:

   CREATE -> OK
   SETATTR (mode set to 0400) -> OK
   WRITE -> NFS3ERR_ACCES

That write would succeed locally (because the mode doesn't matter to a local application that already holds the file open). It would fail over NFSv3, which doesn't know about the open--except that there's a hack for this case: NFSv3 servers allow IO operations to ignore the mode, if the operation comes from the owner of the file. NFSv3 clients are then careful to perform necessary access checks on open to ensure that this owner-override rule doesn't grant too many permissions.

That allows NFSv3 applications to see behavior that's mostly like a local filesystem, without opening much of a security hole (since the owner could always chmod anyway).

So, knfsd is making this special exception--but gluster (which I believe it's exporting in this case, via fuse?)--probably doesn't.... I'm not sure what you can do about that.

--b.

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Resending because it has not been accepted by [email protected] because of the HTML... Sorry if you get it twice.

Thanks again.

Omar

--- Forwarded message ---

Hi Bruce, others.

I come back to this issue again in the hope that these evidences I bring can help to give some light to the problem.

We have exactly the same environment that last year when I came to you, but with upgraded versions of gluster (3.8) and nfs-kernel-server (1.2.8) this time over an updated Ubuntu 16.04.

(NOTE: Environment: glusterfs-cluster made of ZFS datasets exports via glusterfs a distributed volume + nfs-kernel server mounts via fuse that glusterfs volume and reexports it via nfs-v3 + nfs-common client mounts via nfs that nfs exported volume)

The errors I get by the user side are exactly the same, but in the server side, I have different messages, which I forward to you.

* /TESTS/ and errors by the user side:

u056@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod
444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: cannot create regular file ‘kk.txt’: Permission denied
ls: cannot access kk.txt: No such file or directory
-r--r--r-- 1 u056 admincdc 7 oct 19  2016 444.txt
u056@l056:~$ ls -lrt
total 33
---------- 1 u056 admincdc    0 ene  4  1970 kk.txt
drwx------ 2 u056 alumno   4096 oct  3  2014 Desktop
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Público
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Descargas
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Documentos
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Vídeos
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Música
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Plantillas
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:22 Imágenes
-r--r--r-- 1 u056 admincdc    7 oct 19 17:34 444.txt

u056@l056:~$ mount | grep cuentas09
cuentas09:/home-3/u056 on /home/u056 type nfs (rw,noatime,intr,fsc,nolock,rsize=262140,wsize=262140,vers=4,addr=138.4.30.80,clientaddr=138.4.31.56)
u056@l056:~$ df -h | grep cuentas09
cuentas09:/home-3/u056              40T   55G   40T   1% /home/u056
u056@l056:~$

* /LOGS/ by the glusterfs client side (server side), versions of the software, info of the kernel and related kernel modules:

root@cuentas09-lab:/var/log/glusterfs# tail home-3.log
[2016-10-19 15:33:38.091300] I [io-stats.c:1574:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---
[2016-10-19 15:33:38.091331] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3:       Filename : /u056/444.txt
[2016-10-19 15:33:38.091344] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3:   BytesWritten : 7 bytes
[2016-10-19 15:33:38.091355] I [io-stats.c:1606:io_stats_dump_fd]
0-home-lab-3: Write 000004b+ : 1
[2016-10-19 15:34:13.199285] I [io-stats.c:1574:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---
[2016-10-19 15:34:13.200291] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3:       Filename : /u056/444.txt
[2016-10-19 15:34:13.200313] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3:   BytesWritten : 7 bytes
[2016-10-19 15:34:13.200325] I [io-stats.c:1606:io_stats_dump_fd]
0-home-lab-3: Write 000004b+ : 1
[2016-10-19 15:34:13.211696] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-0:
remote operation failed. Path: /u056/kk.txt
(d8fc54a3-f2eb-4538-889a-17afdcfbb255) [Permission denied]
[2016-10-19 15:34:13.211758] W [fuse-bridge.c:989:fuse_fd_cbk]
0-glusterfs-fuse: 630: OPEN() /u056/kk.txt => -1 (Permission denied)

root@cuentas09-lab:/var/log/glusterfs# dpkg -l | grep -e gluster -e nfs
ii  glusterfs-client 3.8.5-ubuntu1~xenial1             amd64
clustered file-system (client package)
ii  glusterfs-common 3.8.5-ubuntu1~xenial1             amd64
GlusterFS common libraries and translator modules
ii  libnfsidmap2:amd64 0.25-5                            amd64
NFS idmapping library
ii  nfs-common 1:1.2.8-9ubuntu12                 amd64        NFS
support files common to client and server
ii  nfs-kernel-server 1:1.2.8-9ubuntu12                 amd64
support for NFS kernel server

root@cuentas09-lab:/var/log/glusterfs# uname -a
Linux cuentas09-lab.lab.dit.upm.es 4.4.0-43-generic #63-Ubuntu SMP
Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

root@cuentas09-lab:/var/log/glusterfs# lsmod | grep nfs
nfsd                  319488  13
auth_rpcgss            61440  1 nfsd
nfs_acl                16384  1 nfsd
lockd                  94208  1 nfsd
grace                  16384  2 nfsd,lockd
sunrpc                335872  21 nfsd,auth_rpcgss,lockd,nfs_acl
root@cuentas09-lab:/var/log/glusterfs#

* /LOGS/ by the nfs-kernel server side and basic configuration of the nfs-kernel-server:

root@cuentas09-lab:~# less /var/log/kern.log
root@cuentas09-lab:~# tail -30 !$
tail -30 /var/log/kern.log
Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631498]
[<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631504]
[<ffffffff81831c4f>] ret_from_fork+0x3f/0x70
Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631507]
[<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631510] ---[ end trace
34db7650fa22d1d0 ]---
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814898] ------------[
cut here ]------------
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814948] WARNING: CPU: 0
PID: 1373 at /build/linux-BwgxJb/linux-4.4.0/fs/nfsd/nfs4proc.c:464
nfsd4_open+0x515/0x780 [nfsd]()
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814951]
nfsd4_process_open2 failed to open newly-created file! status=13
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814954] Modules linked
in: vmw_vsock_vmci_transport vsock ppdev vmw_balloon coretemp joydev
input_leds serio_raw irda 8250_fintek parport_pc shpchp parport
i2c_piix4 crc_ccitt vmw_vmci mac_hid ib_iser nfsd rdma_cm iw_cm
ib_cm auth_rpcgss nfs_acl lockd grace ib_sa sunrpc ib_mad ib_core
ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4
btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq
async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath
linear vmwgfx ttm psmouse drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops mptspi mptscsih mptbase drm e1000
scsi_transport_spi pata_acpi floppy fjes
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815018] CPU: 0 PID:
1373 Comm: nfsd Tainted: G        W       4.4.0-43-generic
#63-Ubuntu
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815021] Hardware name:
VMware, Inc. VMware Virtual Platform/440BX Desktop Reference
Platform, BIOS 6.00 09/22/2009
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815024]
0000000000000286 00000000c0e912c3 ffff8800b93c7c80 ffffffff813f1f93
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815028]
ffff8800b93c7cc8 ffffffffc047b668 ffff8800b93c7cb8 ffffffff81081212
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815031]
ffff8800b9e43240 ffff8800b9e44068 000000000d000000 ffff8800b85e6000
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815035] Call Trace:
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815047]
[<ffffffff813f1f93>] dump_stack+0x63/0x90
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815055]
[<ffffffff81081212>] warn_slowpath_common+0x82/0xc0
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815058]
[<ffffffff810812ac>] warn_slowpath_fmt+0x5c/0x80
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815073]
[<ffffffffc04665db>] ? nfs4_free_ol_stateid+0x3b/0x40 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815085]
[<ffffffffc0459d05>] nfsd4_open+0x515/0x780 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815098]
[<ffffffffc045a2fa>] nfsd4_proc_compound+0x38a/0x660 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815108]
[<ffffffffc0446e78>] nfsd_dispatch+0xb8/0x200 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815132]
[<ffffffffc039eeac>] svc_process_common+0x40c/0x650 [sunrpc]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815151]
[<ffffffffc03a0273>] svc_process+0x103/0x1c0 [sunrpc]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815160]
[<ffffffffc04468cf>] nfsd+0xef/0x160 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815170]
[<ffffffffc04467e0>] ? nfsd_destroy+0x60/0x60 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815174]
[<ffffffff810a0928>] kthread+0xd8/0xf0
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815178]
[<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815184]
[<ffffffff81831c4f>] ret_from_fork+0x3f/0x70
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815187]
[<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815190] ---[ end trace
34db7650fa22d1d1 ]---
root@cuentas09-lab:~#

root@cuentas09-lab:~# showmount -e
Export list for cuentas09-lab.lab.dit.upm.es:
/home-4 138.4.30.0/23
/home-3 138.4.30.0/23

root@cuentas09-lab:~# cat /etc/exports | grep -v ^#
/home-3 138.4.30.0/23(rw,fsid=3,insecure,no_subtree_check,async,no_root_squash)
/home-4 138.4.30.0/23(rw,fsid=4,insecure,no_subtree_check,async,no_root_squash)
root@cuentas09-lab:~#

root@cuentas09-lab:~# cat /etc/default/nfs-kernel-server | grep -v ^#
RPCNFSDCOUNT="8 -V 3"
RPCNFSDPRIORITY=0
RPCMOUNTDOPTS="--manage-gids"
NEED_SVCGSSD=""
RPCSVCGSSDOPTS=""
RPCNFSDOPTS="-d -V 3"
root@cuentas09-lab:~#

* LOGS by the glusterfs server side:

# for i in 10 11 12 13; do echo recipiente$i; ssh recipiente$i tail
/var/log/glusterfs/bricks/data-recipiente$i-gluster-home-lab-3.log;
done
recipiente10
[2016-10-19 15:27:48.172858] I [io-stats.c:1606:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3: Write 000008b+ : 1
[2016-10-19 15:29:32.626450] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:29:32.626545] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3:       Filename :
/u056/.local/share/tracker/data/tracker-store.journal
[2016-10-19 15:29:32.626572] I [io-stats.c:1594:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3:   BytesWritten : 8 bytes
[2016-10-19 15:29:32.626594] I [io-stats.c:1606:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3: Write 000008b+ : 1
[2016-10-19 15:30:23.997592] E [MSGID: 115070]
[server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 372:
OPEN /u056/kk.txt (cc97d150-c725-42b6-9aa5-50328f281e06) ==>
(Permission denied) [Permission denied]
[2016-10-19 15:32:28.423380] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:32:28.423478] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3:       Filename :
/u056/.bashrc
[2016-10-19 15:32:28.423504] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3:       Lifetime : 280secs,
232488usecs
[2016-10-19 15:34:13.176805] E [MSGID: 115070]
[server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 424:
OPEN /u056/kk.txt (d8fc54a3-f2eb-4538-889a-17afdcfbb255) ==>
(Permission denied) [Permission denied]
recipiente11
[2016-10-19 15:33:38.057524] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:       Filename :
/u056/444.txt
[2016-10-19 15:33:38.057548] I [io-stats.c:1594:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:   BytesWritten : 7 bytes
[2016-10-19 15:33:38.057570] I [io-stats.c:1606:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3: Write 000004b+ : 1
[2016-10-19 15:34:13.164629] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:34:13.164722] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:       Filename :
/u056/444.txt
[2016-10-19 15:34:13.164769] I [io-stats.c:1594:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:   BytesWritten : 7 bytes
[2016-10-19 15:34:13.164791] I [io-stats.c:1606:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3: Write 000004b+ : 1
[2016-10-19 15:36:29.066078] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:36:29.066180] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:       Filename :
/u056/444.txt
[2016-10-19 15:36:29.066206] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:       Lifetime : 136secs,
894418usecs
recipiente12
[2016-10-19 15:29:06.022950] I [io-stats.c:1594:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:   BytesWritten : 986 bytes
[2016-10-19 15:29:06.022972] I [io-stats.c:1606:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3: Write 000512b+ : 1
[2016-10-19 15:29:32.646111] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:29:32.646214] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:       Filename :
/u056/.profile
[2016-10-19 15:32:28.423115] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:32:28.423219] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:       Filename :
/u056/.bash_logout
[2016-10-19 15:32:28.423247] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:       Lifetime : 202secs,
412682usecs
[2016-10-19 15:32:28.423345] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:32:28.423378] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:       Filename :
/u056/.bash_history
[2016-10-19 15:32:28.423400] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:       Lifetime : 202secs,
399903usecs
recipiente13
[2016-10-19 15:04:26.825754] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:04:26.825788] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3:       Filename : /u056/pp
[2016-10-19 15:04:26.825799] I [MSGID: 101055]
[client_t.c:415:gf_client_unref] 0-home-lab-3-server: Shutting down
connection cuentas09-lab.lab.dit.upm.es-1578-2016/10/19-05:00:48:464910-home-lab-3-client-3-0-0
[2016-10-19 15:04:26.825813] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3:       Lifetime : 25804secs,
917102usecs
[2016-10-19 15:04:26.825991] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:04:26.826026] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3:       Filename :
/u056/.local/share/evolution/tasks/system/tasks.ics
[2016-10-19 15:04:26.826050] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3:       Lifetime : 25774secs,
172800usecs
[2016-10-19 15:04:48.333183] I [MSGID: 115029]
[server-handshake.c:692:server_setvolume] 0-home-lab-3-server:
accepted client from cuentas09-lab.lab.dit.upm.es-1610-2016/10/19-15:04:46:807972-home-lab-3-client-3-0-0
(version: 3.8.5)
[2016-10-19 15:27:27.763705] E [MSGID: 115070]
[server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 149:
OPEN /u056/.Xauthority-c (a4fa9bc4-a09e-4eb8-968b-b77cb6d49fd3) ==>
(Permission denied) [Permission denied]
[2016-10-19 15:29:12.202792] E [MSGID: 115070]
[server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 267:
OPEN /u056/.Xauthority-c (ce222a86-9f74-42d1-8e18-1f80f607d766) ==>
(Permission denied) [Permission denied]
root@admin2:~#

As you can see, the glusterfs client part of the server fuse-mounting the volume shows a "/u056/kk.txt => -1 (Permission denied)" error that comes from a similar error in the glusterfs cluster, while at the nfs-kernel server log we can see a kernel trace directly related to the glusterfs fuse mount of the volume that says "nfsd4_process_open2 failed to open newly-created file! status=13".

This error can be reproduced any number of times exactly the same, is not sporadic.

Other test I've done at the client side with no logs at all at the server side, but the same error for the user, is the following:

u056@l056:~$ rm -f 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; echo "kk" > 444.txt
-bash: 444.txt: Permission denied
u056@l056:~$

We have been trying to reproduce this problem in a virtual environment using KVM, nfs-kernel-server, glusterfs, and ZFS over VFS, being unable to do it... This leads us to think that creating glusterfs volumes over ZFS over phisical disks has something to do with this problem, but we have no clue on how to track it down to the ZFS level, as there are no traces on the kernel.log or the syslog that we can use.

Thank you for your time.

Omar

Quoted 200 lines · J. Bruce Fields

Quoted 46 lines · Soumya Koduri

Quoted 44 lines · J. Bruce Fields

Quoted 43 lines · Soumya Koduri

Quoted 20 lines · J. Bruce Fields

Quoted 19 lines · Soumya Koduri

Quoted 17 lines · J. Bruce Fields

Quoted 16 lines · Soumya Koduri

Quoted 9 lines · Malahal Naineni

IIRC, permission checks are done in open(). write/read syscalls should NOT do much access checks (at least based on POSIX). This is why once an open is done, you remove permissions for that process, but it should still be able to read/write based on the open flags it did when it opened the file.

I don't know all the details of this defect, but gluster seems to be doing what it is supposed to do.

Right. Thanks for the correction. I assumed the behavior should be same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But looks like (from 'man creat') the open() call that creates a read-only file may well return a read/write file descriptor, which is the reason the following WRITE can succeed.

Quoted 9 lines

Quoted 6 lines

Quoted 4 lines

I forgot another complication, which is that knsfd actually does a temporary open before each read or write--I assume that's getting translated into fuse and gluster open operations?

yes. It is the OPEN done as part of NFS WRITE which fails with EACCESS error (with both NFSv3 and NFSv4 mounts).

Makes sense for v3, but I wouldn't normally expect the extra temporary open on v4 WRITEs. Could you share any details?

I re-tried the test on v4 mount using Fedora23 machine, acting as both NFS server and client (Linux#4.2.3-300.fc23.x86_64). Please find the pkt trace attached.

56 07:23:25.567134 ::1 -> ::1 NFS 288 V4 Call
WRITE StateID: 0xf934 Offset: 0 Len: 7
57 07:23:25.567233 192.168.122.17 -> 192.168.122.202 GlusterFS 188
V330 GETXATTR Call
58 07:23:25.567732 192.168.122.202 -> 192.168.122.17 GlusterFS 112
V330 GETXATTR Reply (Call In 57)
59 07:23:25.567881 192.168.122.17 -> 192.168.122.202 GlusterFS 164
V330 OPEN Call

Quoted 2 lines

Quoted 1 line

Remind me what kernel version your server is on?

NFS server is on fedora23 VM - Linux version 4.2.3-300.fc23.x86_64

We did reshuffle the code that does the temporary open on WRITE around there--but it looks right to me, and I can't reproduce an open on WRITE on that kernel myself.

Maybe there's some further fuse or gluster debugging that would help show where that open is coming from.

--b.

Quoted 142 lines

Thanks,
Soumya

Quoted 139 lines

--b.

Quoted 135 lines

60 07:23:25.568354 192.168.122.202 -> 192.168.122.17 GlusterFS 116
V330 OPEN Reply (Call In 59)
61 07:23:25.568570 ::1 -> ::1 NFS 144 V4 Reply
(Call In 56) WRITE Status: NFS4ERR_ACCESS

Thanks,
Soumya

Quoted 127 lines

--b.

Quoted 15 lines

63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS
164 V330 OPEN Call
65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS
164 [RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
116 V330 OPEN Reply (Call In 64)
67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
116 [RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call In 64)
68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
Reply (Call In 63) Error: NFS3ERR_ACCES

Quoted 105 lines

Thanks,
Soumya

Quoted 102 lines

In which case it might be worth experimenting with NFSv4 or with Jeff Layton's filehandle-caching patches. Neither's a real fix, but that could help confirm whether it's the temporary opens that are a problem.

--b.

Quoted 96 lines

Thanks,
Soumya

Quoted 92 lines

Regards, Malahal.

Quoted 89 lines · Soumya Koduri [[email protected]] wrote:

As mentioned by Bruce, GlusterFS doesn't have owner-override rule except for setattr.

I did few experiments to check why this test case passes on plain glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.

NFS-Ganesha (for most of the FSALs) seem to be passing the actual request credentials to the back-end filesystem only for CREATE(-like) and UNLINK fops. For all the remaining fops, it does the access check at its end and then perform the operation with root credentials. That's the reason WRITE succeeded in your case as NFS-Ganesha (like kernel-NFS) skipped the access check if the request caller_uid proved to be the file's owner.

In case of native GlusterFS FUSE mount, there is no OPEN fop involved. WRITE is performed on the fd returned by CREATE. And strangely GlusterFS seem to be doing certain access checks only during OPEN but not for WRITE (this seems like a bug and probably needs to be fixed in Gluster).

Thanks,
Soumya

Quoted 60 lines · Omar Walid Llorente

Thank you Bruce, others, for the responses. I send attached a complete capture of the issue, including the glusterfs transactions.

Hope this helps to clear where may it be...

Omar

Quoted 52 lines · J. Bruce Fields

On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
packets (you can find them in attached file nfs-problem-nks.pcap.zip).
Sorry for being so late.

What I did was the following:

1st) Create the RO file:
cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:
root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
may be helpful to see what GlusterFS operations are being processed
as part of NFS WRITE call (which has failed in this case).

Could you please try taking the packet trace on the machine where
NFS server is running (without filtering out based on the port
number).

Also I tried out the same test on Fedora22 machine, but haven't run
into any issue. What are the fuse mount options you have used to
mount gluster volume?
Oh, I think this is a simple problem (but maybe hard to fix). The
capture shows NFSv3 traffic like:

CREATE -> OK
SETATTR (mode set to 0400) -> OK
WRITE -> NFS3ERR_ACCES

That write would succeed locally (because the mode doesn't matter to a
local application that already holds the file open). It would fail over
NFSv3, which doesn't know about the open--except that there's a hack for
this case: NFSv3 servers allow IO operations to ignore the mode, if the
operation comes from the owner of the file. NFSv3 clients are then
careful to perform necessary access checks on open to ensure that this
owner-override rule doesn't grant too many permissions.

That allows NFSv3 applications to see behavior that's mostly like a
local filesystem, without opening much of a security hole (since the
owner could always chmod anyway).

So, knfsd is making this special exception--but gluster (which I believe
it's exporting in this case, via fuse?)--probably doesn't.... I'm not
sure what you can do about that.

--b.

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 8 lines

----------------------------------------------------------------
Centro de Cálculo         Depto. Ingeniería Sistemas Telemáticos
E-mail: [email protected]        Universidad Politécnica de Madrid
Fax:(+34) 913367333                 E.T.S. Ing. Telecomunicación
Tel:(+34) 915495700-Ext.3005                28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------

Quoted 11 lines · Omar Walid Llorente

Resending because it has not been accepted by [email protected] because of the HTML... Sorry if you get it twice.

Thanks again.

Omar

--- Forwarded message ---

Hi Bruce, others.

Sory for the delayed response. This is a fairly complicated setup that may require people with nfs, gluster, and zfs experience to debug.

The server does need to be able to bypass certain permission checks on open so that it can open a newly created read-only file for write. Probably one of those lower layers is insisting on permission checks where it shouldn't. That's all I can say from a quick look.

--b.

Quoted 545 lines

I come back to this issue again in the hope that these evidences I bring can help to give some light to the problem.

We have exactly the same environment that last year when I came to you, but with upgraded versions of gluster (3.8) and nfs-kernel-server (1.2.8) this time over an updated Ubuntu 16.04.

(NOTE: Environment: glusterfs-cluster made of ZFS datasets exports via glusterfs a distributed volume + nfs-kernel server mounts via fuse that glusterfs volume and reexports it via nfs-v3 + nfs-common client mounts via nfs that nfs exported volume)

The errors I get by the user side are exactly the same, but in the server side, I have different messages, which I forward to you.

* /TESTS/ and errors by the user side:

u056@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod
444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: cannot create regular file ‘kk.txt’: Permission denied
ls: cannot access kk.txt: No such file or directory
-r--r--r-- 1 u056 admincdc 7 oct 19  2016 444.txt
u056@l056:~$ ls -lrt
total 33
---------- 1 u056 admincdc    0 ene  4  1970 kk.txt
drwx------ 2 u056 alumno   4096 oct  3  2014 Desktop
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Público
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Descargas
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Documentos
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Vídeos
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Música
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Plantillas
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:22 Imágenes
-r--r--r-- 1 u056 admincdc    7 oct 19 17:34 444.txt

u056@l056:~$ mount | grep cuentas09
cuentas09:/home-3/u056 on /home/u056 type nfs (rw,noatime,intr,fsc,nolock,rsize=262140,wsize=262140,vers=4,addr=138.4.30.80,clientaddr=138.4.31.56)
u056@l056:~$ df -h | grep cuentas09
cuentas09:/home-3/u056              40T   55G   40T   1% /home/u056
u056@l056:~$

* /LOGS/ by the glusterfs client side (server side), versions of the software, info of the kernel and related kernel modules:

root@cuentas09-lab:/var/log/glusterfs# tail home-3.log
[2016-10-19 15:33:38.091300] I [io-stats.c:1574:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---
[2016-10-19 15:33:38.091331] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3:       Filename : /u056/444.txt
[2016-10-19 15:33:38.091344] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3:   BytesWritten : 7 bytes
[2016-10-19 15:33:38.091355] I [io-stats.c:1606:io_stats_dump_fd]
0-home-lab-3: Write 000004b+ : 1
[2016-10-19 15:34:13.199285] I [io-stats.c:1574:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---
[2016-10-19 15:34:13.200291] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3:       Filename : /u056/444.txt
[2016-10-19 15:34:13.200313] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3:   BytesWritten : 7 bytes
[2016-10-19 15:34:13.200325] I [io-stats.c:1606:io_stats_dump_fd]
0-home-lab-3: Write 000004b+ : 1
[2016-10-19 15:34:13.211696] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-0:
remote operation failed. Path: /u056/kk.txt
(d8fc54a3-f2eb-4538-889a-17afdcfbb255) [Permission denied]
[2016-10-19 15:34:13.211758] W [fuse-bridge.c:989:fuse_fd_cbk]
0-glusterfs-fuse: 630: OPEN() /u056/kk.txt => -1 (Permission denied)

root@cuentas09-lab:/var/log/glusterfs# dpkg -l | grep -e gluster -e nfs
ii  glusterfs-client 3.8.5-ubuntu1~xenial1             amd64
clustered file-system (client package)
ii  glusterfs-common 3.8.5-ubuntu1~xenial1             amd64
GlusterFS common libraries and translator modules
ii  libnfsidmap2:amd64 0.25-5                            amd64
NFS idmapping library
ii  nfs-common 1:1.2.8-9ubuntu12                 amd64        NFS
support files common to client and server
ii  nfs-kernel-server 1:1.2.8-9ubuntu12                 amd64
support for NFS kernel server

root@cuentas09-lab:/var/log/glusterfs# uname -a
Linux cuentas09-lab.lab.dit.upm.es 4.4.0-43-generic #63-Ubuntu SMP
Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

root@cuentas09-lab:/var/log/glusterfs# lsmod | grep nfs
nfsd                  319488  13
auth_rpcgss            61440  1 nfsd
nfs_acl                16384  1 nfsd
lockd                  94208  1 nfsd
grace                  16384  2 nfsd,lockd
sunrpc                335872  21 nfsd,auth_rpcgss,lockd,nfs_acl
root@cuentas09-lab:/var/log/glusterfs#

* /LOGS/ by the nfs-kernel server side and basic configuration of the nfs-kernel-server:

root@cuentas09-lab:~# less /var/log/kern.log
root@cuentas09-lab:~# tail -30 !$
tail -30 /var/log/kern.log
Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631498]
[<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631504]
[<ffffffff81831c4f>] ret_from_fork+0x3f/0x70
Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631507]
[<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631510] ---[ end trace
34db7650fa22d1d0 ]---
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814898] ------------[
cut here ]------------
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814948] WARNING: CPU: 0
PID: 1373 at /build/linux-BwgxJb/linux-4.4.0/fs/nfsd/nfs4proc.c:464
nfsd4_open+0x515/0x780 [nfsd]()
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814951]
nfsd4_process_open2 failed to open newly-created file! status=13
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814954] Modules linked
in: vmw_vsock_vmci_transport vsock ppdev vmw_balloon coretemp joydev
input_leds serio_raw irda 8250_fintek parport_pc shpchp parport
i2c_piix4 crc_ccitt vmw_vmci mac_hid ib_iser nfsd rdma_cm iw_cm
ib_cm auth_rpcgss nfs_acl lockd grace ib_sa sunrpc ib_mad ib_core
ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4
btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq
async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath
linear vmwgfx ttm psmouse drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops mptspi mptscsih mptbase drm e1000
scsi_transport_spi pata_acpi floppy fjes
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815018] CPU: 0 PID:
1373 Comm: nfsd Tainted: G        W       4.4.0-43-generic
#63-Ubuntu
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815021] Hardware name:
VMware, Inc. VMware Virtual Platform/440BX Desktop Reference
Platform, BIOS 6.00 09/22/2009
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815024]
0000000000000286 00000000c0e912c3 ffff8800b93c7c80 ffffffff813f1f93
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815028]
ffff8800b93c7cc8 ffffffffc047b668 ffff8800b93c7cb8 ffffffff81081212
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815031]
ffff8800b9e43240 ffff8800b9e44068 000000000d000000 ffff8800b85e6000
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815035] Call Trace:
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815047]
[<ffffffff813f1f93>] dump_stack+0x63/0x90
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815055]
[<ffffffff81081212>] warn_slowpath_common+0x82/0xc0
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815058]
[<ffffffff810812ac>] warn_slowpath_fmt+0x5c/0x80
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815073]
[<ffffffffc04665db>] ? nfs4_free_ol_stateid+0x3b/0x40 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815085]
[<ffffffffc0459d05>] nfsd4_open+0x515/0x780 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815098]
[<ffffffffc045a2fa>] nfsd4_proc_compound+0x38a/0x660 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815108]
[<ffffffffc0446e78>] nfsd_dispatch+0xb8/0x200 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815132]
[<ffffffffc039eeac>] svc_process_common+0x40c/0x650 [sunrpc]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815151]
[<ffffffffc03a0273>] svc_process+0x103/0x1c0 [sunrpc]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815160]
[<ffffffffc04468cf>] nfsd+0xef/0x160 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815170]
[<ffffffffc04467e0>] ? nfsd_destroy+0x60/0x60 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815174]
[<ffffffff810a0928>] kthread+0xd8/0xf0
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815178]
[<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815184]
[<ffffffff81831c4f>] ret_from_fork+0x3f/0x70
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815187]
[<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815190] ---[ end trace
34db7650fa22d1d1 ]---
root@cuentas09-lab:~#

root@cuentas09-lab:~# showmount -e
Export list for cuentas09-lab.lab.dit.upm.es:
/home-4 138.4.30.0/23
/home-3 138.4.30.0/23

root@cuentas09-lab:~# cat /etc/exports | grep -v ^#
/home-3 138.4.30.0/23(rw,fsid=3,insecure,no_subtree_check,async,no_root_squash)
/home-4 138.4.30.0/23(rw,fsid=4,insecure,no_subtree_check,async,no_root_squash)
root@cuentas09-lab:~#

root@cuentas09-lab:~# cat /etc/default/nfs-kernel-server | grep -v ^#
RPCNFSDCOUNT="8 -V 3"
RPCNFSDPRIORITY=0
RPCMOUNTDOPTS="--manage-gids"
NEED_SVCGSSD=""
RPCSVCGSSDOPTS=""
RPCNFSDOPTS="-d -V 3"
root@cuentas09-lab:~#

* LOGS by the glusterfs server side:

# for i in 10 11 12 13; do echo recipiente$i; ssh recipiente$i tail
/var/log/glusterfs/bricks/data-recipiente$i-gluster-home-lab-3.log;
done
recipiente10
[2016-10-19 15:27:48.172858] I [io-stats.c:1606:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3: Write 000008b+ : 1
[2016-10-19 15:29:32.626450] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:29:32.626545] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3:       Filename :
/u056/.local/share/tracker/data/tracker-store.journal
[2016-10-19 15:29:32.626572] I [io-stats.c:1594:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3:   BytesWritten : 8 bytes
[2016-10-19 15:29:32.626594] I [io-stats.c:1606:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3: Write 000008b+ : 1
[2016-10-19 15:30:23.997592] E [MSGID: 115070]
[server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 372:
OPEN /u056/kk.txt (cc97d150-c725-42b6-9aa5-50328f281e06) ==>
(Permission denied) [Permission denied]
[2016-10-19 15:32:28.423380] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:32:28.423478] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3:       Filename :
/u056/.bashrc
[2016-10-19 15:32:28.423504] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3:       Lifetime : 280secs,
232488usecs
[2016-10-19 15:34:13.176805] E [MSGID: 115070]
[server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 424:
OPEN /u056/kk.txt (d8fc54a3-f2eb-4538-889a-17afdcfbb255) ==>
(Permission denied) [Permission denied]
recipiente11
[2016-10-19 15:33:38.057524] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:       Filename :
/u056/444.txt
[2016-10-19 15:33:38.057548] I [io-stats.c:1594:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:   BytesWritten : 7 bytes
[2016-10-19 15:33:38.057570] I [io-stats.c:1606:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3: Write 000004b+ : 1
[2016-10-19 15:34:13.164629] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:34:13.164722] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:       Filename :
/u056/444.txt
[2016-10-19 15:34:13.164769] I [io-stats.c:1594:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:   BytesWritten : 7 bytes
[2016-10-19 15:34:13.164791] I [io-stats.c:1606:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3: Write 000004b+ : 1
[2016-10-19 15:36:29.066078] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:36:29.066180] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:       Filename :
/u056/444.txt
[2016-10-19 15:36:29.066206] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:       Lifetime : 136secs,
894418usecs
recipiente12
[2016-10-19 15:29:06.022950] I [io-stats.c:1594:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:   BytesWritten : 986 bytes
[2016-10-19 15:29:06.022972] I [io-stats.c:1606:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3: Write 000512b+ : 1
[2016-10-19 15:29:32.646111] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:29:32.646214] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:       Filename :
/u056/.profile
[2016-10-19 15:32:28.423115] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:32:28.423219] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:       Filename :
/u056/.bash_logout
[2016-10-19 15:32:28.423247] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:       Lifetime : 202secs,
412682usecs
[2016-10-19 15:32:28.423345] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:32:28.423378] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:       Filename :
/u056/.bash_history
[2016-10-19 15:32:28.423400] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:       Lifetime : 202secs,
399903usecs
recipiente13
[2016-10-19 15:04:26.825754] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:04:26.825788] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3:       Filename : /u056/pp
[2016-10-19 15:04:26.825799] I [MSGID: 101055]
[client_t.c:415:gf_client_unref] 0-home-lab-3-server: Shutting down
connection cuentas09-lab.lab.dit.upm.es-1578-2016/10/19-05:00:48:464910-home-lab-3-client-3-0-0
[2016-10-19 15:04:26.825813] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3:       Lifetime : 25804secs,
917102usecs
[2016-10-19 15:04:26.825991] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:04:26.826026] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3:       Filename :
/u056/.local/share/evolution/tasks/system/tasks.ics
[2016-10-19 15:04:26.826050] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3:       Lifetime : 25774secs,
172800usecs
[2016-10-19 15:04:48.333183] I [MSGID: 115029]
[server-handshake.c:692:server_setvolume] 0-home-lab-3-server:
accepted client from cuentas09-lab.lab.dit.upm.es-1610-2016/10/19-15:04:46:807972-home-lab-3-client-3-0-0
(version: 3.8.5)
[2016-10-19 15:27:27.763705] E [MSGID: 115070]
[server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 149:
OPEN /u056/.Xauthority-c (a4fa9bc4-a09e-4eb8-968b-b77cb6d49fd3) ==>
(Permission denied) [Permission denied]
[2016-10-19 15:29:12.202792] E [MSGID: 115070]
[server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 267:
OPEN /u056/.Xauthority-c (ce222a86-9f74-42d1-8e18-1f80f607d766) ==>
(Permission denied) [Permission denied]
root@admin2:~#

As you can see, the glusterfs client part of the server fuse-mounting the volume shows a "/u056/kk.txt => -1 (Permission denied)" error that comes from a similar error in the glusterfs cluster, while at the nfs-kernel server log we can see a kernel trace directly related to the glusterfs fuse mount of the volume that says "nfsd4_process_open2 failed to open newly-created file! status=13".

This error can be reproduced any number of times exactly the same, is not sporadic.

Other test I've done at the client side with no logs at all at the server side, but the same error for the user, is the following:

u056@l056:~$ rm -f 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; echo "kk" > 444.txt
-bash: 444.txt: Permission denied
u056@l056:~$

We have been trying to reproduce this problem in a virtual environment using KVM, nfs-kernel-server, glusterfs, and ZFS over VFS, being unable to do it... This leads us to think that creating glusterfs volumes over ZFS over phisical disks has something to do with this problem, but we have no clue on how to track it down to the ZFS level, as there are no traces on the kernel.log or the syslog that we can use.

Thank you for your time.

Omar

Quoted 195 lines · J. Bruce Fields

Quoted 42 lines · Soumya Koduri

Quoted 41 lines · J. Bruce Fields

Quoted 40 lines · Soumya Koduri

Quoted 18 lines · J. Bruce Fields

Quoted 17 lines · Soumya Koduri

Quoted 16 lines · J. Bruce Fields

Quoted 15 lines · Soumya Koduri

Quoted 9 lines · Malahal Naineni

IIRC, permission checks are done in open(). write/read syscalls should NOT do much access checks (at least based on POSIX). This is why once an open is done, you remove permissions for that process, but it should still be able to read/write based on the open flags it did when it opened the file.

I don't know all the details of this defect, but gluster seems to be doing what it is supposed to do.

Right. Thanks for the correction. I assumed the behavior should be same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But looks like (from 'man creat') the open() call that creates a read-only file may well return a read/write file descriptor, which is the reason the following WRITE can succeed.

Quoted 9 lines

Quoted 6 lines

Quoted 4 lines

I forgot another complication, which is that knsfd actually does a temporary open before each read or write--I assume that's getting translated into fuse and gluster open operations?

yes. It is the OPEN done as part of NFS WRITE which fails with EACCESS error (with both NFSv3 and NFSv4 mounts).

Makes sense for v3, but I wouldn't normally expect the extra temporary open on v4 WRITEs. Could you share any details?

I re-tried the test on v4 mount using Fedora23 machine, acting as both NFS server and client (Linux#4.2.3-300.fc23.x86_64). Please find the pkt trace attached.

56 07:23:25.567134 ::1 -> ::1 NFS 288 V4 Call
WRITE StateID: 0xf934 Offset: 0 Len: 7
57 07:23:25.567233 192.168.122.17 -> 192.168.122.202 GlusterFS 188
V330 GETXATTR Call
58 07:23:25.567732 192.168.122.202 -> 192.168.122.17 GlusterFS 112
V330 GETXATTR Reply (Call In 57)
59 07:23:25.567881 192.168.122.17 -> 192.168.122.202 GlusterFS 164
V330 OPEN Call

Quoted 2 lines

Quoted 1 line

Remind me what kernel version your server is on?

NFS server is on fedora23 VM - Linux version 4.2.3-300.fc23.x86_64

We did reshuffle the code that does the temporary open on WRITE around there--but it looks right to me, and I can't reproduce an open on WRITE on that kernel myself.

Maybe there's some further fuse or gluster debugging that would help show where that open is coming from.

--b.

Quoted 141 lines

Thanks,
Soumya

Quoted 138 lines

--b.

Quoted 135 lines

60 07:23:25.568354 192.168.122.202 -> 192.168.122.17 GlusterFS 116
V330 OPEN Reply (Call In 59)
61 07:23:25.568570 ::1 -> ::1 NFS 144 V4 Reply
(Call In 56) WRITE Status: NFS4ERR_ACCESS

Thanks,
Soumya

Quoted 127 lines

--b.

Quoted 15 lines

63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS
164 V330 OPEN Call
65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS
164 [RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
116 V330 OPEN Reply (Call In 64)
67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
116 [RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call In 64)
68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
Reply (Call In 63) Error: NFS3ERR_ACCES

Quoted 105 lines

Thanks,
Soumya

Quoted 102 lines

In which case it might be worth experimenting with NFSv4 or with Jeff Layton's filehandle-caching patches. Neither's a real fix, but that could help confirm whether it's the temporary opens that are a problem.

--b.

Quoted 96 lines

Thanks,
Soumya

Quoted 92 lines

Regards, Malahal.

Quoted 89 lines · Soumya Koduri [[email protected]] wrote:

As mentioned by Bruce, GlusterFS doesn't have owner-override rule except for setattr.

I did few experiments to check why this test case passes on plain glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.

NFS-Ganesha (for most of the FSALs) seem to be passing the actual request credentials to the back-end filesystem only for CREATE(-like) and UNLINK fops. For all the remaining fops, it does the access check at its end and then perform the operation with root credentials. That's the reason WRITE succeeded in your case as NFS-Ganesha (like kernel-NFS) skipped the access check if the request caller_uid proved to be the file's owner.

In case of native GlusterFS FUSE mount, there is no OPEN fop involved. WRITE is performed on the fd returned by CREATE. And strangely GlusterFS seem to be doing certain access checks only during OPEN but not for WRITE (this seems like a bug and probably needs to be fixed in Gluster).

Thanks,
Soumya

Quoted 60 lines · Omar Walid Llorente

Thank you Bruce, others, for the responses. I send attached a complete
capture of the issue, including the glusterfs transactions.

Hope this helps to clear where may it be...

Omar

El 10/12/15 a las 15:44, J. Bruce Fields escribió:
On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
packets (you can find them in attached file nfs-problem-nks.pcap.zip).
Sorry for being so late.

What I did was the following:

1st) Create the RO file:
cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:
root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
may be helpful to see what GlusterFS operations are being processed
as part of NFS WRITE call (which has failed in this case).

Could you please try taking the packet trace on the machine where
NFS server is running (without filtering out based on the port
number).

Also I tried out the same test on Fedora22 machine, but haven't run
into any issue. What are the fuse mount options you have used to
mount gluster volume?
Oh, I think this is a simple problem (but maybe hard to fix). The
capture shows NFSv3 traffic like:

CREATE -> OK
SETATTR (mode set to 0400) -> OK
WRITE -> NFS3ERR_ACCES

That write would succeed locally (because the mode doesn't matter to a
local application that already holds the file open). It would fail over
NFSv3, which doesn't know about the open--except that there's a hack for
this case: NFSv3 servers allow IO operations to ignore the mode, if the
operation comes from the owner of the file. NFSv3 clients are then
careful to perform necessary access checks on open to ensure that this
owner-override rule doesn't grant too many permissions.

That allows NFSv3 applications to see behavior that's mostly like a
local filesystem, without opening much of a security hole (since the
owner could always chmod anyway).

So, knfsd is making this special exception--but gluster (which I believe
it's exporting in this case, via fuse?)--probably doesn't.... I'm not
sure what you can do about that.

--b.

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 8 lines

----------------------------------------------------------------
Centro de Cálculo         Depto. Ingeniería Sistemas Telemáticos
E-mail: [email protected]        Universidad Politécnica de Madrid
Fax:(+34) 913367333                 E.T.S. Ing. Telecomunicación
Tel:(+34) 915495700-Ext.3005                28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------

Hi Bruce, thanks for your answer, that is greatly appreciated.

This side of the problem feels a little bit lonely and maybe we can manage to get to know what is happening if you have some time.

We have been doing tests and we have found a way to replicate the problem in a virtual environment without ZFS, only with glusterfs and nfs-kernel-server, where we can give you root password or you can replicate it at home.

It is closely related to non-root users (with root credentials the problem does not happen).

If you think this is good news and feel like checking, tell me and next week we will prepare it and I'll forward you the data for both kind of tests.

Thank you very much for your time.

Omar

Quoted 567 lines · J. Bruce Fields

Quoted 11 lines · Omar Walid Llorente

Resending because it has not been accepted by [email protected] because of the HTML... Sorry if you get it twice.

Thanks again.

Omar

--- Forwarded message ---

Hi Bruce, others.

Sory for the delayed response. This is a fairly complicated setup that may require people with nfs, gluster, and zfs experience to debug.

The server does need to be able to bypass certain permission checks on open so that it can open a newly created read-only file for write. Probably one of those lower layers is insisting on permission checks where it shouldn't. That's all I can say from a quick look.

--b.

Quoted 545 lines

I come back to this issue again in the hope that these evidences I bring can help to give some light to the problem.

We have exactly the same environment that last year when I came to you, but with upgraded versions of gluster (3.8) and nfs-kernel-server (1.2.8) this time over an updated Ubuntu 16.04.

(NOTE: Environment: glusterfs-cluster made of ZFS datasets exports via glusterfs a distributed volume + nfs-kernel server mounts via fuse that glusterfs volume and reexports it via nfs-v3 + nfs-common client mounts via nfs that nfs exported volume)

The errors I get by the user side are exactly the same, but in the server side, I have different messages, which I forward to you.

* /TESTS/ and errors by the user side:

u056@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod
444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: cannot create regular file ‘kk.txt’: Permission denied
ls: cannot access kk.txt: No such file or directory
-r--r--r-- 1 u056 admincdc 7 oct 19  2016 444.txt
u056@l056:~$ ls -lrt
total 33
---------- 1 u056 admincdc    0 ene  4  1970 kk.txt
drwx------ 2 u056 alumno   4096 oct  3  2014 Desktop
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Público
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Descargas
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Documentos
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Vídeos
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Música
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:20 Plantillas
drwxr-xr-x 2 u056 alumno   4096 sep 23 12:22 Imágenes
-r--r--r-- 1 u056 admincdc    7 oct 19 17:34 444.txt

u056@l056:~$ mount | grep cuentas09
cuentas09:/home-3/u056 on /home/u056 type nfs (rw,noatime,intr,fsc,nolock,rsize=262140,wsize=262140,vers=4,addr=138.4.30.80,clientaddr=138.4.31.56)
u056@l056:~$ df -h | grep cuentas09
cuentas09:/home-3/u056              40T   55G   40T   1% /home/u056
u056@l056:~$

* /LOGS/ by the glusterfs client side (server side), versions of the software, info of the kernel and related kernel modules:

root@cuentas09-lab:/var/log/glusterfs# tail home-3.log
[2016-10-19 15:33:38.091300] I [io-stats.c:1574:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---
[2016-10-19 15:33:38.091331] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3:       Filename : /u056/444.txt
[2016-10-19 15:33:38.091344] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3:   BytesWritten : 7 bytes
[2016-10-19 15:33:38.091355] I [io-stats.c:1606:io_stats_dump_fd]
0-home-lab-3: Write 000004b+ : 1
[2016-10-19 15:34:13.199285] I [io-stats.c:1574:io_stats_dump_fd]
0-home-lab-3: --- fd stats ---
[2016-10-19 15:34:13.200291] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3:       Filename : /u056/444.txt
[2016-10-19 15:34:13.200313] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3:   BytesWritten : 7 bytes
[2016-10-19 15:34:13.200325] I [io-stats.c:1606:io_stats_dump_fd]
0-home-lab-3: Write 000004b+ : 1
[2016-10-19 15:34:13.211696] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-0:
remote operation failed. Path: /u056/kk.txt
(d8fc54a3-f2eb-4538-889a-17afdcfbb255) [Permission denied]
[2016-10-19 15:34:13.211758] W [fuse-bridge.c:989:fuse_fd_cbk]
0-glusterfs-fuse: 630: OPEN() /u056/kk.txt => -1 (Permission denied)

root@cuentas09-lab:/var/log/glusterfs# dpkg -l | grep -e gluster -e nfs
ii  glusterfs-client 3.8.5-ubuntu1~xenial1             amd64
clustered file-system (client package)
ii  glusterfs-common 3.8.5-ubuntu1~xenial1             amd64
GlusterFS common libraries and translator modules
ii  libnfsidmap2:amd64 0.25-5                            amd64
NFS idmapping library
ii  nfs-common 1:1.2.8-9ubuntu12                 amd64        NFS
support files common to client and server
ii  nfs-kernel-server 1:1.2.8-9ubuntu12                 amd64
support for NFS kernel server

root@cuentas09-lab:/var/log/glusterfs# uname -a
Linux cuentas09-lab.lab.dit.upm.es 4.4.0-43-generic #63-Ubuntu SMP
Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

root@cuentas09-lab:/var/log/glusterfs# lsmod | grep nfs
nfsd                  319488  13
auth_rpcgss            61440  1 nfsd
nfs_acl                16384  1 nfsd
lockd                  94208  1 nfsd
grace                  16384  2 nfsd,lockd
sunrpc                335872  21 nfsd,auth_rpcgss,lockd,nfs_acl
root@cuentas09-lab:/var/log/glusterfs#

* /LOGS/ by the nfs-kernel server side and basic configuration of the nfs-kernel-server:

root@cuentas09-lab:~# less /var/log/kern.log
root@cuentas09-lab:~# tail -30 !$
tail -30 /var/log/kern.log
Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631498]
[<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631504]
[<ffffffff81831c4f>] ret_from_fork+0x3f/0x70
Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631507]
[<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
Oct 19 17:30:24 cuentas09-lab kernel: [ 1573.631510] ---[ end trace
34db7650fa22d1d0 ]---
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814898] ------------[
cut here ]------------
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814948] WARNING: CPU: 0
PID: 1373 at /build/linux-BwgxJb/linux-4.4.0/fs/nfsd/nfs4proc.c:464
nfsd4_open+0x515/0x780 [nfsd]()
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814951]
nfsd4_process_open2 failed to open newly-created file! status=13
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.814954] Modules linked
in: vmw_vsock_vmci_transport vsock ppdev vmw_balloon coretemp joydev
input_leds serio_raw irda 8250_fintek parport_pc shpchp parport
i2c_piix4 crc_ccitt vmw_vmci mac_hid ib_iser nfsd rdma_cm iw_cm
ib_cm auth_rpcgss nfs_acl lockd grace ib_sa sunrpc ib_mad ib_core
ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4
btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq
async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath
linear vmwgfx ttm psmouse drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops mptspi mptscsih mptbase drm e1000
scsi_transport_spi pata_acpi floppy fjes
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815018] CPU: 0 PID:
1373 Comm: nfsd Tainted: G        W       4.4.0-43-generic
#63-Ubuntu
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815021] Hardware name:
VMware, Inc. VMware Virtual Platform/440BX Desktop Reference
Platform, BIOS 6.00 09/22/2009
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815024]
0000000000000286 00000000c0e912c3 ffff8800b93c7c80 ffffffff813f1f93
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815028]
ffff8800b93c7cc8 ffffffffc047b668 ffff8800b93c7cb8 ffffffff81081212
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815031]
ffff8800b9e43240 ffff8800b9e44068 000000000d000000 ffff8800b85e6000
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815035] Call Trace:
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815047]
[<ffffffff813f1f93>] dump_stack+0x63/0x90
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815055]
[<ffffffff81081212>] warn_slowpath_common+0x82/0xc0
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815058]
[<ffffffff810812ac>] warn_slowpath_fmt+0x5c/0x80
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815073]
[<ffffffffc04665db>] ? nfs4_free_ol_stateid+0x3b/0x40 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815085]
[<ffffffffc0459d05>] nfsd4_open+0x515/0x780 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815098]
[<ffffffffc045a2fa>] nfsd4_proc_compound+0x38a/0x660 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815108]
[<ffffffffc0446e78>] nfsd_dispatch+0xb8/0x200 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815132]
[<ffffffffc039eeac>] svc_process_common+0x40c/0x650 [sunrpc]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815151]
[<ffffffffc03a0273>] svc_process+0x103/0x1c0 [sunrpc]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815160]
[<ffffffffc04468cf>] nfsd+0xef/0x160 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815170]
[<ffffffffc04467e0>] ? nfsd_destroy+0x60/0x60 [nfsd]
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815174]
[<ffffffff810a0928>] kthread+0xd8/0xf0
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815178]
[<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815184]
[<ffffffff81831c4f>] ret_from_fork+0x3f/0x70
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815187]
[<ffffffff810a0850>] ? kthread_create_on_node+0x1e0/0x1e0
Oct 19 17:34:13 cuentas09-lab kernel: [ 1802.815190] ---[ end trace
34db7650fa22d1d1 ]---
root@cuentas09-lab:~#

root@cuentas09-lab:~# showmount -e
Export list for cuentas09-lab.lab.dit.upm.es:
/home-4 138.4.30.0/23
/home-3 138.4.30.0/23

root@cuentas09-lab:~# cat /etc/exports | grep -v ^#
/home-3 138.4.30.0/23(rw,fsid=3,insecure,no_subtree_check,async,no_root_squash)
/home-4 138.4.30.0/23(rw,fsid=4,insecure,no_subtree_check,async,no_root_squash)
root@cuentas09-lab:~#

root@cuentas09-lab:~# cat /etc/default/nfs-kernel-server | grep -v ^#
RPCNFSDCOUNT="8 -V 3"
RPCNFSDPRIORITY=0
RPCMOUNTDOPTS="--manage-gids"
NEED_SVCGSSD=""
RPCSVCGSSDOPTS=""
RPCNFSDOPTS="-d -V 3"
root@cuentas09-lab:~#

* LOGS by the glusterfs server side:

# for i in 10 11 12 13; do echo recipiente$i; ssh recipiente$i tail
/var/log/glusterfs/bricks/data-recipiente$i-gluster-home-lab-3.log;
done
recipiente10
[2016-10-19 15:27:48.172858] I [io-stats.c:1606:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3: Write 000008b+ : 1
[2016-10-19 15:29:32.626450] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:29:32.626545] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3:       Filename :
/u056/.local/share/tracker/data/tracker-store.journal
[2016-10-19 15:29:32.626572] I [io-stats.c:1594:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3:   BytesWritten : 8 bytes
[2016-10-19 15:29:32.626594] I [io-stats.c:1606:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3: Write 000008b+ : 1
[2016-10-19 15:30:23.997592] E [MSGID: 115070]
[server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 372:
OPEN /u056/kk.txt (cc97d150-c725-42b6-9aa5-50328f281e06) ==>
(Permission denied) [Permission denied]
[2016-10-19 15:32:28.423380] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:32:28.423478] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3:       Filename :
/u056/.bashrc
[2016-10-19 15:32:28.423504] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente10-gluster-home-lab-3:       Lifetime : 280secs,
232488usecs
[2016-10-19 15:34:13.176805] E [MSGID: 115070]
[server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 424:
OPEN /u056/kk.txt (d8fc54a3-f2eb-4538-889a-17afdcfbb255) ==>
(Permission denied) [Permission denied]
recipiente11
[2016-10-19 15:33:38.057524] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:       Filename :
/u056/444.txt
[2016-10-19 15:33:38.057548] I [io-stats.c:1594:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:   BytesWritten : 7 bytes
[2016-10-19 15:33:38.057570] I [io-stats.c:1606:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3: Write 000004b+ : 1
[2016-10-19 15:34:13.164629] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:34:13.164722] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:       Filename :
/u056/444.txt
[2016-10-19 15:34:13.164769] I [io-stats.c:1594:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:   BytesWritten : 7 bytes
[2016-10-19 15:34:13.164791] I [io-stats.c:1606:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3: Write 000004b+ : 1
[2016-10-19 15:36:29.066078] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:36:29.066180] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:       Filename :
/u056/444.txt
[2016-10-19 15:36:29.066206] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente11-gluster-home-lab-3:       Lifetime : 136secs,
894418usecs
recipiente12
[2016-10-19 15:29:06.022950] I [io-stats.c:1594:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:   BytesWritten : 986 bytes
[2016-10-19 15:29:06.022972] I [io-stats.c:1606:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3: Write 000512b+ : 1
[2016-10-19 15:29:32.646111] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:29:32.646214] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:       Filename :
/u056/.profile
[2016-10-19 15:32:28.423115] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:32:28.423219] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:       Filename :
/u056/.bash_logout
[2016-10-19 15:32:28.423247] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:       Lifetime : 202secs,
412682usecs
[2016-10-19 15:32:28.423345] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:32:28.423378] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:       Filename :
/u056/.bash_history
[2016-10-19 15:32:28.423400] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente12-gluster-home-lab-3:       Lifetime : 202secs,
399903usecs
recipiente13
[2016-10-19 15:04:26.825754] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:04:26.825788] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3:       Filename : /u056/pp
[2016-10-19 15:04:26.825799] I [MSGID: 101055]
[client_t.c:415:gf_client_unref] 0-home-lab-3-server: Shutting down
connection cuentas09-lab.lab.dit.upm.es-1578-2016/10/19-05:00:48:464910-home-lab-3-client-3-0-0
[2016-10-19 15:04:26.825813] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3:       Lifetime : 25804secs,
917102usecs
[2016-10-19 15:04:26.825991] I [io-stats.c:1574:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3: --- fd stats ---
[2016-10-19 15:04:26.826026] I [io-stats.c:1579:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3:       Filename :
/u056/.local/share/evolution/tasks/system/tasks.ics
[2016-10-19 15:04:26.826050] I [io-stats.c:1584:io_stats_dump_fd]
0-/data/recipiente13-gluster-home-lab-3:       Lifetime : 25774secs,
172800usecs
[2016-10-19 15:04:48.333183] I [MSGID: 115029]
[server-handshake.c:692:server_setvolume] 0-home-lab-3-server:
accepted client from cuentas09-lab.lab.dit.upm.es-1610-2016/10/19-15:04:46:807972-home-lab-3-client-3-0-0
(version: 3.8.5)
[2016-10-19 15:27:27.763705] E [MSGID: 115070]
[server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 149:
OPEN /u056/.Xauthority-c (a4fa9bc4-a09e-4eb8-968b-b77cb6d49fd3) ==>
(Permission denied) [Permission denied]
[2016-10-19 15:29:12.202792] E [MSGID: 115070]
[server-rpc-fops.c:1472:server_open_cbk] 0-home-lab-3-server: 267:
OPEN /u056/.Xauthority-c (ce222a86-9f74-42d1-8e18-1f80f607d766) ==>
(Permission denied) [Permission denied]
root@admin2:~#

As you can see, the glusterfs client part of the server fuse-mounting the volume shows a "/u056/kk.txt => -1 (Permission denied)" error that comes from a similar error in the glusterfs cluster, while at the nfs-kernel server log we can see a kernel trace directly related to the glusterfs fuse mount of the volume that says "nfsd4_process_open2 failed to open newly-created file! status=13".

This error can be reproduced any number of times exactly the same, is not sporadic.

Other test I've done at the client side with no logs at all at the server side, but the same error for the user, is the following:

u056@l056:~$ rm -f 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; echo "kk" > 444.txt
-bash: 444.txt: Permission denied
u056@l056:~$

We have been trying to reproduce this problem in a virtual environment using KVM, nfs-kernel-server, glusterfs, and ZFS over VFS, being unable to do it... This leads us to think that creating glusterfs volumes over ZFS over phisical disks has something to do with this problem, but we have no clue on how to track it down to the ZFS level, as there are no traces on the kernel.log or the syslog that we can use.

Thank you for your time.

Omar

Quoted 195 lines · J. Bruce Fields

Quoted 42 lines · Soumya Koduri

Quoted 41 lines · J. Bruce Fields

Quoted 40 lines · Soumya Koduri

Quoted 18 lines · J. Bruce Fields

Quoted 17 lines · Soumya Koduri

Quoted 16 lines · J. Bruce Fields

Quoted 15 lines · Soumya Koduri

Quoted 9 lines · Malahal Naineni

IIRC, permission checks are done in open(). write/read syscalls should NOT do much access checks (at least based on POSIX). This is why once an open is done, you remove permissions for that process, but it should still be able to read/write based on the open flags it did when it opened the file.

I don't know all the details of this defect, but gluster seems to be doing what it is supposed to do.

Right. Thanks for the correction. I assumed the behavior should be same for both OPEN+WRITE vs CREATE+WRITE in the below scenario. But looks like (from 'man creat') the open() call that creates a read-only file may well return a read/write file descriptor, which is the reason the following WRITE can succeed.

Quoted 9 lines

Quoted 6 lines

Quoted 4 lines

I forgot another complication, which is that knsfd actually does a temporary open before each read or write--I assume that's getting translated into fuse and gluster open operations?

yes. It is the OPEN done as part of NFS WRITE which fails with EACCESS error (with both NFSv3 and NFSv4 mounts).

Makes sense for v3, but I wouldn't normally expect the extra temporary open on v4 WRITEs. Could you share any details?

I re-tried the test on v4 mount using Fedora23 machine, acting as both NFS server and client (Linux#4.2.3-300.fc23.x86_64). Please find the pkt trace attached.

56 07:23:25.567134 ::1 -> ::1 NFS 288 V4 Call
WRITE StateID: 0xf934 Offset: 0 Len: 7
57 07:23:25.567233 192.168.122.17 -> 192.168.122.202 GlusterFS 188
V330 GETXATTR Call
58 07:23:25.567732 192.168.122.202 -> 192.168.122.17 GlusterFS 112
V330 GETXATTR Reply (Call In 57)
59 07:23:25.567881 192.168.122.17 -> 192.168.122.202 GlusterFS 164
V330 OPEN Call

Quoted 2 lines

Quoted 1 line

Remind me what kernel version your server is on?

NFS server is on fedora23 VM - Linux version 4.2.3-300.fc23.x86_64

We did reshuffle the code that does the temporary open on WRITE around there--but it looks right to me, and I can't reproduce an open on WRITE on that kernel myself.

Maybe there's some further fuse or gluster debugging that would help show where that open is coming from.

--b.

Quoted 141 lines

Thanks,
Soumya

Quoted 138 lines

--b.

Quoted 135 lines

60 07:23:25.568354 192.168.122.202 -> 192.168.122.17 GlusterFS 116
V330 OPEN Reply (Call In 59)
61 07:23:25.568570 ::1 -> ::1 NFS 144 V4 Reply
(Call In 56) WRITE Status: NFS4ERR_ACCESS

Thanks,
Soumya

Quoted 127 lines

--b.

Quoted 15 lines

63 16:59:09.278651000 ::1 -> ::1 NFS 232 V3 WRITE
Call, FH: 0x49a35e54 Offset: 0 Len: 7 FILE_SYNC
64 16:59:09.278926000 192.168.122.1 -> 192.168.122.202 GlusterFS
164 V330 OPEN Call
65 16:59:09.278937000 192.168.122.1 -> 192.168.122.202 GlusterFS
164 [RPC retransmission of #64][TCP Retransmission] V330 OPEN Call
66 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
116 V330 OPEN Reply (Call In 64)
67 16:59:09.279459000 192.168.122.202 -> 192.168.122.1 GlusterFS
116 [RPC duplicate of #66][TCP Retransmission] V330 OPEN Reply (Call In 64)
68 16:59:09.279733000 ::1 -> ::1 NFS 212 V3 WRITE
Reply (Call In 63) Error: NFS3ERR_ACCES

Quoted 105 lines

Thanks,
Soumya

Quoted 102 lines

In which case it might be worth experimenting with NFSv4 or with Jeff Layton's filehandle-caching patches. Neither's a real fix, but that could help confirm whether it's the temporary opens that are a problem.

--b.

Quoted 96 lines

Thanks,
Soumya

Quoted 92 lines

Regards, Malahal.

Quoted 89 lines · Soumya Koduri [[email protected]] wrote:

As mentioned by Bruce, GlusterFS doesn't have owner-override rule
except for setattr.

I did few experiments to check why this test case passes on plain
glusterfs fuse mount & NFS-Ganesha but fails with kernel-NFS.

NFS-Ganesha (for most of the FSALs) seem to be passing the actual
request credentials to the back-end filesystem only for
CREATE(-like) and UNLINK fops. For all the remaining fops, it does
the access check at its end and then perform the operation with root
credentials. That's the reason WRITE succeeded in your case as
NFS-Ganesha (like kernel-NFS) skipped the access check if the
request caller_uid proved to be the file's owner.

In case of native GlusterFS FUSE mount, there is no OPEN fop
involved. WRITE is performed on the fd returned by CREATE. And
strangely GlusterFS seem to be doing certain access checks only
during OPEN but not for WRITE (this seems like a bug and probably
needs to be fixed in Gluster).

Thanks,
Soumya

On 12/14/2015 10:27 PM, Omar Walid Llorente wrote:
Thank you Bruce, others, for the responses. I send attached a complete
capture of the issue, including the glusterfs transactions.

Hope this helps to clear where may it be...

Omar

El 10/12/15 a las 15:44, J. Bruce Fields escribió:
On Thu, Dec 10, 2015 at 05:59:33PM +0530, Soumya Koduri wrote:
On 12/10/2015 04:02 PM, Omar Walid Llorente wrote:
Hi, Jeff, Bruce, finally I got some time to get the capture of the nfs
packets (you can find them in attached file nfs-problem-nks.pcap.zip).
Sorry for being so late.

What I did was the following:

1st) Create the RO file:
cdc@l056:~/prueba-git$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt;

2nd) Init the capture:
root@l056:~# tcpdump -i eth2 -w /tmp/nfs.pcap -s 512 port 2049
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size
512 bytes

GlusterFS protocol is added to wireshark from version 1.8.0 [1]. It
may be helpful to see what GlusterFS operations are being processed
as part of NFS WRITE call (which has failed in this case).

Could you please try taking the packet trace on the machine where
NFS server is running (without filtering out based on the port
number).

Also I tried out the same test on Fedora22 machine, but haven't run
into any issue. What are the fuse mount options you have used to
mount gluster volume?
Oh, I think this is a simple problem (but maybe hard to fix). The
capture shows NFSv3 traffic like:

CREATE -> OK
SETATTR (mode set to 0400) -> OK
WRITE -> NFS3ERR_ACCES

That write would succeed locally (because the mode doesn't matter to a
local application that already holds the file open). It would fail over
NFSv3, which doesn't know about the open--except that there's a hack for
this case: NFSv3 servers allow IO operations to ignore the mode, if the
operation comes from the owner of the file. NFSv3 clients are then
careful to perform necessary access checks on open to ensure that this
owner-override rule doesn't grant too many permissions.

That allows NFSv3 applications to see behavior that's mostly like a
local filesystem, without opening much of a security hole (since the
owner could always chmod anyway).

So, knfsd is making this special exception--but gluster (which I believe
it's exporting in this case, via fuse?)--probably doesn't.... I'm not
sure what you can do about that.

--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 8 lines

----------------------------------------------------------------
Centro de Cálculo         Depto. Ingeniería Sistemas Telemáticos
E-mail: [email protected]        Universidad Politécnica de Madrid
Fax:(+34) 913367333                 E.T.S. Ing. Telecomunicación
Tel:(+34) 915495700-Ext.3005                28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------

Signature · 8 lines

----------------------------------------------------------------
Centro de Cálculo         Depto. Ingeniería Sistemas Telemáticos
E-mail: [email protected]        Universidad Politécnica de Madrid
Fax:(+34) 913367333                 E.T.S. Ing. Telecomunicación
Tel:(+34) 915495700-Ext.3005                28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------

Hi Bruce, thanks for your answer, that is greatly appreciated.

This side of the problem feels a little bit lonely and maybe we can manage to get to know what is happening if you have some time.

We have been doing tests and we have found a way to replicate the problem in a virtual environment without ZFS, only with glusterfs and nfs-kernel-server, where we can give you root password or you can replicate it at home.

It is closely related to non-root users (with root credentials the problem does not happen).

If you think this is good news and feel like checking, tell me and next week we will prepare it and I'll forward you the data for both kind of tests.

So if I'm reading right, the setup is

        nfs client---NFSv3--->knfsd---FUSE-->gluster

I hate to just pass the buck, but my bet is on the issue being either on the gluster side, or on limitations of the fuse protocol itself.

Cc'ing Miklos as I believe he did the original fuse nfsd export implementation. Miklos, does fuse has a way for file owners to override permissions? We need this sometimes because for example, an application may hold a write open on a read-only file, but the server doesn't itself already have an open (which can happen for lots of reasons).

Quoted 4 lines · J. Bruce Fields

So if I'm reading right, the setup is

         nfs client---NFSv3--->knfsd---FUSE-->gluster

Quoted 8 lines

I hate to just pass the buck, but my bet is on the issue being either on the gluster side, or on limitations of the fuse protocol itself.

Cc'ing Miklos as I believe he did the original fuse nfsd export implementation. Miklos, does fuse has a way for file owners to override permissions? We need this sometimes because for example, an application may hold a write open on a read-only file, but the server doesn't itself already have an open (which can happen for lots of reasons).

Is glusterfs using "default_permissions" fuse option?

If so, then the above shouldn't be an issue, since permission checking is done by generic_permission() which works the same for all filesystems.

If not, then there's the same problem as re-exporting NFSv4 as NFSv3 would have: the permissions are not checked in ->permission(), but in e.g. ->atomic_open() so nfsd's override mechanism wouldn't work.

Thanks,
Miklos

In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu SMP Mon

Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse included

(not as a module) so if it would be necessary to change the fuse mount options some help would be needed.

Quoted 4 lines · Omar Walid Llorente

In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu SMP Mon

Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse included

(not as a module) so if it would be necessary to change the fuse mount options some help would be needed.

Can you reproduce this on latest (e.g 4.8) kernel?

Also fuse debug logs (with whaterver kernel) could be useful; not sure how to enable it for glusterfs, usual convention for fuse filesystems is "-d".

Thanks,
Miklos

Yes, of course, we can reproduce the problem on latest (e.g 4.8) kernel.

Tested on a server with fedora 24 (Linux cuentas11.lab.dit.upm.es 4.8.4-200.fc24.x86_64 #1 SMP Tue Oct 25 13:06:04 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux), glusterfs-fuse-3.8.5-1.fc24.x86_64, fuse-2.9.7-1.fc24.x86_64 and nfs-utils-1.3.4-1.rc2.fc24.x86_64 that mounts as a glusterfs-client and exports as a nfs-server a ditributed glusterfs volume (that comes from a 3.7.4 glusterfs server over Ubuntu 14.04.

[root@cuentas11 ~]# cat /proc/mounts | grep home recipiente8hd:/home-lab-3.tcp /home-3 fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0
[root@cuentas11 ~]#

As you may remember the schema is the one that follows (glusterfs volume
name: home-lab-3, glusterfs client mount point: /home-3, nfs client mount point: /home):

nfs client (v1.2.8) ---> NFSv3 ---> knfsd (v1.3.4 over a Linux v4.8.4 kernel) ---> FUSE (v2.9.7) --> glusterfs client (v3.8.5) --> GLUSTER over TCP --> glusterfs server (v3.7.4)

The errors at the nfs-client side are the same:

cdc@l056:~$ mount | grep "/home/cdc"
cuentas11:/home-3/cdc on /home/cdc type nfs
(rw,noatime,nfsvers=3,vers=3,tcp,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.86)

cdc@l056:~$
cdc@l056:~$
cdc@l056:~$

cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied

(quoted diff) +0 -2

-r--r--r-- 1 cdc admincdc 7 nov 18  2016 444.txt
-r--r--r-- 1 cdc admincdc 0 nov 18  2016 kk.txt

cdc@l056:~$
cdc@l056:~$
cdc@l056:~$

cdc@l056:~$ sudo su
[sudo] password for cdc:
root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt

(quoted diff) +0 -2

-r--r--r-- 1 root root 7 nov 18  2016 444.txt
-r--r--r-- 1 root root 7 nov 18  2016 kk.txt

root@l056:/home/cdc#
root@l056:/home/cdc#

root@l056:/home/cdc# exit
cdc@l056:~$

See at the end the logs with and without the debug option activated at the glusterfs client.

Thanks again, and sorry for the long message.

Omar

* Logs on the glusterfs client side with NO DEBUG:

[2016-11-18 15:45:43.681215] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote operation failed. Path: /cdc/kk.txt (cc98e831-3410-4202-8254-2d4b48e8ea68) [Permission denied] [2016-11-18 15:45:43.681297] W [fuse-bridge.c:989:fuse_fd_cbk] 0-glusterfs-fuse: 399: OPEN() /cdc/kk.txt => -1 (Permission denied) [2016-11-18 15:45:55.457922] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 15:45:55.457978] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 15:45:55.457997] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 15:45:55.458011] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 15:45:55.471933] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---

* Logs on the glusterfs client side with DEBUG activated:

[2016-11-18 16:06:17.040092] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:06:17.045986] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.046132] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.046833] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret 0 [Invalid argument] [2016-11-18 16:06:17.047671] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop
[2016-11-18 16:06:17.050539] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.050867] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.051948] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.051871] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.051946] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.062108] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret 0 [Invalid argument] [2016-11-18 16:06:17.063692] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.064015] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.065068] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 4 times between [2016-11-18 16:06:17.040092] and [2016-11-18 16:06:17.064175] [2016-11-18 16:06:17.065066] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.065232] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.065755] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.065897] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.066016] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt missing on subvol home-lab-3-client-0
[2016-11-18 16:06:17.066104] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.066594] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.066669] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.066771] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.066898] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.066906] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067019] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.066915] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067066] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.067107] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:06:17.067151] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:06:17.067179] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067206] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067250] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067275] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067308] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067387] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.068157] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.068281] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.068738] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.068781] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.068805] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt missing on subvol home-lab-3-client-0
[2016-11-18 16:06:17.068827] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.069286] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.069345] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.069452] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.069507] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.069556] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.069585] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.069939] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.069983] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.070005] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:06:17.070038] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:06:17.070098] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070128] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070154] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070178] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070208] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070251] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070537] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled as normal fop
[2016-11-18 16:06:17.070826] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.071100] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
10456189239296 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071142] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
10499693871104 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071183] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
10500911267840 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071343] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
8600432410624 and avail_inodes is: 99.00
[2016-11-18 16:06:17.073943] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.074273] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:06:17.075035] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled as fast fop
[2016-11-18 16:06:17.075913] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled as slow fop
[2016-11-18 16:06:17.076205] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled as fast fop
[2016-11-18 16:06:17.076636] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.077282] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120276.
[2016-11-18 16:06:17.077981] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:06:17.078032] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 16:06:17.078049] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:06:17.078064] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:06:17.079156] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.079794] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.082196] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.082772] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.082318] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.082770] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.083168] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.083282] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:06:17.083320] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.083991] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084043] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0 [2016-11-18 16:06:17.084048] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084092] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3 [2016-11-18 16:06:17.084114] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084167] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1 [2016-11-18 16:06:17.084091] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084377] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2 [2016-11-18 16:06:17.084406] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:06:17.084511] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:06:17.084534] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084683] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084783] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084857] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084889] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.085001] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.086098] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:06:17.087256] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.087173] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.087254] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.087749] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.087795] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.088014] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:06:17.088036] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.088598] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.088639] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0 [2016-11-18 16:06:17.088804] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.088855] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2 [2016-11-18 16:06:17.088819] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089069] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3 [2016-11-18 16:06:17.089380] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089421] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1 [2016-11-18 16:06:17.089443] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:06:17.089463] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:06:17.089575] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089601] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089712] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089738] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.090006] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.090156] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.090648] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled as normal fop
[2016-11-18 16:06:17.090732] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.091499] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.091843] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.093831] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.094711] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.093216] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.094709] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled as fast fop
[2016-11-18 16:06:17.095280] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote operation failed. Path: /cdc/kk.txt (09bfb6fa-82fb-463b-abd9-bb7ef88e10c0) [Permission denied] [2016-11-18 16:06:17.095341] D [MSGID: 0] [client-rpc-fops.c:448:client3_3_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095366] D [MSGID: 0] [dht-inode-read.c:45:dht_open_cbk] 0-home-lab-3-dht: subvolume home-lab-3-client-3 returned -1 [Permission denied]
[2016-11-18 16:06:17.095480] D [MSGID: 0] [dht-inode-read.c:59:dht_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-dht returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095671] D [MSGID: 0] [read-ahead.c:101:ra_open_cbk] 0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-read-ahead returned -1 error: Permission denied [Permission denied] [2016-11-18 16:06:17.095759] D [MSGID: 0] [io-cache.c:605:ioc_open_cbk] 0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: Permission denied [Permission denied] [2016-11-18 16:06:17.095963] D [MSGID: 0] [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095988] D [MSGID: 0] [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.096275] D [MSGID: 0] [io-stats.c:1936:io_stats_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.096362] W [fuse-bridge.c:989:fuse_fd_cbk] 0-glusterfs-fuse: 338: OPEN() /cdc/kk.txt => -1 (Permission denied) [2016-11-18 16:06:17.097104] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.097674] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.099912] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:24.785225] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:06:24.785338] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:06:24.785359] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:06:24.785377] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-3: returning as transport is already disconnected OR there are no frames (0 || 0)

** TEST DONE as user cdc --> ERROR

[2016-11-18 16:06:46.737686] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184] (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce] ))))) 0-: 138.4.2.246:49172: ping timer event already removed [2016-11-18 16:06:49.545698] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184] (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce] ))))) 0-: 138.4.2.177:49157: ping timer event already removed [2016-11-18 16:06:49.546474] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184] (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce] ))))) 0-: 138.4.2.189:49157: ping timer event already removed [2016-11-18 16:06:49.547093] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184] (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce] ))))) 0-: 138.4.2.248:49157: ping timer event already removed

** user CDC becomed root at this point

[2016-11-18 16:07:11.907892] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 7 times between [2016-11-18 16:06:17.086098] and [2016-11-18 16:07:11.906944] [2016-11-18 16:07:11.907884] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.908875] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.909179] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.909256] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop" repeated 2 times between [2016-11-18 16:06:17.047671] and [2016-11-18 16:07:11.908114] [2016-11-18 16:07:11.909254] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.911109] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.911018] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.911107] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop
[2016-11-18 16:07:11.911871] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.912045] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.913067] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 2 times between [2016-11-18 16:07:11.909254] and [2016-11-18 16:07:11.912165] [2016-11-18 16:07:11.913065] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.913193] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.913710] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.913757] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.913788] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt missing on subvol home-lab-3-client-0
[2016-11-18 16:07:11.913810] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.914336] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914379] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.914424] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914448] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.914501] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914617] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.914649] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914698] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.914724] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:07:11.914743] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:07:11.914764] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914791] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914825] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914855] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914880] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914947] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.915569] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.915644] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.916090] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.916134] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.916158] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt missing on subvol home-lab-3-client-0
[2016-11-18 16:07:11.916185] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.916593] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.916780] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.916878] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.916994] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.917076] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.916906] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917117] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.917153] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.917205] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:07:11.917368] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:07:11.917437] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917467] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917496] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917521] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917551] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917726] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917958] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled as normal fop
[2016-11-18 16:07:11.918230] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.918488] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
10456166432768 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918570] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
8600417075200 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918603] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
10500905500672 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918675] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
10499686531072 and avail_inodes is: 99.00
[2016-11-18 16:07:11.919095] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.919377] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.920040] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled as fast fop
[2016-11-18 16:07:11.920889] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled as slow fop
[2016-11-18 16:07:11.921051] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled as fast fop
[2016-11-18 16:07:11.921583] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.922068] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120276.
[2016-11-18 16:07:11.922936] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:07:11.922988] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 16:07:11.923004] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:07:11.923026] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:07:11.923779] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.924504] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.926692] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop" repeated 2 times between [2016-11-18 16:06:17.097104] and [2016-11-18 16:07:11.923946] [2016-11-18 16:07:11.926690] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.926858] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.927327] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.927376] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.927401] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:07:11.927422] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.927940] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.927987] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0 [2016-11-18 16:07:11.928043] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928055] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928058] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928170] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1 [2016-11-18 16:07:11.928174] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3 [2016-11-18 16:07:11.928277] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2 [2016-11-18 16:07:11.928360] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:07:11.928381] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:07:11.928403] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928431] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928456] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928480] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928511] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928753] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.930485] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.929862] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.930483] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.930602] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.931071] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.931124] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.931148] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:07:11.931348] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.931880] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.931927] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0 [2016-11-18 16:07:11.932002] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.931994] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.931991] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932078] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2 [2016-11-18 16:07:11.932116] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1 [2016-11-18 16:07:11.932315] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3 [2016-11-18 16:07:11.932363] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:07:11.932383] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:07:11.932407] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932446] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932471] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932494] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932518] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932725] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.932545] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932722] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled as normal fop
[2016-11-18 16:07:11.933008] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.934086] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.934323] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.934922] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.935497] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.936102] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.936903] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled as fast fop
[2016-11-18 16:07:11.937540] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled as slow fop
[2016-11-18 16:07:11.937682] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled as fast fop
[2016-11-18 16:07:11.938267] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.938757] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120652.
[2016-11-18 16:07:11.938994] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:07:11.939072] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/kk.txt
[2016-11-18 16:07:11.939096] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:07:11.939112] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:07:11.939448] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.940028] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.942130] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0

** TEST DONE as root --> NO PROBLEM

[2016-11-18 16:07:28.794150] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:07:31.794656] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:07:31.794925] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:07:31.794984] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-3: returning as transport is already disconnected OR there are no frames (0 || 0)

** USER ROOT EXITED at this point

Quoted 17 lines · Miklos Szeredi

Quoted 6 lines · Omar Walid Llorente

In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu SMP Mon

Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse

included
(not as a module) so if it would be necessary to change the fuse mount options some help would be needed.

Can you reproduce this on latest (e.g 4.8) kernel?

Also fuse debug logs (with whaterver kernel) could be useful; not sure how to enable it for glusterfs, usual convention for fuse filesystems is "-d".

Thanks,
Miklos

Hi,

Sorry for the delay. I could reproduce the issue on my gluster setup as well. The error is generated at posix-acl xlator layer of glusterfs server.

As Bruce mentioned in the earlier mail, as part of WRITE call, NFS-server issues OPEN() with O_WRONLY flag set. Since the attributes of the file "kk.txt" when created was 0400, the corresponding acl maps to below ACL -

[root@dhcp35-197 vol1]# getfacl kk.txt
# file: kk.txt
# owner: guest
# group: mock
user::r--
group::---
other::---

And hence the permission check for WRITE access fails. Also not just glusterfs, I see similar behavior with ext4 filesystem as well when exported via kernel-NFS server.

So I am not sure if fixing glusterfs
for this use-case is valid.

Thanks,
Soumya

Quoted 1,150 lines · Omar Walid Llorente

Yes, of course, we can reproduce the problem on latest (e.g 4.8) kernel.

Tested on a server with fedora 24 (Linux cuentas11.lab.dit.upm.es 4.8.4-200.fc24.x86_64 #1 SMP Tue Oct 25 13:06:04 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux), glusterfs-fuse-3.8.5-1.fc24.x86_64, fuse-2.9.7-1.fc24.x86_64 and nfs-utils-1.3.4-1.rc2.fc24.x86_64 that mounts as a glusterfs-client and exports as a nfs-server a ditributed glusterfs volume (that comes from a 3.7.4 glusterfs server over Ubuntu 14.04.

[root@cuentas11 ~]# cat /proc/mounts | grep home recipiente8hd:/home-lab-3.tcp /home-3 fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0
[root@cuentas11 ~]#

As you may remember the schema is the one that follows (glusterfs volume
name: home-lab-3, glusterfs client mount point: /home-3, nfs client mount point: /home):

nfs client (v1.2.8) ---> NFSv3 ---> knfsd (v1.3.4 over a Linux v4.8.4 kernel) ---> FUSE (v2.9.7) --> glusterfs client (v3.8.5) --> GLUSTER over TCP --> glusterfs server (v3.7.4)

The errors at the nfs-client side are the same:

cdc@l056:~$ mount | grep "/home/cdc"
cuentas11:/home-3/cdc on /home/cdc type nfs
(rw,noatime,nfsvers=3,vers=3,tcp,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.86)

cdc@l056:~$
cdc@l056:~$
cdc@l056:~$

cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied

(quoted diff) +0 -2

-r--r--r-- 1 cdc admincdc 7 nov 18  2016 444.txt
-r--r--r-- 1 cdc admincdc 0 nov 18  2016 kk.txt

cdc@l056:~$
cdc@l056:~$
cdc@l056:~$

cdc@l056:~$ sudo su
[sudo] password for cdc:
root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt

(quoted diff) +0 -2

-r--r--r-- 1 root root 7 nov 18  2016 444.txt
-r--r--r-- 1 root root 7 nov 18  2016 kk.txt

root@l056:/home/cdc#
root@l056:/home/cdc#

root@l056:/home/cdc# exit
cdc@l056:~$

See at the end the logs with and without the debug option activated at the glusterfs client.

Thanks again, and sorry for the long message.

Omar

* Logs on the glusterfs client side with NO DEBUG:

[2016-11-18 15:45:43.681215] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote operation failed. Path: /cdc/kk.txt (cc98e831-3410-4202-8254-2d4b48e8ea68) [Permission denied] [2016-11-18 15:45:43.681297] W [fuse-bridge.c:989:fuse_fd_cbk] 0-glusterfs-fuse: 399: OPEN() /cdc/kk.txt => -1 (Permission denied) [2016-11-18 15:45:55.457922] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 15:45:55.457978] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 15:45:55.457997] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 15:45:55.458011] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 15:45:55.471933] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---

* Logs on the glusterfs client side with DEBUG activated:

[2016-11-18 16:06:17.040092] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:06:17.045986] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.046132] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.046833] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret 0 [Invalid argument] [2016-11-18 16:06:17.047671] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop
[2016-11-18 16:06:17.050539] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.050867] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.051948] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.051871] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.051946] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.062108] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret 0 [Invalid argument] [2016-11-18 16:06:17.063692] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.064015] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.065068] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 4 times between [2016-11-18 16:06:17.040092] and [2016-11-18 16:06:17.064175] [2016-11-18 16:06:17.065066] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.065232] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.065755] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.065897] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.066016] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt missing on subvol home-lab-3-client-0
[2016-11-18 16:06:17.066104] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.066594] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.066669] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.066771] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.066898] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.066906] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067019] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.066915] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067066] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.067107] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:06:17.067151] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:06:17.067179] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067206] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067250] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067275] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067308] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067387] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.068157] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.068281] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.068738] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.068781] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.068805] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt missing on subvol home-lab-3-client-0
[2016-11-18 16:06:17.068827] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.069286] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.069345] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.069452] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.069507] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.069556] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.069585] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.069939] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.069983] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.070005] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:06:17.070038] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:06:17.070098] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070128] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070154] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070178] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070208] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070251] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070537] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled as normal fop
[2016-11-18 16:06:17.070826] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.071100] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
10456189239296 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071142] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
10499693871104 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071183] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
10500911267840 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071343] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
8600432410624 and avail_inodes is: 99.00
[2016-11-18 16:06:17.073943] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.074273] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:06:17.075035] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled as fast fop
[2016-11-18 16:06:17.075913] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled as slow fop
[2016-11-18 16:06:17.076205] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled as fast fop
[2016-11-18 16:06:17.076636] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.077282] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120276.
[2016-11-18 16:06:17.077981] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:06:17.078032] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 16:06:17.078049] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:06:17.078064] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:06:17.079156] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.079794] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.082196] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.082772] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.082318] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.082770] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.083168] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.083282] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:06:17.083320] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.083991] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084043] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0 [2016-11-18 16:06:17.084048] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084092] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3 [2016-11-18 16:06:17.084114] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084167] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1 [2016-11-18 16:06:17.084091] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084377] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2 [2016-11-18 16:06:17.084406] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:06:17.084511] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:06:17.084534] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084683] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084783] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084857] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084889] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.085001] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.086098] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:06:17.087256] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.087173] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.087254] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.087749] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.087795] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.088014] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:06:17.088036] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.088598] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.088639] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0 [2016-11-18 16:06:17.088804] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.088855] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2 [2016-11-18 16:06:17.088819] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089069] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3 [2016-11-18 16:06:17.089380] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089421] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1 [2016-11-18 16:06:17.089443] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:06:17.089463] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:06:17.089575] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089601] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089712] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089738] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.090006] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.090156] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.090648] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled as normal fop
[2016-11-18 16:06:17.090732] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.091499] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.091843] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.093831] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.094711] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.093216] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.094709] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled as fast fop
[2016-11-18 16:06:17.095280] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote operation failed. Path: /cdc/kk.txt (09bfb6fa-82fb-463b-abd9-bb7ef88e10c0) [Permission denied] [2016-11-18 16:06:17.095341] D [MSGID: 0] [client-rpc-fops.c:448:client3_3_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095366] D [MSGID: 0] [dht-inode-read.c:45:dht_open_cbk] 0-home-lab-3-dht: subvolume home-lab-3-client-3 returned -1 [Permission denied]
[2016-11-18 16:06:17.095480] D [MSGID: 0] [dht-inode-read.c:59:dht_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-dht returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095671] D [MSGID: 0] [read-ahead.c:101:ra_open_cbk] 0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-read-ahead returned -1 error: Permission denied [Permission denied] [2016-11-18 16:06:17.095759] D [MSGID: 0] [io-cache.c:605:ioc_open_cbk] 0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: Permission denied [Permission denied] [2016-11-18 16:06:17.095963] D [MSGID: 0] [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095988] D [MSGID: 0] [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.096275] D [MSGID: 0] [io-stats.c:1936:io_stats_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.096362] W [fuse-bridge.c:989:fuse_fd_cbk] 0-glusterfs-fuse: 338: OPEN() /cdc/kk.txt => -1 (Permission denied) [2016-11-18 16:06:17.097104] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.097674] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.099912] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:24.785225] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:06:24.785338] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:06:24.785359] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:06:24.785377] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-3: returning as transport is already disconnected OR there are no frames (0 || 0)

** TEST DONE as user cdc --> ERROR

[2016-11-18 16:06:46.737686] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184] (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce] ))))) 0-: 138.4.2.246:49172: ping timer event already removed [2016-11-18 16:06:49.545698] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184] (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce] ))))) 0-: 138.4.2.177:49157: ping timer event already removed [2016-11-18 16:06:49.546474] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184] (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce] ))))) 0-: 138.4.2.189:49157: ping timer event already removed [2016-11-18 16:06:49.547093] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184] (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce] ))))) 0-: 138.4.2.248:49157: ping timer event already removed

** user CDC becomed root at this point

[2016-11-18 16:07:11.907892] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 7 times between [2016-11-18 16:06:17.086098] and [2016-11-18 16:07:11.906944] [2016-11-18 16:07:11.907884] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.908875] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.909179] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.909256] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop" repeated 2 times between [2016-11-18 16:06:17.047671] and [2016-11-18 16:07:11.908114] [2016-11-18 16:07:11.909254] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.911109] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.911018] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.911107] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop
[2016-11-18 16:07:11.911871] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.912045] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.913067] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 2 times between [2016-11-18 16:07:11.909254] and [2016-11-18 16:07:11.912165] [2016-11-18 16:07:11.913065] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.913193] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.913710] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.913757] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.913788] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt missing on subvol home-lab-3-client-0
[2016-11-18 16:07:11.913810] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.914336] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914379] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.914424] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914448] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.914501] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914617] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.914649] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914698] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.914724] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:07:11.914743] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:07:11.914764] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914791] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914825] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914855] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914880] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914947] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.915569] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.915644] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.916090] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.916134] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.916158] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt missing on subvol home-lab-3-client-0
[2016-11-18 16:07:11.916185] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.916593] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.916780] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.916878] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.916994] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.917076] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.916906] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917117] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.917153] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.917205] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:07:11.917368] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:07:11.917437] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917467] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917496] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917521] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917551] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917726] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917958] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled as normal fop
[2016-11-18 16:07:11.918230] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.918488] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
10456166432768 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918570] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
8600417075200 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918603] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
10500905500672 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918675] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
10499686531072 and avail_inodes is: 99.00
[2016-11-18 16:07:11.919095] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.919377] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.920040] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled as fast fop
[2016-11-18 16:07:11.920889] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled as slow fop
[2016-11-18 16:07:11.921051] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled as fast fop
[2016-11-18 16:07:11.921583] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.922068] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120276.
[2016-11-18 16:07:11.922936] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:07:11.922988] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 16:07:11.923004] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:07:11.923026] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:07:11.923779] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.924504] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.926692] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop" repeated 2 times between [2016-11-18 16:06:17.097104] and [2016-11-18 16:07:11.923946] [2016-11-18 16:07:11.926690] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.926858] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.927327] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.927376] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.927401] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:07:11.927422] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.927940] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.927987] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0 [2016-11-18 16:07:11.928043] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928055] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928058] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928170] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1 [2016-11-18 16:07:11.928174] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3 [2016-11-18 16:07:11.928277] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2 [2016-11-18 16:07:11.928360] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:07:11.928381] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:07:11.928403] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928431] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928456] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928480] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928511] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928753] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.930485] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.929862] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.930483] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.930602] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.931071] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.931124] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.931148] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:07:11.931348] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.931880] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.931927] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0 [2016-11-18 16:07:11.932002] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.931994] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.931991] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932078] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2 [2016-11-18 16:07:11.932116] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1 [2016-11-18 16:07:11.932315] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3 [2016-11-18 16:07:11.932363] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:07:11.932383] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:07:11.932407] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932446] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932471] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932494] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932518] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932725] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.932545] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932722] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled as normal fop
[2016-11-18 16:07:11.933008] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.934086] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.934323] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.934922] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.935497] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.936102] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.936903] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled as fast fop
[2016-11-18 16:07:11.937540] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled as slow fop
[2016-11-18 16:07:11.937682] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled as fast fop
[2016-11-18 16:07:11.938267] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.938757] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120652.
[2016-11-18 16:07:11.938994] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:07:11.939072] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/kk.txt
[2016-11-18 16:07:11.939096] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:07:11.939112] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:07:11.939448] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.940028] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.942130] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0

** TEST DONE as root --> NO PROBLEM

[2016-11-18 16:07:28.794150] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:07:31.794656] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:07:31.794925] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:07:31.794984] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-3: returning as transport is already disconnected OR there are no frames (0 || 0)

** USER ROOT EXITED at this point

Quoted 17 lines · Miklos Szeredi

Quoted 6 lines · Omar Walid Llorente

In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu SMP Mon

Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse

included
(not as a module) so if it would be necessary to change the fuse mount options some help would be needed.

Can you reproduce this on latest (e.g 4.8) kernel?

Also fuse debug logs (with whaterver kernel) could be useful; not sure how to enable it for glusterfs, usual convention for fuse filesystems is "-d".

Thanks,
Miklos

Quoted 19 lines · Soumya Koduri

Hi,

Sorry for the delay. I could reproduce the issue on my gluster setup as well. The error is generated at posix-acl xlator layer of glusterfs server.

As Bruce mentioned in the earlier mail, as part of WRITE call, NFS-server issues OPEN() with O_WRONLY flag set. Since the attributes of the file "kk.txt" when created was 0400, the corresponding acl maps to below ACL -

[root@dhcp35-197 vol1]# getfacl kk.txt
# file: kk.txt
# owner: guest
# group: mock
user::r--
group::---
other::---

Quoted 3 lines

And hence the permission check for WRITE access fails. Also not just glusterfs, I see similar behavior with ext4 filesystem as well when exported via kernel-NFS server.

That definitely shouldn't be happening, how are you reproducing?

--b.

Quoted 1,158 lines

So I am not sure if fixing glusterfs
for this use-case is valid.

Thanks,
Soumya

Quoted 1,150 lines · Omar Walid Llorente

Yes, of course, we can reproduce the problem on latest (e.g 4.8) kernel.

Tested on a server with fedora 24 (Linux cuentas11.lab.dit.upm.es 4.8.4-200.fc24.x86_64 #1 SMP Tue Oct 25 13:06:04 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux), glusterfs-fuse-3.8.5-1.fc24.x86_64, fuse-2.9.7-1.fc24.x86_64 and nfs-utils-1.3.4-1.rc2.fc24.x86_64 that mounts as a glusterfs-client and exports as a nfs-server a ditributed glusterfs volume (that comes from a 3.7.4 glusterfs server over Ubuntu 14.04.

[root@cuentas11 ~]# cat /proc/mounts | grep home recipiente8hd:/home-lab-3.tcp /home-3 fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0
[root@cuentas11 ~]#

As you may remember the schema is the one that follows (glusterfs volume
name: home-lab-3, glusterfs client mount point: /home-3, nfs client mount point: /home):

nfs client (v1.2.8) ---> NFSv3 ---> knfsd (v1.3.4 over a Linux v4.8.4 kernel) ---> FUSE (v2.9.7) --> glusterfs client (v3.8.5) --> GLUSTER over TCP --> glusterfs server (v3.7.4)

The errors at the nfs-client side are the same:

cdc@l056:~$ mount | grep "/home/cdc"
cuentas11:/home-3/cdc on /home/cdc type nfs
(rw,noatime,nfsvers=3,vers=3,tcp,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.86)

cdc@l056:~$
cdc@l056:~$
cdc@l056:~$

cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied

(quoted diff) +0 -2

-r--r--r-- 1 cdc admincdc 7 nov 18  2016 444.txt
-r--r--r-- 1 cdc admincdc 0 nov 18  2016 kk.txt

cdc@l056:~$
cdc@l056:~$
cdc@l056:~$

cdc@l056:~$ sudo su
[sudo] password for cdc:
root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt

(quoted diff) +0 -2

-r--r--r-- 1 root root 7 nov 18  2016 444.txt
-r--r--r-- 1 root root 7 nov 18  2016 kk.txt

root@l056:/home/cdc#
root@l056:/home/cdc#

root@l056:/home/cdc# exit
cdc@l056:~$

See at the end the logs with and without the debug option activated at the glusterfs client.

Thanks again, and sorry for the long message.

Omar

* Logs on the glusterfs client side with NO DEBUG:

[2016-11-18 15:45:43.681215] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote operation failed. Path: /cdc/kk.txt (cc98e831-3410-4202-8254-2d4b48e8ea68) [Permission denied] [2016-11-18 15:45:43.681297] W [fuse-bridge.c:989:fuse_fd_cbk] 0-glusterfs-fuse: 399: OPEN() /cdc/kk.txt => -1 (Permission denied) [2016-11-18 15:45:55.457922] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 15:45:55.457978] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 15:45:55.457997] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 15:45:55.458011] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 15:45:55.471933] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---

* Logs on the glusterfs client side with DEBUG activated:

[2016-11-18 16:06:17.040092] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:06:17.045986] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.046132] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.046833] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret 0 [Invalid argument] [2016-11-18 16:06:17.047671] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop
[2016-11-18 16:06:17.050539] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.050867] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.051948] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.051871] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.051946] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.062108] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret 0 [Invalid argument] [2016-11-18 16:06:17.063692] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.064015] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.065068] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 4 times between [2016-11-18 16:06:17.040092] and [2016-11-18 16:06:17.064175] [2016-11-18 16:06:17.065066] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.065232] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.065755] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.065897] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.066016] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt missing on subvol home-lab-3-client-0
[2016-11-18 16:06:17.066104] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.066594] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.066669] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.066771] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.066898] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.066906] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067019] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.066915] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067066] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.067107] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:06:17.067151] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:06:17.067179] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067206] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067250] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067275] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067308] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067387] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.068157] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.068281] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.068738] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.068781] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.068805] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt missing on subvol home-lab-3-client-0
[2016-11-18 16:06:17.068827] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.069286] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.069345] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.069452] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.069507] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.069556] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.069585] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.069939] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.069983] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.070005] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:06:17.070038] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:06:17.070098] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070128] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070154] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070178] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070208] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070251] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070537] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled as normal fop
[2016-11-18 16:06:17.070826] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.071100] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
10456189239296 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071142] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
10499693871104 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071183] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
10500911267840 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071343] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
8600432410624 and avail_inodes is: 99.00
[2016-11-18 16:06:17.073943] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.074273] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:06:17.075035] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled as fast fop
[2016-11-18 16:06:17.075913] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled as slow fop
[2016-11-18 16:06:17.076205] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled as fast fop
[2016-11-18 16:06:17.076636] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.077282] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120276.
[2016-11-18 16:06:17.077981] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:06:17.078032] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 16:06:17.078049] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:06:17.078064] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:06:17.079156] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.079794] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.082196] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.082772] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.082318] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.082770] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.083168] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.083282] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:06:17.083320] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.083991] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084043] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0 [2016-11-18 16:06:17.084048] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084092] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3 [2016-11-18 16:06:17.084114] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084167] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1 [2016-11-18 16:06:17.084091] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084377] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2 [2016-11-18 16:06:17.084406] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:06:17.084511] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:06:17.084534] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084683] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084783] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084857] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084889] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.085001] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.086098] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:06:17.087256] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.087173] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.087254] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.087749] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.087795] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.088014] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:06:17.088036] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.088598] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.088639] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0 [2016-11-18 16:06:17.088804] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.088855] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2 [2016-11-18 16:06:17.088819] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089069] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3 [2016-11-18 16:06:17.089380] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089421] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1 [2016-11-18 16:06:17.089443] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:06:17.089463] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:06:17.089575] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089601] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089712] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089738] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.090006] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.090156] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.090648] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled as normal fop
[2016-11-18 16:06:17.090732] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.091499] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.091843] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.093831] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.094711] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.093216] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.094709] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled as fast fop
[2016-11-18 16:06:17.095280] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote operation failed. Path: /cdc/kk.txt (09bfb6fa-82fb-463b-abd9-bb7ef88e10c0) [Permission denied] [2016-11-18 16:06:17.095341] D [MSGID: 0] [client-rpc-fops.c:448:client3_3_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095366] D [MSGID: 0] [dht-inode-read.c:45:dht_open_cbk] 0-home-lab-3-dht: subvolume home-lab-3-client-3 returned -1 [Permission denied]
[2016-11-18 16:06:17.095480] D [MSGID: 0] [dht-inode-read.c:59:dht_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-dht returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095671] D [MSGID: 0] [read-ahead.c:101:ra_open_cbk] 0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-read-ahead returned -1 error: Permission denied [Permission denied] [2016-11-18 16:06:17.095759] D [MSGID: 0] [io-cache.c:605:ioc_open_cbk] 0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: Permission denied [Permission denied] [2016-11-18 16:06:17.095963] D [MSGID: 0] [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095988] D [MSGID: 0] [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.096275] D [MSGID: 0] [io-stats.c:1936:io_stats_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.096362] W [fuse-bridge.c:989:fuse_fd_cbk] 0-glusterfs-fuse: 338: OPEN() /cdc/kk.txt => -1 (Permission denied) [2016-11-18 16:06:17.097104] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.097674] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.099912] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:24.785225] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:06:24.785338] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:06:24.785359] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:06:24.785377] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-3: returning as transport is already disconnected OR there are no frames (0 || 0)

** TEST DONE as user cdc --> ERROR

[2016-11-18 16:06:46.737686] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184] (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce] ))))) 0-: 138.4.2.246:49172: ping timer event already removed [2016-11-18 16:06:49.545698] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184] (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce] ))))) 0-: 138.4.2.177:49157: ping timer event already removed [2016-11-18 16:06:49.546474] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184] (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce] ))))) 0-: 138.4.2.189:49157: ping timer event already removed [2016-11-18 16:06:49.547093] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184] (--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce] ))))) 0-: 138.4.2.248:49157: ping timer event already removed

** user CDC becomed root at this point

[2016-11-18 16:07:11.907892] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 7 times between [2016-11-18 16:06:17.086098] and [2016-11-18 16:07:11.906944] [2016-11-18 16:07:11.907884] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.908875] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.909179] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.909256] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop" repeated 2 times between [2016-11-18 16:06:17.047671] and [2016-11-18 16:07:11.908114] [2016-11-18 16:07:11.909254] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.911109] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.911018] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.911107] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop
[2016-11-18 16:07:11.911871] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.912045] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.913067] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 2 times between [2016-11-18 16:07:11.909254] and [2016-11-18 16:07:11.912165] [2016-11-18 16:07:11.913065] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.913193] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.913710] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.913757] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.913788] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt missing on subvol home-lab-3-client-0
[2016-11-18 16:07:11.913810] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.914336] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914379] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.914424] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914448] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.914501] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914617] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.914649] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914698] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.914724] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:07:11.914743] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:07:11.914764] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914791] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914825] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914855] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914880] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914947] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.915569] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.915644] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.916090] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.916134] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.916158] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt missing on subvol home-lab-3-client-0
[2016-11-18 16:07:11.916185] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.916593] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.916780] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.916878] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.916994] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.917076] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.916906] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917117] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.917153] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.917205] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:07:11.917368] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:07:11.917437] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917467] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917496] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917521] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917551] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917726] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917958] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled as normal fop
[2016-11-18 16:07:11.918230] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.918488] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
10456166432768 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918570] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
8600417075200 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918603] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
10500905500672 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918675] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
10499686531072 and avail_inodes is: 99.00
[2016-11-18 16:07:11.919095] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.919377] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.920040] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled as fast fop
[2016-11-18 16:07:11.920889] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled as slow fop
[2016-11-18 16:07:11.921051] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled as fast fop
[2016-11-18 16:07:11.921583] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.922068] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120276.
[2016-11-18 16:07:11.922936] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:07:11.922988] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 16:07:11.923004] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:07:11.923026] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:07:11.923779] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.924504] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.926692] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop" repeated 2 times between [2016-11-18 16:06:17.097104] and [2016-11-18 16:07:11.923946] [2016-11-18 16:07:11.926690] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.926858] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.927327] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.927376] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.927401] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:07:11.927422] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.927940] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.927987] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0 [2016-11-18 16:07:11.928043] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928055] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928058] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928170] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1 [2016-11-18 16:07:11.928174] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3 [2016-11-18 16:07:11.928277] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2 [2016-11-18 16:07:11.928360] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:07:11.928381] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:07:11.928403] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928431] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928456] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928480] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928511] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928753] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.930485] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.929862] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.930483] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.930602] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.931071] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.931124] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.931148] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:07:11.931348] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.931880] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.931927] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0 [2016-11-18 16:07:11.932002] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.931994] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.931991] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932078] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2 [2016-11-18 16:07:11.932116] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1 [2016-11-18 16:07:11.932315] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3 [2016-11-18 16:07:11.932363] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:07:11.932383] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:07:11.932407] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932446] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932471] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932494] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932518] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932725] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.932545] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932722] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled as normal fop
[2016-11-18 16:07:11.933008] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.934086] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.934323] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.934922] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.935497] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.936102] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.936903] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled as fast fop
[2016-11-18 16:07:11.937540] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled as slow fop
[2016-11-18 16:07:11.937682] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled as fast fop
[2016-11-18 16:07:11.938267] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.938757] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120652.
[2016-11-18 16:07:11.938994] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:07:11.939072] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/kk.txt
[2016-11-18 16:07:11.939096] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:07:11.939112] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:07:11.939448] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.940028] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.942130] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0

** TEST DONE as root --> NO PROBLEM

[2016-11-18 16:07:28.794150] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:07:31.794656] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:07:31.794925] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:07:31.794984] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-3: returning as transport is already disconnected OR there are no frames (0 || 0)

** USER ROOT EXITED at this point

Quoted 17 lines · Miklos Szeredi

Quoted 6 lines · Omar Walid Llorente

In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu SMP Mon

Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse

included
(not as a module) so if it would be necessary to change the fuse mount options some help would be needed.

Can you reproduce this on latest (e.g 4.8) kernel?

Also fuse debug logs (with whaterver kernel) could be useful; not sure how to enable it for glusterfs, usual convention for fuse filesystems is "-d".

Thanks,
Miklos

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 20 lines · J. Bruce Fields

Quoted 19 lines · Soumya Koduri

Hi,

Sorry for the delay. I could reproduce the issue on my gluster setup as well. The error is generated at posix-acl xlator layer of glusterfs server.

As Bruce mentioned in the earlier mail, as part of WRITE call, NFS-server issues OPEN() with O_WRONLY flag set. Since the attributes of the file "kk.txt" when created was 0400, the corresponding acl maps to below ACL -

[root@dhcp35-197 vol1]# getfacl kk.txt
# file: kk.txt
# owner: guest
# group: mock
user::r--
group::---
other::---

Quoted 5 lines

Quoted 3 lines

And hence the permission check for WRITE access fails. Also not just glusterfs, I see similar behavior with ext4 filesystem as well when exported via kernel-NFS server.

That definitely shouldn't be happening, how are you reproducing?

Sorry. Yesterday when I was testing, both the exports (one on glusterfs fuse mount and another on ext4) had same fsid. So when I was trying to mount and access the share on ext4, it had redirected to glusterfs fuse mount and hence the test failed.

I re-checked with ext4 and I do not see any failures. So probably then we need to put a fix in glusterfs to bypass this check for file owner during OPEN.

@Omar,
Could you please file a bug against glusterfs component in RH bugzilla [1] for the same?

Thanks,
Soumya

[1] https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS

Quoted 1,176 lines

--b.

Quoted 1,168 lines

So I am not sure if fixing glusterfs
for this use-case is valid.

Thanks,
Soumya

Quoted 1,160 lines · Omar Walid Llorente

Yes, of course, we can reproduce the problem on latest (e.g 4.8) kernel.

Tested on a server with fedora 24 (Linux cuentas11.lab.dit.upm.es 4.8.4-200.fc24.x86_64 #1 SMP Tue Oct 25 13:06:04 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux), glusterfs-fuse-3.8.5-1.fc24.x86_64, fuse-2.9.7-1.fc24.x86_64 and nfs-utils-1.3.4-1.rc2.fc24.x86_64 that mounts as a glusterfs-client and exports as a nfs-server a ditributed glusterfs volume (that comes from a 3.7.4 glusterfs server over Ubuntu 14.04.

[root@cuentas11 ~]# cat /proc/mounts | grep home recipiente8hd:/home-lab-3.tcp /home-3 fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072

0 0
[root@cuentas11 ~]#

As you may remember the schema is the one that follows (glusterfs volume
name: home-lab-3, glusterfs client mount point: /home-3, nfs client mount point: /home):

nfs client (v1.2.8) ---> NFSv3 ---> knfsd (v1.3.4 over a Linux v4.8.4 kernel) ---> FUSE (v2.9.7) --> glusterfs client (v3.8.5) --> GLUSTER over TCP --> glusterfs server (v3.7.4)

The errors at the nfs-client side are the same:

cdc@l056:~$ mount | grep "/home/cdc"
cuentas11:/home-3/cdc on /home/cdc type nfs
(rw,noatime,nfsvers=3,vers=3,tcp,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.86) 

cdc@l056:~$
cdc@l056:~$
cdc@l056:~$

cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied

(quoted diff) +0 -2

-r--r--r-- 1 cdc admincdc 7 nov 18  2016 444.txt
-r--r--r-- 1 cdc admincdc 0 nov 18  2016 kk.txt

cdc@l056:~$
cdc@l056:~$
cdc@l056:~$

cdc@l056:~$ sudo su
[sudo] password for cdc:
root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt

(quoted diff) +0 -2

-r--r--r-- 1 root root 7 nov 18  2016 444.txt
-r--r--r-- 1 root root 7 nov 18  2016 kk.txt

root@l056:/home/cdc#
root@l056:/home/cdc#

root@l056:/home/cdc# exit
cdc@l056:~$

See at the end the logs with and without the debug option activated at the glusterfs client.

Thanks again, and sorry for the long message.

Omar

* Logs on the glusterfs client side with NO DEBUG:

[2016-11-18 15:45:43.681215] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote operation failed. Path: /cdc/kk.txt (cc98e831-3410-4202-8254-2d4b48e8ea68) [Permission denied] [2016-11-18 15:45:43.681297] W [fuse-bridge.c:989:fuse_fd_cbk] 0-glusterfs-fuse: 399: OPEN() /cdc/kk.txt => -1 (Permission denied) [2016-11-18 15:45:55.457922] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 15:45:55.457978] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 15:45:55.457997] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 15:45:55.458011] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 15:45:55.471933] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---

* Logs on the glusterfs client side with DEBUG activated:

[2016-11-18 16:06:17.040092] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:06:17.045986] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.046132] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.046833] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret 0 [Invalid argument] [2016-11-18 16:06:17.047671] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop
[2016-11-18 16:06:17.050539] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.050867] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.051948] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.051871] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.051946] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.062108] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret 0 [Invalid argument] [2016-11-18 16:06:17.063692] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.064015] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.065068] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 4 times between [2016-11-18 16:06:17.040092] and [2016-11-18 16:06:17.064175] [2016-11-18 16:06:17.065066] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.065232] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.065755] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.065897] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.066016] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt missing on subvol home-lab-3-client-0
[2016-11-18 16:06:17.066104] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.066594] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.066669] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.066771] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.066898] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.066906] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067019] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.066915] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067066] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.067107] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:06:17.067151] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:06:17.067179] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067206] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067250] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067275] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067308] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067387] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.068157] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.068281] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.068738] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.068781] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.068805] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt missing on subvol home-lab-3-client-0
[2016-11-18 16:06:17.068827] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.069286] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.069345] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.069452] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.069507] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.069556] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.069585] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.069939] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.069983] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.070005] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:06:17.070038] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:06:17.070098] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070128] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070154] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070178] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070208] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070251] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070537] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled as normal fop
[2016-11-18 16:06:17.070826] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.071100] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
10456189239296 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071142] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
10499693871104 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071183] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
10500911267840 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071343] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
8600432410624 and avail_inodes is: 99.00
[2016-11-18 16:06:17.073943] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.074273] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:06:17.075035] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled as fast fop
[2016-11-18 16:06:17.075913] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled as slow fop
[2016-11-18 16:06:17.076205] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled as fast fop
[2016-11-18 16:06:17.076636] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.077282] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120276.
[2016-11-18 16:06:17.077981] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:06:17.078032] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 16:06:17.078049] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:06:17.078064] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:06:17.079156] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.079794] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.082196] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.082772] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.082318] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.082770] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.083168] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.083282] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:06:17.083320] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.083991] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084043] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0 [2016-11-18 16:06:17.084048] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084092] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3 [2016-11-18 16:06:17.084114] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084167] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1 [2016-11-18 16:06:17.084091] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084377] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2 [2016-11-18 16:06:17.084406] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:06:17.084511] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:06:17.084534] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084683] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084783] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084857] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084889] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.085001] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.086098] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:06:17.087256] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.087173] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.087254] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.087749] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.087795] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.088014] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:06:17.088036] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.088598] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.088639] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0 [2016-11-18 16:06:17.088804] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.088855] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2 [2016-11-18 16:06:17.088819] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089069] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3 [2016-11-18 16:06:17.089380] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089421] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1 [2016-11-18 16:06:17.089443] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:06:17.089463] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:06:17.089575] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089601] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089712] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089738] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.090006] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.090156] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.090648] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled as normal fop
[2016-11-18 16:06:17.090732] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.091499] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.091843] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.093831] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.094711] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.093216] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.094709] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled as fast fop
[2016-11-18 16:06:17.095280] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote operation failed. Path: /cdc/kk.txt (09bfb6fa-82fb-463b-abd9-bb7ef88e10c0) [Permission denied] [2016-11-18 16:06:17.095341] D [MSGID: 0] [client-rpc-fops.c:448:client3_3_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095366] D [MSGID: 0] [dht-inode-read.c:45:dht_open_cbk] 0-home-lab-3-dht: subvolume home-lab-3-client-3 returned -1 [Permission denied]
[2016-11-18 16:06:17.095480] D [MSGID: 0] [dht-inode-read.c:59:dht_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-dht returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095671] D [MSGID: 0] [read-ahead.c:101:ra_open_cbk] 0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-read-ahead returned -1 error: Permission denied [Permission denied] [2016-11-18 16:06:17.095759] D [MSGID: 0] [io-cache.c:605:ioc_open_cbk] 0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: Permission denied [Permission denied] [2016-11-18 16:06:17.095963] D [MSGID: 0] [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095988] D [MSGID: 0] [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.096275] D [MSGID: 0] [io-stats.c:1936:io_stats_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.096362] W [fuse-bridge.c:989:fuse_fd_cbk] 0-glusterfs-fuse: 338: OPEN() /cdc/kk.txt => -1 (Permission denied) [2016-11-18 16:06:17.097104] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.097674] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.099912] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:24.785225] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:06:24.785338] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:06:24.785359] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:06:24.785377] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-3: returning as transport is already disconnected OR there are no frames (0 || 0)

** TEST DONE as user cdc --> ERROR

[2016-11-18 16:06:46.737686] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]

(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]

))))) 0-: 138.4.2.246:49172: ping timer event already removed [2016-11-18 16:06:49.545698] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]

(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]

))))) 0-: 138.4.2.177:49157: ping timer event already removed [2016-11-18 16:06:49.546474] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]

(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]

))))) 0-: 138.4.2.189:49157: ping timer event already removed [2016-11-18 16:06:49.547093] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]

(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]

))))) 0-: 138.4.2.248:49157: ping timer event already removed

** user CDC becomed root at this point

[2016-11-18 16:07:11.907892] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 7 times between [2016-11-18 16:06:17.086098] and [2016-11-18 16:07:11.906944] [2016-11-18 16:07:11.907884] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.908875] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.909179] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.909256] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop" repeated 2 times between [2016-11-18 16:06:17.047671] and [2016-11-18 16:07:11.908114] [2016-11-18 16:07:11.909254] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.911109] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.911018] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.911107] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop
[2016-11-18 16:07:11.911871] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.912045] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.913067] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 2 times between [2016-11-18 16:07:11.909254] and [2016-11-18 16:07:11.912165] [2016-11-18 16:07:11.913065] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.913193] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.913710] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.913757] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.913788] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt missing on subvol home-lab-3-client-0
[2016-11-18 16:07:11.913810] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.914336] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914379] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.914424] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914448] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.914501] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914617] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.914649] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914698] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.914724] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:07:11.914743] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:07:11.914764] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914791] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914825] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914855] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914880] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914947] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.915569] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.915644] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.916090] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.916134] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.916158] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt missing on subvol home-lab-3-client-0
[2016-11-18 16:07:11.916185] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.916593] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.916780] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.916878] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.916994] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.917076] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.916906] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917117] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.917153] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.917205] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:07:11.917368] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:07:11.917437] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917467] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917496] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917521] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917551] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917726] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917958] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled as normal fop
[2016-11-18 16:07:11.918230] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.918488] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
10456166432768 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918570] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
8600417075200 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918603] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
10500905500672 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918675] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
10499686531072 and avail_inodes is: 99.00
[2016-11-18 16:07:11.919095] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.919377] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.920040] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled as fast fop
[2016-11-18 16:07:11.920889] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled as slow fop
[2016-11-18 16:07:11.921051] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled as fast fop
[2016-11-18 16:07:11.921583] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.922068] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120276.
[2016-11-18 16:07:11.922936] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:07:11.922988] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 16:07:11.923004] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:07:11.923026] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:07:11.923779] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.924504] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.926692] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop" repeated 2 times between [2016-11-18 16:06:17.097104] and [2016-11-18 16:07:11.923946] [2016-11-18 16:07:11.926690] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.926858] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.927327] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.927376] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.927401] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:07:11.927422] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.927940] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.927987] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0 [2016-11-18 16:07:11.928043] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928055] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928058] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928170] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1 [2016-11-18 16:07:11.928174] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3 [2016-11-18 16:07:11.928277] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2 [2016-11-18 16:07:11.928360] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:07:11.928381] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:07:11.928403] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928431] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928456] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928480] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928511] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928753] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.930485] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.929862] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.930483] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.930602] D [MSGID: 0] [dht-common.c:2684:dht_lookup] 0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.931071] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.931124] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.931148] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:07:11.931348] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.931880] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.931927] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0 [2016-11-18 16:07:11.932002] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.931994] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.931991] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932078] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2 [2016-11-18 16:07:11.932116] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1 [2016-11-18 16:07:11.932315] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3 [2016-11-18 16:07:11.932363] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:07:11.932383] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:07:11.932407] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932446] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932471] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932494] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932518] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932725] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.932545] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932722] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: MKNOD scheduled as normal fop
[2016-11-18 16:07:11.933008] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.934086] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.934323] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.934922] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop
[2016-11-18 16:07:11.935497] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.936102] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.936903] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: OPEN scheduled as fast fop
[2016-11-18 16:07:11.937540] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: WRITE scheduled as slow fop
[2016-11-18 16:07:11.937682] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: FSTAT scheduled as fast fop
[2016-11-18 16:07:11.938267] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.938757] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120652.
[2016-11-18 16:07:11.938994] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:07:11.939072] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/kk.txt
[2016-11-18 16:07:11.939096] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:07:11.939112] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:07:11.939448] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.940028] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.942130] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0

** TEST DONE as root --> NO PROBLEM

[2016-11-18 16:07:28.794150] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:07:31.794656] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:07:31.794925] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
[2016-11-18 16:07:31.794984] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-3: returning as transport is already disconnected OR there are no frames (0 || 0)

** USER ROOT EXITED at this point

Quoted 17 lines · Miklos Szeredi

Quoted 6 lines · Omar Walid Llorente

In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu SMP Mon

Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse

included
(not as a module) so if it would be necessary to change the fuse mount options some help would be needed.

Can you reproduce this on latest (e.g 4.8) kernel?

Also fuse debug logs (with whaterver kernel) could be useful; not sure how to enable it for glusterfs, usual convention for fuse filesystems is "-d".

Thanks,
Miklos

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Soumya, all.

Sorry for the delay answering.

Following your sugestion, we are planning to file a bug at bugzilla [1], but in order to minimize the content of the explanation, we wanted to include a link to the file archive of this nfs-kernel-server list, but we only find a page that doesn't work (from http://dir.gmane.org/gmane.linux.nfs we go to
http://news.gmane.org/gmane.linux.nfs, but it gives a "Page Not Found" answer).

Does anybody know if is this link available anywhere else or if it can be repaired?

Thanks a lot to everyone. We hope we can file the bug this week.

Omar

Quoted 1,417 lines · Soumya Koduri

Quoted 20 lines · J. Bruce Fields

Quoted 19 lines · Soumya Koduri

Hi,

Sorry for the delay. I could reproduce the issue on my gluster setup as well. The error is generated at posix-acl xlator layer of glusterfs server.

As Bruce mentioned in the earlier mail, as part of WRITE call, NFS-server issues OPEN() with O_WRONLY flag set. Since the attributes of the file "kk.txt" when created was 0400, the corresponding acl maps to below ACL -

[root@dhcp35-197 vol1]# getfacl kk.txt
# file: kk.txt
# owner: guest
# group: mock
user::r--
group::---
other::---

Quoted 5 lines

Quoted 3 lines

And hence the permission check for WRITE access fails. Also not just glusterfs, I see similar behavior with ext4 filesystem as well when exported via kernel-NFS server.

That definitely shouldn't be happening, how are you reproducing?

Sorry. Yesterday when I was testing, both the exports (one on glusterfs fuse mount and another on ext4) had same fsid. So when I was trying to mount and access the share on ext4, it had redirected to glusterfs fuse mount and hence the test failed.

I re-checked with ext4 and I do not see any failures. So probably then we need to put a fix in glusterfs to bypass this check for file owner during OPEN.

@Omar,
Could you please file a bug against glusterfs component in RH bugzilla [1] for the same?

Thanks,
Soumya

[1] https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS

Quoted 1,370 lines

--b.

Quoted 1,362 lines

So I am not sure if fixing glusterfs
for this use-case is valid.

Thanks,
Soumya

Quoted 1,354 lines · Omar Walid Llorente

Yes, of course, we can reproduce the problem on latest (e.g 4.8) kernel.

Tested on a server with fedora 24 (Linux cuentas11.lab.dit.upm.es 4.8.4-200.fc24.x86_64 #1 SMP Tue Oct 25 13:06:04 UTC 2016 x86_64 x86_64
x86_64 GNU/Linux), glusterfs-fuse-3.8.5-1.fc24.x86_64, fuse-2.9.7-1.fc24.x86_64 and nfs-utils-1.3.4-1.rc2.fc24.x86_64 that mounts as a glusterfs-client and exports as a nfs-server a ditributed glusterfs volume (that comes from a 3.7.4 glusterfs server over Ubuntu 14.04.

[root@cuentas11 ~]# cat /proc/mounts | grep home recipiente8hd:/home-lab-3.tcp /home-3 fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072

0 0
[root@cuentas11 ~]#

As you may remember the schema is the one that follows (glusterfs volume
name: home-lab-3, glusterfs client mount point: /home-3, nfs client mount point: /home):

nfs client (v1.2.8) ---> NFSv3 ---> knfsd (v1.3.4 over a Linux v4.8.4 kernel) ---> FUSE (v2.9.7) --> glusterfs client (v3.8.5) --> GLUSTER over TCP --> glusterfs server (v3.7.4)

The errors at the nfs-client side are the same:

cdc@l056:~$ mount | grep "/home/cdc"
cuentas11:/home-3/cdc on /home/cdc type nfs
(rw,noatime,nfsvers=3,vers=3,tcp,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.86)

cdc@l056:~$
cdc@l056:~$
cdc@l056:~$

cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied

(quoted diff) +0 -2

-r--r--r-- 1 cdc admincdc 7 nov 18  2016 444.txt
-r--r--r-- 1 cdc admincdc 0 nov 18  2016 kk.txt

cdc@l056:~$
cdc@l056:~$
cdc@l056:~$

cdc@l056:~$ sudo su
[sudo] password for cdc:
root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt

(quoted diff) +0 -2

-r--r--r-- 1 root root 7 nov 18  2016 444.txt
-r--r--r-- 1 root root 7 nov 18  2016 kk.txt

root@l056:/home/cdc#
root@l056:/home/cdc#

root@l056:/home/cdc# exit
cdc@l056:~$

See at the end the logs with and without the debug option activated at the glusterfs client.

Thanks again, and sorry for the long message.

Omar

* Logs on the glusterfs client side with NO DEBUG:

[2016-11-18 15:45:43.681215] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote
operation failed. Path: /cdc/kk.txt (cc98e831-3410-4202-8254-2d4b48e8ea68) [Permission denied] [2016-11-18 15:45:43.681297] W [fuse-bridge.c:989:fuse_fd_cbk] 0-glusterfs-fuse: 399: OPEN() /cdc/kk.txt => -1 (Permission denied) [2016-11-18 15:45:55.457922] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 15:45:55.457978] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 15:45:55.457997] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 15:45:55.458011] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 15:45:55.471933] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---

* Logs on the glusterfs client side with DEBUG activated:

[2016-11-18 16:06:17.040092] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
STAT scheduled
as fast fop
[2016-11-18 16:06:17.045986] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.046132] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.046833] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret 0 [Invalid argument] [2016-11-18 16:06:17.047671] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop
[2016-11-18 16:06:17.050539] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.050867] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.051948] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.051871] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.051946] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.062108] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret 0 [Invalid argument] [2016-11-18 16:06:17.063692] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.064015] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.065068] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 4 times between [2016-11-18 16:06:17.040092] and [2016-11-18 16:06:17.064175] [2016-11-18 16:06:17.065066] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.065232] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.065755] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.065897] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.066016] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
missing on subvol home-lab-3-client-0
[2016-11-18 16:06:17.066104] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.066594] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.066669] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.066771] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.066898] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.066906] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.067019] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.066915] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.067066] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.067107] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:06:17.067151] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:06:17.067179] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1
error: No such
file or directory [No such file or directory]
[2016-11-18 16:06:17.067206] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067250] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.067275] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067308] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.067387] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:06:17.068157] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.068281] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.068738] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.068781] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.068805] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
missing on subvol home-lab-3-client-0
[2016-11-18 16:06:17.068827] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.069286] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.069345] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.069452] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.069507] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.069556] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.069585] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.069939] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.069983] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.070005] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:06:17.070038] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:06:17.070098] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1
error: No such
file or directory [No such file or directory]
[2016-11-18 16:06:17.070128] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070154] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.070178] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070208] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.070251] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:06:17.070537] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
MKNOD scheduled
as normal fop
[2016-11-18 16:06:17.070826] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.071100] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
10456189239296 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071142] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
10499693871104 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071183] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
10500911267840 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071343] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
8600432410624 and avail_inodes is: 99.00
[2016-11-18 16:06:17.073943] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.074273] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
STAT scheduled
as fast fop
[2016-11-18 16:06:17.075035] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
OPEN scheduled
as fast fop
[2016-11-18 16:06:17.075913] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
WRITE scheduled
as slow fop
[2016-11-18 16:06:17.076205] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
FSTAT scheduled
as fast fop
[2016-11-18 16:06:17.076636] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.077282] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120276.
[2016-11-18 16:06:17.077981] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:06:17.078032] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 16:06:17.078049] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:06:17.078064] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:06:17.079156] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.079794] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.082196] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.082772] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.082318] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.082770] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.083168] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.083282] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:06:17.083320] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.083991] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.084043] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.084048] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.084092] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.084114] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.084167] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.084091] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.084377] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.084406] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:06:17.084511] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:06:17.084534] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
error: No such
file or directory [No such file or directory]
[2016-11-18 16:06:17.084683] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084783] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.084857] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084889] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.085001] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:06:17.086098] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
STAT scheduled
as fast fop
[2016-11-18 16:06:17.087256] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.087173] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.087254] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.087749] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.087795] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.088014] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:06:17.088036] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.088598] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.088639] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.088804] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.088855] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.088819] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.089069] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.089380] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.089421] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.089443] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:06:17.089463] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:06:17.089575] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
error: No such
file or directory [No such file or directory]
[2016-11-18 16:06:17.089601] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089712] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.089738] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.090006] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.090156] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:06:17.090648] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
MKNOD scheduled
as normal fop
[2016-11-18 16:06:17.090732] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.091499] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.091843] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.093831] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.094711] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.093216] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.094709] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
OPEN scheduled
as fast fop
[2016-11-18 16:06:17.095280] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote
operation failed. Path: /cdc/kk.txt (09bfb6fa-82fb-463b-abd9-bb7ef88e10c0) [Permission denied] [2016-11-18 16:06:17.095341] D [MSGID: 0] [client-rpc-fops.c:448:client3_3_open_cbk] 0-stack-trace:
stack-address:
0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error:
Permission denied
[Permission denied]
[2016-11-18 16:06:17.095366] D [MSGID: 0] [dht-inode-read.c:45:dht_open_cbk] 0-home-lab-3-dht: subvolume home-lab-3-client-3 returned -1 [Permission denied] [2016-11-18 16:06:17.095480] D [MSGID: 0] [dht-inode-read.c:59:dht_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-dht returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095671] D [MSGID: 0] [read-ahead.c:101:ra_open_cbk]
0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-read-ahead returned -1 error: Permission denied [Permission denied] [2016-11-18 16:06:17.095759] D [MSGID: 0] [io-cache.c:605:ioc_open_cbk]
0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: Permission denied [Permission denied] [2016-11-18 16:06:17.095963] D [MSGID: 0] [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095988] D [MSGID: 0] [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.096275] D [MSGID: 0] [io-stats.c:1936:io_stats_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.096362] W [fuse-bridge.c:989:fuse_fd_cbk] 0-glusterfs-fuse: 338: OPEN() /cdc/kk.txt => -1 (Permission denied) [2016-11-18 16:06:17.097104] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.097674] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.099912] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:24.785225] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-0: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:06:24.785338] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-1: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:06:24.785359] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-2: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:06:24.785377] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-3: returning as transport is already disconnected OR
there are no frames (0 || 0)

** TEST DONE as user cdc --> ERROR

[2016-11-18 16:06:46.737686] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]

(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]

))))) 0-: 138.4.2.246:49172: ping timer event already removed [2016-11-18 16:06:49.545698] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]

(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]

))))) 0-: 138.4.2.177:49157: ping timer event already removed [2016-11-18 16:06:49.546474] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]

(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]

))))) 0-: 138.4.2.189:49157: ping timer event already removed [2016-11-18 16:06:49.547093] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]

(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]

))))) 0-: 138.4.2.248:49157: ping timer event already removed

** user CDC becomed root at this point

[2016-11-18 16:07:11.907892] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 7 times between [2016-11-18 16:06:17.086098] and [2016-11-18 16:07:11.906944] [2016-11-18 16:07:11.907884] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.908875] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.909179] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.909256] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop" repeated 2 times between [2016-11-18 16:06:17.047671] and [2016-11-18 16:07:11.908114]
[2016-11-18 16:07:11.909254] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
STAT scheduled
as fast fop
[2016-11-18 16:07:11.911109] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.911018] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.911107] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop
[2016-11-18 16:07:11.911871] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.912045] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.913067] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 2 times between [2016-11-18 16:07:11.909254] and [2016-11-18 16:07:11.912165] [2016-11-18 16:07:11.913065] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.913193] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.913710] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.913757] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.913788] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
missing on subvol home-lab-3-client-0
[2016-11-18 16:07:11.913810] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.914336] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.914379] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.914424] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.914448] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.914501] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.914617] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.914649] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.914698] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.914724] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:07:11.914743] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:07:11.914764] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
error: No such
file or directory [No such file or directory]
[2016-11-18 16:07:11.914791] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914825] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.914855] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914880] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.914947] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:07:11.915569] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.915644] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.916090] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.916134] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.916158] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
missing on subvol home-lab-3-client-0
[2016-11-18 16:07:11.916185] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.916593] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.916780] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.916878] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.916994] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.917076] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.916906] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.917117] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.917153] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.917205] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:07:11.917368] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:07:11.917437] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
error: No such
file or directory [No such file or directory]
[2016-11-18 16:07:11.917467] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917496] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.917521] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917551] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.917726] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:07:11.917958] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
MKNOD scheduled
as normal fop
[2016-11-18 16:07:11.918230] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.918488] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
10456166432768 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918570] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
8600417075200 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918603] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
10500905500672 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918675] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
10499686531072 and avail_inodes is: 99.00
[2016-11-18 16:07:11.919095] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.919377] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
STAT scheduled
as fast fop
[2016-11-18 16:07:11.920040] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
OPEN scheduled
as fast fop
[2016-11-18 16:07:11.920889] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
WRITE scheduled
as slow fop
[2016-11-18 16:07:11.921051] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
FSTAT scheduled
as fast fop
[2016-11-18 16:07:11.921583] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.922068] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120276.
[2016-11-18 16:07:11.922936] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:07:11.922988] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 16:07:11.923004] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:07:11.923026] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:07:11.923779] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
STAT scheduled
as fast fop
[2016-11-18 16:07:11.924504] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.926692] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop" repeated 2 times between [2016-11-18 16:06:17.097104] and [2016-11-18 16:07:11.923946]
[2016-11-18 16:07:11.926690] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.926858] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.927327] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.927376] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.927401] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:07:11.927422] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.927940] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.927987] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.928043] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.928055] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.928058] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.928170] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.928174] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.928277] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.928360] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:07:11.928381] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:07:11.928403] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
error: No such
file or directory [No such file or directory]
[2016-11-18 16:07:11.928431] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928456] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.928480] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928511] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.928753] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:07:11.930485] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.929862] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
STAT scheduled
as fast fop
[2016-11-18 16:07:11.930483] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.930602] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.931071] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.931124] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.931148] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:07:11.931348] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.931880] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.931927] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.932002] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.931994] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.931991] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.932078] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.932116] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.932315] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.932363] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:07:11.932383] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:07:11.932407] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
error: No such
file or directory [No such file or directory]
[2016-11-18 16:07:11.932446] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932471] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.932494] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932518] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.932725] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.932545] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:07:11.932722] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
MKNOD scheduled
as normal fop
[2016-11-18 16:07:11.933008] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.934086] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.934323] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.934922] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
STAT scheduled
as fast fop
[2016-11-18 16:07:11.935497] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.936102] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.936903] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
OPEN scheduled
as fast fop
[2016-11-18 16:07:11.937540] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
WRITE scheduled
as slow fop
[2016-11-18 16:07:11.937682] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
FSTAT scheduled
as fast fop
[2016-11-18 16:07:11.938267] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.938757] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120652.
[2016-11-18 16:07:11.938994] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:07:11.939072] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/kk.txt
[2016-11-18 16:07:11.939096] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:07:11.939112] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:07:11.939448] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.940028] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.942130] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0

** TEST DONE as root --> NO PROBLEM

[2016-11-18 16:07:28.794150] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-2: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:07:31.794656] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-0: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:07:31.794925] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-1: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:07:31.794984] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-3: returning as transport is already disconnected OR
there are no frames (0 || 0)

** USER ROOT EXITED at this point

Quoted 20 lines · Miklos Szeredi

Quoted 7 lines · Omar Walid Llorente

In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu SMP Mon

Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse

included
(not as a module) so if it would be necessary to change the fuse mount
options some help would be needed.

Can you reproduce this on latest (e.g 4.8) kernel?

Also fuse debug logs (with whaterver kernel) could be useful; not sure
how to enable it for glusterfs, usual convention for fuse filesystems is "-d".

Thanks,
Miklos

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 8 lines

----------------------------------------------------------------
Centro de Cálculo         Depto. Ingeniería Sistemas Telemáticos
E-mail: [email protected]        Universidad Politécnica de Madrid
Fax:(+34) 913367333                 E.T.S. Ing. Telecomunicación
Tel:(+34) 915495700-Ext.3005                28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------

Quoted 12 lines · Omar Walid Llorente

Hi Soumya, all.

Sorry for the delay answering.

Following your sugestion, we are planning to file a bug at bugzilla [1], but in order to minimize the content of the explanation, we wanted to include a link to the file archive of this nfs-kernel-server list, but we only find a page that doesn't work (from http://dir.gmane.org/gmane.linux.nfs we go to
http://news.gmane.org/gmane.linux.nfs, but it gives a "Page Not Found" answer).

Quoted 4 lines

Does anybody know if is this link available anywhere else or if it can be repaired?

Thanks a lot to everyone. We hope we can file the bug this week.

http://marc.info/?t=144801803400001&r=1&w=2

Or you can link by message id like:

http://marc.info/[email protected]

which I sometimes try to for posterity's sake since the message id should still be useful even if the one archive dies.

--b.

Quoted 1,424 lines

Omar

Quoted 1,412 lines · Soumya Koduri

Quoted 20 lines · J. Bruce Fields

Quoted 19 lines · Soumya Koduri

Hi,

Sorry for the delay. I could reproduce the issue on my gluster setup as well. The error is generated at posix-acl xlator layer of glusterfs server.

As Bruce mentioned in the earlier mail, as part of WRITE call, NFS-server issues OPEN() with O_WRONLY flag set. Since the attributes of the file "kk.txt" when created was 0400, the corresponding acl maps to below ACL -

[root@dhcp35-197 vol1]# getfacl kk.txt
# file: kk.txt
# owner: guest
# group: mock
user::r--
group::---
other::---

Quoted 4 lines

Quoted 3 lines

And hence the permission check for WRITE access fails. Also not just glusterfs, I see similar behavior with ext4 filesystem as well when exported via kernel-NFS server.

That definitely shouldn't be happening, how are you reproducing?

Sorry. Yesterday when I was testing, both the exports (one on glusterfs fuse mount and another on ext4) had same fsid. So when I was trying to mount and access the share on ext4, it had redirected to glusterfs fuse mount and hence the test failed.

I re-checked with ext4 and I do not see any failures. So probably then we need to put a fix in glusterfs to bypass this check for file owner during OPEN.

@Omar,
Could you please file a bug against glusterfs component in RH bugzilla [1] for the same?

Thanks,
Soumya

[1] https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS

Quoted 1,368 lines

--b.

Quoted 1,361 lines

So I am not sure if fixing glusterfs
for this use-case is valid.

Thanks,
Soumya

Quoted 1,353 lines · Omar Walid Llorente

Yes, of course, we can reproduce the problem on latest (e.g 4.8) kernel.

Tested on a server with fedora 24 (Linux cuentas11.lab.dit.upm.es 4.8.4-200.fc24.x86_64 #1 SMP Tue Oct 25 13:06:04 UTC 2016 x86_64 x86_64
x86_64 GNU/Linux), glusterfs-fuse-3.8.5-1.fc24.x86_64, fuse-2.9.7-1.fc24.x86_64 and nfs-utils-1.3.4-1.rc2.fc24.x86_64 that mounts as a glusterfs-client and exports as a nfs-server a ditributed glusterfs volume (that comes from a 3.7.4 glusterfs server over Ubuntu 14.04.

[root@cuentas11 ~]# cat /proc/mounts | grep home recipiente8hd:/home-lab-3.tcp /home-3 fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072

0 0
[root@cuentas11 ~]#

As you may remember the schema is the one that follows (glusterfs volume
name: home-lab-3, glusterfs client mount point: /home-3, nfs client mount point: /home):

nfs client (v1.2.8) ---> NFSv3 ---> knfsd (v1.3.4 over a Linux v4.8.4 kernel) ---> FUSE (v2.9.7) --> glusterfs client (v3.8.5) --> GLUSTER over TCP --> glusterfs server (v3.7.4)

The errors at the nfs-client side are the same:

cdc@l056:~$ mount | grep "/home/cdc"
cuentas11:/home-3/cdc on /home/cdc type nfs
(rw,noatime,nfsvers=3,vers=3,tcp,intr,fsc,nolock,rsize=262140,wsize=262140,addr=138.4.30.86)

cdc@l056:~$
cdc@l056:~$
cdc@l056:~$

cdc@l056:~$ rm -f kk.txt 444.txt; echo "prueba" > 444.txt; chmod 444
444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt
cp: failed to close ‘kk.txt’: Permission denied

(quoted diff) +0 -2

-r--r--r-- 1 cdc admincdc 7 nov 18  2016 444.txt
-r--r--r-- 1 cdc admincdc 0 nov 18  2016 kk.txt

cdc@l056:~$
cdc@l056:~$
cdc@l056:~$

cdc@l056:~$ sudo su
[sudo] password for cdc:
root@l056:/home/cdc# rm -f kk.txt 444.txt; echo "prueba" > 444.txt;
chmod 444 444.txt; cp -p 444.txt kk.txt; ls -ld 444.txt kk.txt

(quoted diff) +0 -2

-r--r--r-- 1 root root 7 nov 18  2016 444.txt
-r--r--r-- 1 root root 7 nov 18  2016 kk.txt

root@l056:/home/cdc#
root@l056:/home/cdc#

root@l056:/home/cdc# exit
cdc@l056:~$

See at the end the logs with and without the debug option activated at the glusterfs client.

Thanks again, and sorry for the long message.

Omar

* Logs on the glusterfs client side with NO DEBUG:

[2016-11-18 15:45:43.681215] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote
operation failed. Path: /cdc/kk.txt (cc98e831-3410-4202-8254-2d4b48e8ea68) [Permission denied] [2016-11-18 15:45:43.681297] W [fuse-bridge.c:989:fuse_fd_cbk] 0-glusterfs-fuse: 399: OPEN() /cdc/kk.txt => -1 (Permission denied) [2016-11-18 15:45:55.457922] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 15:45:55.457978] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 15:45:55.457997] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 15:45:55.458011] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 15:45:55.471933] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---

* Logs on the glusterfs client side with DEBUG activated:

[2016-11-18 16:06:17.040092] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
STAT scheduled
as fast fop
[2016-11-18 16:06:17.045986] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.046132] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.046833] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret 0 [Invalid argument] [2016-11-18 16:06:17.047671] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop
[2016-11-18 16:06:17.050539] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.050867] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.051948] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.051871] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.051946] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.062108] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret 0 [Invalid argument] [2016-11-18 16:06:17.063692] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.064015] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.065068] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 4 times between [2016-11-18 16:06:17.040092] and [2016-11-18 16:06:17.064175] [2016-11-18 16:06:17.065066] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.065232] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.065755] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.065897] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.066016] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
missing on subvol home-lab-3-client-0
[2016-11-18 16:06:17.066104] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.066594] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.066669] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.066771] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.066898] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.066906] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.067019] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.066915] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.067066] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.067107] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:06:17.067151] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:06:17.067179] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1
error: No such
file or directory [No such file or directory]
[2016-11-18 16:06:17.067206] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067250] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.067275] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.067308] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.067387] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:06:17.068157] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.068281] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.068738] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.068781] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.068805] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
missing on subvol home-lab-3-client-0
[2016-11-18 16:06:17.068827] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.069286] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.069345] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.069452] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.069507] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.069556] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.069585] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.069939] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.069983] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.070005] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:06:17.070038] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:06:17.070098] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e0498, home-lab-3-dht returned -1
error: No such
file or directory [No such file or directory]
[2016-11-18 16:06:17.070128] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070154] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.070178] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.070208] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.070251] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e0498, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:06:17.070537] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
MKNOD scheduled
as normal fop
[2016-11-18 16:06:17.070826] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:06:17.071100] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
10456189239296 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071142] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
10499693871104 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071183] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
10500911267840 and avail_inodes is: 99.00
[2016-11-18 16:06:17.071343] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
8600432410624 and avail_inodes is: 99.00
[2016-11-18 16:06:17.073943] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.074273] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
STAT scheduled
as fast fop
[2016-11-18 16:06:17.075035] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
OPEN scheduled
as fast fop
[2016-11-18 16:06:17.075913] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
WRITE scheduled
as slow fop
[2016-11-18 16:06:17.076205] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
FSTAT scheduled
as fast fop
[2016-11-18 16:06:17.076636] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.077282] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120276.
[2016-11-18 16:06:17.077981] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:06:17.078032] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 16:06:17.078049] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:06:17.078064] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:06:17.079156] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.079794] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.082196] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.082772] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.082318] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.082770] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.083168] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.083282] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:06:17.083320] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.083991] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.084043] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.084048] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.084092] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.084114] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.084167] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.084091] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.084377] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.084406] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:06:17.084511] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:06:17.084534] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
error: No such
file or directory [No such file or directory]
[2016-11-18 16:06:17.084683] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084783] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.084857] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.084889] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.085001] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:06:17.086098] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
STAT scheduled
as fast fop
[2016-11-18 16:06:17.087256] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.087173] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:06:17.087254] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.087749] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.087795] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:06:17.088014] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:06:17.088036] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:06:17.088598] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.088639] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
[2016-11-18 16:06:17.088804] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.088855] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
[2016-11-18 16:06:17.088819] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.089069] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
[2016-11-18 16:06:17.089380] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:06:17.089421] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
[2016-11-18 16:06:17.089443] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:06:17.089463] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:06:17.089575] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
error: No such
file or directory [No such file or directory]
[2016-11-18 16:06:17.089601] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.089712] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.089738] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:06:17.090006] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:06:17.090156] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:06:17.090648] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
MKNOD scheduled
as normal fop
[2016-11-18 16:06:17.090732] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:06:17.091499] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:06:17.091843] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.093831] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.094711] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:06:17.093216] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.094709] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
OPEN scheduled
as fast fop
[2016-11-18 16:06:17.095280] E [MSGID: 114031] [client-rpc-fops.c:444:client3_3_open_cbk] 0-home-lab-3-client-3: remote
operation failed. Path: /cdc/kk.txt (09bfb6fa-82fb-463b-abd9-bb7ef88e10c0) [Permission denied] [2016-11-18 16:06:17.095341] D [MSGID: 0] [client-rpc-fops.c:448:client3_3_open_cbk] 0-stack-trace:
stack-address:
0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error:
Permission denied
[Permission denied]
[2016-11-18 16:06:17.095366] D [MSGID: 0] [dht-inode-read.c:45:dht_open_cbk] 0-home-lab-3-dht: subvolume home-lab-3-client-3 returned -1 [Permission denied] [2016-11-18 16:06:17.095480] D [MSGID: 0] [dht-inode-read.c:59:dht_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-dht returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095671] D [MSGID: 0] [read-ahead.c:101:ra_open_cbk]
0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-read-ahead returned -1 error: Permission denied [Permission denied] [2016-11-18 16:06:17.095759] D [MSGID: 0] [io-cache.c:605:ioc_open_cbk]
0-stack-trace: stack-address: 0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: Permission denied [Permission denied] [2016-11-18 16:06:17.095963] D [MSGID: 0] [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.095988] D [MSGID: 0] [defaults.c:1137:default_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.096275] D [MSGID: 0] [io-stats.c:1936:io_stats_open_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: Permission denied [Permission denied]
[2016-11-18 16:06:17.096362] W [fuse-bridge.c:989:fuse_fd_cbk] 0-glusterfs-fuse: 338: OPEN() /cdc/kk.txt => -1 (Permission denied) [2016-11-18 16:06:17.097104] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:06:17.097674] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:17.099912] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:06:24.785225] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-0: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:06:24.785338] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-1: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:06:24.785359] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-2: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:06:24.785377] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-3: returning as transport is already disconnected OR
there are no frames (0 || 0)

** TEST DONE as user cdc --> ERROR

[2016-11-18 16:06:46.737686] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]

(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]

))))) 0-: 138.4.2.246:49172: ping timer event already removed [2016-11-18 16:06:49.545698] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]

(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]

))))) 0-: 138.4.2.177:49157: ping timer event already removed [2016-11-18 16:06:49.546474] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]

(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]

))))) 0-: 138.4.2.189:49157: ping timer event already removed [2016-11-18 16:06:49.547093] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fa8f0851da3] (--> /lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x84)[0x7fa8f061f184]

(--> /lib64/libgfrpc.so.0(+0x13787)[0x7fa8f061f787] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x29a)[0x7fa8f061c07a] (--> /usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xefce)[0x7fa8e22fffce]

))))) 0-: 138.4.2.248:49157: ping timer event already removed

** user CDC becomed root at this point

[2016-11-18 16:07:11.907892] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 7 times between [2016-11-18 16:06:17.086098] and [2016-11-18 16:07:11.906944] [2016-11-18 16:07:11.907884] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.908875] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.909179] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.909256] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop" repeated 2 times between [2016-11-18 16:06:17.047671] and [2016-11-18 16:07:11.908114]
[2016-11-18 16:07:11.909254] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
STAT scheduled
as fast fop
[2016-11-18 16:07:11.911109] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.911018] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.911107] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: UNLINK scheduled as normal fop
[2016-11-18 16:07:11.911871] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.912045] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.913067] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: STAT scheduled as fast fop" repeated 2 times between [2016-11-18 16:07:11.909254] and [2016-11-18 16:07:11.912165] [2016-11-18 16:07:11.913065] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.913193] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.913710] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.913757] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.913788] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
missing on subvol home-lab-3-client-0
[2016-11-18 16:07:11.913810] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.914336] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.914379] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.914424] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.914448] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.914501] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.914617] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.914649] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.914698] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.914724] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:07:11.914743] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:07:11.914764] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
error: No such
file or directory [No such file or directory]
[2016-11-18 16:07:11.914791] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914825] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.914855] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.914880] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.914947] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:07:11.915569] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.915644] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.916090] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.916134] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/444.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.916158] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/444.txt
missing on subvol home-lab-3-client-0
[2016-11-18 16:07:11.916185] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.916593] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.916780] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.916878] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.916994] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.917076] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.916906] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.917117] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.917153] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/444.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.917205] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-0 cached_subvol null [2016-11-18 16:07:11.917368] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/444.txt [2016-11-18 16:07:11.917437] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
error: No such
file or directory [No such file or directory]
[2016-11-18 16:07:11.917467] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917496] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.917521] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.917551] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.917726] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:07:11.917958] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
MKNOD scheduled
as normal fop
[2016-11-18 16:07:11.918230] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/444.txt on home-lab-3-client-0
[2016-11-18 16:07:11.918488] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-1': avail_percent is: 96.00 and avail_space is:
10456166432768 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918570] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-3': avail_percent is: 95.00 and avail_space is:
8600417075200 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918603] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-2': avail_percent is: 96.00 and avail_space is:
10500905500672 and avail_inodes is: 99.00
[2016-11-18 16:07:11.918675] D [MSGID: 0] [dht-diskusage.c:92:dht_du_info_cbk] 0-home-lab-3-dht: subvolume 'home-lab-3-client-0': avail_percent is: 95.00 and avail_space is:
10499686531072 and avail_inodes is: 99.00
[2016-11-18 16:07:11.919095] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.919377] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
STAT scheduled
as fast fop
[2016-11-18 16:07:11.920040] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
OPEN scheduled
as fast fop
[2016-11-18 16:07:11.920889] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
WRITE scheduled
as slow fop
[2016-11-18 16:07:11.921051] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
FSTAT scheduled
as fast fop
[2016-11-18 16:07:11.921583] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.922068] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120276.
[2016-11-18 16:07:11.922936] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:07:11.922988] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/444.txt
[2016-11-18 16:07:11.923004] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:07:11.923026] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:07:11.923779] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
STAT scheduled
as fast fop
[2016-11-18 16:07:11.924504] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.926692] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop" repeated 2 times between [2016-11-18 16:06:17.097104] and [2016-11-18 16:07:11.923946]
[2016-11-18 16:07:11.926690] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.926858] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.927327] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.927376] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.927401] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:07:11.927422] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.927940] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.927987] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.928043] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.928055] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.928058] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.928170] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.928174] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.928277] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.928360] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:07:11.928381] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:07:11.928403] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
error: No such
file or directory [No such file or directory]
[2016-11-18 16:07:11.928431] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928456] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.928480] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.928511] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.928753] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:07:11.930485] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.929862] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
STAT scheduled
as fast fop
[2016-11-18 16:07:11.930483] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: LOOKUP scheduled as fast fop
[2016-11-18 16:07:11.930602] D [MSGID: 0] [dht-common.c:2684:dht_lookup]
0-home-lab-3-dht: Calling fresh lookup for /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.931071] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.931124] D [MSGID: 0] [dht-common.c:2271:dht_lookup_cbk] 0-home-lab-3-dht: fresh_lookup returned for /cdc/kk.txt with op_ret -1 [No such file or directory] [2016-11-18 16:07:11.931148] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk] 0-home-lab-3-dht: Entry /cdc/kk.txt missing on subvol home-lab-3-client-3
[2016-11-18 16:07:11.931348] D [MSGID: 0] [dht-common.c:2055:dht_lookup_everywhere] 0-home-lab-3-dht: winding lookup call to 4 subvols
[2016-11-18 16:07:11.931880] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-0 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.931927] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-0
[2016-11-18 16:07:11.932002] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-3 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.931994] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-2 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.931991] D [MSGID: 0] [client-rpc-fops.c:2945:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-client-1 returned -1 error: No
such file or directory [No such file or directory] [2016-11-18 16:07:11.932078] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-2
[2016-11-18 16:07:11.932116] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-1
[2016-11-18 16:07:11.932315] D [MSGID: 0] [dht-common.c:1862:dht_lookup_everywhere_cbk] 0-home-lab-3-dht: returned
with op_ret -1 and op_errno 2 (/cdc/kk.txt) from subvol home-lab-3-client-3
[2016-11-18 16:07:11.932363] D [MSGID: 0] [dht-common.c:1527:dht_lookup_everywhere_done] 0-home-lab-3-dht: STATUS:
hashed_subvol home-lab-3-client-3 cached_subvol null [2016-11-18 16:07:11.932383] D [MSGID: 0] [dht-common.c:1588:dht_lookup_everywhere_done] 0-home-lab-3-dht: There was no cached file and unlink on hashed is not skipped /cdc/kk.txt [2016-11-18 16:07:11.932407] D [MSGID: 0] [dht-common.c:1591:dht_lookup_everywhere_done] 0-stack-trace:
stack-address: 0x7fa8ee0e1310, home-lab-3-dht returned -1
error: No such
file or directory [No such file or directory]
[2016-11-18 16:07:11.932446] D [MSGID: 0] [io-cache.c:256:ioc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932471] D [MSGID: 0] [quick-read.c:447:qr_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-quick-read returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.932494] D [MSGID: 0] [md-cache.c:863:mdc_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-md-cache returned -1 error: No such file or directory [No such file or directory]
[2016-11-18 16:07:11.932518] D [MSGID: 0] [defaults.c:1266:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3-io-threads returned -1 error: No such file or
directory [No such file or directory]
[2016-11-18 16:07:11.932725] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2016-11-18 16:07:11.932545] D [MSGID: 0] [io-stats.c:2116:io_stats_lookup_cbk] 0-stack-trace: stack-address:
0x7fa8ee0e1310, home-lab-3 returned -1 error: No such file or directory
[No such file or directory]
[2016-11-18 16:07:11.932722] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
MKNOD scheduled
as normal fop
[2016-11-18 16:07:11.933008] D [MSGID: 0] [dht-common.c:5559:dht_mknod_wind_to_avail_subvol] 0-home-lab-3-dht:
creating /cdc/kk.txt on home-lab-3-client-3
[2016-11-18 16:07:11.934086] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph -1418558531
[2016-11-18 16:07:11.934323] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.934922] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
STAT scheduled
as fast fop
[2016-11-18 16:07:11.935497] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.936102] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.936903] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
OPEN scheduled
as fast fop
[2016-11-18 16:07:11.937540] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
WRITE scheduled
as slow fop
[2016-11-18 16:07:11.937682] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads:
FSTAT scheduled
as fast fop
[2016-11-18 16:07:11.938267] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.938757] D [fuse-bridge.c:52:fuse_invalidate] 0-fuse: Invalidate inode id 140363155120652.
[2016-11-18 16:07:11.938994] I [io-stats.c:1574:io_stats_dump_fd] 0-home-lab-3: --- fd stats ---
[2016-11-18 16:07:11.939072] I [io-stats.c:1579:io_stats_dump_fd]
0-home-lab-3: Filename : /cdc/kk.txt
[2016-11-18 16:07:11.939096] I [io-stats.c:1594:io_stats_dump_fd]
0-home-lab-3: BytesWritten : 7 bytes
[2016-11-18 16:07:11.939112] I [io-stats.c:1606:io_stats_dump_fd] 0-home-lab-3: Write 000004b+ : 1
[2016-11-18 16:07:11.939448] D [MSGID: 0] [io-threads.c:349:iot_schedule] 0-home-lab-3-io-threads: SETATTR scheduled as normal fop
[2016-11-18 16:07:11.940028] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0
[2016-11-18 16:07:11.942130] D [fuse-bridge.c:5264:notify] 0-fuse: got event 19 on graph 0

** TEST DONE as root --> NO PROBLEM

[2016-11-18 16:07:28.794150] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-2: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:07:31.794656] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-0: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:07:31.794925] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-1: returning as transport is already disconnected OR
there are no frames (0 || 0)
[2016-11-18 16:07:31.794984] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-home-lab-3-client-3: returning as transport is already disconnected OR
there are no frames (0 || 0)

** USER ROOT EXITED at this point

Quoted 20 lines · Miklos Szeredi

Quoted 7 lines · Omar Walid Llorente

In any case, the kernel we are using (3.13.0-96-generic #143-Ubuntu SMP Mon

Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) has fuse

included
(not as a module) so if it would be necessary to change the fuse mount
options some help would be needed.

Can you reproduce this on latest (e.g 4.8) kernel?

Also fuse debug logs (with whaterver kernel) could be useful; not sure
how to enable it for glusterfs, usual convention for fuse filesystems is "-d".

Thanks,
Miklos

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 8 lines

----------------------------------------------------------------
Centro de Cálculo         Depto. Ingeniería Sistemas Telemáticos
E-mail: [email protected]        Universidad Politécnica de Madrid
Fax:(+34) 913367333                 E.T.S. Ing. Telecomunicación
Tel:(+34) 915495700-Ext.3005                28040 Madrid (Spain)
Tel:(+34) 915495762-Ext.3005
Tel:(+34) 913367366-Ext.3005
----------------------------------------------------------------