Skip to contentSkip to conversation

Random IO errors on nfs clients running linux > 4.20

DISCUSSION Alberto Gonzalez Iniesta 4 participants 16 messages Apr 29, 2020 – Oct 26, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello NFS maintainers,

I'm sorry for reporting this (a little bit) late, but it took us (Miguel in Cc:) some time to track this issue to an exact kernel update.

We're running a +200 clients NFS server with Ubuntu 16.04 and 18.04 clients. The server runs Debian 8.11 (jessie) with Linux 3.16.0 and nfs-kernel-server 1:1.2.8-9+deb8u1. It has been working some years now without issues.

But since we started moving clients from Ubuntu 16.04 to Ubuntu 18.04 some of them started experiencing failures while working on NFS mounts. The failures are arbitrary and sometimes it may take more than 20 minutes to come out (which made finding out which kernel version introduced this a pain). We are almost sure that some directories are more prone to suffer from this than others (maybe related to path length/chars?).

The error is also not very "verbose", from an strace:

execve("/bin/ls", ["ls", "-lR", "Becas y ayudas/"], 0x7ffccb7f5b20 /* 16 vars */) = 0 [lots of uninteresting output]
openat(AT_FDCWD, "Becas y ayudas/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3

fstat(3, {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
fstat(3, {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
fstat(1, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0

ioctl(1, TCGETS, 0x7ffd8b725c80) = -1 ENOTTY (Inappropriate ioctl for device)
getdents(3, /* 35 entries */, 32768) = 1936
[lots of lstats)
lstat("Becas y ayudas/Convocatorias", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
getdents(3, 0x561af78de890, 32768) = -1 EIO (Input/output error)

(I can send you the full output if you need it)

We can run the previous "ls -lR" 20 times and get no error, or get this "ls: leyendo el directorio 'Becas y ayudas/': Error de entrada/salida" (ls: reading directorio 'Becas y ayudas/': Input/Output Error") every now and then.

The error happens (obviously?) with ls, rsync and the users's GUI tools.

There's nothing in dmesg (or elsewhere).
These are the kernels with tried:

4.18.0-25   -> Can't reproduce
4.19.0      -> Can't reproduce
4.20.17     -> Happening (hard to reproduce)
5.0.0-15    -> Happening (hard to reproduce)
5.3.0-45    -> Happening (more frequently)
5.6.0-rc7   -> Reproduced a couple of times after boot, then nothing

We did long (as in daylong) testing trying to reproduce this with all those kernel versions, so we are pretty sure 4.18 and 4.19 don't experience this and our Ubuntu 16.04 clients don't have any issue.

I know we aren't providing much info but we are really looking forward to doing all the testing required (we already spent lots of time in it).

Thanks for your work.

Regards,

Alberto

Signature · 2 lines
Alberto González Iniesta             | Universidad a Distancia
[email protected]            | de Madrid
#1 lore raw
Quoted 4 lines · Alberto Gonzalez Iniesta

Hello NFS maintainers,

I know we aren't providing much info but we are really looking forward to doing all the testing required (we already spent lots of time in it).

Hi,

Sorry, I was providing way too little info...
We're using NFSv4 with kerberos, mounts are done with:
mount -t nfs4 -o sec=krb5p,exec,noauto pluto.XXXX:/publico /media/pluto

Regards,

Alberto

Signature · 5 lines
Alberto Gonzalez Iniesta    | Formación, consultoría y soporte técnico
mailto/sip: [email protected] | en GNU/Linux y software libre
Encrypted mail preferred    | http://inittab.com

Key fingerprint = 5347 CBD8 3E30 A9EB 4D7D  4BF2 009B 3375 6B9A AA55
#2 lore raw
In reply to Alberto Gonzalez Iniesta
Quoted 20 lines · Alberto Gonzalez Iniesta

I'm sorry for reporting this (a little bit) late, but it took us (Miguel in Cc:) some time to track this issue to an exact kernel update.

We're running a +200 clients NFS server with Ubuntu 16.04 and 18.04 clients. The server runs Debian 8.11 (jessie) with Linux 3.16.0 and nfs-kernel-server 1:1.2.8-9+deb8u1. It has been working some years now without issues.

But since we started moving clients from Ubuntu 16.04 to Ubuntu 18.04 some of them started experiencing failures while working on NFS mounts. The failures are arbitrary and sometimes it may take more than 20 minutes to come out (which made finding out which kernel version introduced this a pain). We are almost sure that some directories are more prone to suffer from this than others (maybe related to path length/chars?).

The error is also not very "verbose", from an strace:

execve("/bin/ls", ["ls", "-lR", "Becas y ayudas/"], 0x7ffccb7f5b20 /* 16 vars */) = 0 [lots of uninteresting output]
openat(AT_FDCWD, "Becas y ayudas/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3

Quoted 8 lines
fstat(3, {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
fstat(3, {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
fstat(1, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0

ioctl(1, TCGETS, 0x7ffd8b725c80) = -1 ENOTTY (Inappropriate ioctl for device)
getdents(3, /* 35 entries */, 32768) = 1936
[lots of lstats)
lstat("Becas y ayudas/Convocatorias", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
getdents(3, 0x561af78de890, 32768) = -1 EIO (Input/output error)

Ideas off the top of my head....

It'd be really useful to get a network trace--something like tcpdump -s0 -wtmp.pcap -i<interface>, then reproduce the problem, then look through it to see if you can find the READDIR or STAT or whatever that results in the unexpected EIO. But if takes a while to reproduce, that may be difficult.

Is there anything in the logs?

It might be worth turning on some more debug logging--see the "rpcdebug" command.

--b.

Quoted 35 lines

(I can send you the full output if you need it)

We can run the previous "ls -lR" 20 times and get no error, or get this "ls: leyendo el directorio 'Becas y ayudas/': Error de entrada/salida" (ls: reading directorio 'Becas y ayudas/': Input/Output Error") every now and then.

The error happens (obviously?) with ls, rsync and the users's GUI tools.

There's nothing in dmesg (or elsewhere).
These are the kernels with tried:

4.18.0-25   -> Can't reproduce
4.19.0      -> Can't reproduce
4.20.17     -> Happening (hard to reproduce)
5.0.0-15    -> Happening (hard to reproduce)
5.3.0-45    -> Happening (more frequently)
5.6.0-rc7   -> Reproduced a couple of times after boot, then nothing

We did long (as in daylong) testing trying to reproduce this with all those kernel versions, so we are pretty sure 4.18 and 4.19 don't experience this and our Ubuntu 16.04 clients don't have any issue.

I know we aren't providing much info but we are really looking forward to doing all the testing required (we already spent lots of time in it).

Thanks for your work.

Regards,

Alberto

Signature · 2 lines
Alberto González Iniesta             | Universidad a Distancia
[email protected]            | de Madrid
#3 lore raw
Quoted 38 lines · J. Bruce Fields
Quoted 20 lines · Alberto Gonzalez Iniesta

I'm sorry for reporting this (a little bit) late, but it took us (Miguel in Cc:) some time to track this issue to an exact kernel update.

We're running a +200 clients NFS server with Ubuntu 16.04 and 18.04 clients. The server runs Debian 8.11 (jessie) with Linux 3.16.0 and nfs-kernel-server 1:1.2.8-9+deb8u1. It has been working some years now without issues.

But since we started moving clients from Ubuntu 16.04 to Ubuntu 18.04 some of them started experiencing failures while working on NFS mounts. The failures are arbitrary and sometimes it may take more than 20 minutes to come out (which made finding out which kernel version introduced this a pain). We are almost sure that some directories are more prone to suffer from this than others (maybe related to path length/chars?).

The error is also not very "verbose", from an strace:

execve("/bin/ls", ["ls", "-lR", "Becas y ayudas/"], 0x7ffccb7f5b20 /* 16 vars */) = 0 [lots of uninteresting output]
openat(AT_FDCWD, "Becas y ayudas/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3

Quoted 8 lines
fstat(3, {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
fstat(3, {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
fstat(1, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0

ioctl(1, TCGETS, 0x7ffd8b725c80) = -1 ENOTTY (Inappropriate ioctl for device)
getdents(3, /* 35 entries */, 32768) = 1936
[lots of lstats)
lstat("Becas y ayudas/Convocatorias", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
getdents(3, 0x561af78de890, 32768) = -1 EIO (Input/output error)

Ideas off the top of my head....

It'd be really useful to get a network trace--something like tcpdump -s0 -wtmp.pcap -i<interface>, then reproduce the problem, then look through it to see if you can find the READDIR or STAT or whatever that results in the unexpected EIO. But if takes a while to reproduce, that may be difficult.

Quoted 4 lines

Is there anything in the logs?

It might be worth turning on some more debug logging--see the "rpcdebug" command.

Hi, Bruce et at.

I'm sorry this reply took so long, but with debugging enabled the error was harder to reproduce.

I'm attaching 3 log files (2 with just "nfs" debugging and one with "nfs" and "rpc" debugging modules enabled).

I'm also attaching a pcap file, don't know if it would be useful since we run "sec=krb5p".

Let me know if there's anything else I can test/provide. These tests were done with Ubuntu's 5.3.0-53-generic.

Thanks,

Alberto

Quoted 34 lines
Quoted 34 lines

(I can send you the full output if you need it)

We can run the previous "ls -lR" 20 times and get no error, or get this "ls: leyendo el directorio 'Becas y ayudas/': Error de entrada/salida" (ls: reading directorio 'Becas y ayudas/': Input/Output Error") every now and then.

The error happens (obviously?) with ls, rsync and the users's GUI tools.

There's nothing in dmesg (or elsewhere).
These are the kernels with tried:

4.18.0-25   -> Can't reproduce
4.19.0      -> Can't reproduce
4.20.17     -> Happening (hard to reproduce)
5.0.0-15    -> Happening (hard to reproduce)
5.3.0-45    -> Happening (more frequently)
5.6.0-rc7   -> Reproduced a couple of times after boot, then nothing

We did long (as in daylong) testing trying to reproduce this with all those kernel versions, so we are pretty sure 4.18 and 4.19 don't experience this and our Ubuntu 16.04 clients don't have any issue.

I know we aren't providing much info but we are really looking forward to doing all the testing required (we already spent lots of time in it).

Thanks for your work.

Regards,

Alberto

Signature · 2 lines
Alberto González Iniesta             | Universidad a Distancia
[email protected]            | de Madrid
Signature · 2 lines
Alberto González Iniesta             | Universidad a Distancia
[email protected]            | de Madrid
Sep  8 16:03:23 portatil264 kernel: [15032.854134] RPC:  3283 call_decode result 0
Sep  8 16:03:23 portatil264 kernel: [15032.854140] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
Sep  8 16:03:23 portatil264 kernel: [15032.854142] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
Sep  8 16:03:23 portatil264 kernel: [15032.854145] RPC:       wake_up_first(00000000d3f50f4d "ForeChannel Slot table")
Sep  8 16:03:23 portatil264 kernel: [15032.854147] nfs4_free_slot: slotid 1 highest_used_slotid 0
Sep  8 16:03:23 portatil264 kernel: [15032.854148] nfs41_sequence_process: Error 0 free the slot 
Sep  8 16:03:23 portatil264 kernel: [15032.854150] RPC:       wake_up_first(00000000d3f50f4d "ForeChannel Slot table")
Sep  8 16:03:23 portatil264 kernel: [15032.854152] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 16:03:23 portatil264 kernel: [15032.854154] RPC:  3283 return 0, status 0
Sep  8 16:03:23 portatil264 kernel: [15032.854155] RPC:  3283 release task
Sep  8 16:03:23 portatil264 kernel: [15032.854159] RPC:       freeing buffer of size 4584 at 00000000a3649daf
Sep  8 16:03:23 portatil264 kernel: [15032.854162] RPC:  3283 release request 0000000079df89b2
Sep  8 16:03:23 portatil264 kernel: [15032.854164] RPC:       wake_up_first(00000000c5ee49ee "xprt_backlog")
Sep  8 16:03:23 portatil264 kernel: [15032.854166] RPC:       rpc_release_client(00000000b930c343)
Sep  8 16:03:23 portatil264 kernel: [15032.854169] RPC:  3283 freeing task
Sep  8 16:03:23 portatil264 kernel: [15032.854179] NFS: nfs_update_inode(0:53/31982477 fh_crc=0x47b6a915 ct=1 info=0x427e7f)
Sep  8 16:03:23 portatil264 kernel: [15032.854183] NFS: (0:53/31982477) revalidation complete
Sep  8 16:03:23 portatil264 kernel: [15032.854186] NFS: dentry_delete(innovacion/Proyectos de investigación, 28080c)
Sep  8 16:03:23 portatil264 kernel: [15032.854240] NFS: readdir(departamentos/innovacion) starting at cookie 19
Sep  8 16:03:23 portatil264 kernel: [15032.854259] _nfs4_proc_readdir: dentry = departamentos/innovacion, cookie = 0
Sep  8 16:03:23 portatil264 kernel: [15032.854262] RPC:       new task initialized, procpid 26029
Sep  8 16:03:23 portatil264 kernel: [15032.854263] RPC:       allocated task 00000000493c7c0e
Sep  8 16:03:23 portatil264 kernel: [15032.854267] RPC:  3284 __rpc_execute flags=0x4080
Sep  8 16:03:23 portatil264 kernel: [15032.854269] --> nfs41_call_sync_prepare data->seq_server 00000000eb43854e
Sep  8 16:03:23 portatil264 kernel: [15032.854271] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
Sep  8 16:03:23 portatil264 kernel: [15032.854273] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
Sep  8 16:03:23 portatil264 kernel: [15032.854276] RPC:  3284 call_start nfs4 proc READDIR (sync)
Sep  8 16:03:23 portatil264 kernel: [15032.854278] RPC:  3284 call_reserve (status 0)
Sep  8 16:03:23 portatil264 kernel: [15032.854281] RPC:  3284 reserved req 0000000079df89b2 xid 5dfa3876
Sep  8 16:03:23 portatil264 kernel: [15032.854282] RPC:  3284 call_reserveresult (status 0)
Sep  8 16:03:23 portatil264 kernel: [15032.854284] RPC:  3284 call_refresh (status 0)
Sep  8 16:03:23 portatil264 kernel: [15032.854287] RPC:  3284 call_refreshresult (status 0)
Sep  8 16:03:23 portatil264 kernel: [15032.854289] RPC:  3284 call_allocate (status 0)
Sep  8 16:03:23 portatil264 kernel: [15032.854292] RPC:  3284 allocated buffer of size 4144 at 00000000a3649daf
Sep  8 16:03:23 portatil264 kernel: [15032.854297] RPC:  3284 call_encode (status 0)
Sep  8 16:03:23 portatil264 kernel: [15032.854301] RPC:       gss_get_mic_v2
Sep  8 16:03:23 portatil264 kernel: [15032.854319] encode_sequence: sessionid=1582481124:15749:9224:0 seqid=195 slotid=0 max_slotid=0 cache_this=0
Sep  8 16:03:23 portatil264 kernel: [15032.854323] encode_readdir: cookie = 0, verifier = 00000000:00000000, bitmap = 0018091a:00b0a23a:00000000
Sep  8 16:03:23 portatil264 kernel: [15032.854325] RPC:       gss_wrap_kerberos_v2
Sep  8 16:03:23 portatil264 kernel: [15032.854350] RPC:  3284 call_transmit (status 0)
Sep  8 16:03:23 portatil264 kernel: [15032.854352] RPC:  3284 xprt_prepare_transmit
Sep  8 16:03:23 portatil264 kernel: [15032.854389] RPC:       xs_tcp_send_request(272) = 0
Sep  8 16:03:23 portatil264 kernel: [15032.854392] RPC:       wake_up_first(000000001bb91292 "xprt_sending")
Sep  8 16:03:23 portatil264 kernel: [15032.854395] RPC:  3284 sleep_on(queue "xprt_pending" time 4298650492)
Sep  8 16:03:23 portatil264 kernel: [15032.854397] RPC:  3284 added to queue 00000000a9a9ecc7 "xprt_pending"
Sep  8 16:03:23 portatil264 kernel: [15032.854398] RPC:  3284 setting alarm for 60000 ms
Sep  8 16:03:23 portatil264 kernel: [15032.854401] RPC:  3284 sync task going to sleep
Sep  8 16:03:23 portatil264 kernel: [15032.935579] RPC:       xs_data_ready...
Sep  8 16:03:23 portatil264 kernel: [15032.939135] RPC:       xs_data_ready...
Sep  8 16:03:23 portatil264 kernel: [15032.941013] decode_attr_owner: uid=138184
Sep  8 16:03:23 portatil264 kernel: [15033.016025] RPC:       xs_data_ready...
Sep  8 16:03:23 portatil264 kernel: [15033.016117] RPC:       xs_data_ready...
Sep  8 16:03:23 portatil264 kernel: [15033.016124] RPC:  3284 xid 5dfa3876 complete (4064 bytes received)
Sep  8 16:03:23 portatil264 kernel: [15033.016129] RPC:  3284 __rpc_wake_up_task (now 4298650533)
Sep  8 16:03:23 portatil264 kernel: [15033.016131] RPC:  3284 disabling timer
Sep  8 16:03:23 portatil264 kernel: [15033.016136] RPC:  3284 removed from queue 00000000a9a9ecc7 "xprt_pending"
Sep  8 16:03:23 portatil264 kernel: [15033.016149] RPC:       __rpc_wake_up_task done
Sep  8 16:03:23 portatil264 kernel: [15033.016231] RPC:  3284 sync task resuming
Sep  8 16:03:23 portatil264 kernel: [15033.016234] RPC:  3284 call_status (status 0)
Sep  8 16:03:23 portatil264 kernel: [15033.016235] RPC:  3284 call_decode (status 0)
Sep  8 16:03:23 portatil264 kernel: [15033.016240] RPC:       gss_verify_mic_v2
Sep  8 16:03:23 portatil264 kernel: [15033.016276] RPC:  3284 call_decode result -5
Sep  8 16:03:23 portatil264 kernel: [15033.016281] nfs41_sequence_process: Error 1 free the slot 
Sep  8 16:03:23 portatil264 kernel: [15033.016286] RPC:       wake_up_first(00000000d3f50f4d "ForeChannel Slot table")
Sep  8 16:03:23 portatil264 kernel: [15033.016288] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 16:03:23 portatil264 kernel: [15033.016290] RPC:  3284 return 0, status -5
Sep  8 16:03:23 portatil264 kernel: [15033.016291] RPC:  3284 release task
Sep  8 16:03:23 portatil264 kernel: [15033.016295] RPC:       freeing buffer of size 4144 at 00000000a3649daf
Sep  8 16:03:23 portatil264 kernel: [15033.016298] RPC:  3284 release request 0000000079df89b2
Sep  8 16:03:23 portatil264 kernel: [15033.016300] RPC:       wake_up_first(00000000c5ee49ee "xprt_backlog")
Sep  8 16:03:23 portatil264 kernel: [15033.016302] RPC:       rpc_release_client(00000000b930c343)
Sep  8 16:03:23 portatil264 kernel: [15033.016304] RPC:  3284 freeing task
Sep  8 16:03:23 portatil264 kernel: [15033.016309] _nfs4_proc_readdir: returns -5
Sep  8 16:03:23 portatil264 kernel: [15033.016318] NFS: readdir(departamentos/innovacion) returns -5
Sep  8 16:03:23 portatil264 kernel: [15033.016788] NFS: permission(0:53/28049412), mask=0x81, res=0
Sep  8 16:03:23 portatil264 kernel: [15033.016793] NFS: nfs_lookup_revalidate_done(publico/departamentos) is valid
Sep  8 16:03:23 portatil264 kernel: [15033.016797] NFS: permission(0:53/28051004), mask=0x81, res=0
Sep  8 16:03:23 portatil264 kernel: [15033.016801] NFS: revalidating (0:53/30801922)
Sep  8 16:03:23 portatil264 kernel: [15033.016808] RPC:       new task initialized, procpid 26029
Sep  8 16:03:23 portatil264 kernel: [15033.016810] RPC:       allocated task 000000005afe768d
Sep  8 16:03:23 portatil264 kernel: [15033.016814] RPC:  3285 __rpc_execute flags=0x4080
Sep  8 16:03:23 portatil264 kernel: [15033.016817] --> nfs41_call_sync_prepare data->seq_server 00000000eb43854e
Sep  8 16:03:23 portatil264 kernel: [15033.016820] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
Sep  8 16:03:23 portatil264 kernel: [15033.016823] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
Sep  8 16:03:23 portatil264 kernel: [15033.016826] RPC:  3285 call_start nfs4 proc GETATTR (sync)
Sep  8 16:03:23 portatil264 kernel: [15033.016829] RPC:  3285 call_reserve (status 0)
Sep  8 16:03:23 portatil264 kernel: [15033.016832] RPC:  3285 reserved req 0000000079df89b2 xid 5efa3876
Sep  8 16:03:23 portatil264 kernel: [15033.016834] RPC:  3285 call_reserveresult (status 0)
Sep  8 16:03:23 portatil264 kernel: [15033.016836] RPC:  3285 call_refresh (status 0)
Sep  8 16:03:23 portatil264 kernel: [15033.016840] RPC:  3285 call_refreshresult (status 0)
Sep  8 16:03:23 portatil264 kernel: [15033.016844] RPC:  3285 call_allocate (status 0)
Sep  8 16:03:23 portatil264 kernel: [15033.016848] RPC:  3285 allocated buffer of size 4584 at 000000007624e941
Sep  8 16:03:23 portatil264 kernel: [15033.016850] RPC:  3285 call_encode (status 0)
Sep  8 16:03:23 portatil264 kernel: [15033.016857] RPC:       gss_get_mic_v2
Sep  8 16:03:23 portatil264 kernel: [15033.016876] encode_sequence: sessionid=1582481124:15749:9224:0 seqid=196 slotid=0 max_slotid=0 cache_this=0
Sep  8 16:03:23 portatil264 kernel: [15033.016879] RPC:       gss_wrap_kerberos_v2
Sep  8 16:03:23 portatil264 kernel: [15033.016907] RPC:  3285 call_transmit (status 0)
Sep  8 16:03:23 portatil264 kernel: [15033.016908] RPC:  3285 xprt_prepare_transmit
Sep  8 16:03:23 portatil264 kernel: [15033.016941] RPC:       xs_tcp_send_request(248) = 0
Sep  8 16:03:23 portatil264 kernel: [15033.016944] RPC:       wake_up_first(000000001bb91292 "xprt_sending")
Sep  8 16:03:23 portatil264 kernel: [15033.016947] RPC:  3285 sleep_on(queue "xprt_pending" time 4298650533)
Sep  8 16:03:23 portatil264 kernel: [15033.016948] RPC:  3285 added to queue 00000000a9a9ecc7 "xprt_pending"
Sep  8 16:03:23 portatil264 kernel: [15033.016949] RPC:  3285 setting alarm for 60000 ms
Sep  8 16:03:23 portatil264 kernel: [15033.016952] RPC:  3285 sync task going to sleep
Sep  8 11:03:08 portatil264 kernel: [  294.429356] NFS: permission(0:53/28049412), mask=0x81, res=0
Sep  8 11:03:08 portatil264 kernel: [  294.429365] NFS: nfs_lookup_revalidate_done(publico/departamentos) is valid
Sep  8 11:03:08 portatil264 kernel: [  294.429369] NFS: permission(0:53/28051004), mask=0x81, res=0
Sep  8 11:03:08 portatil264 kernel: [  294.429372] NFS: nfs_lookup_revalidate_done(departamentos/innovacion) is valid
Sep  8 11:03:08 portatil264 kernel: [  294.429377] NFS: revalidating (0:53/30801924)
Sep  8 11:03:08 portatil264 kernel: [  294.429392] --> nfs41_call_sync_prepare data->seq_server 00000000368715a5
Sep  8 11:03:08 portatil264 kernel: [  294.429396] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
Sep  8 11:03:08 portatil264 kernel: [  294.429399] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
Sep  8 11:03:08 portatil264 kernel: [  294.429456] encode_sequence: sessionid=1582481124:15703:9206:0 seqid=50 slotid=0 max_slotid=0 cache_this=0
Sep  8 11:03:09 portatil264 kernel: [  294.480184] decode_attr_type: type=040000
Sep  8 11:03:09 portatil264 kernel: [  294.480187] decode_attr_change: change attribute=6810276040818143508
Sep  8 11:03:09 portatil264 kernel: [  294.480189] decode_attr_size: file size=4096
Sep  8 11:03:09 portatil264 kernel: [  294.480190] decode_attr_fsid: fsid=(0x0/0x0)
Sep  8 11:03:09 portatil264 kernel: [  294.480192] decode_attr_fileid: fileid=30801924
Sep  8 11:03:09 portatil264 kernel: [  294.480193] decode_attr_fs_locations: fs_locations done, error = 0
Sep  8 11:03:09 portatil264 kernel: [  294.480194] decode_attr_mode: file mode=02770
Sep  8 11:03:09 portatil264 kernel: [  294.480196] decode_attr_nlink: nlink=19
Sep  8 11:03:09 portatil264 kernel: [  294.480208] decode_attr_owner: uid=0
Sep  8 11:03:09 portatil264 kernel: [  294.480212] decode_attr_group: gid=709
Sep  8 11:03:09 portatil264 kernel: [  294.480213] decode_attr_rdev: rdev=(0x0:0x0)
Sep  8 11:03:09 portatil264 kernel: [  294.480215] decode_attr_space_used: space used=4096
Sep  8 11:03:09 portatil264 kernel: [  294.480216] decode_attr_time_access: atime=1599472206
Sep  8 11:03:09 portatil264 kernel: [  294.480217] decode_attr_time_metadata: ctime=1585640954
Sep  8 11:03:09 portatil264 kernel: [  294.480218] decode_attr_time_modify: mtime=1585640954
Sep  8 11:03:09 portatil264 kernel: [  294.480220] decode_attr_mounted_on_fileid: fileid=30801924
Sep  8 11:03:09 portatil264 kernel: [  294.480221] decode_getfattr_attrs: xdr returned 0
Sep  8 11:03:09 portatil264 kernel: [  294.480223] decode_getfattr_generic: xdr returned 0
Sep  8 11:03:09 portatil264 kernel: [  294.480228] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
Sep  8 11:03:09 portatil264 kernel: [  294.480230] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
Sep  8 11:03:09 portatil264 kernel: [  294.480232] nfs4_free_slot: slotid 1 highest_used_slotid 0
Sep  8 11:03:09 portatil264 kernel: [  294.480233] nfs41_sequence_process: Error 0 free the slot 
Sep  8 11:03:09 portatil264 kernel: [  294.480234] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 11:03:09 portatil264 kernel: [  294.480246] NFS: nfs_update_inode(0:53/30801924 fh_crc=0x21ccf1a0 ct=1 info=0x427e7f)
Sep  8 11:03:09 portatil264 kernel: [  294.480249] NFS: (0:53/30801924) revalidation complete
Sep  8 11:03:09 portatil264 kernel: [  294.480252] NFS: dentry_delete(departamentos/innovacion, 28084c)
Sep  8 11:03:09 portatil264 kernel: [  294.480271] NFS: permission(0:53/28049412), mask=0x81, res=0
Sep  8 11:03:09 portatil264 kernel: [  294.480273] NFS: nfs_lookup_revalidate_done(publico/departamentos) is valid
Sep  8 11:03:09 portatil264 kernel: [  294.480275] NFS: permission(0:53/28051004), mask=0x81, res=0
Sep  8 11:03:09 portatil264 kernel: [  294.480277] NFS: revalidating (0:53/30801924)
Sep  8 11:03:09 portatil264 kernel: [  294.480282] --> nfs41_call_sync_prepare data->seq_server 00000000368715a5
Sep  8 11:03:09 portatil264 kernel: [  294.480284] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
Sep  8 11:03:09 portatil264 kernel: [  294.480285] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
Sep  8 11:03:09 portatil264 kernel: [  294.480306] encode_sequence: sessionid=1582481124:15703:9206:0 seqid=51 slotid=0 max_slotid=0 cache_this=0
Sep  8 11:03:09 portatil264 kernel: [  294.562935] decode_attr_type: type=040000
Sep  8 11:03:09 portatil264 kernel: [  294.562937] decode_attr_change: change attribute=6810276040818143508
Sep  8 11:03:09 portatil264 kernel: [  294.562939] decode_attr_size: file size=4096
Sep  8 11:03:09 portatil264 kernel: [  294.562940] decode_attr_fsid: fsid=(0x0/0x0)
Sep  8 11:03:09 portatil264 kernel: [  294.562941] decode_attr_fileid: fileid=30801924
Sep  8 11:03:09 portatil264 kernel: [  294.562942] decode_attr_fs_locations: fs_locations done, error = 0
Sep  8 11:03:09 portatil264 kernel: [  294.562943] decode_attr_mode: file mode=02770
Sep  8 11:03:09 portatil264 kernel: [  294.562944] decode_attr_nlink: nlink=19
Sep  8 11:03:09 portatil264 kernel: [  294.562950] decode_attr_owner: uid=0
Sep  8 11:03:09 portatil264 kernel: [  294.562954] decode_attr_group: gid=709
Sep  8 11:03:09 portatil264 kernel: [  294.562955] decode_attr_rdev: rdev=(0x0:0x0)
Sep  8 11:03:09 portatil264 kernel: [  294.562956] decode_attr_space_used: space used=4096
Sep  8 11:03:09 portatil264 kernel: [  294.562957] decode_attr_time_access: atime=1599472206
Sep  8 11:03:09 portatil264 kernel: [  294.562958] decode_attr_time_metadata: ctime=1585640954
Sep  8 11:03:09 portatil264 kernel: [  294.562961] decode_attr_time_modify: mtime=1585640954
Sep  8 11:03:09 portatil264 kernel: [  294.562962] decode_attr_mounted_on_fileid: fileid=30801924
Sep  8 11:03:09 portatil264 kernel: [  294.562963] decode_getfattr_attrs: xdr returned 0
Sep  8 11:03:09 portatil264 kernel: [  294.562964] decode_getfattr_generic: xdr returned 0
Sep  8 11:03:09 portatil264 kernel: [  294.562968] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
Sep  8 11:03:09 portatil264 kernel: [  294.562970] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
Sep  8 11:03:09 portatil264 kernel: [  294.562971] nfs4_free_slot: slotid 1 highest_used_slotid 0
Sep  8 11:03:09 portatil264 kernel: [  294.562973] nfs41_sequence_process: Error 0 free the slot 
Sep  8 11:03:09 portatil264 kernel: [  294.562974] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 11:03:09 portatil264 kernel: [  294.562982] NFS: nfs_update_inode(0:53/30801924 fh_crc=0x21ccf1a0 ct=1 info=0x427e7f)
Sep  8 11:03:09 portatil264 kernel: [  294.562984] NFS: (0:53/30801924) revalidation complete
Sep  8 11:03:09 portatil264 kernel: [  294.562987] NFS: nfs_lookup_revalidate_done(departamentos/innovacion) is valid
Sep  8 11:03:09 portatil264 kernel: [  294.562991] NFS: permission(0:53/30801924), mask=0x24, res=0
Sep  8 11:03:09 portatil264 kernel: [  294.562995] NFS: open dir(departamentos/innovacion)
Sep  8 11:03:09 portatil264 kernel: [  294.563017] NFS: readdir(departamentos/innovacion) starting at cookie 0
Sep  8 11:03:09 portatil264 kernel: [  294.563029] _nfs4_proc_readdir: dentry = departamentos/innovacion, cookie = 0
Sep  8 11:03:09 portatil264 kernel: [  294.563034] --> nfs41_call_sync_prepare data->seq_server 00000000368715a5
Sep  8 11:03:09 portatil264 kernel: [  294.563036] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
Sep  8 11:03:09 portatil264 kernel: [  294.563037] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
Sep  8 11:03:09 portatil264 kernel: [  294.563057] encode_sequence: sessionid=1582481124:15703:9206:0 seqid=52 slotid=0 max_slotid=0 cache_this=0
Sep  8 11:03:09 portatil264 kernel: [  294.563060] encode_readdir: cookie = 0, verifier = 00000000:00000000, bitmap = 0018091a:00b0a23a:00000000
Sep  8 11:03:09 portatil264 kernel: [  294.643573] nfs41_sequence_process: Error 1 free the slot 
Sep  8 11:03:09 portatil264 kernel: [  294.643576] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 11:03:09 portatil264 kernel: [  294.643583] _nfs4_proc_readdir: returns -5
Sep  8 11:03:09 portatil264 kernel: [  294.643589] NFS: readdir(departamentos/innovacion) returns -5
Sep  8 11:03:09 portatil264 kernel: [  294.644109] NFS: dentry_delete(departamentos/innovacion, 28084c)
Sep  8 11:03:10 portatil264 kernel: [  296.012504] nfs4_renew_state: start
Sep  8 11:03:10 portatil264 kernel: [  296.012510] nfs4_schedule_state_renewal: requeueing work. Lease period = 59
Sep  8 11:03:10 portatil264 kernel: [  296.012513] nfs4_renew_state: done
Sep  8 11:02:03 portatil264 kernel: [  229.334583] FS-Cache: Loaded
Sep  8 11:02:03 portatil264 kernel: [  229.374147] FS-Cache: Netfs 'nfs' registered for caching
Sep  8 11:02:04 portatil264 kernel: [  229.528716] NFS: Registering the id_resolver key type
Sep  8 11:02:04 portatil264 kernel: [  229.528731] Key type id_resolver registered
Sep  8 11:02:04 portatil264 kernel: [  229.528732] Key type id_legacy registered
Sep  8 11:02:52 portatil264 kernel: [  278.316243] NFS: permission(0:53/28049412), mask=0x81, res=-10
Sep  8 11:02:52 portatil264 kernel: [  278.316247] NFS: revalidating (0:53/28049412)
Sep  8 11:02:52 portatil264 kernel: [  278.316260] --> nfs41_call_sync_prepare data->seq_server 00000000368715a5
Sep  8 11:02:52 portatil264 kernel: [  278.316263] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
Sep  8 11:02:52 portatil264 kernel: [  278.316265] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
Sep  8 11:02:52 portatil264 kernel: [  278.316309] encode_sequence: sessionid=1582481124:15703:9206:0 seqid=44 slotid=0 max_slotid=0 cache_this=0
Sep  8 11:02:52 portatil264 kernel: [  278.373736] decode_attr_type: type=040000
Sep  8 11:02:52 portatil264 kernel: [  278.373738] decode_attr_change: change attribute=6769953312283491654
Sep  8 11:02:52 portatil264 kernel: [  278.373739] decode_attr_size: file size=4096
Sep  8 11:02:52 portatil264 kernel: [  278.373741] decode_attr_fsid: fsid=(0x0/0x0)
Sep  8 11:02:52 portatil264 kernel: [  278.373742] decode_attr_fileid: fileid=28049412
Sep  8 11:02:52 portatil264 kernel: [  278.373743] decode_attr_fs_locations: fs_locations done, error = 0
Sep  8 11:02:52 portatil264 kernel: [  278.373745] decode_attr_mode: file mode=0755
Sep  8 11:02:52 portatil264 kernel: [  278.373746] decode_attr_nlink: nlink=6
Sep  8 11:02:52 portatil264 kernel: [  278.373757] decode_attr_owner: uid=0
Sep  8 11:02:52 portatil264 kernel: [  278.373761] decode_attr_group: gid=0
Sep  8 11:02:52 portatil264 kernel: [  278.373763] decode_attr_rdev: rdev=(0x0:0x0)
Sep  8 11:02:52 portatil264 kernel: [  278.373764] decode_attr_space_used: space used=4096
Sep  8 11:02:52 portatil264 kernel: [  278.373765] decode_attr_time_access: atime=1599538321
Sep  8 11:02:52 portatil264 kernel: [  278.373766] decode_attr_time_metadata: ctime=1576252587
Sep  8 11:02:52 portatil264 kernel: [  278.373767] decode_attr_time_modify: mtime=1576252587
Sep  8 11:02:52 portatil264 kernel: [  278.373768] decode_attr_mounted_on_fileid: fileid=28049412
Sep  8 11:02:52 portatil264 kernel: [  278.373770] decode_getfattr_attrs: xdr returned 0
Sep  8 11:02:52 portatil264 kernel: [  278.373771] decode_getfattr_generic: xdr returned 0
Sep  8 11:02:52 portatil264 kernel: [  278.373776] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
Sep  8 11:02:52 portatil264 kernel: [  278.373778] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
Sep  8 11:02:52 portatil264 kernel: [  278.373780] nfs4_free_slot: slotid 1 highest_used_slotid 0
Sep  8 11:02:52 portatil264 kernel: [  278.373781] nfs41_sequence_process: Error 0 free the slot 
Sep  8 11:02:52 portatil264 kernel: [  278.373782] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 11:02:52 portatil264 kernel: [  278.373794] NFS: nfs_update_inode(0:53/28049412 fh_crc=0xc875a563 ct=2 info=0x427e7f)
Sep  8 11:02:52 portatil264 kernel: [  278.373797] NFS: (0:53/28049412) revalidation complete
Sep  8 11:02:52 portatil264 kernel: [  278.373800] NFS: permission(0:53/28049412), mask=0x1, res=0
Sep  8 11:02:52 portatil264 kernel: [  278.373806] NFS: nfs_lookup_revalidate_done(publico/departamentos) is valid
Sep  8 11:02:52 portatil264 kernel: [  278.373812] --> nfs41_call_sync_prepare data->seq_server 00000000368715a5
Sep  8 11:02:52 portatil264 kernel: [  278.373814] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
Sep  8 11:02:52 portatil264 kernel: [  278.373815] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
Sep  8 11:02:52 portatil264 kernel: [  278.373834] encode_sequence: sessionid=1582481124:15703:9206:0 seqid=45 slotid=0 max_slotid=0 cache_this=0
Sep  8 11:02:52 portatil264 kernel: [  278.430991] decode_attr_type: type=00
Sep  8 11:02:52 portatil264 kernel: [  278.430994] decode_attr_change: change attribute=6792489597889629835
Sep  8 11:02:52 portatil264 kernel: [  278.430995] decode_attr_size: file size=4096
Sep  8 11:02:52 portatil264 kernel: [  278.430996] decode_attr_fsid: fsid=(0x0/0x0)
Sep  8 11:02:52 portatil264 kernel: [  278.430997] decode_attr_fileid: fileid=0
Sep  8 11:02:52 portatil264 kernel: [  278.430998] decode_attr_fs_locations: fs_locations done, error = 0
Sep  8 11:02:52 portatil264 kernel: [  278.430999] decode_attr_mode: file mode=00
Sep  8 11:02:52 portatil264 kernel: [  278.431000] decode_attr_nlink: nlink=1
Sep  8 11:02:52 portatil264 kernel: [  278.431001] decode_attr_rdev: rdev=(0x0:0x0)
Sep  8 11:02:52 portatil264 kernel: [  278.431002] decode_attr_space_used: space used=0
Sep  8 11:02:52 portatil264 kernel: [  278.431003] decode_attr_time_access: atime=0
Sep  8 11:02:52 portatil264 kernel: [  278.431004] decode_attr_time_metadata: ctime=1581499725
Sep  8 11:02:52 portatil264 kernel: [  278.431005] decode_attr_time_modify: mtime=1581499725
Sep  8 11:02:52 portatil264 kernel: [  278.431006] decode_attr_mounted_on_fileid: fileid=0
Sep  8 11:02:52 portatil264 kernel: [  278.431007] decode_getfattr_attrs: xdr returned 0
Sep  8 11:02:52 portatil264 kernel: [  278.431008] decode_getfattr_generic: xdr returned 0
Sep  8 11:02:52 portatil264 kernel: [  278.431012] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
Sep  8 11:02:52 portatil264 kernel: [  278.431014] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
Sep  8 11:02:52 portatil264 kernel: [  278.431016] nfs4_free_slot: slotid 1 highest_used_slotid 0
Sep  8 11:02:52 portatil264 kernel: [  278.431017] nfs41_sequence_process: Error 0 free the slot 
Sep  8 11:02:52 portatil264 kernel: [  278.431018] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 11:02:52 portatil264 kernel: [  278.431026] NFS: nfs_update_inode(0:53/28051004 fh_crc=0x3843eff0 ct=1 info=0x26040)
Sep  8 11:02:52 portatil264 kernel: [  278.431033] NFS: permission(0:53/28051004), mask=0x1, res=0
Sep  8 11:02:52 portatil264 kernel: [  278.431037] NFS: lookup(departamentos/innovacion)
Sep  8 11:02:52 portatil264 kernel: [  278.431039] NFS call  lookup innovacion
Sep  8 11:02:52 portatil264 kernel: [  278.431043] --> nfs41_call_sync_prepare data->seq_server 00000000368715a5
Sep  8 11:02:52 portatil264 kernel: [  278.431045] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
Sep  8 11:02:52 portatil264 kernel: [  278.431047] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
Sep  8 11:02:52 portatil264 kernel: [  278.431066] encode_sequence: sessionid=1582481124:15703:9206:0 seqid=46 slotid=0 max_slotid=0 cache_this=0
Sep  8 11:02:53 portatil264 kernel: [  278.585742] decode_attr_type: type=040000
Sep  8 11:02:53 portatil264 kernel: [  278.585745] decode_attr_change: change attribute=6810276040818143508
Sep  8 11:02:53 portatil264 kernel: [  278.585746] decode_attr_size: file size=4096
Sep  8 11:02:53 portatil264 kernel: [  278.585747] decode_attr_fsid: fsid=(0x0/0x0)
Sep  8 11:02:53 portatil264 kernel: [  278.585748] decode_attr_fileid: fileid=30801924
Sep  8 11:02:53 portatil264 kernel: [  278.585749] decode_attr_fs_locations: fs_locations done, error = 0
Sep  8 11:02:53 portatil264 kernel: [  278.585750] decode_attr_mode: file mode=02770
Sep  8 11:02:53 portatil264 kernel: [  278.585751] decode_attr_nlink: nlink=19
Sep  8 11:02:53 portatil264 kernel: [  278.585758] decode_attr_owner: uid=0
Sep  8 11:02:53 portatil264 kernel: [  278.898245] decode_attr_group: gid=709
Sep  8 11:02:53 portatil264 kernel: [  278.898249] decode_attr_rdev: rdev=(0x0:0x0)
Sep  8 11:02:53 portatil264 kernel: [  278.898250] decode_attr_space_used: space used=4096
Sep  8 11:02:53 portatil264 kernel: [  278.898251] decode_attr_time_access: atime=1599472206
Sep  8 11:02:53 portatil264 kernel: [  278.898252] decode_attr_time_metadata: ctime=1585640954
Sep  8 11:02:53 portatil264 kernel: [  278.898253] decode_attr_time_modify: mtime=1585640954
Sep  8 11:02:53 portatil264 kernel: [  278.898255] decode_attr_mounted_on_fileid: fileid=30801924
Sep  8 11:02:53 portatil264 kernel: [  278.898256] decode_getfattr_attrs: xdr returned 0
Sep  8 11:02:53 portatil264 kernel: [  278.898258] decode_getfattr_generic: xdr returned 0
Sep  8 11:02:53 portatil264 kernel: [  278.898266] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
Sep  8 11:02:53 portatil264 kernel: [  278.898268] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
Sep  8 11:02:53 portatil264 kernel: [  278.898270] nfs4_free_slot: slotid 1 highest_used_slotid 0
Sep  8 11:02:53 portatil264 kernel: [  278.898271] nfs41_sequence_process: Error 0 free the slot 
Sep  8 11:02:53 portatil264 kernel: [  278.898273] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 11:02:53 portatil264 kernel: [  278.898281] NFS reply lookup: 0
Sep  8 11:02:53 portatil264 kernel: [  278.898292] NFS: nfs_fhget(0:53/30801924 fh_crc=0x21ccf1a0 ct=1)
Sep  8 11:02:53 portatil264 kernel: [  278.898302] NFS: dentry_delete(departamentos/innovacion, 20080c)
Sep  8 11:02:53 portatil264 kernel: [  278.898330] NFS: permission(0:53/28049412), mask=0x81, res=0
Sep  8 11:02:53 portatil264 kernel: [  278.898334] NFS: nfs_lookup_revalidate_done(publico/departamentos) is valid
Sep  8 11:02:53 portatil264 kernel: [  278.898336] NFS: permission(0:53/28051004), mask=0x81, res=0
Sep  8 11:02:53 portatil264 kernel: [  278.898338] NFS: revalidating (0:53/30801924)
Sep  8 11:02:53 portatil264 kernel: [  278.898349] --> nfs41_call_sync_prepare data->seq_server 00000000368715a5
Sep  8 11:02:53 portatil264 kernel: [  278.898351] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
Sep  8 11:02:53 portatil264 kernel: [  278.898352] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
Sep  8 11:02:53 portatil264 kernel: [  278.898393] encode_sequence: sessionid=1582481124:15703:9206:0 seqid=47 slotid=0 max_slotid=0 cache_this=0
Sep  8 11:02:53 portatil264 kernel: [  278.984901] decode_attr_type: type=040000
Sep  8 11:02:53 portatil264 kernel: [  278.984904] decode_attr_change: change attribute=6810276040818143508
Sep  8 11:02:53 portatil264 kernel: [  278.984905] decode_attr_size: file size=4096
Sep  8 11:02:53 portatil264 kernel: [  278.984906] decode_attr_fsid: fsid=(0x0/0x0)
Sep  8 11:02:53 portatil264 kernel: [  278.984907] decode_attr_fileid: fileid=30801924
Sep  8 11:02:53 portatil264 kernel: [  278.984909] decode_attr_fs_locations: fs_locations done, error = 0
Sep  8 11:02:53 portatil264 kernel: [  278.984910] decode_attr_mode: file mode=02770
Sep  8 11:02:53 portatil264 kernel: [  278.984911] decode_attr_nlink: nlink=19
Sep  8 11:02:53 portatil264 kernel: [  278.984919] decode_attr_owner: uid=0
Sep  8 11:02:53 portatil264 kernel: [  278.984922] decode_attr_group: gid=709
Sep  8 11:02:53 portatil264 kernel: [  278.984923] decode_attr_rdev: rdev=(0x0:0x0)
Sep  8 11:02:53 portatil264 kernel: [  278.984924] decode_attr_space_used: space used=4096
Sep  8 11:02:53 portatil264 kernel: [  278.984926] decode_attr_time_access: atime=1599472206
Sep  8 11:02:53 portatil264 kernel: [  278.984927] decode_attr_time_metadata: ctime=1585640954
Sep  8 11:02:53 portatil264 kernel: [  278.984928] decode_attr_time_modify: mtime=1585640954
Sep  8 11:02:53 portatil264 kernel: [  278.984929] decode_attr_mounted_on_fileid: fileid=30801924
Sep  8 11:02:53 portatil264 kernel: [  278.984930] decode_getfattr_attrs: xdr returned 0
Sep  8 11:02:53 portatil264 kernel: [  278.984931] decode_getfattr_generic: xdr returned 0
Sep  8 11:02:53 portatil264 kernel: [  278.984935] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
Sep  8 11:02:53 portatil264 kernel: [  278.984937] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
Sep  8 11:02:53 portatil264 kernel: [  278.984939] nfs4_free_slot: slotid 1 highest_used_slotid 0
Sep  8 11:02:53 portatil264 kernel: [  278.984940] nfs41_sequence_process: Error 0 free the slot 
Sep  8 11:02:53 portatil264 kernel: [  278.984941] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 11:02:53 portatil264 kernel: [  278.984951] NFS: nfs_update_inode(0:53/30801924 fh_crc=0x21ccf1a0 ct=1 info=0x427e7f)
Sep  8 11:02:53 portatil264 kernel: [  278.984954] NFS: (0:53/30801924) revalidation complete
Sep  8 11:02:53 portatil264 kernel: [  278.984957] NFS: nfs_lookup_revalidate_done(departamentos/innovacion) is valid
Sep  8 11:02:53 portatil264 kernel: [  278.984964] --> nfs41_call_sync_prepare data->seq_server 00000000368715a5
Sep  8 11:02:53 portatil264 kernel: [  278.984966] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
Sep  8 11:02:53 portatil264 kernel: [  278.984968] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
Sep  8 11:02:53 portatil264 kernel: [  278.984989] encode_sequence: sessionid=1582481124:15703:9206:0 seqid=48 slotid=0 max_slotid=0 cache_this=0
Sep  8 11:02:53 portatil264 kernel: [  279.083208] decode_attr_type: type=00
Sep  8 11:02:53 portatil264 kernel: [  279.083210] decode_attr_change: change attribute=6810276040818143508
Sep  8 11:02:53 portatil264 kernel: [  279.083211] decode_attr_size: file size=4096
Sep  8 11:02:53 portatil264 kernel: [  279.083212] decode_attr_fsid: fsid=(0x0/0x0)
Sep  8 11:02:53 portatil264 kernel: [  279.083213] decode_attr_fileid: fileid=0
Sep  8 11:02:53 portatil264 kernel: [  279.083215] decode_attr_fs_locations: fs_locations done, error = 0
Sep  8 11:02:53 portatil264 kernel: [  279.083216] decode_attr_mode: file mode=00
Sep  8 11:02:53 portatil264 kernel: [  279.083217] decode_attr_nlink: nlink=1
Sep  8 11:02:53 portatil264 kernel: [  279.083218] decode_attr_rdev: rdev=(0x0:0x0)
Sep  8 11:02:53 portatil264 kernel: [  279.083219] decode_attr_space_used: space used=0
Sep  8 11:02:53 portatil264 kernel: [  279.083220] decode_attr_time_access: atime=0
Sep  8 11:02:53 portatil264 kernel: [  279.083221] decode_attr_time_metadata: ctime=1585640954
Sep  8 11:02:53 portatil264 kernel: [  279.083222] decode_attr_time_modify: mtime=1585640954
Sep  8 11:02:53 portatil264 kernel: [  279.083223] decode_attr_mounted_on_fileid: fileid=0
Sep  8 11:02:53 portatil264 kernel: [  279.083224] decode_getfattr_attrs: xdr returned 0
Sep  8 11:02:53 portatil264 kernel: [  279.083225] decode_getfattr_generic: xdr returned 0
Sep  8 11:02:53 portatil264 kernel: [  279.083230] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=31
Sep  8 11:02:53 portatil264 kernel: [  279.083232] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
Sep  8 11:02:53 portatil264 kernel: [  279.083233] nfs4_free_slot: slotid 1 highest_used_slotid 0
Sep  8 11:02:53 portatil264 kernel: [  279.083234] nfs41_sequence_process: Error 0 free the slot 
Sep  8 11:02:53 portatil264 kernel: [  279.083236] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 11:02:53 portatil264 kernel: [  279.083244] NFS: nfs_update_inode(0:53/30801924 fh_crc=0x21ccf1a0 ct=1 info=0x26040)
Sep  8 11:02:53 portatil264 kernel: [  279.083249] NFS: permission(0:53/30801924), mask=0x24, res=0
Sep  8 11:02:53 portatil264 kernel: [  279.083253] NFS: open dir(departamentos/innovacion)
Sep  8 11:02:53 portatil264 kernel: [  279.083278] NFS: readdir(departamentos/innovacion) starting at cookie 0
Sep  8 11:02:53 portatil264 kernel: [  279.083293] _nfs4_proc_readdir: dentry = departamentos/innovacion, cookie = 0
Sep  8 11:02:53 portatil264 kernel: [  279.083298] --> nfs41_call_sync_prepare data->seq_server 00000000368715a5
Sep  8 11:02:53 portatil264 kernel: [  279.083300] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=31
Sep  8 11:02:53 portatil264 kernel: [  279.083302] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
Sep  8 11:02:53 portatil264 kernel: [  279.083326] encode_sequence: sessionid=1582481124:15703:9206:0 seqid=49 slotid=0 max_slotid=0 cache_this=0
Sep  8 11:02:53 portatil264 kernel: [  279.083329] encode_readdir: cookie = 0, verifier = 00000000:00000000, bitmap = 0018091a:00b0a23a:00000000
Sep  8 11:02:53 portatil264 kernel: [  279.188491] nfs41_sequence_process: Error 1 free the slot 
Sep  8 11:02:53 portatil264 kernel: [  279.188494] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 11:02:53 portatil264 kernel: [  279.188501] _nfs4_proc_readdir: returns -5
Sep  8 11:02:53 portatil264 kernel: [  279.188507] NFS: readdir(departamentos/innovacion) returns -5
Sep  8 11:02:53 portatil264 kernel: [  279.189204] NFS: dentry_delete(departamentos/innovacion, 28080c)
#4 lore raw
Quoted 46 lines · Alberto Gonzalez Iniesta
Quoted 38 lines · J. Bruce Fields
Quoted 20 lines · Alberto Gonzalez Iniesta

I'm sorry for reporting this (a little bit) late, but it took us (Miguel in Cc:) some time to track this issue to an exact kernel update.

We're running a +200 clients NFS server with Ubuntu 16.04 and 18.04 clients. The server runs Debian 8.11 (jessie) with Linux 3.16.0 and nfs-kernel-server 1:1.2.8-9+deb8u1. It has been working some years now without issues.

But since we started moving clients from Ubuntu 16.04 to Ubuntu 18.04 some of them started experiencing failures while working on NFS mounts. The failures are arbitrary and sometimes it may take more than 20 minutes to come out (which made finding out which kernel version introduced this a pain). We are almost sure that some directories are more prone to suffer from this than others (maybe related to path length/chars?).

The error is also not very "verbose", from an strace:

execve("/bin/ls", ["ls", "-lR", "Becas y ayudas/"], 0x7ffccb7f5b20 /* 16 vars */) = 0 [lots of uninteresting output]
openat(AT_FDCWD, "Becas y ayudas/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3

Quoted 8 lines
fstat(3, {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
fstat(3, {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
fstat(1, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0

ioctl(1, TCGETS, 0x7ffd8b725c80) = -1 ENOTTY (Inappropriate ioctl for device)
getdents(3, /* 35 entries */, 32768) = 1936
[lots of lstats)
lstat("Becas y ayudas/Convocatorias", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
getdents(3, 0x561af78de890, 32768) = -1 EIO (Input/output error)

Ideas off the top of my head....

It'd be really useful to get a network trace--something like tcpdump -s0 -wtmp.pcap -i<interface>, then reproduce the problem, then look through it to see if you can find the READDIR or STAT or whatever that results in the unexpected EIO. But if takes a while to reproduce, that may be difficult.

Quoted 4 lines

Is there anything in the logs?

It might be worth turning on some more debug logging--see the "rpcdebug" command.

Hi, Bruce et at.

Quoted 8 lines

I'm sorry this reply took so long, but with debugging enabled the error was harder to reproduce.

I'm attaching 3 log files (2 with just "nfs" debugging and one with "nfs" and "rpc" debugging modules enabled).

I'm also attaching a pcap file, don't know if it would be useful since we run "sec=krb5p".

Yeah, not too useful.

Quoted 33 lines

Let me know if there's anything else I can test/provide. These tests were done with Ubuntu's 5.3.0-53-generic.

Thanks,

Alberto

Quoted 26 lines
Quoted 26 lines

(I can send you the full output if you need it)

We can run the previous "ls -lR" 20 times and get no error, or get this "ls: leyendo el directorio 'Becas y ayudas/': Error de entrada/salida" (ls: reading directorio 'Becas y ayudas/': Input/Output Error") every now and then.

The error happens (obviously?) with ls, rsync and the users's GUI tools.

There's nothing in dmesg (or elsewhere).
These are the kernels with tried:

4.18.0-25   -> Can't reproduce
4.19.0      -> Can't reproduce
4.20.17     -> Happening (hard to reproduce)
5.0.0-15    -> Happening (hard to reproduce)
5.3.0-45    -> Happening (more frequently)
5.6.0-rc7   -> Reproduced a couple of times after boot, then nothing

We did long (as in daylong) testing trying to reproduce this with all those kernel versions, so we are pretty sure 4.18 and 4.19 don't experience this and our Ubuntu 16.04 clients don't have any issue.

I know we aren't providing much info but we are really looking forward to doing all the testing required (we already spent lots of time in it).

Thanks for your work.

So all I notice from this one is the readdir EIO came from call_decode. I suspect that means it failed in the xdr decoding. Looks like xdr decoding of the actual directory data (which is the complicated part) is done later, so this means it failed decoding the header or verifier, which is a little odd:

Quoted 5 lines
Sep  8 16:03:23 portatil264 kernel: [15033.016276] RPC:  3284 call_decode result -5
Sep  8 16:03:23 portatil264 kernel: [15033.016281] nfs41_sequence_process: Error 1 free the slot 
Sep  8 16:03:23 portatil264 kernel: [15033.016286] RPC:       wake_up_first(00000000d3f50f4d "ForeChannel Slot table")
Sep  8 16:03:23 portatil264 kernel: [15033.016288] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 16:03:23 portatil264 kernel: [15033.016290] RPC:  3284 return 0, status -5
Quoted 8 lines
Sep  8 16:03:23 portatil264 kernel: [15033.016291] RPC:  3284 release task
Sep  8 16:03:23 portatil264 kernel: [15033.016295] RPC:       freeing buffer of size 4144 at 00000000a3649daf
Sep  8 16:03:23 portatil264 kernel: [15033.016298] RPC:  3284 release request 0000000079df89b2
Sep  8 16:03:23 portatil264 kernel: [15033.016300] RPC:       wake_up_first(00000000c5ee49ee "xprt_backlog")
Sep  8 16:03:23 portatil264 kernel: [15033.016302] RPC:       rpc_release_client(00000000b930c343)
Sep  8 16:03:23 portatil264 kernel: [15033.016304] RPC:  3284 freeing task
Sep  8 16:03:23 portatil264 kernel: [15033.016309] _nfs4_proc_readdir: returns -5
Sep  8 16:03:23 portatil264 kernel: [15033.016318] NFS: readdir(departamentos/innovacion) returns -5

--b.

#5 lore raw
Quoted 26 lines · J. Bruce Fields
Quoted 25 lines · Alberto Gonzalez Iniesta
Quoted 24 lines · J. Bruce Fields
Quoted 23 lines · Alberto Gonzalez Iniesta

We can run the previous "ls -lR" 20 times and get no error, or get this "ls: leyendo el directorio 'Becas y ayudas/': Error de entrada/salida" (ls: reading directorio 'Becas y ayudas/': Input/Output Error") every now and then.

The error happens (obviously?) with ls, rsync and the users's GUI tools.

There's nothing in dmesg (or elsewhere).
These are the kernels with tried:

4.18.0-25   -> Can't reproduce
4.19.0      -> Can't reproduce
4.20.17     -> Happening (hard to reproduce)
5.0.0-15    -> Happening (hard to reproduce)
5.3.0-45    -> Happening (more frequently)
5.6.0-rc7   -> Reproduced a couple of times after boot, then nothing

We did long (as in daylong) testing trying to reproduce this with all those kernel versions, so we are pretty sure 4.18 and 4.19 don't experience this and our Ubuntu 16.04 clients don't have any issue.

I know we aren't providing much info but we are really looking forward to doing all the testing required (we already spent lots of time in it).

Quoted 21 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

Thanks for your work.

So all I notice from this one is the readdir EIO came from call_decode. I suspect that means it failed in the xdr decoding. Looks like xdr decoding of the actual directory data (which is the complicated part) is done later, so this means it failed decoding the header or verifier, which is a little odd:

Quoted 13 lines
Sep  8 16:03:23 portatil264 kernel: [15033.016276] RPC:  3284 call_decode result -5
Sep  8 16:03:23 portatil264 kernel: [15033.016281] nfs41_sequence_process: Error 1 free the slot 
Sep  8 16:03:23 portatil264 kernel: [15033.016286] RPC:       wake_up_first(00000000d3f50f4d "ForeChannel Slot table")
Sep  8 16:03:23 portatil264 kernel: [15033.016288] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 16:03:23 portatil264 kernel: [15033.016290] RPC:  3284 return 0, status -5
Sep  8 16:03:23 portatil264 kernel: [15033.016291] RPC:  3284 release task
Sep  8 16:03:23 portatil264 kernel: [15033.016295] RPC:       freeing buffer of size 4144 at 00000000a3649daf
Sep  8 16:03:23 portatil264 kernel: [15033.016298] RPC:  3284 release request 0000000079df89b2
Sep  8 16:03:23 portatil264 kernel: [15033.016300] RPC:       wake_up_first(00000000c5ee49ee "xprt_backlog")
Sep  8 16:03:23 portatil264 kernel: [15033.016302] RPC:       rpc_release_client(00000000b930c343)
Sep  8 16:03:23 portatil264 kernel: [15033.016304] RPC:  3284 freeing task
Sep  8 16:03:23 portatil264 kernel: [15033.016309] _nfs4_proc_readdir: returns -5
Sep  8 16:03:23 portatil264 kernel: [15033.016318] NFS: readdir(departamentos/innovacion) returns -5

Hi, Bruce et al.

Is there anything we can do to help debugging/fixing this? It's still biting our users with a +4.20.x kernel.

Thanks,

Alberto

Signature · 2 lines
Alberto González Iniesta             | Universidad a Distancia
[email protected]            | de Madrid
#6 lore raw
Quoted 53 lines · Alberto Gonzalez Iniesta
Quoted 26 lines · J. Bruce Fields
Quoted 25 lines · Alberto Gonzalez Iniesta
Quoted 24 lines · J. Bruce Fields
Quoted 23 lines · Alberto Gonzalez Iniesta

We can run the previous "ls -lR" 20 times and get no error, or get this "ls: leyendo el directorio 'Becas y ayudas/': Error de entrada/salida" (ls: reading directorio 'Becas y ayudas/': Input/Output Error") every now and then.

The error happens (obviously?) with ls, rsync and the users's GUI tools.

There's nothing in dmesg (or elsewhere).
These are the kernels with tried:

4.18.0-25   -> Can't reproduce
4.19.0      -> Can't reproduce
4.20.17     -> Happening (hard to reproduce)
5.0.0-15    -> Happening (hard to reproduce)
5.3.0-45    -> Happening (more frequently)
5.6.0-rc7   -> Reproduced a couple of times after boot, then nothing

We did long (as in daylong) testing trying to reproduce this with all those kernel versions, so we are pretty sure 4.18 and 4.19 don't experience this and our Ubuntu 16.04 clients don't have any issue.

I know we aren't providing much info but we are really looking forward to doing all the testing required (we already spent lots of time in it).

Quoted 21 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

Thanks for your work.

So all I notice from this one is the readdir EIO came from call_decode. I suspect that means it failed in the xdr decoding. Looks like xdr decoding of the actual directory data (which is the complicated part) is done later, so this means it failed decoding the header or verifier, which is a little odd:

Quoted 13 lines
Sep  8 16:03:23 portatil264 kernel: [15033.016276] RPC:  3284 call_decode result -5
Sep  8 16:03:23 portatil264 kernel: [15033.016281] nfs41_sequence_process: Error 1 free the slot 
Sep  8 16:03:23 portatil264 kernel: [15033.016286] RPC:       wake_up_first(00000000d3f50f4d "ForeChannel Slot table")
Sep  8 16:03:23 portatil264 kernel: [15033.016288] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 16:03:23 portatil264 kernel: [15033.016290] RPC:  3284 return 0, status -5
Sep  8 16:03:23 portatil264 kernel: [15033.016291] RPC:  3284 release task
Sep  8 16:03:23 portatil264 kernel: [15033.016295] RPC:       freeing buffer of size 4144 at 00000000a3649daf
Sep  8 16:03:23 portatil264 kernel: [15033.016298] RPC:  3284 release request 0000000079df89b2
Sep  8 16:03:23 portatil264 kernel: [15033.016300] RPC:       wake_up_first(00000000c5ee49ee "xprt_backlog")
Sep  8 16:03:23 portatil264 kernel: [15033.016302] RPC:       rpc_release_client(00000000b930c343)
Sep  8 16:03:23 portatil264 kernel: [15033.016304] RPC:  3284 freeing task
Sep  8 16:03:23 portatil264 kernel: [15033.016309] _nfs4_proc_readdir: returns -5
Sep  8 16:03:23 portatil264 kernel: [15033.016318] NFS: readdir(departamentos/innovacion) returns -5
Quoted 4 lines

Hi, Bruce et al.

Is there anything we can do to help debugging/fixing this? It's still biting our users with a +4.20.x kernel.

Alberto, can you share a snippet of a raw network capture that shows the READDIR Reply that fails to decode?

--
Chuck Lever

#7 lore raw
Quoted 20 lines · Chuck Lever
Quoted 20 lines
Quoted 19 lines

So all I notice from this one is the readdir EIO came from call_decode. I suspect that means it failed in the xdr decoding. Looks like xdr decoding of the actual directory data (which is the complicated part) is done later, so this means it failed decoding the header or verifier, which is a little odd:

Quoted 13 lines
Sep  8 16:03:23 portatil264 kernel: [15033.016276] RPC:  3284 call_decode result -5
Sep  8 16:03:23 portatil264 kernel: [15033.016281] nfs41_sequence_process: Error 1 free the slot 
Sep  8 16:03:23 portatil264 kernel: [15033.016286] RPC:       wake_up_first(00000000d3f50f4d "ForeChannel Slot table")
Sep  8 16:03:23 portatil264 kernel: [15033.016288] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 16:03:23 portatil264 kernel: [15033.016290] RPC:  3284 return 0, status -5
Sep  8 16:03:23 portatil264 kernel: [15033.016291] RPC:  3284 release task
Sep  8 16:03:23 portatil264 kernel: [15033.016295] RPC:       freeing buffer of size 4144 at 00000000a3649daf
Sep  8 16:03:23 portatil264 kernel: [15033.016298] RPC:  3284 release request 0000000079df89b2
Sep  8 16:03:23 portatil264 kernel: [15033.016300] RPC:       wake_up_first(00000000c5ee49ee "xprt_backlog")
Sep  8 16:03:23 portatil264 kernel: [15033.016302] RPC:       rpc_release_client(00000000b930c343)
Sep  8 16:03:23 portatil264 kernel: [15033.016304] RPC:  3284 freeing task
Sep  8 16:03:23 portatil264 kernel: [15033.016309] _nfs4_proc_readdir: returns -5
Sep  8 16:03:23 portatil264 kernel: [15033.016318] NFS: readdir(departamentos/innovacion) returns -5
Quoted 7 lines
Quoted 4 lines

Hi, Bruce et al.

Is there anything we can do to help debugging/fixing this? It's still biting our users with a +4.20.x kernel.

Alberto, can you share a snippet of a raw network capture that shows the READDIR Reply that fails to decode?

Hi, Chuck.

Thanks for your reply. We're using "sec=krb5p", which makes the network capture useless :-(
I'm afraid it's quite difficult to disable that, since we have +200 NFS clients on production.

Signature · 2 lines
Alberto González Iniesta             | Universidad a Distancia
[email protected]            | de Madrid
#8 lore raw
Quoted 30 lines · Alberto Gonzalez Iniesta
Quoted 25 lines · Chuck Lever
Quoted 24 lines
Quoted 19 lines

So all I notice from this one is the readdir EIO came from call_decode. I suspect that means it failed in the xdr decoding. Looks like xdr decoding of the actual directory data (which is the complicated part) is done later, so this means it failed decoding the header or verifier, which is a little odd:

Quoted 13 lines
Sep  8 16:03:23 portatil264 kernel: [15033.016276] RPC:  3284 call_decode result -5
Sep  8 16:03:23 portatil264 kernel: [15033.016281] nfs41_sequence_process: Error 1 free the slot 
Sep  8 16:03:23 portatil264 kernel: [15033.016286] RPC:       wake_up_first(00000000d3f50f4d "ForeChannel Slot table")
Sep  8 16:03:23 portatil264 kernel: [15033.016288] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 16:03:23 portatil264 kernel: [15033.016290] RPC:  3284 return 0, status -5
Sep  8 16:03:23 portatil264 kernel: [15033.016291] RPC:  3284 release task
Sep  8 16:03:23 portatil264 kernel: [15033.016295] RPC:       freeing buffer of size 4144 at 00000000a3649daf
Sep  8 16:03:23 portatil264 kernel: [15033.016298] RPC:  3284 release request 0000000079df89b2
Sep  8 16:03:23 portatil264 kernel: [15033.016300] RPC:       wake_up_first(00000000c5ee49ee "xprt_backlog")
Sep  8 16:03:23 portatil264 kernel: [15033.016302] RPC:       rpc_release_client(00000000b930c343)
Sep  8 16:03:23 portatil264 kernel: [15033.016304] RPC:  3284 freeing task
Sep  8 16:03:23 portatil264 kernel: [15033.016309] _nfs4_proc_readdir: returns -5
Sep  8 16:03:23 portatil264 kernel: [15033.016318] NFS: readdir(departamentos/innovacion) returns -5

Hi, Bruce et al.

Is there anything we can do to help debugging/fixing this? It's still biting our users with a +4.20.x kernel.

Quoted 7 lines
Quoted 2 lines

Alberto, can you share a snippet of a raw network capture that shows the READDIR Reply that fails to decode?

Hi, Chuck.

Thanks for your reply. We're using "sec=krb5p", which makes the network capture useless :-(

You can plug keytabs into Wireshark to enable it to decrypt the traffic.

--
Chuck Lever

#9 lore raw
Quoted 33 lines · Chuck Lever
Quoted 33 lines · Alberto Gonzalez Iniesta
Quoted 27 lines · Chuck Lever
Quoted 20 lines
Quoted 19 lines

So all I notice from this one is the readdir EIO came from call_decode. I suspect that means it failed in the xdr decoding. Looks like xdr decoding of the actual directory data (which is the complicated part) is done later, so this means it failed decoding the header or verifier, which is a little odd:

Quoted 13 lines
Sep  8 16:03:23 portatil264 kernel: [15033.016276] RPC:  3284 call_decode result -5
Sep  8 16:03:23 portatil264 kernel: [15033.016281] nfs41_sequence_process: Error 1 free the slot 
Sep  8 16:03:23 portatil264 kernel: [15033.016286] RPC:       wake_up_first(00000000d3f50f4d "ForeChannel Slot table")
Sep  8 16:03:23 portatil264 kernel: [15033.016288] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 16:03:23 portatil264 kernel: [15033.016290] RPC:  3284 return 0, status -5
Sep  8 16:03:23 portatil264 kernel: [15033.016291] RPC:  3284 release task
Sep  8 16:03:23 portatil264 kernel: [15033.016295] RPC:       freeing buffer of size 4144 at 00000000a3649daf
Sep  8 16:03:23 portatil264 kernel: [15033.016298] RPC:  3284 release request 0000000079df89b2
Sep  8 16:03:23 portatil264 kernel: [15033.016300] RPC:       wake_up_first(00000000c5ee49ee "xprt_backlog")
Sep  8 16:03:23 portatil264 kernel: [15033.016302] RPC:       rpc_release_client(00000000b930c343)
Sep  8 16:03:23 portatil264 kernel: [15033.016304] RPC:  3284 freeing task
Sep  8 16:03:23 portatil264 kernel: [15033.016309] _nfs4_proc_readdir: returns -5
Sep  8 16:03:23 portatil264 kernel: [15033.016318] NFS: readdir(departamentos/innovacion) returns -5
Quoted 4 lines

Hi, Bruce et al.

Is there anything we can do to help debugging/fixing this? It's still biting our users with a +4.20.x kernel.

Alberto, can you share a snippet of a raw network capture that shows the READDIR Reply that fails to decode?

Quoted 7 lines
Quoted 4 lines

Hi, Chuck.

Thanks for your reply. We're using "sec=krb5p", which makes the network capture useless :-(

You can plug keytabs into Wireshark to enable it to decrypt the traffic.

Cool, I'll look into that then.

Thanks!

Signature · 2 lines
Alberto González Iniesta             | Universidad a Distancia
[email protected]            | de Madrid
#10 lore raw
In reply to Chuck Lever
Quoted 33 lines · Chuck Lever
Quoted 33 lines · Alberto Gonzalez Iniesta
Quoted 27 lines · Chuck Lever
Quoted 20 lines
Quoted 19 lines

So all I notice from this one is the readdir EIO came from call_decode. I suspect that means it failed in the xdr decoding. Looks like xdr decoding of the actual directory data (which is the complicated part) is done later, so this means it failed decoding the header or verifier, which is a little odd:

Quoted 13 lines
Sep  8 16:03:23 portatil264 kernel: [15033.016276] RPC:  3284 call_decode result -5
Sep  8 16:03:23 portatil264 kernel: [15033.016281] nfs41_sequence_process: Error 1 free the slot 
Sep  8 16:03:23 portatil264 kernel: [15033.016286] RPC:       wake_up_first(00000000d3f50f4d "ForeChannel Slot table")
Sep  8 16:03:23 portatil264 kernel: [15033.016288] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 16:03:23 portatil264 kernel: [15033.016290] RPC:  3284 return 0, status -5
Sep  8 16:03:23 portatil264 kernel: [15033.016291] RPC:  3284 release task
Sep  8 16:03:23 portatil264 kernel: [15033.016295] RPC:       freeing buffer of size 4144 at 00000000a3649daf
Sep  8 16:03:23 portatil264 kernel: [15033.016298] RPC:  3284 release request 0000000079df89b2
Sep  8 16:03:23 portatil264 kernel: [15033.016300] RPC:       wake_up_first(00000000c5ee49ee "xprt_backlog")
Sep  8 16:03:23 portatil264 kernel: [15033.016302] RPC:       rpc_release_client(00000000b930c343)
Sep  8 16:03:23 portatil264 kernel: [15033.016304] RPC:  3284 freeing task
Sep  8 16:03:23 portatil264 kernel: [15033.016309] _nfs4_proc_readdir: returns -5
Sep  8 16:03:23 portatil264 kernel: [15033.016318] NFS: readdir(departamentos/innovacion) returns -5
Quoted 4 lines

Hi, Bruce et al.

Is there anything we can do to help debugging/fixing this? It's still biting our users with a +4.20.x kernel.

Alberto, can you share a snippet of a raw network capture that shows the READDIR Reply that fails to decode?

Quoted 6 lines
Quoted 4 lines

Hi, Chuck.

Thanks for your reply. We're using "sec=krb5p", which makes the network capture useless :-(

You can plug keytabs into Wireshark to enable it to decrypt the traffic.

Just skimming that range of history, there's some changes to the handling of gss sequence numbers, I wonder if there's a chance he could be hitting that? You had a workload that would lead to calls dropping out of the sequence number window, didn't you, Chuck? Is there a quick way to check whether that's happening?

--b.

#11 lore raw
Quoted 43 lines · Bruce Fields
Quoted 38 lines · Chuck Lever
Quoted 37 lines · Alberto Gonzalez Iniesta
Quoted 20 lines · Chuck Lever
Quoted 20 lines
Quoted 19 lines

So all I notice from this one is the readdir EIO came from call_decode. I suspect that means it failed in the xdr decoding. Looks like xdr decoding of the actual directory data (which is the complicated part) is done later, so this means it failed decoding the header or verifier, which is a little odd:

Quoted 13 lines
Sep  8 16:03:23 portatil264 kernel: [15033.016276] RPC:  3284 call_decode result -5
Sep  8 16:03:23 portatil264 kernel: [15033.016281] nfs41_sequence_process: Error 1 free the slot 
Sep  8 16:03:23 portatil264 kernel: [15033.016286] RPC:       wake_up_first(00000000d3f50f4d "ForeChannel Slot table")
Sep  8 16:03:23 portatil264 kernel: [15033.016288] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 16:03:23 portatil264 kernel: [15033.016290] RPC:  3284 return 0, status -5
Sep  8 16:03:23 portatil264 kernel: [15033.016291] RPC:  3284 release task
Sep  8 16:03:23 portatil264 kernel: [15033.016295] RPC:       freeing buffer of size 4144 at 00000000a3649daf
Sep  8 16:03:23 portatil264 kernel: [15033.016298] RPC:  3284 release request 0000000079df89b2
Sep  8 16:03:23 portatil264 kernel: [15033.016300] RPC:       wake_up_first(00000000c5ee49ee "xprt_backlog")
Sep  8 16:03:23 portatil264 kernel: [15033.016302] RPC:       rpc_release_client(00000000b930c343)
Sep  8 16:03:23 portatil264 kernel: [15033.016304] RPC:  3284 freeing task
Sep  8 16:03:23 portatil264 kernel: [15033.016309] _nfs4_proc_readdir: returns -5
Sep  8 16:03:23 portatil264 kernel: [15033.016318] NFS: readdir(departamentos/innovacion) returns -5
Quoted 7 lines
Quoted 4 lines

Hi, Bruce et al.

Is there anything we can do to help debugging/fixing this? It's still biting our users with a +4.20.x kernel.

Alberto, can you share a snippet of a raw network capture that shows the READDIR Reply that fails to decode?

Hi, Chuck.

Thanks for your reply. We're using "sec=krb5p", which makes the network capture useless :-(

Quoted 7 lines
Quoted 1 line

You can plug keytabs into Wireshark to enable it to decrypt the traffic.

Just skimming that range of history, there's some changes to the handling of gss sequence numbers, I wonder if there's a chance he could be hitting that? You had a workload that would lead to calls dropping out of the sequence number window, didn't you, Chuck? Is there a quick way to check whether that's happening?

The server is supposed to drop the connection when that happens, though I'm not sure 4.20's NFSD was perfect in that regard. Connection loss in itself wouldn't result in EIO.

--
Chuck Lever

#12 lore raw
Quoted 51 lines · Chuck Lever
Quoted 43 lines · Bruce Fields
Quoted 38 lines · Chuck Lever
Quoted 37 lines · Alberto Gonzalez Iniesta
Quoted 20 lines · Chuck Lever
Quoted 20 lines
Quoted 19 lines

So all I notice from this one is the readdir EIO came from call_decode. I suspect that means it failed in the xdr decoding. Looks like xdr decoding of the actual directory data (which is the complicated part) is done later, so this means it failed decoding the header or verifier, which is a little odd:

Quoted 13 lines
Sep  8 16:03:23 portatil264 kernel: [15033.016276] RPC:  3284 call_decode result -5
Sep  8 16:03:23 portatil264 kernel: [15033.016281] nfs41_sequence_process: Error 1 free the slot 
Sep  8 16:03:23 portatil264 kernel: [15033.016286] RPC:       wake_up_first(00000000d3f50f4d "ForeChannel Slot table")
Sep  8 16:03:23 portatil264 kernel: [15033.016288] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 16:03:23 portatil264 kernel: [15033.016290] RPC:  3284 return 0, status -5
Sep  8 16:03:23 portatil264 kernel: [15033.016291] RPC:  3284 release task
Sep  8 16:03:23 portatil264 kernel: [15033.016295] RPC:       freeing buffer of size 4144 at 00000000a3649daf
Sep  8 16:03:23 portatil264 kernel: [15033.016298] RPC:  3284 release request 0000000079df89b2
Sep  8 16:03:23 portatil264 kernel: [15033.016300] RPC:       wake_up_first(00000000c5ee49ee "xprt_backlog")
Sep  8 16:03:23 portatil264 kernel: [15033.016302] RPC:       rpc_release_client(00000000b930c343)
Sep  8 16:03:23 portatil264 kernel: [15033.016304] RPC:  3284 freeing task
Sep  8 16:03:23 portatil264 kernel: [15033.016309] _nfs4_proc_readdir: returns -5
Sep  8 16:03:23 portatil264 kernel: [15033.016318] NFS: readdir(departamentos/innovacion) returns -5
Quoted 7 lines
Quoted 4 lines

Hi, Bruce et al.

Is there anything we can do to help debugging/fixing this? It's still biting our users with a +4.20.x kernel.

Alberto, can you share a snippet of a raw network capture that shows the READDIR Reply that fails to decode?

Hi, Chuck.

Thanks for your reply. We're using "sec=krb5p", which makes the network capture useless :-(

Quoted 7 lines
Quoted 1 line

You can plug keytabs into Wireshark to enable it to decrypt the traffic.

Just skimming that range of history, there's some changes to the handling of gss sequence numbers, I wonder if there's a chance he could be hitting that? You had a workload that would lead to calls dropping out of the sequence number window, didn't you, Chuck? Is there a quick way to check whether that's happening?

Quoted 3 lines

The server is supposed to drop the connection when that happens, though I'm not sure 4.20's NFSD was perfect in that regard. Connection loss in itself wouldn't result in EIO.

In case this is relevant, server is running 3.16.0. Clients (with issues) +4.20.

Signature · 2 lines
Alberto González Iniesta             | Universidad a Distancia
[email protected]            | de Madrid
#13 lore raw
Quoted 56 lines · Alberto Gonzalez Iniesta
Quoted 51 lines · Chuck Lever
Quoted 50 lines · Bruce Fields
Quoted 33 lines · Chuck Lever
Quoted 33 lines · Alberto Gonzalez Iniesta
Quoted 27 lines · Chuck Lever
Quoted 20 lines
Quoted 19 lines

So all I notice from this one is the readdir EIO came from call_decode. I suspect that means it failed in the xdr decoding. Looks like xdr decoding of the actual directory data (which is the complicated part) is done later, so this means it failed decoding the header or verifier, which is a little odd:

Quoted 13 lines
Sep  8 16:03:23 portatil264 kernel: [15033.016276] RPC:  3284 call_decode result -5
Sep  8 16:03:23 portatil264 kernel: [15033.016281] nfs41_sequence_process: Error 1 free the slot 
Sep  8 16:03:23 portatil264 kernel: [15033.016286] RPC:       wake_up_first(00000000d3f50f4d "ForeChannel Slot table")
Sep  8 16:03:23 portatil264 kernel: [15033.016288] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 16:03:23 portatil264 kernel: [15033.016290] RPC:  3284 return 0, status -5
Sep  8 16:03:23 portatil264 kernel: [15033.016291] RPC:  3284 release task
Sep  8 16:03:23 portatil264 kernel: [15033.016295] RPC:       freeing buffer of size 4144 at 00000000a3649daf
Sep  8 16:03:23 portatil264 kernel: [15033.016298] RPC:  3284 release request 0000000079df89b2
Sep  8 16:03:23 portatil264 kernel: [15033.016300] RPC:       wake_up_first(00000000c5ee49ee "xprt_backlog")
Sep  8 16:03:23 portatil264 kernel: [15033.016302] RPC:       rpc_release_client(00000000b930c343)
Sep  8 16:03:23 portatil264 kernel: [15033.016304] RPC:  3284 freeing task
Sep  8 16:03:23 portatil264 kernel: [15033.016309] _nfs4_proc_readdir: returns -5
Sep  8 16:03:23 portatil264 kernel: [15033.016318] NFS: readdir(departamentos/innovacion) returns -5
Quoted 4 lines

Hi, Bruce et al.

Is there anything we can do to help debugging/fixing this? It's still biting our users with a +4.20.x kernel.

Alberto, can you share a snippet of a raw network capture that shows the READDIR Reply that fails to decode?

Quoted 6 lines
Quoted 4 lines

Hi, Chuck.

Thanks for your reply. We're using "sec=krb5p", which makes the network capture useless :-(

You can plug keytabs into Wireshark to enable it to decrypt the traffic.

Just skimming that range of history, there's some changes to the handling of gss sequence numbers, I wonder if there's a chance he could be hitting that? You had a workload that would lead to calls dropping out of the sequence number window, didn't you, Chuck? Is there a quick way to check whether that's happening?

Quoted 6 lines
Quoted 3 lines

The server is supposed to drop the connection when that happens, though I'm not sure 4.20's NFSD was perfect in that regard. Connection loss in itself wouldn't result in EIO.

In case this is relevant, server is running 3.16.0. Clients (with issues) +4.20.

Ah, I see. Well that's an old kernel. Have you engaged your distributor? They might be able to provide builds with debugging instrumentation, for example, if we can give them some instructions or a patch.

My experience tells me that this is probably an issue with either the server's GSS wrap function, or the client's GSS unwrap function, if you don't ever see this failure without krb5p in the picture.

If you don't see this problem on older clients, than I would start looking at the client's GSS unwrap function, fwiw.

--
Chuck Lever

#14 lore raw
In reply to Alberto Gonzalez Iniesta
Quoted 49 lines · Alberto Gonzalez Iniesta
Quoted 26 lines · J. Bruce Fields
Quoted 25 lines · Alberto Gonzalez Iniesta
Quoted 24 lines · J. Bruce Fields
Quoted 23 lines · Alberto Gonzalez Iniesta

We can run the previous "ls -lR" 20 times and get no error, or get this "ls: leyendo el directorio 'Becas y ayudas/': Error de entrada/salida" (ls: reading directorio 'Becas y ayudas/': Input/Output Error") every now and then.

The error happens (obviously?) with ls, rsync and the users's GUI tools.

There's nothing in dmesg (or elsewhere).
These are the kernels with tried:

4.18.0-25   -> Can't reproduce
4.19.0      -> Can't reproduce
4.20.17     -> Happening (hard to reproduce)
5.0.0-15    -> Happening (hard to reproduce)
5.3.0-45    -> Happening (more frequently)
5.6.0-rc7   -> Reproduced a couple of times after boot, then nothing

We did long (as in daylong) testing trying to reproduce this with all those kernel versions, so we are pretty sure 4.18 and 4.19 don't experience this and our Ubuntu 16.04 clients don't have any issue.

I know we aren't providing much info but we are really looking forward to doing all the testing required (we already spent lots of time in it).

Quoted 21 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

Thanks for your work.

So all I notice from this one is the readdir EIO came from call_decode. I suspect that means it failed in the xdr decoding. Looks like xdr decoding of the actual directory data (which is the complicated part) is done later, so this means it failed decoding the header or verifier, which is a little odd:

Quoted 13 lines
Sep  8 16:03:23 portatil264 kernel: [15033.016276] RPC:  3284 call_decode result -5
Sep  8 16:03:23 portatil264 kernel: [15033.016281] nfs41_sequence_process: Error 1 free the slot 
Sep  8 16:03:23 portatil264 kernel: [15033.016286] RPC:       wake_up_first(00000000d3f50f4d "ForeChannel Slot table")
Sep  8 16:03:23 portatil264 kernel: [15033.016288] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
Sep  8 16:03:23 portatil264 kernel: [15033.016290] RPC:  3284 return 0, status -5
Sep  8 16:03:23 portatil264 kernel: [15033.016291] RPC:  3284 release task
Sep  8 16:03:23 portatil264 kernel: [15033.016295] RPC:       freeing buffer of size 4144 at 00000000a3649daf
Sep  8 16:03:23 portatil264 kernel: [15033.016298] RPC:  3284 release request 0000000079df89b2
Sep  8 16:03:23 portatil264 kernel: [15033.016300] RPC:       wake_up_first(00000000c5ee49ee "xprt_backlog")
Sep  8 16:03:23 portatil264 kernel: [15033.016302] RPC:       rpc_release_client(00000000b930c343)
Sep  8 16:03:23 portatil264 kernel: [15033.016304] RPC:  3284 freeing task
Sep  8 16:03:23 portatil264 kernel: [15033.016309] _nfs4_proc_readdir: returns -5
Sep  8 16:03:23 portatil264 kernel: [15033.016318] NFS: readdir(departamentos/innovacion) returns -5
Quoted 4 lines

Hi, Bruce et al.

Is there anything we can do to help debugging/fixing this? It's still biting our users with a +4.20.x kernel.

Sorry, I just don't know what this is off the top of my head. If I had the time, stuff I might try:

- look at the wire traffic with wireshark: try to figure out which operation this is happening on, and if there's anything unusual about the reply. May be difficult if a lot of traffic is required to reproduce.
- if you're using krb5, try without just to see if that makes a difference.
- look at history: gitk v4.19..v4.20.17 fs/nfs net/sunrpc, see if anything looks relevant; or even just try a git bisect (difficult again given the intermittent failure).
- trace through the code to work out where call_decode might be returning -EIO, try to narrow it down with tracepoints or even just debugging printk's.

--b.

#15 lore raw
Quoted 9 lines · J. Bruce Fields

Sorry, I just don't know what this is off the top of my head. If I had the time, stuff I might try:

- look at the wire traffic with wireshark: try to figure out which operation this is happening on, and if there's anything unusual about the reply. May be difficult if a lot of traffic is required to reproduce.
- if you're using krb5, try without just to see if that makes a difference.

Using "sec=krb5p" here... And with lots of active clients. :-/

Oh! Could I add another entry in /etc/exports with krb5 without afecting current krb5p clients?

Quoted 6 lines

- look at history: gitk v4.19..v4.20.17 fs/nfs net/sunrpc, see if anything looks relevant; or even just try a git bisect (difficult again given the intermittent failure).
- trace through the code to work out where call_decode might be returning -EIO, try to narrow it down with tracepoints or even just debugging printk's.

I'm afraid I don't have the skills to carry out those tasks, haven't coded in C for decades, not to mention kernel/nfs hacking. I guess we'll have to wait until someone else hits this too, or we update the server and (hopefully/miraculously) fixes it self :-)

Thanks a lot,

Alberto

Signature · 2 lines
Alberto González Iniesta             | Universidad a Distancia
[email protected]            | de Madrid