loop on file cat + file copy when files are on CIFS share
Hi Christoph, David
Sorry my last mail didn't arrive at the top of the list so I resend it with a new title
I don't know if it had already been reported but after building Linux 6.14-rc1 I constat the following behaviour:
'cat' command is going on a loop when I cat a file which reside on cifs share
And so 'cp' command does the same: it copy the content of a file on cifs share and loop writing it to the destination I did test with a file named 'toto' and containing only ascii string 'toto'.
When I started copying it from cifs share to local filesystem, I had to CTRL+C the copy of this 5 bytes file after some time because the destination file was using all the filesystem free space and containing billions of 'toto' lines
Here is an example with cat:
CIFS SHARE is mounted as /mnt/fbx/FBX-24T
CIFS mount options:
grep cifs /proc/mounts
//10.0.10.100/FBX24T /mnt/fbx/FBX-24T cifs rw,nosuid,nodev,noexec,relatime,vers=3.1.1,cache=none,upcall_target=app,username=fbx,domain=HOMELAN,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.10.100,file_mode=0666,dir_mode=0755,iocharset=utf8,soft,nounix,serverino,mapposix,mfsymlinks,reparse=nfs,nativesocket,symlink=mfsymlinks,rsize=65536,wsize=65536,bsize=16777216,retrans=1,echo_interval=60,actimeo=1,closetimeo=1 0 0
KERNEL: uname -a
Linux 14RV-SERVER.14rv.lan 6.14.0.1-ast-rc2-amd64 #0 SMP PREEMPT_DYNAMIC Wed Feb 12 18:23:00 CET 2025 x86_64 GNU/Linux
To be reproduced:
echo toto >/mnt/fbx/FBX-24T/toto
ls -l /mnt/fbx/FBX-24T/toto
-rw-rw-rw- 1 root root 5 20 mars 09:20 /mnt/fbx/FBX-24T/toto
cat /mnt/fbx/FBX-24T/toto
toto
toto
toto
toto
toto
toto
toto
^C
strace cat /mnt/fbx/FBX-24T/totoexecve("/usr/bin/cat", ["cat", "/mnt/fbx/FBX-24T/toto"], 0x7ffc39b41848 /* 19 vars */) = 0
brk(NULL) = 0x55755b1c1000mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f55f95d6000access("/etc/ld.so.preload", R_OK) = -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "glibc-hwcaps/x86-64-v3/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type) openat(AT_FDCWD, "glibc-hwcaps/x86-64-v2/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type) openat(AT_FDCWD, "tls/haswell/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type) openat(AT_FDCWD, "tls/haswell/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type) openat(AT_FDCWD, "tls/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type) openat(AT_FDCWD, "tls/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "haswell/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type) openat(AT_FDCWD, "haswell/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/glibc-hwcaps/x86-64-v3/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type) newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/glibc-hwcaps/x86-64-v3", 0x7fff25937800, 0) = -1 ENOENT (Aucun fichier ou dossier de ce type) openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/glibc-hwcaps/x86-64-v2/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type) newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/glibc-hwcaps/x86-64-v2", 0x7fff25937800, 0) = -1 ENOENT (Aucun fichier ou dossier de ce type) openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/tls/haswell/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type) newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/tls/haswell/x86_64", 0x7fff25937800, 0) = -1 ENOENT (Aucun fichier ou dossier de ce type) openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/tls/haswell/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type) newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/tls/haswell", 0x7fff25937800, 0) = -1 ENOENT (Aucun fichier ou dossier de ce type) openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/tls/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type) newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/tls/x86_64", 0x7fff25937800, 0) = -1 ENOENT (Aucun fichier ou dossier de ce type) openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/tls/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type) newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/tls", 0x7fff25937800, 0) = -1 ENOENT (Aucun fichier ou dossier de ce type) openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/haswell/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type) newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/haswell/x86_64", 0x7fff25937800, 0) = -1 ENOENT (Aucun fichier ou dossier de ce type) openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/haswell/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type) newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/haswell", 0x7fff25937800, 0) = -1 ENOENT (Aucun fichier ou dossier de ce type) openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type) newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/x86_64", 0x7fff25937800, 0) = -1 ENOENT (Aucun fichier ou dossier de ce type) openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Aucun fichier ou dossier de ce type) newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64", {st_mode=S_IFDIR|S_ISGID|0755, st_size=4570, ...}, 0) = 0openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=148466, ...}, AT_EMPTY_PATH) = 0mmap(NULL, 148466, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f55f95b1000
close(3) = 0openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20t\2\0\0\0\0\0"..., 832) = 832pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=1922136, ...}, AT_EMPTY_PATH) = 0pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784mmap(NULL, 1970000, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f55f93d0000mmap(0x7f55f93f6000, 1396736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f55f93f6000
mmap(0x7f55f954b000, 339968, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17b000) = 0x7f55f954b000 mmap(0x7f55f959e000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1ce000) = 0x7f55f959e000 mmap(0x7f55f95a4000, 53072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f55f95a4000
close(3) = 0mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f55f93cd000 arch_prctl(ARCH_SET_FS, 0x7f55f93cd740) = 0set_tid_address(0x7f55f93cda10) = 38427set_robust_list(0x7f55f93cda20, 24) = 0rseq(0x7f55f93ce060, 0x20, 0, 0x53053053) = 0mprotect(0x7f55f959e000, 16384, PROT_READ) = 0mprotect(0x55754475e000, 4096, PROT_READ) = 0mprotect(0x7f55f960e000, 8192, PROT_READ) = 0prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0munmap(0x7f55f95b1000, 148466) = 0getrandom("\x19\x6b\x9e\x55\x7e\x09\x74\x5f", 8, GRND_NONBLOCK) = 8
brk(NULL) = 0x55755b1c1000
brk(0x55755b1e2000) = 0x55755b1e2000openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=3048928, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 3048928, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f55f9000000
close(3) = 0newfstatat(1, "", {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}, AT_EMPTY_PATH) = 0 openat(AT_FDCWD, "/mnt/fbx/FBX-24T/toto", O_RDONLY) = 3newfstatat(3, "", {st_mode=S_IFREG|0666, st_size=5, ...}, AT_EMPTY_PATH) = 0 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0mmap(NULL, 16785408, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f55f7ffe000 read(3, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16777216) = 16711680 write(1, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16711680toto ) = 16711680read(3, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16777216) = 16711680 write(1, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16711680toto ) = 16711680read(3, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16777216) = 16711680 write(1, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16711680toto ) = 16711680read(3, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16777216) = 16711680 write(1, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16711680toto ) = 16711680read(3, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16777216) = 16711680 write(1, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16711680toto ) = 16711680read(3, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16777216) = 16711680 write(1, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16711680toto ) = 16711680read(3, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16777216) = 16711680 write(1, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16711680toto ^Cstrace: Process 38427 detached
<detached ...>
Please let me know if it had already been fixed or reported and if you're able to reproduce this issue.
Thanks for help
Kind regards
Nicolas Baranger
Dear maintener
I get no answer from linux-cifs and netfs list so I'm sending this new mail Today I've just download and build mainline kernel: Linux 6.14.0 (no -rc)
I still constat the bug describe in my previous mail. What can I do to be able to use CIFS share ?
Thanks for help,
Kind regard
Nicolas Baranger
Quoted 143 lines · Nicolas Baranger
Hi Christoph, David
Sorry my last mail didn't arrive at the top of the list so I resend it with a new title
I don't know if it had already been reported but after building Linux 6.14-rc1 I constat the following behaviour:
'cat' command is going on a loop when I cat a file which reside on cifs share
And so 'cp' command does the same: it copy the content of a file on cifs share and loop writing it to the destination I did test with a file named 'toto' and containing only ascii string 'toto'.
When I started copying it from cifs share to local filesystem, I had to CTRL+C the copy of this 5 bytes file after some time because the destination file was using all the filesystem free space and containing billions of 'toto' lines
Here is an example with cat:
CIFS SHARE is mounted as
/mnt/fbx/FBX-24TCIFS mount options:
grep cifs/proc/mounts
//10.0.10.100/FBX24T/mnt/fbx/FBX-24Tcifs rw,nosuid,nodev,noexec,relatime,vers=3.1.1,cache=none,upcall_target=app,username=fbx,domain=HOMELAN,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.10.100,file_mode=0666,dir_mode=0755,iocharset=utf8,soft,nounix,serverino,mapposix,mfsymlinks,reparse=nfs,nativesocket,symlink=mfsymlinks,rsize=65536,wsize=65536,bsize=16777216,retrans=1,echo_interval=60,actimeo=1,closetimeo=1 0 0KERNEL: uname -a
Linux 14RV-SERVER.14rv.lan 6.14.0.1-ast-rc2-amd64 #0 SMPPREEMPT_DYNAMICWed Feb 12 18:23:00 CET 2025x86_64GNU/LinuxTo be reproduced:
echo toto >/mnt/fbx/FBX-24T/totols -l
/mnt/fbx/FBX-24T/toto
-rw-rw-rw- 1 root root 5 20 mars 09:20/mnt/fbx/FBX-24T/totocat
/mnt/fbx/FBX-24T/toto
toto
toto
toto
toto
toto
toto
toto
^Cstrace cat
/mnt/fbx/FBX-24T/totoexecve("/usr/bin/cat", ["cat", "/mnt/fbx/FBX-24T/toto"], 0x7ffc39b41848 /* 19 vars */)= 0
brk(NULL) =0x55755b1c1000mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)=0x7f55f95d6000access("/etc/ld.so.preload", R_OK)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "glibc-hwcaps/x86-64-v3/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "glibc-hwcaps/x86-64-v2/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "tls/haswell/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC)=
-1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "tls/haswell/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "tls/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "tls/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "haswell/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "haswell/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "x86_64/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/glibc-hwcaps/x86-64-v3/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/glibc-hwcaps/x86-64-v3", 0x7fff25937800, 0)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/glibc-hwcaps/x86-64-v2/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/glibc-hwcaps/x86-64-v2", 0x7fff25937800, 0)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/tls/haswell/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/tls/haswell/x86_64", 0x7fff25937800, 0)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/tls/haswell/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/tls/haswell", 0x7fff25937800, 0)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/tls/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/tls/x86_64", 0x7fff25937800, 0)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/tls/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/tls", 0x7fff25937800, 0)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/haswell/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/haswell/x86_64", 0x7fff25937800, 0)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/haswell/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/haswell", 0x7fff25937800, 0)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/x86_64", 0x7fff25937800, 0)= -1 ENOENT (Aucun fichier ou dossier de ce type)openat(AT_FDCWD, "/usr/local/cuda-12.6/lib64/libc.so.6", O_RDONLY|O_CLOEXEC)= -1 ENOENT (Aucun fichier ou dossier de ce type)newfstatat(AT_FDCWD, "/usr/local/cuda-12.6/lib64", {st_mode=S_IFDIR|S_ISGID|0755, st_size=4570, ...}, 0)= 0openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC)= 3newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=148466, ...}, AT_EMPTY_PATH)= 0mmap(NULL, 148466, PROT_READ, MAP_PRIVATE, 3, 0)=0x7f55f95b1000
close(3) = 0openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC)= 3read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20t\2\0\0\0\0\0"..., 832)= 832pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64)= 784newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=1922136, ...}, AT_EMPTY_PATH)= 0pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64)= 784mmap(NULL, 1970000, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0)=0x7f55f93d0000mmap(0x7f55f93f6000, 1396736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000)=0x7f55f93f6000mmap(0x7f55f954b000, 339968, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17b000) = 0x7f55f954b000 mmap(0x7f55f959e000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1ce000) = 0x7f55f959e000 mmap(0x7f55f95a4000, 53072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f55f95a4000close(3) = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)=0x7f55f93cd000arch_prctl(ARCH_SET_FS, 0x7f55f93cd740)= 0set_tid_address(0x7f55f93cda10)= 38427set_robust_list(0x7f55f93cda20, 24)= 0rseq(0x7f55f93ce060, 0x20, 0, 0x53053053)= 0mprotect(0x7f55f959e000, 16384, PROT_READ)= 0mprotect(0x55754475e000, 4096, PROT_READ)= 0mprotect(0x7f55f960e000, 8192, PROT_READ)= 0prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY})= 0munmap(0x7f55f95b1000, 148466)= 0getrandom("\x19\x6b\x9e\x55\x7e\x09\x74\x5f", 8, GRND_NONBLOCK)= 8
brk(NULL) =0x55755b1c1000
brk(0x55755b1e2000) =0x55755b1e2000openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC)= 3newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=3048928, ...}, AT_EMPTY_PATH)= 0mmap(NULL, 3048928, PROT_READ, MAP_PRIVATE, 3, 0)=0x7f55f9000000
close(3) = 0newfstatat(1, "", {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}, AT_EMPTY_PATH)= 0openat(AT_FDCWD, "/mnt/fbx/FBX-24T/toto", O_RDONLY)= 3newfstatat(3, "", {st_mode=S_IFREG|0666, st_size=5, ...}, AT_EMPTY_PATH)= 0fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL)= 0
mmap(NULL, 16785408, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) =0x7f55f7ffe000read(3, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16777216)= 16711680write(1, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16711680toto )= 16711680read(3, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16777216)= 16711680write(1, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16711680toto )= 16711680read(3, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16777216)= 16711680write(1, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16711680toto )= 16711680read(3, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16777216)= 16711680write(1, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16711680toto )= 16711680read(3, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16777216)= 16711680write(1, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16711680toto )= 16711680read(3, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16777216)= 16711680write(1, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16711680toto )= 16711680read(3, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16777216)= 16711680 write(1, "toto\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16711680toto ^Cstrace: Process 38427 detached
<detached ...>Please let me know if it had already been fixed or reported and if you're able to reproduce this issue.
Thanks for help
Kind regards
Nicolas Baranger
Hi Nicolas,
please wait a bit, many file system developers where at a conference this week.
Hi Christoph
Thanks for answer and help
Did someone reproduced the issue (very easy) ?
CIFS SHARE is mounted as /mnt/fbx/FBX-24T
echo toto >/mnt/fbx/FBX-24T/toto
ls -l /mnt/fbx/FBX-24T/toto
-rw-rw-rw- 1 root root 5 20 mars 09:20 /mnt/fbx/FBX-24T/toto
cat /mnt/fbx/FBX-24T/toto
toto
toto
toto
toto
toto
toto
toto
^C
CIFS mount options:
grep cifs /proc/mounts
//10.0.10.100/FBX24T /mnt/fbx/FBX-24T cifs rw,nosuid,nodev,noexec,relatime,vers=3.1.1,cache=none,upcall_target=app,username=fbx,domain=HOMELAN,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.10.100,file_mode=0666,dir_mode=0755,iocharset=utf8,soft,nounix,serverino,mapposix,mfsymlinks,reparse=nfs,nativesocket,symlink=mfsymlinks,rsize=65536,wsize=65536,bsize=16777216,retrans=1,echo_interval=60,actimeo=1,closetimeo=1 0 0
KERNEL: uname -a
Linux 14RV-SERVER.14rv.lan 6.14.0-rc2-amd64 #0 SMP PREEMPT_DYNAMIC Wed Feb 12 18:23:00 CET 2025 x86_64 GNU/Linux
Kind regards
Nicolas Baranger
Quoted 4 lines · Christoph Hellwig
Hi Nicolas,
please wait a bit, many file system developers where at a conference this week.
Hi Nicolas,
I'll look into it as soon as I recover from my illness. Sorry for the delay.
Quoted 42 lines · Nicolas Baranger
Hi Christoph
Thanks for answer and help
Did someone reproduced the issue (very easy) ?CIFS SHARE is mounted as
/mnt/fbx/FBX-24T
echo toto >/mnt/fbx/FBX-24T/totols -l
/mnt/fbx/FBX-24T/toto
-rw-rw-rw- 1 root root 5 20 mars 09:20/mnt/fbx/FBX-24T/totocat
/mnt/fbx/FBX-24T/toto
toto
toto
toto
toto
toto
toto
toto
^CCIFS mount options:
grep cifs/proc/mounts
//10.0.10.100/FBX24T/mnt/fbx/FBX-24Tcifs rw,nosuid,nodev,noexec,relatime,vers=3.1.1,cache=none,upcall_target=app,username=fbx,domain=HOMELAN,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.10.100,file_mode=0666,dir_mode=0755,iocharset=utf8,soft,nounix,serverino,mapposix,mfsymlinks,reparse=nfs,nativesocket,symlink=mfsymlinks,rsize=65536,wsize=65536,bsize=16777216,retrans=1,echo_interval=60,actimeo=1,closetimeo=1 0 0KERNEL: uname -a
Linux 14RV-SERVER.14rv.lan 6.14.0-rc2-amd64 #0 SMPPREEMPT_DYNAMICWed Feb 12 18:23:00 CET 2025x86_64GNU/LinuxKind regards
Nicolas BarangerQuoted 4 lines · Christoph Hellwig
Hi Nicolas,
please wait a bit, many file system developers where at a conference this week.
Hi Paulo
Thanks for answer and help
Quoted 1 line
I'll look into it as soon as I recover from my illness.
Hope you're doing better
I had to rollback to linux 6.13.8 to be able to use the SMB share and here is what I constat (don't know if it's a normal behavior but if yes, SMB seems to be a very very unefficient protocol)
I think the issue can be buffer related:
On Linux 6.13.8 the copy and cat of the 5 bytes 'toto' file containing only ascii string 'toto' is working fine but here is what I capture with tcpdump during transfert of toto file:
https://xba.soartist.net/t6.pcap
131 tcp packets to transfer a 5 byte file...
Isn't there a problem ?
Openning the pcap file with wireshark show a lot of lines:
25 0.005576 10.0.10.100 10.0.10.25 SMB2 1071 Read Response, Error: STATUS_END_OF_FILE
It seems that those lines appears after the 5 bytes 'toto' file had been transferred, and it continue until the last ACK recieved
I will try to reboot on Linux 6.14.0 mainline to see if I have the same behavior or to see what I get in the packet capture (system is in production, I cannot reboot on a failing kernel when I want, it should be organised... sorry)
Let me know if you reproduced the issue
Kind regards
Nicolas Baranger
Quoted 41 lines · Paulo Alcantara
Hi Nicolas,
I'll look into it as soon as I recover from my illness. Sorry for the delay.
On 4 April 2025 08:50:27 UTC, Nicolas Baranger <[email protected]> wrote: Hi Christoph
Thanks for answer and help
Did someone reproduced the issue (very easy) ?CIFS SHARE is mounted as
/mnt/fbx/FBX-24T
echo toto >/mnt/fbx/FBX-24T/totols -l
/mnt/fbx/FBX-24T/toto
-rw-rw-rw- 1 root root 5 20 mars 09:20/mnt/fbx/FBX-24T/totocat
/mnt/fbx/FBX-24T/toto
toto
toto
toto
toto
toto
toto
toto
^CCIFS mount options:
grep cifs/proc/mounts
//10.0.10.100/FBX24T/mnt/fbx/FBX-24Tcifs rw,nosuid,nodev,noexec,relatime,vers=3.1.1,cache=none,upcall_target=app,username=fbx,domain=HOMELAN,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.10.100,file_mode=0666,dir_mode=0755,iocharset=utf8,soft,nounix,serverino,mapposix,mfsymlinks,reparse=nfs,nativesocket,symlink=mfsymlinks,rsize=65536,wsize=65536,bsize=16777216,retrans=1,echo_interval=60,actimeo=1,closetimeo=1 0 0KERNEL: uname -a
Linux 14RV-SERVER.14rv.lan 6.14.0-rc2-amd64 #0 SMPPREEMPT_DYNAMICWed Feb 12 18:23:00 CET 2025x86_64GNU/LinuxKind regards
Nicolas BarangerLe 2025-03-28 11:45, Christoph Hellwig a écrit :
Hi Nicolas,
please wait a bit, many file system developers where at a conference this week.
Hi Nicolas,
Sorry for the delay as I've got busy with some downstream work.
Quoted 2 lines · Nicolas Baranger
Quoted 1 line
I'll look into it as soon as I recover from my illness.
Hope you're doing better
I'm fully recovered now, thanks :-)
Quoted 5 lines
I had to rollback to linux 6.13.8 to be able to use the SMB share and here is what I constat
(don't know if it's a normal behavior but if yes, SMB seems to be a very very unefficient protocol)
Quoted 12 lines
I think the issue can be buffer related:
On Linux 6.13.8 the copy and cat of the 5 bytes 'toto' file containing only ascii string 'toto' is working fine but here is what I capture with tcpdump during transfert of toto file:
https://xba.soartist.net/t6.pcap
131 tcp packets to transfer a 5 byte file...
Isn't there a problem ?
Openning the pcap file with wireshark show a lot of lines:
25 0.005576 10.0.10.100 10.0.10.25 SMB2 1071 Read Response, Error:STATUS_END_OF_FILE
It seems that those lines appears after the 5 bytes 'toto' file had been transferred, and it continue until the last ACK recieved
Thanks for the trace. I was finally able to reproduce your issue and will provide you with a fix soon.
Hi Paulo
Resending this mail with content-type: text, sorry !
Thanks again for answer and help, it's good to hear you're back to health.
Quoted 1 line
Thanks for the trace. I was finally able to reproduce your issue and will provide you with a fix soon.
Perfect... And thanks !
If you need more traces or details on (both?) issues :
- 1) infinite loop issue during 'cat' or 'copy' since Linux 6.14.0
- 2) (don't know if it's related) the very high number of several bytes TCP packets transmitted in SMB transaction (more than a hundred) for a 5 bytes file transfert under Linux 6.13.8
Do not hesitate to ask, I would be happy to help.
Kind regards
Nicolas
Quoted 7 lines · Paulo Alcantara
Hi Nicolas,
Sorry for the delay as I've got busy with some downstream work.
Nicolas Baranger <[email protected]> writes:
I'll look into it as soon as I recover from my illness. Hope you're doing better
I'm fully recovered now, thanks :-)
Quoted 5 lines
I had to rollback to linux 6.13.8 to be able to use the SMB share and here is what I constat
(don't know if it's a normal behavior but if yes, SMB seems to be a very very unefficient protocol)
Quoted 12 lines
I think the issue can be buffer related:
On Linux 6.13.8 the copy and cat of the 5 bytes 'toto' file containing only ascii string 'toto' is working fine but here is what I capture with tcpdump during transfert of toto file:
https://xba.soartist.net/t6.pcap
131 tcp packets to transfer a 5 byte file...
Isn't there a problem ?
Openning the pcap file with wireshark show a lot of lines:
25 0.005576 10.0.10.100 10.0.10.25 SMB2 1071 Read Response, Error:STATUS_END_OF_FILE
It seems that those lines appears after the 5 bytes 'toto' file had been transferred, and it continue until the last ACK recieved
Thanks for the trace. I was finally able to reproduce your issue and will provide you with a fix soon.
Quoted 7 lines · Nicolas Baranger
If you need more traces or details on (both?) issues :
- 1) infinite loop issue during 'cat' or 'copy' since Linux 6.14.0
- 2) (don't know if it's related) the very high number of several bytes TCP packets transmitted in SMB transaction (more than a hundred) for a 5 bytes file transfert under Linux 6.13.8
According to your mount options and network traces, cat(1) is attempting to read 16M from 'toto' file, in which case netfslib will create 256 subrequests to handle 64K (rsize=65536) reads from 'toto' file.
The first 64K read at offset 0 succeeds and server returns 5 bytes, the client then sets NETFS_SREQ_HIT_EOF to indicate that this subrequest hit the EOF. The next subrequests will still be processed by netfslib and sent to the server, but they all fail with STATUS_END_OF_FILE.
So, the problem is with short DIO reads in netfslib that are not being handled correctly. It is returning a fixed number of bytes read to every read(2) call in your cat command, 16711680 bytes which is the offset of last subrequest. This will make cat(1) retry forever as netfslib is failing to return the correct number of bytes read, including EOF.
While testing a potential fix, I also found other problems with DIO in cifs.ko, so I'm working with Dave to get the proper fixes for both netfslib and cifs.ko.
I've noticed that you disabled caching with 'cache=none', is there any particular reason for that?
Have you also set rsize, wsize and bsize mount options? If so, why?
If you want to keep 'cache=none', then a possible workaround for you would be making rsize and wsize always greater than bsize. The default values (rsize=4194304,wsize=4194304,bsize=1048576) would do it.
Hi Paolo
Thanks for answer, all explanations and help
I'm happy you found those 2 bugs and starting to patch them.
Reading your answer, I want to remember that I already found a bug in cifs DIO starting from Linux 6.10 (when cifs statring to use netfs to do its IO) and it was fixed by David and Christoph full story here: https://lore.kernel.org/all/[email protected]/T/
Quoted 2 lines
I've noticed that you disabled caching with 'cache=none', is there any particular reason for that?
Yes, it's related with the precedent use case describes in the other bug:
For backuping servers, I've got some KSMBD cifs share on which there are some 4TB+ sparses files (back-files) which are LUKS + BTRFS formatted.
The cifs share is mounted on servers and each server mount its own back-file as a block device and make its backup inside this crypted disk file
Due to performance issues, it is required that the disk files are using 4KB block and are mounted in servers using losetup DIO option (+ 4K block size options)
When I use something else than 'cache=none', sometimes the BTRFS filesystem on the back file get corrupted and I also need to mount the BTRFS filesystem with 'space_cache=v2' to avoid filesystem corruption
Quoted 1 line
Have you also set rsize, wsize and bsize mount options? If so, why?
After a lot of testing, the mounts buffers values: rsize=65536, wsize=65536, bsize=16777216, are the one which provide the best performances with no corruptions on the back-file filesystem and with these options a ~2TB backup is possible in few hours during timeframe ~1 -> ~5 AM each night
For me it's important that kernel async DIO on netfs continue to work as it's used by all my production backup system (transfer speed ratio compared with and without DIO is between 10 to 25)
I will try the patch "[PATCH] netfs: Fix setting of transferred bytes with short DIO reads", thanks
Let me know if you need further explanations,
Kind regards
Nicolas Baranger
Quoted 39 lines · Paulo Alcantara
Quoted 8 lines · Nicolas Baranger
If you need more traces or details on (both?) issues :
- 1) infinite loop issue during 'cat' or 'copy' since Linux 6.14.0
- 2) (don't know if it's related) the very high number of several bytes
TCP packets transmitted in SMB transaction (more than a hundred) for a 5 bytes file transfert under Linux 6.13.8According to your mount options and network traces, cat(1) is attempting to read 16M from 'toto' file, in which case netfslib will create 256 subrequests to handle 64K (rsize=65536) reads from 'toto' file.
The first 64K read at offset 0 succeeds and server returns 5 bytes, the client then sets
NETFS_SREQ_HIT_EOFto indicate that this subrequest hit the EOF. The next subrequests will still be processed by netfslib and sent to the server, but they all fail withSTATUS_END_OF_FILE.So, the problem is with short DIO reads in netfslib that are not being handled correctly. It is returning a fixed number of bytes read to every read(2) call in your cat command, 16711680 bytes which is the offset of last subrequest. This will make cat(1) retry forever as netfslib is failing to return the correct number of bytes read, including EOF.
While testing a potential fix, I also found other problems with DIO in cifs.ko, so I'm working with Dave to get the proper fixes for both netfslib and cifs.ko.
I've noticed that you disabled caching with 'cache=none', is there any particular reason for that?
Have you also set rsize, wsize and bsize mount options? If so, why?
If you want to keep 'cache=none', then a possible workaround for you would be making rsize and wsize always greater than bsize. The default values (rsize=4194304,wsize=4194304,bsize=1048576) would do it.
Hi Paolo
Thanks again for help.
I'm sorry, I made a mistake in my answer yesterday:
Quoted 1 line
After a lot of testing, the mounts buffers values: rsize=65536, wsize=65536, bsize=16777216,...
The actual values in /etc/fstab are:
rsize=4194304,wsize=4194304,bsize=16777216
But negociated values in /proc/mounts are:
rsize=65536,wsize=65536,bsize=16777216
And don't know if it's related but I have:
grep -i maxbuf /proc/fs/cifs/DebugData
CIFSMaxBufSize: 16384
I've just force a manual 'mount -o remount' and now I have in /proc/mounts the good values (SMB version is 3.1.1). Where does this behavior comes from ?
After some search, it appears that when the CIFS share is mounted by systemd option x-systemd.automount (for example doing 'ls' in the mount point directory), negociated values are:
rsize=65536,wsize=65536,bsize=16777216
If I umount / remount manually, the negociated values are those defined in /etc/fstab !
Don't know if it's a normal behavior but it is a source of errors / mistake and makes troubleshooting performance issues harder
Kind regards
Nicolas
Quoted 69 lines · Nicolas Baranger
Hi Paolo
Thanks for answer, all explanations and help
I'm happy you found those 2 bugs and starting to patch them.
Reading your answer, I want to remember that I already found a bug in cifs DIO starting from Linux 6.10 (when cifs statring to use netfs to do its IO) and it was fixed by David and Christoph full story here: https://lore.kernel.org/all/[email protected]/T/Quoted 2 lines
I've noticed that you disabled caching with 'cache=none', is there any particular reason for that?
Yes, it's related with the precedent use case describes in the other bug:
For backuping servers, I've got some KSMBD cifs share on which there are some 4TB+ sparses files (back-files) which are LUKS + BTRFS formatted.
The cifs share is mounted on servers and each server mount its own back-file as a block device and make its backup inside this crypted disk file
Due to performance issues, it is required that the disk files are using 4KB block and are mounted in servers using losetup DIO option (+ 4K block size options)
When I use something else than 'cache=none', sometimes the BTRFS filesystem on the back file get corrupted and I also need to mount the BTRFS filesystem with 'space_cache=v2' to avoid filesystem corruptionQuoted 1 line
Have you also set rsize, wsize and bsize mount options? If so, why?
After a lot of testing, the mounts buffers values: rsize=65536, wsize=65536, bsize=16777216, are the one which provide the best performances with no corruptions on the back-file filesystem and with these options a ~2TB backup is possible in few hours during timeframe ~1 -> ~5 AM each night
For me it's important that kernel async DIO on netfs continue to work as it's used by all my production backup system (transfer speed ratio compared with and without DIO is between 10 to 25)
I will try the patch "[PATCH] netfs: Fix setting of transferred bytes with short DIO reads", thanks
Let me know if you need further explanations,
Kind regards
Nicolas BarangerLe 2025-04-22 01:45, Paulo Alcantara a écrit :
Nicolas Baranger <[email protected]> writes:
If you need more traces or details on (both?) issues :
- 1) infinite loop issue during 'cat' or 'copy' since Linux 6.14.0
- 2) (don't know if it's related) the very high number of several bytes TCP packets transmitted in SMB transaction (more than a hundred) for a 5 bytes file transfert under Linux 6.13.8
According to your mount options and network traces, cat(1) is attempting to read 16M from 'toto' file, in which case netfslib will create 256 subrequests to handle 64K (rsize=65536) reads from 'toto' file.The first 64K read at offset 0 succeeds and server returns 5 bytes, the client then sets
NETFS_SREQ_HIT_EOFto indicate that this subrequest hit the EOF. The next subrequests will still be processed by netfslib and sent to the server, but they all fail withSTATUS_END_OF_FILE.So, the problem is with short DIO reads in netfslib that are not being handled correctly. It is returning a fixed number of bytes read to every read(2) call in your cat command, 16711680 bytes which is the offset of last subrequest. This will make cat(1) retry forever as netfslib is failing to return the correct number of bytes read, including EOF.
While testing a potential fix, I also found other problems with DIO in cifs.ko, so I'm working with Dave to get the proper fixes for both netfslib and cifs.ko.
I've noticed that you disabled caching with 'cache=none', is there any particular reason for that?
Have you also set rsize, wsize and bsize mount options? If so, why?
If you want to keep 'cache=none', then a possible workaround for you would be making rsize and wsize always greater than bsize. The default values (rsize=4194304,wsize=4194304,bsize=1048576) would do it.
[Resending mail in plain text version, sorry !]
Hi Paolo
Thanks again for help and sorry for this new mail but I think it could be relevant
In fact, I think there is somethings wrong:
After a remount, I sucessfully get the good buffers size values in /proc/mounts (those defined in /etc/fstab).
grep cifs /proc/mounts
//10.0.10.100/FBX24T /mnt/fbx/FBX-24T cifs rw,nosuid,nodev,noexec,relatime,vers=3.1.1,cache=none,upcall_target=app,username=*****,domain=*****,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.10.100,file_mode=0666,dir_mode=0755,iocharset=utf8,soft,nounix,serverino,mapposix,mfsymlinks,reparse=nfs,rsize=4194304,wsize=4194304,bsize=16777216,retrans=1,echo_interval=60,actimeo=1,closetimeo=1 0 0
uname -r
6.13.8.1-ast-nba0-amd64
But here is what I constat: a 'dd' with a block size smaller than 65536 is working fine:
LANG=en_US.UTF-8
dd if=/dev/urandom of=/mnt/fbx/FBX-24T/dd.test3 bs=65536 status=progress conv=notrunc oflag=direct count=128 128+0 records in
128+0 records out
8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.100398 s, 83.6 MB/s
But a 'dd' with a block size bigger than 65536 is not working:
LANG=en_US.UTF-8
dd if=/dev/urandom of=/mnt/fbx/FBX-24T/dd.test3 bs=65537 status=progress conv=notrunc oflag=direct count=128
dd: error writing '/mnt/fbx/FBX-24T/dd.test3'
dd: closing output file '/mnt/fbx/FBX-24T/dd.test3': Invalid argument
And kernel report:
Apr 24 10:01:37 14RV-SERVER.14rv.lan kernel: CIFS: VFS: \\10.0.10.100 Error -32 sending data on socket to server
If I let systemd option x-systemd.automount mount the share it configure /proc/mount with rsize=65536,wsize=65536 and I'm able to send datas whatever is the size of each packet of datas in the transfer stream. Example:
grep cifs /proc/mounts
//10.0.10.100/FBX24T /mnt/fbx/FBX-24T cifs rw,nosuid,nodev,noexec,relatime,vers=3.1.1,cache=none,upcall_target=app,username=*****,domain=*****,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.10.100,file_mode=0666,dir_mode=0755,iocharset=utf8,soft,nounix,serverino,mapposix,mfsymlinks,reparse=nfs,rsize=65536,wsize=65536,bsize=16777216,retrans=1,echo_interval=60,actimeo=1,closetimeo=1 0 0
dd if=/dev/urandom of=/mnt/fbx/FBX-24T/dd.test3 bs=64M status=progress conv=notrunc oflag=direct count=128 8589934592 bytes (8.6 GB, 8.0 GiB) copied, 42 s, 203 MB/s
128+0 records in
128+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 42.2399 s, 203 MB/s
To conclude, if I force an fstab value bigger than 65536 to be concidered and used (visible in /proc/mounts), transfer failed if I don't stream the transfer in packets of maximum 65536 bytes and if I let systemd configure rsize and wsize at 65536, I can stream the transfer in blocks of all size and specially of bigger size (*1024 in the example)
Let me know if you need further testing
Kind regards
Nicolas
Quoted 99 lines · Nicolas Baranger
Hi Paolo
Thanks again for help.
I'm sorry, I made a mistake in my answer yesterday:
Quoted 1 line
After a lot of testing, the mounts buffers values: rsize=65536, wsize=65536, bsize=16777216,...
The actual values in
/etc/fstabare:
rsize=4194304,wsize=4194304,bsize=16777216But negociated values in
/proc/mountsare:
rsize=65536,wsize=65536,bsize=16777216And don't know if it's related but I have:
grep -i maxbuf/proc/fs/cifs/DebugData
CIFSMaxBufSize: 16384I've just force a manual 'mount -o remount' and now I have in
/proc/mountsthe good values (SMB version is 3.1.1). Where does this behavior comes from ?After some search, it appears that when the CIFS share is mounted by systemd option x-systemd.automount (for example doing 'ls' in the mount point directory), negociated values are:
rsize=65536,wsize=65536,bsize=16777216
If I umount / remount manually, the negociated values are those defined in/etc/fstab!Don't know if it's a normal behavior but it is a source of errors / mistake and makes troubleshooting performance issues harder
Kind regards
NicolasLe 2025-04-23 18:28, Nicolas Baranger a écrit :
Hi Paolo
Thanks for answer, all explanations and help
I'm happy you found those 2 bugs and starting to patch them.
Reading your answer, I want to remember that I already found a bug in cifs DIO starting from Linux 6.10 (when cifs statring to use netfs to do its IO) and it was fixed by David and Christoph full story here: https://lore.kernel.org/all/[email protected]/T/I've noticed that you disabled caching with 'cache=none', is there any
particular reason for that?
Yes, it's related with the precedent use case describes in the other bug:
For backuping servers, I've got some KSMBD cifs share on which there are some 4TB+ sparses files (back-files) which are LUKS + BTRFS formatted.
The cifs share is mounted on servers and each server mount its own back-file as a block device and make its backup inside this crypted disk file
Due to performance issues, it is required that the disk files are using 4KB block and are mounted in servers using losetup DIO option (+ 4K block size options)
When I use something else than 'cache=none', sometimes the BTRFS filesystem on the back file get corrupted and I also need to mount the BTRFS filesystem with 'space_cache=v2' to avoid filesystem corruptionHave you also set rsize, wsize and bsize mount options? If so, why?
After a lot of testing, the mounts buffers values: rsize=65536, wsize=65536, bsize=16777216, are the one which provide the best performances with no corruptions on the back-file filesystem and with these options a ~2TB backup is possible in few hours during timeframe ~1 -> ~5 AM each nightFor me it's important that kernel async DIO on netfs continue to work as it's used by all my production backup system (transfer speed ratio compared with and without DIO is between 10 to 25)
I will try the patch "[PATCH] netfs: Fix setting of transferred bytes with short DIO reads", thanks
Let me know if you need further explanations,
Kind regards
Nicolas BarangerLe 2025-04-22 01:45, Paulo Alcantara a écrit :
Nicolas Baranger <[email protected]> writes:
If you need more traces or details on (both?) issues :
- 1) infinite loop issue during 'cat' or 'copy' since Linux 6.14.0
- 2) (don't know if it's related) the very high number of several bytes TCP packets transmitted in SMB transaction (more than a hundred) for a 5 bytes file transfert under Linux 6.13.8
According to your mount options and network traces, cat(1) is attempting to read 16M from 'toto' file, in which case netfslib will create 256 subrequests to handle 64K (rsize=65536) reads from 'toto' file.The first 64K read at offset 0 succeeds and server returns 5 bytes, the client then sets
NETFS_SREQ_HIT_EOFto indicate that this subrequest hit the EOF. The next subrequests will still be processed by netfslib and sent to the server, but they all fail withSTATUS_END_OF_FILE.So, the problem is with short DIO reads in netfslib that are not being handled correctly. It is returning a fixed number of bytes read to every read(2) call in your cat command, 16711680 bytes which is the offset of last subrequest. This will make cat(1) retry forever as netfslib is failing to return the correct number of bytes read, including EOF.
While testing a potential fix, I also found other problems with DIO in cifs.ko, so I'm working with Dave to get the proper fixes for both netfslib and cifs.ko.
I've noticed that you disabled caching with 'cache=none', is there any particular reason for that?
Have you also set rsize, wsize and bsize mount options? If so, why?
If you want to keep 'cache=none', then a possible workaround for you would be making rsize and wsize always greater than bsize. The default values (rsize=4194304,wsize=4194304,bsize=1048576) would do it.
Hi Nicolas,
Thanks for the very detailed information and testing.
Quoted 9 lines · Nicolas Baranger
In fact, I think there is somethings wrong:
After a remount, I sucessfully get the good buffers size values in
/proc/mounts(those defined in/etc/fstab).grep cifs
/proc/mounts
//10.0.10.100/FBX24T/mnt/fbx/FBX-24Tcifs rw,nosuid,nodev,noexec,relatime,vers=3.1.1,cache=none,upcall_target=app,username=*****,domain=*****,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.10.100,file_mode=0666,dir_mode=0755,iocharset=utf8,soft,nounix,serverino,mapposix,mfsymlinks,reparse=nfs,rsize=4194304,wsize=4194304,bsize=16777216,retrans=1,echo_interval=60,actimeo=1,closetimeo=1 0 0
Interesting. When you do 'mount -o remount ...' but don't pass rsize= and wsize=, the client is suppposed to reuse the existing values of rsize and wsize set in the current superblock. The above values of rsize, wsize and bsize are also the default ones in case you don't pass them at all.
I'll look into that when time allows it.
Quoted 15 lines
But here is what I constat: a 'dd' with a block size smaller than 65536 is working fine:
LANG=en_US.UTF-8dd if=/dev/urandom of=/mnt/fbx/FBX-24T/dd.test3 bs=65536 status=progress conv=notrunc oflag=direct count=128
128+0 records in
128+0 records out
8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.100398 s, 83.6 MB/sBut a 'dd' with a block size bigger than 65536 is not working:
LANG=en_US.UTF-8
Quoted 8 lines
dd if=/dev/urandom of=/mnt/fbx/FBX-24T/dd.test3 bs=65537 status=progress conv=notrunc oflag=direct count=128
dd: error writing '/mnt/fbx/FBX-24T/dd.test3'
dd: closing output file '/mnt/fbx/FBX-24T/dd.test3': Invalid argumentAnd kernel report:
Apr 24 10:01:37 14RV-SERVER.14rv.lan kernel: CIFS: VFS: \\10.0.10.100Error -32 sending data on socket to server
This seems related to unaligned DIO reads and writes. With O_DIRECT, the client will set FILE_NO_INTERMEDIATE_BUFFERING when opening the file, telling the server to not do any buffering when reading from or writing to the file. Some servers will fail the read or write request if the file offset or length isn't a multiple of block size, where the block size is >= 512 && <= PAGE_SIZE, as specified in MS-FSA 2.1.5.[34].
Since you're passing bs= with a value that is not multiple of block size, then the server is failing the request with STATUS_INVALID_PARAMETER as specified in MS-FSA.
I've tested it against Windows Server 2022 and it seems to enforce the alignment only for DIO reads. While samba doesn't enforce it at all.
win2k22:
$ dd if=/mnt/1/foo of=/dev/null status=none iflag=direct count=128 bs=65536 $ dd if=/mnt/1/foo of=/dev/null status=none iflag=direct count=128 bs=65537 dd: error reading '/mnt/1/foo': Invalid argument $ dd if=/mnt/1/foo of=/dev/null status=none iflag=direct count=128 bs=$((65536+512))
$ xfs_io -d -f -c "pread 0 4096" /mnt/1/foo read 4096/4096 bytes at offset 0 4 KiB, 1 ops; 0.0009 sec (4.260 MiB/sec and 1090.5125 ops/sec) $ xfs_io -d -f -c "pread 1 4096" /mnt/1/foo pread: Invalid argument
samba:
$ dd if=/mnt/1/foo of=/dev/null status=none iflag=direct count=128 bs=65536 $ dd if=/mnt/1/foo of=/dev/null status=none iflag=direct count=128 bs=65537 $ dd if=/mnt/1/foo of=/dev/null status=none iflag=direct count=128 bs=$((65536+512))
$ xfs_io -d -f -c "pread 0 4096" /mnt/1/foo read 4096/4096 bytes at offset 0 4 KiB, 1 ops; 0.0071 sec (557.880 KiB/sec and 139.4700 ops/sec) $ xfs_io -d -f -c "pread 1 4096" /mnt/1/foo read 4096/4096 bytes at offset 1 4 KiB, 1 ops; 0.0010 sec (3.864 MiB/sec and 989.1197 ops/sec)
Note that the netfslib fix is for short DIO reads, so this bug is related to unaligned DIO reads and writes and need to be fixed in the client. I'll let you know when I have patches for that.
Hi Nicolas,
Could you try my cifs.dio branch [1] which contains the following fixes
afea8b581c75 ("netfs: Fix wait/wake to be consistent about the waitqueue used") ae9f3deaa17a ("netfs: Fix the request's work item to not require a ref") b2a47dc3ead6 ("netfs: Fix setting of transferred bytes with short DIO reads") c59f7c9661b9 ("smb: client: ensure aligned IO sizes")
Let me know if you find any issues with it. Thanks.
Hi Paulo
I'm testing this branch and going back with results
Thanks again !
Nicolas
Quoted 12 lines · Paulo Alcantara
Hi Nicolas,
Could you try my cifs.dio branch [1] which contains the following fixes
afea8b581c75("netfs: Fix wait/wake to be consistent about the waitqueue used")ae9f3deaa17a("netfs: Fix the request's work item to not require a ref")b2a47dc3ead6("netfs: Fix setting of transferred bytes with short DIO reads")c59f7c9661b9("smb: client: ensure aligned IO sizes")Let me know if you find any issues with it. Thanks.
Quoted 6 lines
when the CIFS share is mounted by
systemd option x-systemd.automount (for example doing 'ls' in the mount point directory), negociated values are:
rsize=65536,wsize=65536,bsize=16777216
If I umount / remount manually, the negociated values are those defined in/etc/fstab!
That does seems broken (and obviously can hurt performance significantly as most servers negotiate an rsize and wsize of at least 4MB).
It looks like it can be overridden by creating the file to configure the smb3 systemd automounts in /etc/systemd/system but it is odd that it overrides the default that would be used for normal mounts or /etc/fstab automounts
Quoted 132 lines · Nicolas Baranger
Hi Paolo
Thanks again for help.
I'm sorry, I made a mistake in my answer yesterday:
Quoted 2 lines
After a lot of testing, the mounts buffers values: rsize=65536, wsize=65536, bsize=16777216,...
The actual values in
/etc/fstabare:
rsize=4194304,wsize=4194304,bsize=16777216But negociated values in
/proc/mountsare:
rsize=65536,wsize=65536,bsize=16777216And don't know if it's related but I have:
grep -i maxbuf/proc/fs/cifs/DebugData
CIFSMaxBufSize: 16384I've just force a manual 'mount -o remount' and now I have in
/proc/mountsthe good values (SMB version is 3.1.1). Where does this behavior comes from ?After some search, it appears that when the CIFS share is mounted by systemd option x-systemd.automount (for example doing 'ls' in the mount point directory), negociated values are:
rsize=65536,wsize=65536,bsize=16777216
If I umount / remount manually, the negociated values are those defined in/etc/fstab!Don't know if it's a normal behavior but it is a source of errors / mistake and makes troubleshooting performance issues harder
Kind regards
NicolasQuoted 91 lines · Nicolas Baranger
Hi Paolo
Thanks for answer, all explanations and help
I'm happy you found those 2 bugs and starting to patch them. Reading your answer, I want to remember that I already found a bug in cifs DIO starting from Linux 6.10 (when cifs statring to use netfs to do its IO) and it was fixed by David and Christoph
full story here:
https://lore.kernel.org/all/[email protected]/T/Quoted 2 lines
I've noticed that you disabled caching with 'cache=none', is there any particular reason for that?
Yes, it's related with the precedent use case describes in the other bug:
For backuping servers, I've got some KSMBD cifs share on which there are some 4TB+ sparses files (back-files) which are LUKS + BTRFS formatted.
The cifs share is mounted on servers and each server mount its own back-file as a block device and make its backup inside this crypted disk file
Due to performance issues, it is required that the disk files are using 4KB block and are mounted in servers using losetup DIO option (+ 4K block size options)
When I use something else than 'cache=none', sometimes the BTRFS filesystem on the back file get corrupted and I also need to mount the BTRFS filesystem with 'space_cache=v2' to avoid filesystem corruptionQuoted 1 line
Have you also set rsize, wsize and bsize mount options? If so, why?
After a lot of testing, the mounts buffers values: rsize=65536, wsize=65536, bsize=16777216, are the one which provide the best performances with no corruptions on the back-file filesystem and with these options a ~2TB backup is possible in few hours during timeframe ~1 -> ~5 AM each night
For me it's important that kernel async DIO on netfs continue to work as it's used by all my production backup system (transfer speed ratio compared with and without DIO is between 10 to 25)
I will try the patch "[PATCH] netfs: Fix setting of transferred bytes with short DIO reads", thanks
Let me know if you need further explanations,
Kind regards
Nicolas BarangerLe 2025-04-22 01:45, Paulo Alcantara a écrit :
Nicolas Baranger <[email protected]> writes:
If you need more traces or details on (both?) issues :
- 1) infinite loop issue during 'cat' or 'copy' since Linux 6.14.0
- 2) (don't know if it's related) the very high number of several bytes TCP packets transmitted in SMB transaction (more than a hundred) for a 5
bytes file transfert under Linux 6.13.8
According to your mount options and network traces, cat(1) is attempting
to read 16M from 'toto' file, in which case netfslib will create 256 subrequests to handle 64K (rsize=65536) reads from 'toto' file.The first 64K read at offset 0 succeeds and server returns 5 bytes, the client then sets
NETFS_SREQ_HIT_EOFto indicate that this subrequest hit
the EOF. The next subrequests will still be processed by netfslib and sent to the server, but they all fail withSTATUS_END_OF_FILE.So, the problem is with short DIO reads in netfslib that are not being handled correctly. It is returning a fixed number of bytes read to every read(2) call in your cat command, 16711680 bytes which is the offset of last subrequest. This will make cat(1) retry forever as netfslib is failing to return the correct number of bytes read, including EOF.
While testing a potential fix, I also found other problems with DIO in cifs.ko, so I'm working with Dave to get the proper fixes for both netfslib and cifs.ko.
I've noticed that you disabled caching with 'cache=none', is there any particular reason for that?
Have you also set rsize, wsize and bsize mount options? If so, why?
If you want to keep 'cache=none', then a possible workaround for you would be making rsize and wsize always greater than bsize. The default values (rsize=4194304,wsize=4194304,bsize=1048576) would do it.
Signature · 3 lines
Thanks, Steve