CIFS slowness & crashes
I mailed [email protected] (the guy who wrote the driver) about this a month ago, but didn't get any reply. Is anyone working on that driver anymore?
The problems that I wrote him about were:
1. CIFS VFS hangs entirely if the server crashes or otherwise goes offline. Every process touching the mount halts too and cannot be killed (but they are not zombies). System loads start climbing and eventually the entire system will die (after system loads reach about 500). It is not possible to umount with either smbumount (hangs) nor umount -f (prints errors but doesn't umount anything). It won't recover without reboot, even if the server becomes back online.
This problem has been around as long as I have used SMBFS or CIFS. There has only been slight variation from one version to another. Sometimes it is possible to umount them (after some pretty long timeout), sometimes it is not. It seems as if the problem was being fixed, but none of the fixes really worked.
2. Occassionally the transmission speeds go extremely low for no apparent reason. While writing this, I am getting 0.39 Mo/s over a gigabit network. Using FTP to read the same file gives 40 Mo/s, which is the speed that the file can be read locally on the server too. Remounting the CIFS does not help, nor does restarting Samba. However, using SMBFS I can get 20 Mo/s which is a bit better but still far from what it should be. It is important to mention that sometimes CIFS does work faster (about as quickly as SMBFS) and that this misbehavior occurs randomly.
During CIFS transfer, both computers seem to be idling. The CPU usage (including I/O wait) is almost none. During SMBFS transfer the server smbd process uses about 15 % CPU and the client is almost idle. The client is P4 3.4 GHz and the server is Athlon64 3000+.
I also tested with a Windows XP client machine and found out that this slowness issue does not happen with it, using the very same Samba server that the Linux CIFS mount is using.
- Tronic -
Quoted 4 lines · Lasse Kärkkäinen / Tronic
I mailed [email protected] (the guy who wrote the driver) about this a month ago, but didn't get any reply. Is anyone working on that driver anymore?
As far as I know Steve is still maintaining cifs. If you wrote him and didn't get a response, then try again after a while (you might have included him on CC for this mail) - maintainers don't always have time to answer all mail in a timely fashion (or at all), and it's your responsability to resend - that's not news.
You could also have written to the [email protected] mailinglist (or copied it - it's listed in MAINTAINERS under "COMMON INTERNET FILE SYSTEM (CIFS)").
[adding Stephen French to CC]
Personally I'd probably have send the mail
To: Steve French <[email protected]>
Cc: [email protected]
Cc: [email protected]
Quoted 38 lines
The problems that I wrote him about were:
1. CIFS VFS hangs entirely if the server crashes or otherwise goes offline. Every process touching the mount halts too and cannot be killed (but they are not zombies). System loads start climbing and eventually the entire system will die (after system loads reach about 500). It is not possible to umount with either smbumount (hangs) nor umount -f (prints errors but doesn't umount anything). It won't recover without reboot, even if the server becomes back online.
This problem has been around as long as I have used SMBFS or CIFS. There has only been slight variation from one version to another. Sometimes it is possible to umount them (after some pretty long timeout), sometimes it is not. It seems as if the problem was being fixed, but none of the fixes really worked.
2. Occassionally the transmission speeds go extremely low for no apparent reason. While writing this, I am getting 0.39 Mo/s over a gigabit network. Using FTP to read the same file gives 40 Mo/s, which is the speed that the file can be read locally on the server too. Remounting the CIFS does not help, nor does restarting Samba. However, using SMBFS I can get 20 Mo/s which is a bit better but still far from what it should be. It is important to mention that sometimes CIFS does work faster (about as quickly as SMBFS) and that this misbehavior occurs randomly.
During CIFS transfer, both computers seem to be idling. The CPU usage (including I/O wait) is almost none. During SMBFS transfer the server smbd process uses about 15 % CPU and the client is almost idle. The client is P4 3.4 GHz and the server is Athlon64 3000+.
I also tested with a Windows XP client machine and found out that this slowness issue does not happen with it, using the very same Samba server that the Linux CIFS mount is using.
- Tronic -
Quoted 10 lines · Jesper Juhl
Quoted 4 lines · Lasse Kärkkäinen / Tronic
I mailed [email protected] (the guy who wrote the driver) about this a month ago, but didn't get any reply. Is anyone working on that driver anymore?
As far as I know Steve is still maintaining cifs. If you wrote him and didn't get a response, then try again after a while (you might have included him on CC for this mail) - maintainers don't always have time to answer all mail in a timely fashion (or at all), and it's your responsability to resend - that's not news.
CIFS is still being actively improved/maintained. I was out on vacation for over a week. It might have gotten missed in the flood of email I had to catch up on. I am still maintaining cifs and evaluating patches from others as well. As you can see from the web git page, changesets are still being added. You can see the list of CIFS changesets (most recent first) by the query:
http://www.kernel.org/git/?p=linux%2Fkernel%2Fgit%2Fsfrench%2Fcifs-2.6.git&a=search&s=CIFS
Quoted 10 lines
You could also have written to the [email protected] mailinglist (or copied it - it's listed in
MAINTAINERSunder "COMMON INTERNET FILE SYSTEM (CIFS)").[adding Stephen French to CC]
Personally I'd probably have send the mail
To: Steve French <[email protected]>
Cc: [email protected]
Cc: [email protected]
Yes. CCing those lists is recommended. The best list to send to is and which I and others in this area monitor regularly though is [email protected]
Quoted 1 line
Quoted 1 line
not possible to umount with either smbumount (hangs)
smbumount can not be called on cifs vfs mounts (it is for the older smbfs). Although there is a somewhat similar umount.cifs utility (umount.cifs.c in the samba 3 source) it will be obsolete when more general user umounting (of their own mounts) is permitted in the kernel vfs itself.
Quoted 2 lines
nor umount -f
Quoted 1 line
(prints errors but doesn't umount anything).
What are the errors? What is the version of cifs.ko module?
Quoted 8 lines
It won't recover without
Quoted 7 lines
reboot, even if the server becomes back online.
This problem has been around as long as I have used SMBFS or CIFS. There has only been slight variation from one version to another. Sometimes it is possible to umount them (after some pretty long timeout), sometimes it is not. It seems as if the problem was being fixed, but none of the fixes really worked.
My tests of reconnection after server crash or network reconnection with smbfs works (for the past year or two) and also of course for cifs. cifs also reconnects state (open files) not just the connection to \\server\share
Quoted 4 lines
Quoted 4 lines
2. Occassionally the transmission speeds go extremely low for no apparent reason. While writing this, I am getting 0.39 Mo/s over a gigabit network.
My informal tests (cifs->samba) showed a maximum of about 20% utilization of gigabit doing large file writes and about double that for large file reads with a single cifs client to Samba over gigabit. Should be somewhat simalar to Windows server.
The most common cause of widely varying speeds are the following:
1) memory fragmentation - some versions of the kernel badly fragment slab allocations greater than 4 pages (cifs by default allocates 16.5K buffers - which results in larger size allocations when more than 5 processes are accessing the mount and the cifs buffer pool is exceeded)
2) write behind due to oplock - smbfs does not do oplock, cifs does - which means that cifs client caching can cause a large amount of writebehind data to accumulate (with great performance for a while) - then when memory gets tight due to the inode caching in linux's mm layer, the cifs client is asked to write out a large amount of file data at one time (which it does synchronously).
Both of these are being improved. You can bypass the inode caching on the client by using the cifs mount option
"forcedirectio"
Quoted 1 line
Using FTP to read the same file gives 40 Mo/s,
Similarly NFS was somewhat better than that (as long as the file was in memory already on the server).
Quoted 3 lines · Steve French
Yes. CCing those lists is recommended. The best list to send to is and which I and others in this area monitor regularly though is [email protected]
If that's the best list to send to, then I suggest it be added to the CIFS MAINTAINERS entry as pr the patch below (also attached since gmail will probably mangle it).
Add [email protected] to CIFS entry in MAINTAINERS
---
MAINTAINERS | 1 + 1 files changed, 1 insertion(+)
MAINTAINERS 2005-07-22 01:42:09.000000000 +0200 renamed from MAINTAINERS 2005-07-14 20:33:32.000000000 +0200 +1 -0
@@ -532,6 +532,7 @@ COMMON INTERNET FILE SYSTEM (CIFS) P: Steve French M: [email protected] +L: [email protected] L: [email protected] W: http://us1.samba.org/samba/Linux_CIFS_client.html S: Supported
Signature · 3 lines
Jesper Juhl <[email protected]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
From: Jesper Juhl <[email protected]>
Add [email protected] to CIFS entry in MAINTAINERS
MAINTAINERS +1 -0
@@ -532,6 +532,7 @@ COMMON INTERNET FILE SYSTEM (CIFS) P: Steve French M: [email protected] +L: [email protected] L: [email protected] W: http://us1.samba.org/samba/Linux_CIFS_client.html S: Supported
Quoted 2 lines
Quoted 1 line
nor umount -f
What are the errors? What is the version of cifs.ko module?
umount2: Device or resource busy
umount: /tmpmnt: device is busy
umount2: Device or resource busy
umount: /tmpmnt: device is busy
Without -f it doesn't print those umount2 errors, just the other two.
The version is whatever comes with 2.6.12 or 2.6.13rc1.
Quoted 4 lines
My tests of reconnection after server crash or network reconnection with smbfs works (for the past year or two) and also of course for cifs. cifs also reconnects state (open files) not just the connection to \\server\share
Reconnection usually (or perhaps always, with new versions) works, but nothing cannot be done if the server goes permanently offline (nor until it comes back online). When the server is down, the programs that were using the CIFS mount or that try to access it will halt.
Quoted 4 lines
My informal tests (
cifs->samba) showed a maximum of about 20% utilization of gigabit doing large file writes and about double that for large file reads with a single cifs client to Samba over gigabit. Should be somewhat simalar to Windows server.
This seems quite bad. Is it waiting for packet confirmations or what?
However, I have never got anything better than about 40 Mo/s with a gigabit network so far. Jumbo frames and using direct cross-over cable between the clients make no difference. Still, all protocols that I have tried, except for SMB and CIFS, can reach that easily. I'll try to get two Windows machines that I can test with. Perhaps the problem is with Linux network drivers.
Anyway, 20 Mo/s or something like that would not be a big problem. The real problem occurs when the speed drops under 3 Mo/s.
Quoted 5 lines
The most common cause of widely varying speeds are the following:
1) memory fragmentation - some versions of the kernel badly fragment slab allocations greater than 4 pages (cifs by default allocates 16.5K buffers - which results in larger size allocations when more than 5 processes are accessing the mount and the cifs buffer pool is exceeded)
Wouldn't this show up as increased system loads?
Quoted 10 lines
2) write behind due to oplock - smbfs does not do oplock, cifs does - which means that cifs client caching can cause a large amount of writebehind data to accumulate (with great performance for a while) - then when memory gets tight due to the inode caching in linux's mm layer, the cifs client is asked to write out a large amount of file data at one time (which it does synchronously).
Both of these are being improved. You can bypass the inode caching on the client by using the cifs mount option
"forcedirectio"
This option has little or no effect. Still runs slowly (2.4 Mo/s right now).
My benchmark is reading data in 50 Mio chunks, as quickly as it can, calculating the average read speed. The files being read are bigger than 4 Gio. I haven't tried writing anything (the shares that I play with are read-only).
For the record, I am using O_RDONLY | O_LARGEFILE on open and then pread
for all reads. However, I am getting similar results with all programs,
so I don't think that the reading method really matters that much.
- Tronic -
Quoted 4 lines
Quoted 4 lines
2. Occassionally the transmission speeds go extremely low for no apparent reason. While writing this, I am getting 0.39 Mo/s over a gigabit network.
It would help to know whether you are doing mostly writing to or reading from the server.
Forgot to mention that another thing that may help (besides the large number of improvements that already went into 2.6.12 already to drastically reduce the expensive large buffer usage, and the even newer experimental CIFS write code) are two configuration changes - either dropping the buffer size to relieve pressure on the slab (e.g. to just under 8K - perhaps 7800 bytes) or increasing the number of large buffers in the pool (they are insmod parms of cifs.ko - you can do modinfo on cifs.ko to see them) so fewer allocations are done. This would only be needed if multiple processes were accessing the mount at one time.