Performance degradation with ISAM tables
Subsystems (2)
Hello,
I'm executing, in a linux server, a program that reads data from dbf files in a windows share. I've tried to execute the same program in an upgraded server and I've found that it runs much much much slower.
These are the versions of the first (older) server:
kernel: 2.6.37.6-24
cifs.ko: 1.68
and of the second (newer):
kernel: 3.4.11-2.16
cifs.ko: 1.78
The windows machine runs windows 2000 server and has EnableOplocks disabled in the LanManServer service.
The windows share is mounted as a cifs file system with some differences in the parameters assigned by default:
first server: rsize=16384,wsize=57344
second server: sec=ntlm,nounix,rsize=61440,wsize=65536
I've captured the network traffic in both servers, and these are the service response time statistics:
First server:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT
Close 37 0.000140 0.001635 0.000404 Read AndX 11554 0.000196 0.017179 0.002099 Write AndX 12 0.000255 0.000657 0.000510 NT Create AndX 43 0.000190 0.000406 0.000237
Transaction2 Commands Calls Min SRT Max SRT Avg SRT
FIND_FIRST2 83 0.000747 0.001953 0.001735 FIND_NEXT2 880 0.000486 0.002885 0.001711 QUERY_PATH_INFO 25678 0.000170 0.005451 0.000250 QUERY_FILE_INFO 67 0.000169 0.000206 0.000182
NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================
Second server:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT
Close 118 0.000128 0.006194 0.000517 Trans 1 0.000872 0.000872 0.000872 Read AndX 3490 0.000260 0.019713 0.009729 Write AndX 12 0.000508 0.000522 0.000515 NT Create AndX 118 0.000253 0.001496 0.000342
Transaction2 Commands Calls Min SRT Max SRT Avg SRT
FIND_FIRST2 171 0.000761 0.002862 0.002404 FIND_NEXT2 326 0.002342 0.003058 0.002429 QUERY_PATH_INFO 318093 0.000168 0.041616 0.000320 QUERY_FILE_INFO 137 0.000235 0.002855 0.000370
NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================
where there is a great difference in the number of calls, specially in the number of QUERY_PATH_INFO commands executed. And the average SRT are also slower in the newer server.
I would like to know what is the cause and what can I do to correct this?
Senen de Diego
Quoted 61 lines · On Thu, 13 Dec 2012 23:53:00 +0100 Senén de Diego…
Hello,
I'm executing, in a linux server, a program that reads data from dbf files in a windows share. I've tried to execute the same program in an upgraded server and I've found that it runs much much much slower.
These are the versions of the first (older) server:
kernel: 2.6.37.6-24
cifs.ko: 1.68and of the second (newer):
kernel: 3.4.11-2.16
cifs.ko: 1.78The windows machine runs windows 2000 server and has EnableOplocks disabled in the LanManServer service.
The windows share is mounted as a cifs file system with some differences in the parameters assigned by default:
first server: rsize=16384,wsize=57344
second server: sec=ntlm,nounix,rsize=61440,wsize=65536I've captured the network traffic in both servers, and these are the service response time statistics:
First server:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRTClose 37 0.000140 0.001635 0.000404 Read AndX 11554 0.000196 0.017179 0.002099 Write AndX 12 0.000255 0.000657 0.000510 NT Create AndX 43 0.000190 0.000406 0.000237Transaction2 Commands Calls Min SRT Max SRT Avg SRT
FIND_FIRST2 83 0.000747 0.001953 0.001735 FIND_NEXT2 880 0.000486 0.002885 0.001711 QUERY_PATH_INFO 25678 0.000170 0.005451 0.000250 QUERY_FILE_INFO 67 0.000169 0.000206 0.000182NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================Second server:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRTClose 118 0.000128 0.006194 0.000517 Trans 1 0.000872 0.000872 0.000872 Read AndX 3490 0.000260 0.019713 0.009729 Write AndX 12 0.000508 0.000522 0.000515 NT Create AndX 118 0.000253 0.001496 0.000342Transaction2 Commands Calls Min SRT Max SRT Avg SRT
FIND_FIRST2 171 0.000761 0.002862 0.002404 FIND_NEXT2 326 0.002342 0.003058 0.002429 QUERY_PATH_INFO 318093 0.000168 0.041616 0.000320 QUERY_FILE_INFO 137 0.000235 0.002855 0.000370
Quoted 10 lines
NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================where there is a great difference in the number of calls, specially in the number of
QUERY_PATH_INFOcommands executed. And the average SRT are also slower in the newer server.I would like to know what is the cause and what can I do to correct this?
The difference between those two kernel represents more than a year of development time. The best first step you can take is to try a more recent kernel (like something 3.7-ish) to see if that might help.
The next problem is that it's difficult for us to infer anything about your workload. You'll probably need to do some investigative work to figure out what's actually slow. In particular, if you can phrase the problem in terms of particular syscalls being slower, then we might be able to help you more.
Even better is to come up with a small reproducer program that demonstrates the slowdown.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 57 lines · Jeff Layton
Quoted 47 lines · On Thu, 13 Dec 2012 23:53:00 +0100 Senén de Diego…
Hello,
I'm executing, in a linux server, a program that reads data from dbf files in a windows share. I've tried to execute the same program in an upgraded server and I've found that it runs much much much slower. These are the versions of the first (older) server:
kernel: 2.6.37.6-24
cifs.ko: 1.68
and of the second (newer):
kernel: 3.4.11-2.16
cifs.ko: 1.78
The windows machine runs windows 2000 server and has EnableOplocks disabled in the LanManServer service.
The windows share is mounted as a cifs file system with some differences in the parameters assigned by default:
first server: rsize=16384,wsize=57344
second server: sec=ntlm,nounix,rsize=61440,wsize=65536
I've captured the network traffic in both servers, and these are the service response time statistics:
First server:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT
Close 37 0.000140 0.001635 0.000404
Read AndX 11554 0.000196 0.017179 0.002099
Write AndX 12 0.000255 0.000657 0.000510
NT Create AndX 43 0.000190 0.000406 0.000237
Transaction2 Commands Calls Min SRT Max SRT Avg SRTFIND_FIRST283 0.000747 0.001953 0.001735FIND_NEXT2880 0.000486 0.002885 0.001711QUERY_PATH_INFO25678 0.000170 0.005451 0.000250QUERY_FILE_INFO67 0.000169 0.000206 0.000182
NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================
Second server:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT
Close 118 0.000128 0.006194 0.000517
Trans 1 0.000872 0.000872 0.000872
Read AndX 3490 0.000260 0.019713 0.009729
Write AndX 12 0.000508 0.000522 0.000515
NT Create AndX 118 0.000253 0.001496 0.000342
Transaction2 Commands Calls Min SRT Max SRT Avg SRTFIND_FIRST2171 0.000761 0.002862 0.002404FIND_NEXT2326 0.002342 0.003058 0.002429Quoted 8 lines
QUERY_PATH_INFO318093 0.000168 0.041616 0.000320QUERY_FILE_INFO137 0.000235 0.002855 0.000370
NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================
where there is a great difference in the number of calls, specially in the number ofQUERY_PATH_INFOcommands executed. And the average SRT are also slower in the newer server.
I would like to know what is the cause and what can I do to correct this?
Quoted 3 lines
The difference between those two kernel represents more than a year of development time. The best first step you can take is to try a more recent kernel (like something 3.7-ish) to see if that might help.
I've tried with a 3.6.10 kernel (the last available in my distro) and it's still slow:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT
Close 116 0.000137 1.101347 0.009745 Trans 3 0.000517 0.000722 0.000642 Echo 1 0.000203 0.000203 0.000203 Read AndX 122893 0.000206 0.154745 0.000599 Write AndX 12 0.000253 0.005756 0.000789 NT Create AndX 118 0.000266 0.000865 0.000458
Transaction2 Commands Calls Min SRT Max SRT Avg SRT
FIND_FIRST2 170 0.000843 0.003413 0.002590 FIND_NEXT2 325 0.002373 0.006695 0.002908 QUERY_PATH_INFO 320423 0.000181 2.066630 0.000485 QUERY_FILE_INFO 135 0.000252 0.000768 0.000395
NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================
Quoted 7 lines
The next problem is that it's difficult for us to infer anything about your workload. You'll probably need to do some investigative work to figure out what's actually slow. In particular, if you can phrase the problem in terms of particular syscalls being slower, then we might be able to help you more.
Even better is to come up with a small reproducer program that demonstrates the slowdown.
The file access is done by a proprietary jdbc driver inside a jvm. I really don't know how to do what you suggest. The SMB service response time statitics are all I've got.
Could you correlate these with the SMB counters (stats) on the client in /proc/fs/cifs/ which also should include bytes written/read?
Based on your description it looks like reads totally dominate this workload. The obvious thing to check is the read size - a few quick things does "forcedirectio" (or "cache=none" for newer kernels) help - in that case the network read size which usually match what the client application requests (which is usually a bad thing because we want
large i/o over the network and apps tend to read too small). Second
thing to try is a larger rsize - IIRC Windows should generally support up to rsize of about 127K (for Samba the rsize can be larger due to Unix extensions, we set the rsize smaller than 127K to Windows to work around bugs in other non-Samba, non-Windows servers).
Quoted 111 lines · Senén de Diego
Quoted 59 lines · Jeff Layton
Quoted 48 lines · On Thu, 13 Dec 2012 23:53:00 +0100 Senén de Diego…
Hello,
I'm executing, in a linux server, a program that reads data from dbf files in a windows share. I've tried to execute the same program in an upgraded server and I've found that it runs much much much slower. These are the versions of the first (older) server:
kernel: 2.6.37.6-24
cifs.ko: 1.68
and of the second (newer):
kernel: 3.4.11-2.16
cifs.ko: 1.78
The windows machine runs windows 2000 server and has EnableOplocks disabled in the LanManServer service.
The windows share is mounted as a cifs file system with some differences in the parameters assigned by default:
first server: rsize=16384,wsize=57344
second server: sec=ntlm,nounix,rsize=61440,wsize=65536
I've captured the network traffic in both servers, and these are the service response time statistics:
First server:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT
Close 37 0.000140 0.001635 0.000404
Read AndX 11554 0.000196 0.017179 0.002099
Write AndX 12 0.000255 0.000657 0.000510
NT Create AndX 43 0.000190 0.000406 0.000237
Transaction2 Commands Calls Min SRT Max SRT Avg SRTFIND_FIRST283 0.000747 0.001953 0.001735FIND_NEXT2880 0.000486 0.002885 0.001711QUERY_PATH_INFO25678 0.000170 0.005451 0.000250QUERY_FILE_INFO67 0.000169 0.000206 0.000182
NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================
Second server:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT
Close 118 0.000128 0.006194 0.000517
Trans 1 0.000872 0.000872 0.000872
Read AndX 3490 0.000260 0.019713 0.009729
Write AndX 12 0.000508 0.000522 0.000515
NT Create AndX 118 0.000253 0.001496 0.000342
Transaction2 Commands Calls Min SRT Max SRT Avg SRTFIND_FIRST2171 0.000761 0.002862 0.002404FIND_NEXT2326 0.002342 0.003058 0.002429Quoted 8 lines
QUERY_PATH_INFO318093 0.000168 0.041616 0.000320QUERY_FILE_INFO137 0.000235 0.002855 0.000370
NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================
where there is a great difference in the number of calls, specially in the number ofQUERY_PATH_INFOcommands executed. And the average SRT are also slower in the newer server.
I would like to know what is the cause and what can I do to correct this?Quoted 3 lines
The difference between those two kernel represents more than a year of development time. The best first step you can take is to try a more recent kernel (like something 3.7-ish) to see if that might help.
I've tried with a 3.6.10 kernel (the last available in my distro) and it's still slow:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRTClose 116 0.000137 1.101347 0.009745 Trans 3 0.000517 0.000722 0.000642 Echo 1 0.000203 0.000203 0.000203 Read AndX 122893 0.000206 0.154745 0.000599 Write AndX 12 0.000253 0.005756 0.000789 NT Create AndX 118 0.000266 0.000865 0.000458Transaction2 Commands Calls Min SRT Max SRT Avg SRT
FIND_FIRST2 170 0.000843 0.003413 0.002590 FIND_NEXT2 325 0.002373 0.006695 0.002908 QUERY_PATH_INFO 320423 0.000181 2.066630 0.000485 QUERY_FILE_INFO 135 0.000252 0.000768 0.000395NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================Quoted 7 lines
The next problem is that it's difficult for us to infer anything about your workload. You'll probably need to do some investigative work to figure out what's actually slow. In particular, if you can phrase the problem in terms of particular syscalls being slower, then we might be able to help you more.
Even better is to come up with a small reproducer program that demonstrates the slowdown.The file access is done by a proprietary jdbc driver inside a jvm. I really don't know how to do what you suggest. The SMB service response time statitics are all I've got.
--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
Thanks, Steve
Quoted 14 lines · On Fri, 14 Dec 2012 17:20:28 -0600 Steve French…
Could you correlate these with the SMB counters (stats) on the client in
/proc/fs/cifs/which also should include bytes written/read?Based on your description it looks like reads totally dominate this workload. The obvious thing to check is the read size - a few quick things does "forcedirectio" (or "cache=none" for newer kernels) help - in that case the network read size which usually match what the client application requests (which is usually a bad thing because we want
large i/o over the network and apps tend to read too small). Second
thing to try is a larger rsize - IIRC Windows should generally support up to rsize of about 127K (for Samba the rsize can be larger due to Unix extensions, we set the rsize smaller than 127K to Windows to work around bugs in other non-Samba, non-Windows servers).
Actually, based on the original email, it sounds to me more like this is a very metadata-intensive workload, likely dominated by lookups and/or stat calls. I'd be surprised if reads were slower on the new kernels, but it can be hard to tell...
In any case, you may want to try a larger actimeo= mount option and see if it helps. It defaults to 1s, but you have at least one QUERY_PATH_INFO call that took >2s for the server to respond to. Maybe try actimeo=5 or so?
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
The next problem is that it's difficult for us to infer anything about your workload. You'll probably need to do some investigative work to figure out what's actually slow. In particular, if you can phrase the problem in terms of particular syscalls being slower, then we might be able to help you more.
Even better is to come up with a small reproducer program that demonstrates the slowdown.
Quoted 5 lines
Quoted 5 lines
The file access is done by a proprietary jdbc driver inside a jvm. I really don't know how to do what you suggest. The SMB service response time statitics are all I've got.
Then you have your work cut out for you. Trying to figure out what the heck a particular java program is doing can be an exercise in pain management.
You might want to consider bisecting the problem down to a narrower range of kernel versions. If your distro provides pre-built kernels, you can try different versions within the range you've already ID'ed. See if you can figure out when the performance regression crept in.
Signature · 1 line
Jeff Layton <[email protected]>