Performance degradation with ISAM tables

DISCUSSION Senén de Diego 3 participants 5 messages Dec 13, 2012 – Dec 15, 2012 4 messages not in the archive

Subsystems (2)

SMB client (CIFS) Kernel SMB3 server (ksmbd)

lore

Share & tools

lore

Senén de Diego

author

2012-12-13 22:53 UTC #0 lore raw

Hello,

I'm executing, in a linux server, a program that reads data from dbf files in a windows share. I've tried to execute the same program in an upgraded server and I've found that it runs much much much slower.

These are the versions of the first (older) server:
kernel: 2.6.37.6-24
cifs.ko: 1.68

and of the second (newer):
kernel: 3.4.11-2.16
cifs.ko: 1.78

The windows machine runs windows 2000 server and has EnableOplocks disabled in the LanManServer service.

The windows share is mounted as a cifs file system with some differences in the parameters assigned by default:
first server: rsize=16384,wsize=57344
second server: sec=ntlm,nounix,rsize=61440,wsize=65536

I've captured the network traffic in both servers, and these are the service response time statistics:

First server:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT

Close                         37   0.000140   0.001635   0.000404
Read AndX                  11554   0.000196   0.017179   0.002099
Write AndX                    12   0.000255   0.000657   0.000510
NT Create AndX                43   0.000190   0.000406   0.000237

Transaction2 Commands Calls Min SRT Max SRT Avg SRT

FIND_FIRST2                   83   0.000747   0.001953   0.001735
FIND_NEXT2                   880   0.000486   0.002885   0.001711
QUERY_PATH_INFO            25678   0.000170   0.005451   0.000250
QUERY_FILE_INFO               67   0.000169   0.000206   0.000182

NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================

Second server:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT

Close                        118   0.000128   0.006194   0.000517
Trans                          1   0.000872   0.000872   0.000872
Read AndX                   3490   0.000260   0.019713   0.009729
Write AndX                    12   0.000508   0.000522   0.000515
NT Create AndX               118   0.000253   0.001496   0.000342

Transaction2 Commands Calls Min SRT Max SRT Avg SRT

FIND_FIRST2                  171   0.000761   0.002862   0.002404
FIND_NEXT2                   326   0.002342   0.003058   0.002429
QUERY_PATH_INFO           318093   0.000168   0.041616   0.000320
QUERY_FILE_INFO              137   0.000235   0.002855   0.000370

NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================

where there is a great difference in the number of calls, specially in the number of QUERY_PATH_INFO commands executed. And the average SRT are also slower in the newer server.

I would like to know what is the cause and what can I do to correct this?

Senen de Diego

A message that is not in the archive (sent off-list or not yet received): 50CA5C4C.60103-qHrp9QoZuCS1Z/[email protected]

Jeff Layton

2012-12-14 02:39 UTC #2 lore raw

Quoted 61 lines · On Thu, 13 Dec 2012 23:53:00 +0100 Senén de Diego…

Hello,
I'm executing, in a linux server, a program that reads data from dbf files in a windows share. I've tried to execute the same program in an upgraded server and I've found that it runs much much much slower.
These are the versions of the first (older) server:
kernel: 2.6.37.6-24
cifs.ko: 1.68
and of the second (newer):
kernel: 3.4.11-2.16
cifs.ko: 1.78
The windows machine runs windows 2000 server and has EnableOplocks disabled in the LanManServer service.
The windows share is mounted as a cifs file system with some differences in the parameters assigned by default:
first server: rsize=16384,wsize=57344
second server: sec=ntlm,nounix,rsize=61440,wsize=65536
I've captured the network traffic in both servers, and these are the service response time statistics:
First server:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT
Close                         37   0.000140   0.001635   0.000404
Read AndX                  11554   0.000196   0.017179   0.002099
Write AndX                    12   0.000255   0.000657   0.000510
NT Create AndX                43   0.000190   0.000406   0.000237
Transaction2 Commands Calls Min SRT Max SRT Avg SRT
FIND_FIRST2                   83   0.000747   0.001953   0.001735
FIND_NEXT2                   880   0.000486   0.002885   0.001711
QUERY_PATH_INFO            25678   0.000170   0.005451   0.000250
QUERY_FILE_INFO               67   0.000169   0.000206   0.000182
NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================
Second server:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT
Close                        118   0.000128   0.006194   0.000517
Trans                          1   0.000872   0.000872   0.000872
Read AndX                   3490   0.000260   0.019713   0.009729
Write AndX                    12   0.000508   0.000522   0.000515
NT Create AndX               118   0.000253   0.001496   0.000342
Transaction2 Commands Calls Min SRT Max SRT Avg SRT
FIND_FIRST2                  171   0.000761   0.002862   0.002404
FIND_NEXT2                   326   0.002342   0.003058   0.002429
QUERY_PATH_INFO           318093   0.000168   0.041616   0.000320
QUERY_FILE_INFO              137   0.000235   0.002855   0.000370

Quoted 10 lines

NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================
where there is a great difference in the number of calls, specially in the number of QUERY_PATH_INFO commands executed. And the average SRT are also slower in the newer server.
I would like to know what is the cause and what can I do to correct this?

The difference between those two kernel represents more than a year of development time. The best first step you can take is to try a more recent kernel (like something 3.7-ish) to see if that might help.

The next problem is that it's difficult for us to infer anything about your workload. You'll probably need to do some investigative work to figure out what's actually slow. In particular, if you can phrase the problem in terms of particular syscalls being slower, then we might be able to help you more.

Even better is to come up with a small reproducer program that demonstrates the slowdown.

Signature · 1 line

Jeff Layton <[email protected]>

A message that is not in the archive (sent off-list or not yet received): 20121213213929.0eea2f5f-4QP7MXygkU+dMjc06nkz3ljfA9RmPOcC@public.gmane.org

Senén de Diego

author

2012-12-14 23:11 UTC #4 lore raw

Quoted 57 lines · Jeff Layton

Quoted 47 lines · On Thu, 13 Dec 2012 23:53:00 +0100 Senén de Diego…
Hello,
I'm executing, in a linux server, a program that reads data from dbf files in a windows share. I've tried to execute the same program in an upgraded server and I've found that it runs much much much slower. These are the versions of the first (older) server:
kernel: 2.6.37.6-24
cifs.ko: 1.68
and of the second (newer):
kernel: 3.4.11-2.16
cifs.ko: 1.78
The windows machine runs windows 2000 server and has EnableOplocks disabled in the LanManServer service.
The windows share is mounted as a cifs file system with some differences in the parameters assigned by default:
first server: rsize=16384,wsize=57344
second server: sec=ntlm,nounix,rsize=61440,wsize=65536
I've captured the network traffic in both servers, and these are the service response time statistics:
First server:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT
Close 37 0.000140 0.001635 0.000404
Read AndX 11554 0.000196 0.017179 0.002099
Write AndX 12 0.000255 0.000657 0.000510
NT Create AndX 43 0.000190 0.000406 0.000237
Transaction2 Commands Calls Min SRT Max SRT Avg SRT FIND_FIRST2 83 0.000747 0.001953 0.001735
FIND_NEXT2 880 0.000486 0.002885 0.001711
QUERY_PATH_INFO 25678 0.000170 0.005451 0.000250 QUERY_FILE_INFO 67 0.000169 0.000206 0.000182
NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================
Second server:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT
Close 118 0.000128 0.006194 0.000517
Trans 1 0.000872 0.000872 0.000872
Read AndX 3490 0.000260 0.019713 0.009729
Write AndX 12 0.000508 0.000522 0.000515
NT Create AndX 118 0.000253 0.001496 0.000342
Transaction2 Commands Calls Min SRT Max SRT Avg SRT FIND_FIRST2 171 0.000761 0.002862 0.002404
FIND_NEXT2 326 0.002342 0.003058 0.002429
Quoted 8 lines
QUERY_PATH_INFO 318093 0.000168 0.041616 0.000320 QUERY_FILE_INFO 137 0.000235 0.002855 0.000370
NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================
where there is a great difference in the number of calls, specially in the number of QUERY_PATH_INFO commands executed. And the average SRT are also slower in the newer server.
I would like to know what is the cause and what can I do to correct this?

Quoted 3 lines

The difference between those two kernel represents more than a year of development time. The best first step you can take is to try a more recent kernel (like something 3.7-ish) to see if that might help.

I've tried with a 3.6.10 kernel (the last available in my distro) and it's still slow:

=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT

Close                        116   0.000137   1.101347   0.009745
Trans                          3   0.000517   0.000722   0.000642
Echo                           1   0.000203   0.000203   0.000203
Read AndX                 122893   0.000206   0.154745   0.000599
Write AndX                    12   0.000253   0.005756   0.000789
NT Create AndX               118   0.000266   0.000865   0.000458

Transaction2 Commands Calls Min SRT Max SRT Avg SRT

FIND_FIRST2                  170   0.000843   0.003413   0.002590
FIND_NEXT2                   325   0.002373   0.006695   0.002908
QUERY_PATH_INFO           320423   0.000181   2.066630   0.000485
QUERY_FILE_INFO              135   0.000252   0.000768   0.000395

NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================

Quoted 7 lines

The next problem is that it's difficult for us to infer anything about your workload. You'll probably need to do some investigative work to figure out what's actually slow. In particular, if you can phrase the problem in terms of particular syscalls being slower, then we might be able to help you more.
Even better is to come up with a small reproducer program that demonstrates the slowdown.

The file access is done by a proprietary jdbc driver inside a jvm. I really don't know how to do what you suggest. The SMB service response time statitics are all I've got.

A message that is not in the archive (sent off-list or not yet received): 50CBB21B.3080908-qHrp9QoZuCS1Z/[email protected]

Steve French

2012-12-14 23:20 UTC #6 lore raw

Could you correlate these with the SMB counters (stats) on the client in /proc/fs/cifs/ which also should include bytes written/read?

Based on your description it looks like reads totally dominate this workload. The obvious thing to check is the read size - a few quick things does "forcedirectio" (or "cache=none" for newer kernels) help - in that case the network read size which usually match what the client application requests (which is usually a bad thing because we want
large i/o over the network and apps tend to read too small). Second
thing to try is a larger rsize - IIRC Windows should generally support up to rsize of about 127K (for Samba the rsize can be larger due to Unix extensions, we set the rsize smaller than 127K to Windows to work around bugs in other non-Samba, non-Windows servers).

Quoted 111 lines · Senén de Diego

Quoted 59 lines · Jeff Layton
Quoted 48 lines · On Thu, 13 Dec 2012 23:53:00 +0100 Senén de Diego…
Hello,
I'm executing, in a linux server, a program that reads data from dbf files in a windows share. I've tried to execute the same program in an upgraded server and I've found that it runs much much much slower. These are the versions of the first (older) server:
kernel: 2.6.37.6-24
cifs.ko: 1.68
and of the second (newer):
kernel: 3.4.11-2.16
cifs.ko: 1.78
The windows machine runs windows 2000 server and has EnableOplocks disabled in the LanManServer service.
The windows share is mounted as a cifs file system with some differences in the parameters assigned by default:
first server: rsize=16384,wsize=57344
second server: sec=ntlm,nounix,rsize=61440,wsize=65536
I've captured the network traffic in both servers, and these are the service response time statistics:
First server:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT
Close 37 0.000140 0.001635 0.000404
Read AndX 11554 0.000196 0.017179 0.002099
Write AndX 12 0.000255 0.000657 0.000510
NT Create AndX 43 0.000190 0.000406 0.000237
Transaction2 Commands Calls Min SRT Max SRT Avg SRT FIND_FIRST2 83 0.000747 0.001953 0.001735
FIND_NEXT2 880 0.000486 0.002885 0.001711
QUERY_PATH_INFO 25678 0.000170 0.005451 0.000250 QUERY_FILE_INFO 67 0.000169 0.000206 0.000182
NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================
Second server:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT
Close 118 0.000128 0.006194 0.000517
Trans 1 0.000872 0.000872 0.000872
Read AndX 3490 0.000260 0.019713 0.009729
Write AndX 12 0.000508 0.000522 0.000515
NT Create AndX 118 0.000253 0.001496 0.000342
Transaction2 Commands Calls Min SRT Max SRT Avg SRT FIND_FIRST2 171 0.000761 0.002862 0.002404
FIND_NEXT2 326 0.002342 0.003058 0.002429
Quoted 8 lines
QUERY_PATH_INFO 318093 0.000168 0.041616 0.000320 QUERY_FILE_INFO 137 0.000235 0.002855 0.000370
NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================
where there is a great difference in the number of calls, specially in the number of QUERY_PATH_INFO commands executed. And the average SRT are also slower in the newer server.
I would like to know what is the cause and what can I do to correct this?
Quoted 3 lines
The difference between those two kernel represents more than a year of development time. The best first step you can take is to try a more recent kernel (like something 3.7-ish) to see if that might help.
I've tried with a 3.6.10 kernel (the last available in my distro) and it's still slow:
=================================================================
SMB SRT Statistics:
Filter:
Commands Calls Min SRT Max SRT Avg SRT
Close                        116   0.000137   1.101347   0.009745
Trans                          3   0.000517   0.000722   0.000642
Echo                           1   0.000203   0.000203   0.000203
Read AndX                 122893   0.000206   0.154745   0.000599
Write AndX                    12   0.000253   0.005756   0.000789
NT Create AndX               118   0.000266   0.000865   0.000458
Transaction2 Commands Calls Min SRT Max SRT Avg SRT
FIND_FIRST2                  170   0.000843   0.003413   0.002590
FIND_NEXT2                   325   0.002373   0.006695   0.002908
QUERY_PATH_INFO           320423   0.000181   2.066630   0.000485
QUERY_FILE_INFO              135   0.000252   0.000768   0.000395
NT Transaction Commands Calls Min SRT Max SRT Avg SRT
=================================================================
Quoted 7 lines
The next problem is that it's difficult for us to infer anything about your workload. You'll probably need to do some investigative work to figure out what's actually slow. In particular, if you can phrase the problem in terms of particular syscalls being slower, then we might be able to help you more.
Even better is to come up with a small reproducer program that demonstrates the slowdown.
The file access is done by a proprietary jdbc driver inside a jvm. I really don't know how to do what you suggest. The SMB service response time statitics are all I've got.
--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 3 lines

Thanks,

Steve

A message that is not in the archive (sent off-list or not yet received): CAH2r5mv6njgwQpSUMD106BExG8b9ErRPdpeoC2eSj0mff3Hs6w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Jeff Layton

2012-12-15 14:39 UTC #8 lore raw

Quoted 14 lines · On Fri, 14 Dec 2012 17:20:28 -0600 Steve French…

Could you correlate these with the SMB counters (stats) on the client in /proc/fs/cifs/ which also should include bytes written/read?
Based on your description it looks like reads totally dominate this workload. The obvious thing to check is the read size - a few quick things does "forcedirectio" (or "cache=none" for newer kernels) help - in that case the network read size which usually match what the client application requests (which is usually a bad thing because we want
large i/o over the network and apps tend to read too small). Second
thing to try is a larger rsize - IIRC Windows should generally support up to rsize of about 127K (for Samba the rsize can be larger due to Unix extensions, we set the rsize smaller than 127K to Windows to work around bugs in other non-Samba, non-Windows servers).

Actually, based on the original email, it sounds to me more like this is a very metadata-intensive workload, likely dominated by lookups and/or stat calls. I'd be surprised if reads were slower on the new kernels, but it can be hard to tell...

In any case, you may want to try a larger actimeo= mount option and see if it helps. It defaults to 1s, but you have at least one QUERY_PATH_INFO call that took >2s for the server to respond to. Maybe try actimeo=5 or so?

Quoted 9 lines

Quoted 9 lines
Quoted 8 lines
The next problem is that it's difficult for us to infer anything about your workload. You'll probably need to do some investigative work to figure out what's actually slow. In particular, if you can phrase the problem in terms of particular syscalls being slower, then we might be able to help you more.
Even better is to come up with a small reproducer program that demonstrates the slowdown.

Quoted 5 lines

Quoted 5 lines
The file access is done by a proprietary jdbc driver inside a jvm. I really don't know how to do what you suggest. The SMB service response time statitics are all I've got.

Then you have your work cut out for you. Trying to figure out what the heck a particular java program is doing can be an exercise in pain management.

You might want to consider bisecting the problem down to a narrower range of kernel versions. If your distro provides pre-built kernels, you can try different versions within the range you've already ID'ed. See if you can figure out when the performance regression crept in.

Signature · 1 line

Jeff Layton <[email protected]>