Skip to contentSkip to conversation

Terrible performance between 2.6.16 and 2.6.5

DISCUSSION Frank Steiner 2 participants 3 messages Nov 30, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

I just upgraded our NFS server from SuSE 9.2 (kernel 2.6.8) to SuSE 10.1 (2.6.16.21). We have some SLES 9 (2.6.5) and some SLES 10 (2.6.16) clients acessing the NFS server.

When the server ran 2.6.8, the performance was always fine. After upgrading the SLES 10 clients still have nice performance, but the SLES 9 clients are terrible.

Here's a test call I'm doing:
find /mnt/tmp/i586/9/SuSE-updates/ -name \*.rpm -exec rpm -qp {} \;
It just queries all the rpm packages for their name.

When I call this on a SLES 10 client I can monitor the network traffic on the server lies between 200k and some little peaks at 1.5M.

Calling it on a SLES9 client looks fine at first, then after some RPMs the network traffic raises to about 20-30 MB/sec! Guess what the server says when 30 clients search for new RPMs with an algorithm similar to this.

When I call the "find" a second time on the same SLES 9 clients it runs fine for some longer time, like it had cached the accesses or if it had some buffer that would fill and then slow the read down.

On all clients we mount with "ro,tcp,hard,rsize=16384,wsize=16384". Changing sync/async on the server and the clients didn't help.

Are there any known issues? Any ideas how I could debug this? We can't throw away SLES 9 on these clients, so I need to solve this :-(

cu,
Frank

Signature · 14 lines
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#1 lore raw
Quoted 25 lines · Frank Steiner

Hi,

I just upgraded our NFS server from SuSE 9.2 (kernel 2.6.8) to SuSE 10.1 (2.6.16.21). We have some SLES 9 (2.6.5) and some SLES 10 (2.6.16) clients acessing the NFS server.

When the server ran 2.6.8, the performance was always fine. After upgrading the SLES 10 clients still have nice performance, but the SLES 9 clients are terrible.

Here's a test call I'm doing:
find /mnt/tmp/i586/9/SuSE-updates/ -name \*.rpm -exec rpm -qp {} \;
It just queries all the rpm packages for their name.

When I call this on a SLES 10 client I can monitor the network traffic on the server lies between 200k and some little peaks at 1.5M.

Calling it on a SLES9 client looks fine at first, then after some RPMs the network traffic raises to about 20-30 MB/sec! Guess what the server says when 30 clients search for new RPMs with an algorithm similar to this.

When I call the "find" a second time on the same SLES 9 clients it runs fine for some longer time, like it had cached the accesses or if it had some buffer that would fill and then slow the read down.

Quoted 8 lines

On all clients we mount with "ro,tcp,hard,rsize=16384,wsize=16384". Changing sync/async on the server and the clients didn't help.

Are there any known issues? Any ideas how I could debug this? We can't throw away SLES 9 on these clients, so I need to solve this :-(

cu,
Frank

Frank,

What is the type of the underlying disk subsystem? Did the underlying
disk subsystem's performance change or is it the same? You may need
to run bonnie or something similar to verify that the disk hardware is not where the issue is.

I have seen the MPT driver in the later kernels be quite a bit slower (3x) with certain combinations of devices.

And there are a fair number of issues were various other disk devices from time to time get much slower in newer kernels, but are fine in older kernels.

Roger

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#2 lore raw

Roger Heflin wrote

Quoted 4 lines

What is the type of the underlying disk subsystem? Did the underlying
disk subsystem's performance change or is it the same? You may need
to run bonnie or something similar to verify that the disk hardware is not where the issue is.

Not, that's the same external FC raid as before. But I will do some local disk tests!

Quoted 6 lines

I have seen the MPT driver in the later kernels be quite a bit slower (3x) with certain combinations of devices.

And there are a fair number of issues were various other disk devices from time to time get much slower in newer kernels, but are fine in older kernels.

I don't think it's the disk itself, because the SuSE 10.1/SLES 10 clients don't show this performance problem when running the same command on the same directories. Therefore I'm pretty sure that the SLES 9 kernel has some issues with the newer NFS system in 2.6.16.

cu,
Frank

Signature · 14 lines
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs