nfs performance problem
Subsystems (2)
Hi All
I have really slow working nfs.
Server is build on:
- dual pentium Xeon 2GHz,
- 2 3ware 7500 controllers,
- raid 5 on 7 160GB discs on controler 1
- raid 5 on 6 160GB disks on cotroller 2
Serwer is used for ~500k maildirs,
all clients (~20) are freebsd boxes, when nfs transfer is around 1MB/s for writing and 1.5MB/s for reading listing user directories even with very few file stakes up to 15 seconds.
I tried nfs-ALL patch, but it didn' t help.
I run 256 nfs daemons,
echo 2097152 > /proc/sys/net/core/rmem_max
echo 2097152 > /proc/sys/net/core/rmem_default
nsfstat output:
Server rpc stats:
calls badcalls badauth badclnt xdrcall 25271043 17885061 0 17885061 0
Server nfs v3:
null getattr setattr lookup access readlink
25 0% 1159 0% 89718 0% 11702395 46% 10953437 43% 34868 0%
read write create mkdir symlink mknod
925727 3% 226250 0% 44964 0% 1089 0% 2299 0% 0 0%
remove rmdir rename link readdir readdirplus
35040 0% 42 0% 121064 0% 535 0% 534720 2% 0 0%
fsstat fsinfo pathconf commit
585179 2% 36 0% 0 0% 12493 0%
mount over tcp gives frequent errors on freebsd clients side:
kernel: nfs send error 32 from nfs server xxx.xxx.xxx.xxx:/export
thanks for any help,
rafal
Quoted 11 lines · poczta.dotcom.pl
Hi All
=20
I have really slow working nfs.
Server is build on:
- dual pentium Xeon 2GHz,
- 2 3ware 7500 controllers,
- raid 5 on 7 160GB discs on controler 1=20
- raid 5 on 6 160GB disks on cotroller 2=20
=20
Serwer is used for ~500k maildirs,=20
all clients (~20) are freebsd boxes, when nfs transfer is around 1MB/s =
for writing and 1.5MB/s for reading=20
Quoted 1 line
listing user directories even with very few file stakes up to 15 second=
s.
Quoted 2 lines
=20
I tried nfs-ALL patch, but it didn' t help.
How is local performance?
What filesystem do you use?
What's the output from "iostat -x -d -k 60 3"?
--=20
Ragnar Kj=F8rstad
Big Storage
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
----- Original Message -----
From: "Ragnar Kj=F8rstad" <[email protected]>
To: "poczta.dotcom.pl" <[email protected]>
Cc: <[email protected]>
Sent: Tuesday, November 05, 2002 8:17 PM
Subject: Re: [NFS] nfs performance problem
Quoted 10 lines · poczta.dotcom.pl
Quoted 9 lines
Hi All
I have really slow working nfs.
Server is build on:
- dual pentium Xeon 2GHz,
- 2 3ware 7500 controllers,
- raid 5 on 7 160GB discs on controler 1
- raid 5 on 6 160GB disks on cotroller 2
Quoted 2 lines
Quoted 2 lines
Serwer is used for ~500k maildirs,
all clients (~20) are freebsd boxes, when nfs transfer is around 1MB/=
s
for writing and 1.5MB/s for reading
Quoted 1 line
Quoted 1 line
listing user directories even with very few file stakes up to 15
seconds.
Quoted 4 lines
Quoted 2 lines
I tried nfs-ALL patch, but it didn' t help.
How is local performance?
local read/writes are ok
Quoted 1 line
What filesystem do you use?
reiserfs on top of lvm (to be able to get snapshots)
Quoted 2 lines
What's the output from "iostat -x -d -k 60 3"?
below is output from 'iostat -d 60 3' (- my iostat does not take '-k' option,
and '-d' gives me empty output) - mayby this is enough?
Linux 2.4.19 (hostname) 11/05/02
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 16.16 205.52 82.84 3481842 1403528
dev8-1 160.58 1266.39 1346.12 21455002 22805744
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 9.23 134.93 16.13 8096 968
dev8-1 150.92 753.20 1715.87 45192 102952
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 10.02 135.07 26.53 8104 1592
dev8-1 152.78 763.33 1767.20 45800 106032
rafal
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
I have 3ware cards as well (2 7500 Series 8 drive), just like yours. ALso with 120 GIG drives (7200)
RPM. Whever I get 150-200 tps in iostat then my NFS runs SO SLOW. I also only get 1 MB, to 1.5 MB
over NFS. The local speed seems to be ok, not great though. MY OPINION is
that this is because of seek time on the raid
array. I am serving very small files, just like you are. I am requesting
about 200-300 files per second from
each NFS server. So even though our throughput of only 1.5 MB isn't high. The number of files per second is
actually quite high, and causes things to slow down because of seek time issues. PLEASE GIVE US CACHEFS SOMEONE??
Does anyone have experience with IDE Raid arrays that get over 250 tps in iostat that work fine? I would
be VERY VERY VERY interested to find out.
L8r...
Matt
Quoted 10 lines · poczta.dotcom.pl
Quoted 9 lines
Hi All
I have really slow working nfs.
Server is build on:
- dual pentium Xeon 2GHz,
- 2 3ware 7500 controllers,
- raid 5 on 7 160GB discs on controler 1
- raid 5 on 6 160GB disks on cotroller 2
Quoted 2 lines
Quoted 2 lines
Serwer is used for ~500k maildirs,
all clients (~20) are freebsd boxes, when nfs transfer is around 1MB/s
for writing and 1.5MB/s for reading
Quoted 1 line
Quoted 1 line
listing user directories even with very few file stakes up to 15
seconds.
Quoted 4 lines
Quoted 2 lines
I tried nfs-ALL patch, but it didn' t help.
How is local performance?
local read/writes are ok
Quoted 1 line
What filesystem do you use?
reiserfs on top of lvm (to be able to get snapshots)
Quoted 2 lines
What's the output from "iostat -x -d -k 60 3"?
below is output from 'iostat -d 60 3' (- my iostat does not take '-k' option,
and '-d' gives me empty output) - mayby this is enough?
Linux 2.4.19 (hostname) 11/05/02
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 16.16 205.52 82.84 3481842 1403528
dev8-1 160.58 1266.39 1346.12 21455002 22805744
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 9.23 134.93 16.13 8096 968
dev8-1 150.92 753.20 1715.87 45192 102952
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 10.02 135.07 26.53 8104 1592
dev8-1 152.78 763.33 1767.20 45800 106032
rafal
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 4 lines · Matt Heaton
each NFS server. So even though our throughput of only 1.5 MB isn't high. The number of files per second is
actually quite high, and causes things to slow down because of seek time issues. PLEASE GIVE US CACHEFS SOMEONE??
How is cachefs going to help? The kernel is already trying to cache data as much as possible. Once you're trying to serve more data than you have RAM, this are naturally going to degreate quite significantly as the system becomes seek bound.
Quoted 3 lines
Does anyone have experience with IDE Raid arrays that get over 250 tps in iostat that work fine? I would
be VERY VERY VERY interested to find out.
Use raid1+0 and you'll be much happier, as read requests will be balanced over multiple drives (mirroring means the same data can be read from all of the mirrors). Additionally, you'll have much lower CPU utilization and writes won't cause all disks in the array to seek for strip updates. Read the archives for the past couple of weeks for another example of the performance increase when going from raid5 to raid1+0.
-ben
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Cachefs will help quite a lot in my opinion because it doesn't just store the files in RAM,
it uses the hard drive. So if you have an NFS client with an extra 5 gig that you can
designate as cache then reads to the NFS server will go down DRAMATICALLY as it will hit local cache on the NFS clients drive.
I agree raid 1+0 should be much faster for writes and a little for read, but RAID 5 still
reads from all drives simultaneously (Has to read parity in too I know), but can read
all 7 drives at once instead of only 4 drives at once in a raid 1+0 configuration with 8 drives
in the array. I have never used 1+0 so I am only talking about physical drive layout rather
than any personal experience. Are my assumptions correct that raid 5 does in fact read
from all drives at the same time? If so, reading might be a LITTLE faster on raid 1+0 than
raid 5, but it shouldn't be HUGE. When I contacted 3ware, they basically said the same thing.
I do agree that writes are MUCH faster on 1+0 than raid 5.
Any thoughts?
L8r...
Matt
Quoted 2 lines · Matt Heaton
Quoted 1 line
each NFS server. So even though our throughput of only 1.5 MB isn't
high.
Quoted 7 lines
Quoted 3 lines
The number of files per second is
actually quite high, and causes things to slow down because of seek time issues. PLEASE GIVE US CACHEFS SOMEONE??How is cachefs going to help? The kernel is already trying to cache data as much as possible. Once you're trying to serve more data than you have RAM, this are naturally going to degreate quite significantly as the
system
Quoted 3 lines
becomes seek bound.
Quoted 1 line
Does anyone have experience with IDE Raid arrays that get over 250 tps
in
Quoted 13 lines
Quoted 2 lines
iostat that work fine? I would
be VERY VERY VERY interested to find out.Use raid1+0 and you'll be much happier, as read requests will be balanced over multiple drives (mirroring means the same data can be read from all of the mirrors). Additionally, you'll have much lower CPU utilization and writes won't cause all disks in the array to seek for strip updates. Read the archives for the past couple of weeks for another example of the performance increase when going from raid5 to raid1+0.
-ben
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
You're wrong. Reads (not sequential, but scattered) on a 1+0 setup will be faster as the raid1 driver optimizes requests somewhat, plus 2 (or more for larger raid1s of disks) drives will be able to service requests for the same stripe offset on different disks. There is no way a raid5 can service two requests for the same stripe offset at different offsets in the array. And even for a read-heavy workload, there are still writes to update metadata (and journal unless you've got a separate journal device), and the impact of those is that *all* reads to the array have to suffer from seeks when even the smallest write is active.
On a side note, make sure you have the filesystem mounted with the noatime flag if you can afford losing atimes.
-ben
Quoted 59 lines · Matt Heaton
Cachefs will help quite a lot in my opinion because it doesn't just store the files in RAM,
it uses the hard drive. So if you have an NFS client with an extra 5 gig that you can
designate as cache then reads to the NFS server will go down DRAMATICALLY as it will hit local cache on the NFS clients drive.I agree raid 1+0 should be much faster for writes and a little for read, but RAID 5 still
reads from all drives simultaneously (Has to read parity in too I know), but can read
all 7 drives at once instead of only 4 drives at once in a raid 1+0 configuration with 8 drives
in the array. I have never used 1+0 so I am only talking about physical drive layout rather
than any personal experience. Are my assumptions correct that raid 5 does in fact read
from all drives at the same time? If so, reading might be a LITTLE faster on raid 1+0 than
raid 5, but it shouldn't be HUGE. When I contacted 3ware, they basically said the same thing.
I do agree that writes are MUCH faster on 1+0 than raid 5.Any thoughts?
L8r...
Matt
Quoted 2 lines · Matt Heaton
Quoted 1 line
each NFS server. So even though our throughput of only 1.5 MB isn't
high.
Quoted 7 lines
Quoted 3 lines
The number of files per second is
actually quite high, and causes things to slow down because of seek time issues. PLEASE GIVE US CACHEFS SOMEONE??How is cachefs going to help? The kernel is already trying to cache data as much as possible. Once you're trying to serve more data than you have RAM, this are naturally going to degreate quite significantly as the
system
Quoted 3 lines
becomes seek bound.
Quoted 1 line
Does anyone have experience with IDE Raid arrays that get over 250 tps
in
Quoted 13 lines
Quoted 2 lines
iostat that work fine? I would
be VERY VERY VERY interested to find out.Use raid1+0 and you'll be much happier, as read requests will be balanced over multiple drives (mirroring means the same data can be read from all of the mirrors). Additionally, you'll have much lower CPU utilization and writes won't cause all disks in the array to seek for strip updates. Read the archives for the past couple of weeks for another example of the performance increase when going from raid5 to raid1+0.
-ben
Signature · 10 lines
"Do you seek knowledge in time travel?" ------------------------------------------------------- This sf.net email is sponsored by: See the NEW Palm Tungsten T handheld. Power & Color in a compact size! http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 7 lines · poczta.dotcom.pl
Quoted 1 line
How is local performance?
=20
local read/writes are ok
=20Quoted 1 line
What filesystem do you use?
=20
reiserfs on top of lvm (to be able to get snapshots)
What reiserfs-version? 3.5 or 3.6?=20
What on-disk format? 3.5 or 3.6?
There should be an entry in the kernel-log from when the filesystem is mounted.=20
Reiserfs-3.5 has some known performance-problems related to NFS - that may be your problem.
Quoted 5 lines
Quoted 2 lines
What's the output from "iostat -x -d -k 60 3"?
below is output from 'iostat -d 60 3' (- my iostat does not take '-k' option,
and '-d' gives me empty output) - mayby this is enough?
And not -x either?
iostat fr=E5 sysstat-4.0.3-2 (from RedHat) includes a "-x" option for extended output. Most importantly, it will tell how much of the time each device is busy.
It's unclear from your posts if this is a io-related problem or not. Unnless you're running reiserfs-3.5 my guess is that it is the IO-performance that is the problem. 3ware 7500 controllers have rather=20 poor performance on RAID5 - especially for writes.=20
If that's correct, then switching to a different RAID-level or replacing the 3ware-controller should solve the problem. SCSI- or FC- RAIDS are easily 10 or 20 times faster than the 3ware RAID for some types of operations, and if you want something cheaper there is also different types of IDE-RAIDs.
--=20
Ragnar Kj=F8rstad
Big Storage
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
----- Original Message -----
From: "Ragnar Kj=F8rstad" <[email protected]>
To: "poczta.dotcom.pl" <[email protected]>
Cc: <[email protected]>
Sent: Wednesday, November 06, 2002 12:32 AM
Subject: Re: [NFS] nfs performance problem
Quoted 11 lines · poczta.dotcom.pl
Quoted 7 lines
Quoted 1 line
How is local performance?
local read/writes are ok
Quoted 1 line
What filesystem do you use?
reiserfs on top of lvm (to be able to get snapshots)
What reiserfs-version? 3.5 or 3.6?
What on-disk format? 3.5 or 3.6?
3.6.25
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
What's the output from "iostat -x -d -k 60 3"?
below is output from 'iostat -d 60 3' (- my iostat does not take '-k=
'
Quoted 4 lines
Quoted 2 lines
option,
and '-d' gives me empty output) - mayby this is enough?And not -x either?
now I have 4.0.6,
iostat -x -k 60 3 - outputs only column titles every 60 seconds but lin= e
where numbers should be is blank,
the same I have on other machine (fresh gentoo linux)
Quoted 3 lines
iostat fr=E5 sysstat-4.0.3-2 (from RedHat) includes a "-x" option for extended output. Most importantly, it will tell how much of the time each device is busy.
anyway local writes are ok
Quoted 5 lines
It's unclear from your posts if this is a io-related problem or not. Unnless you're running reiserfs-3.5 my guess is that it is the IO-performance that is the problem. 3ware 7500 controllers have rather poor performance on RAID5 - especially for writes.
reiserfs 3.6.25, raid 5,
ok, I can understand raid 5 is not fast but getting below 2Mbytes/s is really poor :-(
Quoted 2 lines
If that's correct, then switching to a different RAID-level or replacin=
g
Quoted 5 lines
the 3ware-controller should solve the problem. SCSI- or FC- RAIDS are easily 10 or 20 times faster than the 3ware RAID for some types of operations, and if you want something cheaper there is also different types of IDE-RAIDs.
what kind of IDE-RAID would You suggest?
thanks
rafal mycielski
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 8 lines · myciel
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
What filesystem do you use?
reiserfs on top of lvm (to be able to get snapshots)
What reiserfs-version? 3.5 or 3.6?
What on-disk format? 3.5 or 3.6?=20
3.6.25
That's the reiserfs-version.
The on-disk format must be either "3.5" or "3.6".
There should be a message at mount-time telling you wich one, or you can use "debugreiserfs <device>" to check it.
Quoted 2 lines
Quoted 2 lines
Unnless you're running reiserfs-3.5 my guess is that it is the IO-performance that is the problem. 3ware 7500 controllers have rathe=
r
Quoted 5 lines
Quoted 1 line
poor performance on RAID5 - especially for writes.
=20
reiserfs 3.6.25, raid 5,
ok, I can understand raid 5 is not fast but getting below 2Mbytes/s is really poor :-(
Well, updating a single byte is a very very expensive operation on raid5. I agree that 2MB/s is worse than one should expect though, so there may be something else going on.
The fact that you say local writes are faster could indicate that the problem is not only io-related.=20
Maybe there is some packet-loss? That kills performance on nfs. Is there anything in the kernel-log on the clients to indicate the problem?
Quoted 1 line
Quoted 1 line
If that's correct, then switching to a different RAID-level or replac=
ing
Quoted 6 lines
Quoted 4 lines
the 3ware-controller should solve the problem. SCSI- or FC- RAIDS are easily 10 or 20 times faster than the 3ware RAID for some types of operations, and if you want something cheaper there is also different types of IDE-RAIDs.
=20
what kind of IDE-RAID would You suggest?
A BigStorage IDE-RAID of course :)
I'll get back to you off-list about that.
But it's still not clear if your problems are raid-related or network-related. Possible a combination.
--=20
Ragnar Kj=F8rstad
Big Storage
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 9 lines · myciel
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
What filesystem do you use?
reiserfs on top of lvm (to be able to get snapshots)
What reiserfs-version? 3.5 or 3.6?
What on-disk format? 3.5 or 3.6?
Quoted 7 lines
Quoted 1 line
3.6.25
That's the reiserfs-version.
The on-disk format must be either "3.5" or "3.6".
There should be a message at mount-time telling you wich one, or you can use "debugreiserfs <device>" to check it.
format 3.6 with standard journal
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Unnless you're running reiserfs-3.5 my guess is that it is the IO-performance that is the problem. 3ware 7500 controllers have rather poor performance on RAID5 - especially for writes.
reiserfs 3.6.25, raid 5,
ok, I can understand raid 5 is not fast but getting below 2Mbytes/s is really poor :-(
Quoted 8 lines
Well, updating a single byte is a very very expensive operation on raid5. I agree that 2MB/s is worse than one should expect though, so there may be something else going on.
The fact that you say local writes are faster could indicate that the problem is not only io-related.
Maybe there is some packet-loss?
I don't see any packet losses - at least as I can see with ping big packets like 16k or 32k
Quoted 4 lines
That kills performance on nfs.
Is there anything in the kernel-log on the clients to indicate the problem?
yesterday evening I switched from udp to tcp,
performace is much better but I get a lot of kernel messages in syslog:
Nov 6 12:40:45 intler kernel: RPC request reserved 272 but used 276 Nov 6 12:40:50 intler kernel: RPC request reserved 240 but used 244 Nov 6 12:40:54 intler kernel: RPC request reserved 244 but used 248
what does above mean?
Quoted 5 lines
Quoted 2 lines
what kind of IDE-RAID would You suggest?
A BigStorage IDE-RAID of course :)
I'll get back to you off-list about that.
ok
rafal mycielski
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs