NAS server avalanche overload
Subsystems (2)
We have a NAS server currently running a basically stock 2.4.22 kernel, on a single 2.4GHz Xeon HT with 512MB RAM and 4 SATA 200GB disks on a 3ware 8506-12 in RAID-5.
It serves two machines via switched gigabit (1500 mtu, no switch support) which each run large numbers of User-mode Linux processes. Each kernel (there are ~60 total) has one or more open files at all times, backing the normal ext3 filesystems for these virtual machines. They use copy-on-write such that the main distro files are on local disks and only the differences against this are stored, per machine, on the NAS. /home filesystems and such are flat, sparse files. No attempt is made at this point to turn atime off (but I can patch the kernel to change the default if necessary).
The problem we're having is that every once in a while the entire system grinds to a screeching halt with the load average on the NAS box spiking to 17-18 (with 16 nfsd processes, this means every last one is wedged), which quickly causes the load on the two client machines to spike as requests they're making get stuck. This eventually clears up, but can last anywhere from 15 seconds to 15+ minutes. In the meantime, however, any disk-based operation inside the virtual machines can take a minute or more to complete.
I've been trying for a long time to track this down with no luck, so now it's time to see if anyone here has any ideas.
First major datapoint: early in the debugging cycle a large-ish number of RRD datasets were kept on the NAS box, being updated regularly in an attempt to spot the culprit. This instead made the problem significantly more frequent. Moving the archives to another machine and off NFS entirely immediately trimmed 100-200 I/O's per second average off the NAS box, and the problem eased greatly.
Second: the whole process can easily be replicated by running bonnie++ on any of the machines (the NAS, the client, or a virtual machine), and it appears clearly related to the I/O's per second, but only in cases where I/O's are not linear. *Reading* a huge file either locally or over NFS will cause a very mild form of the overload, but *writing* can cause it almost instantaneously.
I've tried playing around with bdflush parameters, but without a dramatically clearer mental picture of how that whole subsystem works, I have no real chance of coming up with the best direction to move. A gradual search isn't really feasible because the spikes are unpredictable, and artificially generated loads (writing huge files) are *too* stressful to see any differences.
I've graphed this thing utterly to death, and anyone interested in checking it out can see tonight's fiasco at:
http://narsil.pdxcolo.net/graphs/?start=200403022200&duration=1hr
The aforementioned switch away from NAS-based RRD archives can be seen quite easily at:
http://narsil.pdxcolo.net/graphs/?start=20040208&duration=1week
The graph pages are designed for a full 1600x1200 screen (mine), so it may be hard to see everything clearly on smaller screens. Try adding &width=100&height=50 maybe. The most relevant link is the NAS debug page (nasdebug.php?...), which shows more information than the main graphs page.
What I'd like to know is if anyone has any idea what's really going on here, or suggestions as to what other data I might gather that would help diagnose the problem. Easy solutions (add RAM, tweak a sysctl, etc.) would be *greatly* appreciated ;-)
Signature · 14 lines
- Omega aka Erik Walthinsen [email protected] ------------------------------------------------------- SF.Net is sponsored by: Speed Start Your Linux Apps Now. Build and deploy apps & Web services for Linux with a free DVD software kit from IBM. Click Now! http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 2 lines · Greg Banks
Are you using the "async" export option on the server? It causes similar symptoms when used with large NFS writes. Use "sync".
Mount options as reported by /proc/mounts are:
rw,noatime,rsize=3D4096,wsize=3D4096,intr,soft,noac,tcp
I'm pretty sure the default here is async, as I had sync on there earlier and it actually caused a noticeable drop in performance.
What I'm wondering is if the default bdflush settings are putting a hard cap on how much data can be write-cached, forcing the system to block writes too early. With 512MB of RAM, say half available as write-cache, even at the rate of 5MB/sec, we should be able to run for almost a minute with complete disk starvation before things start to wedge. And since this doesn't look like complete starvation at all (graphs show I/O's are completing the whole time), it should last even longer.
If anyone has any ideas on what to tweak in bdflush, it seems that there *is* some pattern in the spikes, with them occurring at 11:25pm and 12:00am every day for at least the last 3 days.
Quoted 1 line · Philippe Gramouli=E9 said:
Is there anything that prevent you from running a 2.4.25 kernel ?
It's a production machine with those 60+ virtual machines running on it, so the only opportunity I have to change anything of this sort is during our quarterly downtime, the next one being early April.
Quoted 4 lines · Jay (John G)
Hi. I have no experience with your particular setup but have had similar problems when our clients were running a pre-2.4.20 kernel and using UDP for the NFS mounts. If that fits your client setup then try either upgrading the kernel or switching to TCP.
We're using TCP, as it also had performance advantages in our early tests.
Quoted 5 lines · David Dougall
My experience is that ext3 is dreadfully slow and RAID5 is dreadfully slow. These 2 combined can cause significant problems. The suggestions that have come from the list before are to change to RAID10 and use another filesystem such as reiserfs or xfs. I saw significant speedup moving away from ext3.
The NAS itself is using reiserfs, only the virtual machines are using ext3. The question there is what kind of read/write load differences one might have between the two. Certainly there's a possibility that the journaling writes have something to do with it, but I wouldn't think they would cluster to the degree things seem to be.
RAID 1+0 is an option with the 8506-12, but the migration is extremely painful. We have to acquire a whole new set of disks (would probably get 6), construct the array, then copy half a TB of data across. Much of the data is sparse files, so the process would take even longer. At least a large chunk of it is non-production files (mirrors), so probably 1/2 to 2/3 can be done without downtime.
- Omega
aka Erik Walthinsen
[email protected]
-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 6 lines · Erik Walthinsen
Quoted 2 lines · Greg Banks
Are you using the "async" export option on the server? It causes similar symptoms when used with large NFS writes. Use "sync".
Mount options as reported by
/proc/mountsare:
rw,noatime,rsize=4096,wsize=4096,intr,soft,noac,tcp
And the export options are? cat /etc/exports on the server.
The noatime option has no effect over NFS. Your [rw]sizes are really quite small, try 8K. Also, try turning off noac.
Quoted 2 lines
I'm pretty sure the default here is async, as I had sync on there earlier and it actually caused a noticeable drop in performance.
So did you get the collapse with sync ?
Quoted 7 lines
What I'm wondering is if the default bdflush settings are putting a hard cap on how much data can be write-cached, forcing the system to block writes too early. With 512MB of RAM, say half available as write-cache, even at the rate of 5MB/sec, we should be able to run for almost a minute with complete disk starvation before things start to wedge. And since this doesn't look like complete starvation at all (graphs show I/O's are completing the whole time), it should last even longer.
The problem I've seen is that the data is written out from the page cache with the BKL held, which prevents any nfsd thread from waking up and responding to incoming requests, and NFS traffic drops to zero. In addition, if any of the nfsd's owned some other lock when this happened, some local processes can be blocked too. This is an inevitable result of the "async" export option.
Quoted 3 lines
If anyone has any ideas on what to tweak in bdflush, it seems that there *is* some pattern in the spikes, with them occurring at 11:25pm and 12:00am every day for at least the last 3 days.
You could try reducing the 1st parameter in /proc/sys/vm/bdflush to say 5 and decrease the 5th parameter by a similar factor. This will activate kupdated more frequently and it will write data out earlier. But, did I mention the "sync" export option?
Greg.
Signature · 12 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI. ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line · Greg Banks
And the export options are? cat
/etc/exportson the server.
/array/01-moria *.nas.pdxcolo.net(rw,no_root_squash,async)
Quoted 2 lines
The noatime option has no effect over NFS. Your [rw]sizes are really quite small, try 8K. Also, try turning off noac.
The rwsizes were set based on a set of experiments with different sizes, with 4k yielding by far the best bandwidth performance. The limiting factor there is that the gig switch we have atm doesn't handle jumbo frames. If increasing the rwsizes will reduce actual IO/sec load, then it's worth trying even if it does reduce max bandwidth. In reality, the theoretical max is almost never approached, so it's probably not a huge deal.
Quoted 1 line
So did you get the collapse with sync ?
The sync/async testing was done before migrating everything to the NAS, and these spikes started showing up as load increased many months later on. Can only try sync after the next downtime coming up in about a month.
Quoted 6 lines
The problem I've seen is that the data is written out from the page cache with the BKL held, which prevents any nfsd thread from waking up and responding to incoming requests, and NFS traffic drops to zero. In addition, if any of the nfsd's owned some other lock when this happened, some local processes can be blocked too. This is an inevitable result of the "async" export option.
That sounds like the kind of scenario I've been imagining. Are there any (stable) patches to get rid of the BKL in this case, or do I have to wait until we move to 2.6 for that? Alternately, would reducing the number of nfsd's help? Since there are only 2 heavy and 2-3 light physical clients, is 16 overkill?
Quoted 4 lines
You could try reducing the 1st parameter in
/proc/sys/vm/bdflushto say 5 and decrease the 5th parameter by a similar factor. This will activate kupdated more frequently and it will write data out earlier. But, did I mention the "sync" export option?
OK, I'll give that a shot and see if it makes a dent in tonight's spike(s). . . . Oddly enough, I just checked the graphs and a spike is going on now, but looks like I caught the tail end only. I made the bdflush changes, but cannot determine whether it's the end of the spike or a bdflush-related termination.
- Omega
aka Erik Walthinsen
[email protected]
-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 4 lines · Erik Walthinsen
Quoted 1 line · Greg Banks
And the export options are? cat
/etc/exportson the server./array/01-moria *.nas.pdxcolo.net(rw,no_root_squash,async)
Aha.
Quoted 6 lines
Quoted 2 lines
The noatime option has no effect over NFS. Your [rw]sizes are really quite small, try 8K. Also, try turning off noac.
The rwsizes were set based on a set of experiments with different sizes, with 4k yielding by far the best bandwidth performance. The limiting factor there is that the gig switch we have atm doesn't handle jumbo frames.
That really shouldn't make a difference; both 4K and 8K IOs will end up being split over multiple ethernet frames.
Quoted 4 lines
If increasing the rwsizes will reduce actual IO/sec load, then it's worth trying even if it does reduce max bandwidth. In reality, the theoretical max is almost never approached, so it's probably not a huge deal.
Generally speaking, larger IOs are more efficient for heavy streaming reads and writes. The best parameters will depend on your workload.
Quoted 8 lines
Quoted 6 lines
The problem I've seen is that the data is written out from the page cache with the BKL held, which prevents any nfsd thread from waking up and responding to incoming requests, and NFS traffic drops to zero. In addition, if any of the nfsd's owned some other lock when this happened, some local processes can be blocked too. This is an inevitable result of the "async" export option.
That sounds like the kind of scenario I've been imagining. Are there any (stable) patches to get rid of the BKL in this case,
Not AFAIK. Trond? It would in any case be a fairly adventurous patch.
Quoted 2 lines
or do I have to
wait until we move to 2.6 for that?
Sorry, I haven't tried this on 2.6 yet.
Quoted 2 lines
Alternately, would reducing the
number of nfsd's help?
No, that will either have no effect or reduce the throughput when things are going well.
Quoted 2 lines
Since there are only 2 heavy and 2-3 light
physical clients, is 16 overkill?
Probably not.
You need more than 1 nfsd per client, because (assuming you don't have the sync *mount* option on the clients) the clients will be issuing multiple (up to 16 for 2.4.x) rpc calls in parallel each. The number that is useful is limited (at least on my machines) by the nfsd's doing time consuming memcpy()s with the BKL; absent this even more nfsds would be useful.
Quoted 6 lines
Quoted 1 line
You could try reducing the 1st parameter in /proc/sys/vm/bdflush[...]
OK, I'll give that a shot and see if it makes a dent in tonight's spike(s). . . . Oddly enough, I just checked the graphs and a spike is going on now, but looks like I caught the tail end only. I made the bdflush changes, but cannot determine whether it's the end of the spike or a bdflush-related termination.
With the bdflush changes I mentioned you'll still get spikes, just hopefully they'll be short enough that you won't notice. What you want to do is get them so short that the clients don't hit a major RPC timeout.
Also, making the changes won't help a spike in progress.
Greg.
Signature · 12 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI. ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
P=E5 on , 03/03/2004 klokka 20:40, skreiv Greg Banks:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The problem I've seen is that the data is written out from the page cache with the BKL held, which prevents any nfsd thread from waking u=
p
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
and responding to incoming requests, and NFS traffic drops to zero. In addition, if any of the nfsd's owned some other lock when this happened, some local processes can be blocked too. This is an inevitable result of the "async" export option.
That sounds like the kind of scenario I've been imagining. Are there any (stable) patches to get rid of the BKL in this case,=20
=20
Not AFAIK. Trond? It would in any case be a fairly adventurous patch.
I'm not aware of anything, but this is more of a question for Neil.
I *really* doubt anyone has bothered to do it for 2.4.x...
Cheers,
Trond
-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 11 lines · Greg Banks
Quoted 3 lines
Alternately, would reducing the
number of nfsd's help?No, that will either have no effect or reduce the throughput when things are going well.
Quoted 2 lines
Since there are only 2 heavy and 2-3 light
physical clients, is 16 overkill?Probably not.
Never seen a situation were less nfs threads was useful but I haven't worked much with Linux as a server.
Quoted 7 lines
You need more than 1 nfsd per client, because (assuming you don't have the sync *mount* option on the clients) the clients will be issuing multiple (up to 16 for 2.4.x) rpc calls in parallel each. The number that is useful is limited (at least on my machines) by the nfsd's doing time consuming memcpy()s with the BKL; absent this even more nfsds would be useful.
Bumping the number of threads to 32 or 64 or more would be the first thing I would try.
Ian
-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 2 lines · Ian Kent
Bumping the number of threads to 32 or 64 or more would be the first thing I would try.
Only problem is that if the spike occurs anyway, the load will be 33 or 65 instead of 17. I don't know if it's ctxswitching itself to death while all these nfsd's are blocking, but if so, doubling or quadrupling the number of blocked processes will just make the spike worse.
Also, I've learned to be extremely careful about messing with nfsd while clients are connected. I've had cases where just running exportfs *seems* to have permanently killed the client's session, yet the client of course still has the filesystem mounted. This required a complete *hard* kill of every UML instance on the client, which did a significant amount of damage to the filesystems that were mounted at the time.
- Omega
aka Erik Walthinsen
[email protected]
-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 8 lines · Erik Walthinsen
Quoted 2 lines · Ian Kent
Bumping the number of threads to 32 or 64 or more would be the first thing I would try.
Only problem is that if the spike occurs anyway, the load will be 33 or 65 instead of 17. I don't know if it's ctxswitching itself to death while all these nfsd's are blocking,
If it's the same as I've seen, an nfsd will be spinning in schedule() trying to reacquire the BKL. Changing the number of nfsds is not going to affect the amount of CPU wasted (i.e. pretty much all of it) just the number of runnable processes.
Greg.
Signature · 12 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI. ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 8 lines · Erik Walthinsen
Quoted 2 lines · Ian Kent
Bumping the number of threads to 32 or 64 or more would be the first thing I would try.
Only problem is that if the spike occurs anyway, the load will be 33 or 65 instead of 17. I don't know if it's ctxswitching itself to death while all these nfsd's are blocking, but if so, doubling or quadrupling the number of blocked processes will just make the spike worse.
I'm assuming there is not a problem with the NFS implementation of course. Never the less you generally need many more than 16 threads for a busy NFS server.
Quoted 3 lines
Also, I've learned to be extremely careful about messing with nfsd while clients are connected. I've had cases where just running exportfs
I wouldn't change this on the fly either. I'm suggesting this type of change be made when during scheduled downtime.
Ian
-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs