knfsd: miscellaneous performance-related fixes
Subsystems (3)
PATCH 1/1 knfsd: miscellaneous performance-related fixes
G'day,
Five more patches which fix various issues found in knfsd, mostly directly performance-related.
1 of 5 knfsd: make readahead params cache SMP-friendly
the racache is a global contention point and performance bottleneck in
some read-dominated workloads, alleviate that
2 of 5 knfsd: cache ipmap per TCP socket
for TCP we can avoid doing a hash lookup on the ip_map cache for every RPC call
3 of 5 knfsd: avoid nfsd CPU scheduler overload
avoid enormous load averages and CPU scheduler overload with workloads which generate extremely high call rates
4 of 5 knfsd: add RPC pool thread stats
add some statistics to the svc_pool
5 of 5 knfsd: remove nfsd threadstats
remove the 'th' stats line from /proc/net/rc/nfsd, it's a global contention point in high call rate workloads
Greg.
Signature · 13 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 4 lines · On Tuesday August 8, [email protected] wrote:
G'day,
Five more patches which fix various issues found in knfsd, mostly directly performance-related.
Thanks. I'll have a bit more of a look in a day or two, but just quickly:
Quoted 5 lines
1 of 5 knfsd: make readahead params cache SMP-friendly
the racache is a global contention point and performance
bottleneck in some read-dominated workloads, alleviate that
Looks good, though might become irrelevant if the 'new' readahead code goes ahead (the racache can just be discarded).
Quoted 3 lines
2 of 5 knfsd: cache ipmap per TCP socket
for TCP we can avoid doing a hash lookup on theip_mapcache for every RPC call
Looks sane, but I'm wondering why:
+static inline int cache_valid(struct cache_head *h) +{
+ return (h->expiry_time != 0 && test_bit(CACHE_VALID, &h->flags));
+}
+
we need to test expiry_time here. Is not the test on CACHE_VALID enough?
Quoted 4 lines
3 of 5 knfsd: avoid nfsd CPU scheduler overload
avoid enormous load averages and CPU scheduler overload with workloads which generate extremely high call rates
Not sure I quite understand all the implications of this yet. There would have to be lots of active sockets because each socket can only be waking one thread. But that is entirely possible with TCP. So a request comes in on a TCP socket and we decide not to queue it to a process just yet, so it stays on the pool queue.
That means it doesn't get handled until some process in the queue finishes it's request.
These seems to suggest quite some room for unfairness on request handling.
That may not be a problem, but I'm not sure...
Maybe when one thread wakes up it should kick the next one to wake up???
Or am I missing something.
Quoted 3 lines
4 of 5 knfsd: add RPC pool thread stats
add some statistics to thesvc_pool
Could we get a paragraph or 3 explaining how to interpret these stats? And we have to do something about svc_put. With this patch as it stands, if you:
write some sockets to 'portlist' (with no threads running) look at the pool stats
try to start thread
the ports will have been forgotten because looking at the pool stats destroyed the ports...
Quoted 5 lines
5 of 5 knfsd: remove nfsd threadstats
remove the 'th' stats line from/proc/net/rc/nfsd, it's a global contention point in high call rate workloads
Yes..... I can appreciate all the problems that you identify. But before removing this I would like to have something to replace it. Some that gives some sort of vague idea about whether you have a suitable number of threads or not.
Does your pool stats provide that at all?
What I would really like it auto-scaling of the number of threads to match the load. I wonder if that is a good idea?
NeilBrown
Quoted 5 lines
Greg.
Signature · 2 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI.
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 7 lines · Neil Brown
Quoted 4 lines · On Tuesday August 8, [email protected] wrote:
G'day,
Five more patches which fix various issues found in knfsd, mostly directly performance-related.
Thanks. I'll have a bit more of a look in a day or two,
Suddenly we've all got a lot of reading to do ;-)
Quoted 11 lines
but just
quickly:Quoted 5 lines
1 of 5 knfsd: make readahead params cache SMP-friendly
the racache is a global contention point and performance
bottleneck in some read-dominated workloads, alleviate thatLooks good, though might become irrelevant if the 'new' readahead code goes ahead (the racache can just be discarded).
I haven't looked at the new readahead code. Does it no longer store anything attached to struct file?
Quoted 13 lines
Quoted 4 lines
2 of 5 knfsd: cache ipmap per TCP socket
for TCP we can avoid doing a hash lookup on theip_mapcache for every RPC callLooks sane, but I'm wondering why:
(quoted diff) +5
-0+static inline int cache_valid(struct cache_head *h) +{ + return (h->expiry_time != 0 && test_bit(CACHE_VALID, &h->flags)); +} +
Quoted 2 lines
we need to test
expiry_timehere. Is not the test onCACHE_VALIDenough?
Hmm, it seemed necessary when I wrote that line.
Looking at current cache.c code, when a cache entry is replaced with a new one in sunrpc_cache_update(), the old one is left in a state where CACHE_VALID is set but h->expiry_time == 0. It's in that state while it's waiting for the last ref to be dropped.
Before the patch, that state was transient and we would soon after drop that reference. After the patch we keep a ref alive for a long time attached to the svc_sock, so if the ip_map enters that state we need to drop that ref and lookup again. That was why I needed to check for both the VALID flag and a zero expiry time.
The test case that caused the addition of that logic was a 2nd mount from the same IP address. In the codebase before your cache rewrite, that would cause ip_map_lookup() to replace the existing valid ip_map as described. I haven't followed the current logic of ip_map_lookup() to see what it does, but it does still seem to be an issue.
Quoted 7 lines
Quoted 3 lines
3 of 5 knfsd: avoid nfsd CPU scheduler overload
avoid enormous load averages and CPU scheduler overload with workloads which generate extremely high call ratesNot sure I quite understand all the implications of this yet. There would have to be lots of active sockets because each socket can only be waking one thread. But that is entirely possible with TCP.
It's real easy when you have 200 active clients on TCP. It's a whole lot easier when you have 2000.
Quoted 7 lines
So a request comes in on a TCP socket and we decide not to queue it to a process just yet, so it stays on the pool queue.
That means it doesn't get handled until some process in the queue finishes it's request.
These seems to suggest quite some room for unfairness on request handling.
That may not be a problem, but I'm not sure...
I don't think it changes the order in which sockets get serviced, just how many we try to do at the same time. Maybe I've missed some subtle side effect.
Quoted 2 lines
Maybe when one thread wakes up it should kick the next one to wake up???
?? Not sure what you mean.
We want to keep the number of threads woken to a minimum. Ideally, a small subset (around 5) of the threads on a pool are hot in cache and handle all the requests when the calls are quickly satisfied from memory, and the others kick in only when calls need to block on disk traffic.
Quoted 7 lines
Or am I missing something.
Quoted 3 lines
4 of 5 knfsd: add RPC pool thread stats
add some statistics to thesvc_poolCould we get a paragraph or 3 explaining how to interpret these stats?
For each pool, we see
id <poolid>
ps <packets> <socks-queued> <threads-woken> <overloads-avoided> <threads-timedout>
<poolid>
the pool id, i.e. 0..npools-1
<packets>
how many times more data arrived on an NFS socket (more precisely, how many times svc_sock_enqueue was called)
<socks-queued>
how many times a socket was queued, because there were not enough nfsd threads to service it
<threads-woken>
how many times an nfsd thread was woken to handle a socket
<overloads-avoided>
how many times it was necessary not to wake a thread, because too many threads had been woken recently for the cpus in this pool to run them
<threads-timedout>
how many times a thread timed out waiting for a socket to need handling
Quoted 14 lines
And we have to do something about
svc_put. With this patch as it stands, if you:
write some sockets to 'portlist' (with no threads running) look at the pool stats
try to start thread
the ports will have been forgotten because looking at the pool stats destroyed the ports...Quoted 5 lines
5 of 5 knfsd: remove nfsd threadstats
remove the 'th' stats line from/proc/net/rc/nfsd, it's a global contention point in high call rate workloads
Quoted 5 lines
Yes..... I can appreciate all the problems that you identify. But before removing this I would like to have something to replace it. Some that gives some sort of vague idea about whether you have a suitable number of threads or not.
Does your pool stats provide that at all?
This patch, no. Probably the most useful stat in this patch is the packets per pool, which gives you some idea of how balanced across pools your traffic is.
I'm still working on another patch (it only recently became possible with the ktime_t code) which replaces threadstats fully. That patch adds nanosecond resolution busy and idle counters to each thread, then aggregates them per-pool when you read the pool_stats file. After export to userspace and rate conversion, these numbers tell us directly what percentage of the nfsds are currently being used.
That patch worked on two platforms some weeks ago, but I'm not sure it's ready for primetime. I can post it for comment if you like?
Quoted 2 lines
What I would really like it auto-scaling of the number of threads to match the load. I wonder if that is a good idea?
Yes it is.
The original purpose of the pool_stats file was to drive a userspace balancing daemon to do that, and I have written most of such a daemon. The trouble is that measuring demand for nfsds from what meagre stats it was possible to measure in a 2.6.9 kernel proved difficult. The new pool idle counter provides a direct measure of the excess nfsd capacity, which it will be then easy to write a control loop for.
Greg.
Signature · 13 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 11 lines · On Tuesday August 8, [email protected] wrote:
Quoted 7 lines · Neil Brown
Quoted 4 lines · On Tuesday August 8, [email protected] wrote:
G'day,
Five more patches which fix various issues found in knfsd, mostly directly performance-related.
Thanks. I'll have a bit more of a look in a day or two,
Suddenly we've all got a lot of reading to do ;-)
I've taken to reading kids fantasy novels recently. Nice and light, yet still entertaining (e.g. The Quentaris Chronicles).
Quoted 10 lines
I'm still working on another patch (it only recently became possible with the
ktime_tcode) which replaces threadstats fully. That patch adds nanosecond resolution busy and idle counters to each thread, then aggregates them per-pool when you read thepool_statsfile. After export to userspace and rate conversion, these numbers tell us directly what percentage of the nfsds are currently being used.That patch worked on two platforms some weeks ago, but I'm not sure it's ready for primetime. I can post it for comment if you like?
Please!
I always feel a bit guilty complaining about a patch that has been thoroughly tested and polished. Seeing things early removed the guilt :-)
Quoted 4 lines
Quoted 3 lines
What I would really like it auto-scaling of the number of threads to match the load. I wonder if that is a good idea?
Quoted 8 lines
Yes it is.
The original purpose of the
pool_statsfile was to drive a userspace balancing daemon to do that, and I have written most of such a daemon. The trouble is that measuring demand for nfsds from what meagre stats it was possible to measure in a 2.6.9 kernel proved difficult. The new pool idle counter provides a direct measure of the excess nfsd capacity, which it will be then easy to write a control loop for.
sounds grand!
NeilBrown
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 8 lines · Neil Brown
Quoted 7 lines · On Tuesday August 8, [email protected] wrote:
I'm still working on another patch (it only recently became possible with the
ktime_tcode) which replaces threadstats fully. That patch adds nanosecond resolution busy and idle counters to each thread, then aggregates them per-pool when you read thepool_statsfile. After export to userspace and rate conversion, these numbers tell us directly what percentage of the nfsds are currently being used.
Quoted 6 lines
Quoted 2 lines
That patch worked on two platforms some weeks ago, but I'm not sure it's ready for primetime. I can post it for comment if you like?
Please!
I always feel a bit guilty complaining about a patch that has been thoroughly tested and polished. Seeing things early removed the guilt :-)
Ok, for comment only...this patch won't apply for several reasons but hopefully you can get the gist of what I was trying to do. --
knfsd: Maintain per-thread nanosecond resolution counters for time nfsd threads spend idle (i.e. queued on the pool or waking up) and busy (doing cpu work to service requests or blocking waiting for disk I/O). Accumulate those counter per-pool on demand and export them via /proc/fs/nfsd/pool_stats.
The expectation is that userspace will rate-convert the counters from nanoseconds to nanosec-per-sec and then scale down to a multiple percentage, by analogy with CPU utilisation stats. For example, a 4 CPU machine can report:
150% user 100% sys 50% intr 100 % idle
Where the total is Ncpus * 100%. By analogy, a machine with 4 nfsd threads can report:
300% busy 100% idle
(where the total is Nthreads * 100%) meaning that on average 3 of the threads are doing useful NFS work and 1 is not being used.
By presenting thread usage as just two numbers, we make it easy for a sysadmin to understand whether we have too many threads for the currently running workload. For example, a machine with 16 threads which reports 327% busy 1273% idle has about 12 threads more than it needs to satisfy the workload.
Also, the %idle number is potentially an ideal input for an automatic control loop. That control loop would attempt to keep the %idle number at some predetermined small level (I like 100%), by adding new threads (if the %idle falls too low) or removing threads (if %idle climbs too high).
Please consider this patch a beta release only.
---
fs/nfsd/stats.c | 25 ++++++++++++------ include/linux/sunrpc/svc.h | 14 ++++++++++ net/sunrpc/svc.c | 17 ++++++++++++ net/sunrpc/svcsock.c | 45 +++++++++++++++++++++++++++++++--- 4 files changed, 89 insertions(+), 12 deletions(-)
fs/nfsd/stats.c +16 -9
@@ -423,16 +423,25 @@ void nfsd_stats_update_op(struct svc_rqs } } -static inline ktime_t ktime_get (void ) +static inline ktime_t svc_elapsed_ktime (struct svc_rqst *rqstp,int newstate) { -struct timespec ts; -ktime_get_ts (&ts); -return timespec_to_ktime (ts); +ktime_t now, elapsed; +int oldstate = rqstp->rq_state; + + now =ktime_get (); + elapsed =ktime_sub (now, rqstp->rq_timestamp); + rqstp->rq_timestamp = now; + + rqstp->rq_times[oldstate] =ktime_add (rqstp->rq_times[oldstate], + elapsed); + rqstp->rq_state = newstate; + +return elapsed; }void nfsd_stats_pre (struct svc_rqst *rqstp) { - rqstp->rq_timestamp =ktime_get (); +svc_elapsed_ktime (rqstp,/*going busy*/ 1 ); rqstp->rq_export_stats =NULL ; rqstp->rq_client_stats =NULL ; }
@@ -458,15 +467,13 @@ static inline int time_bucket(const ktimvoid nfsd_stats_post (struct svc_rqst *rqstp) {int tb =-1 ; -ktime_t now, svctime; +ktime_t svctime;if (rqstp->rq_export_stats ==NULL && rqstp->rq_client_stats ==NULL )return ;/* calculate service time and update the stats */ - now =ktime_get (); - svctime =ktime_sub (now, rqstp->rq_timestamp); - rqstp->rq_timestamp = now; + svctime =svc_elapsed_ktime (rqstp,/*going idle*/ 0 ); tb =time_bucket (svctime);
include/linux/sunrpc/svc.h +14 -0
@@ -30,6 +30,11 @@ struct svc_pool_stats {unsigned long threads_woken;unsigned long overloads_avoided;unsigned long threads_timedout; +ktime_t dead_times[2 ];/* Counts accumulated idle +* and busy times for threads +* which have died, to avoid +* the /proc counters going +* backward. */ };/*
@@ -251,6 +256,8 @@ struct svc_rqst {wait_queue_head_t rq_wait;/* synchronization */ struct task_struct *rq_task;/* service thread */ int rq_waking;/* 1 if thread is being woken */ +int rq_state;/* 0=idle 1=busy */ +ktime_t rq_times[2 ];/* cumulative time spent idle,busy */ ktime_t rq_timestamp;/* time of last idle<->busy transition */ struct nfsd_stats_hentry *rq_export_stats;struct nfsd_stats_hentry *rq_client_stats;
@@ -440,5 +447,12 @@ static inline struct svc_pool *svc_pool_ #endif } +static inline ktime_t ktime_get (void ) +{ +struct timespec ts; +ktime_get_ts (&ts); +return timespec_to_ktime (ts); +} +#endif /* SUNRPC_SVC_H */
net/sunrpc/svcsock.c +42 -3
61 lines not on this page · raw message
net/sunrpc/svc.c +17 -0
@@ -395,6 +395,8 @@ __svc_create_thread(svc_thread_fn func,spin_unlock_bh (&pool->sp_lock); rqstp->rq_server = serv; rqstp->rq_pool = pool; + rqstp->rq_state =0 ;/* idle */ + rqstp->rq_timestamp =ktime_get ();#if SVC_HAVE_MULTIPLE_POOLS if (serv->sv_nrpools >1 )
@@ -540,12 +542,27 @@ svc_exit_thread(struct svc_rqst *rqstp) {struct svc_serv *serv = rqstp->rq_server;struct svc_pool *pool = rqstp->rq_pool; +ktime_t now;svc_release_buffer (rqstp);kfree (rqstp->rq_resp);kfree (rqstp->rq_argp);kfree (rqstp->rq_auth_data); + +/* interpolate time since last state change */ + now =ktime_get (); + rqstp->rq_times[rqstp->rq_state] =ktime_add ( + rqstp->rq_times[rqstp->rq_state], +ktime_sub (now, + rqstp->rq_timestamp));spin_lock_bh (&pool->sp_lock); +/* remember accumulated time in dead_times */ + pool->sp_stats.dead_times[0 ] =ktime_add ( + pool->sp_stats.dead_times[0 ], + rqstp->rq_times[0 ]); + pool->sp_stats.dead_times[1 ] =ktime_add ( + pool->sp_stats.dead_times[1 ], + rqstp->rq_times[1 ]);list_del (&rqstp->rq_all);spin_unlock_bh (&pool->sp_lock);kfree (rqstp);
Greg.
Signature · 13 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 3 lines · On Wednesday August 9, [email protected] wrote:
Ok, for comment only...this patch won't apply for several reasons but hopefully you can get the gist of what I was trying to do.
Thanks. It does give a good gist.
Trying to remember back to why I did the current stats the way I did, and comparing with this, the big difference seems to be how burst-behaviour is recognised.
By counting only idle and busy time, your stats would not be able to distinguish between a load of (say) 10 requests each requiring 1ms arriving at once, and those same ten arriving at 1ms intervals.
The first scenario can benefit from having 10 threads (lower latency). The second would not.
My stats might have been able to show that but reporting that some times 10 threads were used.
Is that an issue? I'm not certain, but I have a feeling that NFS traffic is likely to be fairly bursty. In the interests of minimising latency I think we want to auto-configure to be ready to catch that burst.
I imagine the ideal auto-configure regime would be to ramp up the number of threads fairly quickly on demand, and then have them slowly drop away when the demand isn't there.
The minimal stat I would want would be something like a count of the number of times that 'svc_sock_enqueue' found that pool->sp_threads was list_empty. While that number is increasing we add threads at some set rate (1 per second?). When it stops increasing we drop threads very slowly (1 per hour?).
Possibly the %idle number could be used to change the rate of rise/decline. But I'm not convinced that it can be used all by itself.
NeilBrown
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 7 lines · Neil Brown
Quoted 3 lines · On Wednesday August 9, [email protected] wrote:
Ok, for comment only...this patch won't apply for several reasons but hopefully you can get the gist of what I was trying to do.
Thanks. It does give a good gist.
Quoted 7 lines
Trying to remember back to why I did the current stats the way I did, and comparing with this, the big difference seems to be how burst-behaviour is recognised.
By counting only idle and busy time, your stats would not be able to distinguish between a load of (say) 10 requests each requiring 1ms arriving at once, and those same ten arriving at 1ms intervals.
That's right, my busy+idle stats effectively integrate the time spent servicing requests after they're dequeued, so time spent limited by availability of nfsds is not measured.
However the sockets-queued stat from my other stats patch lets you detect that situation.
My first stab at automatic control was to use sockets-queued, expressed as %age of incoming packets, to decide when to increase the number of nfsds. To decide when to decrease, I tweaked the svc_recv() timeout down to 30 seconds and measured the rate at which those occur as a %age of packets arriving. Then, it got...interesting (see below).
Quoted 2 lines
The first scenario can benefit from having 10 threads (lower latency). The second would not.
Yes, this is one aspect of the burst problem.
Quoted 2 lines
Is that an issue? I'm not certain, but I have a feeling that NFS traffic is likely to be fairly bursty.
Yes, burstiness is a problem for the automatic control algorithm when there are a small to medium number of clients (with large numbers of clients, the burstiness tends to average out).
To avoid oscillations you need damping, and that limits your ability to respond to the start of bursts. So you need to configure spare capacity to handle bursts in the short term. You also need some hysteresis for the long term, so that bursts ratchet up the number of threads and the second burst has the capacity it needs.
But this is true no matter which metrics you use to make your control decisions.
Quoted 3 lines
In the interests of minimising
latency I think we want to auto-configure to be ready to catch that burst.
Agreed.
Quoted 3 lines
I imagine the ideal auto-configure regime would be to ramp up the number of threads fairly quickly on demand, and then have them slowly drop away when the demand isn't there.
Precisely.
On Irix, the documented behaviour of the default tunables is:
* limit the rate of thread creation so that a pool-equivalent can go from minimum threads (1) to maximum threads (32) in 1 second.
* threads die when they have been completely idle for 65 seconds.
These were the settings chosen to be most generally useful across the widest range of workloads.
Quoted 3 lines
The minimal stat I would want would be something like a count of the number of times that '
svc_sock_enqueue' found thatpool->sp_threadswaslist_empty.
Sockets-queued in the "add pool stats" patch.
Quoted 3 lines
While that number is increasing we add threads at
some set rate (1 per second?). When it stops increasing we drop threads very slowly (1 per hour?).
This is very close to what I originally planned, and is similar to Irix' documented behaviour.
The problem with this approach is accounting for the case of a sustained heavy load. Imagine that for many consecutive control intervals, more packets arrive than can be handled by the number of CPUs available. Socket queues grow until an equilibrium is reached when TCP closes windows sufficiently that clients can't send any more calls.
In this situation the machine is hitting the "queuing sockets because no threads are available" case on every single packet. The algorithm you describe thus decides to add a thread every time it makes a control decision, and the number of threads grows until the load stops or we reach the limit on the number of threads.
Those new threads are not actually helping because there isn't enough CPU to run them. In fact, without the "avoid CPU overload" patch the machine gets very sick because CPU run queues are enormous and all the spare CPU time goes into the scheduler.
On Linux, each of these new threads is now allocating 1MiB+2pages, at exactly the same time when memory is under pressure from all the queued skbuffs.
To avoid this situation, the control loop decision needs to be made more complex (this is why I added an overloads-avoided metric to the pool stats). But then you end up juggling several different metrics trying to figure out whether the machine is idle, or in one of the possible equilibrium states, or in a burst, or in sustained demand increase, or what.
In short, it gets messy and I was hoping using %idle would make it clean.
Quoted 3 lines
Possibly the %idle number could be used to change the rate of rise/decline. But I'm not convinced that it can be used all by itself.
I haven't done all the experiments yet, so I can't comment.
I will say that the busy/idle stats are one useful way of measuring how well an automatic control loop is working. The other would be to simulate a bursty client and measure average latency vs burst length at the client (I haven't done this in my client yet).
BTW, the problem that has me worried about the busy/idle stats is the efficiency and scalability of getting the monotonic clock sample on 32bit x86 platforms. Current measurements indicate a major problem there, and I might need to try to tweak the HPET driver before the busy/idle stats are usable.
I have the skeleton of a user-space control loop written, I can post that if you like.
Greg.
Signature · 13 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 6 lines · Greg Banks
I'm still working on another patch (it only recently became possible with the
ktime_tcode) which replaces threadstats fully. That patch adds nanosecond resolution busy and idle counters to each thread, then aggregates them per-pool when you read thepool_statsfile. After export to userspace and rate conversion, these numbers tell us directly what percentage of the nfsds are currently being used.
Would it be possible to use that to continue to support the existing interface (the "th" line)?
It'd be nice if we could avoid ripping out a working user interface that someone might be using....
--b.
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 10 lines · J. Bruce Fields
Quoted 6 lines · Greg Banks
I'm still working on another patch (it only recently became possible with the
ktime_tcode) which replaces threadstats fully. That patch adds nanosecond resolution busy and idle counters to each thread, then aggregates them per-pool when you read thepool_statsfile. After export to userspace and rate conversion, these numbers tell us directly what percentage of the nfsds are currently being used.Would it be possible to use that to continue to support the existing interface (the "th" line)?
No, the formats are really quite different.
Quoted 2 lines
It'd be nice if we could avoid ripping out a working user interface that someone might be using....
I agree, but I don't believe a) it's working or b) anyone could be getting any use out of it.
An earlier version of the patch left the data structures in place and just reported them as zeros in the /proc file. Would that be preferrable?
Greg.
Signature · 13 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 6 lines · Greg Banks
Quoted 2 lines · J. Bruce Fields
It'd be nice if we could avoid ripping out a working user interface that someone might be using....
I agree, but I don't believe a) it's working or b) anyone could be getting any use out of it.
OK. So the advice at
http://nfs.sourceforge.net/nfs-howto/ar01s05.html#nfsd_daemon_instances
is wrong?
Quoted 3 lines
An earlier version of the patch left the data structures in place and just reported them as zeros in the /proc file. Would that be preferrable?
I'm not sure that makes any difference.
--b.
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 5 lines · J. Bruce Fields
Quoted 4 lines · Greg Banks
Quoted 2 lines · J. Bruce Fields
It'd be nice if we could avoid ripping out a working user interface that someone might be using....
Quoted 8 lines
Quoted 2 lines
I agree, but I don't believe a) it's working or b) anyone could be getting any use out of it.
OK. So the advice at
http://nfs.sourceforge.net/nfs-howto/ar01s05.html#nfsd_daemon_instances
is wrong?
Let's see.
Quoted 5 lines
Most startup scripts, Linux and otherwise, start 8 instances of nfsd. In the early days of NFS, Sun decided on this number as a rule of thumb, and everyone else copied. There are no good measures of how many instances are optimal, but a more heavily-trafficked server may require more.
Correct.
Quoted 2 lines
You should use at the very least one daemon per processor, but four to eight per processor may be a better rule of thumb.
Wrong. This rule might work up to 4 or 8 cpus, but only by coincidence. Try running knfsd on a 512 cpu machine; you don't need anything like 512 to 4096 nfsd threads. A better rule of thumb would be 1 to 4 nfsds per simultaneously active client. Of course that number is a lot harder to measure with the server as it stands today.
Quoted 7 lines
If you are using a 2.4 or higher kernel and you want to see how heavily each nfsd thread is being used, you can look at the file
/proc/net/rpc/nfsd. The last ten numbers on the th line in that file indicate the number of seconds that the thread usage was at that percentage of the maximum allowable. If you have a large number in the top three deciles, you may wish to increase the number of nfsd instances.
This is true, except that
1. the numbers are undercounted (the mechanism tends to err towards incrementing a lower bucket), and
2. the numbers are never reset and are scaled to the number of nfsd daemons, so to tell whether your change in the number of nfsds was helpful you need to reload the nfsd module or reboot.
Quoted 2 lines
This is done upon starting nfsd using the number of instances as the command line option,
Or by echoing a number into /proc/fs/nfsd/threads.
Quoted 3 lines
and is specified in the NFS startup script (
/etc/rc.d/init.d/nfson Red Hat) as RPCNFSDCOUNT. See the nfsd(8) man page for more information.
On SUSE the file is /etc/sysconfig/nfs and the variable is USE_KERNEL_NFSD_NUMBER.
Quoted 5 lines
Quoted 3 lines
An earlier version of the patch left the data structures in place and just reported them as zeros in the /proc file. Would that be preferrable?
I'm not sure that makes any difference.
Fair enough.
Greg.
Signature · 13 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs