XFS thread inflation in 2.6.23rc
In 2.6.23rc I have a new kernel thread running from XFS:
30137 ? S< 0:00 [xfs_mru_cache]
Is that one really needed? Can it be started only on demand when that MRU feature is used?
Thanks,
-Andi
Quoted 7 lines · Andi Kleen
In 2.6.23rc I have a new kernel thread running from XFS:
30137 ? S< 0:00 [
xfs_mru_cache]Is that one really needed? Can it be started only on demand when that MRU feature is used?
It uses a single threaded workqueue for reaping objects and the thread comes along with that. Creating the workqueue on demand would require creating a kernel thread inside a transaction and that's not some thing we want to do. It can't really be put into an existing thread/workqueue because of deadlock problems which is why it has it's own workqueue....
Besides, what's the point of having nice constructs like dedicated workqueues if people complain when they get used to solve problems?
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Quoted 4 lines · On Wednesday 08 August 2007 14:13:59 David Chinner wrote:
Quoted 3 lines · Andi Kleen
In 2.6.23rc I have a new kernel thread running from XFS:
Quoted 8 lines
Quoted 4 lines
30137 ? S< 0:00 [
xfs_mru_cache]Is that one really needed? Can it be started only on demand when that MRU feature is used?
It uses a single threaded workqueue for reaping objects and the thread comes along with that. Creating the workqueue on demand would require creating a kernel thread inside a transaction and that's not some thing we want to do.
Why not? I can't think of any possible problem except memory allocation recursion, but even that should be handled.
Quoted 2 lines
Besides, what's the point of having nice constructs like dedicated workqueues
It's a resource that shouldn't be overused.
Especially for such a obscure feature -- i remember reviewing your rationale for the MRU cache and the probability of this applying to 99.9+% of users ever is pretty small. If you insist adding such things make them as least as unobtrusive as possible.
Quoted 1 line
if people complain when they get used to solve problems?
Does XFS really need that many threads? Seems doubtful to me.
Ok part of the problem is that the workqueues are a little dumb. e.g. it's highly doubtful per SMT thread workqueues really make any sense. It would be probably enough to have one per socket or one per node. But that's a separate issue from just gratuitously adding new ones.
-Andi
Quoted 10 lines · Andi Kleen
Quoted 9 lines · On Wednesday 08 August 2007 14:13:59 David Chinner wrote:
Quoted 7 lines · Andi Kleen
In 2.6.23rc I have a new kernel thread running from XFS:
30137 ? S< 0:00 [
xfs_mru_cache]Is that one really needed? Can it be started only on demand when that MRU feature is used?
Quoted 6 lines
Quoted 3 lines
It uses a single threaded workqueue for reaping objects and the thread comes along with that. Creating the workqueue on demand would require creating a kernel thread inside a transaction and that's not some thing we want to do.
Why not? I can't think of any possible problem except memory allocation recursion, but even that should be handled.
Memory allocation failure + dirty transaction == filesystem shutdown.
Bad, bad, bad, bad.
Quoted 4 lines
Quoted 2 lines
Besides, what's the point of having nice constructs like dedicated workqueues
It's a resource that shouldn't be overused.
A workqueue + thread uses, what, 10-15k of memory? That's the cost of about 10 cached inodes. It is insignificant...
Quoted 4 lines
Especially for such a obscure feature -- i remember reviewing your rationale for the MRU cache and the probability of this applying to 99.9+% of users ever is pretty small. If you insist adding such things make them as least as unobtrusive as possible.
Sure, it was written for a small market, but it's useful for other workloads. e.g. got a heavy tmp file load? Mark /tmp as a filestream directory and all your temp files end up in one location on disk and don't pollute and fragment everything else. Got a PVR that records multiple streams at once? I wouldn't reject this out of hand as not useful. It was created for the specific purpose of workload isolation and that can be used in many different ways....
Quoted 3 lines
Quoted 1 line
if people complain when they get used to solve problems?
Does XFS really need that many threads? Seems doubtful to me.
XFS is much more threaded than any other linux filesystem. It does lots of stuff asynchronously and that's where the threads and workqueues come in. They fit in very closely with the nature of XFS which is to do as much stuff concurrently as possible on as many CPUs as possible.
Hmmm. I guess you are really not going to like the patch I have that moves the AIL pushing to a new thread to solve some of scalability issues in the transaction subsystem......
Quoted 4 lines
Ok part of the problem is that the workqueues are a little dumb. e.g. it's highly doubtful per SMT thread workqueues really make any sense. It would be probably enough to have one per socket or one per node.
Agreed, but that is a workqueue implementation detail, not a reason for preventing ppl from using workqueues.
Quoted 1 line
But that's a separate issue from just gratuitously adding new ones.
Call it what you will. Threads and workqueues are there to be used and they cost very little for the flexibility and concurrency they provide us.
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Quoted 1 line · On Wednesday 08 August 2007 15:14:04 David Chinner wrote:
Memory allocation failure + dirty transaction == filesystem shutdown.
You mean if the workqueue creation would fail?
Surely not having the MRU cache is not a catastrophe and would allow the transaction to commit anyways?
The other alternative would be to start it when a directory with the flag is first seen. That should be before any transactions.
The other thing is that it would be nasty to switch do_fork to use GFP_NOFS, but I doubt that's really needed.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Besides, what's the point of having nice constructs like dedicated workqueues
It's a resource that shouldn't be overused.
A workqueue + thread uses, what, 10-15k of memory? That's the cost of about 10 cached inodes. It is insignificant...
A little bloat here and a little bloat there and soon we're talking about serious memory.
e.g. on a dual core box in a standard configuration we're going towards ~50 kernel threads out of the box now and that's just too much IMNSHO. Given there are other really bad wastes of memory (don't get me started on some of the hash tables), but it's not totally insignificant.
Threads also have some CPU costs even when sleeping, e.g. there is still code which walks all threads to do various things.
Quoted 3 lines
Hmmm. I guess you are really not going to like the patch I have that moves the AIL pushing to a new thread to solve some of scalability issues in the transaction subsystem......
Per CPU or single? If the later definitely the workqueues would need to be fixed first.
-Andi
Quoted 5 lines · Andi Kleen
Quoted 1 line · On Wednesday 08 August 2007 15:14:04 David Chinner wrote:
Memory allocation failure + dirty transaction == filesystem shutdown.
You mean if the workqueue creation would fail?
Yeah.
Quoted 2 lines
Surely not having the MRU cache is not a catastrophe and would allow the transaction to commit anyways?
Right now the only errors that come from filestream association are fatal errors. i.e. they should force a shutdown if the transaction is dirty. One of these errors is already an ENOMEM.
Now we'd have a case where we'd had a "error that is not an error" and handle that in some way. It gets complex and that's something I try to avoid if at all possible.
Quoted 2 lines
The other alternative would be to start it when a directory with the flag is first seen. That should be before any transactions.
On lookup? No thanks.
On first create? Possible, but it's still painful and it introduces overhead into every single create operation.
Quoted 11 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Besides, what's the point of having nice constructs like dedicated workqueues
It's a resource that shouldn't be overused.
A workqueue + thread uses, what, 10-15k of memory? That's the cost of about 10 cached inodes. It is insignificant...
A little bloat here and a little bloat there and soon we're talking about serious memory.
e.g. on a dual core box in a standard configuration we're going towards ~50 kernel threads out of the box now and that's just too much IMNSHO.
My idle desktop machine has 180 processes running on it, 60 of them kernel threads. It doesn't concern me one bit.
Andi, you are complaining to the wrong person about thread counts. I live in the world of excessive parallelism and multithreaded I/O. I regularly see 4-8p boxes running hundreds to thousands of I/O threads on a single filesystem. These are the workloads XFS is designed for and we optimise for. A single extra thread is noise....
Quoted 5 lines
Quoted 3 lines
Hmmm. I guess you are really not going to like the patch I have that moves the AIL pushing to a new thread to solve some of scalability issues in the transaction subsystem......
Per CPU or single?
Single.
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Quoted 5 lines · David Chinner
Andi, you are complaining to the wrong person about thread counts. I live in the world of excessive parallelism and multithreaded I/O. I regularly see 4-8p boxes running hundreds to thousands of I/O threads on a single filesystem. These are the workloads XFS is designed for and we optimise for. A single extra thread is noise....
The other thing this is doing, though, is having an impact on power usage, I think. Fedora is trying to identify things which wake the CPU in an effort to reduce power usage if possible.
See for example:
http://article.gmane.org/gmane.linux.redhat.fedora.testers/50480
quoting:
"One of the biggest power problems is apps that wake up the CPU unnecessarily. With the tickless kernel for x86 (and soon x86_64), every time this happens it's a chance for power savings lost.
HOW YOU CAN HELP
----------------
We'd like to get as many reports of misbehaving apps as possible.
1) Install the 'powertop' package
2) Run it in a terminal window on a reasonably idle system with your normal combination of apps. (If you run it when you're actively compiling a kernel, watching a movie, or doing other CPU-intensive things, the results aren't as useful.)
3) Note the results..."
and here are the results w/ xfs mounted:
< Detailed C-state information is only available on Mobile CPUs (laptops) > P-states (frequencies)
2.21 Ghz 0.0%
2.00 Ghz 0.0%
1.80 Ghz 100.0%
Wakeups-from-idle per second : 31.7
no ACPI power usage estimate available
Top causes for wakeups:
37.0% ( 20.0) mount : xfs_mru_cache_create
...
looks like having this running (note: i'm not using filestreams on this mounted filesystem) may be impacting the power usage of the CPU? Or maybe SGI doesn' care about that, if the goal is 100% utilization of these machines on altixes etc... but still - something to consider I think.
Thanks,
-Eric
Quoted 9 lines · Eric Sandeen
Quoted 5 lines · David Chinner
Andi, you are complaining to the wrong person about thread counts. I live in the world of excessive parallelism and multithreaded I/O. I regularly see 4-8p boxes running hundreds to thousands of I/O threads on a single filesystem. These are the workloads XFS is designed for and we optimise for. A single extra thread is noise....
Quoted 3 lines
The other thing this is doing, though, is having an impact on power usage, I think. Fedora is trying to identify things which wake the CPU in an effort to reduce power usage if possible.
Sure, but an idle thread doesn't use any power, and the workqueue is now only used when there are streams active.
Quoted 11 lines
and here are the results w/ xfs mounted:
< Detailed C-state information is only available on Mobile CPUs (laptops) > P-states (frequencies)
2.21 Ghz 0.0%
2.00 Ghz 0.0%
1.80 Ghz 100.0%
Wakeups-from-idle per second : 31.7
no ACPI power usage estimate available
Top causes for wakeups:
37.0% ( 20.0) mount :xfs_mru_cache_create
http://oss.sgi.com/archives/xfs/2007-08/msg00009.html
"On-demand reaping of the MRU cache
Instead of running the mru cache reaper all the time based on a timeout, we should only run it when the cache has active objects. This allows CPUs to sleep when there is no activity rather than be woken repeatedly just to check if there is anything to do."
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Quoted 8 lines · David Chinner
http://oss.sgi.com/archives/xfs/2007-08/msg00009.html
"On-demand reaping of the MRU cache
Instead of running the mru cache reaper all the time based on a timeout, we should only run it when the cache has active objects. This allows CPUs to sleep when there is no activity rather than be woken repeatedly just to check if there is anything to do."
Ah, cool. Missed that, sorry. Will re-run powertop with that just to keep you on your toes. ;-)
-Eric