per-cpu stats in block device: overkill?
Hi Jens, Tejun, Jerome,
I've been auditing alloc_per_cpu users, and got to genhd. The code is fairly complex, but I can't help wondering if per-cpu counters are overkill. After all, we have a single queue lock.
The reason I care is that I'm changing alloc_per_cpu to use the static per-cpu area: at 40/80 bytes (32/64 bit) per stat, we'd be restricted to a few hundred disks unless the percpu area is enlarged (in current patches, a cmdline param). Or, I can change genhd to use big_percpu_alloc which will use the current inefficient dynamic per-cpu system until we get dynamic per-cpu regions (if ever).
Cheers,
Rusty.
Hello, Rusty.
Quoted 5 lines · Rusty Russell
Hi Jens, Tejun, Jerome,
I've been auditing
alloc_per_cpuusers, and got to genhd. The code is fairly complex, but I can't help wondering if per-cpu counters are overkill. After all, we have a single queue lock.
Yeah, maybe.
Quoted 6 lines
The reason I care is that I'm changing
alloc_per_cputo use the static per-cpu area: at 40/80 bytes (32/64 bit) per stat, we'd be restricted to a few hundred disks unless the percpu area is enlarged (in current patches, a cmdline param). Or, I can change genhd to usebig_percpu_allocwhich will use the current inefficient dynamic per-cpu system until we get dynamic per-cpu regions (if ever).
I'm working on local counter (local_t) allocator which is used to replace percpu allocation in percpu_counter and used as basis for percpu_ref which replaces module ref counting and will be used to simplify block/char lifetime rules.
The local counter allocator allocates per-cpu pages and the space overhead is minimal. If per-cpu stats in genhd is necessary, I think converting it to percpu local counter allocation should do it.
BTW, why make percpu area static?
Thanks.
Signature · 1 line
tejun
Quoted 4 lines · On Monday 22 December 2008 12:25:54 Tejun Heo wrote:
I'm working on local counter (
local_t) allocator which is used to replace percpu allocation inpercpu_counterand used as basis forpercpu_refwhich replaces module ref counting and will be used to simplify block/char lifetime rules.
Hi Tejun,
Interesting! Thanks to Christoph's dynamic percpu efforts, I've
been revising efforts to make alloc_percpu use the same efficient mechanism that static percpu vars use. We actually have this code already, tucked away in module.c.
This work is basically complete; the step I started this morning
is to remove the per_cpu__ prefix hackery from the per-cpu ops (in favour of sparse annotations). This leads to cpu_local_inc et. al. being usable for alloc_percpu-created percpu vars, not just static ones.
Quoted 3 lines
The local counter allocator allocates per-cpu pages and the space overhead is minimal. If per-cpu stats in genhd is necessary, I think converting it to percpu local counter allocation should do it.
Interesting; an allyesconfig boot uses 194 per-cpu allocs fromlib/percpu_counter.c at the moment. The module.c allocator is fairly space
efficient: 4 bytes per "block" (ie. each allocation or hole) but slow, which I figure is OK. Packing is good though.
Quoted 1 line
BTW, why make percpu area static?
Good question. Archs use a simple offset for per-cpu areas: some hold
this in a register (eg. %fs for x86-32). This means that the layout must be "congruent" (ie. have the same inter-cpu spacing) if we allocate a new per-cpu area (hard for non-NUMA).
For 5 years I waited for this to be fixed, and avoided exposing the per-cpu
core, and the alloc_percpu stuff was a standin implementation. But Christoph L. showed that even with the size limit, there are numerous places which want small per-cpu allocations which are optimally accessed, so I restarted work. See Message-Id: <[email protected]> "[PATCH 1/7] Improvealloc_percpu: make the per cpu reserve configurable and larger." and thread.
In addition, Mathieu and I have been discussing local_t: it's wandered
off its original purpose and we're debating what to do about it. See
Message-ID: <[email protected]> "local_add_return" and thread.
I look forward to always-cogent your thoughts on these issues! Rusty.