SLUB with Queueing beats SLAB in hackbench
Subsystems (4)
cover SLUB with Queueing beats SLAB in hackbench
The following patchset cleans some pieces up and then equips SLUB with per cpu queues that work similar to SLABs queues. With that approach SLUB wins in hackbench:
#!/bin/bash
uname -a
echo "./hackbench 100 process 200000"
./hackbench 100 process 200000
echo "./hackbench 100 process 20000"
./hackbench 100 process 20000
echo "./hackbench 100 process 20000"
./hackbench 100 process 20000
echo "./hackbench 100 process 20000"
./hackbench 100 process 20000
echo "./hackbench 10 process 20000"
./hackbench 10 process 20000
echo "./hackbench 10 process 20000"
./hackbench 10 process 20000
echo "./hackbench 10 process 20000"
./hackbench 10 process 20000
echo "./hackbench 1 process 20000"
./hackbench 1 process 20000
echo "./hackbench 1 process 20000"
./hackbench 1 process 20000
echo "./hackbench 1 process 20000"
./hackbench 1 process 20000
Procs NR SLAB SLUB SLUB+Queuing
----------------------------------------------------
100 200000 2741.3 2764.7 2231.9
100 20000 279.3 270.3 219.0
100 20000 278.0 273.1 219.2
100 20000 279.0 271.7 218.8
10 20000 34.0 35.6 28.8
10 20000 30.3 35.2 28.4
10 20000 32.9 34.6 28.4
1 20000 6.4 6.7 6.5
1 20000 6.3 6.8 6.5
1 20000 6.4 6.9 6.4
SLUB+Q is a merging of SLUB with some queuing concepts from SLAB and a new way of managing objects in the slabs using bitmaps. It uses a percpu queue so that free operations can be properly buffered and a bitmap for managing the free/allocated state in the slabs. It is slightly more inefficient than SLUB (due to the need to place large bitmaps --sized a few words--in some slab pages if there are more than BITS_PER_LONG objects in a slab) but in general does not increase space use too much.
The SLAB scheme of not touching the object during management is adopted. SLUB+Q can efficiently free and allocate cache cold objects without causing cache misses.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/16 ipc/sem.c: Bugfix for semop() not reporting successful operation
[Necessary to make 2.6.35-rc3 not deadlock. Not sure if this is the "right"(tm) fix]
The last change to improve the scalability moved the actual wake-up out of the section that is protected by spin_lock(sma->sem_perm.lock).
This means that IN_WAKEUP can be in queue.status even when the spinlock is acquired by the current task. Thus the same loop that is performed when queue.status is read without the spinlock acquired must be performed when the spinlock is acquired.
--- ipc/sem.c | 36 ++++++++++++++++++++++++++++++------ 1 files changed, 30 insertions(+), 6 deletions(-)
ipc/sem.c +30 -6
@@ -1256,6 +1256,32 @@ out:return un; } + +/** get_queue_result - Retrieve the result code from sem_queue +* @q: Pointer to queue structure +* +* The function retrieve the return code from the pending queue. If +* IN_WAKEUP is found in q->status, then we must loop until the value +* is replaced with the final value: This may happen if a task is +* woken up by an unrelated event (e.g. signal) and in parallel the task +* is woken up by another task because it got the requested semaphores. +* +* The function can be called with or without holding the semaphore spinlock. +*/ +static int get_queue_result (struct sem_queue *q) +{ +int error; + + error = q->status; +while (unlikely (error ==IN_WAKEUP )) { +cpu_relax (); + error = q->status; + } + +return error; +} + + SYSCALL_DEFINE4(semtimedop, int, semid, struct sembuf __user *, tsops, unsigned, nsops, const struct timespec __user *, timeout) {
@@ -1409,11 +1435,7 @@ SYSCALL_DEFINE4(semtimedop, int, semid, struct sembuf __user *, tsops,else schedule (); - error = queue.status; -while (unlikely (error ==IN_WAKEUP )) { -cpu_relax (); - error = queue.status; - } + error =get_queue_result (&queue);if (error != -EINTR ) {/* fast path: update_queue already obtained all requested
@@ -1427,10 +1449,12 @@ SYSCALL_DEFINE4(semtimedop, int, semid, struct sembuf __user *, tsops, goto out_free; } + error =get_queue_result (&queue); +/* * If queue.status != -EINTR we are woken up by another process */ - error = queue.status; +if (error != -EINTR ) { goto out_unlock_free; }
Signature · 8 lines
1.7.0.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 01/16] [PATCH] ipc/sem.c: Bugfix for semop() not reporting successful operation
Quoted 6 lines · On Fri, 25 Jun 2010 16:20:27 -0500 Christoph Lameter <[email protected]>…
[Necessary to make 2.6.35-rc3 not deadlock. Not sure if this is the "right"(tm) fix]
The last change to improve the scalability moved the actual wake-up out of the section that is protected by
spin_lock(sma->sem_perm.lock).
Quoted 7 lines
This means that
IN_WAKEUPcan be in queue.status even when the spinlock is acquired by the current task. Thus the same loop that is performed when queue.status is read without the spinlock acquired must be performed when the spinlock is acquired.Signed-off-by Manfred Spraul <[email protected]>Signed-off-by Christoph Lameter <[email protected]>
Hmm, I'm sorry if I don't understand the code...
Quoted 27 lines
--- ipc/sem.c | 36 ++++++++++++++++++++++++++++++------ 1 files changed, 30 insertions(+), 6 deletions(-)ipc/sem.c +14
-0@@ -1256,6 +1256,32 @@ out:return un; } + +/** get_queue_result - Retrieve the result code from sem_queue +* @q: Pointer to queue structure +* +* The function retrieve the return code from the pending queue. If +* IN_WAKEUP is found in q->status, then we must loop until the value +* is replaced with the final value: This may happen if a task is +* woken up by an unrelated event (e.g. signal) and in parallel the task +* is woken up by another task because it got the requested semaphores. +* +* The function can be called with or without holding the semaphore spinlock. +*/ +static int get_queue_result (struct sem_queue *q) +{
ipc/sem.c out: · line 1282
Quoted 10 lines
ipc/sem.c +10
-0+int error; + + error = q->status; +while (unlikely (error ==IN_WAKEUP )) { +cpu_relax (); + error = q->status; + } + +return error; +}
no memory barrier is required ?
Thanks,
-Kame
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 01/16] [PATCH] ipc/sem.c: Bugfix for semop() not reporting successful operation
Quoted 52 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · On Fri, 25 Jun 2010 16:20:27 -0500 Christoph Lameter<[email protected]>…
[Necessary to make 2.6.35-rc3 not deadlock. Not sure if this is the "right"(tm) fix]
The last change to improve the scalability moved the actual wake-up out of the section that is protected by
spin_lock(sma->sem_perm.lock).Quoted 8 lines
This means that
IN_WAKEUPcan be in queue.status even when the spinlock is acquired by the current task. Thus the same loop that is performed when queue.status is read without the spinlock acquired must be performed when the spinlock is acquired.Signed-off-by Manfred Spraul<[email protected]>Signed-off-by Christoph Lameter<[email protected]>Hmm, I'm sorry if I don't understand the code...
Quoted 30 lines
--- ipc/sem.c | 36 ++++++++++++++++++++++++++++++------ 1 files changed, 30 insertions(+), 6 deletions(-)ipc/sem.c +18
-0@@ -1256,6 +1256,32 @@ out:return un; } + +/** get_queue_result - Retrieve the result code from sem_queue +* @q: Pointer to queue structure +* +* The function retrieve the return code from the pending queue. If +* IN_WAKEUP is found in q->status, then we must loop until the value +* is replaced with the final value: This may happen if a task is +* woken up by an unrelated event (e.g. signal) and in parallel the task +* is woken up by another task because it got the requested semaphores. +* +* The function can be called with or without holding the semaphore spinlock. +*/ +static int get_queue_result (struct sem_queue *q) +{ +int error; + + error = q->status; +while (unlikely (error ==IN_WAKEUP )) {
Quoted 10 lines
ipc/sem.c out:
Quoted 7 lines
ipc/sem.c +6
-0+cpu_relax (); + error = q->status; + } + +return error; +}no memory barrier is required ?
No.q->status is the only field that is read in the exit path of sys_semtimedop():
After that, q->status is used as the return value of sys_semtimedop(), without accessing any other field.
Thus no memory barrier is required: there is just no other read/write operation against which the read of q->status must be serialized.
There is a smp_wmb() wake_up_sem_queue_do(), to ensure that all writes that are done by the cpu that does the wake-up are completed before q->status is set to the final value.
--
Manfred
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 01/16] [PATCH] ipc/sem.c: Bugfix for semop() not reporting successful operation
Quoted 71 lines · On Mon, 28 Jun 2010 18:45:25 +0200 Manfred Spraul <[email protected]>…
Quoted 52 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · On Fri, 25 Jun 2010 16:20:27 -0500 Christoph Lameter<[email protected]>…
[Necessary to make 2.6.35-rc3 not deadlock. Not sure if this is the "right"(tm) fix]
The last change to improve the scalability moved the actual wake-up out of the section that is protected by
spin_lock(sma->sem_perm.lock).Quoted 8 lines
This means that
IN_WAKEUPcan be in queue.status even when the spinlock is acquired by the current task. Thus the same loop that is performed when queue.status is read without the spinlock acquired must be performed when the spinlock is acquired.Signed-off-by Manfred Spraul<[email protected]>Signed-off-by Christoph Lameter<[email protected]>Hmm, I'm sorry if I don't understand the code...
Quoted 30 lines
--- ipc/sem.c | 36 ++++++++++++++++++++++++++++++------ 1 files changed, 30 insertions(+), 6 deletions(-)ipc/sem.c +18
-0@@ -1256,6 +1256,32 @@ out:return un; } + +/** get_queue_result - Retrieve the result code from sem_queue +* @q: Pointer to queue structure +* +* The function retrieve the return code from the pending queue. If +* IN_WAKEUP is found in q->status, then we must loop until the value +* is replaced with the final value: This may happen if a task is +* woken up by an unrelated event (e.g. signal) and in parallel the task +* is woken up by another task because it got the requested semaphores. +* +* The function can be called with or without holding the semaphore spinlock. +*/ +static int get_queue_result (struct sem_queue *q) +{ +int error; + + error = q->status; +while (unlikely (error ==IN_WAKEUP )) {Quoted 10 lines
ipc/sem.c out:
Quoted 7 lines
ipc/sem.c +6
-0+cpu_relax (); + error = q->status; + } + +return error; +}no memory barrier is required ?
No.
q->statusis the only field that is read in the exit path ofsys_semtimedop():
After that,q->statusis used as the return value ofsys_semtimedop(), without accessing any other field.
Thus no memory barrier is required: there is just no other read/write operation against which the read ofq->statusmust be serialized.
Quoted 4 lines
There is a
smp_wmb()wake_up_sem_queue_do(), to ensure that all writes that are done by the cpu that does the wake-up are completed beforeq->statusis set to the final value.
Thanks. BTW, cpu_relax() always includes asm("":::"memory") for avoiding optimization ?
Thanks,
-Kame
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 01/16] [PATCH] ipc/sem.c: Bugfix for semop() not reporting successful operation
In reply to Christoph LameterQuoted 2 lines · Christoph Lameter
[Necessary to make 2.6.35-rc3 not deadlock. Not sure if this is the "right"(tm) fix]
Is this related to the SLUB patches? Regardless, lets add Andrew and linux-kernel on CC.
Quoted 83 lines
The last change to improve the scalability moved the actual wake-up out of the section that is protected by
spin_lock(sma->sem_perm.lock).This means that
IN_WAKEUPcan be in queue.status even when the spinlock is acquired by the current task. Thus the same loop that is performed when queue.status is read without the spinlock acquired must be performed when the spinlock is acquired.Signed-off-by Manfred Spraul <[email protected]>Signed-off-by Christoph Lameter <[email protected]>---
ipc/sem.c | 36 ++++++++++++++++++++++++++++++------1 files changed, 30 insertions(+), 6 deletions(-)ipc/sem.c +26
-0@@ -1256,6 +1256,32 @@ out: return un; } + +/** get_queue_result - Retrieve the result code from sem_queue +* @q: Pointer to queue structure +* +* The function retrieve the return code from the pending queue. If +* IN_WAKEUP is found in q->status, then we must loop until the value +* is replaced with the final value: This may happen if a task is +* woken up by an unrelated event (e.g. signal) and in parallel the task +* is woken up by another task because it got the requested semaphores. +* +* The function can be called with or without holding the semaphore spinlock. +*/ +static int get_queue_result (struct sem_queue *q) +{ + int error; + + error = q->status; + while(unlikely(error == IN_WAKEUP)) { + cpu_relax(); + error = q->status; + } + + return error; +} + + SYSCALL_DEFINE4(semtimedop, int, semid, struct sembuf __user *, tsops, unsigned, nsops, const struct timespec __user *, timeout) {@@ -1409,11 +1435,7 @@ SYSCALL_DEFINE4(semtimedop, int, semid, struct sembuf __user *, tsops,else
schedule();- error = queue.status;
-while(unlikely(error == IN_WAKEUP)){
-cpu_relax();
- error = queue.status;
- }
+ error =get_queue_result(&queue);if (error !=
-EINTR) {
/* fast path:update_queuealready obtained all requestedipc/sem.c +3
-2@@ -1427,10 +1449,12 @@ SYSCALL_DEFINE4(semtimedop, int, semid, struct sembuf __user *, tsops, goto out_free; } + error = get_queue_result(&queue); + /* * If queue.status != -EINTR we are woken up by another process */ - error = queue.status; + if (error != -EINTR) { goto out_unlock_free; } --1.7.0.1
Quoted 6 lines
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 01/16] [PATCH] ipc/sem.c: Bugfix for semop() not reporting successful operation
This is a patch from Manfred. Required to make 2.6.35-rc3 work.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 01/16] [PATCH] ipc/sem.c: Bugfix for semop() not reporting successful operation
Quoted 2 lines · On Tue, 29 Jun 2010 10:42:42 -0500 (CDT) Christoph Lameter…
This is a patch from Manfred. Required to make 2.6.35-rc3 work.
My current version of the patch is below.
I believe that Luca has still seen problems with this patch applied so its current status is "stuck, awaiting developments".
Is that a correct determination?
Thanks.
From: Manfred Spraul <[email protected]>
The last change to improve the scalability moved the actual wake-up out of the section that is protected by spin_lock(sma->sem_perm.lock).
This means that IN_WAKEUP can be in queue.status even when the spinlock is acquired by the current task. Thus the same loop that is performed when queue.status is read without the spinlock acquired must be performed when the spinlock is acquired.
Addresses https://bugzilla.kernel.org/show_bug.cgi?id=16255
---
ipc/sem.c | 37 +++++++++++++++++++++++++++++++------ 1 file changed, 31 insertions(+), 6 deletions(-)
ipc/sem.c renamed from ipc/sem.c~ipc-semc-bugfix-for-semop-not-reporting-successful-operation +31 -6
@@ -1256,6 +1256,33 @@ out:return un; } + +/** +* get_queue_result - Retrieve the result code from sem_queue +* @q: Pointer to queue structure +* +* Retrieve the return code from the pending queue. If IN_WAKEUP is found in +* q->status, then we must loop until the value is replaced with the final +* value: This may happen if a task is woken up by an unrelated event (e.g. +* signal) and in parallel the task is woken up by another task because it got +* the requested semaphores. +* +* The function can be called with or without holding the semaphore spinlock. +*/ +static int get_queue_result (struct sem_queue *q) +{ +int error; + + error = q->status; +while (unlikely (error ==IN_WAKEUP )) { +cpu_relax (); + error = q->status; + } + +return error; +} + + SYSCALL_DEFINE4(semtimedop, int, semid, struct sembuf __user *, tsops, unsigned, nsops, const struct timespec __user *, timeout) {
@@ -1409,11 +1436,7 @@ SYSCALL_DEFINE4(semtimedop, int, semid,else schedule (); - error = queue.status; -while (unlikely (error ==IN_WAKEUP )) { -cpu_relax (); - error = queue.status; - } + error =get_queue_result (&queue);if (error != -EINTR ) {/* fast path: update_queue already obtained all requested
@@ -1427,10 +1450,12 @@ SYSCALL_DEFINE4(semtimedop, int, semid, goto out_free; } + error =get_queue_result (&queue); +/* * If queue.status != -EINTR we are woken up by another process */ - error = queue.status; +if (error != -EINTR ) { goto out_unlock_free; }
_
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 01/16] [PATCH] ipc/sem.c: Bugfix for semop() not reporting successful operation
Hi Andrew,
Quoted 7 lines · Andrew Morton
Quoted 3 lines · On Tue, 29 Jun 2010 10:42:42 -0500 (CDT) Christoph…
This is a patch from Manfred. Required to make 2.6.35-rc3 work.
Quoted 7 lines
My current version of the patch is below.
I believe that Luca has still seen problems with this patch applied so its current status is "stuck, awaiting developments".
Is that a correct determination?
I would propose that you forward a patch to Linus - either the one you have in your tree or the v2 that I've just posted.
With stock 2.6.35-rc3, my semtimedop() stress tests produces an oops or an invalid return value (i.e.:semtimedop() returns with "1") within a fraction of a second.
With either of the patches applied, my test apps show the expected behavior.
--
Manfred
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andrew Morton
maintainerRe: [S+Q 01/16] [PATCH] ipc/sem.c: Bugfix for semop() not reporting successful operation
Quoted 12 lines · On Wed, 30 Jun 2010 21:38:43 +0200 Manfred Spraul <[email protected]>…
Hi Andrew,
Quoted 9 lines · Andrew Morton
Quoted 3 lines · On Tue, 29 Jun 2010 10:42:42 -0500 (CDT) Christoph…
This is a patch from Manfred. Required to make 2.6.35-rc3 work.
My current version of the patch is below.
Quoted 8 lines
Quoted 5 lines
I believe that Luca has still seen problems with this patch applied so its current status is "stuck, awaiting developments".
Is that a correct determination?
I would propose that you forward a patch to Linus - either the one you have in your tree or the v2 that I've just posted.
OK, I added the incremental change:
ipc/sem.c renamed from ipc/sem.c~ipc-semc-bugfix-for-semop-not-reporting-successful-operation-update +8 -1
@@ -1440,7 +1440,14 @@ SYSCALL_DEFINE4(semtimedop, int, semid,if (error != -EINTR ) {/* fast path: update_queue already obtained all requested -* resources */ +* resources. +* Perform a smp_mb(): User space could assume that semop() +* is a memory barrier: Without the mb(), the cpu could +* speculatively read in user space stale data that was +* overwritten by the previous owner of the semaphore. +*/ +smp_mb (); + goto out_free; }
_
Quoted 5 lines
With stock 2.6.35-rc3, my
semtimedop()stress tests produces an oops or an invalid return value (i.e.:semtimedop() returns with "1") within a fraction of a second.With either of the patches applied, my test apps show the expected behavior.
OK, I'll queue it up.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 4/16 slub: Use a constant for a unspecified node.
In reply to Christoph Lameterkmalloc_node() and friends can be passed a constant -1 to indicate that no choice was made for the node from which the object needs to come.
Use NUMA_NO_NODE instead of -1.
--- include/linux/slab.h | 2 ++ mm/slub.c | 10 +++++----- 2 files changed, 7 insertions(+), 5 deletions(-)
mm/slub.c +5 -5
@@ -1073,7 +1073,7 @@ static inline struct page *alloc_slab_pa flags |= __GFP_NOTRACK; -if (node ==-1 ) +if (node ==NUMA_NO_NODE )return alloc_pages (flags, order);else return alloc_pages_exact_node (node, flags, order);
@@ -1727,7 +1727,7 @@ static __always_inline void *slab_alloc(void *kmem_cache_alloc (struct kmem_cache *s,gfp_t gfpflags) { -void *ret =slab_alloc (s, gfpflags,-1 , _RET_IP_); +void *ret =slab_alloc (s, gfpflags,NUMA_NO_NODE , _RET_IP_);trace_kmem_cache_alloc (_RET_IP_, ret, s->objsize, s->size, gfpflags);
@@ -1738,7 +1738,7 @@ EXPORT_SYMBOL(kmem_cache_alloc);#ifdef CONFIG_TRACING void *kmem_cache_alloc_notrace (struct kmem_cache *s,gfp_t gfpflags) { -return slab_alloc (s, gfpflags,-1 , _RET_IP_); +return slab_alloc (s, gfpflags,NUMA_NO_NODE , _RET_IP_); }EXPORT_SYMBOL (kmem_cache_alloc_notrace);#endif
@@ -2728,7 +2728,7 @@ void *__kmalloc(size_t size, gfp_t flagsif (unlikely (ZERO_OR_NULL_PTR (s)))return s; - ret =slab_alloc (s, flags,-1 , _RET_IP_); + ret =slab_alloc (s, flags,NUMA_NO_NODE , _RET_IP_);trace_kmalloc (_RET_IP_, ret, size, s->size, flags);
@@ -3312,7 +3312,7 @@ void *__kmalloc_track_caller(size_t sizeif (unlikely (ZERO_OR_NULL_PTR (s)))return s; - ret =slab_alloc (s, gfpflags,-1 , caller); + ret =slab_alloc (s, gfpflags,NUMA_NO_NODE , caller);/* Honor the call site pointer we recieved. */ trace_kmalloc (caller, ret, size, s->size, gfpflags);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 04/16] slub: Use a constant for a unspecified node.
Quoted 9 lines · On Fri, 25 Jun 2010 16:20:30 -0500 Christoph Lameter <[email protected]>…
kmalloc_node()and friends can be passed a constant -1 to indicate that no choice was made for the node from which the object needs to come.Use
NUMA_NO_NODEinstead of -1.Signed-off-by David Rientjes <[email protected]>Signed-off-by Christoph Lameter <[email protected]>
How about more updates ?
Hmm, by grep (mmotm)
==
static struct page *get_partial(struct kmem_cache *s, gfp_t flags, int node)
{
struct page *page;
int searchnode = (node == -1) ? numa_node_id() : node;==
==
static inline int node_match(struct kmem_cache_cpu *c, int node) {
#ifdef CONFIG_NUMA
if (node != -1 && c->node != node)
return 0;
#endif
return 1;
}==
==
debug:
if (!alloc_debug_processing(s, c->page, object, addr))
goto another_slab;
c->page->inuse++;
c->page->freelist = get_freepointer(s, object);
c->node = -1;
goto unlock_out;
}==
Thanks,
-Kame
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 04/16] slub: Use a constant for a unspecified node.
Quoted 7 lines · KAMEZAWA Hiroyuki
Quoted 4 lines · On Fri, 25 Jun 2010 16:20:30 -0500 Christoph Lameter <[email protected]>…
kmalloc_node()and friends can be passed a constant -1 to indicate that no choice was made for the node from which the object needs to come.
Quoted 8 lines
Quoted 5 lines
Use
NUMA_NO_NODEinstead of -1.Signed-off-by David Rientjes <[email protected]>Signed-off-by Christoph Lameter <[email protected]>Reviewd-by KAMEZAWA Hiroyuki <[email protected]>How about more updates ?
Would be a great idea. Can you take over this patch and add the missing pieces? I dont have too much time in the next weeks. Also am on vacation.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 5/16 SLUB: Constants need UL
In reply to Christoph LameterUL suffix is missing in some constants. Conform to how slab.h uses constants.
--- mm/slub.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/slub.c +2 -2
@@ -162,8 +162,8 @@#define MAX_OBJS_PER_PAGE 65535/* since page.objects is u16 */ /* Internal SLUB flags */ -#define __OBJECT_POISON 0x80000000/* Poison object */ -#define __SYSFS_ADD_DEFERRED 0x40000000/* Not yet visible via sysfs */ +#define __OBJECT_POISON 0x80000000UL/* Poison object */ +#define __SYSFS_ADD_DEFERRED 0x40000000UL/* Not yet visible via sysfs */ static int kmem_size =sizeof (struct kmem_cache );
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 05/16] SLUB: Constants need UL
Quoted 3 lines · Christoph Lameter
UL suffix is missing in some constants. Conform to how
slab.huses constants.Signed-off-by Christoph Lameter <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 05/16] SLUB: Constants need UL
In reply to Christoph LameterQuoted 4 lines · On Fri, 25 Jun 2010 16:20:31 -0500 Christoph Lameter <[email protected]>…
UL suffix is missing in some constants. Conform to how
slab.huses constants.Signed-off-by Christoph Lameter <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 6/16 slub: Use kmem_cache flags to detect if slab is in debugging mode.
In reply to Christoph LameterThe cacheline with the flags is reachable from the hot paths after the percpu allocator changes went in. So there is no need anymore to put a flag into each slab page. Get rid of the SlubDebug flag and use the flags in kmem_cache instead.
--- include/linux/page-flags.h | 1 - mm/slub.c | 33 ++++++++++++--------------------- 2 files changed, 12 insertions(+), 22 deletions(-)
include/linux/page-flags.h +0 -1
@@ -215,7 +215,6 @@ PAGEFLAG(SwapBacked, swapbacked) __CLEAR __PAGEFLAG(SlobFree, slob_free)__PAGEFLAG (SlubFrozen, slub_frozen) -__PAGEFLAG (SlubDebug, slub_debug)/* * Private page markings that may be used by the filesystem that owns the page
mm/slub.c +12 -21
@@ -107,11 +107,17 @@* the fast path and disables lockless freelists. */ +#define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \ + SLAB_TRACE | SLAB_DEBUG_FREE) + +static inline int kmem_cache_debug (struct kmem_cache *s) +{#ifdef CONFIG_SLUB_DEBUG -#define SLABDEBUG 1 +return unlikely (s->flags &SLAB_DEBUG_FLAGS );#else -#define SLABDEBUG 0 +return 0 ;#endif +}/* * Issues still to be resolved:
@@ -1157,9 +1163,6 @@ static struct page *new_slab(struct kmeminc_slabs_node (s,page_to_nid (page), page->objects); page->slab = s; page->flags |=1 << PG_slab; -if (s->flags & (SLAB_DEBUG_FREE |SLAB_RED_ZONE |SLAB_POISON | -SLAB_STORE_USER |SLAB_TRACE )) -__SetPageSlubDebug (page); start =page_address (page);
@@ -1186,14 +1189,13 @@ static void __free_slab(struct kmem_cachint order =compound_order (page);int pages =1 << order; -if (unlikely (SLABDEBUG &&PageSlubDebug (page))) { +if (kmem_cache_debug (s)) {void *p;slab_pad_check (s, page);for_each_object (p, s,page_address (page), page->objects)check_object (s, page, p,0 ); -__ClearPageSlubDebug (page); }kmemcheck_free_shadow (page,compound_order (page));
@@ -1415,8 +1417,7 @@ static void unfreeze_slab(struct kmem_castat (s, tail ?DEACTIVATE_TO_TAIL :DEACTIVATE_TO_HEAD ); }else {stat (s,DEACTIVATE_FULL ); -if (SLABDEBUG &&PageSlubDebug (page) && - (s->flags &SLAB_STORE_USER )) +if (kmem_cache_debug (s) && (s->flags &SLAB_STORE_USER ))add_full (n, page); }slab_unlock (page);
@@ -1624,7 +1625,7 @@ load_freelist: object = c->page->freelist;if (unlikely (!object)) goto another_slab; -if (unlikely (SLABDEBUG &&PageSlubDebug (c->page))) +if (kmem_cache_debug (s)) goto debug; c->freelist =get_freepointer (s, object);
@@ -1783,7 +1784,7 @@ static void __slab_free(struct kmem_cachstat (s,FREE_SLOWPATH );slab_lock (page); -if (unlikely (SLABDEBUG &&PageSlubDebug (page))) +if (kmem_cache_debug (s)) goto debug; checks_ok:
@@ -3395,16 +3396,6 @@ static void validate_slab_slab(struct km }else printk (KERN_INFO "SLUB %s: Skipped busy slab 0x%p\n" , s->name, page); - -if (s->flags &DEBUG_DEFAULT_FLAGS ) { -if (!PageSlubDebug (page)) -printk (KERN_ERR "SLUB %s: SlubDebug not set " -"on slab 0x%p\n" , s->name, page); - }else { -if (PageSlubDebug (page)) -printk (KERN_ERR "SLUB %s: SlubDebug set on " -"slab 0x%p\n" , s->name, page); - } } static int validate_slab_node(struct kmem_cache *s,
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 06/16] slub: Use kmem_cache flags to detect if slab is in debugging mode.
Quoted 6 lines · Christoph Lameter
The cacheline with the flags is reachable from the hot paths after the percpu allocator changes went in. So there is no need anymore to put a flag into each slab page. Get rid of the SlubDebug flag and use the flags in
kmem_cacheinstead.Signed-off-by Christoph Lameter <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 7/16 slub: discard_slab_unlock
In reply to Christoph LameterThe sequence of unlocking a slab and freeing occurs multiple times. Put the common into a single function.
--- mm/slub.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
mm/slub.c +10 -6
@@ -1260,6 +1260,13 @@ static __always_inline int slab_trylock(return rc; } +static void discard_slab_unlock (struct kmem_cache *s, +struct page *page) +{ +slab_unlock (page); +discard_slab (s, page); +} +/* * Management of partially allocated slabs */
@@ -1437,9 +1444,8 @@ static void unfreeze_slab(struct kmem_caadd_partial (n, page,1 );slab_unlock (page); }else { -slab_unlock (page);stat (s,FREE_SLAB ); -discard_slab (s, page); +discard_slab_unlock (s, page); } } }
@@ -1822,9 +1828,8 @@ slab_empty:remove_partial (s, page);stat (s,FREE_REMOVE_PARTIAL ); } -slab_unlock (page);stat (s,FREE_SLAB ); -discard_slab (s, page); +discard_slab_unlock (s, page);return ; debug:
@@ -2893,8 +2898,7 @@ int kmem_cache_shrink(struct kmem_cache*/ list_del (&page->lru); n->nr_partial--; -slab_unlock (page); -discard_slab (s, page); +discard_slab_unlock (s, page); }else {list_move (&page->lru, slabs_by_inuse + page->inuse);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerRe: [S+Q 07/16] slub: discard_slab_unlock
Quoted 3 lines · Christoph Lameter
The sequence of unlocking a slab and freeing occurs multiple times. Put the common into a single function.
Did you want to respond to the comments I made about this patch at
http://marc.info/?l=linux-mm&m=127689747432061 ? Specifically, how it makes seeing if there are unmatched slab_lock() -> slab_unlock() pairs more difficult.
Quoted 47 lines
Signed-off-by Christoph Lameter <[email protected]>--- mm/slub.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)Index:
linux-2.6/mm/slub.c
===================================================================mm/slub.c +9
-4@@ -1260,6 +1260,13 @@ static __always_inline int slab_trylock(return rc; } +static void discard_slab_unlock (struct kmem_cache *s, +struct page *page) +{ +slab_unlock (page); +discard_slab (s, page); +} +/* * Management of partially allocated slabs */ @@ -1437,9 +1444,8 @@ static void unfreeze_slab(struct kmem_caadd_partial (n, page,1 );slab_unlock (page); }else { -slab_unlock (page);stat (s,FREE_SLAB ); -discard_slab (s, page); +discard_slab_unlock (s, page); } } }@@ -1822,9 +1828,8 @@ slab_empty:remove_partial (s, page);stat (s,FREE_REMOVE_PARTIAL ); } -slab_unlock (page);stat (s,FREE_SLAB ); -discard_slab (s, page); +discard_slab_unlock (s, page);return ; debug:
mm/slub.c int kmem_cache_shrink(struct kmem_cache · line 2904
Quoted 9 lines
mm/slub.c +1
-2@@ -2893,8 +2898,7 @@ int kmem_cache_shrink(struct kmem_cache*/ list_del (&page->lru); n->nr_partial--; -slab_unlock (page); -discard_slab (s, page); +discard_slab_unlock (s, page); }else {list_move (&page->lru, slabs_by_inuse + page->inuse);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 07/16] slub: discard_slab_unlock
Quoted 8 lines · David Rientjes
Quoted 3 lines
The sequence of unlocking a slab and freeing occurs multiple times. Put the common into a single function.
Did you want to respond to the comments I made about this patch at
http://marc.info/?l=linux-mm&m=127689747432061 ? Specifically, how it makes seeing if there are unmatchedslab_lock()->slab_unlock()pairs more difficult.
I dont think so. The name includes slab_unlock at the end. We could drop this but its a frequent action necessary when disposing of a slab page.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 8/16 slub: remove dynamic dma slab allocation
In reply to Christoph LameterRemove the dynamic dma slab allocation since this causes too many issues with nested locks etc etc. The change avoids passing gfpflags into many functions.
--- mm/slub.c | 153 ++++++++++++++++---------------------------------------------- 1 file changed, 41 insertions(+), 112 deletions(-)
mm/slub.c +41 -112
@@ -2070,7 +2070,7 @@ init_kmem_cache_node(struct kmem_cache_n static DEFINE_PER_CPU(struct kmem_cache_cpu, kmalloc_percpu[KMALLOC_CACHES]); -static inline int alloc_kmem_cache_cpus (struct kmem_cache *s,gfp_t flags) +static inline int alloc_kmem_cache_cpus (struct kmem_cache *s) {if (s < kmalloc_caches +KMALLOC_CACHES && s >= kmalloc_caches)/*
@@ -2097,7 +2097,7 @@ static inline int alloc_kmem_cache_cpus(* when allocating for the kmalloc_node_cache. This is used for bootstrapping * memory on a fresh node that has no slab structures yet. */ -static void early_kmem_cache_node_alloc (gfp_t gfpflags,int node) +static void early_kmem_cache_node_alloc (int node) {struct page *page;struct kmem_cache_node *n;
@@ -2105,7 +2105,7 @@ static void early_kmem_cache_node_alloc(BUG_ON (kmalloc_caches->size <sizeof (struct kmem_cache_node )); - page =new_slab (kmalloc_caches, gfpflags, node); + page =new_slab (kmalloc_caches,GFP_KERNEL , node);BUG_ON (!page);if (page_to_nid (page) != node) {
@@ -2149,7 +2149,7 @@ static void free_kmem_cache_nodes(struct } } -static int init_kmem_cache_nodes (struct kmem_cache *s,gfp_t gfpflags) +static int init_kmem_cache_nodes (struct kmem_cache *s) {int node;
@@ -2157,11 +2157,11 @@ static int init_kmem_cache_nodes(structstruct kmem_cache_node *n;if (slab_state ==DOWN ) { -early_kmem_cache_node_alloc (gfpflags, node); +early_kmem_cache_node_alloc (node);continue ; } n =kmem_cache_alloc_node (kmalloc_caches, - gfpflags, node); +GFP_KERNEL , node);if (!n) {free_kmem_cache_nodes (s);
@@ -2178,7 +2178,7 @@ static void free_kmem_cache_nodes(struct { } -static int init_kmem_cache_nodes (struct kmem_cache *s,gfp_t gfpflags) +static int init_kmem_cache_nodes (struct kmem_cache *s) {init_kmem_cache_node (&s->local_node, s);return 1 ;
@@ -2318,7 +2318,7 @@ static int calculate_sizes(struct kmem_c } -static int kmem_cache_open (struct kmem_cache *s,gfp_t gfpflags, +static int kmem_cache_open (struct kmem_cache *s,const char *name,size_t size,size_t align,unsigned long flags,void (*ctor)(void *))
@@ -2354,10 +2354,10 @@ static int kmem_cache_open(struct kmem_c#ifdef CONFIG_NUMA s->remote_node_defrag_ratio =1000 ;#endif -if (!init_kmem_cache_nodes (s, gfpflags & ~SLUB_DMA )) +if (!init_kmem_cache_nodes (s)) goto error; -if (alloc_kmem_cache_cpus (s, gfpflags & ~SLUB_DMA )) +if (alloc_kmem_cache_cpus (s))return 1 ;free_kmem_cache_nodes (s);
@@ -2517,6 +2517,10 @@ EXPORT_SYMBOL(kmem_cache_destroy); struct kmem_cache kmalloc_caches[KMALLOC_CACHES] __cacheline_aligned;EXPORT_SYMBOL (kmalloc_caches); +#ifdef CONFIG_ZONE_DMA +static struct kmem_cache kmalloc_dma_caches[SLUB_PAGE_SHIFT ]; +#endif +static int __init setup_slub_min_order (char *str) {get_option (&str, &slub_min_order);
@@ -2553,116 +2557,26 @@ static int __init setup_slub_nomerge(cha__setup ("slub_nomerge" , setup_slub_nomerge); -static struct kmem_cache *create_kmalloc_cache (struct kmem_cache *s, -const char *name,int size,gfp_t gfp_flags) +static void create_kmalloc_cache (struct kmem_cache *s, +const char *name,int size,unsigned int flags) { -unsigned int flags =0 ; - -if (gfp_flags &SLUB_DMA ) - flags =SLAB_CACHE_DMA ; -/* * This function is called with IRQs disabled during early-boot on * single CPU so there's no need to take slub_lock here. */ -if (!kmem_cache_open (s, gfp_flags, name, size,ARCH_KMALLOC_MINALIGN , +if (!kmem_cache_open (s, name, size,ARCH_KMALLOC_MINALIGN , flags,NULL )) goto panic;list_add (&s->list, &slab_caches); -if (sysfs_slab_add (s)) - goto panic; -return s; +if (!sysfs_slab_add (s)) +return ; panic:panic ("Creation of kmalloc slab %s size=%d failed.\n" , name, size); } -#ifdef CONFIG_ZONE_DMA -static struct kmem_cache *kmalloc_caches_dma[SLUB_PAGE_SHIFT ]; - -static void sysfs_add_func (struct work_struct *w) -{ -struct kmem_cache *s; - -down_write (&slub_lock); -list_for_each_entry (s, &slab_caches, list) { -if (s->flags & __SYSFS_ADD_DEFERRED) { - s->flags &= ~__SYSFS_ADD_DEFERRED; -sysfs_slab_add (s); - } - } -up_write (&slub_lock); -} - -static DECLARE_WORK(sysfs_add_work, sysfs_add_func); - -static noinline struct kmem_cache *dma_kmalloc_cache(int index, gfp_t flags) -{ -struct kmem_cache *s; -char *text; -size_t realsize; -unsigned long slabflags; -int i; - - s = kmalloc_caches_dma[index]; -if (s) -return s; - -/* Dynamically create dma cache */ -if (flags & __GFP_WAIT) -down_write (&slub_lock); -else { -if (!down_write_trylock (&slub_lock)) - goto out; - } - -if (kmalloc_caches_dma[index]) - goto unlock_out; - - realsize = kmalloc_caches[index].objsize; - text =kasprintf (flags & ~SLUB_DMA ,"kmalloc_dma-%d" , - (unsigned int )realsize); - - s =NULL ; -for (i =0 ; i <KMALLOC_CACHES ; i++) -if (!kmalloc_caches[i].size) -break ; - -BUG_ON (i >=KMALLOC_CACHES ); - s = kmalloc_caches + i; - -/* -* Must defer sysfs creation to a workqueue because we don't know -* what context we are called from. Before sysfs comes up, we don't -* need to do anything because our sysfs initcall will start by -* adding all existing slabs to sysfs. -*/ - slabflags =SLAB_CACHE_DMA |SLAB_NOTRACK ; -if (slab_state >=SYSFS ) - slabflags |= __SYSFS_ADD_DEFERRED; - -if (!text || !kmem_cache_open (s, flags, text, - realsize,ARCH_KMALLOC_MINALIGN , slabflags,NULL )) { - s->size =0 ; -kfree (text); - goto unlock_out; - } - -list_add (&s->list, &slab_caches); - kmalloc_caches_dma[index] = s; - -if (slab_state >=SYSFS ) -schedule_work (&sysfs_add_work); - -unlock_out: -up_write (&slub_lock); -out: -return kmalloc_caches_dma[index]; -} -#endif -/* * Conversion table for small slabs sizes / 8 to the index in the * kmalloc array. This is necessary for slabs < 192 since we have non power
@@ -2715,7 +2629,7 @@ static struct kmem_cache *get_slab(size_#ifdef CONFIG_ZONE_DMA if (unlikely ((flags &SLUB_DMA ))) -return dma_kmalloc_cache (index, flags); +return &kmalloc_dma_caches[index];#endif return &kmalloc_caches[index];
@@ -3053,7 +2967,7 @@ void __init kmem_cache_init(void)* kmem_cache_open for slab_state == DOWN. */ create_kmalloc_cache (&kmalloc_caches[0 ],"kmem_cache_node" , -sizeof (struct kmem_cache_node ),GFP_NOWAIT ); +sizeof (struct kmem_cache_node ),0 ); kmalloc_caches[0 ].refcount =-1 ; caches++;
@@ -3066,18 +2980,18 @@ void __init kmem_cache_init(void)/* Caches that are not of the two-to-the-power-of size */ if (KMALLOC_MIN_SIZE <=32 ) {create_kmalloc_cache (&kmalloc_caches[1 ], -"kmalloc-96" ,96 ,GFP_NOWAIT ); +"kmalloc-96" ,96 ,0 ); caches++; }if (KMALLOC_MIN_SIZE <=64 ) {create_kmalloc_cache (&kmalloc_caches[2 ], -"kmalloc-192" ,192 ,GFP_NOWAIT ); +"kmalloc-192" ,192 ,0 ); caches++; }for (i =KMALLOC_SHIFT_LOW ; i <SLUB_PAGE_SHIFT ; i++) {create_kmalloc_cache (&kmalloc_caches[i], -"kmalloc" ,1 << i,GFP_NOWAIT ); +"kmalloc" ,1 << i,0 ); caches++; }
@@ -3124,7 +3038,7 @@ void __init kmem_cache_init(void)/* Provide the correct kmalloc names now that the caches are up */ for (i =KMALLOC_SHIFT_LOW ; i <SLUB_PAGE_SHIFT ; i++) - kmalloc_caches[i]. name = + kmalloc_caches[i].name =kasprintf (GFP_NOWAIT ,"kmalloc-%d" ,1 << i);#ifdef CONFIG_SMP
@@ -3147,6 +3061,21 @@ void __init kmem_cache_init(void)void __init kmem_cache_init_late (void ) { +#ifdef CONFIG_ZONE_DMA +int i; + +for (i =0 ; i <SLUB_PAGE_SHIFT ; i++) { +struct kmem_cache *s = &kmalloc_caches[i]; + +if (s && s->size) { +char *name =kasprintf (GFP_KERNEL , +"dma-kmalloc-%d" , s->objsize); + +create_kmalloc_cache (&kmalloc_dma_caches[i], + name, s->objsize,SLAB_CACHE_DMA ); + } + } +#endif }/*
@@ -3241,7 +3170,7 @@ struct kmem_cache *kmem_cache_create(con s =kmalloc (kmem_size,GFP_KERNEL );if (s) { -if (kmem_cache_open (s,GFP_KERNEL , name, +if (kmem_cache_open (s, name, size, align, flags, ctor)) {list_add (&s->list, &slab_caches);up_write (&slub_lock);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerRe: [S+Q 08/16] slub: remove dynamic dma slab allocation
Quoted 32 lines · Christoph Lameter
Remove the dynamic dma slab allocation since this causes too many issues with nested locks etc etc. The change avoids passing gfpflags into many functions.
Signed-off-by Christoph Lameter <[email protected]>--- mm/slub.c | 153 ++++++++++++++++---------------------------------------------- 1 file changed, 41 insertions(+), 112 deletions(-)Index:
linux-2.6/mm/slub.c
===================================================================mm/slub.c +2
-2@@ -2070,7 +2070,7 @@ init_kmem_cache_node(struct kmem_cache_n static DEFINE_PER_CPU(struct kmem_cache_cpu, kmalloc_percpu[KMALLOC_CACHES]); -static inline int alloc_kmem_cache_cpus (struct kmem_cache *s,gfp_t flags) +static inline int alloc_kmem_cache_cpus (struct kmem_cache *s) {if (s < kmalloc_caches +KMALLOC_CACHES && s >= kmalloc_caches)/* @@ -2097,7 +2097,7 @@ static inline int alloc_kmem_cache_cpus(* when allocating for the kmalloc_node_cache. This is used for bootstrapping * memory on a fresh node that has no slab structures yet. */ -static void early_kmem_cache_node_alloc (gfp_t gfpflags,int node) +static void early_kmem_cache_node_alloc (int node) {struct page *page;struct kmem_cache_node *n;
mm/slub.c static void early_kmem_cache_node_alloc( · line 2111
Quoted 8 lines
mm/slub.c +1
-1@@ -2105,7 +2105,7 @@ static void early_kmem_cache_node_alloc(BUG_ON (kmalloc_caches->size <sizeof (struct kmem_cache_node )); - page =new_slab (kmalloc_caches, gfpflags, node); + page =new_slab (kmalloc_caches,GFP_KERNEL , node);BUG_ON (!page);if (page_to_nid (page) != node) {
This still passes GFP_KERNEL to the page allocator when not allowed by gfp_allowed_mask for early (non SLAB_CACHE_DMA) users of create_kmalloc_cache().
Quoted 13 lines
mm/slub.c +1
-1@@ -2149,7 +2149,7 @@ static void free_kmem_cache_nodes(struct } } -static int init_kmem_cache_nodes (struct kmem_cache *s,gfp_t gfpflags) +static int init_kmem_cache_nodes (struct kmem_cache *s) {int node;@@ -2157,11 +2157,11 @@ static int init_kmem_cache_nodes(structstruct kmem_cache_node *n;if (slab_state ==DOWN ) {
mm/slub.c static int init_kmem_cache_nodes(struct · line 2167
Quoted 10 lines
mm/slub.c +2
-2-early_kmem_cache_node_alloc (gfpflags, node); +early_kmem_cache_node_alloc (node);continue ; } n =kmem_cache_alloc_node (kmalloc_caches, - gfpflags, node); +GFP_KERNEL , node);if (!n) {free_kmem_cache_nodes (s);
slab_state != DOWN is still not an indication that GFP_KERNEL is safe; in fact, all users of GFP_KERNEL from kmem_cache_init() are unsafe. These need to be GFP_NOWAIT.
Quoted 217 lines
mm/slub.c +19
-105@@ -2178,7 +2178,7 @@ static void free_kmem_cache_nodes(struct { } -static int init_kmem_cache_nodes (struct kmem_cache *s,gfp_t gfpflags) +static int init_kmem_cache_nodes (struct kmem_cache *s) {init_kmem_cache_node (&s->local_node, s);return 1 ;@@ -2318,7 +2318,7 @@ static int calculate_sizes(struct kmem_c } -static int kmem_cache_open (struct kmem_cache *s,gfp_t gfpflags, +static int kmem_cache_open (struct kmem_cache *s,const char *name,size_t size,size_t align,unsigned long flags,void (*ctor)(void *))@@ -2354,10 +2354,10 @@ static int kmem_cache_open(struct kmem_c#ifdef CONFIG_NUMA s->remote_node_defrag_ratio =1000 ;#endif -if (!init_kmem_cache_nodes (s, gfpflags & ~SLUB_DMA )) +if (!init_kmem_cache_nodes (s)) goto error; -if (alloc_kmem_cache_cpus (s, gfpflags & ~SLUB_DMA )) +if (alloc_kmem_cache_cpus (s))return 1 ;free_kmem_cache_nodes (s);@@ -2517,6 +2517,10 @@ EXPORT_SYMBOL(kmem_cache_destroy); struct kmem_cache kmalloc_caches[KMALLOC_CACHES] __cacheline_aligned;EXPORT_SYMBOL (kmalloc_caches); +#ifdef CONFIG_ZONE_DMA +static struct kmem_cache kmalloc_dma_caches[SLUB_PAGE_SHIFT ]; +#endif +static int __init setup_slub_min_order (char *str) {get_option (&str, &slub_min_order);@@ -2553,116 +2557,26 @@ static int __init setup_slub_nomerge(cha__setup ("slub_nomerge" , setup_slub_nomerge); -static struct kmem_cache *create_kmalloc_cache (struct kmem_cache *s, -const char *name,int size,gfp_t gfp_flags) +static void create_kmalloc_cache (struct kmem_cache *s, +const char *name,int size,unsigned int flags) { -unsigned int flags =0 ; - -if (gfp_flags &SLUB_DMA ) - flags =SLAB_CACHE_DMA ; -/* * This function is called with IRQs disabled during early-boot on * single CPU so there's no need to take slub_lock here. */ -if (!kmem_cache_open (s, gfp_flags, name, size,ARCH_KMALLOC_MINALIGN , +if (!kmem_cache_open (s, name, size,ARCH_KMALLOC_MINALIGN , flags,NULL )) goto panic;list_add (&s->list, &slab_caches); -if (sysfs_slab_add (s)) - goto panic; -return s; +if (!sysfs_slab_add (s)) +return ; panic:panic ("Creation of kmalloc slab %s size=%d failed.\n" , name, size); } -#ifdef CONFIG_ZONE_DMA -static struct kmem_cache *kmalloc_caches_dma[SLUB_PAGE_SHIFT ]; - -static void sysfs_add_func (struct work_struct *w) -{ -struct kmem_cache *s; - -down_write (&slub_lock); -list_for_each_entry (s, &slab_caches, list) { -if (s->flags & __SYSFS_ADD_DEFERRED) { - s->flags &= ~__SYSFS_ADD_DEFERRED; -sysfs_slab_add (s); - } - } -up_write (&slub_lock); -} - -static DECLARE_WORK(sysfs_add_work, sysfs_add_func); - -static noinline struct kmem_cache *dma_kmalloc_cache(int index, gfp_t flags) -{ -struct kmem_cache *s; -char *text; -size_t realsize; -unsigned long slabflags; -int i; - - s = kmalloc_caches_dma[index]; -if (s) -return s; - -/* Dynamically create dma cache */ -if (flags & __GFP_WAIT) -down_write (&slub_lock); -else { -if (!down_write_trylock (&slub_lock)) - goto out; - } - -if (kmalloc_caches_dma[index]) - goto unlock_out; - - realsize = kmalloc_caches[index].objsize; - text =kasprintf (flags & ~SLUB_DMA ,"kmalloc_dma-%d" , - (unsigned int )realsize); - - s =NULL ; -for (i =0 ; i <KMALLOC_CACHES ; i++) -if (!kmalloc_caches[i].size) -break ; - -BUG_ON (i >=KMALLOC_CACHES ); - s = kmalloc_caches + i; - -/* -* Must defer sysfs creation to a workqueue because we don't know -* what context we are called from. Before sysfs comes up, we don't -* need to do anything because our sysfs initcall will start by -* adding all existing slabs to sysfs. -*/ - slabflags =SLAB_CACHE_DMA |SLAB_NOTRACK ; -if (slab_state >=SYSFS ) - slabflags |= __SYSFS_ADD_DEFERRED; - -if (!text || !kmem_cache_open (s, flags, text, - realsize,ARCH_KMALLOC_MINALIGN , slabflags,NULL )) { - s->size =0 ; -kfree (text); - goto unlock_out; - } - -list_add (&s->list, &slab_caches); - kmalloc_caches_dma[index] = s; - -if (slab_state >=SYSFS ) -schedule_work (&sysfs_add_work); - -unlock_out: -up_write (&slub_lock); -out: -return kmalloc_caches_dma[index]; -} -#endif -/* * Conversion table for small slabs sizes / 8 to the index in the * kmalloc array. This is necessary for slabs < 192 since we have non power @@ -2715,7 +2629,7 @@ static struct kmem_cache *get_slab(size_#ifdef CONFIG_ZONE_DMA if (unlikely ((flags &SLUB_DMA ))) -return dma_kmalloc_cache (index, flags); +return &kmalloc_dma_caches[index];#endif return &kmalloc_caches[index];@@ -3053,7 +2967,7 @@ void __init kmem_cache_init(void)* kmem_cache_open for slab_state == DOWN. */ create_kmalloc_cache (&kmalloc_caches[0 ],"kmem_cache_node" , -sizeof (struct kmem_cache_node ),GFP_NOWAIT ); +sizeof (struct kmem_cache_node ),0 ); kmalloc_caches[0 ].refcount =-1 ; caches++;@@ -3066,18 +2980,18 @@ void __init kmem_cache_init(void)/* Caches that are not of the two-to-the-power-of size */ if (KMALLOC_MIN_SIZE <=32 ) {create_kmalloc_cache (&kmalloc_caches[1 ], -"kmalloc-96" ,96 ,GFP_NOWAIT ); +"kmalloc-96" ,96 ,0 ); caches++; }if (KMALLOC_MIN_SIZE <=64 ) {create_kmalloc_cache (&kmalloc_caches[2 ], -"kmalloc-192" ,192 ,GFP_NOWAIT ); +"kmalloc-192" ,192 ,0 ); caches++; }for (i =KMALLOC_SHIFT_LOW ; i <SLUB_PAGE_SHIFT ; i++) {create_kmalloc_cache (&kmalloc_caches[i], -"kmalloc" ,1 << i,GFP_NOWAIT ); +"kmalloc" ,1 << i,0 ); caches++; }@@ -3124,7 +3038,7 @@ void __init kmem_cache_init(void)/* Provide the correct kmalloc names now that the caches are up */ for (i =KMALLOC_SHIFT_LOW ; i <SLUB_PAGE_SHIFT ; i++) - kmalloc_caches[i]. name = + kmalloc_caches[i].name =kasprintf (GFP_NOWAIT ,"kmalloc-%d" ,1 << i);#ifdef CONFIG_SMP @@ -3147,6 +3061,21 @@ void __init kmem_cache_init(void)void __init kmem_cache_init_late (void ) {
mm/slub.c void __init kmem_cache_init(void) · line 3073
Quoted 10 lines
mm/slub.c +10
-0+#ifdef CONFIG_ZONE_DMA +int i; + +for (i =0 ; i <SLUB_PAGE_SHIFT ; i++) { +struct kmem_cache *s = &kmalloc_caches[i]; + +if (s && s->size) { +char *name =kasprintf (GFP_KERNEL , +"dma-kmalloc-%d" , s->objsize); +
You're still not handling the case where !name, which kasprintf() can return both here and in kmem_cache_init(). Nameless caches aren't allowed for CONFIG_SLUB_DEBUG.
Quoted 19 lines
mm/slub.c +6
-2+create_kmalloc_cache (&kmalloc_dma_caches[i], + name, s->objsize, SLAB_CACHE_DMA); + } + } +#endif }/* @@ -3241,7 +3170,7 @@ struct kmem_cache *kmem_cache_create(con s =kmalloc (kmem_size,GFP_KERNEL );if (s) { -if (kmem_cache_open (s,GFP_KERNEL , name, +if (kmem_cache_open (s, name, size, align, flags, ctor)) {list_add (&s->list, &slab_caches);up_write (&slub_lock); --
Quoted 5 lines
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 08/16] slub: remove dynamic dma slab allocation
Quoted 9 lines · David Rientjes
mm/slub.c static void early_kmem_cache_node_alloc( · line 2111
Quoted 5 lines
mm/slub.c +1
-1- page =new_slab (kmalloc_caches, gfpflags, node); + page =new_slab (kmalloc_caches,GFP_KERNEL , node);BUG_ON (!page);if (page_to_nid (page) != node) {This still passes
GFP_KERNELto the page allocator when not allowed bygfp_allowed_maskfor early (nonSLAB_CACHE_DMA) users ofcreate_kmalloc_cache().
Right a later patch changes that. I could fold that hunk in here.
Quoted 12 lines
Quoted 12 lines
mm/slub.c +2
-2@@ -2157,11 +2157,11 @@ static int init_kmem_cache_nodes(structstruct kmem_cache_node *n;if (slab_state ==DOWN ) { -early_kmem_cache_node_alloc (gfpflags, node); +early_kmem_cache_node_alloc (node);continue ; } n =kmem_cache_alloc_node (kmalloc_caches, - gfpflags, node); +GFP_KERNEL , node);
Quoted 6 lines
mm/slub.c static int init_kmem_cache_nodes(struct · line 2167
Quoted 2 lines
mm/slub.c +0
-0if (!n) {free_kmem_cache_nodes (s);
slab_state!= DOWN is still not an indication thatGFP_KERNELis safe; in fact, all users ofGFP_KERNELfromkmem_cache_init()are unsafe. These need to beGFP_NOWAIT.
slab_state == DOWN is a sure indicator that kmem_cache_alloc_node is not functional. That is what we need to know here.
Quoted 6 lines
Quoted 6 lines
mm/slub.c +6
-0+#ifdef CONFIG_ZONE_DMA +int i; + +for (i =0 ; i <SLUB_PAGE_SHIFT ; i++) { +struct kmem_cache *s = &kmalloc_caches[i]; +
Quoted 8 lines
mm/slub.c void __init kmem_cache_init(void) · line 3073
Quoted 4 lines
mm/slub.c +4
-0+if (s && s->size) { +char *name =kasprintf (GFP_KERNEL , +"dma-kmalloc-%d" , s->objsize); +You're still not handling the case where !name, which
kasprintf()can return both here and inkmem_cache_init(). Nameless caches aren't allowed forCONFIG_SLUB_DEBUG.
It was not handled before either. I can come up with a patch but frankly this is a rare corner case that does not have too high priority to get done.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 08/16] slub: remove dynamic dma slab allocation
In reply to Christoph LameterQuoted 5 lines · On Fri, 25 Jun 2010 16:20:34 -0500 Christoph Lameter <[email protected]>…
Remove the dynamic dma slab allocation since this causes too many issues with nested locks etc etc. The change avoids passing gfpflags into many functions.
Signed-off-by Christoph Lameter <[email protected]>
Uh...I think just using GFP_KERNEL drops too much requests-from-user-via-gfp_mask.
How about this ?
gfp_mask = (gfp_mask & GFP_RECLAIM_MASK);
if you want to drop __GFP_DMA.
Thanks,
-Kame
Quoted 304 lines
--- mm/slub.c | 153 ++++++++++++++++---------------------------------------------- 1 file changed, 41 insertions(+), 112 deletions(-)Index:
linux-2.6/mm/slub.c
===================================================================mm/slub.c +41
-113@@ -2070,7 +2070,7 @@ init_kmem_cache_node(struct kmem_cache_n static DEFINE_PER_CPU(struct kmem_cache_cpu, kmalloc_percpu[KMALLOC_CACHES]); -static inline int alloc_kmem_cache_cpus (struct kmem_cache *s,gfp_t flags) +static inline int alloc_kmem_cache_cpus (struct kmem_cache *s) {if (s < kmalloc_caches +KMALLOC_CACHES && s >= kmalloc_caches)/* @@ -2097,7 +2097,7 @@ static inline int alloc_kmem_cache_cpus(* when allocating for the kmalloc_node_cache. This is used for bootstrapping * memory on a fresh node that has no slab structures yet. */ -static void early_kmem_cache_node_alloc (gfp_t gfpflags,int node) +static void early_kmem_cache_node_alloc (int node) {struct page *page;struct kmem_cache_node *n;@@ -2105,7 +2105,7 @@ static void early_kmem_cache_node_alloc(BUG_ON (kmalloc_caches->size <sizeof (struct kmem_cache_node )); - page =new_slab (kmalloc_caches, gfpflags, node); + page =new_slab (kmalloc_caches,GFP_KERNEL , node);BUG_ON (!page);if (page_to_nid (page) != node) {@@ -2149,7 +2149,7 @@ static void free_kmem_cache_nodes(struct } } -static int init_kmem_cache_nodes (struct kmem_cache *s,gfp_t gfpflags) +static int init_kmem_cache_nodes (struct kmem_cache *s) {int node;@@ -2157,11 +2157,11 @@ static int init_kmem_cache_nodes(structstruct kmem_cache_node *n;if (slab_state ==DOWN ) { -early_kmem_cache_node_alloc (gfpflags, node); +early_kmem_cache_node_alloc (node);continue ; } n =kmem_cache_alloc_node (kmalloc_caches, - gfpflags, node); +GFP_KERNEL , node);if (!n) {free_kmem_cache_nodes (s);@@ -2178,7 +2178,7 @@ static void free_kmem_cache_nodes(struct { } -static int init_kmem_cache_nodes (struct kmem_cache *s,gfp_t gfpflags) +static int init_kmem_cache_nodes (struct kmem_cache *s) {init_kmem_cache_node (&s->local_node, s);return 1 ;@@ -2318,7 +2318,7 @@ static int calculate_sizes(struct kmem_c } -static int kmem_cache_open (struct kmem_cache *s,gfp_t gfpflags, +static int kmem_cache_open (struct kmem_cache *s,const char *name,size_t size,size_t align,unsigned long flags,void (*ctor)(void *))@@ -2354,10 +2354,10 @@ static int kmem_cache_open(struct kmem_c#ifdef CONFIG_NUMA s->remote_node_defrag_ratio =1000 ;#endif -if (!init_kmem_cache_nodes (s, gfpflags & ~SLUB_DMA )) +if (!init_kmem_cache_nodes (s)) goto error; -if (alloc_kmem_cache_cpus (s, gfpflags & ~SLUB_DMA )) +if (alloc_kmem_cache_cpus (s))return 1 ;free_kmem_cache_nodes (s);@@ -2517,6 +2517,10 @@ EXPORT_SYMBOL(kmem_cache_destroy); struct kmem_cache kmalloc_caches[KMALLOC_CACHES] __cacheline_aligned;EXPORT_SYMBOL (kmalloc_caches); +#ifdef CONFIG_ZONE_DMA +static struct kmem_cache kmalloc_dma_caches[SLUB_PAGE_SHIFT ]; +#endif +static int __init setup_slub_min_order (char *str) {get_option (&str, &slub_min_order);@@ -2553,116 +2557,26 @@ static int __init setup_slub_nomerge(cha__setup ("slub_nomerge" , setup_slub_nomerge); -static struct kmem_cache *create_kmalloc_cache (struct kmem_cache *s, -const char *name,int size,gfp_t gfp_flags) +static void create_kmalloc_cache (struct kmem_cache *s, +const char *name,int size,unsigned int flags) { -unsigned int flags =0 ; - -if (gfp_flags &SLUB_DMA ) - flags =SLAB_CACHE_DMA ; -/* * This function is called with IRQs disabled during early-boot on * single CPU so there's no need to take slub_lock here. */ -if (!kmem_cache_open (s, gfp_flags, name, size,ARCH_KMALLOC_MINALIGN , +if (!kmem_cache_open (s, name, size,ARCH_KMALLOC_MINALIGN , flags,NULL )) goto panic;list_add (&s->list, &slab_caches); -if (sysfs_slab_add (s)) - goto panic; -return s; +if (!sysfs_slab_add (s)) +return ; panic:panic ("Creation of kmalloc slab %s size=%d failed.\n" , name, size); } -#ifdef CONFIG_ZONE_DMA -static struct kmem_cache *kmalloc_caches_dma[SLUB_PAGE_SHIFT ]; - -static void sysfs_add_func (struct work_struct *w) -{ -struct kmem_cache *s; - -down_write (&slub_lock); -list_for_each_entry (s, &slab_caches, list) { -if (s->flags & __SYSFS_ADD_DEFERRED) { - s->flags &= ~__SYSFS_ADD_DEFERRED; -sysfs_slab_add (s); - } - } -up_write (&slub_lock); -} - -static DECLARE_WORK(sysfs_add_work, sysfs_add_func); - -static noinline struct kmem_cache *dma_kmalloc_cache(int index, gfp_t flags) -{ -struct kmem_cache *s; -char *text; -size_t realsize; -unsigned long slabflags; -int i; - - s = kmalloc_caches_dma[index]; -if (s) -return s; - -/* Dynamically create dma cache */ -if (flags & __GFP_WAIT) -down_write (&slub_lock); -else { -if (!down_write_trylock (&slub_lock)) - goto out; - } - -if (kmalloc_caches_dma[index]) - goto unlock_out; - - realsize = kmalloc_caches[index].objsize; - text =kasprintf (flags & ~SLUB_DMA ,"kmalloc_dma-%d" , - (unsigned int )realsize); - - s =NULL ; -for (i =0 ; i <KMALLOC_CACHES ; i++) -if (!kmalloc_caches[i].size) -break ; - -BUG_ON (i >=KMALLOC_CACHES ); - s = kmalloc_caches + i; - -/* -* Must defer sysfs creation to a workqueue because we don't know -* what context we are called from. Before sysfs comes up, we don't -* need to do anything because our sysfs initcall will start by -* adding all existing slabs to sysfs. -*/ - slabflags =SLAB_CACHE_DMA |SLAB_NOTRACK ; -if (slab_state >=SYSFS ) - slabflags |= __SYSFS_ADD_DEFERRED; - -if (!text || !kmem_cache_open (s, flags, text, - realsize,ARCH_KMALLOC_MINALIGN , slabflags,NULL )) { - s->size =0 ; -kfree (text); - goto unlock_out; - } - -list_add (&s->list, &slab_caches); - kmalloc_caches_dma[index] = s; - -if (slab_state >=SYSFS ) -schedule_work (&sysfs_add_work); - -unlock_out: -up_write (&slub_lock); -out: -return kmalloc_caches_dma[index]; -} -#endif -/* * Conversion table for small slabs sizes / 8 to the index in the * kmalloc array. This is necessary for slabs < 192 since we have non power @@ -2715,7 +2629,7 @@ static struct kmem_cache *get_slab(size_#ifdef CONFIG_ZONE_DMA if (unlikely ((flags &SLUB_DMA ))) -return dma_kmalloc_cache (index, flags); +return &kmalloc_dma_caches[index];#endif return &kmalloc_caches[index];@@ -3053,7 +2967,7 @@ void __init kmem_cache_init(void)* kmem_cache_open for slab_state == DOWN. */ create_kmalloc_cache (&kmalloc_caches[0 ],"kmem_cache_node" , -sizeof (struct kmem_cache_node ),GFP_NOWAIT ); +sizeof (struct kmem_cache_node ),0 ); kmalloc_caches[0 ].refcount =-1 ; caches++;@@ -3066,18 +2980,18 @@ void __init kmem_cache_init(void)/* Caches that are not of the two-to-the-power-of size */ if (KMALLOC_MIN_SIZE <=32 ) {create_kmalloc_cache (&kmalloc_caches[1 ], -"kmalloc-96" ,96 ,GFP_NOWAIT ); +"kmalloc-96" ,96 ,0 ); caches++; }if (KMALLOC_MIN_SIZE <=64 ) {create_kmalloc_cache (&kmalloc_caches[2 ], -"kmalloc-192" ,192 ,GFP_NOWAIT ); +"kmalloc-192" ,192 ,0 ); caches++; }for (i =KMALLOC_SHIFT_LOW ; i <SLUB_PAGE_SHIFT ; i++) {create_kmalloc_cache (&kmalloc_caches[i], -"kmalloc" ,1 << i,GFP_NOWAIT ); +"kmalloc" ,1 << i,0 ); caches++; }@@ -3124,7 +3038,7 @@ void __init kmem_cache_init(void)/* Provide the correct kmalloc names now that the caches are up */ for (i =KMALLOC_SHIFT_LOW ; i <SLUB_PAGE_SHIFT ; i++) - kmalloc_caches[i]. name = + kmalloc_caches[i].name =kasprintf (GFP_NOWAIT ,"kmalloc-%d" ,1 << i);#ifdef CONFIG_SMP @@ -3147,6 +3061,21 @@ void __init kmem_cache_init(void)void __init kmem_cache_init_late (void ) { +#ifdef CONFIG_ZONE_DMA +int i; + +for (i =0 ; i <SLUB_PAGE_SHIFT ; i++) { +struct kmem_cache *s = &kmalloc_caches[i]; + +if (s && s->size) { +char *name =kasprintf (GFP_KERNEL , +"dma-kmalloc-%d" , s->objsize); + +create_kmalloc_cache (&kmalloc_dma_caches[i], + name, s->objsize,SLAB_CACHE_DMA ); + } + } +#endif }/* @@ -3241,7 +3170,7 @@ struct kmem_cache *kmem_cache_create(con s =kmalloc (kmem_size,GFP_KERNEL );if (s) { -if (kmem_cache_open (s,GFP_KERNEL , name, +if (kmem_cache_open (s, name, size, align, flags, ctor)) {list_add (&s->list, &slab_caches);up_write (&slub_lock); --
Quoted 4 lines
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 08/16] slub: remove dynamic dma slab allocation
Quoted 2 lines · KAMEZAWA Hiroyuki
Uh...I think just using
GFP_KERNELdrops too much
requests-from-user-via-gfp_mask.
Sorry I do not understand what the issue is? The dma slabs are allocated while user space is not active yet.
Please do not quote diff hunks that you do not comment on. I am on a slow link (vacation) and its awkward to check for comments...
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 08/16] slub: remove dynamic dma slab allocation
Quoted 8 lines · On Tue, 29 Jun 2010 10:41:59 -0500 (CDT) Christoph Lameter…
Quoted 2 lines · KAMEZAWA Hiroyuki
Uh...I think just using
GFP_KERNELdrops too much
requests-from-user-via-gfp_mask.Sorry I do not understand what the issue is? The dma slabs are allocated while user space is not active yet.
Sorry, I misunderstood the patch. It seems ok, now.
Quoted 2 lines
Please do not quote diff hunks that you do not comment on. I am on a slow link (vacation) and its awkward to check for comments...
Sure.
Thanks,
-Kame
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 9/16 make allocpercpu usable during early boot
In reply to Christoph Lameterallocpercpu() may be used during early boot after the page allocator has been bootstrapped but when interrupts are still off. Make sure that we do not do GFP_KERNEL allocations if this occurs.
--- mm/percpu.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/percpu.c +3 -2
@@ -275,7 +275,8 @@ static void __maybe_unused pcpu_next_pop* memory is always zeroed. * * CONTEXT: -* Does GFP_KERNEL allocation. +* Does GFP_KERNEL allocation (May be called early in boot when +* interrupts are still disabled. Will then do GFP_NOWAIT alloc). * * RETURNS: * Pointer to the allocated area on success, NULL on failure.
@@ -286,7 +287,7 @@ static void *pcpu_mem_alloc(size_t size)return NULL ;if (size <=PAGE_SIZE ) -return kzalloc (size,GFP_KERNEL ); +return kzalloc (size,GFP_KERNEL & gfp_allowed_mask);else {void *ptr =vmalloc (size);if (ptr)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 09/16] [percpu] make allocpercpu usable during early boot
Quoted 6 lines · Christoph Lameter
allocpercpu()may be used during early boot after the page allocator has been bootstrapped but when interrupts are still off. Make sure that we do not doGFP_KERNELallocations if this occurs.Signed-off-by Christoph Lameter <[email protected]>
Christoph, how do you wanna route these patches? I already have the other two patches in the percpu tree, I can push this there too, which then you can pull into the allocator tree.
Thanks.
Signature · 7 lines
tejun -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerRe: [S+Q 09/16] [percpu] make allocpercpu usable during early boot
Quoted 5 lines · Tejun Heo
Quoted 4 lines · Christoph Lameter
allocpercpu()may be used during early boot after the page allocator has been bootstrapped but when interrupts are still off. Make sure that we do not doGFP_KERNELallocations if this occurs.
Quoted 9 lines
Quoted 2 lines
Signed-off-by Christoph Lameter <[email protected]>Acked-by Tejun Heo <[email protected]>Christoph, how do you wanna route these patches? I already have the other two patches in the percpu tree, I can push this there too, which then you can pull into the allocator tree.
I think that's great for patches 2 and 3 in this series, but this patch is only a bandaid for allocations done in early boot whereas the real fix should be within a lower layer such as the slab or page allocator since the irq context on the boot cpu is not specific only to percpu.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 09/16] [percpu] make allocpercpu usable during early boot
In reply to Tejun HeoQuoted 3 lines · Tejun Heo
Christoph, how do you wanna route these patches? I already have the other two patches in the percpu tree, I can push this there too, which then you can pull into the allocator tree.
Please push via your trees. Lets keep stuff subsystem specific if possible.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Tejun Heo
maintainerRe: [S+Q 09/16] [percpu] make allocpercpu usable during early boot
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Tejun Heo
Christoph, how do you wanna route these patches? I already have the other two patches in the percpu tree, I can push this there too, which then you can pull into the allocator tree.
Please push via your trees. Lets keep stuff subsystem specific if possible.
Sure, please feel free to pull from the following tree.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu.git for-next
I haven't committed the gfp_allowed_mask patch yet. I'll commit it once it gets resolved.
Thanks.
Signature · 7 lines
tejun -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 09/16] [percpu] make allocpercpu usable during early boot
Quoted 2 lines · Tejun Heo
I haven't committed the
gfp_allowed_maskpatch yet. I'll commit it once it gets resolved.
Dont commit. I dropped it myself.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerRe: [S+Q 09/16] [percpu] make allocpercpu usable during early boot
In reply to Christoph LameterQuoted 4 lines · Christoph Lameter
allocpercpu()may be used during early boot after the page allocator has been bootstrapped but when interrupts are still off. Make sure that we do not doGFP_KERNELallocations if this occurs.
Why isn't this being handled at a lower level, specifically in the slab allocator to prevent GFP_KERNEL from being used when irqs are disabled? We'll otherwise need to audit all slab allocations from the boot cpu for correctness.
Quoted 22 lines
Signed-off-by Christoph Lameter <[email protected]>--- mm/percpu.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)Index:
linux-2.6/mm/percpu.c
===================================================================mm/percpu.c +2
-1@@ -275,7 +275,8 @@ static void __maybe_unused pcpu_next_pop* memory is always zeroed. * * CONTEXT: -* Does GFP_KERNEL allocation. +* Does GFP_KERNEL allocation (May be called early in boot when +* interrupts are still disabled. Will then do GFP_NOWAIT alloc). * * RETURNS: * Pointer to the allocated area on success, NULL on failure.
mm/percpu.c static void *pcpu_mem_alloc(size_t size) · line 293
Quoted 8 lines
mm/percpu.c +1
-1@@ -286,7 +287,7 @@ static void *pcpu_mem_alloc(size_t size)return NULL ;if (size <=PAGE_SIZE ) -return kzalloc (size,GFP_KERNEL ); +return kzalloc (size,GFP_KERNEL & gfp_allowed_mask);else {void *ptr =vmalloc (size);if (ptr)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 09/16] [percpu] make allocpercpu usable during early boot
Quoted 9 lines · David Rientjes
Quoted 3 lines · Christoph Lameter
allocpercpu()may be used during early boot after the page allocator has been bootstrapped but when interrupts are still off. Make sure that we do not doGFP_KERNELallocations if this occurs.Why isn't this being handled at a lower level, specifically in the slab allocator to prevent
GFP_KERNELfrom being used when irqs are disabled? We'll otherwise need to audit all slab allocations from the boot cpu for correctness.
It is handled at a lower level when slab allocates from the page allocator. But the checking logic for the proper flags passed to the slab allocator does not mask the bits and it seems that this approach is the way people want it to be. So we have to explicitly mask GFP_KERNEL in these locations.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 09/16] [percpu] make allocpercpu usable during early boot
In reply to Christoph LameterQuoted 26 lines · Christoph Lameter
allocpercpu()may be used during early boot after the page allocator has been bootstrapped but when interrupts are still off. Make sure that we do not doGFP_KERNELallocations if this occurs.Signed-off-by Christoph Lameter <[email protected]>---
mm/percpu.c | 5 +++--1 file changed, 3 insertions(+), 2 deletions(-)Index:
linux-2.6/mm/percpu.c
===================================================================mm/percpu.c 2010-06-23 14:44:05.000000000 -0500 renamed from mm/percpu.c 2010-06-23 14:43:54.000000000 -0500 +2
-1@@ -275,7 +275,8 @@ static void __maybe_unused pcpu_next_pop * memory is always zeroed. * * CONTEXT: - * Does GFP_KERNEL allocation. + * Does GFP_KERNEL allocation (May be called early in boot when + * interrupts are still disabled. Will then do GFP_NOWAIT alloc). * * RETURNS: * Pointer to the allocated area on success, NULL on failure.
mm/percpu.c 2010-06-23 14:44:05.000000000 -0500 static void *pcpu_mem_alloc(size_t size) · line 293
Quoted 8 lines
mm/percpu.c 2010-06-23 14:44:05.000000000 -0500 renamed from mm/percpu.c 2010-06-23 14:43:54.000000000 -0500 +1
-1@@ -286,7 +287,7 @@ static void *pcpu_mem_alloc(size_t size) return NULL; if (size <= PAGE_SIZE) - return kzalloc(size, GFP_KERNEL); + return kzalloc(size, GFP_KERNEL & gfp_allowed_mask); else { void *ptr = vmalloc(size); if (ptr)
This looks wrong to me. All slab allocators should do gfp_allowed_mask magic under the hood. Maybe it's triggering kmalloc_large() path that needs the masking too?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 09/16] [percpu] make allocpercpu usable during early boot
Quoted 8 lines · Pekka Enberg
Quoted 4 lines
+ return
kzalloc(size, GFP_KERNEL & gfp_allowed_mask); else {
void *ptr = vmalloc(size);
if (ptr)This looks wrong to me. All slab allocators should do
gfp_allowed_maskmagic under the hood. Maybe it's triggeringkmalloc_large()path that needs the masking too?
They do gfp_allowed_mask magic. But the checks at function entry of the slabs do not mask the masks so we get false positives without this. All my protest against the checks doing it this IMHO broken way were ignored.
Re: [S+Q 09/16] [percpu] make allocpercpu usable during early boot
Quoted 9 lines · Pekka Enberg
Quoted 8 lines
Quoted 4 lines
+ return
kzalloc(size, GFP_KERNEL & gfp_allowed_mask); else {
void *ptr = vmalloc(size);
if (ptr)This looks wrong to me. All slab allocators should do
gfp_allowed_maskmagic under the hood. Maybe it's triggeringkmalloc_large()path that needs the masking too?
Quoted 3 lines · Christoph Lameter
They do
gfp_allowed_maskmagic. But the checks at function entry of the slabs do not mask the masks so we get false positives without this. All my protest against the checks doing it this IMHO broken way were ignored.
Which checks are those? Are they in SLUB proper or are they introduced in one of the SLEB patches? We definitely don't want to expose gfp_allowed_mask here.
Pekka
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 09/16] [percpu] make allocpercpu usable during early boot
Quoted 12 lines · Pekka Enberg
Quoted 9 lines · Pekka Enberg
Quoted 8 lines
Quoted 4 lines
+ return
kzalloc(size, GFP_KERNEL & gfp_allowed_mask); else {
void *ptr = vmalloc(size);
if (ptr)This looks wrong to me. All slab allocators should do
gfp_allowed_maskmagic under the hood. Maybe it's triggeringkmalloc_large()path that needs the masking too?On Tue, Jun 29, 2010 at 6:45 PM, Christoph Lameter <[email protected]> wrote:
Quoted 7 lines
Quoted 3 lines
They do
gfp_allowed_maskmagic. But the checks at function entry of the slabs do not mask the masks so we get false positives without this. All my protest against the checks doing it this IMHO broken way were ignored.Which checks are those? Are they in SLUB proper or are they introduced in one of the SLEB patches? We definitely don't want to expose
gfp_allowed_maskhere.
Argh. The reason for the trouble here is because I moved the masking of the gfp flags out of the hot path.
The masking of the bits adds to the cache footprint of the hotpaths now in all slab allocators. Gosh. Why is there constant contamination of the hot paths with the stuff?
We only need this masking in the hot path if the debugging hooks need it. Otherwise its fine to defer this to the slow paths.
So how do I get that in there? Add "& gfp_allowed_mask" to the gfp mask passed to the debugging hooks?
Or add a debug_hooks_alloc() function and make it empty if no debugging functions are enabled?
Re: [S+Q 09/16] [percpu] make allocpercpu usable during early boot
Quoted 19 lines · Christoph Lameter
Quoted 8 lines · Pekka Enberg
Quoted 8 lines · Pekka Enberg
Quoted 7 lines
Quoted 4 lines
+ return kzalloc(size, GFP_KERNEL & gfp_allowed_mask); else { void *ptr = vmalloc(size); if (ptr)This looks wrong to me. All slab allocators should do
gfp_allowed_maskmagic under the hood. Maybe it's triggeringkmalloc_large()path that needs the masking too?Quoted 8 lines
Quoted 3 lines · Christoph Lameter
They do
gfp_allowed_maskmagic. But the checks at function entry of the slabs do not mask the masks so we get false positives without this. All my protest against the checks doing it this IMHO broken way were ignored.Which checks are those? Are they in SLUB proper or are they introduced in one of the SLEB patches? We definitely don't want to expose
gfp_allowed_maskhere.
Quoted 6 lines
Argh. The reason for the trouble here is because I moved the masking of the gfp flags out of the hot path.
The masking of the bits adds to the cache footprint of the hotpaths now in all slab allocators. Gosh. Why is there constant contamination of the hot paths with the stuff?
We should definitely move gfp_allowed_mask masking out of fast-paths. I think the ideal solution is to only do it deep in the page allocator.
Quoted 8 lines
We only need this masking in the hot path if the debugging hooks need it. Otherwise its fine to defer this to the slow paths.
So how do I get that in there? Add "&
gfp_allowed_mask" to the gfp mask passed to the debugging hooks?Or add a
debug_hooks_alloc()function and make it empty if no debugging functions are enabled?
I think it's best to have separate debugging hooks that are empty if debugging is disabled, yes.
Pekka
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 10/16 slub: Remove static kmem_cache_cpu array for boot
In reply to Christoph LameterThe percpu allocator can now handle allocations in early boot. So drop the static kmem_cache_cpu array.
Early memory allocations require the use of GFP_NOWAIT instead of GFP_KERNEL. Mask GFP_KERNEL with gfp_allowed_mask to get to GFP_NOWAIT in a boot scenario.
--- mm/slub.c | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-)
mm/slub.c +6 -15
@@ -2068,23 +2068,14 @@ init_kmem_cache_node(struct kmem_cache_n#endif } -static DEFINE_PER_CPU(struct kmem_cache_cpu, kmalloc_percpu[KMALLOC_CACHES]); -static inline int alloc_kmem_cache_cpus (struct kmem_cache *s) { -if (s < kmalloc_caches +KMALLOC_CACHES && s >= kmalloc_caches) -/* -* Boot time creation of the kmalloc array. Use static per cpu data -* since the per cpu allocator is not available yet. -*/ - s->cpu_slab = kmalloc_percpu + (s - kmalloc_caches); -else - s->cpu_slab = alloc_percpu(struct kmem_cache_cpu); +BUILD_BUG_ON (PERCPU_DYNAMIC_EARLY_SIZE < +SLUB_PAGE_SHIFT *sizeof (struct kmem_cache )); -if (!s->cpu_slab) -return 0 ; + s->cpu_slab = alloc_percpu(struct kmem_cache_cpu); -return 1 ; +return s->cpu_slab !=NULL ; }#ifdef CONFIG_NUMA
@@ -2105,7 +2096,7 @@ static void early_kmem_cache_node_alloc(BUG_ON (kmalloc_caches->size <sizeof (struct kmem_cache_node )); - page =new_slab (kmalloc_caches,GFP_KERNEL , node); + page =new_slab (kmalloc_caches,GFP_KERNEL & gfp_allowed_mask, node);BUG_ON (!page);if (page_to_nid (page) != node) {
@@ -2161,7 +2152,7 @@ static int init_kmem_cache_nodes(structcontinue ; } n =kmem_cache_alloc_node (kmalloc_caches, -GFP_KERNEL , node); +GFP_KERNEL & gfp_allowed_mask, node);if (!n) {free_kmem_cache_nodes (s);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerRe: [S+Q 10/16] slub: Remove static kmem_cache_cpu array for boot
Quoted 47 lines · Christoph Lameter
The percpu allocator can now handle allocations in early boot. So drop the static
kmem_cache_cpuarray.Early memory allocations require the use of
GFP_NOWAITinstead ofGFP_KERNEL. MaskGFP_KERNELwithgfp_allowed_maskto get toGFP_NOWAITin a boot scenario.Cc Tejun Heo <[email protected]>Signed-off-by Christoph Lameter <[email protected]>--- mm/slub.c | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-)Index:
linux-2.6.34/mm/slub.c
===================================================================mm/slub.c +4
-13@@ -2068,23 +2068,14 @@ init_kmem_cache_node(struct kmem_cache_n#endif } -static DEFINE_PER_CPU(struct kmem_cache_cpu, kmalloc_percpu[KMALLOC_CACHES]); -static inline int alloc_kmem_cache_cpus (struct kmem_cache *s) { -if (s < kmalloc_caches +KMALLOC_CACHES && s >= kmalloc_caches) -/* -* Boot time creation of the kmalloc array. Use static per cpu data -* since the per cpu allocator is not available yet. -*/ - s->cpu_slab = kmalloc_percpu + (s - kmalloc_caches); -else - s->cpu_slab = alloc_percpu(struct kmem_cache_cpu); +BUILD_BUG_ON (PERCPU_DYNAMIC_EARLY_SIZE < +SLUB_PAGE_SHIFT *sizeof (struct kmem_cache )); -if (!s->cpu_slab) -return 0 ; + s->cpu_slab = alloc_percpu(struct kmem_cache_cpu); -return 1 ; +return s->cpu_slab !=NULL ; }#ifdef CONFIG_NUMA
mm/slub.c static void early_kmem_cache_node_alloc( · line 2102
Quoted 8 lines
mm/slub.c +1
-1@@ -2105,7 +2096,7 @@ static void early_kmem_cache_node_alloc(BUG_ON (kmalloc_caches->size <sizeof (struct kmem_cache_node )); - page =new_slab (kmalloc_caches,GFP_KERNEL , node); + page =new_slab (kmalloc_caches,GFP_KERNEL & gfp_allowed_mask, node);BUG_ON (!page);if (page_to_nid (page) != node) {
This needs to be merged into the preceding patch since it had broken new slab allocations during early boot while irqs are still disabled; it also seems deserving of a big fat comment about why it's required in this situation.
mm/slub.c static int init_kmem_cache_nodes(struct · line 2158
Quoted 9 lines
mm/slub.c +1
-1@@ -2161,7 +2152,7 @@ static int init_kmem_cache_nodes(structcontinue ; } n =kmem_cache_alloc_node (kmalloc_caches, -GFP_KERNEL , node); +GFP_KERNEL & gfp_allowed_mask, node);if (!n) {free_kmem_cache_nodes (s);
Likewise.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 10/16] slub: Remove static kmem_cache_cpu array for boot
Quoted 7 lines · David Rientjes
Quoted 7 lines
mm/slub.c +1
-1@@ -2105,7 +2096,7 @@ static void early_kmem_cache_node_alloc(BUG_ON (kmalloc_caches->size <sizeof (struct kmem_cache_node )); - page =new_slab (kmalloc_caches,GFP_KERNEL , node); + page =new_slab (kmalloc_caches,GFP_KERNEL & gfp_allowed_mask, node);
Quoted 7 lines
mm/slub.c static void early_kmem_cache_node_alloc( · line 2102
Quoted 2 lines
mm/slub.c +0
-0BUG_ON (!page);if (page_to_nid (page) != node) {This needs to be merged into the preceding patch since it had broken new slab allocations during early boot while irqs are still disabled; it also seems deserving of a big fat comment about why it's required in this situation.
AFAICT The earlier patch did not break anything but leave existing behavior the way it was. Breakage would occur in this patch because it results in allocations occurring earlier during boot.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 11/16 slub: Dynamically size kmalloc cache allocations
In reply to Christoph Lameterkmalloc caches are statically defined and may take up a lot of space just because the sizes of the node array has to be dimensioned for the largest node count supported.
This patch makes the size of the kmem_cache structure dynamic throughout by creating a kmem_cache slab cache for the kmem_cache objects. The bootstrap occurs by allocating the initial one or two kmem_cache objects from the page allocator.
--- include/linux/slub_def.h | 7 - mm/slub.c | 182 +++++++++++++++++++++++++++++++++++------------ 2 files changed, 139 insertions(+), 50 deletions(-)
include/linux/slub_def.h +2 -5
@@ -136,19 +136,16 @@ struct kmem_cache {#ifdef CONFIG_ZONE_DMA #define SLUB_DMA __GFP_DMA -/* Reserve extra caches for potential DMA use */ -#define KMALLOC_CACHES (2 * SLUB_PAGE_SHIFT)#else /* Disable DMA functionality */ #define SLUB_DMA (__force gfp_t)0 -#define KMALLOC_CACHES SLUB_PAGE_SHIFT#endif /* * We keep the general caches in an array of slab caches that are used for * 2^x bytes of allocations. */ -extern struct kmem_cache kmalloc_caches[KMALLOC_CACHES ]; +extern struct kmem_cache *kmalloc_caches[SLUB_PAGE_SHIFT ];/* * Sorry that the following has to be that ugly but some versions of GCC
@@ -213,7 +210,7 @@ static __always_inline struct kmem_cacheif (index ==0 )return NULL ; -return &kmalloc_caches[index]; +return kmalloc_caches[index]; }void *kmem_cache_alloc (struct kmem_cache *,gfp_t );
mm/slub.c +137 -45
301 lines not on this page · raw message
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 12/16 SLUB: Add SLAB style per cpu queueing
In reply to Christoph LameterThis patch adds SLAB style cpu queueing and uses a new way for
managing objects in the slabs using bitmaps. It uses a percpu queue so that
free operations can be properly buffered and a bitmap for managing the free/allocated state in the slabs. It uses slightly more memory (due to the need to place large bitmaps --sized a few words--in some slab pages) but in general does compete well in terms of space use. The storage format using bitmaps avoids the SLAB management structure that SLAB needs for each slab page and therefore the metadata is more compact and easily fits into a cacheline.
The SLAB scheme of not touching the object during management is adopted. SLUB can now efficiently free and allocate cache cold objects.
The queueing scheme addresses also the issue that the free slowpath was taken too frequently.
This patch only implements staticallly sized per cpu queues.
--- include/linux/slub_def.h | 10 mm/slub.c | 920 ++++++++++++++++++++--------------------------- 2 files changed, 416 insertions(+), 514 deletions(-)
include/linux/slub_def.h +6 -4
@@ -34,13 +34,16 @@ enum stat_item {ORDER_FALLBACK ,/* Number of times fallback was necessary */ NR_SLUB_STAT_ITEMS }; +#define QUEUE_SIZE 50 +#define BATCH_SIZE 25 +struct kmem_cache_cpu { -void **freelist;/* Pointer to first free per cpu object */ -struct page *page;/* The slab from which we are allocating */ -int node;/* The node of the page (or -1 for debug) */ #ifdef CONFIG_SLUB_STATS unsigned stat[NR_SLUB_STAT_ITEMS ];#endif +int objects;/* Number of objects available */ +int node;/* The node of the page (or -1 for debug) */ +void *object[QUEUE_SIZE ];/* List of objects */ }; struct kmem_cache_node {
@@ -72,7 +75,6 @@ struct kmem_cache {unsigned long flags;int size;/* The size of an object including meta data */ int objsize;/* The size of an object without meta data */ -int offset;/* Free pointer offset. */ struct kmem_cache_order_objects oo;/* Allocation and freeing of slabs */
mm/slub.c +410 -510
1,432 lines not on this page · raw message
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 12/16] SLUB: Add SLAB style per cpu queueing
Quoted 10 lines · Christoph Lameter
This patch adds SLAB style cpu queueing and uses a new way for
managing objects in the slabs using bitmaps. It uses a percpu queue so that
free operations can be properly buffered and a bitmap for managing the free/allocated state in the slabs. It uses slightly more memory (due to the need to place large bitmaps --sized a few words--in some slab pages) but in general does compete well in terms of space use. The storage format using bitmaps avoids the SLAB management structure that SLAB needs for each slab page and therefore the metadata is more compact and easily fits into a cacheline.
Quoted 2 lines
The SLAB scheme of not touching the object during management is adopted. SLUB can now efficiently free and allocate cache cold objects.
BTW. this was never the problem with SLUB, because SLQB didn't have the big performance regression on tpcc. SLUB IIRC had to touch more cachelines per operation.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 12/16] SLUB: Add SLAB style per cpu queueing
Quoted 6 lines · Nick Piggin
Quoted 2 lines
The SLAB scheme of not touching the object during management is adopted. SLUB can now efficiently free and allocate cache cold objects.
BTW. this was never the problem with SLUB, because SLQB didn't have the big performance regression on tpcc. SLUB IIRC had to touch more cachelines per operation.
Wish you were more detailed here. SLUB was designed for minimal cacheline footprint and always had an edge there. AFACT SLQB was able to address tpcc in some ways was because the issue with the hotpath on free was addressed. These were issues with atomic ops on free not cache footprint.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 13/16 SLUB: Resize the new cpu queues
In reply to Christoph LameterAllow resizing of cpu queue and batch size. This is done in the basic steps that are also followed by SLAB.
The statically allocated per cpu areas are removed since the per cpu allocator is already available when kmem_cache_init is called. We can dynamically size the per cpu data during bootstrap.
Careful: This means that the ->cpu pointer is becoming volatile. References to the ->cpu pointer either
A. Occur with interrupts disabled. This guarantees that nothing on the
processor itself interferes. This only serializes access to a single processor specific area.
B. Occur with slub_lock taken for operations on all per cpu areas.
Taking the slub_lock guarantees that no resizing operation will occur while accessing the percpu areas. The data in the percpu areas is volatile even with slub_lock since the alloc and free functions do not take slub_lock and will operate on fields of kmem_cache_cpu.
C. Are racy: This is true for the statistics. The ->cpu pointer must always
point to a valid kmem_cache_cpu area.
--- include/linux/slub_def.h | 9 -- mm/slub.c | 199 +++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 178 insertions(+), 30 deletions(-)
include/linux/slub_def.h +4 -5
@@ -34,16 +34,13 @@ enum stat_item {ORDER_FALLBACK ,/* Number of times fallback was necessary */ NR_SLUB_STAT_ITEMS }; -#define QUEUE_SIZE 50 -#define BATCH_SIZE 25 -struct kmem_cache_cpu {#ifdef CONFIG_SLUB_STATS unsigned stat[NR_SLUB_STAT_ITEMS ];#endif int objects;/* Number of objects available */ int node;/* The node of the page (or -1 for debug) */ -void *object[QUEUE_SIZE ];/* List of objects */ +void *object[];/* Dynamic alloc will allow larger sizes */ }; struct kmem_cache_node {
@@ -70,12 +67,14 @@ struct kmem_cache_order_objects {* Slab cache management. */ struct kmem_cache { -struct kmem_cache_cpu *cpu_slab; +struct kmem_cache_cpu *cpu;/* Used for retriving partial slabs etc */ unsigned long flags;int size;/* The size of an object including meta data */ int objsize;/* The size of an object without meta data */ struct kmem_cache_order_objects oo; +int queue;/* per cpu queue size */ +int batch;/* batch size */ /* Allocation and freeing of slabs */ struct kmem_cache_order_objects max;
mm/slub.c +174 -25
400 lines not on this page · raw message
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 14/16 SLUB: Get rid of useless function count_free()
In reply to Christoph Lametercount_free() == available()
--- mm/slub.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)
mm/slub.c +3 -8
@@ -1617,11 +1617,6 @@ static inline int node_match(struct kmemreturn 1 ; } -static int count_free (struct page *page) -{ -return available (page); -} -static unsigned long count_partial (struct kmem_cache_node *n,int (*get_count)(struct page *)) {
@@ -1670,7 +1665,7 @@ slab_out_of_memory(struct kmem_cache *s,if (!n)continue ; - nr_free =count_partial (n, count_free); + nr_free =count_partial (n, available); nr_slabs =node_nr_slabs (n); nr_objs =node_nr_objs (n);
@@ -3805,7 +3800,7 @@ static ssize_t show_slab_objects(struct x =atomic_long_read (&n->total_objects); else if (flags & SO_OBJECTS) x =atomic_long_read (&n->total_objects) - -count_partial (n, count_free); +count_partial (n, available); else x =atomic_long_read (&n->nr_slabs);
@@ -4694,7 +4689,7 @@ static int s_show(struct seq_file *m, vo nr_partials += n->nr_partial; nr_slabs +=atomic_long_read (&n->nr_slabs); nr_objs +=atomic_long_read (&n->total_objects); - nr_free +=count_partial (n, count_free); + nr_free +=count_partial (n, available); } nr_inuse = nr_objs - nr_free;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 15/16 SLUB: Remove MAX_OBJS limitation
In reply to Christoph LameterThere is no need anymore for the "inuse" field in the page struct. Extend the objects field to 32 bit allowing a practically unlimited number of objects.
--- include/linux/mm_types.h | 5 +---- mm/slub.c | 7 ------- 2 files changed, 1 insertion(+), 11 deletions(-)
include/linux/mm_types.h +1 -4
@@ -40,10 +40,7 @@ struct page {* to show when page is mapped * & limit reverse map searches. */ -struct {/* SLUB */ -u16 inuse; -u16 objects; - }; +u32 objects;/* SLUB */ }; union { struct {
mm/slub.c +0 -7
@@ -144,7 +144,6 @@ static inline int kmem_cache_debug(struc#define OO_SHIFT 16#define OO_MASK ((1 << OO_SHIFT) - 1) -#define MAX_OBJS_PER_PAGE 65535/* since page.objects is u16 */ /* Internal SLUB flags */ #define __OBJECT_POISON 0x80000000UL/* Poison object */
@@ -767,9 +766,6 @@ static int verify_slab(struct kmem_cache max_objects = ((void *)page->freelist - start) / s->size; } -if (max_objects >MAX_OBJS_PER_PAGE ) - max_objects =MAX_OBJS_PER_PAGE ; -if (page->objects != max_objects) {slab_err (s, page,"Wrong number of objects. Found %d but " "should be %d" , page->objects, max_objects);
@@ -1958,9 +1954,6 @@ static inline int slab_order(int size, iint rem;int min_order = slub_min_order; -if ((PAGE_SIZE << min_order) / size >MAX_OBJS_PER_PAGE ) -return get_order (size *MAX_OBJS_PER_PAGE ) -1 ; -for (order =max (min_order,fls (min_objects * size -1 ) -PAGE_SHIFT ); order <= max_order; order++) {
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 16/16 slub: Drop allocator announcement
In reply to Christoph LameterPeople get confused and some of the items listed no longer have the same relevance in the queued form of SLUB.
--- mm/slub.c | 7 ------- 1 file changed, 7 deletions(-)
mm/slub.c +0 -7
@@ -3124,13 +3124,6 @@ void __init kmem_cache_init(void)#ifdef CONFIG_SMP register_cpu_notifier (&slab_notifier);#endif -printk (KERN_INFO -"SLUB: Genslabs=%d, HWalign=%d, Order=%d-%d, MinObjects=%d," -" CPUs=%d, Nodes=%d\n" , - caches,cache_line_size (), - slub_min_order, slub_max_order, slub_min_objects, - nr_cpu_ids, nr_node_ids); - }void __init kmem_cache_init_late (void )
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/2 percpu: make @dyn_size always mean min dyn_size in first chunk init functions
In reply to Christoph LameterIn pcpu_alloc_info() and pcpu_embed_first_chunk(), @dyn_size was ssize_t, -1 meant auto-size, 0 forced 0 and positive meant minimum size. There's no use case for forcing 0 and the upcoming early alloc support always requires non-zero dynamic size. Make @dyn_size always mean minimum dyn_size.
include/linux/percpu.h +2 -2
@@ -105,7 +105,7 @@ extern struct pcpu_alloc_info * __init pextern void __init pcpu_free_alloc_info (struct pcpu_alloc_info *ai);extern struct pcpu_alloc_info *__init pcpu_build_alloc_info ( -size_t reserved_size,ssize_t dyn_size, +size_t reserved_size,size_t dyn_size,size_t atom_size,pcpu_fc_cpu_distance_fn_t cpu_distance_fn);
@@ -113,7 +113,7 @@ extern int __init pcpu_setup_first_chunk void *base_addr);#ifdef CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK -extern int __init pcpu_embed_first_chunk(size_t reserved_size, ssize_t dyn_size, +extern int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, size_t atom_size, pcpu_fc_cpu_distance_fn_t cpu_distance_fn, pcpu_fc_alloc_fn_t alloc_fn,
mm/percpu.c +9 -24
@@ -988,20 +988,6 @@ phys_addr_t per_cpu_ptr_to_phys(void *adreturn page_to_phys (pcpu_addr_to_page (addr)); } -static inline size_t pcpu_calc_fc_sizes (size_t static_size, -size_t reserved_size, -ssize_t *dyn_sizep) -{ -size_t size_sum; - - size_sum =PFN_ALIGN (static_size + reserved_size + - (*dyn_sizep >=0 ? *dyn_sizep :0 )); -if (*dyn_sizep !=0 ) - *dyn_sizep = size_sum - static_size - reserved_size; - -return size_sum; -} -/** * pcpu_alloc_alloc_info - allocate percpu allocation info * @nr_groups: the number of groups
@@ -1060,7 +1046,7 @@ void __init pcpu_free_alloc_info(struct/** * pcpu_build_alloc_info - build alloc_info considering distances between CPUs * @reserved_size: the size of reserved percpu area in bytes -* @dyn_size: free size for dynamic allocation in bytes, -1 for auto +* @dyn_size: free size for dynamic allocation in bytes * @atom_size: allocation atom size * @cpu_distance_fn: callback to determine distance between cpus, optional *
@@ -1079,7 +1065,7 @@ void __init pcpu_free_alloc_info(struct* failure, ERR_PTR value is returned. */ struct pcpu_alloc_info *__init pcpu_build_alloc_info ( -size_t reserved_size,ssize_t dyn_size, +size_t reserved_size,size_t dyn_size,size_t atom_size,pcpu_fc_cpu_distance_fn_t cpu_distance_fn) {
@@ -1098,13 +1084,15 @@ struct pcpu_alloc_info * __init pcpu_buimemset (group_map,0 ,sizeof (group_map));memset (group_cnt,0 ,sizeof (group_map)); + size_sum =PFN_ALIGN (static_size + reserved_size + dyn_size); + dyn_size = size_sum - static_size - reserved_size; +/* * Determine min_unit_size, alloc_size and max_upa such that * alloc_size is multiple of atom_size and is the smallest * which can accomodate 4k aligned segments which are equal to * or larger than min_unit_size. */ - size_sum =pcpu_calc_fc_sizes (static_size, reserved_size, &dyn_size); min_unit_size =max_t (size_t, size_sum,PCPU_MIN_UNIT_SIZE ); alloc_size =roundup (min_unit_size, atom_size);
@@ -1508,7 +1496,7 @@ early_param("percpu_alloc", percpu_alloc/** * pcpu_embed_first_chunk - embed the first percpu chunk into bootmem * @reserved_size: the size of reserved percpu area in bytes -* @dyn_size: free size for dynamic allocation in bytes, -1 for auto +* @dyn_size: minimum free size for dynamic allocation in bytes * @atom_size: allocation atom size * @cpu_distance_fn: callback to determine distance between cpus, optional * @alloc_fn: function to allocate percpu page
@@ -1529,10 +1517,7 @@ early_param("percpu_alloc", percpu_alloc* vmalloc space is not orders of magnitude larger than distances * between node memory addresses (ie. 32bit NUMA machines). * -* When @dyn_size is positive, dynamic area might be larger than -* specified to fill page alignment. When @dyn_size is auto, -* @dyn_size is just big enough to fill page alignment after static -* and reserved areas. +* @dyn_size specifies the minimum dynamic area size. * * If the needed size is smaller than the minimum or specified unit * size, the leftover is returned using @free_fn.
@@ -1540,7 +1525,7 @@ early_param("percpu_alloc", percpu_alloc* RETURNS: * 0 on success, -errno on failure. */ -int __init pcpu_embed_first_chunk(size_t reserved_size, ssize_t dyn_size, +int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, size_t atom_size, pcpu_fc_cpu_distance_fn_t cpu_distance_fn, pcpu_fc_alloc_fn_t alloc_fn,
@@ -1671,7 +1656,7 @@ int __init pcpu_page_first_chunk(size_tsnprintf (psize_str,sizeof (psize_str),"%luK" ,PAGE_SIZE >>10 ); - ai =pcpu_build_alloc_info (reserved_size,-1 ,PAGE_SIZE ,NULL ); + ai =pcpu_build_alloc_info (reserved_size,0 ,PAGE_SIZE ,NULL );if (IS_ERR (ai))return PTR_ERR (ai);BUG_ON (ai->nr_groups !=1 );
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerRe: [S+Q 02/16] [PATCH 1/2] percpu: make @dyn_size always mean min dyn_size in first chunk init functions
Quoted 1 line · Christoph Lameter
In
pcpu_alloc_info()
You mean pcpu_build_alloc_info()?
This should have a "From: Tejun Heo <[email protected]>" line, right?
Quoted 14 lines
and
pcpu_embed_first_chunk(), @dyn_size wasssize_t, -1 meant auto-size, 0 forced 0 and positive meant minimum size. There's no use case for forcing 0 and the upcoming early alloc support always requires non-zero dynamic size. Make @dyn_size always mean minimumdyn_size.Signed-off-by Tejun Heo <[email protected]>Signed-off-by Christoph Lameter <[email protected]>Index:
linux-2.6/include/linux/percpu.h
===================================================================
include/linux/percpu.h extern struct pcpu_alloc_info * __init p · line 111
Quoted 8 lines
include/linux/percpu.h +1
-1@@ -105,7 +105,7 @@ extern struct pcpu_alloc_info * __init pextern void __init pcpu_free_alloc_info (struct pcpu_alloc_info *ai);extern struct pcpu_alloc_info *__init pcpu_build_alloc_info ( -size_t reserved_size,ssize_t dyn_size, +size_t reserved_size,size_t dyn_size,size_t atom_size,pcpu_fc_cpu_distance_fn_t cpu_distance_fn);
This can just be removed entirely, it's unnecessarily global.
Quoted 35 lines
include/linux/percpu.h +1
-1@@ -113,7 +113,7 @@ extern int __init pcpu_setup_first_chunk void *base_addr);#ifdef CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK -extern int __init pcpu_embed_first_chunk(size_t reserved_size, ssize_t dyn_size, +extern int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, size_t atom_size, pcpu_fc_cpu_distance_fn_t cpu_distance_fn, pcpu_fc_alloc_fn_t alloc_fn,Index:
linux-2.6/mm/percpu.c
===================================================================mm/percpu.c +0
-14@@ -988,20 +988,6 @@ phys_addr_t per_cpu_ptr_to_phys(void *adreturn page_to_phys (pcpu_addr_to_page (addr)); } -static inline size_t pcpu_calc_fc_sizes (size_t static_size, -size_t reserved_size, -ssize_t *dyn_sizep) -{ -size_t size_sum; - - size_sum =PFN_ALIGN (static_size + reserved_size + - (*dyn_sizep >=0 ? *dyn_sizep :0 )); -if (*dyn_sizep !=0 ) - *dyn_sizep = size_sum - static_size - reserved_size; - -return size_sum; -} -/** * pcpu_alloc_alloc_info - allocate percpu allocation info * @nr_groups: the number of groups
mm/percpu.c void __init pcpu_free_alloc_info(struct · line 1049
Quoted 5 lines
mm/percpu.c +1
-1@@ -1060,7 +1046,7 @@ void __init pcpu_free_alloc_info(struct/** * pcpu_build_alloc_info - build alloc_info considering distances between CPUs * @reserved_size: the size of reserved percpu area in bytes -* @dyn_size: free size for dynamic allocation in bytes, -1 for auto +* @dyn_size: free size for dynamic allocation in bytes
It's the minimum free size, it's not necessarily the exact size due to round-up.
Quoted 13 lines
mm/percpu.c +1
-1* @atom_size: allocation atom size * @cpu_distance_fn: callback to determine distance between cpus, optional * @@ -1079,7 +1065,7 @@ void __init pcpu_free_alloc_info(struct* failure, ERR_PTR value is returned. */ struct pcpu_alloc_info *__init pcpu_build_alloc_info ( -size_t reserved_size,ssize_t dyn_size, +size_t reserved_size,size_t dyn_size,size_t atom_size,pcpu_fc_cpu_distance_fn_t cpu_distance_fn) {
mm/percpu.c struct pcpu_alloc_info * __init pcpu_bui · line 1088
Quoted 5 lines
mm/percpu.c +2
-0@@ -1098,13 +1084,15 @@ struct pcpu_alloc_info * __init pcpu_buimemset (group_map,0 ,sizeof (group_map));memset (group_cnt,0 ,sizeof (group_map)); + size_sum =PFN_ALIGN (static_size + reserved_size + dyn_size); + dyn_size = size_sum - static_size - reserved_size;
Ok, so the only purpose of "dyn_size" is to store in the struct pcpu_alloc_info later. Before this patch, ai->dyn_size would always be 0 if that's what was passed to pcpu_build_alloc_info(), but due to this arithmetic it now requires that static_size + reserved_size to be pfn aligned. Where is that enforced or do we not care?
Quoted 44 lines
mm/percpu.c +4
-7+/* * Determine min_unit_size, alloc_size and max_upa such that * alloc_size is multiple of atom_size and is the smallest * which can accomodate 4k aligned segments which are equal to * or larger than min_unit_size. */ - size_sum =pcpu_calc_fc_sizes (static_size, reserved_size, &dyn_size); min_unit_size =max_t (size_t, size_sum,PCPU_MIN_UNIT_SIZE ); alloc_size =roundup (min_unit_size, atom_size);@@ -1508,7 +1496,7 @@ early_param("percpu_alloc", percpu_alloc/** * pcpu_embed_first_chunk - embed the first percpu chunk into bootmem * @reserved_size: the size of reserved percpu area in bytes -* @dyn_size: free size for dynamic allocation in bytes, -1 for auto +* @dyn_size: minimum free size for dynamic allocation in bytes * @atom_size: allocation atom size * @cpu_distance_fn: callback to determine distance between cpus, optional * @alloc_fn: function to allocate percpu page @@ -1529,10 +1517,7 @@ early_param("percpu_alloc", percpu_alloc* vmalloc space is not orders of magnitude larger than distances * between node memory addresses (ie. 32bit NUMA machines). * -* When @dyn_size is positive, dynamic area might be larger than -* specified to fill page alignment. When @dyn_size is auto, -* @dyn_size is just big enough to fill page alignment after static -* and reserved areas. +* @dyn_size specifies the minimum dynamic area size. * * If the needed size is smaller than the minimum or specified unit * size, the leftover is returned using @free_fn. @@ -1540,7 +1525,7 @@ early_param("percpu_alloc", percpu_alloc* RETURNS: * 0 on success, -errno on failure. */ -int __init pcpu_embed_first_chunk(size_t reserved_size, ssize_t dyn_size, +int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, size_t atom_size, pcpu_fc_cpu_distance_fn_t cpu_distance_fn, pcpu_fc_alloc_fn_t alloc_fn,
mm/percpu.c int __init pcpu_page_first_chunk(size_t · line 1662
Quoted 8 lines
mm/percpu.c +1
-1@@ -1671,7 +1656,7 @@ int __init pcpu_page_first_chunk(size_tsnprintf (psize_str,sizeof (psize_str),"%luK" ,PAGE_SIZE >>10 ); - ai =pcpu_build_alloc_info (reserved_size,-1 ,PAGE_SIZE ,NULL ); + ai =pcpu_build_alloc_info (reserved_size,0 ,PAGE_SIZE ,NULL );if (IS_ERR (ai))return PTR_ERR (ai);BUG_ON (ai->nr_groups !=1 );
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Tejun Heo
maintainerRe: [S+Q 02/16] [PATCH 1/2] percpu: make @dyn_size always mean min dyn_size in first chunk init functions
Hello,
Quoted 5 lines · David Rientjes
Quoted 1 line · Christoph Lameter
In
pcpu_alloc_info()You mean
pcpu_build_alloc_info()?
Yeap.
Quoted 11 lines
include/linux/percpu.h extern struct pcpu_alloc_info * __init p · line 111
Quoted 9 lines
include/linux/percpu.h +1
-1@@ -105,7 +105,7 @@ extern struct pcpu_alloc_info * __init pextern void __init pcpu_free_alloc_info (struct pcpu_alloc_info *ai);extern struct pcpu_alloc_info *__init pcpu_build_alloc_info ( -size_t reserved_size,ssize_t dyn_size, +size_t reserved_size,size_t dyn_size,size_t atom_size,pcpu_fc_cpu_distance_fn_t cpu_distance_fn);This can just be removed entirely, it's unnecessarily global.
Oh yeah, it's not used outside mm/percpu.c anymore. I'll make it static.
Quoted 4 lines
Quoted 4 lines
/**
*pcpu_alloc_alloc_info- allocate percpu allocation info
* @nr_groups: the number of groups(quoted diff) +0
-0@@ -1060,7 +1046,7 @@ void __init pcpu_free_alloc_info(struct
Quoted 8 lines
mm/percpu.c void __init pcpu_free_alloc_info(struct · line 1049
Quoted 5 lines
mm/percpu.c +1
-1/** * pcpu_build_alloc_info - build alloc_info considering distances between CPUs * @reserved_size: the size of reserved percpu area in bytes -* @dyn_size: free size for dynamic allocation in bytes, -1 for auto +* @dyn_size: free size for dynamic allocation in bytes It's the minimum free size, it's not necessarily the exact size due to round-up.
Will update.
Quoted 10 lines
Quoted 10 lines
mm/percpu.c +1
-1struct pcpu_alloc_info * __init pcpu_build_alloc_info( - size_t reserved_size, ssize_t dyn_size, + size_t reserved_size, size_t dyn_size, size_t atom_size,mm/percpu.c +0
-0pcpu_fc_cpu_distance_fn_t cpu_distance_fn) {@@ -1098,13 +1084,15 @@ struct pcpu_alloc_info * __init pcpu_buimemset (group_map,0 ,sizeof (group_map));memset (group_cnt,0 ,sizeof (group_map));
Quoted 8 lines
mm/percpu.c struct pcpu_alloc_info * __init pcpu_bui · line 1088
Quoted 2 lines
mm/percpu.c +2
-0+ size_sum =PFN_ALIGN (static_size + reserved_size + dyn_size); + dyn_size = size_sum - static_size - reserved_size;Ok, so the only purpose of "
dyn_size" is to store in the structpcpu_alloc_infolater. Before this patch,ai->dyn_sizewould always be 0 if that's what was passed topcpu_build_alloc_info(), but due to this arithmetic it now requires thatstatic_size+reserved_sizeto be pfn aligned. Where is that enforced or do we not care?
I'm not really following you, but
* Nobody called pcpu_build_alloc_info() w/ zero dyn_size. It was either -1 or positive minimum size.
* None of static_size, reserved_size or dyn_size needs to be page aligned.
Thanks for the review.
Signature · 7 lines
tejun -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/2 percpu: make @dyn_size always mean min dyn_size in first chunk init functions
In pcpu_build_alloc_info() and pcpu_embed_first_chunk(), @dyn_size was ssize_t, -1 meant auto-size, 0 forced 0 and positive meant minimum size. There's no use case for forcing 0 and the upcoming early alloc support always requires non-zero dynamic size. Make @dyn_size always mean minimum dyn_size.
While at it, make pcpu_build_alloc_info() static which doesn't have any external caller as suggested by David Rientjes.
---
Here's the updated patch. I've pushed out this and the second patch to linux-next. Please feel free to pull from the following git tree.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu.git for-next
I'll apply 09 once how it's gonna be handled is determined.
Thanks.
include/linux/percpu.h | 7 +------ mm/percpu.c | 35 ++++++++++------------------------- 2 files changed, 11 insertions(+), 31 deletions(-)
include/linux/percpu.h +1 -6
@@ -104,16 +104,11 @@ extern struct pcpu_alloc_info * __init p int nr_units);extern void __init pcpu_free_alloc_info (struct pcpu_alloc_info *ai); -extern struct pcpu_alloc_info *__init pcpu_build_alloc_info ( -size_t reserved_size,ssize_t dyn_size, -size_t atom_size, -pcpu_fc_cpu_distance_fn_t cpu_distance_fn); -extern int __init pcpu_setup_first_chunk (const struct pcpu_alloc_info *ai,void *base_addr);#ifdef CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK -extern int __init pcpu_embed_first_chunk(size_t reserved_size, ssize_t dyn_size, +extern int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, size_t atom_size, pcpu_fc_cpu_distance_fn_t cpu_distance_fn, pcpu_fc_alloc_fn_t alloc_fn,
mm/percpu.c +10 -25
@@ -1013,20 +1013,6 @@ phys_addr_t per_cpu_ptr_to_phys(void *adreturn page_to_phys (pcpu_addr_to_page (addr)); } -static inline size_t pcpu_calc_fc_sizes (size_t static_size, -size_t reserved_size, -ssize_t *dyn_sizep) -{ -size_t size_sum; - - size_sum =PFN_ALIGN (static_size + reserved_size + - (*dyn_sizep >=0 ? *dyn_sizep :0 )); -if (*dyn_sizep !=0 ) - *dyn_sizep = size_sum - static_size - reserved_size; - -return size_sum; -} -/** * pcpu_alloc_alloc_info - allocate percpu allocation info * @nr_groups: the number of groups
@@ -1085,7 +1071,7 @@ void __init pcpu_free_alloc_info(struct/** * pcpu_build_alloc_info - build alloc_info considering distances between CPUs * @reserved_size: the size of reserved percpu area in bytes -* @dyn_size: free size for dynamic allocation in bytes, -1 for auto +* @dyn_size: minimum free size for dynamic allocation in bytes * @atom_size: allocation atom size * @cpu_distance_fn: callback to determine distance between cpus, optional *
@@ -1103,8 +1089,8 @@ void __init pcpu_free_alloc_info(struct* On success, pointer to the new allocation_info is returned. On * failure, ERR_PTR value is returned. */ -struct pcpu_alloc_info *__init pcpu_build_alloc_info ( -size_t reserved_size,ssize_t dyn_size, +static struct pcpu_alloc_info *__init pcpu_build_alloc_info ( +size_t reserved_size,size_t dyn_size,size_t atom_size,pcpu_fc_cpu_distance_fn_t cpu_distance_fn) {
@@ -1123,13 +1109,15 @@ struct pcpu_alloc_info * __init pcpu_buimemset (group_map,0 ,sizeof (group_map));memset (group_cnt,0 ,sizeof (group_cnt)); + size_sum =PFN_ALIGN (static_size + reserved_size + dyn_size); + dyn_size = size_sum - static_size - reserved_size; +/* * Determine min_unit_size, alloc_size and max_upa such that * alloc_size is multiple of atom_size and is the smallest * which can accomodate 4k aligned segments which are equal to * or larger than min_unit_size. */ - size_sum =pcpu_calc_fc_sizes (static_size, reserved_size, &dyn_size); min_unit_size =max_t (size_t, size_sum,PCPU_MIN_UNIT_SIZE ); alloc_size =roundup (min_unit_size, atom_size);
@@ -1532,7 +1520,7 @@ early_param("percpu_alloc", percpu_alloc/** * pcpu_embed_first_chunk - embed the first percpu chunk into bootmem * @reserved_size: the size of reserved percpu area in bytes -* @dyn_size: free size for dynamic allocation in bytes, -1 for auto +* @dyn_size: minimum free size for dynamic allocation in bytes * @atom_size: allocation atom size * @cpu_distance_fn: callback to determine distance between cpus, optional * @alloc_fn: function to allocate percpu page
@@ -1553,10 +1541,7 @@ early_param("percpu_alloc", percpu_alloc* vmalloc space is not orders of magnitude larger than distances * between node memory addresses (ie. 32bit NUMA machines). * -* When @dyn_size is positive, dynamic area might be larger than -* specified to fill page alignment. When @dyn_size is auto, -* @dyn_size is just big enough to fill page alignment after static -* and reserved areas. +* @dyn_size specifies the minimum dynamic area size. * * If the needed size is smaller than the minimum or specified unit * size, the leftover is returned using @free_fn.
@@ -1564,7 +1549,7 @@ early_param("percpu_alloc", percpu_alloc* RETURNS: * 0 on success, -errno on failure. */ -int __init pcpu_embed_first_chunk(size_t reserved_size, ssize_t dyn_size, +int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size, size_t atom_size, pcpu_fc_cpu_distance_fn_t cpu_distance_fn, pcpu_fc_alloc_fn_t alloc_fn,
@@ -1695,7 +1680,7 @@ int __init pcpu_page_first_chunk(size_tsnprintf (psize_str,sizeof (psize_str),"%luK" ,PAGE_SIZE >>10 ); - ai =pcpu_build_alloc_info (reserved_size,-1 ,PAGE_SIZE ,NULL ); + ai =pcpu_build_alloc_info (reserved_size,0 ,PAGE_SIZE ,NULL );if (IS_ERR (ai))return PTR_ERR (ai);BUG_ON (ai->nr_groups !=1 );
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 02/16] [PATCH 1/2 UPDATED] percpu: make @dyn_size always mean min dyn_size in first chunk init functions
Quoted 11 lines · Tejun Heo
In
pcpu_build_alloc_info()andpcpu_embed_first_chunk(), @dyn_size wasssize_t, -1 meant auto-size, 0 forced 0 and positive meant minimum size. There's no use case for forcing 0 and the upcoming early alloc support always requires non-zero dynamic size. Make @dyn_size always mean minimumdyn_size.While at it, make
pcpu_build_alloc_info()static which doesn't have any external caller as suggested by David Rientjes.Signed-off-by Tejun Heo <[email protected]>Cc David Rientjes <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerRe: [S+Q 02/16] [PATCH 1/2] percpu: make @dyn_size always mean min dyn_size in first chunk init functions
In reply to Tejun HeoQuoted 19 lines · Tejun Heo
Quoted 10 lines
Quoted 10 lines
struct
pcpu_alloc_info*__initpcpu_build_alloc_info(mm/percpu.c +1
-1- size_t reserved_size, ssize_t dyn_size, + size_t reserved_size, size_t dyn_size, size_t atom_size, pcpu_fc_cpu_distance_fn_t cpu_distance_fn) {@@ -1098,13 +1084,15 @@ struct pcpu_alloc_info * __init pcpu_buimemset (group_map,0 ,sizeof (group_map));memset (group_cnt,0 ,sizeof (group_map));Quoted 8 lines
mm/percpu.c struct pcpu_alloc_info * __init pcpu_bui · line 1088
Quoted 2 lines
mm/percpu.c +2
-0+ size_sum =PFN_ALIGN (static_size + reserved_size + dyn_size); + dyn_size = size_sum - static_size - reserved_size;Ok, so the only purpose of "
dyn_size" is to store in the structpcpu_alloc_infolater. Before this patch,ai->dyn_sizewould always be 0 if that's what was passed topcpu_build_alloc_info(), but due to this arithmetic it now requires thatstatic_size+reserved_sizeto be pfn aligned. Where is that enforced or do we not care?
Quoted 5 lines
I'm not really following you, but
* Nobody called
pcpu_build_alloc_info()w/ zerodyn_size. It was either -1 or positive minimum size.
Ok, the commit description said that passing pcpu_build_alloc_info() a dyn_size of 0 would force it to be 0, although the arithmetic introduced by this patch would not have necessarily set ai->dyn_size to be 0 when passed if static_size + reserved_size was not page aligned (size_sum could be greater than static_size + reserved_size). Since there are no users passing a dyn_size of 0, my concern is addressed.
Thanks!
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [S+Q 02/16] [PATCH 1/2] percpu: make @dyn_size always mean min dyn_size in first chunk init functions
In reply to David RientjesHmmm... Ok seems to have been manged by quilt send.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/2 percpu: allow limited allocation before slab is online
In reply to Christoph LameterThis patch updates percpu allocator such that it can serve limited amount of allocation before slab comes online. This is primarily to allow slab to depend on working percpu allocator.
Two parameters, PERCPU_DYNAMIC_EARLY_SIZE and SLOTS, determine how much memory space and allocation map slots are reserved. If this reserved area is exhausted, WARN_ON_ONCE() will trigger and allocation will fail till slab comes online.
The following changes are made to implement early alloc.
* pcpu_mem_alloc() now checks slab_is_available()
* Chunks are allocated using pcpu_mem_alloc()
* Init paths make sure ai->dyn_size is at least as large as PERCPU_DYNAMIC_EARLY_SIZE.
* Initial alloc maps are allocated in __initdata and copied to kmalloc'd areas once slab is online.
--- include/linux/percpu.h | 13 ++++++++++++ init/main.c | 1 include/linux/percpu.h | 13 ++++++++++++ init/main.c | 1 mm/percpu.c | 52 +++++++++++++++++++++++++++++++++++++------------ 3 files changed, 54 insertions(+), 12 deletions(-)
mm/percpu.c +40 -12
@@ -282,6 +282,9 @@ static void __maybe_unused pcpu_next_pop*/ static void *pcpu_mem_alloc (size_t size) { +if (WARN_ON_ONCE (!slab_is_available ())) +return NULL ; +if (size <=PAGE_SIZE )return kzalloc (size,GFP_KERNEL );else {
@@ -392,13 +395,6 @@ static int pcpu_extend_area_map(struct p old_size = chunk->map_alloc *sizeof (chunk->map[0 ]);memcpy (new, chunk->map, old_size); -/* -* map_alloc < PCPU_DFL_MAP_ALLOC indicates that the chunk is -* one of the first chunks and still using static map. -*/ -if (chunk->map_alloc >=PCPU_DFL_MAP_ALLOC ) - old = chunk->map; - chunk->map_alloc = new_alloc; chunk->map = new; new =NULL ;
@@ -604,7 +600,7 @@ static struct pcpu_chunk *pcpu_alloc_chu {struct pcpu_chunk *chunk; - chunk =kzalloc (pcpu_chunk_struct_size,GFP_KERNEL ); + chunk =pcpu_mem_alloc (pcpu_chunk_struct_size);if (!chunk)return NULL ;
@@ -1084,7 +1080,9 @@ struct pcpu_alloc_info * __init pcpu_buimemset (group_map,0 ,sizeof (group_map));memset (group_cnt,0 ,sizeof (group_map)); - size_sum =PFN_ALIGN (static_size + reserved_size + dyn_size); +/* calculate size_sum and ensure dyn_size is enough for early alloc */ + size_sum =PFN_ALIGN (static_size + reserved_size + +max_t (size_t, dyn_size,PERCPU_DYNAMIC_EARLY_SIZE )); dyn_size = size_sum - static_size - reserved_size;/*
@@ -1314,7 +1312,8 @@ int __init pcpu_setup_first_chunk(const void *base_addr) {static char cpus_buf[4096 ]__initdata ; -static int smap[2 ], dmap[2 ]; +static int smap[PERCPU_DYNAMIC_EARLY_SLOTS ]__initdata ; +static int dmap[PERCPU_DYNAMIC_EARLY_SLOTS ]__initdata ;size_t dyn_size = ai->dyn_size;size_t size_sum = ai->static_size + ai->reserved_size + dyn_size;struct pcpu_chunk *schunk, *dchunk =NULL ;
@@ -1337,14 +1336,13 @@ int __init pcpu_setup_first_chunk(const }while (0 )/* sanity checks */ -BUILD_BUG_ON (ARRAY_SIZE (smap) >=PCPU_DFL_MAP_ALLOC || -ARRAY_SIZE (dmap) >=PCPU_DFL_MAP_ALLOC );PCPU_SETUP_BUG_ON (ai->nr_groups <=0 );PCPU_SETUP_BUG_ON (!ai->static_size);PCPU_SETUP_BUG_ON (!base_addr);PCPU_SETUP_BUG_ON (ai->unit_size < size_sum);PCPU_SETUP_BUG_ON (ai->unit_size & ~PAGE_MASK );PCPU_SETUP_BUG_ON (ai->unit_size <PCPU_MIN_UNIT_SIZE ); +PCPU_SETUP_BUG_ON (ai->dyn_size <PERCPU_DYNAMIC_EARLY_SIZE );PCPU_SETUP_BUG_ON (pcpu_verify_alloc_info (ai) <0 );/* process group information and build config tables accordingly */
@@ -1782,3 +1780,33 @@ void __init setup_per_cpu_areas(void) __per_cpu_offset[cpu] = delta + pcpu_unit_offsets[cpu]; }#endif /* CONFIG_HAVE_SETUP_PER_CPU_AREA */ + +/* +* First and reserved chunks are initialized with temporary allocation +* map in initdata so that they can be used before slab is online. +* This function is called after slab is brought up and replaces those +* with properly allocated maps. +*/ +void __init percpu_init_late (void ) +{ +struct pcpu_chunk *target_chunks[] = + { pcpu_first_chunk, pcpu_reserved_chunk,NULL }; +struct pcpu_chunk *chunk; +unsigned long flags; +int i; + +for (i =0 ; (chunk = target_chunks[i]); i++) { +int *map; +const size_t size =PERCPU_DYNAMIC_EARLY_SLOTS *sizeof (map[0 ]); + +BUILD_BUG_ON (size >PAGE_SIZE ); + + map =pcpu_mem_alloc (size); +BUG_ON (!map); + +spin_lock_irqsave (&pcpu_lock, flags); +memcpy (map, chunk->map, size); + chunk->map = map; +spin_unlock_irqrestore (&pcpu_lock, flags); + } +}
init/main.c +1 -0
@@ -522,6 +522,7 @@ static void __init mm_init(void)page_cgroup_init_flatmem ();mem_init ();kmem_cache_init (); +percpu_init_late ();pgtable_cache_init ();vmalloc_init (); }
include/linux/percpu.h +13 -0
@@ -45,6 +45,16 @@#define PCPU_MIN_UNIT_SIZE PFN_ALIGN(64 << 10)/* +* Percpu allocator can serve percpu allocations before slab is +* initialized which allows slab to depend on the percpu allocator. +* The following two parameters decide how much resource to +* preallocate for this. Keep PERCPU_DYNAMIC_RESERVE equal to or +* larger than PERCPU_DYNAMIC_EARLY_SIZE. +*/ +#define PERCPU_DYNAMIC_EARLY_SLOTS 128 +#define PERCPU_DYNAMIC_EARLY_SIZE (12 << 10) + +/* * PERCPU_DYNAMIC_RESERVE indicates the amount of free area to piggy * back on the first chunk for dynamic percpu allocation if arch is * manually allocating and mapping it for faster access (as a part of
@@ -140,6 +150,7 @@ extern bool is_kernel_percpu_address(uns#ifndef CONFIG_HAVE_SETUP_PER_CPU_AREA extern void __init setup_per_cpu_areas (void );#endif +extern void __init percpu_init_late (void );#else /* CONFIG_SMP */
@@ -153,6 +164,8 @@ static inline bool is_kernel_percpu_addrstatic inline void __init setup_per_cpu_areas (void ) { } +static inline void __init percpu_init_late (void ) { } +static inline void *pcpu_lpage_remapped (void *kaddr) {return NULL ;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Christoph Lameter
The following patchset cleans some pieces up and then equips SLUB with per cpu queues that work similar to SLABs queues. With that approach SLUB wins in hackbench:
Hackbench I don't think is that interesting. SLQB was beating SLAB too.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · Nick Piggin
Quoted 3 lines · Christoph Lameter
The following patchset cleans some pieces up and then equips SLUB with per cpu queues that work similar to SLABs queues. With that approach SLUB wins in hackbench:
Hackbench I don't think is that interesting. SLQB was beating SLAB too.
We've seen regressions pop up with hackbench so I think it's interesting. Not the most interesting one, for sure, nor conclusive.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 6 lines · Pekka Enberg
Quoted 2 lines
Hackbench I don't think is that interesting. SLQB was beating SLAB too.
We've seen regressions pop up with hackbench so I think it's interesting. Not the most interesting one, for sure, nor conclusive.
Hackbench was frequently cited in performance tests. Which benchmarks would be of interest? I am off this week so dont expect a fast response from me.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · Pekka Enberg
Quoted 5 lines
Quoted 2 lines
Hackbench I don't think is that interesting. SLQB was beating SLAB too.
We've seen regressions pop up with hackbench so I think it's interesting. Not the most interesting one, for sure, nor conclusive.
Quoted 3 lines · Christoph Lameter
Hackbench was frequently cited in performance tests. Which benchmarks would be of interest? I am off this week so dont expect a fast response from me.
I guess "netperf TCP_RR" is the most interesting one because that's a known benchmark where SLUB performs poorly when compared to SLAB. Mel's extensive slab benchmarks are also worth looking at:
http://lkml.indiana.edu/hypermail/linux/kernel/0902.0/00745.html
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerQuoted 10 lines · Pekka Enberg
Quoted 3 lines
Hackbench was frequently cited in performance tests. Which benchmarks would be of interest? I am off this week so dont expect a fast response from me.
I guess "netperf
TCP_RR" is the most interesting one because that's a known benchmark where SLUB performs poorly when compared to SLAB. Mel's extensive slab benchmarks are also worth looking at:http://lkml.indiana.edu/hypermail/linux/kernel/0902.0/00745.html
In addition to that benchmark, which regresses on systems with larger numbers of cpus, you had posted results for slub vs slab for kernbench, aim9, and sysbench before slub was ever merged. If you're going to use slab-like queueing in slub, it would be interesting to see if these particular benchmarks regress once again.
Quoted 5 lines · David Rientjes
In addition to that benchmark, which regresses on systems with larger numbers of cpus, you had posted results for slub vs slab for kernbench, aim9, and sysbench before slub was ever merged. If you're going to use slab-like queueing in slub, it would be interesting to see if these particular benchmarks regress once again.
I do not have access to Itanium systems anymore. I hope Mike can run some benchmarks?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · Christoph Lameter
Quoted 5 lines · David Rientjes
In addition to that benchmark, which regresses on systems with larger numbers of cpus, you had posted results for slub vs slab for kernbench, aim9, and sysbench before slub was ever merged. If you're going to use slab-like queueing in slub, it would be interesting to see if these particular benchmarks regress once again.
I do not have access to Itanium systems anymore. I hope Mike can run some benchmarks?
Sure, but I won't have a lot of time as we're pushing out the first customer UV systems and that's keeping me pretty busy.
If it's all packaged up and ready to run that would help a lot.
Thanks,
Mike
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Pekka Enberg
I guess "netperf
TCP_RR" is the most interesting one because that's a known benchmark where SLUB performs poorly when compared to SLAB. Mel's extensive slab benchmarks are also worth looking at:
I will look at it when I get time but I am vacation right now and sitting in the hospital with my son who managed to get himself there on the first day of the "vacation". Guess it will take a week or so at least.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · Pekka Enberg
Quoted 7 lines · Nick Piggin
Quoted 3 lines · Christoph Lameter
The following patchset cleans some pieces up and then equips SLUB with per cpu queues that work similar to SLABs queues. With that approach SLUB wins in hackbench:
Hackbench I don't think is that interesting. SLQB was beating SLAB too.
We've seen regressions pop up with hackbench so I think it's interesting. Not the most interesting one, for sure, nor conclusive.
Looks like most of the stuff up to 12 is a good idea.
Christoph, is there any test where this is likely to lose substantial ground to SLUB without queueing? Can we characterize that? We're in danger now of getting into the situation where we can't drop SLUB for the same reasons we can't drop SLAB - big performance regressions.
Signature · 8 lines
Mathematics is the supreme nostalgia of our time. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>