IB/ehca: Replace vmalloc with kmalloc
PATCH 1/3 IB/ehca: Replace vmalloc with kmalloc
From: Anton Blanchard <antonb at au1.ibm.com>
To improve performance of driver ressource allocation, replace the vmalloc() call with kmalloc().
--- drivers/infiniband/hw/ehca/ipz_pt_fn.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
drivers/infiniband/hw/ehca/ipz_pt_fn.c +3 -3
@@ -220,7 +220,7 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue, queue->small_page =NULL ;/* allocate queue page pointers */ - queue->queue_pages =vmalloc (nr_of_pages *sizeof (void *)); + queue->queue_pages =kmalloc (nr_of_pages *sizeof (void *),GFP_KERNEL );if (!queue->queue_pages) {ehca_gen_err ("Couldn't allocate queue page list" );return 0 ;
@@ -240,7 +240,7 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue, ipz_queue_ctor_exit0:ehca_gen_err ("Couldn't alloc pages queue=%p " "nr_of_pages=%x" , queue, nr_of_pages); -vfree (queue->queue_pages); +kfree (queue->queue_pages);return 0 ; }
@@ -262,7 +262,7 @@ int ipz_queue_dtor(struct ehca_pd *pd, struct ipz_queue *queue)free_page ((unsigned long )queue->queue_pages[i]); } -vfree (queue->queue_pages); +kfree (queue->queue_pages);return 1 ; }
Signature · 1 line
1.5.5
drivers/infiniband/hw/ehca/ipz_pt_fn.c int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue, · line 223
Quoted 1 line
drivers/infiniband/hw/ehca/ipz_pt_fn.c +1
-0+ queue->queue_pages =kmalloc (nr_of_pages *sizeof (void *),GFP_KERNEL );
How big might this buffer be? Any chance of allocation failure due to memory fragmentation?
- R.
Hi Roland,
thanks for the quick review. I was hoping you could apply these changes for 2.6.30 because this will be the codebase for the next OFED release. The patch is well tested in HPC environment and we haven't seen any issues.
Regarding Antons patch you are right. If a user allocates an unrealistically large queue pair it could happen that kmalloc() is not able to allocate the memory. In this case we will return ENOMEM to the user so the kernel will not be affected at all. We plan to add vmalloc() call in case kmalloc() fails for the next kernel release.
Mit freundlichen Grüßen / Kind regards
Stefan Roscher
eHCA/eHEA Linux Driver Development
IBM Systems &Technology Group, Systems Software Development / FW I/O Firmware Entwicklung 2
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland
Schoenaicher Str. 220
71032 Boeblingen
Phone: +49-7031-16-2015
E-Mail: [email protected]
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland Research & Development GmbH / Vorsitzender des
Aufsichtsrats: Martin Jetter
Geschäftsführung: Herbert Kircher
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
From:
Roland Dreier <[email protected]>
To:
Stefan Roscher <[email protected]>
Cc:
"LinuxPPC-Dev" <[email protected]>, LKML <[email protected]>, "OF-EWG" <[email protected]>, Roland Dreier <[email protected]>, Joachim Fenkes/Germany/IBM@IBMDE, Christoph Raisch/Germany/IBM@IBMDE, Alexander Schmidt1/Germany/IBM@IBMDE, Stefan Roscher/Germany/IBM@IBMDE, Hoang-Nam Nguyen/Germany/IBM@IBMDE Date:
21.04.2009 19:34
Subject:
Re: [PATCH 1/3] IB/ehca: Replace vmalloc with kmalloc
Quoted 1 line
+
queue->queue_pages= kmalloc(nr_of_pages * sizeof(void
*), GFP_KERNEL);
How big might this buffer be? Any chance of allocation failure due to memory fragmentation?
- R.
In case of large queue pairs there is the possibillity of allocation failures due to memory fragmentationo with kmalloc().To ensure the memory is allocated even if kmalloc() can not find chunks which are big enough, we try to allocate the memory with vmalloc().
---
Quoted 6 lines · Roland Dreier
Review
Quoted 1 line
(quoted diff) +1
-0+ queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);How big might this buffer be? Any chance of allocation failure due to memory fragmentation?
- R.
Hey Roland,
yes you are right and here is the patch to circumvent the described problem. It will apply on top of the patchset.
regards Stefan
drivers/infiniband/hw/ehca/ipz_pt_fn.c | 17 +++++++++++++---- 1 files changed, 13 insertions(+), 4 deletions(-)
drivers/infiniband/hw/ehca/ipz_pt_fn.c +13 -4
@@ -222,8 +222,11 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue,/* allocate queue page pointers */ queue->queue_pages =kmalloc (nr_of_pages *sizeof (void *),GFP_KERNEL );if (!queue->queue_pages) { -ehca_gen_err ("Couldn't allocate queue page list" ); -return 0 ; + queue->queue_pages =vmalloc (nr_of_pages *sizeof (void *)); +if (!queue->queue_pages) { +ehca_gen_err ("Couldn't allocate queue page list" ); +return 0 ; + } }memset (queue->queue_pages,0 , nr_of_pages *sizeof (void *));
@@ -240,7 +243,10 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue, ipz_queue_ctor_exit0:ehca_gen_err ("Couldn't alloc pages queue=%p " "nr_of_pages=%x" , queue, nr_of_pages); -kfree (queue->queue_pages); +if (is_vmalloc_addr (queue->queue_pages)) +vfree (queue->queue_pages); +else +kfree (queue->queue_pages);return 0 ; }
@@ -262,7 +268,10 @@ int ipz_queue_dtor(struct ehca_pd *pd, struct ipz_queue *queue)free_page ((unsigned long )queue->queue_pages[i]); } -kfree (queue->queue_pages); +if (is_vmalloc_addr (queue->queue_pages)) +vfree (queue->queue_pages); +else +kfree (queue->queue_pages);return 1 ; }
Signature · 1 line
1.5.5
Quoted 6 lines · michael
Hi,
I don't take the point, if it is not import use the vmalloc. Why you try with a kmalloc
alloc first? and why do not use kzalloc?
Because kmalloc() is faster than vmalloc() causing a huge performance win when someone allocates a large number of queue pairs. We fall back to vmalloc() only if kmalloc() can't deliver the memory chunk. We don't need kzalloc because we fill the list right after the alloc.
regards Stefan
Hi,
Quoted 11 lines · Stefan Roscher
In case of large queue pairs there is the possibillity of allocation failures due to memory fragmentationo with kmalloc().To ensure the memory is allocated even if
kmalloc()can not find chunks which are big enough, we try to allocate the memory
withvmalloc().Signed-off-by Stefan Roscher <[email protected]>---
Quoted 2 lines · Roland Dreier
Quoted 1 line
(quoted diff) +1
-0+ queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);
Quoted 10 lines
Quoted 5 lines
How big might this buffer be? Any chance of allocation failure due to memory fragmentation?
- R.
Hey Roland, yes you are right and here is the patch to circumvent the described problem. It will apply on top of the patchset.
regards Stefan
I don't take the point, if it is not import use the vmalloc. Why you try with a kmalloc alloc first? and why do not use kzalloc?
Quoted 37 lines
drivers/infiniband/hw/ehca/ipz_pt_fn.c | 17 +++++++++++++---- 1 files changed, 13 insertions(+), 4 deletions(-)drivers/infiniband/hw/ehca/ipz_pt_fn.c +9
-3@@ -222,8 +222,11 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue,/* allocate queue page pointers */ queue->queue_pages =kmalloc (nr_of_pages *sizeof (void *),GFP_KERNEL );if (!queue->queue_pages) { -ehca_gen_err ("Couldn't allocate queue page list" ); -return 0 ; + queue->queue_pages =vmalloc (nr_of_pages *sizeof (void *)); +if (!queue->queue_pages) { +ehca_gen_err ("Couldn't allocate queue page list" ); +return 0 ; + } }memset (queue->queue_pages,0 , nr_of_pages *sizeof (void *));@@ -240,7 +243,10 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue, ipz_queue_ctor_exit0:ehca_gen_err ("Couldn't alloc pages queue=%p " "nr_of_pages=%x" , queue, nr_of_pages); -kfree (queue->queue_pages); +if (is_vmalloc_addr (queue->queue_pages)) +vfree (queue->queue_pages); +else +kfree (queue->queue_pages);return 0 ; }@@ -262,7 +268,10 @@ int ipz_queue_dtor(struct ehca_pd *pd, struct ipz_queue *queue)free_page ((unsigned long )queue->queue_pages[i]);
drivers/infiniband/hw/ehca/ipz_pt_fn.c int ipz_queue_dtor(struct ehca_pd *pd, struct ipz_queue *queue) · line 277
Quoted 10 lines
drivers/infiniband/hw/ehca/ipz_pt_fn.c +4
-1} -kfree (queue->queue_pages); +if (is_vmalloc_addr (queue->queue_pages)) +vfree (queue->queue_pages); +else +kfree (queue->queue_pages);return 1 ; }
Regards Michael
Hi,
Quoted 7 lines · Stefan Roscher
Quoted 3 lines · michael
Hi,
Quoted 8 lines
Quoted 3 lines
I don't take the point, if it is not import use the vmalloc. Why you try with a kmalloc alloc first? and why do not use kzalloc?
Because
kmalloc()is faster thanvmalloc()causing a huge performance win when someone allocates a large number of queue pairs. We fall back tovmalloc()only ifkmalloc()can't deliver the memory chunk.
Sorry I catch later the performace issue.
Quoted 4 lines
We don't need kzalloc because we fill the list right after the alloc.
regards Stefan
Regards Michael
Quoted 6 lines
_______________________________________________
Linuxppc-dev mailing list
[email protected]
https://ozlabs.org/mailman/listinfo/linuxppc-dev
Hi Roland,
did you have a chance to take a look at the patchset and will you apply it, or are there any outstanding issues we need to address?
Regards,
Alex
Quoted 67 lines · On Wed, 22 Apr 2009 16:02:28 +0200 Stefan Roscher…
In case of large queue pairs there is the possibillity of allocation failures due to memory fragmentationo with kmalloc().To ensure the memory is allocated even if
kmalloc()can not find chunks which are big enough, we try to allocate the memory withvmalloc().Signed-off-by Stefan Roscher <[email protected]>---
Quoted 6 lines · Roland Dreier
Review
Quoted 1 line
(quoted diff) +1
-0+ queue->queue_pages = kmalloc(nr_of_pages * sizeof(void *), GFP_KERNEL);How big might this buffer be? Any chance of allocation failure due to memory fragmentation?
- R.
Hey Roland,
yes you are right and here is the patch to circumvent the described problem. It will apply on top of the patchset.
regards Stefandrivers/infiniband/hw/ehca/ipz_pt_fn.c | 17 +++++++++++++---- 1 files changed, 13 insertions(+), 4 deletions(-)drivers/infiniband/hw/ehca/ipz_pt_fn.c +13
-4@@ -222,8 +222,11 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue,/* allocate queue page pointers */ queue->queue_pages =kmalloc (nr_of_pages *sizeof (void *),GFP_KERNEL );if (!queue->queue_pages) { -ehca_gen_err ("Couldn't allocate queue page list" ); -return 0 ; + queue->queue_pages =vmalloc (nr_of_pages *sizeof (void *)); +if (!queue->queue_pages) { +ehca_gen_err ("Couldn't allocate queue page list" ); +return 0 ; + } }memset (queue->queue_pages,0 , nr_of_pages *sizeof (void *));@@ -240,7 +243,10 @@ int ipz_queue_ctor(struct ehca_pd *pd, struct ipz_queue *queue, ipz_queue_ctor_exit0:ehca_gen_err ("Couldn't alloc pages queue=%p " "nr_of_pages=%x" , queue, nr_of_pages); -kfree (queue->queue_pages); +if (is_vmalloc_addr (queue->queue_pages)) +vfree (queue->queue_pages); +else +kfree (queue->queue_pages);return 0 ; }@@ -262,7 +268,10 @@ int ipz_queue_dtor(struct ehca_pd *pd, struct ipz_queue *queue)free_page ((unsigned long )queue->queue_pages[i]); } -kfree (queue->queue_pages); +if (is_vmalloc_addr (queue->queue_pages)) +vfree (queue->queue_pages); +else +kfree (queue->queue_pages);return 1 ; }
Quoted 2 lines
did you have a chance to take a look at the patchset and will you apply it, or are there any outstanding issues we need to address?
I guess it's OK, but definitely 2.6.31 material. I guess I'll stick it linux-next soon.
- R.
Quoted 7 lines · On Tue, 28 Apr 2009 07:01:32 -0700 Roland Dreier <[email protected]> wrote:
Quoted 2 lines
did you have a chance to take a look at the patchset and will you apply it, or are there any outstanding issues we need to address?
I guess it's OK, but definitely 2.6.31 material. I guess I'll stick it linux-next soon.
- R.
Okay with us, thank you very much!
Alex
Quoted 4 lines · Stefan Roscher
From: Anton Blanchard <antonb at au1.ibm.com>
To improve performance of driver ressource allocation, replace the
vmalloc()call withkmalloc().
Just curious, but how big are these allocations? Why was vmalloc() even ever used if we know they'll be small?
-- Dave
Quoted 12 lines · Dave Hansen
Quoted 4 lines · Stefan Roscher
From: Anton Blanchard <antonb at au1.ibm.com>
To improve performance of driver ressource allocation, replace the
vmalloc()call withkmalloc().Just curious, but how big are these allocations? Why was
vmalloc()even ever used if we know they'll be small?-- Dave
The theoretical maximum size can be 512k, but for common queue pairs less than 128k is used.Because of the theoretical maximum we implemented vmalloc() first, but recognized a huge performance impact.
-- Stefan
Roland Dreier
Appliedthanks, applied.
Quoted 2 lines
From: Anton Blanchard <antonb at au1.ibm.com>
Signed-off-by: Stefan Roscher <stefan.roscher at de.ibm.com>
please use '@' signs so these are real email addresses.
- R.