be2iscsi: Fixes for powerpc compile
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · Jayamohan Kalickal
Quoted 2 lines
+
max_size=ALIGN(sizeof(struct be_cmd_bhs), 64)*sess->cmds_max;
+ vaddr = pci_alloc_consistent(phba->pcidev,Do you just want a dma/pci pool? It will align structs for you too, I think.
I didn't want to do it because we do not really alloc/free once a connection is established.
Yes, it can be done that way but certainly would not wan to do that in the IO hot path (even if it picking from pool).
I was just going by some other drivers like lpfc where they create the pool at pci probe time, then also preallocate from the pool at some setup/init time. In the case of lpfc it allocates scsi bufs for each command in its slave_alloc when new devices are added.
It is not a big deal. If people are ok with it as is, I am fine.
Implementing pci_pool is not a problem. But, my experience with this chip (or for that matter any chip that can do huge IOPS ,say, more then 600 -700 KIOPS),
every line of code on the hot path would have an impact and hence I strive to
move it out of the hot path.
The code is not yet optimized for speed yet but I definitely want to push whatever
I can to init time from the IO path.
Thanks
Jay
_____
From: Mike Christie [mailto:[email protected]]
To: Jayamohan Kalickal [mailto:[email protected]]
Cc: [email protected], [email protected], [email protected]
Sent: Wed, 09 Sep 2009 08:38:47 -0700
Subject: Re: [PATCH 1/1] be2iscsi: Fixes for powerpc compile
Quoted 9 lines · Jayamohan Kalickal
Quoted 2 lines
+
max_size=ALIGN(sizeof(struct be_cmd_bhs), 64)*sess->cmds_max;
+ vaddr = pci_alloc_consistent(phba->pcidev,Do you just want a dma/pci pool? It will align structs for you too, I think.
I didn't want to do it because we do not really alloc/free once a connection is established.
Yes, it can be done that way but certainly would not wan to do that in the IO hot path (even if it picking from pool).
I was just going by some other drivers like lpfc where they create the pool at pci probe time, then also preallocate from the pool at some setup/init time. In the case of lpfc it allocates scsi bufs for each command in its slave_alloc when new devices are added.
It is not a big deal. If people are ok with it as is, I am fine. --
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
___________________________________________________________________________________
This message, together with any attachment(s), contains confidential and proprietary information of ServerEngines Corporation and is intended only for the designated recipient(s) named above. Any unauthorized review, printing, retention, copying, disclosure or distribution is strictly prohibited. If you are not the intended recipient of this message, please immediately advise the sender by reply email message and delete all copies of this message and any attachment(s). Thank you.
Quoted 7 lines · Jayamohan Kalickal
Implementing
pci_poolis not a problem. But, my experience with this chip (or for that matter any chip that can do huge IOPS ,say, more then 600 -700 KIOPS),
every line of code on the hot path would have an impact and hence I strive to
move it out of the hot path.The code is not yet optimized for speed yet but I definitely want to push whatever
I can to init time from the IO path.
I'll fold in the patch as is because we urgently need a fix for the virt_to_bus() problem.
pci_pool was designed to allow for good IOP throughput (that's why lpfc is using it). The pci_alloc_consistent() you've chosen often only allocates at the page level, so it can be hugely wasteful of resources; plus coherent memory is a precious resource on some platforms, so I can see you running out on them. Could you at least try a pci_pool implementation ... if it really turns out to be a performance problem we can keep the current implementation despite the potential resource issues.
Thanks,
James
Ok. Reading Mike's comment once again and looking at the lpfc code ,I think, maybe, this is what Mike had in mind, but I could have misunderstood.
During session create, I will create the pool as well as alloc. This will take care of the allocation at page level and scarce coherent resources problems mentioned by JamesB. Also, this would not have any impact on performance. I see this done for "lpfc_mbuf_pool" in lpfc/lpfc_mem.c.
Thanks
Jay
_____
From: James Bottomley [mailto:[email protected]]
To: Jayamohan Kalickal [mailto:[email protected]]
Cc: Mike Christie [mailto:[email protected]], [email protected], [email protected]
Sent: Thu, 10 Sep 2009 10:15:40 -0700
Subject: Re: [PATCH 1/1] be2iscsi: Fixes for powerpc compile
Quoted 7 lines · Jayamohan Kalickal
Implementing
pci_poolis not a problem. But, my experience with this chip (or for that matter any chip that can do huge IOPS ,say, more then 600 -700 KIOPS),
every line of code on the hot path would have an impact and hence I strive to
move it out of the hot path.The code is not yet optimized for speed yet but I definitely want to push whatever
I can to init time from the IO path.
I'll fold in the patch as is because we urgently need a fix for the virt_to_bus() problem.
pci_pool was designed to allow for good IOP throughput (that's why lpfc is using it). The pci_alloc_consistent() you've chosen often only allocates at the page level, so it can be hugely wasteful of resources; plus coherent memory is a precious resource on some platforms, so I can see you running out on them. Could you at least try a pci_pool implementation ... if it really turns out to be a performance problem we can keep the current implementation despite the potential resource issues.
Thanks,
James
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
___________________________________________________________________________________
This message, together with any attachment(s), contains confidential and proprietary information of ServerEngines Corporation and is intended only for the designated recipient(s) named above. Any unauthorized review, printing, retention, copying, disclosure or distribution is strictly prohibited. If you are not the intended recipient of this message, please immediately advise the sender by reply email message and delete all copies of this message and any attachment(s). Thank you.
Quoted 7 lines · Jayamohan Kalickal
Ok. Reading Mike's comment once again and looking at the lpfc code ,I think, maybe, this is what Mike had in mind, but I could have misunderstood.
During session create, I will create the pool as well as alloc. This will take care of the allocation at page level and scarce coherent resources problems mentioned by JamesB. Also, this would not have any impact on performance. I see this done for "
lpfc_mbuf_pool" inlpfc/lpfc_mem.c.Thanks
Jay
As most things - it depends on what's being allocated. I recommended against a pci pool when I was looking at the allocations for queue memory, SGLs, etc as that is largely static (allocate/assign once, rarely dealloc/deassign) and very much likes to be 4k pages, page aligned, and as contiguous as possible.
However, the DMA buffers for the Command and Response buffers is a different story. They are in the fast path, re-alloc per command, are smaller than a page, etc and fit the pci pool profile very well.
-- james s
Thanks for your comments. It looks like we are in agreement.
I have implemented with pci_pool (using the approach of "lpfc_mbuf_pool"
in lpfc/lpfc_mem.c).
That should take care of the two issues raised earlier. I will send the patch shortly
Thanks
Jay
_____
From: James Smart [mailto:[email protected]]
To: Jayamohan Kalickal [mailto:[email protected]]
Cc: James Bottomley [mailto:[email protected]], Mike Christie [mailto:[email protected]], [email protected] [mailto:[email protected]], [email protected] [mailto:[email protected]]
Sent: Fri, 11 Sep 2009 05:15:42 -0700
Subject: Re: [PATCH 1/1] be2iscsi: Fixes for powerpc compile
Quoted 9 lines · Jayamohan Kalickal
Ok. Reading Mike's comment once again and looking at the lpfc code ,I think, maybe, this is what Mike had in mind, but I could have misunderstood.
During session create, I will create the pool as well as alloc. This will take care of the allocation at page level and scarce coherent resources problems mentioned by JamesB. Also, this would not have any impact on performance. I see this done for "
lpfc_mbuf_pool" inlpfc/lpfc_mem.c.Thanks
Jay
As most things - it depends on what's being allocated. I recommended against a pci pool when I was looking at the allocations for queue memory, SGLs, etc as that is largely static (allocate/assign once, rarely dealloc/deassign) and very much likes to be 4k pages, page aligned, and as contiguous as possible.
However, the DMA buffers for the Command and Response buffers is a different story. They are in the fast path, re-alloc per command, are smaller than a page, etc and fit the pci pool profile very well.
-- james s
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
___________________________________________________________________________________
This message, together with any attachment(s), contains confidential and proprietary information of ServerEngines Corporation and is intended only for the designated recipient(s) named above. Any unauthorized review, printing, retention, copying, disclosure or distribution is strictly prohibited. If you are not the intended recipient of this message, please immediately advise the sender by reply email message and delete all copies of this message and any attachment(s). Thank you.