Add p2p via dmabuf to habanalabs
Subsystems (3)
v5 cover Add p2p via dmabuf to habanalabs
Hi,
This is v5 of this patch-set following again a long email thread.
It contains fixes to the implementation according to the review that Jason did on v4. Jason, I appreciate your feedback. If you can take another look to see I didn't miss anything that would be great.
The details of the fixes are in the changelog in the commit message of the second patch.
There was one issue with your proposal to set the orig_nents to 0. I did that, but I also had to restore it to nents before calling sg_free_table because that function uses orig_nents to iterate.
Thanks,
Oded
Oded Gabbay (1):
habanalabs: define uAPI to export FD for DMA-BUF
Tomer Tayar (1):
habanalabs: add support for dma-buf exporter
drivers/misc/habanalabs/Kconfig | 1 + drivers/misc/habanalabs/common/habanalabs.h | 22 + drivers/misc/habanalabs/common/memory.c | 522 +++++++++++++++++++- drivers/misc/habanalabs/gaudi/gaudi.c | 1 + drivers/misc/habanalabs/goya/goya.c | 1 + include/uapi/misc/habanalabs.h | 28 +- 6 files changed, 570 insertions(+), 5 deletions(-)
Signature · 1 line
2.25.1
PATCH v5 1/2 habanalabs: define uAPI to export FD for DMA-BUF
User process might want to share the device memory with another driver/device, and to allow it to access it over PCIe (P2P).
To enable this, we utilize the dma-buf mechanism and add a dma-buf exporter support, so the other driver can import the device memory and access it.
The device memory is allocated using our existing allocation uAPI, where the user will get a handle that represents the allocation.
The user will then need to call the new
uAPI (HL_MEM_OP_EXPORT_DMABUF_FD) and give the handle as a parameter.
The driver will return a FD that represents the DMA-BUF object that was created to match that allocation.
--- include/uapi/misc/habanalabs.h | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-)
include/uapi/misc/habanalabs.h +27 -1
@@ -808,6 +808,10 @@ union hl_wait_cs_args {#define HL_MEM_OP_UNMAP 3/* Opcode to map a hw block */ #define HL_MEM_OP_MAP_BLOCK 4 +/* Opcode to create DMA-BUF object for an existing device memory allocation +* and to export an FD of that DMA-BUF back to the caller +*/ +#define HL_MEM_OP_EXPORT_DMABUF_FD 5/* Memory flags */ #define HL_MEM_CONTIGUOUS 0x1
@@ -879,11 +883,26 @@ struct hl_mem_in {/* Virtual address returned from HL_MEM_OP_MAP */ __u64 device_virt_addr; } unmap; + +/* HL_MEM_OP_EXPORT_DMABUF_FD */ +struct { +/* Handle returned from HL_MEM_OP_ALLOC. In Gaudi, +* where we don't have MMU for the device memory, the +* driver expects a physical address (instead of +* a handle) in the device memory space. +*/ +__u64 handle; +/* Size of memory allocation. Relevant only for GAUDI */ +__u64 mem_size; + } export_dmabuf_fd; };/* HL_MEM_OP_* */ __u32 op; -/* HL_MEM_* flags */ +/* HL_MEM_* flags. +* For the HL_MEM_OP_EXPORT_DMABUF_FD opcode, this field holds the +* DMA-BUF file/FD flags. +*/ __u32 flags;/* Context ID - Currently not in use */ __u32 ctx_id;
@@ -920,6 +939,13 @@ struct hl_mem_out {__u32 pad; }; + +/* Returned in HL_MEM_OP_EXPORT_DMABUF_FD. Represents the +* DMA-BUF object that was created to describe a memory +* allocation on the device's memory space. The FD should be +* passed to the importer driver +*/ +__u64 fd; }; };
Signature · 1 line
2.25.1
PATCH v5 2/2 habanalabs: add support for dma-buf exporter
In reply to Oded GabbayFrom: Tomer Tayar <[email protected]>
Implement the calls to the dma-buf kernel api to create a dma-buf object backed by FD.
We block the option to mmap the DMA-BUF object because we don't support DIRECT_IO and implicit P2P. We only implement support for explicit P2P through importing the FD of the DMA-BUF.
In the export phase, we provide to the DMA-BUF object an array of pages that represent the device's memory area. During the map callback, we convert the array of pages into an SGT. We split/merge the pages according to the dma max segment size of the importer.
To get the DMA address of the PCI bar, we use the dma_map_resources() kernel API, because our device memory is not backed by page struct and this API doesn't need page struct to map the physical address to a DMA address.
We set the orig_nents member of the SGT to be 0, to indicate to other drivers that we don't support CPU mappings.
Note that in Habanalabs's ASICs, the device memory is pinned and immutable. Therefore, there is no need for dynamic mappings and pinning callbacks.
Also note that in GAUDI we don't have an MMU towards the device memory and the user works on physical addresses. Therefore, the user doesn't pass through the kernel driver to allocate memory there. As a result, only for GAUDI we receive from the user a device memory physical address (instead of a handle) and a size.
We check the p2p distance using pci_p2pdma_distance_many() and refusing to map dmabuf in case the distance doesn't allow p2p.
---
Changes in v5:
- Use dma_get_max_seg_size() to get the max segment size of the importer. Use that value when we convert the pages array into an SGT, to make sure each SG is not larger than that size.
- Set orig_nents to be 0 after initializing the SGT. This will hopefully tell other drivers to avoid using CPU mapping with this SGT.
- Remove the allocation of pages array in case we receive an address from the user (GAUDI use-case). This was redundant.
drivers/misc/habanalabs/Kconfig | 1 + drivers/misc/habanalabs/common/habanalabs.h | 22 + drivers/misc/habanalabs/common/memory.c | 522 +++++++++++++++++++- drivers/misc/habanalabs/gaudi/gaudi.c | 1 + drivers/misc/habanalabs/goya/goya.c | 1 + 5 files changed, 543 insertions(+), 4 deletions(-)
drivers/misc/habanalabs/Kconfig +1 -0
@@ -8,6 +8,7 @@ config HABANA_AI depends on PCI && HAS_IOMEM select GENERIC_ALLOCATOR select HWMON + select DMA_SHARED_BUFFER help Enables PCIe card driver for Habana's AI Processors (AIP) that are designed to accelerate Deep Learning inference and training workloads.
drivers/misc/habanalabs/common/habanalabs.h +22 -0
@@ -26,6 +26,7 @@#include <linux/sched/signal.h> #include <linux/io-64-nonatomic-lo-hi.h> #include <linux/coresight.h> +#include <linux/dma-buf.h> #define HL_NAME "habanalabs"
@@ -1329,6 +1330,23 @@ struct hl_pending_cb {u32 hw_queue_id; }; +/** +* struct hl_dmabuf_wrapper - a dma-buf wrapper object. +* @dmabuf: pointer to dma-buf object. +* @ctx: pointer to the dma-buf owner's context. +* @phys_pg_pack: pointer to physical page pack if the dma-buf was exported for +* memory allocation handle. +* @device_address: physical address of the device's memory. Relevant only +* if phys_pg_pack is NULL (dma-buf was exported from address). +* The total size can be taken from the dmabuf object. +*/ +struct hl_dmabuf_wrapper { +struct dma_buf *dmabuf; +struct hl_ctx *ctx; +struct hl_vm_phys_pg_pack *phys_pg_pack; +uint64_t device_address; +}; +/** * struct hl_ctx - user/kernel context. * @mem_hash: holds mapping from virtual address to virtual memory area
@@ -1637,6 +1655,7 @@ struct hl_vm_hw_block_list_node {* @npages: num physical pages in the pack. * @total_size: total size of all the pages in this list. * @mapping_cnt: number of shared mappings. +* @exporting_cnt: number of dma-buf exporting. * @asid: the context related to this list. * @page_size: size of each page in the pack. * @flags: HL_MEM_* flags related to this list.
@@ -1651,6 +1670,7 @@ struct hl_vm_phys_pg_pack {u64 npages;u64 total_size;atomic_t mapping_cnt; +u32 exporting_cnt;u32 asid;u32 page_size;u32 flags;
@@ -2314,6 +2334,7 @@ struct hl_mmu_funcs {* the error will be ignored by the driver during * device initialization. Mainly used to debug and * workaround firmware bugs +* @dram_pci_bar_start: start bus address of PCIe bar towards DRAM. * @last_successful_open_jif: timestamp (jiffies) of the last successful * device open. * @last_open_session_duration_jif: duration (jiffies) of the last device open
@@ -2451,6 +2472,7 @@ struct hl_device {u64 max_power;u64 clock_gating_mask;u64 boot_error_status_mask; +u64 dram_pci_bar_start;u64 last_successful_open_jif;u64 last_open_session_duration_jif;u64 open_counter;
drivers/misc/habanalabs/common/memory.c +518 -4
574 lines not on this page · raw message
drivers/misc/habanalabs/gaudi/gaudi.c +1 -0
@@ -773,6 +773,7 @@ static int gaudi_early_init(struct hl_device *hdev) } prop->dram_pci_bar_size =pci_resource_len (pdev,HBM_BAR_ID ); + hdev->dram_pci_bar_start =pci_resource_start (pdev,HBM_BAR_ID );/* If FW security is enabled at this point it means no access to ELBI */ if (hdev->asic_prop.fw_security_enabled) {
drivers/misc/habanalabs/goya/goya.c +1 -0
@@ -620,6 +620,7 @@ static int goya_early_init(struct hl_device *hdev) } prop->dram_pci_bar_size =pci_resource_len (pdev,DDR_BAR_ID ); + hdev->dram_pci_bar_start =pci_resource_start (pdev,DDR_BAR_ID );/* If FW security is enabled at this point it means no access to ELBI */ if (hdev->asic_prop.fw_security_enabled) {
Signature · 1 line
2.25.1
Quoted 7 lines · Oded Gabbay
Hi,
This is v5 of this patch-set following again a long email thread.It contains fixes to the implementation according to the review that Jason did on v4. Jason, I appreciate your feedback. If you can take another look to see I didn't miss anything that would be great.
Quoted 6 lines
The details of the fixes are in the changelog in the commit message of the second patch.
There was one issue with your proposal to set the
orig_nentsto 0. I did that, but I also had to restore it to nents before callingsg_free_tablebecause that function usesorig_nentsto iterate.