Skip to content

Fixing DMA mask issues in habanalabs driver

Oded Gabbay lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest8 patches 4 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/8habanalabs: initialize device CPU queues after MMU initdrivers/misc/habanalabs/asid.c drivers/misc/habanalabs/device.c drivers/misc/habanalabs/goya/goya.c+40 -480 replies
2/8habanalabs: de-couple MMU and VM module initializationdrivers/misc/habanalabs/device.c drivers/misc/habanalabs/memory.c drivers/misc/habanalabs/mmu.c+22 -200 replies
3/8habanalabs: initialize MMU context for driverdrivers/misc/habanalabs/context.c drivers/misc/habanalabs/mmu.c+13 -40 replies
4/8habanalabs: add MMU mappings for Goya CPUdrivers/misc/habanalabs/debugfs.c drivers/misc/habanalabs/goya/goya.c drivers/misc/habanalabs/goya/goyaP.h +1 more+137 -140 replies
5/8habanalabs: set Goya CPU to use ASIC MMUdrivers/misc/habanalabs/goya/goya.c+20 -30 replies
6/8habanalabs: remove DMA mask hack for Goyadrivers/misc/habanalabs/goya/goya.c+4 -150 replies
7/8habanalabs: add WARN in case of bad MMU mappingdrivers/misc/habanalabs/mmu.c+4 -00 replies
8/8habanalabs: enable 64-bit DMA mask in POWER9drivers/misc/habanalabs/goya/goya.c drivers/misc/habanalabs/habanalabs.h drivers/misc/habanalabs/habanalabs_drv.c +1 more+21 -24 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 0/8] Fixing DMA mask issues in habanalabs driver

This patch-set changes the way the Goya internal CPU access memory on the Host machine. This is needed to prevent the non-standard way the driver used the PCI DMA set mask kernel API so far.

The DMA set mask should be called only once at the start of the driver. This is because changing the DMA mask to a new value after allocations were made using a previous mask value, might cause the previous allocations to become unaccessible (usually if there is IOMMU present).

The driver did that because of a limitation in Goya's internal CPU. The limitation was that the internal CPU can only access 40-bit addresses, while the entire ASIC can access 50-bit addresses. Therefore, the driver set the DMA mask to 39-bits, allocated memory for the internal CPU on the host and then changed the DMA mask to 48-bits.

This patch-set eliminates the double DMA set by using Goya's MMU to overcome the limitation. The driver now sets the DMA mask only once to 48-bits and allocates a single DMA region of 2MB for the internal CPU. It then maps that region in Goya's MMU to a device virtual address under 40-bits.

In addition, this patch-set enables the use of 64-bit mask on POWER9 systems. POWER9 DMA mask can be set ONLY to 32-bit or 64-bit. To use 64-bit, the device must set bit 59 to 1 in all its outbound transactions. This is achieved by setting a special configuration in Goya's PCIe controller. The configuration must be done only in POWER9 machines, as it will make the device non-functional on other architectures (e.g. x86-64, ARM).

Thanks,
Oded

Oded Gabbay (8):
habanalabs: initialize device CPU queues after MMU init
habanalabs: de-couple MMU and VM module initialization
habanalabs: initialize MMU context for driver
habanalabs: add MMU mappings for Goya CPU
habanalabs: set Goya CPU to use ASIC MMU
habanalabs: remove DMA mask hack for Goya
habanalabs: add WARN in case of bad MMU mapping
habanalabs: enable 64-bit DMA mask in POWER9

 drivers/misc/habanalabs/asid.c           |   2 +-
 drivers/misc/habanalabs/context.c        |   7 +
 drivers/misc/habanalabs/debugfs.c        |   7 +-
 drivers/misc/habanalabs/device.c         |  45 +++--
 drivers/misc/habanalabs/goya/goya.c      | 234 +++++++++++++++++------
 drivers/misc/habanalabs/goya/goyaP.h     |  12 +-
 drivers/misc/habanalabs/habanalabs.h     |   9 +-
 drivers/misc/habanalabs/habanalabs_drv.c |   7 +
 drivers/misc/habanalabs/memory.c         |  13 +-
 drivers/misc/habanalabs/mmu.c            |  20 +-
 drivers/misc/habanalabs/pci.c            |   7 +-
 11 files changed, 259 insertions(+), 104 deletions(-)
Signature · 1 line
2.17.1