Skip to contentSkip to conversation

Apple M1 DART IOMMU driver

PATCH Superseded Sven Peter 4 participants 17 messages Mar 28, 2021 – Apr 19, 2021

Subsystems (4)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Sven Peter

author
#0 lore raw

v2 cover Apple M1 DART IOMMU driver

Hi,

Here's v2 of my Apple M1 DART IOMMU driver series as a follow up to the original version [1].

Short summary: this series adds support for the iommu found in Apple's new M1 SoC which is required to use DMA on most peripherals. So far this code has been tested with dwc3 in host and device mode on a M1 Mac Mini on top of the latest version of Hector's bringup series [2,3] together with my m1n1 bootloader branch to bring up USB [4]. It will also apply (but not be very useful) on top of iommu/next and v5.12-rc3.

Thanks everyone for the suggestions and discussions so far. I believe they have already significantly improved the state of this driver and our understanding of the DART iommu!

The part I'm most unsure about is the way I keep track of the multiple iommu nodes attached to a device. I would love to especially get some feedback there.

Changes for v2:
- fixed devicetree binding linting issues pointed out by Rob Herring and reworked that file.
- made DART-specific code in io-pgtable.c unconditional and removed flag from Kconfig as proposed by Robin Murphy.
- allowed multiple DART nodes in the "iommus" property as proposed by Rob Herring and Robin Murphy. this resulted in significant changes to apple-iommu-dart.c.
- the domain aperture is now forced to 32bit if translation is enabled after the original suggestion to limit the aperture by Mark Kettenis and the follow-up discussion and investigation with Mark Kettenis, Arnd Bergmann, Robin Murphy and Rob Herring. This change also simplified the code in io-pgtable.c and made some of the improvements suggested during review not apply anymore.
- added support for bypassed and isolated domain modes.
- reject IOMMU_MMIO and IOMMU_NOEXEC since it's unknown how to set these up for now or if the hardware even supports these flags.
- renamed some registers to be less confusing (mainly s/DOMAIN/STREAM/ to prevent confusion with linux's iommu domain concept).

I have also fixed my email provider so this time the series should actually be a single thread and not contain any HTML by accident anymore...

Best,

Sven

[1] https://lore.kernel.org/linux-iommu/[email protected]/
[2] https://lore.kernel.org/linux-arch/[email protected]/
[3] https://github.com/AsahiLinux/linux/tree/upstream-bringup-v4
[4] https://github.com/svenpeter42/m1n1/tree/usb-dwc3-serial-wip

Sven Peter (3):
iommu: io-pgtable: add DART pagetable format
dt-bindings: iommu: add DART iommu bindings
iommu: dart: Add DART iommu driver

 .../devicetree/bindings/iommu/apple,dart.yaml |  81 ++
 MAINTAINERS                                   |   7 +
 drivers/iommu/Kconfig                         |  14 +
 drivers/iommu/Makefile                        |   1 +
 drivers/iommu/apple-dart-iommu.c              | 858 ++++++++++++++++++
 drivers/iommu/io-pgtable-arm.c                |  59 ++
 drivers/iommu/io-pgtable.c                    |   1 +
 include/linux/io-pgtable.h                    |   6 +
 8 files changed, 1027 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/iommu/apple,dart.yaml
 create mode 100644 drivers/iommu/apple-dart-iommu.c
Signature · 6 lines
2.25.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
author
#1 lore raw

PATCH v2 1/3 iommu: io-pgtable: add DART pagetable format

Apple's DART iommu uses a pagetable format that shares some similarities with the ones already implemented by io-pgtable.c. Add a new format variant to support the required differences so that we don't have to duplicate the pagetable handling code.

---
 drivers/iommu/io-pgtable-arm.c | 59 ++++++++++++++++++++++++++++++++++
 drivers/iommu/io-pgtable.c     |  1 +
 include/linux/io-pgtable.h     |  6 ++++
 3 files changed, 66 insertions(+)
drivers/iommu/io-pgtable-arm.c +59 -0
index 87def58e79b5..2f63443fd115 100644
@@ -127,6 +127,9 @@
 #define ARM_MALI_LPAE_MEMATTR_IMP_DEF	0x88ULL
 #define ARM_MALI_LPAE_MEMATTR_WRITE_ALLOC 0x8DULL
 
+#define APPLE_DART_PTE_PROT_NO_WRITE (1<<7)
+#define APPLE_DART_PTE_PROT_NO_READ (1<<8)
+
 /* IOPTE accessors */
 #define iopte_deref(pte,d) __va(iopte_to_paddr(pte, d))
 
@@ -381,6 +384,15 @@ static arm_lpae_iopte arm_lpae_prot_to_pte(struct arm_lpae_io_pgtable *data,
 {
 	arm_lpae_iopte pte;
 
+	if (data->iop.fmt == ARM_APPLE_DART) {
+		pte = 0;
+		if (!(prot & IOMMU_WRITE))
+			pte |= APPLE_DART_PTE_PROT_NO_WRITE;
+		if (!(prot & IOMMU_READ))
+			pte |= APPLE_DART_PTE_PROT_NO_READ;
+		return pte;
+	}
+
 	if (data->iop.fmt == ARM_64_LPAE_S1 ||
 	    data->iop.fmt == ARM_32_LPAE_S1) {
 		pte = ARM_LPAE_PTE_nG;
@@ -1043,6 +1055,48 @@ arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie)
 	return NULL;
 }
 
+static struct io_pgtable *
+apple_dart_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie)
+{
+	struct arm_lpae_io_pgtable *data;
+
+	if (cfg->ias > 36)
+		return NULL;
+	if (cfg->oas > 36)
+		return NULL;
+
+	if (!cfg->coherent_walk)
+		return NULL;
+
+	cfg->pgsize_bitmap &= SZ_16K;
+	if (!cfg->pgsize_bitmap)
+		return NULL;
+
+	if (cfg->quirks)
+		return NULL;
+
+	data = arm_lpae_alloc_pgtable(cfg);
+	if (!data)
+		return NULL;
+
+	data->start_level = 2;
+	data->pgd_bits = 11;
+	data->bits_per_level = 11;
+
+	data->pgd = __arm_lpae_alloc_pages(ARM_LPAE_PGD_SIZE(data), GFP_KERNEL,
+					   cfg);
+	if (!data->pgd)
+		goto out_free_data;
+
+	cfg->apple_dart_cfg.ttbr = virt_to_phys(data->pgd);
+
+	return &data->iop;
+
+out_free_data:
+	kfree(data);
+	return NULL;
+}
+
 struct io_pgtable_init_fns io_pgtable_arm_64_lpae_s1_init_fns = {
 	.alloc	= arm_64_lpae_alloc_pgtable_s1,
 	.free	= arm_lpae_free_pgtable,
@@ -1068,6 +1122,11 @@ struct io_pgtable_init_fns io_pgtable_arm_mali_lpae_init_fns = {
 	.free	= arm_lpae_free_pgtable,
 };
 
+struct io_pgtable_init_fns io_pgtable_apple_dart_init_fns = {
+	.alloc	= apple_dart_alloc_pgtable,
+	.free	= arm_lpae_free_pgtable,
+};
+
 #ifdef CONFIG_IOMMU_IO_PGTABLE_LPAE_SELFTEST
 
 static struct io_pgtable_cfg *cfg_cookie __initdata;
drivers/iommu/io-pgtable.c +1 -0
index 6e9917ce980f..6ec75f3e9c3b 100644
@@ -27,6 +27,7 @@ io_pgtable_init_table[IO_PGTABLE_NUM_FMTS] = {
 #ifdef CONFIG_AMD_IOMMU
 	[AMD_IOMMU_V1] = &io_pgtable_amd_iommu_v1_init_fns,
 #endif
+	[ARM_APPLE_DART] = &io_pgtable_apple_dart_init_fns,
 };
 
 struct io_pgtable_ops *alloc_io_pgtable_ops(enum io_pgtable_fmt fmt,
include/linux/io-pgtable.h +6 -0
index a4c9ca2c31f1..b06925eb20d3 100644
@@ -16,6 +16,7 @@ enum io_pgtable_fmt {
 	ARM_V7S,
 	ARM_MALI_LPAE,
 	AMD_IOMMU_V1,
+	ARM_APPLE_DART,
 	IO_PGTABLE_NUM_FMTS,
 };
 
@@ -136,6 +137,10 @@ struct io_pgtable_cfg {
 			u64	transtab;
 			u64	memattr;
 		} arm_mali_lpae_cfg;
+
+		struct {
+			u64 ttbr;
+		} apple_dart_cfg;
 	};
 };
 
@@ -250,5 +255,6 @@ extern struct io_pgtable_init_fns io_pgtable_arm_64_lpae_s2_init_fns;
 extern struct io_pgtable_init_fns io_pgtable_arm_v7s_init_fns;
 extern struct io_pgtable_init_fns io_pgtable_arm_mali_lpae_init_fns;
 extern struct io_pgtable_init_fns io_pgtable_amd_iommu_v1_init_fns;
+extern struct io_pgtable_init_fns io_pgtable_apple_dart_init_fns;
 
 #endif /* __IO_PGTABLE_H */
Signature · 6 lines
2.25.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#2 lore raw

Re: [PATCH v2 1/3] iommu: io-pgtable: add DART pagetable format

Thank you for the patch! Yet something to improve:

Hi Sven,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on arm-perf/for-next/perf]
[also build test ERROR on linus/master v5.12-rc4 next-20210326] [cannot apply to iommu/next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Sven-Peter/Apple-M1-DART-IOMMU-driver/20210328-154437
base:   https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git for-next/perf
config: nios2-randconfig-r016-20210328 (attached as .config)
compiler: nios2-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/58778f8d813f5d98aedff621f0b236f60fe1dd2b
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Sven-Peter/Apple-M1-DART-IOMMU-driver/20210328-154437 git checkout 58778f8d813f5d98aedff621f0b236f60fe1dd2b
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=nios2

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>):

>> nios2-linux-ld: drivers/iommu/io-pgtable.o:(.rodata+0x1c): undefined reference to `io_pgtable_apple_dart_init_fns'

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

author
#3 lore raw

Re: [PATCH v2 1/3] iommu: io-pgtable: add DART pagetable format

Hi,

Quoted 6 lines · kernel test robot

All errors (new ones prefixed by >>):

Quoted 1 line
Quoted 1 line

nios2-linux-ld: drivers/iommu/io-pgtable.o:(.rodata+0x1c): undefined reference to `io_pgtable_apple_dart_init_fns'

That one was me not being careful enough when removing the Kconfig flag compared to v1.

Fixed by moving

[ARM_APPLE_DART] = &io_pgtable_apple_dart_init_fns,

into the #ifdef CONFIG_IOMMU_IO_PGTABLE_LPAE region in drivers/iommu/io-pgtable.c a few lines above.

Thanks,

Sven

#4 lore raw

Re: [PATCH v2 1/3] iommu: io-pgtable: add DART pagetable format

In reply to Sven Peter
Thank you for the patch! Yet something to improve:

Hi Sven,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on arm-perf/for-next/perf]
[also build test ERROR on linus/master v5.12-rc4 next-20210326] [cannot apply to iommu/next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Sven-Peter/Apple-M1-DART-IOMMU-driver/20210328-154437
base:   https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git for-next/perf
config: m68k-allmodconfig (attached as .config)
compiler: m68k-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/58778f8d813f5d98aedff621f0b236f60fe1dd2b
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Sven-Peter/Apple-M1-DART-IOMMU-driver/20210328-154437 git checkout 58778f8d813f5d98aedff621f0b236f60fe1dd2b
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=m68k

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>):

>> m68k-linux-ld: drivers/iommu/io-pgtable.o:(.rodata+0x1c): undefined reference to `io_pgtable_apple_dart_init_fns'

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

maintainer
#5 lore raw

Re: [PATCH v2 1/3] iommu: io-pgtable: add DART pagetable format

In reply to Sven Peter
Quoted 48 lines · Sven Peter

Apple's DART iommu uses a pagetable format that shares some similarities with the ones already implemented by io-pgtable.c. Add a new format variant to support the required differences so that we don't have to duplicate the pagetable handling code.

---
 drivers/iommu/io-pgtable-arm.c | 59 ++++++++++++++++++++++++++++++++++
 drivers/iommu/io-pgtable.c     |  1 +
 include/linux/io-pgtable.h     |  6 ++++
 3 files changed, 66 insertions(+)
drivers/iommu/io-pgtable-arm.c +14 -0
index 87def58e79b5..2f63443fd115 100644
@@ -127,6 +127,9 @@
 #define ARM_MALI_LPAE_MEMATTR_IMP_DEF	0x88ULL
 #define ARM_MALI_LPAE_MEMATTR_WRITE_ALLOC 0x8DULL
 
+#define APPLE_DART_PTE_PROT_NO_WRITE (1<<7)
+#define APPLE_DART_PTE_PROT_NO_READ (1<<8)
+
 /* IOPTE accessors */
 #define iopte_deref(pte,d) __va(iopte_to_paddr(pte, d))
 
@@ -381,6 +384,15 @@ static arm_lpae_iopte arm_lpae_prot_to_pte(struct arm_lpae_io_pgtable *data,
 {
 	arm_lpae_iopte pte;
 
+	if (data->iop.fmt == ARM_APPLE_DART) {
+		pte = 0;
+		if (!(prot & IOMMU_WRITE))
+			pte |= APPLE_DART_PTE_PROT_NO_WRITE;
+		if (!(prot & IOMMU_READ))
+			pte |= APPLE_DART_PTE_PROT_NO_READ;
+		return pte;
+	}
+
 	if (data->iop.fmt == ARM_64_LPAE_S1 ||
 	    data->iop.fmt == ARM_32_LPAE_S1) {
 		pte = ARM_LPAE_PTE_nG;
@@ -1043,6 +1055,48 @@ arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie)
 	return NULL;
 }
 
+static struct io_pgtable *
+apple_dart_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie)

drivers/iommu/io-pgtable-arm.c arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie) · line 1069

Quoted 10 lines
drivers/iommu/io-pgtable-arm.c +10 -0
+{
+	struct arm_lpae_io_pgtable *data;
+
+	if (cfg->ias > 36)
+		return NULL;
+	if (cfg->oas > 36)
+		return NULL;
+
+	if (!cfg->coherent_walk)
+		return NULL;

This all feels like IOMMU-specific limitations leaking into the page-table code here; it doesn't feel so unlikely that future implementations of this IP might have greater addressing capabilities, for example, and so I don't see why the page-table code needs to police this.

drivers/iommu/io-pgtable-arm.c arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie) · line 1073

Quoted 3 lines
drivers/iommu/io-pgtable-arm.c +3 -0
+	cfg->pgsize_bitmap &= SZ_16K;
+	if (!cfg->pgsize_bitmap)
+		return NULL;

This is worrying (and again, I don't think this belongs here). How is this thing supposed to work if the CPU is using 4k pages?

Will

author
#6 lore raw

Re: [PATCH v2 1/3] iommu: io-pgtable: add DART pagetable format

Quoted 1 line · Will Deacon

On Sun, Mar 28, 2021 at 09:40:07AM +0200, Sven Peter wrote:

[...]

Quoted 10 lines
Quoted 10 lines
drivers/iommu/io-pgtable-arm.c +9 -0
+static struct io_pgtable *
+apple_dart_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie)
+{
+	struct arm_lpae_io_pgtable *data;
+
+	if (cfg->ias > 36)
+		return NULL;
+	if (cfg->oas > 36)
+		return NULL;
Quoted 8 lines

drivers/iommu/io-pgtable-arm.c arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie) · line 1069

Quoted 3 lines
drivers/iommu/io-pgtable-arm.c +3 -0
+
+	if (!cfg->coherent_walk)
+		return NULL;

This all feels like IOMMU-specific limitations leaking into the page-table code here; it doesn't feel so unlikely that future implementations of this IP might have greater addressing capabilities, for example, and so I don't see why the page-table code needs to police this.

That's true, this really doesn't belong here.
I'll fix it for the next version and make sure to keep iommu-specific limitations inside the driver itself.

Quoted 7 lines

drivers/iommu/io-pgtable-arm.c arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie) · line 1073

Quoted 4 lines
drivers/iommu/io-pgtable-arm.c +3 -0
+	cfg->pgsize_bitmap &= SZ_16K;
+	if (!cfg->pgsize_bitmap)
+		return NULL;

This is worrying (and again, I don't think this belongs here). How is this thing supposed to work if the CPU is using 4k pages?

This SoC is just full of fun surprises!
I didn't even think about that case since I've always been using 16k pages so far.

I've checked again and wasn't able to find any way to configure the pagesize of the IOMMU. There seem to be variants of this IP in older iPhones which support a 4k pagesize but to the best of my knowledge this is hard wired and not configurable in software.

When booting with 4k pages I hit the BUG_ON in iova.c that ensures that the iommu pagesize has to be <= the cpu page size.

I see two options here and I'm not sure I like either of them:

1) Just don't support 4k CPU pages together with IOMMU translations and only allow full bypass mode there.
This would however mean that PCIe (i.e. ethernet, usb ports on the Mac mini) and possibly Thunderbolt support would not be possible since these devices don't seem to like iommu bypass mode at all.

2) I've had a brief discussion on IRC with Arnd about this [1] and he pointed out that the dma_map_sg API doesn't make any guarantees about the returned iovas and that it might be possible to make this work at least for devices that go through the normal DMA API.

I've then replaced the page size check with a WARN_ON in iova.c just to see what happens. At least normal devices that go through the DMA API seem to work with my configuration. iommu_dma_alloc took the iommu_dma_alloc_remap path which was called with the cpu page size but then used domain->pgsize_bitmap to increase that to 16k. So this kinda works out, but there are other functions in dma-iommu.c that I believe rely on the fact that the iommu can map single cpu pages. This feels very fragile right now and would probably require some rather invasive changes.

Any driver that tries to use the iommu API directly could be trouble as well if they make similar assumptions.

Is this something you would even want to support in the iommu subsytem and is it even possible to do this in a sane way?

Best,

Sven

[1] https://freenode.irclog.whitequark.org/asahi/2021-04-07#29609786;

#7 lore raw

Re: [PATCH v2 1/3] iommu: io-pgtable: add DART pagetable format

Quoted 21 lines · Sven Peter
Quoted 8 lines · Will Deacon

drivers/iommu/io-pgtable-arm.c arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie) · line 1073

Quoted 3 lines · Sven Peter
drivers/iommu/io-pgtable-arm.c +3 -0
+   cfg->pgsize_bitmap &= SZ_16K;
+   if (!cfg->pgsize_bitmap)
+           return NULL;

This is worrying (and again, I don't think this belongs here). How is this thing supposed to work if the CPU is using 4k pages?

This SoC is just full of fun surprises!
I didn't even think about that case since I've always been using 16k pages so far.

I've checked again and wasn't able to find any way to configure the pagesize of the IOMMU. There seem to be variants of this IP in older iPhones which support a 4k pagesize but to the best of my knowledge this is hard wired and not configurable in software.

When booting with 4k pages I hit the BUG_ON in iova.c that ensures that the iommu pagesize has to be <= the cpu page size.

Quoted 7 lines

I see two options here and I'm not sure I like either of them:

1) Just don't support 4k CPU pages together with IOMMU translations and only allow full bypass mode there.
This would however mean that PCIe (i.e. ethernet, usb ports on the Mac mini) and possibly Thunderbolt support would not be possible since these devices don't seem to like iommu bypass mode at all.

It should be possible to do a fake bypass mode by just programming a static page table for as much address space as you can, and then use swiotlb to address any memory beyond that. This won't perform well because it requires bounce buffers for any high memory, but it can be a last resort if a dart instance cannot do normal bypass mode.

Quoted 5 lines

2) I've had a brief discussion on IRC with Arnd about this [1] and he pointed out that the dma_map_sg API doesn't make any guarantees about the returned iovas and that it might be possible to make this work at least for devices that go through the normal DMA API.

Quoted 8 lines

I've then replaced the page size check with a WARN_ON in iova.c just to see what happens. At least normal devices that go through the DMA API seem to work with my configuration. iommu_dma_alloc took the iommu_dma_alloc_remap path which was called with the cpu page size but then used domain->pgsize_bitmap to increase that to 16k. So this kinda works out, but there are other functions in dma-iommu.c that I believe rely on the fact that the iommu can map single cpu pages. This feels very fragile right now and would probably require some rather invasive changes.

The other second-to-last resort here would be to duplicate the code from the dma-iommu code and implement the dma-mapping API directly on top of the dart hardware instead of the iommu layer. This would probably be much faster than the swiotlb on top of a bypass or a linear map, but it's a really awful abstraction that would require adding special cases into a lot of generic code.

Quoted 2 lines

Any driver that tries to use the iommu API directly could be trouble as well if they make similar assumptions.

I think pretty much all drivers using the iommu API directly already depends on having a matching page size. I don't see any way to use e.g. PCI device assignment using vfio, or a GPU driver with per-process contexts when the iotlb page size is larger than the CPU's.

Quoted 2 lines

Is this something you would even want to support in the iommu subsytem and is it even possible to do this in a sane way?

I don't know how hard it is to do adjust the dma-iommu implementation to allow this, but as long as we can work out the DT binding to support both normal dma-iommu mode with 16KB pages and some kind of passthrough mode (emulated or not) with 4KB pages, it can be left as a possible optimization for later.

Arnd

maintainer
#8 lore raw

Re: [PATCH v2 1/3] iommu: io-pgtable: add DART pagetable format

Quoted 65 lines · Arnd Bergmann
Quoted 21 lines · Sven Peter
Quoted 8 lines · Will Deacon

drivers/iommu/io-pgtable-arm.c arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie) · line 1073

Quoted 3 lines · Sven Peter
drivers/iommu/io-pgtable-arm.c +3 -0
+   cfg->pgsize_bitmap &= SZ_16K;
+   if (!cfg->pgsize_bitmap)
+           return NULL;

This is worrying (and again, I don't think this belongs here). How is this thing supposed to work if the CPU is using 4k pages?

This SoC is just full of fun surprises!
I didn't even think about that case since I've always been using 16k pages so far.

I've checked again and wasn't able to find any way to configure the pagesize of the IOMMU. There seem to be variants of this IP in older iPhones which support a 4k pagesize but to the best of my knowledge this is hard wired and not configurable in software.

When booting with 4k pages I hit the BUG_ON in iova.c that ensures that the iommu pagesize has to be <= the cpu page size.

Quoted 7 lines

I see two options here and I'm not sure I like either of them:

1) Just don't support 4k CPU pages together with IOMMU translations and only allow full bypass mode there.
This would however mean that PCIe (i.e. ethernet, usb ports on the Mac mini) and possibly Thunderbolt support would not be possible since these devices don't seem to like iommu bypass mode at all.

It should be possible to do a fake bypass mode by just programming a static page table for as much address space as you can, and then use swiotlb to address any memory beyond that. This won't perform well because it requires bounce buffers for any high memory, but it can be a last resort if a dart instance cannot do normal bypass mode.

Quoted 5 lines

2) I've had a brief discussion on IRC with Arnd about this [1] and he pointed out that the dma_map_sg API doesn't make any guarantees about the returned iovas and that it might be possible to make this work at least for devices that go through the normal DMA API.

Quoted 8 lines

I've then replaced the page size check with a WARN_ON in iova.c just to see what happens. At least normal devices that go through the DMA API seem to work with my configuration. iommu_dma_alloc took the iommu_dma_alloc_remap path which was called with the cpu page size but then used domain->pgsize_bitmap to increase that to 16k. So this kinda works out, but there are other functions in dma-iommu.c that I believe rely on the fact that the iommu can map single cpu pages. This feels very fragile right now and would probably require some rather invasive changes.

The other second-to-last resort here would be to duplicate the code from the dma-iommu code and implement the dma-mapping API directly on top of the dart hardware instead of the iommu layer. This would probably be much faster than the swiotlb on top of a bypass or a linear map, but it's a really awful abstraction that would require adding special cases into a lot of generic code.

Quoted 2 lines

Any driver that tries to use the iommu API directly could be trouble as well if they make similar assumptions.

I think pretty much all drivers using the iommu API directly already depends on having a matching page size. I don't see any way to use e.g. PCI device assignment using vfio, or a GPU driver with per-process contexts when the iotlb page size is larger than the CPU's.

Quoted 8 lines
Quoted 2 lines

Is this something you would even want to support in the iommu subsytem and is it even possible to do this in a sane way?

I don't know how hard it is to do adjust the dma-iommu implementation to allow this, but as long as we can work out the DT binding to support both normal dma-iommu mode with 16KB pages and some kind of passthrough mode (emulated or not) with 4KB pages, it can be left as a possible optimization for later.

I think one of the main things to modify is the IOVA allocator (drivers/iommu/iova.c). Once that is happy with pages bigger than the CPU page size, then you could probably fake everything else in the DMA layer by offsetting the returned DMA addresses into the 16K page which got mapped.

Will

author
#9 lore raw

PATCH v2 2/3 dt-bindings: iommu: add DART iommu bindings

In reply to Sven Peter

DART (Device Address Resolution Table) is the iommu found on Apple ARM SoCs such as the M1.

---
 .../devicetree/bindings/iommu/apple,dart.yaml | 81 +++++++++++++++++++
 MAINTAINERS                                   |  6 ++
 2 files changed, 87 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/iommu/apple,dart.yaml
Documentation/devicetree/bindings/iommu/apple,dart.yaml new file +81 -0
new file mode 100644
index 000000000000..c0b43d90c157
@@ -0,0 +1,81 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/iommu/apple,dart.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Apple DART IOMMU
+
+maintainers:
+  - Sven Peter <[email protected]>
+
+description: |+
+  Apple SoCs may contain an implementation of their Device Address
+  Resolution Table which provides a mandatory layer of address
+  translations for various masters.
+
+  Each DART instance is capable of handling up to 16 different streams
+  with individual pagetables and page-level read/write protection flags.
+
+  This DART IOMMU also raises interrupts in response to various
+  fault conditions.
+
+properties:
+  compatible:
+    const: apple,t8103-dart
+
+  reg:
+    maxItems: 1
+
+  interrupts:
+    maxItems: 1
+
+  clocks:
+    description:
+      Reference to the gate clock phandle if required for this IOMMU.
+      Optional since not all IOMMUs are attached to a clock gate.
+
+  '#iommu-cells':
+    const: 1
+    description:
+      Has to be one. The single cell describes the stream id emitted by
+      a master to the IOMMU.
+
+required:
+  - compatible
+  - reg
+  - '#iommu-cells'
+  - interrupts
+
+additionalProperties: false
+
+examples:
+  - |+
+    dart1: dart1@82f80000 {
+      compatible = "apple,t8103-dart";
+      reg = <0x82f80000 0x4000>;
+      interrupts = <1 781 4>;
+      #iommu-cells = <1>;
+    };
+
+    master1 {
+      iommus = <&{/dart1} 0>;
+    };
+
+  - |+
+    dart2a: dart2a@82f00000 {
+      compatible = "apple,t8103-dart";
+      reg = <0x82f00000 0x4000>;
+      interrupts = <1 781 4>;
+      #iommu-cells = <1>;
+    };
+    dart2b: dart2@82f80000 {
+      compatible = "apple,t8103-dart";
+      reg = <0x82f80000 0x4000>;
+      interrupts = <1 781 4>;
+      #iommu-cells = <1>;
+    };
+
+    master2 {
+      iommus = <&{/dart2a} 0>, <&{/dart2b} 1>;
+    };
MAINTAINERS +6 -0
index 9ac46317840b..f5397328fa1f 100644
@@ -1236,6 +1236,12 @@ L:	[email protected]
 S:	Odd fixes
 F:	drivers/input/mouse/bcm5974.c
 
+APPLE DART IOMMU DRIVER
+M:	Sven Peter <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	Documentation/devicetree/bindings/iommu/apple,t8103-dart.yaml
+
 APPLE SMC DRIVER
 M:	Henrik Rydberg <[email protected]>
 L:	[email protected]
Signature · 6 lines
2.25.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#10 lore raw

Re: [PATCH v2 2/3] dt-bindings: iommu: add DART iommu bindings

On Sun, Mar 28, 2021 at 9:40 AM Sven Peter <[email protected]> wrote:

I noticed only one detail here:

Documentation/devicetree/bindings/iommu/apple,dart.yaml line 71

Quoted 7 lines
Documentation/devicetree/bindings/iommu/apple,dart.yaml new file +7 -0
+  - |+
+    dart2a: dart2a@82f00000 {
+      compatible = "apple,t8103-dart";
+      reg = <0x82f00000 0x4000>;
+      interrupts = <1 781 4>;
+      #iommu-cells = <1>;
+    };

The name of the iommu should be iommu@82f00000, not dart2a@82f00000.

Arnd
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author
#11 lore raw

Re: [PATCH v2 2/3] dt-bindings: iommu: add DART iommu bindings

Quoted 12 lines · Arnd Bergmann

On Sun, Mar 28, 2021 at 9:40 AM Sven Peter <[email protected]> wrote:

I noticed only one detail here:

Documentation/devicetree/bindings/iommu/apple,dart.yaml line 71

Quoted 7 lines
Documentation/devicetree/bindings/iommu/apple,dart.yaml new file +7 -0
+  - |+
+    dart2a: dart2a@82f00000 {
+      compatible = "apple,t8103-dart";
+      reg = <0x82f00000 0x4000>;
+      interrupts = <1 781 4>;
+      #iommu-cells = <1>;
+    };
Quoted 4 lines

The name of the iommu should be iommu@82f00000, not dart2a@82f00000.

Arnd

Thanks, fixed for v3. I've also just noticed that I forgot to update the filename in MAINTAINERS after I renamed it from apple,t8103-dart.yaml which I've fixed as well.

Sven

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author
#12 lore raw

PATCH v2 3/3 iommu: dart: Add DART iommu driver

In reply to Sven Peter

Apple's new SoCs use iommus for almost all peripherals. These Device Address Resolution Tables must be setup before these peripherals can act as DMA masters.

---
 MAINTAINERS                      |   1 +
 drivers/iommu/Kconfig            |  14 +
 drivers/iommu/Makefile           |   1 +
 drivers/iommu/apple-dart-iommu.c | 858 +++++++++++++++++++++++++++++++
 4 files changed, 874 insertions(+)
 create mode 100644 drivers/iommu/apple-dart-iommu.c
MAINTAINERS +1 -0
index f5397328fa1f..70747b8ac0ee 100644
@@ -1241,6 +1241,7 @@ M:	Sven Peter <[email protected]>
 L:	[email protected]
 S:	Maintained
 F:	Documentation/devicetree/bindings/iommu/apple,t8103-dart.yaml
+F:	drivers/iommu/apple-dart-iommu.c
 
 APPLE SMC DRIVER
 M:	Henrik Rydberg <[email protected]>
drivers/iommu/Kconfig +14 -0
index 192ef8f61310..a1b239147dbc 100644
@@ -249,6 +249,20 @@ config SPAPR_TCE_IOMMU
 	  Enables bits of IOMMU API required by VFIO. The iommu_ops
 	  is not implemented as it is not necessary for VFIO.
 
+config IOMMU_APPLE_DART
+	tristate "Apple DART IOMMU Support"
+	depends on ARM64 || (COMPILE_TEST && !GENERIC_ATOMIC64)
+	select IOMMU_API
+	select IOMMU_IO_PGTABLE
+	select IOMMU_IO_PGTABLE_LPAE
+	help
+	  Support for Apple DART (Device Address Resolution Table) IOMMUs
+	  found in Apple ARM SoCs like the M1.
+	  This IOMMU is required for most peripherals using DMA to access
+	  the main memory.
+
+	  Say Y here if you are using an Apple SoC with a DART IOMMU.
+
 # ARM IOMMU support
 config ARM_SMMU
 	tristate "ARM Ltd. System MMU (SMMU) Support"
drivers/iommu/Makefile +1 -0
index 61bd30cd8369..5f21f0dfec6a 100644
@@ -28,3 +28,4 @@ obj-$(CONFIG_S390_IOMMU) += s390-iommu.o
 obj-$(CONFIG_HYPERV_IOMMU) += hyperv-iommu.o
 obj-$(CONFIG_VIRTIO_IOMMU) += virtio-iommu.o
 obj-$(CONFIG_IOMMU_SVA_LIB) += iommu-sva-lib.o
+obj-$(CONFIG_IOMMU_APPLE_DART) += apple-dart-iommu.o
drivers/iommu/apple-dart-iommu.c new file +858 -0

858 lines not on this page · raw message

Signature · 6 lines
2.25.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
#13 lore raw

Re: [PATCH v2 3/3] iommu: dart: Add DART iommu driver

I love your patch! Yet something to improve:

Hi Sven,

I love your patch! Yet something to improve:

[auto build test ERROR on arm-perf/for-next/perf]
[also build test ERROR on linus/master v5.12-rc4]
[cannot apply to iommu/next next-20210326]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Sven-Peter/Apple-M1-DART-IOMMU-driver/20210328-154437
base:   https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git for-next/perf
config: sparc-allmodconfig (attached as .config)
compiler: sparc64-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/ecad602f1731c2ad5ba1579d56da519e38f57b2b
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Sven-Peter/Apple-M1-DART-IOMMU-driver/20210328-154437 git checkout ecad602f1731c2ad5ba1579d56da519e38f57b2b
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=sparc

If you fix the issue, kindly add following tag as appropriate

All error/warnings (new ones prefixed by >>):

drivers/iommu/apple-dart-iommu.c: In function 'apple_dart_finalize_domain':

>> drivers/iommu/apple-dart-iommu.c:427:34: error: implicit declaration of function 'DMA_BIT_MASK'; did you mean 'BIT_MASK'? [-Werror=implicit-function-declaration]
     427 |  domain->geometry.aperture_end = DMA_BIT_MASK(32);
         |                                  ^~~~~~~~~~~~
         |                                  BIT_MASK
   drivers/iommu/apple-dart-iommu.c: In function 'apple_dart_attach_stream':
>> drivers/iommu/apple-dart-iommu.c:456:11: error: implicit declaration of function 'kzalloc'; did you mean 'kvzalloc'? [-Werror=implicit-function-declaration]
     456 |  stream = kzalloc(sizeof(*stream), GFP_KERNEL);
         |           ^~~~~~~
         |           kvzalloc
>> drivers/iommu/apple-dart-iommu.c:456:9: warning: assignment to 'struct apple_dart_stream *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
     456 |  stream = kzalloc(sizeof(*stream), GFP_KERNEL);
         |         ^
   drivers/iommu/apple-dart-iommu.c: In function 'apple_dart_detach_stream':
>> drivers/iommu/apple-dart-iommu.c:523:5: error: implicit declaration of function 'kfree'; did you mean 'kvfree'? [-Werror=implicit-function-declaration]
     523 |     kfree(stream);
         |     ^~~~~
         |     kvfree
   drivers/iommu/apple-dart-iommu.c: In function 'apple_dart_domain_alloc':
>> drivers/iommu/apple-dart-iommu.c:627:14: warning: assignment to 'struct apple_dart_domain *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
     627 |  dart_domain = kzalloc(sizeof(*dart_domain), GFP_KERNEL);
         |              ^
   drivers/iommu/apple-dart-iommu.c: In function 'apple_dart_of_xlate':
>> drivers/iommu/apple-dart-iommu.c:659:7: warning: assignment to 'struct apple_dart_master_cfg *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
     659 |   cfg = kzalloc(struct_size(cfg, streams, 1), GFP_KERNEL);
         |       ^
>> drivers/iommu/apple-dart-iommu.c:663:13: error: implicit declaration of function 'krealloc'; did you mean 'kvcalloc'? [-Werror=implicit-function-declaration]
     663 |   cfg_new = krealloc(
         |             ^~~~~~~~
         |             kvcalloc
   drivers/iommu/apple-dart-iommu.c:663:11: warning: assignment to 'struct apple_dart_master_cfg *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
     663 |   cfg_new = krealloc(
         |           ^

cc1: some warnings being treated as errors

vim +427 drivers/iommu/apple-dart-iommu.c

397
398 /* must be called with held dart_domain->lock */
399 static int apple_dart_finalize_domain(struct iommu_domain *domain)
400 {
401 struct apple_dart_domain *dart_domain = to_dart_domain(domain);
402 struct apple_dart *dart = dart_domain->dart;
403 struct io_pgtable_cfg pgtbl_cfg;
404
405 if (dart_domain->pgtbl_ops)
406 return 0;
407 if (dart_domain->type != IOMMU_DOMAIN_DMA &&
408 dart_domain->type != IOMMU_DOMAIN_UNMANAGED)
409 return 0;
410
411 pgtbl_cfg = (struct io_pgtable_cfg){
412 .pgsize_bitmap = SZ_16K,
413 .ias = 32,
414 .oas = 36,
415 .coherent_walk = 1,
416 .tlb = &apple_dart_tlb_ops,
417 .iommu_dev = dart->dev,
418 };
419
420 dart_domain->pgtbl_ops =
421 alloc_io_pgtable_ops(ARM_APPLE_DART, &pgtbl_cfg, domain);
422 if (!dart_domain->pgtbl_ops)
423 return -ENOMEM;
424
425 domain->pgsize_bitmap = pgtbl_cfg.pgsize_bitmap;
426 domain->geometry.aperture_start = 0;

 > 427		domain->geometry.aperture_end = DMA_BIT_MASK(32);
   428		domain->geometry.force_aperture = true;
   429	
   430		return 0;
   431	}
   432	
   433	/* must be called with held domain->lock */
   434	static int apple_dart_attach_stream(struct apple_dart_domain *domain,
   435					    struct apple_dart *dart, u32 sid)
   436	{
   437		unsigned long flags;
   438		struct apple_dart_stream *stream;
   439		struct io_pgtable_cfg *pgtbl_cfg;
   440		int ret;
   441	
   442		list_for_each_entry(stream, &domain->streams, stream_head) {
   443			if (stream->dart == dart && stream->sid == sid) {
   444				stream->num_devices++;
   445				return 0;
   446			}
   447		}
   448	
   449		spin_lock_irqsave(&dart->lock, flags);
   450	
   451		if (WARN_ON(dart->used_sids & BIT(sid))) {
   452			ret = -EINVAL;
   453			goto error;
   454		}
   455	
 > 456		stream = kzalloc(sizeof(*stream), GFP_KERNEL);
   457		if (!stream) {
   458			ret = -ENOMEM;
   459			goto error;
   460		}
   461	
   462		stream->dart = dart;
   463		stream->sid = sid;
   464		stream->num_devices = 1;
   465		list_add(&stream->stream_head, &domain->streams);
   466	
   467		dart->used_sids |= BIT(sid);
   468		spin_unlock_irqrestore(&dart->lock, flags);
   469	
   470		apple_dart_hw_clear_all_ttbrs(stream->dart, stream->sid);
   471	
   472		switch (domain->type) {
   473		case IOMMU_DOMAIN_IDENTITY:
   474			apple_dart_hw_enable_bypass(stream->dart, stream->sid);
   475			break;
   476		case IOMMU_DOMAIN_BLOCKED:
   477			apple_dart_hw_enable_isolation(stream->dart, stream->sid);
   478			break;
   479		case IOMMU_DOMAIN_UNMANAGED:
   480		case IOMMU_DOMAIN_DMA:
   481			pgtbl_cfg = &io_pgtable_ops_to_pgtable(domain->pgtbl_ops)->cfg;
   482			apple_dart_hw_set_ttbr(stream->dart, stream->sid, 0,
   483					       pgtbl_cfg->apple_dart_cfg.ttbr);
   484	
   485			apple_dart_hw_enable_translation(stream->dart, stream->sid);
   486			apple_dart_hw_invalidate_tlb_stream(stream->dart, stream->sid);
   487			break;
   488		}
   489	
   490		return 0;
   491	
   492	error:
   493		spin_unlock_irqrestore(&dart->lock, flags);
   494		return ret;
   495	}
   496	
   497	static void apple_dart_disable_stream(struct apple_dart *dart, u32 sid)
   498	{
   499		unsigned long flags;
   500	
   501		apple_dart_hw_enable_isolation(dart, sid);
   502		apple_dart_hw_clear_all_ttbrs(dart, sid);
   503		apple_dart_hw_invalidate_tlb_stream(dart, sid);
   504	
   505		spin_lock_irqsave(&dart->lock, flags);
   506		dart->used_sids &= ~BIT(sid);
   507		spin_unlock_irqrestore(&dart->lock, flags);
   508	}
   509	
   510	/* must be called with held domain->lock */
   511	static void apple_dart_detach_stream(struct apple_dart_domain *domain,
   512					     struct apple_dart *dart, u32 sid)
   513	{
   514		struct apple_dart_stream *stream;
   515	
   516		list_for_each_entry(stream, &domain->streams, stream_head) {
   517			if (stream->dart == dart && stream->sid == sid) {
   518				stream->num_devices--;
   519	
   520				if (stream->num_devices == 0) {
   521					apple_dart_disable_stream(dart, sid);
   522					list_del(&stream->stream_head);
 > 523					kfree(stream);

524 }
525 return;
526 }
527 }
528 }
529

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

author
#14 lore raw

Re: [PATCH v2 3/3] iommu: dart: Add DART iommu driver

Hi,

Quoted 8 lines · kernel test robot

All error/warnings (new ones prefixed by >>):

drivers/iommu/apple-dart-iommu.c: In function 'apple_dart_finalize_domain':

Quoted 1 line
Quoted 1 line

drivers/iommu/apple-dart-iommu.c:427:34: error: implicit declaration of function 'DMA_BIT_MASK'; did you mean 'BIT_MASK'? [-Werror=implicit-function-declaration]

     427 |  domain->geometry.aperture_end = DMA_BIT_MASK(32);
         |                                  ^~~~~~~~~~~~
         |                                  BIT_MASK

Fixed by adding the missing <linux/dma-mapping.h> include for DMA_BIT_MASK.

Quoted 32 lines

drivers/iommu/apple-dart-iommu.c: In function 'apple_dart_attach_stream':

Quoted 1 line
Quoted 1 line

drivers/iommu/apple-dart-iommu.c:456:11: error: implicit declaration of function 'kzalloc'; did you mean 'kvzalloc'? [-Werror=implicit-function-declaration]

     456 |  stream = kzalloc(sizeof(*stream), GFP_KERNEL);
         |           ^~~~~~~
         |           kvzalloc
Quoted 1 line
Quoted 1 line

drivers/iommu/apple-dart-iommu.c:456:9: warning: assignment to 'struct apple_dart_stream *' from 'int' makes pointer from integer without a cast [-Wint-conversion]

     456 |  stream = kzalloc(sizeof(*stream), GFP_KERNEL);
         |         ^

drivers/iommu/apple-dart-iommu.c: In function 'apple_dart_detach_stream':

Quoted 1 line
Quoted 1 line

drivers/iommu/apple-dart-iommu.c:523:5: error: implicit declaration of function 'kfree'; did you mean 'kvfree'? [-Werror=implicit-function-declaration]

     523 |     kfree(stream);
         |     ^~~~~
         |     kvfree

drivers/iommu/apple-dart-iommu.c: In function 'apple_dart_domain_alloc':

Quoted 1 line
Quoted 1 line

drivers/iommu/apple-dart-iommu.c:627:14: warning: assignment to 'struct apple_dart_domain *' from 'int' makes pointer from integer without a cast [-Wint-conversion]

     627 |  dart_domain = kzalloc(sizeof(*dart_domain), GFP_KERNEL);
         |              ^

drivers/iommu/apple-dart-iommu.c: In function 'apple_dart_of_xlate':

Quoted 1 line
Quoted 1 line

drivers/iommu/apple-dart-iommu.c:659:7: warning: assignment to 'struct apple_dart_master_cfg *' from 'int' makes pointer from integer without a cast [-Wint-conversion]

     659 |   cfg = kzalloc(struct_size(cfg, streams, 1), GFP_KERNEL);
         |       ^
Quoted 1 line
Quoted 1 line

drivers/iommu/apple-dart-iommu.c:663:13: error: implicit declaration

of function 'krealloc'; did you mean 'kvcalloc'?
[-Werror=implicit-function-declaration]

     663 |   cfg_new = krealloc(
         |             ^~~~~~~~
         |             kvcalloc
   drivers/iommu/apple-dart-iommu.c:663:11: warning: assignment to 
'struct apple_dart_master_cfg *' from 'int' makes pointer from integer 
without a cast [-Wint-conversion]
     663 |   cfg_new = krealloc(
         |           ^
Quoted 3 lines

cc1: some warnings being treated as errors

Fixed by adding the missing <linux/slab.h> include for krealloc, kfree and kzalloc.

Thanks,

Sven

maintainer
#15 lore raw

Re: [PATCH v2 3/3] iommu: dart: Add DART iommu driver

In reply to Sven Peter
Quoted 12 lines · Sven Peter

Apple's new SoCs use iommus for almost all peripherals. These Device Address Resolution Tables must be setup before these peripherals can act as DMA masters.

---
 MAINTAINERS                      |   1 +
 drivers/iommu/Kconfig            |  14 +
 drivers/iommu/Makefile           |   1 +
 drivers/iommu/apple-dart-iommu.c | 858 +++++++++++++++++++++++++++++++
 4 files changed, 874 insertions(+)
 create mode 100644 drivers/iommu/apple-dart-iommu.c

[...]

Quoted 18 lines
drivers/iommu/apple-dart-iommu.c new file +18 -0
+/* must be called with held domain->lock */
+static int apple_dart_attach_stream(struct apple_dart_domain *domain,
+				    struct apple_dart *dart, u32 sid)
+{
+	unsigned long flags;
+	struct apple_dart_stream *stream;
+	struct io_pgtable_cfg *pgtbl_cfg;
+	int ret;
+
+	list_for_each_entry(stream, &domain->streams, stream_head) {
+		if (stream->dart == dart && stream->sid == sid) {
+			stream->num_devices++;
+			return 0;
+		}
+	}
+
+	spin_lock_irqsave(&dart->lock, flags);
+

drivers/iommu/apple-dart-iommu.c line 460

Quoted 10 lines
drivers/iommu/apple-dart-iommu.c new file +10 -0
+	if (WARN_ON(dart->used_sids & BIT(sid))) {
+		ret = -EINVAL;
+		goto error;
+	}
+
+	stream = kzalloc(sizeof(*stream), GFP_KERNEL);
+	if (!stream) {
+		ret = -ENOMEM;
+		goto error;
+	}

Just in case you missed it, a cocci bot noticed that you're using GFP_KERNEL to allocate while holding a spinlock here:

https://lore.kernel.org/r/alpine.DEB.2.22.394.2104041724340.2958@hadrien

Will

author
#16 lore raw

Re: [PATCH v2 3/3] iommu: dart: Add DART iommu driver

Quoted 48 lines · Will Deacon
Quoted 12 lines · Sven Peter

Apple's new SoCs use iommus for almost all peripherals. These Device Address Resolution Tables must be setup before these peripherals can act as DMA masters.

---
 MAINTAINERS                      |   1 +
 drivers/iommu/Kconfig            |  14 +
 drivers/iommu/Makefile           |   1 +
 drivers/iommu/apple-dart-iommu.c | 858 +++++++++++++++++++++++++++++++
 4 files changed, 874 insertions(+)
 create mode 100644 drivers/iommu/apple-dart-iommu.c

[...]

Quoted 18 lines
drivers/iommu/apple-dart-iommu.c new file +18 -0
+/* must be called with held domain->lock */
+static int apple_dart_attach_stream(struct apple_dart_domain *domain,
+				    struct apple_dart *dart, u32 sid)
+{
+	unsigned long flags;
+	struct apple_dart_stream *stream;
+	struct io_pgtable_cfg *pgtbl_cfg;
+	int ret;
+
+	list_for_each_entry(stream, &domain->streams, stream_head) {
+		if (stream->dart == dart && stream->sid == sid) {
+			stream->num_devices++;
+			return 0;
+		}
+	}
+
+	spin_lock_irqsave(&dart->lock, flags);
+

drivers/iommu/apple-dart-iommu.c line 460

Quoted 10 lines
drivers/iommu/apple-dart-iommu.c new file +10 -0
+	if (WARN_ON(dart->used_sids & BIT(sid))) {
+		ret = -EINVAL;
+		goto error;
+	}
+
+	stream = kzalloc(sizeof(*stream), GFP_KERNEL);
+	if (!stream) {
+		ret = -ENOMEM;
+		goto error;
+	}

Just in case you missed it, a cocci bot noticed that you're using GFP_KERNEL to allocate while holding a spinlock here:

Quoted 2 lines

https://lore.kernel.org/r/alpine.DEB.2.22.394.2104041724340.2958@hadrien

Thanks for the reminder!
I haven't replied yet because that one was found later when the bot picked up a (slightly earlier) version that Marc was using to bring up pcie I believe. I'll fix it for the next version.

Sven