Skip to contentSkip to conversation

iommu: Fix default domain setup

PATCH In review Vasant Hegde 3 participants 9 messages Jun 19, 2023 – Jun 23, 2023 IOMMU

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 iommu: Fix default domain setup

Commit 1000dccd5d13 ("iommu: Allow IOMMU_RESV_DIRECT to work on ARM") accidently restored "group->domain" to "old_domain" while keeping "group->default_domain" to new domain. Also freed new domain.

This works fine during boot as 'old_domain' is NULL. But if we try change domain via sysfs using below command then kernel crashes with "kernel NULL pointer dereference".

Change domain command :
- Go to /sys/kernel/iommu_groups/<group id>/
- Unbind device driver for all devices in the group
- echo "<dom type>" > /sys/kernel/iommu_groups/<group id>/type

Fix above described issue by handling error path properly.

Fixes 1000dccd5d13 ("iommu: Allow IOMMU_RESV_DIRECT to work on ARM")
Cc Jason Gunthorpe <[email protected]>

---
Changes in v2:
- Addressed review comments from Baolu

@Joerg,
This patch applies on top of iommu/next branch.

-Vasant

 drivers/iommu/iommu.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/iommu/iommu.c +5 -0
index 9e0228ef612b..e74e45f42c75 100644
@@ -2915,10 +2915,15 @@ static int iommu_setup_default_domain(struct iommu_group *group,
 		}
 	}
 
+	if (old_dom)
+		iommu_domain_free(old_dom);
+	return ret;
+
 err_restore:
 	if (old_dom) {
 		__iommu_group_set_domain_internal(
 			group, old_dom, IOMMU_SET_DOMAIN_MUST_SUCCEED);
+		group->default_domain = old_dom;
 		iommu_domain_free(dom);
 		old_dom = NULL;
 	}
Signature · 1 line
2.31.1
#1 lore raw
Quoted 7 lines · Vasant Hegde

Commit 1000dccd5d13 ("iommu: Allow IOMMU_RESV_DIRECT to work on ARM") accidently restored "group->domain" to "old_domain" while keeping "group->default_domain" to new domain. Also freed new domain.

This works fine during boot as 'old_domain' is NULL. But if we try change domain via sysfs using below command then kernel crashes with "kernel NULL pointer dereference".

There is more wrong here then.. The error unwind should work even if we do it at the wrong time.

Like this?

drivers/iommu/iommu.c +16 -15
index 8f3464ba204498..6fb4533905c37b 100644
@@ -2960,14 +2960,11 @@ static int iommu_setup_default_domain(struct iommu_group *group,
 		ret = __iommu_group_set_domain_internal(
 			group, dom, IOMMU_SET_DOMAIN_MUST_SUCCEED);
 		if (WARN_ON(ret))
-			goto out_free;
+			goto out_free_old;
 	} else {
 		ret = __iommu_group_set_domain(group, dom);
-		if (ret) {
-			iommu_domain_free(dom);
-			group->default_domain = old_dom;
-			return ret;
-		}
+		if (ret)
+			goto err_restore_def_domain;
 	}
 
 	/*
@@ -2980,21 +2977,25 @@ static int iommu_setup_default_domain(struct iommu_group *group,
 		for_each_group_device(group, gdev) {
 			ret = iommu_create_device_direct_mappings(dom, gdev->dev);
 			if (ret)
-				goto err_restore;
+				goto err_restore_domain;
 		}
 	}
 
-err_restore:
-	if (old_dom) {
-		__iommu_group_set_domain_internal(
-			group, old_dom, IOMMU_SET_DOMAIN_MUST_SUCCEED);
-		iommu_domain_free(dom);
-		old_dom = NULL;
-	}
-out_free:
+out_free_old:
 	if (old_dom)
 		iommu_domain_free(old_dom);
 	return ret;
+
+err_restore_domain:
+	if (old_dom)
+		__iommu_group_set_domain_internal(
+			group, old_dom, IOMMU_SET_DOMAIN_MUST_SUCCEED);
+err_restore_def_domain:
+	if (old_dom) {
+		iommu_domain_free(dom);
+		group->default_domain = old_dom;
+	}
+	return ret;
 }
 
 /*
#2 lore raw

Hi Jason,

Quoted 11 lines · Jason Gunthorpe
Quoted 7 lines · Vasant Hegde

Commit 1000dccd5d13 ("iommu: Allow IOMMU_RESV_DIRECT to work on ARM") accidently restored "group->domain" to "old_domain" while keeping "group->default_domain" to new domain. Also freed new domain.

This works fine during boot as 'old_domain' is NULL. But if we try change domain via sysfs using below command then kernel crashes with "kernel NULL pointer dereference".

There is more wrong here then.. The error unwind should work even if we do it at the wrong time.

Agree. Unwind should work all time. But I'm not sure what's wrong with my patch. Are you saying you prefer goto in all error path?

Quoted 2 lines

Like this?

This also fine.

-Vasant

Quoted 58 lines
drivers/iommu/iommu.c +16 -15
index 8f3464ba204498..6fb4533905c37b 100644
@@ -2960,14 +2960,11 @@ static int iommu_setup_default_domain(struct iommu_group *group,
 		ret = __iommu_group_set_domain_internal(
 			group, dom, IOMMU_SET_DOMAIN_MUST_SUCCEED);
 		if (WARN_ON(ret))
-			goto out_free;
+			goto out_free_old;
 	} else {
 		ret = __iommu_group_set_domain(group, dom);
-		if (ret) {
-			iommu_domain_free(dom);
-			group->default_domain = old_dom;
-			return ret;
-		}
+		if (ret)
+			goto err_restore_def_domain;
 	}
 
 	/*
@@ -2980,21 +2977,25 @@ static int iommu_setup_default_domain(struct iommu_group *group,
 		for_each_group_device(group, gdev) {
 			ret = iommu_create_device_direct_mappings(dom, gdev->dev);
 			if (ret)
-				goto err_restore;
+				goto err_restore_domain;
 		}
 	}
 
-err_restore:
-	if (old_dom) {
-		__iommu_group_set_domain_internal(
-			group, old_dom, IOMMU_SET_DOMAIN_MUST_SUCCEED);
-		iommu_domain_free(dom);
-		old_dom = NULL;
-	}
-out_free:
+out_free_old:
 	if (old_dom)
 		iommu_domain_free(old_dom);
 	return ret;
+
+err_restore_domain:
+	if (old_dom)
+		__iommu_group_set_domain_internal(
+			group, old_dom, IOMMU_SET_DOMAIN_MUST_SUCCEED);
+err_restore_def_domain:
+	if (old_dom) {
+		iommu_domain_free(dom);
+		group->default_domain = old_dom;
+	}
+	return ret;
 }
 
 /*
#3 lore raw
In reply to Vasant Hegde
Quoted 36 lines · Vasant Hegde

Commit 1000dccd5d13 ("iommu: Allow IOMMU_RESV_DIRECT to work on ARM") accidently restored "group->domain" to "old_domain" while keeping "group->default_domain" to new domain. Also freed new domain.

This works fine during boot as 'old_domain' is NULL. But if we try change domain via sysfs using below command then kernel crashes with "kernel NULL pointer dereference".

Change domain command :
- Go to /sys/kernel/iommu_groups/<group id>/
- Unbind device driver for all devices in the group
- echo "<dom type>" > /sys/kernel/iommu_groups/<group id>/type

Fix above described issue by handling error path properly.

Fixes 1000dccd5d13 ("iommu: Allow IOMMU_RESV_DIRECT to work on ARM")
Cc Jason Gunthorpe <[email protected]>

---
Changes in v2:
- Addressed review comments from Baolu

@Joerg,
This patch applies on top of iommu/next branch.

(quoted diff) +0 -1
-Vasant
  drivers/iommu/iommu.c | 5 +++++
  1 file changed, 5 insertions(+)
drivers/iommu/iommu.c +0 -0
index 9e0228ef612b..e74e45f42c75 100644

drivers/iommu/iommu.c static int iommu_setup_default_domain(struct iommu_group *group, · line 2920

Quoted 6 lines
drivers/iommu/iommu.c +3 -0
@@ -2915,10 +2915,15 @@ static int iommu_setup_default_domain(struct iommu_group *group,
  		}
  	}
 
+	if (old_dom)
+		iommu_domain_free(old_dom);
+	return ret;

Add out_free tag here and remove the duplicate code at the end of this function.

drivers/iommu/iommu.c static int iommu_setup_default_domain(struct iommu_group *group, · line 2929

Quoted 9 lines
drivers/iommu/iommu.c +2 -0
+
  err_restore:
  	if (old_dom) {
  		__iommu_group_set_domain_internal(
  			group, old_dom, IOMMU_SET_DOMAIN_MUST_SUCCEED);
+		group->default_domain = old_dom;
  		iommu_domain_free(dom);
  		old_dom = NULL;
  	}

The err_restore branch doesn't work if old_dom is NULL. We have no means to restore a group from a successful first-time attaching to NULL attaching.

Or I overlooked anything?

Best regards,
baolu

#4 lore raw
Quoted 4 lines · Baolu Lu
Quoted 4 lines
  err_restore:
  	if (old_dom) {
  		__iommu_group_set_domain_internal(
  			group, old_dom, IOMMU_SET_DOMAIN_MUST_SUCCEED);
Quoted 8 lines

drivers/iommu/iommu.c static int iommu_setup_default_domain(struct iommu_group *group, · line 2929

Quoted 4 lines
drivers/iommu/iommu.c +1 -0
+		group->default_domain = old_dom;
  		iommu_domain_free(dom);
  		old_dom = NULL;
  	}

The err_restore branch doesn't work if old_dom is NULL. We have no means to restore a group from a successful first-time attaching to NULL attaching.

Yes, this is what I fixed in my alternative version

Jason

#5 lore raw

Hi Baolu, Jason,

Quoted 14 lines · Jason Gunthorpe
Quoted 4 lines · Baolu Lu
Quoted 4 lines
  err_restore:
  	if (old_dom) {
  		__iommu_group_set_domain_internal(
  			group, old_dom, IOMMU_SET_DOMAIN_MUST_SUCCEED);
Quoted 8 lines

drivers/iommu/iommu.c static int iommu_setup_default_domain(struct iommu_group *group, · line 2929

Quoted 4 lines
drivers/iommu/iommu.c +1 -0
+		group->default_domain = old_dom;
  		iommu_domain_free(dom);
  		old_dom = NULL;
  	}

The err_restore branch doesn't work if old_dom is NULL. We have no means to restore a group from a successful first-time attaching to NULL attaching.

Quoted 1 line

Yes, this is what I fixed in my alternative version

Not really. Even your version has

+err_restore_domain:
+ if (old_dom)

Only first time we will have old_dom is NULL and this functions returns error code. iommu_probe_device()/bus_iommu_probe() handler error path properly and frees resources. So I think this is fine.

@Jason,
I'm fine with your version of patch as well. Are you going to send proper
patch -OR- do you want me to respin?

-Vasant

#6 lore raw
Quoted 18 lines · Vasant Hegde

Hi Baolu, Jason,

Quoted 14 lines · Jason Gunthorpe
Quoted 12 lines · Baolu Lu

drivers/iommu/iommu.c static int iommu_setup_default_domain(struct iommu_group *group, · line 2929

Quoted 8 lines
  err_restore:
  	if (old_dom) {
  		__iommu_group_set_domain_internal(
  			group, old_dom, IOMMU_SET_DOMAIN_MUST_SUCCEED);
drivers/iommu/iommu.c +1 -0
+		group->default_domain = old_dom;
  		iommu_domain_free(dom);
  		old_dom = NULL;
  	}

The err_restore branch doesn't work if old_dom is NULL. We have no means to restore a group from a successful first-time attaching to NULL attaching.

Quoted 3 lines
Quoted 1 line

Yes, this is what I fixed in my alternative version

Not really. Even your version has

I though what Baolu means is that it crashes in some of the cases.

Quoted 7 lines
(quoted diff) +2 -0
+err_restore_domain:
+	if (old_dom)

Only first time we will have old_dom is NULL and this functions returns error code. iommu_probe_device()/bus_iommu_probe() handler error path properly and frees resources. So I think this is fine.

Yes, the design is to leave it as-is and know that the error unwind will fail to attach the device and free the domain after release. There is a comment explaining this.

Quoted 3 lines

@Jason,
I'm fine with your version of patch as well. Are you going to send proper
patch -OR- do you want me to respin?

I was hoping you'd test it and can you share the oops message?

Thanks,
Jason

#7 lore raw
Quoted 25 lines · Jason Gunthorpe
Quoted 16 lines · Vasant Hegde

Hi Baolu, Jason,

Quoted 12 lines · Jason Gunthorpe
Quoted 11 lines · Baolu Lu

drivers/iommu/iommu.c static int iommu_setup_default_domain(struct iommu_group *group, · line 2929

Quoted 8 lines
   err_restore:
   	if (old_dom) {
   		__iommu_group_set_domain_internal(
   			group, old_dom, IOMMU_SET_DOMAIN_MUST_SUCCEED);
drivers/iommu/iommu.c +1 -0
+		group->default_domain = old_dom;
   		iommu_domain_free(dom);
   		old_dom = NULL;
   	}

The err_restore branch doesn't work if old_dom is NULL. We have no means to restore a group from a successful first-time attaching to NULL attaching.

Quoted 2 lines
Quoted 1 line

Yes, this is what I fixed in my alternative version

Not really. Even your version has

I though what Baolu means is that it crashes in some of the cases.

Quoted 4 lines
(quoted diff) +2 -0
+err_restore_domain:
+	if (old_dom)
Quoted 7 lines
Quoted 3 lines

Only first time we will have old_dom is NULL and this functions returns error code. iommu_probe_device()/bus_iommu_probe() handler error path properly and frees resources. So I think this is fine.

Yes, the design is to leave it as-is and know that the error unwind will fail to attach the device and free the domain after release. There is a comment explaining this.

I thought that perhaps we could throw a warning to the driver when old_dom is NULL. Something like this:

	if (!old_dom)
		pr_warn("can't restore to old domain\n");
	else
		goto err_restore_domain;

But as Vasant said the only caller can handle this error case gracefully. Then it's fine to keep the code as it is.

Best regards,
baolu

#8 lore raw
In reply to Jason Gunthorpe

Jason,

Quoted 37 lines · Jason Gunthorpe
Quoted 18 lines · Vasant Hegde

Hi Baolu, Jason,

Quoted 14 lines · Jason Gunthorpe
Quoted 12 lines · Baolu Lu

drivers/iommu/iommu.c static int iommu_setup_default_domain(struct iommu_group *group, · line 2929

Quoted 8 lines
  err_restore:
  	if (old_dom) {
  		__iommu_group_set_domain_internal(
  			group, old_dom, IOMMU_SET_DOMAIN_MUST_SUCCEED);
drivers/iommu/iommu.c +1 -0
+		group->default_domain = old_dom;
  		iommu_domain_free(dom);
  		old_dom = NULL;
  	}

The err_restore branch doesn't work if old_dom is NULL. We have no means to restore a group from a successful first-time attaching to NULL attaching.

Quoted 3 lines
Quoted 1 line

Yes, this is what I fixed in my alternative version

Not really. Even your version has

I though what Baolu means is that it crashes in some of the cases.

Quoted 7 lines
(quoted diff) +2 -0
+err_restore_domain:
+	if (old_dom)

Only first time we will have old_dom is NULL and this functions returns error code. iommu_probe_device()/bus_iommu_probe() handler error path properly and frees resources. So I think this is fine.

Yes, the design is to leave it as-is and know that the error unwind will fail to attach the device and free the domain after release. There is a comment explaining this.

Quoted 5 lines
Quoted 3 lines

@Jason,
I'm fine with your version of patch as well. Are you going to send proper
patch -OR- do you want me to respin?

I was hoping you'd test it and can you share the oops message?

This has been tested. Sorry. I should have mentioned it explicitly.

One of the call trace without this fix:

[  350.334395] BUG: kernel NULL pointer dereference, address: 0000000000000000
[  350.341356] #PF: supervisor read access in kernel mode
[  350.346494] #PF: error_code(0x0000) - not-present page
[  350.351625] PGD 0 P4D 0
[  350.354164] Oops: 0000 [#1] PREEMPT SMP NOPTI
[  350.358523] CPU: 1 PID: 3417 Comm: avocado Not tainted 6.4.0-rc4-next-20230602 #3
[  350.366003] Hardware name: Dell Inc. PowerEdge R6515/07PXPY, BIOS 2.3.6
07/06/2021
[  350.373567] RIP: 0010:__iommu_attach_device+0xc/0xa0
[  350.378533] Code: c0 c3 cc cc cc cc 48 89 f0 c3 cc cc cc cc 90 90 90 90 90 90
90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 41 54 55 48 8b 47 08 <48> 8b 00 48
85 c0 74 74 48 89 f5 e8 64 12 49 00 41 89 c4 85 c0 74
[  350.397269] RSP: 0018:ffffabae0220bd48 EFLAGS: 00010246
[  350.402488] RAX: 0000000000000000 RBX: ffff9ac04f70e410 RCX: 0000000000000001
[  350.409620] RDX: ffff9ac044db20c0 RSI: ffff9ac044fa50d0 RDI: ffff9ac04f70e410
[  350.416751] RBP: ffff9ac044fa50d0 R08: 1000000100209001 R09: 00000000000002dc
[  350.423875] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9ac043d54700
[  350.431001] R13: ffff9ac043d54700 R14: 0000000000000001 R15: 0000000000000001
[  350.438133] FS:  00007f02e30ae000(0000) GS:ffff9afeb2440000(0000)
knlGS:0000000000000000
[  350.446217] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  350.451956] CR2: 0000000000000000 CR3: 000000012afca006 CR4: 0000000000770ee0
[  350.459086] PKRU: 55555554
[  350.461792] Call Trace:
[  350.464235]  <TASK>
[  350.466335]  ? __die+0x24/0x70
[  350.469392]  ? page_fault_oops+0x82/0x150
[  350.473404]  ? __iommu_queue_command_sync+0x80/0xc0
[  350.478283]  ? exc_page_fault+0x69/0x150
[  350.482210]  ? asm_exc_page_fault+0x26/0x30
[  350.486396]  ? __iommu_attach_device+0xc/0xa0
[  350.490756]  ? __iommu_attach_device+0x1c/0xa0
[  350.495201]  __iommu_device_set_domain+0x42/0x80
[  350.499820]  __iommu_group_set_domain_internal+0x5d/0x160
[  350.505220]  iommu_setup_default_domain+0x318/0x400
[  350.510097]  iommu_group_store_type+0xb1/0x200
[  350.514536]  kernfs_fop_write_iter+0x12f/0x1c0
[  350.518982]  vfs_write+0x2a2/0x3b0
[  350.522389]  ksys_write+0x63/0xe0
[  350.525707]  do_syscall_64+0x3f/0x90
[  350.529284]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  350.534338] RIP: 0033:0x7f02e2f14a6f
[  350.537918] Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 19 c0 f7 ff 48 8b
54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0
ff ff 77 31 44 89 c7 48 89 44 24 08 e8 5c c0 f7 ff 48
[  350.556663] RSP: 002b:00007ffe5451b6e0 EFLAGS: 00000293 ORIG_RAX:
0000000000000001
[  350.564226] RAX: ffffffffffffffda RBX: 000055f381c20a20 RCX: 00007f02e2f14a6f
[  350.571352] RDX: 0000000000000008 RSI: 000055f382f36220 RDI: 000000000000000d
[  350.578485] RBP: 000055f3829e0340 R08: 0000000000000000 R09: 0000000000000000
[  350.585615] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000008
[  350.592740] R13: 00007f02e30adf80 R14: 000000000000000d R15: 000055f382f36220
[  350.599865]  </TASK>
[  350.602057] Modules linked in: xt_CHECKSUM xt_MASQUERADE xt_conntrack

ipt_REJECT nf_reject_ipv4 nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink bridge stp llc binfmt_misc ipmi_ssif vfat fat intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd kvm irqbypass acpi_ipmi rapl wmi_bmof acpi_cpufreq ccp sg k10temp ipmi_si acpi_power_meter squashfs loop dm_multipath ipmi_devintf ipmi_msghandler ramoops reed_solomon fuse ip_tables ext4 mbcache jbd2 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq libcrc32c raid1 raid0 linear dm_mod sd_mod t10_pi crc64_rocksoft_generic crc64_rocksoft crc64 mgag200 i2c_algo_bit drm_shmem_helper crct10dif_pclmul crc32_pclmul ahci drm_kms_helper crc32c_intel libahci ghash_clmulni_intel mpt3sas sha512_ssse3 drm tg3 raid_class libata i2c_piix4 scsi_transport_sas wmi

[  350.679856] CR2: 0000000000000000
[  350.683174] ---[ end trace 0000000000000000 ]---
[  350.687791] RIP: 0010:__iommu_attach_device+0xc/0xa0
[  350.692758] Code: c0 c3 cc cc cc cc 48 89 f0 c3 cc cc cc cc 90 90 90 90 90 90
90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 41 54 55 48 8b 47 08 <48> 8b 00 48
85 c0 74 74 48 89 f5 e8 64 12 49 00 41 89 c4 85 c0 74
[  350.711495] RSP: 0018:ffffabae0220bd48 EFLAGS: 00010246
[  350.716721] RAX: 0000000000000000 RBX: ffff9ac04f70e410 RCX: 0000000000000001
[  350.723844] RDX: ffff9ac044db20c0 RSI: ffff9ac044fa50d0 RDI: ffff9ac04f70e410
[  350.730968] RBP: ffff9ac044fa50d0 R08: 1000000100209001 R09: 00000000000002dc
[  350.738092] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9ac043d54700
[  350.745215] R13: ffff9ac043d54700 R14: 0000000000000001 R15: 0000000000000001
[  350.752339] FS:  00007f02e30ae000(0000) GS:ffff9afeb2440000(0000)
knlGS:0000000000000000
[  350.760417] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  350.766154] CR2: 0000000000000000 CR3: 000000012afca006 CR4: 0000000000770ee0
[  350.773279] PKRU: 55555554
[  350.775989] Kernel panic - not syncing: Fatal exception
[  351.399507] Kernel Offset: 0x2fe00000 from 0xffffffff81000000 (relocation
range: 0xffffffff80000000-0xffffffffbfffffff)
[  351.410284] ---[ end Kernel panic - not syncing: Fatal exception ]---

-Vasant