Intel IOMMU updates for v7.3
Subsystems (2)
Lu Baolu
maintainerHi Joerg,
The following changes are queued for v7.3-rc1. This update introduces new features, quick fixes and several refactorings, including:
- Consolidation of complex enablement logic into a clean, priority-based state machine.
- Support for the DMA_REMAP_OPT_OUT flag from the VT-d v5.2 specification.
- An update to cache_tag_flush_devtlb_psi() to use full-range constants instead of modifying shared variables for CACHE_TAG_NESTING_DEVTLB.
- A fix for the UCTP context-table slot when copying root entries.
- Fixes for several pre-existing issues reported by Sashiko.
- General code cleanup and refinement.
These patches are based on v7.2-rc6. The full series is also available on GitHub:
https://github.com/LuBaolu/intel-iommu/commits/iommu/vt-d/next
Please consider pulling this series into the iommu/vt-d branch.
Change log:
v2:
- Sashiko reported a critical issue in the previous PATCH 16/20. Remove that patch from this series. It requires more investigation time and will be submitted as a separate patch.
- Add comments to the commit message of PATCH 03/20.
v1: https://lore.kernel.org/linux-iommu/[email protected]/
Best regards,
baolu
Desnes Nunes (1):
iommu/vt-d: Fix UCTP context table slot when copying root entries
Dmitry Antipov (1):
iommu/vt-d: Use kstrtoint_from_user() in dmar_perf_latency_write()
Guanghui Feng (1):
iommu/vt-d: Fix CACHE_TAG_NESTING_DEVTLB polluting shared variables in
flush loop
Kevin Tian (9):
iommu/vt-d: Fix no_iommu to disable platform opt-in iommu/vt-d: Force requesting ACS when tboot is enabled iommu/vt-d: Remove dead code when CONFIG_INTEL_IOMMU is not set iommu/vt-d: Consolidate dmar policy management and force_on logic iommu/vt-d: Use dmar_can_force_on() for platform opt-in iommu/vt-d: Call dmar_can_force_on() for tboot opt-in iommu/vt-d: Remove the 'force_on' variable
iommu/vt-d: Remove dmar_disabled
iommu/vt-d: Support the new DMA_REMAP_OPT_OUT flag bit
Li RongQing (1):
iommu/vt-d: Use logical OR operator for privilege mode check
Lu Baolu (5):
iommu/vt-d: Cache max domain ID to avoid redundant calculation iommu/vt-d: Clear Present bit before tearing down copied context entry iommu/vt-d: Fix iopf_refcount leak on RID domain replacement iommu/vt-d: Tear down scalable-mode context on probe failure iommu/vt-d: Flush context cache with correct SID when tearing down
aliases
ZhaoJinming (1):
iommu/vt-d: Fix copied_tables bitmap leak on error in
copy_translation_tables
drivers/iommu/intel/iommu.h | 73 ++++++++++++++---- include/linux/dmar.h | 1 + drivers/iommu/intel/cache.c | 5 +- drivers/iommu/intel/debugfs.c | 16 +--- drivers/iommu/intel/dmar.c | 96 ++++++++++++++++++++--- drivers/iommu/intel/iommu.c | 139 +++++++++++++++++++--------------- drivers/iommu/intel/pasid.c | 9 ++- drivers/iommu/intel/prq.c | 2 +- drivers/iommu/intel/svm.c | 2 +- 9 files changed, 238 insertions(+), 105 deletions(-)
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 1/19 iommu/vt-d: Fix UCTP context table slot when copying root entries
From: Desnes Nunes <[email protected]>
When translation is already enabled at boot (e.g. kdump), the vt-d driver copies context tables from the previous kernel's root table. In scalable mode, buses that only populate the upper root half (UCTP, devfn >= 0x80) should be written to ctxt_tbls[tbl_idx + 1] through copy_context_table(). However, the current copy path always uses tbl[tbl_idx + 0] in this situa- tion. Since idx wraps to 0 at devfn 0x80 due to a zeroed LCTP, new_ce for LCTP will be NULL and keep pos equals to 0. Thus, UCTP entries will be co- pied into tbl[tbl_idx + 0] instead of tbl[tbl_idx + 1], and written after- wards to root_entry[bus].lo instead of .hi in copy_translation_tables().
In short, devices on bus 0x80 with devfn >= 0x80 fail DMA with fault 0x39, which will break drivers running in kernels with translation pre-enabled. This fixes NO_PASID DMAR faults for UCTP-only buses such as:
DMAR: [DMA Read NO_PASID] Request device [80:14.0] fault addr 0xe81759000
[fault reason 0x39] SM: Present bit in Root Entry is clear
For instance, this fault yielded to locking issues between systemd and xHCI, blocking a system's reboot after a vmcore was captured with kdump:
systemd-udevd[246]: usb3: Worker [255] processing SEQNUM=2193 is taking a long time dracut-initqueue[277]: Timed out while waiting for udev queue to empty. systemd-udevd[246]: usb3: Worker [255] processing SEQNUM=2193 killed systemd-udevd[246]: usb3: Worker [255] terminated by signal 9 (KILL).
...
kdump[569]: saving vmcore complete
...
systemd-shutdown[1]: Rebooting.
INFO: task kworker/0:1:11 blocked for more than 122 seconds.
Not tainted 7.0.0-clean #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/0:1 state:D stack:0 pid:11 tgid:11 ppid:2 task_flags:0x4208160 flags:0x00080000
Workqueue: usb_hub_wq hub_event
Call Trace:
<TASK>
__schedule+0x299/0x5c0
schedule+0x27/0x80
schedule_timeout+0xbd/0x100
__wait_for_common+0x97/0x1b0
? __pfx_schedule_timeout+0x10/0x10
xhci_alloc_dev+0x9e/0x2b0
usb_alloc_dev+0x7a/0x3b0
hub_port_connect+0x285/0x960
hub_port_connect_change+0x94/0x290
port_event+0x4bb/0x840
hub_event+0x141/0x460
process_one_work+0x196/0x390
worker_thread+0x1af/0x320
? __pfx_worker_thread+0x10/0x10
kthread+0xe3/0x120
? __pfx_kthread+0x10/0x10
ret_from_fork+0x199/0x260
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
INFO: task systemd-shutdow:1 blocked for more than 122 seconds.
Not tainted 7.0.0-clean #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-shutdow state:D stack:0 pid:1 tgid:1 ppid:0 task_flags:0x400100 flags:0x00080000
Call Trace:
<TASK>
__schedule+0x299/0x5c0
schedule+0x27/0x80
schedule_preempt_disabled+0x15/0x30
__mutex_lock.constprop.0+0x547/0xac0
device_shutdown+0xac/0x1b0
kernel_restart+0x3a/0x70
__do_sys_reboot+0x147/0x240
do_syscall_64+0x11b/0x6a0
? handle_mm_fault+0x110/0x350
? do_user_addr_fault+0x206/0x680
? irqentry_exit+0x7a/0x4d0
entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7fe2958da917
RSP: 002b:00007ffc5c458618 EFLAGS: 00000206 ORIG_RAX: 00000000000000a9
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fe2958da917
RDX: 0000000001234567 RSI: 0000000028121969 RDI: 00000000fee1dead
RBP: 00007ffc5c458790 R08: 0000000000000069 R09: 00000000ffffffff
R10: 0000000000000000 R11: 0000000000000206 R12: 0000000000000000
R13: 0000000000000000 R14: 00007ffc5c4588b8 R15: 0000000000000000
</TASK>
INFO: task systemd-shutdow:1 is blocked on a mutex likely owned by task kworker/0:1:11.--- drivers/iommu/intel/iommu.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
drivers/iommu/intel/iommu.c +6 -4
@@ -1446,7 +1446,7 @@ static int copy_context_table(struct intel_iommu *iommu,struct context_entry **tbl, int bus, bool ext) { - int tbl_idx, pos = 0, idx, devfn, ret = 0, did; + int tbl_idx, tbl_slot = 0, idx, devfn, ret = 0, did;struct context_entry *new_ce =NULL , ce;struct context_entry *old_ce =NULL ;struct root_entry re;
@@ -1462,10 +1462,9 @@ static int copy_context_table(struct intel_iommu *iommu,if (idx ==0 ) {/* First save what we may have and clean up */ if (new_ce) { - tbl[tbl_idx] = new_ce; + tbl[tbl_idx + tbl_slot] = new_ce;__iommu_flush_cache (iommu, new_ce,VTD_PAGE_SIZE ); - pos =1 ; }if (old_ce)
@@ -1487,6 +1486,9 @@ static int copy_context_table(struct intel_iommu *iommu, } } +/* Track if saving UCTP or LCTP entries in scalable mode */ + tbl_slot = ext && devfn >=0x80 ?1 :0 ; + ret = -ENOMEM ; old_ce =memremap (old_ce_phys,PAGE_SIZE ,MEMREMAP_WB );
@@ -1515,7 +1517,7 @@ static int copy_context_table(struct intel_iommu *iommu, new_ce[idx] = ce; } - tbl[tbl_idx + pos] = new_ce; + tbl[tbl_idx + tbl_slot] = new_ce;__iommu_flush_cache (iommu, new_ce,VTD_PAGE_SIZE );
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 2/19 iommu/vt-d: Use logical OR operator for privilege mode check
In reply to Lu BaoluFrom: Li RongQing <[email protected]>
Replace bitwise OR (|) with logical OR (||) in the privilege mode validation check. While both operators produce the same result for boolean values (0 or 1), using logical OR is semantically correct and makes the intent clearer.
No functional change, but improves code readability.
--- drivers/iommu/intel/prq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/iommu/intel/prq.c +1 -1
@@ -223,7 +223,7 @@ static irqreturn_t prq_event_thread(int irq, void *d) goto prq_advance; } -if (unlikely (req->pm_req && (req->rd_req | req->wr_req))) { +if (unlikely (req->pm_req && (req->rd_req || req->wr_req))) {pr_err ("IOMMU: %s: Page request in Privilege Mode\n" , iommu->name); goto bad_req;
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 3/19 iommu/vt-d: Fix CACHE_TAG_NESTING_DEVTLB polluting shared variables in flush loop
In reply to Lu BaoluFrom: Guanghui Feng <[email protected]>
In cache_tag_flush_range(), the CACHE_TAG_NESTING_DEVTLB case modifies the shared local variables 'addr' and 'mask' before falling through to CACHE_TAG_DEVTLB. This causes all subsequent CACHE_TAG_DEVTLB entries in the same loop iteration to incorrectly use the full-range flush parameters (addr=0, mask=MAX_AGAW_PFN_WIDTH) instead of the precisely calculated PSI range. This is not the intended behavior, as regular DEVTLB entries should always perform targeted range-based invalidation.
Fix this by having CACHE_TAG_NESTING_DEVTLB directly call cache_tag_flush_devtlb_psi() with the full-range constants and break, instead of modifying shared variables and falling through. This ensures CACHE_TAG_DEVTLB always uses the original calculated addr and mask for precise range flush.
This change slightly affects trace_cache_tag_flush_range() behavior. Previously, after addr/mask were overwritten, the tracepoint could record a full-range flush even when the caller requested a narrower range.
The tracepoint should reflect caller intent. Although this helper may widen the actual hardware invalidation range for implementation reasons, that does not change what the caller requested, so logging the requested range is the correct behavior.
If the actual invalidation range sent to hardware is needed, it is already visible via the qi_submit trace event, which records the invalidation descriptors emitted by the driver.
--- drivers/iommu/intel/cache.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/iommu/intel/cache.c +2 -3
@@ -454,9 +454,8 @@ void cache_tag_flush_range(struct dmar_domain *domain, unsigned long start,* affected by a change in S2. So just flush the entire * device cache. */ - addr =0 ; - mask =MAX_AGAW_PFN_WIDTH ; - fallthrough; +cache_tag_flush_devtlb_psi (domain, tag,0 ,MAX_AGAW_PFN_WIDTH ); +break ;case CACHE_TAG_DEVTLB :cache_tag_flush_devtlb_psi (domain, tag, addr, mask);break ;
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 4/19 iommu/vt-d: Use kstrtoint_from_user() in dmar_perf_latency_write()
In reply to Lu BaoluFrom: Dmitry Antipov <[email protected]>
Simplify 'dmar_perf_latency_write()' by using the convenient 'kstrtoint_from_user()'.
--- drivers/iommu/intel/debugfs.c | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-)
drivers/iommu/intel/debugfs.c +4 -12
@@ -690,19 +690,11 @@ static ssize_t dmar_perf_latency_write(struct file *filp, {struct dmar_drhd_unit *drhd;struct intel_iommu *iommu; -int counting; -char buf[64 ]; +int ret, counting; -if (cnt >63 ) - cnt =63 ; - -if (copy_from_user (&buf, ubuf, cnt)) -return -EFAULT ; - - buf[cnt] =0 ; - -if (kstrtoint (buf,0 , &counting)) -return -EINVAL ; + ret =kstrtoint_from_user (ubuf, cnt,0 , &counting); +if (ret) +return ret;switch (counting) {case 0 :
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 5/19 iommu/vt-d: Fix no_iommu to disable platform opt-in
In reply to Lu BaoluFrom: Kevin Tian <[email protected]>
If user explicitly requests to disable iommu (via "iommu=off" or "intel_iommu=off"), there is no reason to force enabling it due to platform opt-in (for external-facing devices). User should be aware of any security implication of doing so.
"intel_iommu=off" implements this policy by setting no_platform_optin to skip platform opt-in in platform_optin_force_iommu().
However, "iommu=off" (no_iommu=1) doesn't set no_platform_optin hence is broken in this aspect:
- detect_intel_iommu() doesn't request ACS if no_iommu=1
- platform_optin_force_iommu() forces iommu on if external-facing devices exist and no_platform_optin is not set
This leads to a bad configuration with ACS disabled while DMA remapping is enabled.
Instead of setting no_platform_optin (will soon be removed) for no_iommu=1, directly check no_iommu in platform_optin_force_iommu().
--- drivers/iommu/intel/iommu.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/iommu/intel/iommu.c +3 -3
@@ -2484,10 +2484,11 @@ static bool has_external_pci(void)static int __init platform_optin_force_iommu (void ) { -if (!dmar_platform_optin () || no_platform_optin || !has_external_pci ()) +if (no_iommu || !dmar_platform_optin () || no_platform_optin || + !has_external_pci ())return 0 ; -if (no_iommu || dmar_disabled) +if (dmar_disabled)pr_info ("Intel-IOMMU force enabled due to platform opt in\n" );/*
@@ -2498,7 +2499,6 @@ static int __init platform_optin_force_iommu(void)iommu_set_default_passthrough (false); dmar_disabled =0 ; - no_iommu =0 ;return 1 ; }
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 6/19 iommu/vt-d: Force requesting ACS when tboot is enabled
In reply to Lu BaoluFrom: Kevin Tian <[email protected]>
Currently the conditions of requesting ACS in detect_intel_iommu() don't include tboot, leading to a possible misconfiguration with ACS disabled (e.g. due to user opts) while iommu is later forced on by tboot_force_iommu().
Fix it by checking tboot in detect_intel_iommu().
--- drivers/iommu/intel/iommu.h | 2 ++ drivers/iommu/intel/dmar.c | 15 +++++++++++++-- drivers/iommu/intel/iommu.c | 2 +- 3 files changed, 16 insertions(+), 3 deletions(-)
drivers/iommu/intel/iommu.h +2 -0
@@ -1354,6 +1354,7 @@ static inline bool ecmd_has_pmu_essential(struct intel_iommu *iommu)extern int dmar_disabled;extern int intel_iommu_enabled; +extern int intel_iommu_tboot_noforce;#else static inline int iommu_calculate_agaw (struct intel_iommu *iommu) {
@@ -1366,6 +1367,7 @@ static inline int iommu_calculate_max_sagaw(struct intel_iommu *iommu)#define dmar_disabled (1)#define intel_iommu_enabled (0)#define intel_iommu_sm (0) +#define intel_iommu_tboot_noforce (0)#endif static inline const char *decode_prq_descriptor(char *str, size_t size,
drivers/iommu/intel/dmar.c +13 -2
@@ -915,6 +915,18 @@ dmar_validate_one_drhd(struct acpi_dmar_header *entry, void *arg)return 0 ; } +static bool dmar_required (void ) +{ +/* tboot supersedes any user/platform opt */ +if (!intel_iommu_tboot_noforce &&tboot_enabled ()) +return true; + +if (!no_iommu && (!dmar_disabled ||dmar_platform_optin ())) +return true; + +return false; +} +void __init detect_intel_iommu (void ) {int ret;
@@ -928,8 +940,7 @@ void __init detect_intel_iommu(void)if (!ret) ret =dmar_walk_dmar_table ((struct acpi_table_dmar *)dmar_tbl, &validate_drhd_cb); -if (!ret && !no_iommu && !iommu_detected && - (!dmar_disabled ||dmar_platform_optin ())) { +if (!ret && !iommu_detected &&dmar_required ()) { iommu_detected =1 ;/* Make sure ACS will be enabled */ pci_request_acs ();
drivers/iommu/intel/iommu.c +1 -1
@@ -57,7 +57,7 @@ static int rwbf_quirk;* (used when kernel is launched w/ TXT) */ static int force_on =0 ; -static int intel_iommu_tboot_noforce; +int intel_iommu_tboot_noforce;static int no_platform_optin;#define ROOT_ENTRY_NR (VTD_PAGE_SIZE/sizeof(struct root_entry))
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 7/19 iommu/vt-d: Remove dead code when CONFIG_INTEL_IOMMU is not set
In reply to Lu BaoluFrom: Kevin Tian <[email protected]>
Those are leftovers and unreachable now: the entire intel directory is built only when CONFIG_INTEL_IOMMU is set.
--- drivers/iommu/intel/iommu.h | 15 --------------- 1 file changed, 15 deletions(-)
drivers/iommu/intel/iommu.h +0 -15
@@ -1340,7 +1340,6 @@ static inline bool intel_domain_is_ss_paging(struct dmar_domain *domain)return domain->domain.ops == &intel_ss_paging_domain_ops; } -#ifdef CONFIG_INTEL_IOMMU extern int intel_iommu_sm;int iommu_calculate_agaw (struct intel_iommu *iommu);int iommu_calculate_max_sagaw (struct intel_iommu *iommu);
@@ -1355,20 +1354,6 @@ static inline bool ecmd_has_pmu_essential(struct intel_iommu *iommu)extern int dmar_disabled;extern int intel_iommu_enabled;extern int intel_iommu_tboot_noforce; -#else -static inline int iommu_calculate_agaw (struct intel_iommu *iommu) -{ -return 0 ; -} -static inline int iommu_calculate_max_sagaw (struct intel_iommu *iommu) -{ -return 0 ; -} -#define dmar_disabled (1) -#define intel_iommu_enabled (0) -#define intel_iommu_sm (0) -#define intel_iommu_tboot_noforce (0) -#endif static inline const char *decode_prq_descriptor (char *str,size_t size,u64 dw0,u64 dw1,u64 dw2,u64 dw3)
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 8/19 iommu/vt-d: Consolidate dmar policy management and force_on logic
In reply to Lu BaoluFrom: Kevin Tian <[email protected]>
Currently the dmar on/off is carried by multiple variables (no_iommu, dmar_disabled, no_platform_optin, etc.) with error-prone force_on logic scattered in multiple places.
Unify/centralize the policy/priority management for various force_on scenarios.
No functional impact except one case - "intel_iommu=off" sets no_platform_optin which is checked in platform_optin_force_iommu() but not in detect_intel_iommu(), leading to ACS unnecessarily requested when iommu could not be forced on later. Now with the unified logic this becomes more consistent.
--- drivers/iommu/intel/iommu.h | 45 ++++++++++++++++++++++++++++ drivers/iommu/intel/dmar.c | 58 ++++++++++++++++++++++++++++++++++--- drivers/iommu/intel/iommu.c | 7 +++++ 3 files changed, 106 insertions(+), 4 deletions(-)
drivers/iommu/intel/iommu.h +45 -0
@@ -1351,6 +1351,51 @@ static inline bool ecmd_has_pmu_essential(struct intel_iommu *iommu)DMA_ECMD_ECCAP3_ESSENTIAL ; } +enum dmar_force_on { +DMAR_FORCEON_PLATFORM , +DMAR_FORCEON_TBOOT +}; + +/* +* On policies are positive, with more positive value being stronger. +* Off policies are negative, with more negative value being stronger. +* +* 'dmar' here refers to DMA remapping instead of the dmar/iommu unit. +* +* - DMAR_FORCE_ON: +* force to turn on (e.g. by tboot or platform opt-in). +* +* - DMAR_ON: +* turn on by build configuration (CONFIG_INTEL_IOMMU_DEFAULT_ON=on) +* or user opts ("intel_iommu=on"). +* +* - DMAR_DEFAULT_OFF +* turn off by build configuration (CONFIG_INTEL_IOMMU_DEFAULT_ON=off). +* +* - DMAR_USER_OFF +* turn off by user opts ("intel_iommu=off" or "iommu=off"). +* +* - '0' is invalid, compared to decide the on/off policy +* +*/ +#define DMAR_FORCE_ON 2 +#define DMAR_ON 1 +#define DMAR_DEFAULT_OFF -1 +#define DMAR_USER_OFF -2 +extern int dmar_policy; + +static inline bool dmar_policy_on (void ) +{ +return dmar_policy >0 ; +} + +static inline bool dmar_policy_off (void ) +{ +return dmar_policy <0 ; +} + +bool dmar_can_force_on (enum dmar_force_on force_on); +extern int dmar_disabled;extern int intel_iommu_enabled;extern int intel_iommu_tboot_noforce;
drivers/iommu/intel/dmar.c +54 -4
74 lines not on this page · raw message
drivers/iommu/intel/iommu.c +7 -0
@@ -199,6 +199,11 @@ static LIST_HEAD(dmar_satc_units);static void intel_iommu_domain_free (struct iommu_domain *domain); +#ifdef CONFIG_INTEL_IOMMU_DEFAULT_ON +int dmar_policy =DMAR_ON ; +#else +int dmar_policy =DMAR_DEFAULT_OFF ; +#endif int dmar_disabled = !IS_ENABLED (CONFIG_INTEL_IOMMU_DEFAULT_ON );int intel_iommu_sm =IS_ENABLED (CONFIG_INTEL_IOMMU_SCALABLE_MODE_DEFAULT_ON );
@@ -240,9 +245,11 @@ static int __init intel_iommu_setup(char *str)while (*str) {if (!strncmp (str,"on" ,2 )) { + dmar_policy =DMAR_ON ; dmar_disabled =0 ;pr_info ("IOMMU enabled\n" ); }else if (!strncmp (str,"off" ,3 )) { + dmar_policy =DMAR_USER_OFF ; dmar_disabled =1 ; no_platform_optin =1 ;pr_info ("IOMMU disabled\n" );
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 9/19 iommu/vt-d: Use dmar_can_force_on() for platform opt-in
In reply to Lu BaoluFrom: Kevin Tian <[email protected]>
So the policy of requesting ACS in detect_intel_iommu() is consistent with that in platform_optin_force_iommu().
While at it, remove no_platform_optin which is unnecessary now.
--- drivers/iommu/intel/iommu.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
drivers/iommu/intel/iommu.c +8 -7
@@ -58,7 +58,6 @@ static int rwbf_quirk;*/ static int force_on =0 ;int intel_iommu_tboot_noforce; -static int no_platform_optin;#define ROOT_ENTRY_NR (VTD_PAGE_SIZE/sizeof(struct root_entry))
@@ -251,7 +250,6 @@ static int __init intel_iommu_setup(char *str) }else if (!strncmp (str,"off" ,3 )) { dmar_policy =DMAR_USER_OFF ; dmar_disabled =1 ; - no_platform_optin =1 ;pr_info ("IOMMU disabled\n" ); }else if (!strncmp (str,"igfx_off" ,8 )) { disable_igfx_iommu =1 ;
@@ -2491,20 +2489,23 @@ static bool has_external_pci(void)static int __init platform_optin_force_iommu (void ) { -if (no_iommu || !dmar_platform_optin () || no_platform_optin || - !has_external_pci ()) +if (!dmar_platform_optin () || !dmar_can_force_on (DMAR_FORCEON_PLATFORM ))return 0 ; -if (dmar_disabled) -pr_info ("Intel-IOMMU force enabled due to platform opt in\n" ); +if (!has_external_pci ()) +return 0 ;/* * If Intel-IOMMU is disabled by default, we will apply identity * map for all devices except those marked as being untrusted. */ -if (dmar_disabled) +if (dmar_policy_off ()) { +pr_info ("Intel-IOMMU force enabled due to platform opt in\n" );iommu_set_default_passthrough (false); + } +/* No concurrent access to dmar_policy at this point. */ + dmar_policy =DMAR_FORCE_ON ; dmar_disabled =0 ;return 1 ;
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 10/19 iommu/vt-d: Call dmar_can_force_on() for tboot opt-in
In reply to Lu BaoluFrom: Kevin Tian <[email protected]>
So the policy of requesting ACS in detect_intel_iommu() is consistent with that in tboot_force_iommu().
Though tboot is the strongest override so far, dmar_can_force_on() may return false due to future extensions. In this case panic the kernel, as is already done when failing to initialize DMA remapping for tboot.
No functional impact at this point.
--- drivers/iommu/intel/iommu.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
drivers/iommu/intel/iommu.c +8 -4
@@ -2550,12 +2550,17 @@ static int __init probe_acpi_namespace_devices(void)static __init int tboot_force_iommu (void ) { -if (!tboot_enabled ()) +if (!tboot_enabled () || intel_iommu_tboot_noforce)return 0 ; -if (no_iommu || dmar_disabled) +if (!dmar_can_force_on (DMAR_FORCEON_TBOOT )) +panic ("tboot: Failed to force IOMMU on\n" ); + +if (dmar_policy_off ())pr_warn ("Forcing Intel-IOMMU to enabled\n" ); +/* No concurrent access to dmar_policy at this point. */ + dmar_policy =DMAR_FORCE_ON ; dmar_disabled =0 ; no_iommu =0 ;
@@ -2572,8 +2577,7 @@ int __init intel_iommu_init(void)* Intel IOMMU is required for a TXT/tboot launch or platform * opt in, so enforce that. */ - force_on = (!intel_iommu_tboot_noforce &&tboot_force_iommu ()) || -platform_optin_force_iommu (); + force_on =tboot_force_iommu () ||platform_optin_force_iommu ();down_write (&dmar_global_lock);if (dmar_table_init ()) {
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 11/19 iommu/vt-d: Remove the 'force_on' variable
In reply to Lu BaoluFrom: Kevin Tian <[email protected]>
The force_on variable is now redundant - same information captured by "dmar_policy == DMAR_FORCE_ON". Replace all force_on checks with dmar_policy_force_on().
--- drivers/iommu/intel/iommu.h | 5 +++++ drivers/iommu/intel/iommu.c | 37 +++++++++++++++++-------------------- 2 files changed, 22 insertions(+), 20 deletions(-)
drivers/iommu/intel/iommu.h +5 -0
@@ -1394,6 +1394,11 @@ static inline bool dmar_policy_off(void)return dmar_policy <0 ; } +static inline bool dmar_policy_force_on (void ) +{ +return dmar_policy ==DMAR_FORCE_ON ; +} +bool dmar_can_force_on (enum dmar_force_on force_on);extern int dmar_disabled;
drivers/iommu/intel/iommu.c +17 -20
@@ -53,10 +53,9 @@ static int rwbf_quirk;#define rwbf_required (iommu) (rwbf_quirk || cap_rwbf((iommu)->cap))/* -* set to 1 to panic kernel if can't successfully enable VT-d -* (used when kernel is launched w/ TXT) +* Skip forcing iommu on and avoid tboot-related kernel panics during +* initialization when set to 1 (via intel_iommu=tboot_noforce). */ -static int force_on =0 ;int intel_iommu_tboot_noforce;#define ROOT_ENTRY_NR (VTD_PAGE_SIZE/sizeof(struct root_entry))
@@ -1713,7 +1712,7 @@ static int __init init_dmars(void)* we always have to disable PMRs or DMA may fail on * this device */ -if (force_on) +if (dmar_policy_force_on ())iommu_disable_protect_mem_regions (iommu);continue ; }
@@ -1805,7 +1804,7 @@ static int init_iommu_hw(void)* we always have to disable PMRs or DMA may fail on * this device */ -if (force_on) +if (dmar_policy_force_on ())iommu_disable_protect_mem_regions (iommu);continue ; }
@@ -1866,7 +1865,7 @@ static void iommu_resume(void *data)unsigned long flag;if (init_iommu_hw ()) { -if (force_on) +if (dmar_policy_force_on ())panic ("tboot: IOMMU setup failed, DMAR can not resume!\n" );else WARN (1 ,"IOMMU setup failed, DMAR can not resume!\n" );
@@ -2134,7 +2133,7 @@ static int intel_iommu_add(struct dmar_drhd_unit *dmaru)/* * we always have to disable PMRs or DMA may fail on this device */ -if (force_on) +if (dmar_policy_force_on ())iommu_disable_protect_mem_regions (iommu);return 0 ; }
@@ -2487,13 +2486,13 @@ static bool has_external_pci(void)return false; } -static int __init platform_optin_force_iommu (void ) +static void __init platform_optin_force_iommu (void ) {if (!dmar_platform_optin () || !dmar_can_force_on (DMAR_FORCEON_PLATFORM )) -return 0 ; +return ;if (!has_external_pci ()) -return 0 ; +return ;/* * If Intel-IOMMU is disabled by default, we will apply identity
@@ -2507,8 +2506,6 @@ static int __init platform_optin_force_iommu(void)/* No concurrent access to dmar_policy at this point. */ dmar_policy =DMAR_FORCE_ON ; dmar_disabled =0 ; - -return 1 ; }static int __init probe_acpi_namespace_devices (void )
@@ -2548,10 +2545,10 @@ static int __init probe_acpi_namespace_devices(void)return 0 ; } -static __init int tboot_force_iommu (void ) +static __init void tboot_force_iommu (void ) {if (!tboot_enabled () || intel_iommu_tboot_noforce) -return 0 ; +return ;if (!dmar_can_force_on (DMAR_FORCEON_TBOOT ))panic ("tboot: Failed to force IOMMU on\n" );
@@ -2563,8 +2560,6 @@ static __init int tboot_force_iommu(void) dmar_policy =DMAR_FORCE_ON ; dmar_disabled =0 ; no_iommu =0 ; - -return 1 ; }int __init intel_iommu_init (void )
@@ -2577,17 +2572,19 @@ int __init intel_iommu_init(void)* Intel IOMMU is required for a TXT/tboot launch or platform * opt in, so enforce that. */ - force_on =tboot_force_iommu () ||platform_optin_force_iommu (); +tboot_force_iommu (); +if (!dmar_policy_force_on ()) +platform_optin_force_iommu ();down_write (&dmar_global_lock);if (dmar_table_init ()) { -if (force_on) +if (dmar_policy_force_on ())panic ("tboot: Failed to initialize DMAR table\n" ); goto out_free_dmar; }if (dmar_dev_scope_init () <0 ) { -if (force_on) +if (dmar_policy_force_on ())panic ("tboot: Failed to initialize DMAR device scope\n" ); goto out_free_dmar; }
@@ -2641,7 +2638,7 @@ int __init intel_iommu_init(void) ret =init_dmars ();if (ret) { -if (force_on) +if (dmar_policy_force_on ())panic ("tboot: Failed to initialize DMARs\n" );pr_err ("Initialization failed\n" ); goto out_free_dmar;
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 12/19 iommu/vt-d: Remove dmar_disabled
In reply to Lu BaoluFrom: Kevin Tian <[email protected]>
It's replaced by dmar_policy_off() now, covering both "iommu=off" and "intel_iommu=off". Also remove unnecessary checks on no_iommu, leaving only one exception in intel_iommu_init() which skips debugfs init for "iommu=off" but not "intel_iommu=off". Keep it to avoid surprise for now.
--- drivers/iommu/intel/iommu.h | 1 - drivers/iommu/intel/iommu.c | 9 ++------- drivers/iommu/intel/svm.c | 2 +- 3 files changed, 3 insertions(+), 9 deletions(-)
drivers/iommu/intel/iommu.h +0 -1
@@ -1401,7 +1401,6 @@ static inline bool dmar_policy_force_on(void)bool dmar_can_force_on (enum dmar_force_on force_on); -extern int dmar_disabled;extern int intel_iommu_enabled;extern int intel_iommu_tboot_noforce;
drivers/iommu/intel/iommu.c +2 -7
@@ -202,7 +202,6 @@ int dmar_policy = DMAR_ON;#else int dmar_policy =DMAR_DEFAULT_OFF ;#endif -int dmar_disabled = !IS_ENABLED (CONFIG_INTEL_IOMMU_DEFAULT_ON );int intel_iommu_sm =IS_ENABLED (CONFIG_INTEL_IOMMU_SCALABLE_MODE_DEFAULT_ON );int intel_iommu_enabled =0 ;
@@ -244,11 +243,9 @@ static int __init intel_iommu_setup(char *str)while (*str) {if (!strncmp (str,"on" ,2 )) { dmar_policy =DMAR_ON ; - dmar_disabled =0 ;pr_info ("IOMMU enabled\n" ); }else if (!strncmp (str,"off" ,3 )) { dmar_policy =DMAR_USER_OFF ; - dmar_disabled =1 ;pr_info ("IOMMU disabled\n" ); }else if (!strncmp (str,"igfx_off" ,8 )) { disable_igfx_iommu =1 ;
@@ -2371,7 +2368,7 @@ void intel_iommu_shutdown(void)struct dmar_drhd_unit *drhd;struct intel_iommu *iommu =NULL ; -if (no_iommu || dmar_disabled) +if (dmar_policy_off ())return ;/*
@@ -2505,7 +2502,6 @@ static void __init platform_optin_force_iommu(void)/* No concurrent access to dmar_policy at this point. */ dmar_policy =DMAR_FORCE_ON ; - dmar_disabled =0 ; }static int __init probe_acpi_namespace_devices (void )
@@ -2558,7 +2554,6 @@ static __init void tboot_force_iommu(void)/* No concurrent access to dmar_policy at this point. */ dmar_policy =DMAR_FORCE_ON ; - dmar_disabled =0 ; no_iommu =0 ; }
@@ -2602,7 +2597,7 @@ int __init intel_iommu_init(void)if (!no_iommu)intel_iommu_debugfs_init (); -if (no_iommu || dmar_disabled) { +if (dmar_policy_off ()) {/* * We exit the function here to ensure IOMMU's remapping and * mempool aren't setup, which means that the IOMMU's PMRs
drivers/iommu/intel/svm.c +1 -1
@@ -115,7 +115,7 @@ static int intel_iommu_sva_supported(struct device *dev)struct device_domain_info *info =dev_iommu_priv_get (dev);struct intel_iommu *iommu; -if (!info || dmar_disabled) +if (!info ||dmar_policy_off ())return -EINVAL ; iommu = info->iommu;
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 13/19 iommu/vt-d: Support the new DMA_REMAP_OPT_OUT flag bit
In reply to Lu BaoluFrom: Kevin Tian <[email protected]>
Some BIOS already provides config options to expose/hide VT-d units as a whole to/from system software. A new demand is to allow exposing VT-d units but requesting system software to disable DMA remapping while sustaining interrupt remapping. This can be communicated now by setting the new DMA_REMAP_OPT_OUT flag bit in the DMAR table, as introduced in VT-d spec v5.2 (section 8.1, DMA Remapping Reporting Structure).
Introduce a new off policy (DMAR_FW_OFF) for DMA_REMAP_OPT_OUT. As the strongest off policy, it cannot be overridden by user opts or any force_on types. If tboot is enabled in the meantime, kernel will panic. It is user responsibility to configure BIOS properly.
One cleanup is left for future - the DMAR flag is parsed multiple times, in detect_intel_iommu(), dmar_platform_optin() (which can be called at run-time), etc. Caching it is a cleaner way.
--- drivers/iommu/intel/iommu.h | 4 ++++ include/linux/dmar.h | 1 + drivers/iommu/intel/dmar.c | 34 ++++++++++++++++++++++++---------- 3 files changed, 29 insertions(+), 10 deletions(-)
drivers/iommu/intel/iommu.h +4 -0
@@ -1375,6 +1375,9 @@ enum dmar_force_on {* - DMAR_USER_OFF * turn off by user opts ("intel_iommu=off" or "iommu=off"). * +* - DMAR_FW_OFF +* turn off due to firmware opt-out (DMAR_REMAP_OPT_OUT) +* * - '0' is invalid, compared to decide the on/off policy * */
@@ -1382,6 +1385,7 @@ enum dmar_force_on {#define DMAR_ON 1#define DMAR_DEFAULT_OFF -1#define DMAR_USER_OFF -2 +#define DMAR_FW_OFF -3extern int dmar_policy;static inline bool dmar_policy_on (void )
include/linux/dmar.h +1 -0
@@ -24,6 +24,7 @@ struct acpi_dmar_header;#define DMAR_INTR_REMAP 0x1#define DMAR_X2APIC_OPT_OUT 0x2#define DMAR_PLATFORM_OPT_IN 0x4 +#define DMAR_REMAP_OPT_OUT 0x8struct intel_iommu ;
drivers/iommu/intel/dmar.c +24 -10
@@ -930,7 +930,9 @@ dmar_validate_one_drhd(struct acpi_dmar_header *entry, void *arg)* * - DMAR_FORCEON_TBOOT: tboot strictly requires DMA remapping for secure * boot hence supersedes any user opts ("iommu=off" or "intel_iommu=off") -* and weaker off policies. +* and weaker off policies. But if firmware forces DMA remapping off (by +* setting DMAR_REMAP_OPT_OUT in the DMAR table), no force_on is allowed. +* Firmware settings must be changed to unblock tboot. * * - DMAR_FORCEON_PLATFORM: external-facing devices requires DMA * remapping to prevent malicious downstream external devices from
@@ -939,6 +941,7 @@ dmar_validate_one_drhd(struct acpi_dmar_header *entry, void *arg)* * In a nutshell, "trusted boot environment" is considered stronger than * "user choices", which in turn is stronger than "platform opt-in hint". +* But they are all meaningless when it's forced off by "firmware". */ bool dmar_can_force_on (enum dmar_force_on force_on) {
@@ -976,31 +979,42 @@ static bool dmar_required(void)void __init detect_intel_iommu (void ) { -int ret;struct dmar_res_callback validate_drhd_cb = { .cb[ACPI_DMAR_TYPE_HARDWARE_UNIT ] = &dmar_validate_one_drhd, .ignore_unhandled = true, }; +struct acpi_table_dmar *dmar; +int ret;down_write (&dmar_global_lock);if (no_iommu) dmar_policy =DMAR_USER_OFF ; ret =dmar_table_detect (); -if (!ret) - ret =dmar_walk_dmar_table ((struct acpi_table_dmar *)dmar_tbl, - &validate_drhd_cb); -if (!ret && !iommu_detected &&dmar_required ()) { +if (!ret) { + dmar = (struct acpi_table_dmar *)dmar_tbl; + ret =dmar_walk_dmar_table (dmar, &validate_drhd_cb); + } + +if (ret) + goto out; + +if (dmar->flags &DMAR_REMAP_OPT_OUT ) { + dmar_policy =DMAR_FW_OFF ; +pr_info ("Firmware forces DMA remapping off\n" ); +pr_info ("Any user opt or tboot/platform force_on will be ignored\n" ); + } + +if (!iommu_detected &&dmar_required ()) { iommu_detected =1 ;/* Make sure ACS will be enabled */ pci_request_acs (); } -if (!ret) { - x86_init.iommu.iommu_init = intel_iommu_init; - x86_platform.iommu_shutdown = intel_iommu_shutdown; - } + x86_init.iommu.iommu_init = intel_iommu_init; + x86_platform.iommu_shutdown = intel_iommu_shutdown; +out:if (dmar_tbl) {acpi_put_table (dmar_tbl); dmar_tbl =NULL ;
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 14/19 iommu/vt-d: Cache max domain ID to avoid redundant calculation
In reply to Lu BaoluThe cap_ndoms() helper calculates the maximum available domain ID from the value of capability register, which can be inefficient if called repeatedly. Cache the maximum supported domain ID in max_domain_id field during initialization to avoid redundant calls to cap_ndoms() throughout the IOMMU driver.
No functionality change.
--- drivers/iommu/intel/iommu.h | 1 + drivers/iommu/intel/dmar.c | 1 + drivers/iommu/intel/iommu.c | 10 +++++----- 3 files changed, 7 insertions(+), 5 deletions(-)
drivers/iommu/intel/iommu.h +1 -0
@@ -700,6 +700,7 @@ struct intel_iommu {/* mutex to protect domain_ida */ struct mutex did_lock;struct ida domain_ida;/* domain id allocator */ +unsigned long max_domain_id;unsigned long *copied_tables;/* bitmap of copied tables */ spinlock_t lock;/* protect context, domain ids */ struct root_entry *root_entry;/* virtual address */
drivers/iommu/intel/dmar.c +1 -0
@@ -1174,6 +1174,7 @@ static int alloc_iommu(struct dmar_drhd_unit *drhd)spin_lock_init (&iommu->lock);ida_init (&iommu->domain_ida);mutex_init (&iommu->did_lock); + iommu->max_domain_id =cap_ndoms (iommu->cap); ver =readl (iommu->reg +DMAR_VER_REG ); pr_info("%s: reg_base_addr %llx ver %d:%d cap %llx ecap %llx\n",
drivers/iommu/intel/iommu.c +5 -5
@@ -1047,7 +1047,7 @@ int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu) } num =ida_alloc_range (&iommu->domain_ida,IDA_START_DID , -cap_ndoms (iommu->cap) -1 ,GFP_KERNEL ); + iommu->max_domain_id -1 ,GFP_KERNEL );if (num <0 ) {pr_err ("%s: No free domain ids\n" , iommu->name); goto err_unlock;
@@ -1111,7 +1111,7 @@ static void copied_context_tear_down(struct intel_iommu *iommu, did_old =context_domain_id (context);context_clear_entry (context); -if (did_old <cap_ndoms (iommu->cap)) { +if (did_old < iommu->max_domain_id) { iommu->flush.flush_context(iommu, did_old, PCI_DEVID(bus, devfn), DMA_CCMD_MASK_NOBIT,
@@ -1511,7 +1511,7 @@ static int copy_context_table(struct intel_iommu *iommu,continue ; did =context_domain_id (&ce); -if (did >=0 && did <cap_ndoms (iommu->cap)) +if (did >=0 && did < iommu->max_domain_id)ida_alloc_range (&iommu->domain_ida, did, did,GFP_KERNEL );set_context_copied (iommu, bus, devfn);
@@ -2431,7 +2431,7 @@ static ssize_t domains_supported_show(struct device *dev, struct device_attribute *attr, char *buf) { struct intel_iommu *iommu = dev_to_intel_iommu(dev); -return sysfs_emit (buf,"%ld\n" ,cap_ndoms (iommu->cap)); +return sysfs_emit (buf,"%ld\n" , iommu->max_domain_id); }static DEVICE_ATTR_RO (domains_supported);
@@ -2442,7 +2442,7 @@ static ssize_t domains_used_show(struct device *dev,unsigned int count =0 ;int id; -for (id =0 ; id <cap_ndoms (iommu->cap); id++) +for (id =0 ; id < iommu->max_domain_id; id++)if (ida_exists (&iommu->domain_ida, id)) count++;
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 15/19 iommu/vt-d: Fix copied_tables bitmap leak on error in copy_translation_tables
In reply to Lu BaoluFrom: ZhaoJinming <[email protected]>
The iommu->copied_tables bitmap was introduced by the IOMMU live update series to track which context entries have been copied from the previous kernel. The allocation via bitmap_zalloc() was added inside copy_translation_tables(), but the error paths were not updated to free it:
1. When old_rt_phys is 0 (invalid root table address)
2. When memremap(old_rt_phys) fails
3. When kcalloc for ctxt_tbls fails (goto out_unmap, which only unmaps old_rt without releasing the bitmap)
The bitmap is only cleaned up by free_dmar_iommu(), which is called from the free_iommu error label in init_dmars(). However, when copy_translation_tables() fails, init_dmars() does not jump to free_iommu -- it logs the error, falls through, and continues with the next IOMMU. As a result, copied_tables is leaked.
Fix this by converting the two early returns to goto a new err_free_bitmap label, and by making out_unmap fall through to it so that the bitmap is always freed on any error path. The success path performs memunmap(old_rt) inline and returns 0 directly, since copied_tables must remain allocated for subsequent use.
--- drivers/iommu/intel/iommu.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)
drivers/iommu/intel/iommu.c +13 -6
@@ -1557,12 +1557,16 @@ static int copy_translation_tables(struct intel_iommu *iommu)return -ENOMEM ; old_rt_phys = rtaddr_reg &VTD_PAGE_MASK ; -if (!old_rt_phys) -return -EINVAL ; +if (!old_rt_phys) { + ret = -EINVAL ; + goto err_free_bitmap; + } old_rt =memremap (old_rt_phys,PAGE_SIZE ,MEMREMAP_WB ); -if (!old_rt) -return -ENOMEM ; +if (!old_rt) { + ret = -ENOMEM ; + goto err_free_bitmap; + }/* This is too big for the stack - allocate it from slab */ ctxt_table_entries = ext ?512 :256 ;
@@ -1606,11 +1610,14 @@ static int copy_translation_tables(struct intel_iommu *iommu)__iommu_flush_cache (iommu, iommu->root_entry,PAGE_SIZE ); - ret =0 ; +memunmap (old_rt); +return 0 ; out_unmap:memunmap (old_rt); - +err_free_bitmap: +bitmap_free (iommu->copied_tables); + iommu->copied_tables =NULL ;return ret; }
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 16/19 iommu/vt-d: Clear Present bit before tearing down copied context entry
In reply to Lu Baolucopied_context_tear_down() zeroes the 128-bit context entry with context_clear_entry() while the Present bit is still set, and only then issues the context-cache and IOTLB invalidations. This leaves a window in which hardware can fetch a torn entry, with some fields already zeroed while Present is still set, leading to unpredictable behaviour or spurious faults. While x86 provides strong write ordering, the compiler may reorder the writes to the two 64-bit halves of the entry, and the hardware fetch is not guaranteed to be atomic with respect to multiple CPU writes.
There is no cacheline flush before the invalidation either, so on an IOMMU without coherent access to the context table the zeroed entry may not be visible to hardware at the point the invalidation is submitted.
Apply the same ownership handshake described in the VT-d spec, Section 6.5.3.3 ("Guidance to Software for Invalidations"): clear only the Present bit, flush it out to the IOMMU, perform the invalidations, and only then zero the remainder of the entry.
--- drivers/iommu/intel/iommu.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/iommu/intel/iommu.c +5 -1
@@ -1109,7 +1109,8 @@ static void copied_context_tear_down(struct intel_iommu *iommu,assert_spin_locked (&iommu->lock); did_old =context_domain_id (context); -context_clear_entry (context); +context_clear_present (context); +__iommu_flush_cache (iommu, context,sizeof (*context));if (did_old < iommu->max_domain_id) { iommu->flush.flush_context(iommu, did_old,
@@ -1120,6 +1121,9 @@ static void copied_context_tear_down(struct intel_iommu *iommu, DMA_TLB_DSI_FLUSH); } +context_clear_entry (context); +__iommu_flush_cache (iommu, context,sizeof (*context)); +clear_context_copied (iommu, bus, devfn); }
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 17/19 iommu/vt-d: Fix iopf_refcount leak on RID domain replacement
In reply to Lu Baoluintel_iommu_attach_device() enables IOPF for the new domain but never disables it for the old one. device_block_translation(), called at the start of the function, tears down translation but does not touch any IOPF state; blocking_domain_attach_dev() has to call iopf_for_domain_remove() explicitly before invoking it for exactly this reason.
identity_domain_attach_dev() has the same problem. Its comment claims that no PRI handling is needed because the device has been put in the blocking state, but the blocking state and the IOPF reference count are independent of each other.
As a result, replacing a domain that has an iopf_handler with another domain at RID level leaks a reference in info->iopf_refcount. The count never drops back to zero, so iopf_queue_remove_device() is never called and iommu_disable_pci_pri() triggers its WARN_ON(info->iopf_refcount) when the device is released.
The PASID paths already handle this correctly by way of iopf_for_domain_replace(); convert the two RID paths to do the same. Using the replace helper rather than a bare remove keeps the enable before the disable, so the reference count does not transiently reach zero and evict the device from the IOPF queue.
--- drivers/iommu/intel/iommu.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
drivers/iommu/intel/iommu.c +8 -5
@@ -3152,13 +3152,13 @@ static int intel_iommu_attach_device(struct iommu_domain *domain,if (ret)return ret; - ret =iopf_for_domain_set (domain, dev); + ret =iopf_for_domain_replace (domain, old, dev);if (ret)return ret; ret =dmar_domain_attach_device (to_dmar_domain (domain), dev);if (ret) -iopf_for_domain_remove (domain, dev); +iopf_for_domain_replace (old, domain, dev);return ret; }
@@ -3861,10 +3861,13 @@ static int identity_domain_attach_dev(struct iommu_domain *domain,return 0 ;/* -* No PRI support with the global identity domain. No need to enable or -* disable PRI in this path as the iommu has been put in the blocking -* state. +* The identity domain has no iopf_handler, so no IOPF reference is +* taken for it. The reference held by the old domain must still be +* released here; putting the device in the blocking state above does +* not affect the IOPF reference count. */ +iopf_for_domain_remove (old, dev); +if (sm_supported (iommu)) ret =intel_pasid_setup_pass_through (iommu, dev,IOMMU_NO_PASID );else
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 18/19 iommu/vt-d: Tear down scalable-mode context on probe failure
In reply to Lu Baoluintel_pasid_setup_sm_context() walks a PCI device’s DMA aliases via pci_for_each_dma_alias() and programs a scalable-mode context entry for each RID. For a device with a dma_alias_mask, the callback is invoked once for the device’s own RID and once for each alias bit, all with the same pci_dev, so device_pasid_table_setup() runs for multiple RIDs.
pci_for_each_dma_alias() stops at the first callback error. Therefore, a failure partway through the walk can leave context entries for already processed RIDs present and still pointing to the device’s PASID table.
On this error path, intel_iommu_probe_device() currently jumps directly to intel_pasid_free_table(), which frees the PASID table without first tearing down those context entries. The IOMMU may then walk a present context entry whose PASID table pointer references freed memory.
intel_iommu_release_device() already performs teardown before freeing the table. Apply the same ordering on the probe failure path.
device_pasid_table_teardown() safely handles RIDs that were never
programmed: iommu_context_addr() returns NULL when no context table has been allocated, and clearing the Present bit of an already non-present entry is a no-op. So unwind is safe for both the alias that failed and any aliases not yet reached.
--- drivers/iommu/intel/iommu.c | 1 + 1 file changed, 1 insertion(+)
drivers/iommu/intel/iommu.c +1 -0
@@ -3333,6 +3333,7 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)return &iommu->iommu; free_table: +intel_pasid_teardown_sm_context (dev);intel_pasid_free_table (dev); clear_rbtree:device_rbtree_remove (info);
Signature · 1 line
2.43.0
Lu Baolu
maintainerPATCH v2 19/19 iommu/vt-d: Flush context cache with correct SID when tearing down aliases
In reply to Lu Baoludomain_context_clear_one() and device_pasid_table_teardown() are both invoked once per DMA alias of a device. Each function locates the context entry using the bus/devfn pair provided by the pci_for_each_dma_alias() callback, then calls intel_context_flush_no_pasid(), which constructs a device-selective context-cache invalidation from info->bus and info->devfn (that is, always the requester ID of the device itself).
As a result, for every alias other than the device’s own RID, the context entry that was just cleared in memory is never invalidated in the context cache. Hardware may continue using that stale cached entry. In the scalable-mode teardown path, intel_pasid_free_table() can then free the PASID directory still referenced by that stale entry, allowing the IOMMU to walk freed memory.
Fix this by passing the source ID of the entry being torn down to intel_context_flush_no_pasid(), instead of deriving it from @info.
--- drivers/iommu/intel/iommu.h | 2 +- drivers/iommu/intel/iommu.c | 2 +- drivers/iommu/intel/pasid.c | 9 ++++++--- 3 files changed, 8 insertions(+), 5 deletions(-)
drivers/iommu/intel/iommu.h +1 -1
@@ -1241,7 +1241,7 @@ void cache_tag_flush_range_np(struct dmar_domain *domain, unsigned long start, unsigned long end);void intel_context_flush_no_pasid (struct device_domain_info *info, -struct context_entry *context,u16 did); +struct context_entry *context,u16 did,u16 sid);int intel_iommu_enable_prq (struct intel_iommu *iommu);int intel_iommu_finish_prq (struct intel_iommu *iommu);
drivers/iommu/intel/iommu.c +1 -1
@@ -1251,7 +1251,7 @@ static void domain_context_clear_one(struct device_domain_info *info, u8 bus, u8context_clear_present (context);__iommu_flush_cache (iommu, context,sizeof (*context));spin_unlock (&iommu->lock); -intel_context_flush_no_pasid (info, context, did); +intel_context_flush_no_pasid (info, context, did,PCI_DEVID (bus, devfn));context_clear_entry (context);__iommu_flush_cache (iommu, context,sizeof (*context)); }
drivers/iommu/intel/pasid.c +6 -3
@@ -751,7 +751,7 @@ static void device_pasid_table_teardown(struct device *dev, u8 bus, u8 devfn)context_clear_present (context);__iommu_flush_cache (iommu, context,sizeof (*context));spin_unlock (&iommu->lock); -intel_context_flush_no_pasid (info, context, did); +intel_context_flush_no_pasid (info, context, did,PCI_DEVID (bus, devfn));context_clear_entry (context);__iommu_flush_cache (iommu, context,sizeof (*context)); }
@@ -955,9 +955,12 @@ static void __context_flush_dev_iotlb(struct device_domain_info *info)* This helper can only be used when IOMMU is working in the legacy mode or * IOMMU is in scalable mode but all PASID table entries of the device are * non-present. +* +* @sid identifies the context entry that was modified, which may be a DMA +* alias of @info->dev rather than its own requester ID. */ void intel_context_flush_no_pasid (struct device_domain_info *info, -struct context_entry *context,u16 did) +struct context_entry *context,u16 did,u16 sid) {struct intel_iommu *iommu = info->iommu;
@@ -967,7 +970,7 @@ void intel_context_flush_no_pasid(struct device_domain_info *info,* when operating in scalable mode. Therefore the @did value doesn't * matter in scalable mode. */ - iommu->flush.flush_context (iommu, did,PCI_DEVID (info->bus, info->devfn), + iommu->flush.flush_context (iommu, did, sid,DMA_CCMD_MASK_NOBIT ,DMA_CCMD_DEVICE_INVL );/*
Signature · 1 line
2.43.0
Quoted 36 lines · Lu Baolu
Desnes Nunes (1):
iommu/vt-d: Fix UCTP context table slot when copying root entriesDmitry Antipov (1):
iommu/vt-d: Usekstrtoint_from_user()indmar_perf_latency_write()Guanghui Feng (1):
iommu/vt-d: FixCACHE_TAG_NESTING_DEVTLBpolluting shared variables in
flush loopKevin Tian (9):
iommu/vt-d: Fixno_iommuto disable platform opt-in iommu/vt-d: Force requesting ACS when tboot is enabled iommu/vt-d: Remove dead code whenCONFIG_INTEL_IOMMUis not set iommu/vt-d: Consolidate dmar policy management andforce_onlogic iommu/vt-d: Usedmar_can_force_on()for platform opt-in iommu/vt-d: Calldmar_can_force_on()for tboot opt-in iommu/vt-d: Remove the 'force_on' variable
iommu/vt-d: Removedmar_disabled
iommu/vt-d: Support the newDMA_REMAP_OPT_OUTflag bitLi RongQing (1):
iommu/vt-d: Use logical OR operator for privilege mode checkLu Baolu (5):
iommu/vt-d: Cache max domain ID to avoid redundant calculation iommu/vt-d: Clear Present bit before tearing down copied context entry iommu/vt-d: Fixiopf_refcountleak on RID domain replacement iommu/vt-d: Tear down scalable-mode context on probe failure iommu/vt-d: Flush context cache with correct SID when tearing down
aliasesZhaoJinming (1):
iommu/vt-d: Fixcopied_tablesbitmap leak on error in
copy_translation_tables
Quoted 10 lines
drivers/iommu/intel/iommu.h | 73 ++++++++++++++---- include/linux/dmar.h | 1 + drivers/iommu/intel/cache.c | 5 +- drivers/iommu/intel/debugfs.c | 16 +--- drivers/iommu/intel/dmar.c | 96 ++++++++++++++++++++--- drivers/iommu/intel/iommu.c | 139 +++++++++++++++++++--------------- drivers/iommu/intel/pasid.c | 9 ++- drivers/iommu/intel/prq.c | 2 +- drivers/iommu/intel/svm.c | 2 +- 9 files changed, 238 insertions(+), 105 deletions(-)
Applied, thanks Baolu.