Skip to contentSkip to conversation

iommu/vt-d: Turn off translations at shutdown

PATCH In review Deepa Dinamani 3 participants 6 messages Nov 10, 2019 – Nov 11, 2019 IOMMU

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 iommu/vt-d: Turn off translations at shutdown

The intel-iommu driver assumes that the iommu state is cleaned up at the start of the new kernel. But, when we try to kexec boot something other than the Linux kernel, the cleanup cannot be relied upon. Hence, cleanup before we go down for reboot.

Keeping the cleanup at initialization also, in case BIOS leaves the IOMMU enabled.

I considered turning off iommu only during kexec reboot, but a clean shutdown seems always a good idea. But if someone wants to make it conditional, such as VMM live update, we can do that. There doesn't seem to be such a condition at this time.

Tested that before, the info message
'DMAR: Translation was enabled for <iommu> but we are not in kdump mode' would be reported for each iommu. The message will not appear when the DMA-remapping is not enabled on entry to the kernel.

---
Changes since v1:
* move shutdown registration to iommu detection

 drivers/iommu/dmar.c        |  5 ++++-
 drivers/iommu/intel-iommu.c | 20 ++++++++++++++++++++
 include/linux/dmar.h        |  2 ++
 3 files changed, 26 insertions(+), 1 deletion(-)
drivers/iommu/dmar.c +4 -1
index eecd6a421667..3acfa6a25fa2 100644
@@ -895,8 +895,11 @@ int __init detect_intel_iommu(void)
 	}
 
 #ifdef CONFIG_X86
-	if (!ret)
+	if (!ret) {
 		x86_init.iommu.iommu_init = intel_iommu_init;
+		x86_platform.iommu_shutdown = intel_iommu_shutdown;
+	}
+
 #endif
 
 	if (dmar_tbl) {
drivers/iommu/intel-iommu.c +20 -0
index fe8097078669..7ac73410ba8e 100644
@@ -4764,6 +4764,26 @@ static void intel_disable_iommus(void)
 		iommu_disable_translation(iommu);
 }
 
+void intel_iommu_shutdown(void)
+{
+	struct dmar_drhd_unit *drhd;
+	struct intel_iommu *iommu = NULL;
+
+	if (no_iommu || dmar_disabled)
+		return;
+
+	down_write(&dmar_global_lock);
+
+	/* Disable PMRs explicitly here. */
+	for_each_iommu(iommu, drhd)
+		iommu_disable_protect_mem_regions(iommu);
+
+	/* Make sure the IOMMUs are switched off */
+	intel_disable_iommus();
+
+	up_write(&dmar_global_lock);
+}
+
 static inline struct intel_iommu *dev_to_intel_iommu(struct device *dev)
 {
 	struct iommu_device *iommu_dev = dev_to_iommu_device(dev);
include/linux/dmar.h +2 -0
index a7cf3599d9a1..f64ca27dc210 100644
@@ -129,6 +129,7 @@ static inline int dmar_res_noop(struct acpi_dmar_header *hdr, void *arg)
 #ifdef CONFIG_INTEL_IOMMU
 extern int iommu_detected, no_iommu;
 extern int intel_iommu_init(void);
+extern void intel_iommu_shutdown(void);
 extern int dmar_parse_one_rmrr(struct acpi_dmar_header *header, void *arg);
 extern int dmar_parse_one_atsr(struct acpi_dmar_header *header, void *arg);
 extern int dmar_check_one_atsr(struct acpi_dmar_header *hdr, void *arg);
@@ -137,6 +138,7 @@ extern int dmar_iommu_hotplug(struct dmar_drhd_unit *dmaru, bool insert);
 extern int dmar_iommu_notify_scope_dev(struct dmar_pci_notify_info *info);
 #else /* !CONFIG_INTEL_IOMMU: */
 static inline int intel_iommu_init(void) { return -ENODEV; }
+static inline void intel_iommu_shutdown(void) { }
 
 #define	dmar_parse_one_rmrr		dmar_res_noop
 #define	dmar_parse_one_atsr		dmar_res_noop
Signature · 1 line
2.17.1
#1 lore raw

Hi,

Quoted 8 lines · Deepa Dinamani

The intel-iommu driver assumes that the iommu state is cleaned up at the start of the new kernel. But, when we try to kexec boot something other than the Linux kernel, the cleanup cannot be relied upon. Hence, cleanup before we go down for reboot.

Keeping the cleanup at initialization also, in case BIOS leaves the IOMMU enabled.

Do you mind shining more light on this statement? I can't get your point here. Currently iommu_shutdown() only happens for reboot, right?

Best regards,
baolu

Quoted 90 lines

I considered turning off iommu only during kexec reboot, but a clean shutdown seems always a good idea. But if someone wants to make it conditional, such as VMM live update, we can do that. There doesn't seem to be such a condition at this time.

Tested that before, the info message
'DMAR: Translation was enabled for <iommu> but we are not in kdump mode' would be reported for each iommu. The message will not appear when the DMA-remapping is not enabled on entry to the kernel.

---
Changes since v1:
* move shutdown registration to iommu detection

  drivers/iommu/dmar.c        |  5 ++++-
  drivers/iommu/intel-iommu.c | 20 ++++++++++++++++++++
  include/linux/dmar.h        |  2 ++
  3 files changed, 26 insertions(+), 1 deletion(-)
drivers/iommu/dmar.c +4 -1
index eecd6a421667..3acfa6a25fa2 100644
@@ -895,8 +895,11 @@ int __init detect_intel_iommu(void)
  	}
 
  #ifdef CONFIG_X86
-	if (!ret)
+	if (!ret) {
  		x86_init.iommu.iommu_init = intel_iommu_init;
+		x86_platform.iommu_shutdown = intel_iommu_shutdown;
+	}
+
  #endif
 
  	if (dmar_tbl) {
drivers/iommu/intel-iommu.c +20 -0
index fe8097078669..7ac73410ba8e 100644
@@ -4764,6 +4764,26 @@ static void intel_disable_iommus(void)
  		iommu_disable_translation(iommu);
  }
 
+void intel_iommu_shutdown(void)
+{
+	struct dmar_drhd_unit *drhd;
+	struct intel_iommu *iommu = NULL;
+
+	if (no_iommu || dmar_disabled)
+		return;
+
+	down_write(&dmar_global_lock);
+
+	/* Disable PMRs explicitly here. */
+	for_each_iommu(iommu, drhd)
+		iommu_disable_protect_mem_regions(iommu);
+
+	/* Make sure the IOMMUs are switched off */
+	intel_disable_iommus();
+
+	up_write(&dmar_global_lock);
+}
+
  static inline struct intel_iommu *dev_to_intel_iommu(struct device *dev)
  {
  	struct iommu_device *iommu_dev = dev_to_iommu_device(dev);
include/linux/dmar.h +2 -0
index a7cf3599d9a1..f64ca27dc210 100644
@@ -129,6 +129,7 @@ static inline int dmar_res_noop(struct acpi_dmar_header *hdr, void *arg)
  #ifdef CONFIG_INTEL_IOMMU
  extern int iommu_detected, no_iommu;
  extern int intel_iommu_init(void);
+extern void intel_iommu_shutdown(void);
  extern int dmar_parse_one_rmrr(struct acpi_dmar_header *header, void *arg);
  extern int dmar_parse_one_atsr(struct acpi_dmar_header *header, void *arg);
  extern int dmar_check_one_atsr(struct acpi_dmar_header *hdr, void *arg);
@@ -137,6 +138,7 @@ extern int dmar_iommu_hotplug(struct dmar_drhd_unit *dmaru, bool insert);
  extern int dmar_iommu_notify_scope_dev(struct dmar_pci_notify_info *info);
  #else /* !CONFIG_INTEL_IOMMU: */
  static inline int intel_iommu_init(void) { return -ENODEV; }
+static inline void intel_iommu_shutdown(void) { }
 
  #define	dmar_parse_one_rmrr		dmar_res_noop
  #define	dmar_parse_one_atsr		dmar_res_noop
#2 lore raw
Quoted 10 lines · Lu Baolu

Hi,

Quoted 6 lines · Deepa Dinamani

The intel-iommu driver assumes that the iommu state is cleaned up at the start of the new kernel. But, when we try to kexec boot something other than the Linux kernel, the cleanup cannot be relied upon. Hence, cleanup before we go down for reboot.

Quoted 5 lines
Quoted 2 lines

Keeping the cleanup at initialization also, in case BIOS leaves the IOMMU enabled.

Do you mind shining more light on this statement? I can't get your point here. Currently iommu_shutdown() only happens for reboot, right?

In this part, I'm saying that I'm leaving intel_iommu_init() alone. I'm not taking out disabling the iommu from intel_iommu_init(). This will help when BIOS has enabled the IOMMU and for whatever reason, the kernel is booting with IOMMU off.

-Deepa
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#3 lore raw

Hi,

Quoted 14 lines · Deepa Dinamani
Quoted 13 lines · Lu Baolu

Hi,

Quoted 8 lines · Deepa Dinamani

The intel-iommu driver assumes that the iommu state is cleaned up at the start of the new kernel. But, when we try to kexec boot something other than the Linux kernel, the cleanup cannot be relied upon. Hence, cleanup before we go down for reboot.

Keeping the cleanup at initialization also, in case BIOS leaves the IOMMU enabled.

Quoted 7 lines
Quoted 2 lines

Do you mind shining more light on this statement? I can't get your point here. Currently iommu_shutdown() only happens for reboot, right?

In this part, I'm saying that I'm leaving intel_iommu_init() alone. I'm not taking out disabling the iommu from intel_iommu_init(). This will help when BIOS has enabled the IOMMU and for whatever reason, the kernel is booting with IOMMU off.

Okay, thanks for the explanation.

Quoted 3 lines

-Deepa

Best regards,
baolu

Acked-by on v2 1/1
#4 lore raw
In reply to Deepa Dinamani

Hi,

Quoted 14 lines · Deepa Dinamani

The intel-iommu driver assumes that the iommu state is cleaned up at the start of the new kernel. But, when we try to kexec boot something other than the Linux kernel, the cleanup cannot be relied upon. Hence, cleanup before we go down for reboot.

Keeping the cleanup at initialization also, in case BIOS leaves the IOMMU enabled.

I considered turning off iommu only during kexec reboot, but a clean shutdown seems always a good idea. But if someone wants to make it conditional, such as VMM live update, we can do that. There doesn't seem to be such a condition at this time.

Quoted 6 lines

Tested that before, the info message
'DMAR: Translation was enabled for <iommu> but we are not in kdump mode' would be reported for each iommu. The message will not appear when the DMA-remapping is not enabled on entry to the kernel.

This patch looks good to me.

Best regards,
baolu

Quoted 71 lines

---
Changes since v1:
* move shutdown registration to iommu detection

  drivers/iommu/dmar.c        |  5 ++++-
  drivers/iommu/intel-iommu.c | 20 ++++++++++++++++++++
  include/linux/dmar.h        |  2 ++
  3 files changed, 26 insertions(+), 1 deletion(-)
drivers/iommu/dmar.c +4 -1
index eecd6a421667..3acfa6a25fa2 100644
@@ -895,8 +895,11 @@ int __init detect_intel_iommu(void)
  	}
 
  #ifdef CONFIG_X86
-	if (!ret)
+	if (!ret) {
  		x86_init.iommu.iommu_init = intel_iommu_init;
+		x86_platform.iommu_shutdown = intel_iommu_shutdown;
+	}
+
  #endif
 
  	if (dmar_tbl) {
drivers/iommu/intel-iommu.c +20 -0
index fe8097078669..7ac73410ba8e 100644
@@ -4764,6 +4764,26 @@ static void intel_disable_iommus(void)
  		iommu_disable_translation(iommu);
  }
 
+void intel_iommu_shutdown(void)
+{
+	struct dmar_drhd_unit *drhd;
+	struct intel_iommu *iommu = NULL;
+
+	if (no_iommu || dmar_disabled)
+		return;
+
+	down_write(&dmar_global_lock);
+
+	/* Disable PMRs explicitly here. */
+	for_each_iommu(iommu, drhd)
+		iommu_disable_protect_mem_regions(iommu);
+
+	/* Make sure the IOMMUs are switched off */
+	intel_disable_iommus();
+
+	up_write(&dmar_global_lock);
+}
+
  static inline struct intel_iommu *dev_to_intel_iommu(struct device *dev)
  {
  	struct iommu_device *iommu_dev = dev_to_iommu_device(dev);
include/linux/dmar.h +1 -0
index a7cf3599d9a1..f64ca27dc210 100644
@@ -129,6 +129,7 @@ static inline int dmar_res_noop(struct acpi_dmar_header *hdr, void *arg)
  #ifdef CONFIG_INTEL_IOMMU
  extern int iommu_detected, no_iommu;
  extern int intel_iommu_init(void);
+extern void intel_iommu_shutdown(void);
  extern int dmar_parse_one_rmrr(struct acpi_dmar_header *header, void *arg);
  extern int dmar_parse_one_atsr(struct acpi_dmar_header *header, void *arg);
  extern int dmar_check_one_atsr(struct acpi_dmar_header *hdr, void *arg);

include/linux/dmar.h extern int dmar_iommu_hotplug(struct dmar_drhd_unit *dmaru, bool insert); · line 144

Quoted 7 lines
include/linux/dmar.h +1 -0
@@ -137,6 +138,7 @@ extern int dmar_iommu_hotplug(struct dmar_drhd_unit *dmaru, bool insert);
  extern int dmar_iommu_notify_scope_dev(struct dmar_pci_notify_info *info);
  #else /* !CONFIG_INTEL_IOMMU: */
  static inline int intel_iommu_init(void) { return -ENODEV; }
+static inline void intel_iommu_shutdown(void) { }
 
  #define	dmar_parse_one_rmrr		dmar_res_noop
  #define	dmar_parse_one_atsr		dmar_res_noop

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

maintainer
#5 lore raw
In reply to Deepa Dinamani
Quoted 20 lines · Deepa Dinamani

The intel-iommu driver assumes that the iommu state is cleaned up at the start of the new kernel. But, when we try to kexec boot something other than the Linux kernel, the cleanup cannot be relied upon. Hence, cleanup before we go down for reboot.

Keeping the cleanup at initialization also, in case BIOS leaves the IOMMU enabled.

I considered turning off iommu only during kexec reboot, but a clean shutdown seems always a good idea. But if someone wants to make it conditional, such as VMM live update, we can do that. There doesn't seem to be such a condition at this time.

Tested that before, the info message
'DMAR: Translation was enabled for <iommu> but we are not in kdump mode' would be reported for each iommu. The message will not appear when the DMA-remapping is not enabled on entry to the kernel.

Quoted 8 lines

---
Changes since v1:
* move shutdown registration to iommu detection

 drivers/iommu/dmar.c        |  5 ++++-
 drivers/iommu/intel-iommu.c | 20 ++++++++++++++++++++
 include/linux/dmar.h        |  2 ++
 3 files changed, 26 insertions(+), 1 deletion(-)

Applied, thanks.
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu