Skip to contentSkip to conversation

iommu/vt-d: Use the SIRTP when enabling remapping

RFC In review Nathan Zimmer 3 participants 4 messages Sep 9, 2014 – Sep 10, 2014 IOMMU

lore
Share & tools
lore
#0 lore raw

RFC 1/1 iommu/vt-d: Use the SIRTP when enabling remapping

The previous change (iommu/vt-d: Don't store SIRTP request) to this area caused a crash in our simulator. In particular is seems that by the time a UART interrupt is sent through the system, we don't see interrupt remapping to be enabled. So the interrupt does not get translated to a logical interrupt and crashes.

OR'ing the SIRTP request to make sure it is seen but hopefully not sticky. This seems like a clean fix, at least on our simulator; if you don't agree, our simulator guy will take a closer look at our iommu model.

Found testing on our simulator, not real hardware.

---
 drivers/iommu/intel_irq_remapping.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/iommu/intel_irq_remapping.c +1 -1
index a872874..f586e41 100644
@@ -455,7 +455,7 @@ static void iommu_set_irq_remapping(struct intel_iommu *iommu, int mode)
 	/* Enable interrupt-remapping */
 	iommu->gcmd |= DMA_GCMD_IRE;
 	iommu->gcmd &= ~DMA_GCMD_CFI;  /* Block compatibility-format MSIs */
-	writel(iommu->gcmd, iommu->reg + DMAR_GCMD_REG);
+	writel(iommu->gcmd | DMA_GCMD_SIRTP, iommu->reg + DMAR_GCMD_REG);
 
 	IOMMU_WAIT_OP(iommu, DMAR_GSTS_REG,
 		      readl, (sts & DMA_GSTS_IRES), sts);
Signature · 1 line
1.8.2.1
#1 lore raw
Quoted 9 lines · Nathan Zimmer

The previous change (iommu/vt-d: Don't store SIRTP request) to this area caused a crash in our simulator. In particular is seems that by the time a UART interrupt is sent through the system, we don't see interrupt remapping to be enabled. So the interrupt does not get translated to a logical interrupt and crashes.

OR'ing the SIRTP request to make sure it is seen but hopefully not sticky. This seems like a clean fix, at least on our simulator; if you don't agree, our simulator guy will take a closer look at our iommu model.

Check the VT-d spec (6.7, Set Interrupt Remapping Table Pointer Operation): "Software must always follow the interrupt-remapping table pointer set operation with a global invalidate of the IEC to ensure hardware references the new structures *before* enabling interrupt remapping." There is also (command register description): "If multiple control fields in this register need to be modified, software must serialize the modifications through multiple writes to this register." This, in combination with the command registers description of bits 24 and 25 strongly suggests that your model is broken.

Quoted 2 lines

Found testing on our simulator, not real hardware.

Funnily, the original issue was found by the QEMU model of VT-d that used to take the last cited sentence very strictly (I asked to remove it due to the preexisting Linux versions).

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
#2 lore raw
Quoted 11 lines · Jan Kiszka
Quoted 9 lines · Nathan Zimmer

The previous change (iommu/vt-d: Don't store SIRTP request) to this area caused a crash in our simulator. In particular is seems that by the time a UART interrupt is sent through the system, we don't see interrupt remapping to be enabled. So the interrupt does not get translated to a logical interrupt and crashes.

OR'ing the SIRTP request to make sure it is seen but hopefully not sticky. This seems like a clean fix, at least on our simulator; if you don't agree, our simulator guy will take a closer look at our iommu model.

Quoted 9 lines

Check the VT-d spec (6.7, Set Interrupt Remapping Table Pointer Operation): "Software must always follow the interrupt-remapping table pointer set operation with a global invalidate of the IEC to ensure hardware references the new structures *before* enabling interrupt remapping." There is also (command register description): "If multiple control fields in this register need to be modified, software must serialize the modifications through multiple writes to this register." This, in combination with the command registers description of bits 24 and 25 strongly suggests that your model is broken.

Thanks, Jan. I'll read up on it. Still working out some kinks with the relatively new IOMMU/interrupt-remapping logic which we added over the last 18 months or so. Sorry for any distraction, and thanks for the help/info.
Gary

Quoted 10 lines
Quoted 3 lines

Found testing on our simulator, not real hardware.

Funnily, the original issue was found by the QEMU model of VT-d that used to take the last cited sentence very strictly (I asked to remove it due to the preexisting Linux versions).

Jan

#3 lore raw

I'm not really a kernel developer and don't know the proper etiquette for this, but just wanted to say thanks to Jan for the guidance. With your help, I was able to track our problem down to a much more fundamental problem of incorrect bit position of the fields in our VT-d GLBCMD register, attributable to our now-manual process of generating register information from Intel socket documentation. (We used to used Intel's SoftSDV to determine register fields, but that's no longer available and we have to use fallible human eyes! Thousands of registers in each new socket...)

Gary

Quoted 38 lines · Gary Kroening
Quoted 11 lines · Jan Kiszka
Quoted 9 lines · Nathan Zimmer

The previous change (iommu/vt-d: Don't store SIRTP request) to this area caused a crash in our simulator. In particular is seems that by the time a UART interrupt is sent through the system, we don't see interrupt remapping to be enabled. So the interrupt does not get translated to a logical interrupt and crashes.

OR'ing the SIRTP request to make sure it is seen but hopefully not sticky. This seems like a clean fix, at least on our simulator; if you don't agree, our simulator guy will take a closer look at our iommu model.

Quoted 9 lines

Check the VT-d spec (6.7, Set Interrupt Remapping Table Pointer Operation): "Software must always follow the interrupt-remapping table pointer set operation with a global invalidate of the IEC to ensure hardware references the new structures *before* enabling interrupt remapping." There is also (command register description): "If multiple control fields in this register need to be modified, software must serialize the modifications through multiple writes to this register." This, in combination with the command registers description of bits 24 and 25 strongly suggests that your model is broken.

Thanks, Jan. I'll read up on it. Still working out some kinks with the relatively new IOMMU/interrupt-remapping logic which we added over the last 18 months or so. Sorry for any distraction, and thanks for the help/info.
Gary

Quoted 10 lines
Quoted 3 lines

Found testing on our simulator, not real hardware.

Funnily, the original issue was found by the QEMU model of VT-d that used to take the last cited sentence very strictly (I asked to remove it due to the preexisting Linux versions).

Jan