intel-iommu.c bug
This bug was reported by Mike Habeck <[email protected]>. The test system was an SGI Altix UV. These are Mike's words:
It appears there is a bug in the iommu code that when 'forcedac' isn't used the nvidia driver is handed back a 44bit dma address even though it's dma_mask is set to 40bits.
I added some debug to the intel_iommu code and I see:
intel_map_sg(): dma_addr_t=0xf81fffff000, pdev->dma_mask=0xffffffffff
Note the dma_addr_t being handed back is 44bits even though the mask is 40bits. This results in the nvidia card generating a bad dma (i.e. the nvidia hw is only capable of generating a 40bit dma address so the upper 4 bits are lost and that results in the iommu hw detecting a bad dma access):
DRHD: handling fault status reg 2
DMAR:[DMA Read] Request device [36:00.0] fault addr 81fffff000 DMAR:[fault reason 06] PTE Read access is not set
If I boot with 'forcedac' then the dma mask is honored and the dma_addr_t handed back is 40bits:
intel_map_sg(): dma_addr_t=0xfffffff000, pdev->dma_mask=0xffffffffff
Without forcedac you'd expect these early maps being handed back to be 32bits. This is the first debug printf (so the first mapping the nvidia device has requested) so I'd expect it to be 0xfffff000... interesting that is what the lower 32bits are in this address being handed back... that 0xf81 upper bits appear to be garbage bits. This might be a hint to help find the bug...
Hi Cliff
Sending this to Suresh who is handling linux iommu.
Ashok
Quoted 39 lines
-----Original Message-----
From: Cliff Wickman [mailto:[email protected]]
Sent: Friday, December 17, 2010 7:03 AM
To: [email protected]
Cc: Raj, Ashok; Li, Shaohua; Keshavamurthy, Anil S; Yu, Fenghua
Subject:intel-iommu.cbugThis bug was reported by Mike Habeck <[email protected]>. The test system was an SGI Altix UV. These are Mike's words:
It appears there is a bug in the iommu code that when 'forcedac' isn't used the nvidia driver is handed back a 44bit dma address even though it's
dma_maskis set to 40bits.I added some debug to the
intel_iommucode and I see:
intel_map_sg(): dma_addr_t=0xf81fffff000, pdev->dma_mask=0xffffffffffNote the
dma_addr_tbeing handed back is 44bits even though the mask is 40bits. This results in the nvidia card generating a bad dma (i.e. the nvidia hw is only capable of generating a 40bit dma address so the upper 4 bits are lost and that results in the iommu hw detecting a bad dma access):DRHD: handling fault status reg 2
DMAR:[DMA Read] Request device [36:00.0] fault addr81fffff000DMAR:[fault reason 06] PTE Read access is not setIf I boot with 'forcedac' then the dma mask is honored and the
dma_addr_thanded back is 40bits:
intel_map_sg(): dma_addr_t=0xfffffff000, pdev->dma_mask=0xffffffffffWithout forcedac you'd expect these early maps being handed back to be 32bits. This is the first debug printf (so the first mapping the nvidia device has requested) so I'd expect it to be
0xfffff000... interesting that is what the lower 32bits are in this address being handed back... that0xf81upper bits appear to be garbage bits. This might be a hint to help find the bug...
Quoted 3 lines · Ashok
Hi Cliff
Sending this to Suresh who is handling linux iommu.
No. It is David Woodhouse (added).
Quoted 43 lines
Ashok
Quoted 39 lines
-----Original Message-----
From: Cliff Wickman [mailto:[email protected]]
Sent: Friday, December 17, 2010 7:03 AM
To: [email protected]
Cc: Raj, Ashok; Li, Shaohua; Keshavamurthy, Anil S; Yu, Fenghua
Subject:intel-iommu.cbugThis bug was reported by Mike Habeck <[email protected]>. The test system was an SGI Altix UV. These are Mike's words:
It appears there is a bug in the iommu code that when 'forcedac' isn't used the nvidia driver is handed back a 44bit dma address even though it's
dma_maskis set to 40bits.I added some debug to the
intel_iommucode and I see:
intel_map_sg(): dma_addr_t=0xf81fffff000, pdev->dma_mask=0xffffffffffNote the
dma_addr_tbeing handed back is 44bits even though the mask is 40bits. This results in the nvidia card generating a bad dma (i.e. the nvidia hw is only capable of generating a 40bit dma address so the upper 4 bits are lost and that results in the iommu hw detecting a bad dma access):DRHD: handling fault status reg 2
DMAR:[DMA Read] Request device [36:00.0] fault addr81fffff000DMAR:[fault reason 06] PTE Read access is not setIf I boot with 'forcedac' then the dma mask is honored and the
dma_addr_thanded back is 40bits:
intel_map_sg(): dma_addr_t=0xfffffff000, pdev->dma_mask=0xffffffffffWithout forcedac you'd expect these early maps being handed back to be 32bits. This is the first debug printf (so the first mapping the nvidia device has requested) so I'd expect it to be
0xfffff000... interesting that is what the lower 32bits are in this address being handed back... that0xf81upper bits appear to be garbage bits. This might be a hint to help find the bug...
Quoted 31 lines · Suresh Siddha
Quoted 31 lines
Quoted 31 lines
-----Original Message-----
From: Cliff Wickman [mailto:[email protected]]
Sent: Friday, December 17, 2010 7:03 AM
To: [email protected]
Cc: Raj, Ashok; Li, Shaohua; Keshavamurthy, Anil S; Yu, Fenghua
Subject:intel-iommu.cbugThis bug was reported by Mike Habeck <[email protected]>. The test system was an SGI Altix UV. These are Mike's words:
It appears there is a bug in the iommu code that when 'forcedac' isn't used the nvidia driver is handed back a 44bit dma address even though it's
dma_maskis set to 40bits.I added some debug to the
intel_iommucode and I see:
intel_map_sg(): dma_addr_t=0xf81fffff000, pdev->dma_mask=0xffffffffffNote the
dma_addr_tbeing handed back is 44bits even though the mask is 40bits. This results in the nvidia card generating a bad dma (i.e. the nvidia hw is only capable of generating a 40bit dma address so the upper 4 bits are lost and that results in the iommu hw detecting a bad dma access):DRHD: handling fault status reg 2
DMAR:[DMA Read] Request device [36:00.0] fault addr81fffff000DMAR:[fault reason 06] PTE Read access is not setIf I boot with 'forcedac' then the dma mask is honored and the
dma_addr_thanded back is 40bits:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
intel_map_sg(): dma_addr_t=0xfffffff000, pdev->dma_mask=0xffffffffffWithout forcedac you'd expect these early maps being handed back to be 32bits. This is the first debug printf (so the first mapping the nvidia device has requested) so I'd expect it to be
0xfffff000... interesting that is what the lower 32bits are in this address being handed back... that0xf81upper bits appear to be garbage bits. This might be a hint to help find the bug...
Hm, you're right; it should be returning addresses under 4GiB until that space is exhausted and it has to use higher addresses, unless you pass 'forcedac' on the command line.
Please could you instrument alloc_iova (in drivers/pci/iova.c to print its arguments, and also pfn_hi and pfn_lo of the iova it returns?
If those are sane, can you print start_vpfn at about line 2909 of intel_iommu.c? And if *that* looks sane, print iov_pfn and the new value of sg->dma_address each time that's set, at round line 1674.
Is this is 32-bit or 64-bit kernel?
Signature · 1 line
dwmw2