Skip to contentSkip to conversation

intel-iommu.c bug

DISCUSSION Cliff Wickman 4 participants 4 messages Dec 17, 2010

lore
Share & tools
lore
#0 lore raw

This bug was reported by Mike Habeck <[email protected]>. The test system was an SGI Altix UV. These are Mike's words:

It appears there is a bug in the iommu code that when 'forcedac' isn't used the nvidia driver is handed back a 44bit dma address even though it's dma_mask is set to 40bits.

I added some debug to the intel_iommu code and I see:
intel_map_sg(): dma_addr_t=0xf81fffff000, pdev->dma_mask=0xffffffffff

Note the dma_addr_t being handed back is 44bits even though the mask is 40bits. This results in the nvidia card generating a bad dma (i.e. the nvidia hw is only capable of generating a 40bit dma address so the upper 4 bits are lost and that results in the iommu hw detecting a bad dma access):

DRHD: handling fault status reg 2
DMAR:[DMA Read] Request device [36:00.0] fault addr 81fffff000 DMAR:[fault reason 06] PTE Read access is not set

If I boot with 'forcedac' then the dma mask is honored and the dma_addr_t handed back is 40bits:

intel_map_sg(): dma_addr_t=0xfffffff000, pdev->dma_mask=0xffffffffff

Without forcedac you'd expect these early maps being handed back to be 32bits. This is the first debug printf (so the first mapping the nvidia device has requested) so I'd expect it to be 0xfffff000... interesting that is what the lower 32bits are in this address being handed back... that 0xf81 upper bits appear to be garbage bits. This might be a hint to help find the bug...

#1 lore raw

Hi Cliff

Sending this to Suresh who is handling linux iommu.

Ashok

Quoted 39 lines

-----Original Message-----
From: Cliff Wickman [mailto:[email protected]]
Sent: Friday, December 17, 2010 7:03 AM
To: [email protected]
Cc: Raj, Ashok; Li, Shaohua; Keshavamurthy, Anil S; Yu, Fenghua
Subject: intel-iommu.c bug

This bug was reported by Mike Habeck <[email protected]>. The test system was an SGI Altix UV. These are Mike's words:

It appears there is a bug in the iommu code that when 'forcedac' isn't used the nvidia driver is handed back a 44bit dma address even though it's dma_mask is set to 40bits.

I added some debug to the intel_iommu code and I see:
intel_map_sg(): dma_addr_t=0xf81fffff000, pdev->dma_mask=0xffffffffff

Note the dma_addr_t being handed back is 44bits even though the mask is 40bits. This results in the nvidia card generating a bad dma (i.e. the nvidia hw is only capable of generating a 40bit dma address so the upper 4 bits are lost and that results in the iommu hw detecting a bad dma access):

DRHD: handling fault status reg 2
DMAR:[DMA Read] Request device [36:00.0] fault addr 81fffff000 DMAR:[fault reason 06] PTE Read access is not set

If I boot with 'forcedac' then the dma mask is honored and the dma_addr_t handed back is 40bits:

intel_map_sg(): dma_addr_t=0xfffffff000, pdev->dma_mask=0xffffffffff

Without forcedac you'd expect these early maps being handed back to be 32bits. This is the first debug printf (so the first mapping the nvidia device has requested) so I'd expect it to be 0xfffff000... interesting that is what the lower 32bits are in this address being handed back... that 0xf81 upper bits appear to be garbage bits. This might be a hint to help find the bug...

#2 lore raw
Quoted 3 lines · Ashok

Hi Cliff

Sending this to Suresh who is handling linux iommu.

No. It is David Woodhouse (added).

Quoted 43 lines

Ashok

Quoted 39 lines

-----Original Message-----
From: Cliff Wickman [mailto:[email protected]]
Sent: Friday, December 17, 2010 7:03 AM
To: [email protected]
Cc: Raj, Ashok; Li, Shaohua; Keshavamurthy, Anil S; Yu, Fenghua
Subject: intel-iommu.c bug

This bug was reported by Mike Habeck <[email protected]>. The test system was an SGI Altix UV. These are Mike's words:

It appears there is a bug in the iommu code that when 'forcedac' isn't used the nvidia driver is handed back a 44bit dma address even though it's dma_mask is set to 40bits.

I added some debug to the intel_iommu code and I see:
intel_map_sg(): dma_addr_t=0xf81fffff000, pdev->dma_mask=0xffffffffff

Note the dma_addr_t being handed back is 44bits even though the mask is 40bits. This results in the nvidia card generating a bad dma (i.e. the nvidia hw is only capable of generating a 40bit dma address so the upper 4 bits are lost and that results in the iommu hw detecting a bad dma access):

DRHD: handling fault status reg 2
DMAR:[DMA Read] Request device [36:00.0] fault addr 81fffff000 DMAR:[fault reason 06] PTE Read access is not set

If I boot with 'forcedac' then the dma mask is honored and the dma_addr_t handed back is 40bits:

intel_map_sg(): dma_addr_t=0xfffffff000, pdev->dma_mask=0xffffffffff

Without forcedac you'd expect these early maps being handed back to be 32bits. This is the first debug printf (so the first mapping the nvidia device has requested) so I'd expect it to be 0xfffff000... interesting that is what the lower 32bits are in this address being handed back... that 0xf81 upper bits appear to be garbage bits. This might be a hint to help find the bug...

#3 lore raw
Quoted 31 lines · Suresh Siddha
Quoted 31 lines
Quoted 31 lines

-----Original Message-----
From: Cliff Wickman [mailto:[email protected]]
Sent: Friday, December 17, 2010 7:03 AM
To: [email protected]
Cc: Raj, Ashok; Li, Shaohua; Keshavamurthy, Anil S; Yu, Fenghua
Subject: intel-iommu.c bug

This bug was reported by Mike Habeck <[email protected]>. The test system was an SGI Altix UV. These are Mike's words:

It appears there is a bug in the iommu code that when 'forcedac' isn't used the nvidia driver is handed back a 44bit dma address even though it's dma_mask is set to 40bits.

I added some debug to the intel_iommu code and I see:
intel_map_sg(): dma_addr_t=0xf81fffff000, pdev->dma_mask=0xffffffffff

Note the dma_addr_t being handed back is 44bits even though the mask is 40bits. This results in the nvidia card generating a bad dma (i.e. the nvidia hw is only capable of generating a 40bit dma address so the upper 4 bits are lost and that results in the iommu hw detecting a bad dma access):

DRHD: handling fault status reg 2
DMAR:[DMA Read] Request device [36:00.0] fault addr 81fffff000 DMAR:[fault reason 06] PTE Read access is not set

If I boot with 'forcedac' then the dma mask is honored and the dma_addr_t handed back is 40bits:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

intel_map_sg(): dma_addr_t=0xfffffff000, pdev->dma_mask=0xffffffffff

Without forcedac you'd expect these early maps being handed back to be 32bits. This is the first debug printf (so the first mapping the nvidia device has requested) so I'd expect it to be 0xfffff000... interesting that is what the lower 32bits are in this address being handed back... that 0xf81 upper bits appear to be garbage bits. This might be a hint to help find the bug...

Hm, you're right; it should be returning addresses under 4GiB until that space is exhausted and it has to use higher addresses, unless you pass 'forcedac' on the command line.

Please could you instrument alloc_iova (in drivers/pci/iova.c to print its arguments, and also pfn_hi and pfn_lo of the iova it returns?

If those are sane, can you print start_vpfn at about line 2909 of intel_iommu.c? And if *that* looks sane, print iov_pfn and the new value of sg->dma_address each time that's set, at round line 1674.

Is this is 32-bit or 64-bit kernel?

Signature · 1 line
dwmw2