Skip to contentSkip to conversation

sba_iommu status?

Linux-ia64 Peter Chubb 3 participants 4 messages Mar 31, 2003

lore
Share & tools
lore
#0 lore raw

Hi,
What is the status of the IOMMU support? I'm finding that
with CONFIG_ZX1 turned on, that PCI cards that can do only 32-bit DMA cause system hangs.

I enabled all the debug options in sba_iommu.c, and note that the normal case bypasses the IOMMU. It's only when a 32-bit DMA to high memory happens that the hang happens.

Here's a sample trace from my driver: I've added a printout of what's in the scatterlist when pci_map_sg() returns. The driver is mapping user-mode virtual addresses by doing essentially in this instance:

	  get_user_pages(current, current->mm, vaddr, 1,
	  0, pages, NULL);
	  sg[0].page = pages[0];
	  sg[0].offset = (unsigned long)vaddr & (PAGE_SIZE - 1);
	  sg[0].length = 1024;
	  n = pci_map_sg(devp, sg, 1, DMA_FROM_DEVICE);

	  (setup and do DMA via sg_dma_address(&sg[0]) and sg_dma_len(&sg[0]))
	  pci_unmap_sg(devp, sg, 1, DMA_FROM_DEVICE);

The code works when I build the kernel as CONFIG_DIG; but not otherwise.
The trace below is typical of what I see.
All DMA mappings except my driver bypass the IOMMU (they're mostly for the SCSI card, which is a 64-bit card, and can do 64-bit DMA)
...
sba_map_sg() START 1 entries, 0xe00000003cb74000,0x400
sba_search_bitmap() o 1 e0000000040d0000 e0000000040d0000 1 1

    e0000000040d0000 2 1
sba_alloc_range(4000) 1 -> 1 hint 0/2
sba_map_single() 0x0000000000000000 -> 0x4000
     pdir 0xe00000003eb80008 a0000000000000ff

sba_map_sg() DONE 1 mapping

sglist[0] dmaaddr = 40004000, len = 1024

sba_unmap_sg() START 1 entries, 0xe00000003cb74000,0x400 sba_unmap_single() iovp 0x40004000/400 sba_unmap_sg() DONE (nents 0)

sba_map_sg() START 1 entries, 0xe00000003cb74400,0x400
sba_search_bitmap() o 1 e0000000040d0000 e0000000040d0000 4 3
sba_alloc_range(4000) 1 -> 2 hint 0/3 sba_map_single() 0x0000000000000000 -> 0x8000

     pdir 0xe00000003eb80010 a0000000000000ff

sba_map_sg() DONE 1 mapping

sba_unmap_sg() START 1 entries, 0xe00000003cb74400,0x400 sba_unmap_single() iovp 0x40008000/400 sba_unmap_sg() DONE (nents 0)

sba_map_sg() START 1 entries, 0xe00000003cb74800,0x400
sba_search_bitmap() o 1 e0000000040d0000 e0000000040d0000 8 7
sba_alloc_range(4000) 1 -> 3 hint 0/4 sba_map_single() 0x0000000040008000 -> 0xc000

     pdir 0xe00000003eb80018 a0000000400080ff

sba_map_sg() DONE 1 mapping

+BEGIN HARDWARE ERROR STATE AT MCA
+Err Record ID: 16 SAL Rev: 0.02
+Time: 03/31/2003 06:22:22 Severity 1
+Processor Device Error Info Section
Processor Error Map: 0x201004000
Processor State Param: 0xa8000000fff21330
Processor LID: 0x0
+ Cache check info[0]
+ Level: L0, Index: 8388608, Operation: Unknown,
+ BUS Check Info [0]
+ Status Info: 2 ,Severity: 0 ,Transaction Type: 0 ,Transaction Size: 0 ,Error : Internal e000000004007308 01 00 00 00 00 00 00 00 01 30 00 00 00 00 80 00 e000000004007318 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e000000004007328 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
CPUID Regs: 0x49656e69756e6547 0x6c65746e 0x0 0x1f000704
...

#1 lore raw
Quoted 3 lines · Peter Chubb

What is the status of the IOMMU support? I'm finding that
with CONFIG_ZX1 turned on, that PCI cards that can do only 32-bit DMA cause system hangs.

2.4 or 2.5?

Signature · 3 lines
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk
#2 lore raw
In reply to Peter Chubb
Quoted 5 lines · Peter Chubb

Hi,
What is the status of the IOMMU support? I'm finding that
with CONFIG_ZX1 turned on, that PCI cards that can do only 32-bit DMA cause system hangs.

It works fine for 2.4.x kernels. The 100BT (tulip, eepro100/e100) NICs and older SCSI (U160 and slower) use IO MMU.
I've run both LAN and Mass Storage tests using CONFIG_IA64_HP_ZX1=y on 2.4.20 though none of the distributions tests/ships with 2.4.20 (well, debian has 2.4.20 available in unstable) or that config.

All drivers that advertise 32-bit capability (pci_set_dma_mask()) or don't advertise anything, will go through IO MMU for physical memory addresses above 4GB. IIRC, only 1GB of physical memory is below 4GB boundary and the rest above.

...

Quoted 2 lines

The code works when I build the kernel as CONFIG_DIG; but not otherwise.

your code looks correct.
I'm wondering why CONFIG_DIG works but not CONFIG_IA64_HP_ZX1. This sounds like 2.5.

Quoted 2 lines

The trace below is typical of what I see. All DMA mappings except my driver bypass the IOMMU

Normally, an MCA with IO MMU means the driver didn't tell the device to use a valid address.

However, this weekend Alex Williamson flushed out another SBA bug (missing wmb() when modifying IO PDIR) exposed by multiple tulip NIC driver testing. With full debug, I wouldn't expect you to hit this bug though. Alex said he would post the patch shortly for 2.4.20/2.5.x.

Quoted 1 line

+BEGIN HARDWARE ERROR STATE AT MCA

Look for
ROPE_N_ERROR 0x0000000000000240

to indicate the IO Pdir entry contained an invalid mapping in the "errdump mca" output from EFI shell.

hth,
grant

#3 lore raw
In reply to Peter Chubb
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Matthew" = Matthew Wilcox <[email protected]> writes:

Quoted 3 lines · Peter Chubb
Quoted 3 lines

What is the status of the IOMMU support? I'm finding that with CONFIG_ZX1 turned on, that PCI cards that can do only 32-bit DMA cause system hangs.

Matthew> 2.4 or 2.5?

Sorry: 2.5.64, on zx2000 or zx6000 (UP or SMP)

Peter C