stop the buffer bouncing
James,
I think I found the reason parisc is bouncing buffers on the a500 when it shouldn't (iommu is present):
include/asm/pci.h defines:
#define PCI_DMA_BUS_IS_PHYS (1)
and drivers/scsi/scsi_lib.c uses it like this:
if (PCI_DMA_BUS_IS_PHYS && host_dev && host_dev->dma_mask)
return *host_dev->dma_mask;Which means anything > ~0U will get bounced.
Sym2 driver uses a 32-bit DMA mask.
And we established this a500 has RAM above :
model 9000/800/A500-6X
Memory Ranges:
0) Start 0x0000000000000000 End 0x00000000efffffff Size 3840 Mb
1) Start 0x00000010f0000000 End 0x00000010ffffffff Size 256 Mb
If someone else can review/test and/or commit, that would be great. Otherwise I can commit this tomorrow. (Assuming CVS is still alive and well)
Patch below is partially inspired by include/asm-ia64/pci.h. Though it might be better to define/declare parisc_bus_is_phys elsewhere if EISA support also eventually needs it.
Ah, almost forgot the good news: a500 is still running multiple instances of wipe (14 or 15, I forgot). Check riot's console the morning...
Log Message:
Fix the PCI_DMA_BUS_IS_PHYS so parisc-linux never bounces buffers on systems with an IOMMU.
thanks,
grant
Index: arch/parisc/kernel/setup.c
===================================================================
RCS file: /var/cvs/linux-2.6/arch/parisc/kernel/setup.c,v retrieving revision 1.9
parisc/kernel/setup.c +4 -0
@@ -53,6 +53,10 @@ struct proc_dir_entry * proc_runway_rootstruct proc_dir_entry * proc_gsc_root =NULL ;struct proc_dir_entry * proc_mckinley_root =NULL ; +#ifndef CONFIG_PA20 +int parisc_bus_is_phys = TRUE;/* Assume no IOMMU is present */ +EXPORT_SYMBOL (parisc_bus_is_phys); +#endif void __init setup_cmdline (char **cmdline_p) {
Index: drivers/parisc/ccio-dma.c
===================================================================
RCS file: /var/cvs/linux-2.6/drivers/parisc/ccio-dma.c,v retrieving revision 1.16
parisc/ccio-dma.c +4 -0
@@ -1560,6 +1560,10 @@ static int ccio_probe(struct parisc_devi parisc_vmerge_boundary =IOVP_SIZE ; parisc_vmerge_max_size =BITS_PER_LONG *IOVP_SIZE ; ioc_count++; + +#ifndef CONFIG_PA20 + parisc_bus_is_phys = FALSE; +#endif return 0 ; }
Index: drivers/parisc/sba_iommu.c
===================================================================
RCS file: /var/cvs/linux-2.6/drivers/parisc/sba_iommu.c,v retrieving revision 1.21
parisc/sba_iommu.c +4 -0
@@ -2114,6 +2116,10 @@ sba_driver_callback(struct parisc_device parisc_vmerge_boundary =IOVP_SIZE ; parisc_vmerge_max_size =IOVP_SIZE *BITS_PER_LONG ; +#ifndef CONFIG_PA20 + parisc_bus_is_phys = FALSE; +#endif +return 0 ; }
Index: include/asm-parisc/pci.h
===================================================================
RCS file: /var/cvs/linux-2.6/include/asm-parisc/pci.h,v retrieving revision 1.12
asm-parisc/pci.h +15 -4
@@ -106,11 +106,22 @@ static __inline__ int pci_is_lmmio(strustruct pci_bus ;struct pci_dev ; -/* The PCI address space does equal the physical memory -* address space. The networking and block device layers use -* this boolean for bounce buffer decisions. +#ifdef CONFIG_PA20 +/* All PA-2.0 machines have an IOMMU. */ +#define PCI_DMA_BUS_IS_PHYS (0) +#else +/* _some_ PA1.1 boxes have an IOMMU (pa7200)...but we can run these bits on +* a PA-2.0 box in narrow mode as well. *sigh* +* FIXME: WAX EISA controller mangles DMA addressing too. +*/ + +/* Set to TRUE if PCI address space equals the physical memory address space. +* The networking and block device layers use this boolean for bounce buffer +* decisions. */ -#define PCI_DMA_BUS_IS_PHYS (1) +extern unsigned int parisc_bus_is_phys; +#define PCI_DMA_BUS_IS_PHYS (parisc_bus_is_phys) +#endif /* ** Most PCI devices (eg Tulip, NCR720) also export the same registers
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
Quoted 2 lines · Grant Grundler
If someone else can review/test and/or commit, that would be great. Otherwise I can commit this tomorrow. (Assuming CVS is still alive and well)
It looks fine to me ... I didn't realise this was broken ...
Quoted 3 lines
Patch below is partially inspired by
include/asm-ia64/pci.h. Though it might be better to define/declareparisc_bus_is_physelsewhere if EISA support also eventually needs it.
Unfortunately, EISA support needs a per bus check for the iommu (since WAX on some of the machines has its own IOMMU but the rest of the system doesn't). I was going to look at introducing this abstraction, but we never managed to get an unencumbered copy of the WAX specs to get it all working.
Thanks for debugging this.
James
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 2 lines · Matthew Wilcox
Er, what? Eh? sym2 only uses a 32-bit DMA mask if you configure it to. And it appears we do tell it to ;-(
yeah...I'm probably responsible for that setting.
It worked the best. :^/
The 39/40 bit or full 64-bit support don't help on parisc-linux anyway.
Quoted 1 line
All machines include
setup.c... that's fine.
cool
Review
Quoted 7 lines
parisc/ccio-dma.c static int ccio_probe(struct parisc_devi · line 1566
Quoted 3 lines
parisc/ccio-dma.c +3
-0+#ifndef CONFIG_PA20 + parisc_bus_is_phys = FALSE; +#endif Ifdefs suck. Try this ...
(quoted diff) +1
-0+ parisc_has_iommu();
Yes - that's a good idea.
Quoted 5 lines
/*
* If the PCI device's view of memory is the same as the CPU's view of memory,
*PCI_DMA_BUS_IS_PHYSis true. The networking and block device layers use
* this boolean for bounce buffer decisions.
*/
Quoted 8 lines
#ifdef
CONFIG_PA20
/* All PA-2.0 machines have an IOMMU. */
#definePCI_DMA_BUS_IS_PHYS0
#defineparisc_has_iommu()do { } while (0)
#else
#definePCI_DMA_BUS_IS_PHYSparisc_bus_is_phys
#defineparisc_has_iommu()do {parisc_bus_is_phys= 1; } while (0)
#endif
This looks way better to me too.
Quoted 3 lines
I'm tempted to define a couple of new config symbols rather than use
CONFIG_PA20-- ifPCI_LBAand CCIO are both deselected (eg 712 build), we know the answer is 1.
Yes, that's true.
Quoted 4 lines
Equally, if GSC is deselected, this must be
a pure PCI machine (if it boots ;-) and they all have an IOMMU, though I guess you'd always select PA20 for that case. In any case, I think a couple of new symbols are in order.
Hrm....I was trying to avoid new config symbols.
We have three cases where I only saw two before:
1) PA20=y, always have IOMMU (SBA or CCIO _MUST_ be enabled).
2) PA11=y, if SBA=n and CCIO=n, never IOMMU.
3) PA11=y, if SBA=y _or_ CCIO=y, requires runtime check
I think this is straight forward enough to code up in pci.h.
Further:
CCIO depends on GSC
SBA depends on PCI
ergo Enabling PA20 and GSC but not PCI means CCIO *MUST* be enabled. ergo Enabling PA20 and PCI but not GSC means SBA *MUST* be enabled.
Could someone else look at Kconfig files to verify the above two rules are enforced?
thanks,
grant
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
Quoted 3 lines · Grant Grundler
yeah...I'm probably responsible for that setting.
It worked the best. :^/
The 39/40 bit or full 64-bit support don't help on parisc-linux anyway.
Why not? 40 bits of addressing supports up to 1TB, which exceeds the highest physical address on any PA machine I know about -- oh, but we can't bypass the IOMMU, right?
Quoted 17 lines
Quoted 4 lines
Equally, if GSC is deselected, this must be
a pure PCI machine (if it boots ;-) and they all have an IOMMU, though I guess you'd always select PA20 for that case. In any case, I think a couple of new symbols are in order.Hrm....I was trying to avoid new config symbols.
We have three cases where I only saw two before:
1) PA20=y, always have IOMMU (SBA or CCIO _MUST_ be enabled).
2) PA11=y, if SBA=n and CCIO=n, never IOMMU.
3) PA11=y, if SBA=y _or_ CCIO=y, requires runtime checkI think this is straight forward enough to code up in
pci.h.Further:
CCIO depends on GSC
SBA depends on PCI
Quoted 5 lines
ergo Enabling PA20 and GSC but not PCI means CCIO *MUST* be enabled. ergo Enabling PA20 and PCI but not GSC means SBA *MUST* be enabled.
Could someone else look at
Kconfigfiles to verify the above two rules are enforced?
Ah, "depends on" doesn't mean what you think it does. It means "if GSC isn't selected, CCIO must be N. if GSC is M, CCIO can be N or M. if GSC is Y, CCIO can be Y, N or M." OK, GSC can't be modular (nor can PCI), but SCSI can, so:
config SCSI_SYM53C8XX_2
tristate "SYM53C8XX Version 2 SCSI support" depends on PCI && SCSI
PCI -> Y N
SCSI v
Y YMN N
M MN N
N N N
The values in the table show the permitted values that SCSI_SYM53C8XX_2 can take on. If there's only one possible answer, it doesn't even ask the question ;-) Think of 'depends on' as 'limit to' and it might be more clear to you.
Signature · 10 lines
"Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain _______________________________________________ parisc-linux mailing list [email protected] http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
Quoted 8 lines · Matthew Wilcox
Quoted 3 lines · Grant Grundler
yeah...I'm probably responsible for that setting.
It worked the best. :^/
The 39/40 bit or full 64-bit support don't help on parisc-linux anyway.Why not? 40 bits of addressing supports up to 1TB, which exceeds the highest physical address on any PA machine I know about -- oh, but we can't bypass the IOMMU, right?
exactly.
Certainly not for CCIO.
On SBA, someone could try writing DVI mode support but I'm not going to. I think DVI mode adds more complexity than performance improvement. But clearly some workloads could benefit. Anything that helps avoid IO TLB thrashing is a good thing.
On ZX1 (also SBA), I suppose it's possible to fully bypass using non-coherent support. But that seems like a step backwards for general use.
...
Quoted 8 lines
Quoted 5 lines
ergo Enabling PA20 and GSC but not PCI means CCIO *MUST* be enabled. ergo Enabling PA20 and PCI but not GSC means SBA *MUST* be enabled.
Could someone else look at
Kconfigfiles to verify the above two rules are enforced?Ah, "depends on" doesn't mean what you think it does. It means "if GSC isn't selected, CCIO must be N.
Oh. You are right...but YKWIM.
Quoted 2 lines
if GSC is M, CCIO can be N or M.
if GSC is Y, CCIO can be Y, N or M."
...
Quoted 4 lines
The values in the table show the permitted values that
SCSI_SYM53C8XX_2can take on. If there's only one possible answer, it doesn't even ask the question ;-) Think of 'depends on' as 'limit to' and it might be more clear to you.
*nod*
I'll have to re-read this and the original docs...but I suspected I didn't fully understand Kconfig anyway. That's contributing to why I'm "defering" it. :^)
thanks,
grant
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
Quoted 4 lines · Matthew Wilcox
Quoted 4 lines
Further:
CCIO depends on GSC
SBA depends on PCI
Quoted 8 lines
Quoted 5 lines
ergo Enabling PA20 and GSC but not PCI means CCIO *MUST* be enabled. ergo Enabling PA20 and PCI but not GSC means SBA *MUST* be enabled.
Could someone else look at
Kconfigfiles to verify the above two rules are enforced?Ah, "depends on" doesn't mean what you think it does. It means "if GSC isn't selected, CCIO must be N.
Actually, in this particular case, this is what I was expecting. I didn't understand some of the other subtleties.
ie I'm still now sure how to encode "depends" statements that involve PA20 and PCI/GSC vis a vis SBA/CCIO support. Seems like it should be possible though.
thanks,
grant
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 2 lines · Joel Soete
It seems to also help c110 pb with ncr53c710:
a big apt-get upgrade + 2 tar in //
I'm guessing the problem is the ncr53c8xx driver does NOT call dma_set_mask(). PCI devices by default get a 32-bit mask. I'll commit a patch so that's true for "parisc" (ie GSC) devices as well. EISA device drivers will need to set an appropriate dma mask. -pa17 should be ready shortly.
grant
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
Quoted 5 lines · Grant Grundler
I'm guessing the problem is the ncr53c8xx driver does NOT call
dma_set_mask(). PCI devices by default get a 32-bit mask. I'll commit a patch so that's true for "parisc" (ie GSC) devices as well. EISA device drivers will need to set an appropriate dma mask. -pa17 should be ready shortly.
EISA devices already get this set:
drivers/eisa/eisa-bus.c: edev->dma_mask = root->dma_mask; /* Default DMA mask */ drivers/eisa/virtual_root.c: .dma_mask = 0xffffffff,
It's purely a parisc_device setup problem. Nice spotting, Grant.
Signature · 10 lines
"Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain _______________________________________________ parisc-linux mailing list [email protected] http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines · Grant Grundler
Ah good. But don't most EISA devices only support 24-bits? Oh wait...maybe that's just an ISA bus limitation that was carried along into EISA?
AFAIK, that's purely an ISA card limitation. If you plug an ISA card into an EISA slot, it'll only be able to DMA to 24-bit addresses. But an ISA card in an EISA slot don't get an eisa_device.
Signature · 10 lines
"Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain _______________________________________________ parisc-linux mailing list [email protected] http://lists.parisc-linux.org/mailman/listinfo/parisc-linux