Skip to contentSkip to conversation

dma_mapping_ops for i386

DISCUSSION Jeremy Fitzhardinge 4 participants 7 messages Jun 26, 2007 – Jun 30, 2007

lore
Share & tools
lore
#0 lore raw

I'm looking at adding dom0 support to the pv-ops kernel. One of the obvious things we need is to support real device drivers, and the associated p->m translations for devices.

I'm thinking the cleanest thing to do is make x86-64's dma-mapping.h with its dma_mapping_ops common to i386 and x86-64, so we can hook the Xen translations in there. Presumably we'll need to do this anyway to support VTd for 32-bit (but I don't know if that's a reasonable thing to do anyway).

What do you think?

J

#1 lore raw
Quoted 11 lines · Jeremy Fitzhardinge

I'm looking at adding dom0 support to the pv-ops kernel. One of the obvious things we need is to support real device drivers, and the associated p->m translations for devices.

I'm thinking the cleanest thing to do is make x86-64's dma-mapping.h with its dma_mapping_ops common to i386 and x86-64, so we can hook the Xen translations in there. Presumably we'll need to do this anyway to support VTd for 32-bit (but I don't know if that's a reasonable thing to do anyway).

What do you think?

Ok, if you can do it without ifdefs.

And no swiotlb on i386; that is something that is completely broken in upstream Xen and needs to be fixed properly anyways.

-Andi

#2 lore raw
Quoted 2 lines · Andi Kleen

Ok, if you can do it without ifdefs.

That should be OK. All the existing i386 mapping operations would just have their own ops structure, right?

Quoted 3 lines

And no swiotlb on i386; that is something that is completely broken in upstream Xen and needs to be fixed properly anyways.

Hm, OK. I'm not really familiar with the issues here. What are they? Looks like Jan has made a number of Xen-ish changes to lib/swiotlb.c; are more changes be needed?

J

#3 lore raw
Quoted 6 lines · On Wednesday 27 June 2007 16:15:17 Jeremy Fitzhardinge wrote:
Quoted 2 lines · Andi Kleen

Ok, if you can do it without ifdefs.

That should be OK. All the existing i386 mapping operations would just have their own ops structure, right?

I just mention it because many people's ideas of merging files seem to add lots of ifdefs which is imho the totally wrong thing to do.

Quoted 7 lines
Quoted 3 lines

And no swiotlb on i386; that is something that is completely broken in upstream Xen and needs to be fixed properly anyways.

Hm, OK. I'm not really familiar with the issues here. What are they? Looks like Jan has made a number of Xen-ish changes to lib/swiotlb.c; are more changes be needed?

See the recent "quiet down swiotlb warnings" thread which uncovered quite some corpses in Xen's current IO setup.

Xen apparently bounces for multi page IOs which get merged from block lists because the block layer doesn't know they are not really continuous in machine memory.

Proper fix is to tell the block layer to not merge in the first place instead.

And probably some similar mechanism for network drivers that limits MTUs.

Maybe we'll still need a simple bouncing mechanism for other obscure devices with large IOs then, but I would very much prefer if it wasn't swiotlb and could be solved some other way.

-Andi

#4 lore raw
Quoted 25 lines · Andi Kleen
Quoted 7 lines · On Wednesday 27 June 2007 16:15:17 Jeremy Fitzhardinge wrote:
Quoted 3 lines · Andi Kleen

Ok, if you can do it without ifdefs.

That should be OK. All the existing i386 mapping operations would just have their own ops structure, right?

I just mention it because many people's ideas of merging files seem to add lots of ifdefs which is imho the totally wrong thing to do.

Quoted 6 lines
Quoted 4 lines

And no swiotlb on i386; that is something that is completely broken in upstream Xen and needs to be fixed properly anyways.

Hm, OK. I'm not really familiar with the issues here. What are they? Looks like Jan has made a number of Xen-ish changes to lib/swiotlb.c; are more changes be needed?

See the recent "quiet down swiotlb warnings" thread which uncovered quite some corpses in Xen's current IO setup.

Quoted 9 lines

Xen apparently bounces for multi page IOs which get merged from block
lists because the block layer doesn't know they are not really continuous in machine memory.

Proper fix is to tell the block layer to not merge in the first place instead.

And probably some similar mechanism for network drivers that limits MTUs.

Well, I think there are two issues here. One is that two pseudo-physical pages won't necessarily be contigious in bus space, because of the pseudo-phys to machine mapping.

The second problem is that devices which can't address all machine physical memory (ie, 32-bit PCI devices on machines with >4G memory) will need to have bouncebuffers established for them. Device drivers won't necessarily be able to do it because they're not really aware of machine addresses.

Quoted 3 lines

Maybe we'll still need a simple bouncing mechanism for other obscure
devices with large IOs then, but I would very much prefer if it wasn't swiotlb and could be solved some other way.

I think 32-bit-only devices are a bigger concern, no?

J

#5 lore raw
In reply to Andi Kleen
Quoted 7 lines · Andi Kleen

See the recent "quiet down swiotlb warnings" thread which uncovered quite some corpses in Xen's current IO setup.

Xen apparently bounces for multi page IOs which get merged from block lists because the block layer doesn't know they are not really continuous in machine memory.

Quoted 5 lines

Proper fix is to tell the block layer to not merge in the first place instead.

And probably some similar mechanism for network drivers that limits MTUs.

Will that guarantee that block and net IOs will not straddle a page boundary?

Cheers,
Muli

#6 lore raw
Quoted 6 lines · Muli Ben-Yehuda
Quoted 3 lines

And probably some similar mechanism for network drivers that limits MTUs.

Will that guarantee that block and net IOs will not straddle a page boundary?

Mostly. There is the thorny case of slab debugging that breaks these nice assumptions.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt