pio barriers
I noticed that in asm-ia64/system.h there's a comment that reads:
/*
...
* Note: "mb()" and its variants cannot be used as a fence to order
* accesses to memory mapped I/O registers. For that, mf.a needs to
* be used. However, we don't want to always use mf.a because (a)
* it's (presumably) much slower than mf and (b) mf.a is supported for
* sequential memory pages only.
*/
Is there a macro (e.g. piob() or mmiob()) to wrap mf.a or are users expected to call it explicitly when they need it? If there is no macro, I'd like to add one, as I think it will be necessary to properly support our NUMA platform.
Thanks,
Jesse
Quoted 11 lines
I noticed that in
asm-ia64/system.hthere's a comment that reads:
/*
...
* Note: "mb()" and its variants cannot be used as a fence to order
* accesses to memory mapped I/O registers. For that, mf.a needs to
* be used. However, we don't want to always use mf.a because (a)
* it's (presumably) much slower than mf and (b) mf.a is supported for
* sequential memory pages only.
*/
Quoted 5 lines
Is there a macro (e.g.
piob()ormmiob()) to wrap mf.a or are users expected to call it explicitly when they need it? If there is no macro, I'd like to add one, as I think it will be necessary to properly support our NUMA platform.
There is a macro in io.h
#define __ia64_mf_a() __asm__ __volatile__ ("mf.a" ::: "memory")However, it is not clear to me when a mf.a is actually required?? Does anyone have an opinion?? I dont see any clear need for it. Seems like places where we might want to use it actually require a flushbus().
Signature · 3 lines
Thanks Jack Steiner (651-683-5302) (vnet 233-5302) [email protected]
Actually, I was thinking that for most platforms, mf.a should do a 'flushbus' (i.e. act as a barrier for all previous pio reads/writes). Of course, for SN, we'll need to do other stuff too, so I'd like to make a machine vector.
Jesse
Quoted 8 lines · Jack Steiner
There is a macro in
io.h#define __ia64_mf_a() __asm__ __volatile__ ("mf.a" ::: "memory")However, it is not clear to me when a mf.a is actually required?? Does anyone have an opinion?? I dont see any clear need for it. Seems like places where we might want to use it actually require a
flushbus().
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jesse Barnes
Jesse> I noticed that in asm-ia64/system.h there's a comment that reads:
Jesse> /*
Jesse> ...
Jesse> * Note: "mb()" and its variants cannot be used as a fence to order Jesse> * accesses to memory mapped I/O registers. For that, mf.a needs to Jesse> * be used. However, we don't want to always use mf.a because (a) Jesse> * it's (presumably) much slower than mf and (b) mf.a is supported for Jesse> * sequential memory pages only.
Jesse> */
Jesse> Is there a macro (e.g. piob() or mmiob()) to wrap mf.a or are users Jesse> expected to call it explicitly when they need it? If there is no Jesse> macro, I'd like to add one, as I think it will be necessary to Jesse> properly support our NUMA platform.
The comment is wrong, or at least misleading (I wrote it, so hopefully nobody is offended... ;-). mf.a is needed for inX/outX emulation, not really for ordering. Uncached accesses are not re-ordered by the CPU and mf will do just fine as far as ordering of cached accesses are concerned.
Platform-acceptance is a tricky business, as it's, well, platform dependent (note that "mf.a" doesn't really guarantee to do anything).
Can you get away with forcing the proper ordering with a dummy-read? If so, I suspect that would be preferable as that is the only platform independent way to do this (as far as I know).
--david
Quoted 5 lines · David Mosberger
The comment is wrong, or at least misleading (I wrote it, so hopefully nobody is offended... ;-). mf.a is needed for inX/outX emulation, not really for ordering. Uncached accesses are not re-ordered by the CPU and mf will do just fine as far as ordering of cached accesses are concerned.
On our platform, uncached accesses can arrive out of orderif they come from different cpus, so simple spinlock protection isn't quite enough to guarantee that pios don't arrive out of order. We'll need a pio barrier macro and (ideally) an spin_unlock variant that includes it.
Quoted 3 lines
Can you get away with forcing the proper ordering with a dummy-read? If so, I suspect that would be preferable as that is the only platform independent way to do this (as far as I know).
Do you mean that a driver should do a dummy read following writes that have to arrive prior to others? I guess that would be just as invasive as introducing a pio barrier or another spinlock...
Jesse
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jesse Barnes
Quoted 5 lines · David Mosberger
Quoted 5 lines
The comment is wrong, or at least misleading (I wrote it, so hopefully nobody is offended... ;-). mf.a is needed for inX/outX emulation, not really for ordering. Uncached accesses are not re-ordered by the CPU and mf will do just fine as far as ordering of cached accesses are concerned.
Jesse> On our platform, uncached accesses can arrive out of orderif they come Jesse> from different cpus, so simple spinlock protection isn't quite enough Jesse> to guarantee that pios don't arrive out of order. We'll need a pio Jesse> barrier macro and (ideally) an spin_unlock variant that includes it.
Yes, I realize that, but it's not the CPU that's reordering the access so mf doesn't help and mf.a really doesn't guarantee anything either (though it may on your platform).
Quoted 3 lines
Quoted 3 lines
Can you get away with forcing the proper ordering with a dummy-read? If so, I suspect that would be preferable as that is the only platform independent way to do this (as far as I know).
Jesse> Do you mean that a driver should do a dummy read following writes that Jesse> have to arrive prior to others? I guess that would be just as Jesse> invasive as introducing a pio barrier or another spinlock...
Invasive, yes. But on some platforms there may be no other way of enforcing order. Perhaps what would be best would be a macro that takes a device address as an argument. Depending on platform, you could then do a dummy read from this address or use a special instruction, such as mf.a, to enforce order.
--david
Quoted 3 lines · David Mosberger
Yes, I realize that, but it's not the CPU that's reordering the access so mf doesn't help and mf.a really doesn't guarantee anything either (though it may on your platform).
Yeah, that's too bad. On MIPS we've got 'sync', which is implemented to do a pio flush. Apparently IA64 doesn't have a nice way to do something similiar though, so oh well.
Quoted 5 lines
Invasive, yes. But on some platforms there may be no other way of enforcing order. Perhaps what would be best would be a macro that takes a device address as an argument. Depending on platform, you could then do a dummy read from this address or use a special instruction, such as mf.a, to enforce order.
Can you think of other platforms that might need a device argument to a potential pio barrier macro? I was tentatively thinking of implementing it without any arguments, but I suppose we could just ignore any arguments for our platform...
Thanks,
Jesse
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jesse Barnes
Jesse> On Mon, Dec 10, 2001 at 04:44:42PM -0800, David Mosberger
Quoted 3 lines · Jesse> wrote:
Quoted 3 lines
Yes, I realize that, but it's not the CPU that's reordering the access so mf doesn't help and mf.a really doesn't guarantee anything either (though it may on your platform).
Jesse> Yeah, that's too bad. On MIPS we've got 'sync', which is Jesse> implemented to do a pio flush. Apparently IA64 doesn't have Jesse> a nice way to do something similiar though, so oh well.
Well, mf.a *does* do something on the 460 chipset. So I think it's mostly a platform issue. Are you saying that on SGI's IA-64 platforms mf.a doesn't do the equivalent of the MIPS sync? (Just curious.)
Quoted 5 lines
Quoted 5 lines
Invasive, yes. But on some platforms there may be no other way of enforcing order. Perhaps what would be best would be a macro that takes a device address as an argument. Depending on platform, you could then do a dummy read from this address or use a special instruction, such as mf.a, to enforce order.
Jesse> Can you think of other platforms that might need a device Jesse> argument to a potential pio barrier macro? I was tentatively Jesse> thinking of implementing it without any arguments, but I Jesse> suppose we could just ignore any arguments for our Jesse> platform...
Well, my concern is mostly about the x86 platform. I assume x86 NUMA machines would have the same ordering problem, so if we propose a fix for the problem, we need to make sure it works on x86 too as otherwise it will not make it into the official tree.
--david