Skip to contentSkip to conversation

Why is wmb() a no-op on x86_64?

DISCUSSION Bryan O'Sullivan 4 participants 8 messages Jan 18, 2006 – Jan 19, 2006

lore
Share & tools
lore
#0 lore raw

Hi, Andi -

I notice that wmb() is a no-op on x86_64 kernels unless CONFIG_UNORDERED_IO is set. Is there any particular reason for this? It's not similarly conditional on other platforms, and as a consequence, in our driver (which requires a write barrier in some situations for correctness), I have to add the following piece of ugliness:

#if defined(CONFIG_X86_64) && !defined(CONFIG_UNORDERED_IO) #define ipath_wmb() asm volatile("sfence" ::: "memory") #else
#define ipath_wmb() wmb()
#endif

	<b
#1 lore raw
Quoted 3 lines · Bryan O'Sullivan

Hi, Andi -

I notice that wmb() is a no-op

Actually it is a compiler optimizer barrier, not a no-op.

Quoted 2 lines

on x86_64 kernels unless
CONFIG_UNORDERED_IO is set.

Because x86 is architecturally defined as having ordered writes (unless you use write combining or non temporal stores which normal kernel code doesn't). So it's not needed.

Quoted 10 lines

Is there any particular reason for this?
It's not similarly conditional on other platforms, and as a consequence, in our driver (which requires a write barrier in some situations for correctness), I have to add the following piece of ugliness:

#if defined(CONFIG_X86_64) && !defined(CONFIG_UNORDERED_IO) #define ipath_wmb() asm volatile("sfence" ::: "memory") #else
#define ipath_wmb() wmb()
#endif

Hmm, I suppose one could add a wc_wmb() or somesuch, but WC is currently deeply architecture specific so I'm not sure how you can even use it portably.

Why do you need the barrier?

-Andi

#2 lore raw
Quoted 1 line · Andi Kleen

Actually it is a compiler optimizer barrier, not a no-op.

Sorry, braino.

Quoted 5 lines

Hmm, I suppose one could add a wc_wmb() or somesuch, but WC is currently deeply architecture specific so I'm not sure how you can even use it portably.

Why do you need the barrier?

On x86_64, we fiddle with the MTRRs to enable write combining, which makes a huge difference to performance. It's not clear to me what we should even do on other architectures, since the only generic entry point that even exposes write combining is pci_mmap_page_range, which is for PCI mmap through userspace, and half the arches I've looked at ignore its write_combine parameter.

	<b
#3 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Bryan" == Bryan O'Sullivan <[email protected]> writes:

Quoted 1 line · Andi Kleen
Quoted 1 line

Why do you need the barrier?

Bryan> On x86_64, we fiddle with the MTRRs to enable write combining, Bryan> which makes a huge difference to performance. It's not clear Bryan> to me what we should even do on other architectures, since the Bryan> only generic entry point that even exposes write combining is Bryan> pci_mmap_page_range, which is for PCI mmap through userspace, Bryan> and half the arches I've looked at ignore its write_combine Bryan> parameter.

A job for mmiowb() perhaps?

Cheers,
Jes

#4 lore raw
Quoted 1 line · Jes Sorensen

A job for mmiowb() perhaps?

That might be suitable. It's a no-op on most platforms, but it's only used by a handful of drivers:

        drivers/scsi/qla1280.c
        drivers/sn/ioc4.c
        drivers/net/bnx2.c
        drivers/net/sky2.c
        drivers/net/s2io.c
        drivers/net/tg3.c

If the semantics were to change so that it really did a write barrier, I doubt any existing users would notice. In fact, based on the comments in some drivers, at least some authors think it does already, when it typically doesn't.

	<b
#5 lore raw
In reply to Jes Sorensen
Quoted 13 lines · Jes Sorensen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Bryan" == Bryan O'Sullivan <[email protected]> writes:

Quoted 1 line · Andi Kleen
Quoted 1 line

Why do you need the barrier?

Bryan> On x86_64, we fiddle with the MTRRs to enable write combining, Bryan> which makes a huge difference to performance. It's not clear Bryan> to me what we should even do on other architectures, since the Bryan> only generic entry point that even exposes write combining is Bryan> pci_mmap_page_range, which is for PCI mmap through userspace, Bryan> and half the arches I've looked at ignore its write_combine Bryan> parameter.

Quoted 1 line

A job for mmiowb() perhaps?

No, normal IO mappings are also not write combining on x86, so it's not needed there.

I guess it's best to just define a wmb_wc() for i386/x86-64 right now and use that in ipath. I suspect it won't compile anywhere else anyways.

-Andi

#6 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Andi" == Andi Kleen <[email protected]> writes:

Quoted 1 line · Jes Sorensen
Quoted 1 line

A job for mmiowb() perhaps?

Andi> No, normal IO mappings are also not write combining on x86, so Andi> it's not needed there.

True, just seemed nasty to have yet another special purpose wmb() variation.

Cheers,
Jes

#7 lore raw
In reply to Jes Sorensen

Bryan> On x86_64, we fiddle with the MTRRs to enable write Bryan> combining, which makes a huge difference to performance. Bryan> It's not clear to me what we should even do on other Bryan> architectures, since the only generic entry point that even Bryan> exposes write combining is pci_mmap_page_range, which is Bryan> for PCI mmap through userspace, and half the arches I've Bryan> looked at ignore its write_combine parameter.

Jes> A job for mmiowb() perhaps?

I don't think the semantics of mmiowb() do what is desired here. mmiowb() is all about ordering writes between separate CPUs, and the issue at hand here is that write-combining buffers might reorder writes from a single CPU.

- R.