x86-64 tools fix question

In the tools there is a declaration:

#if defined(__i386__)
#define rmb() __asm__ __volatile__ ( "lock; addl $0,0(%%esp)" : : :
"memory" )
#define wmb() __asm__ __volatile__ ( "" : : : "memory" ) #else
#error "Define barriers"
#endif

located in:
xcs/xcs.h
tools/python/xen/lowlevel/xu/xu.c

I'm assuming this has a convenient side-effect that it prevents read
reordering. Otherwise I can't figure out why this is being done at all. Now I'm guessing that that using rsp instead of esp since we are in 64bit mode will give the same effect needed.

#elif defined(__x86_64__)
#define rmb() __asm__ __volatile__ ( "lock; addl $0,0(%%rsp)" : : :
"memory" )
#define wmb() __asm__ __volatile__ ( "" : : : "memory" )

I would like to discuss is this correct, dead wrong, or even needed at
all?

Jerone Young
Open Virtualization
IBM Linux Technology Center
[email protected]
512-838-1157 (T/L: 678-1157)



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

x86/64 has proper barrier instructions -- see include/asm-x86_64/system.h in Linux. It is from there that we should pull our definitions. Barrier macros are defined in a few places in the tools -- we ought to pull them all into one single header incorporated by all tools that need it.

-- Keir

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

I was just about to send this out myself :-) I think the consensus was to put them all in xc.h. I posted a patch recently that made everything include asm/system.h. All it should take is replacing asm/system.h with xc.h in the patch and then making the necessary changes to xc.h.

For reference, the proper x86-64 barriers are:

#define mb()     asm volatile("mfence":::"memory")
#define rmb()    asm volatile("lfence":::"memory")
#define wmb()    asm volatile("sfence":::"memory")

Regards,
Anthony Liguori

Regards,

Anthony Liguori



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

We actually should have wmb() be defined as "asm volatile("":::"memory") since we are not using out of order io.

The patch attached removes all the instances and places the definitions in libxc/xc.h as per the discussion last week between Anthony & Keir.

Jerone Young
Open Virtualization
IBM Linux Technology Center
[email protected]
512-838-1157 (T/L: 678-1157)

diff -Nur xen-unstable-0301.orig/tools/blktap/blktaplib.h xen-unstable-0301-test/tools/blktap/blktaplib.h
@@ -10,22 +10,6 @@
 
 #include <stdint.h>
 
-typedef uint8_t            u8;
-typedef uint16_t           u16;
-typedef uint32_t           u32;
-typedef uint64_t           u64;
-typedef int8_t             s8;
-typedef int16_t            s16;
-typedef int32_t            s32;
-typedef int64_t            s64;
-                                                                                
-#if defined(__i386__)
-#define rmb() __asm__ __volatile__ ( "lock; addl $0,0(%%esp)" : : : "memory" )
-#define wmb() __asm__ __volatile__ ( "" : : : "memory" )
-#else
-#error "Define barriers"
-#endif
-    
 #include <sys/user.h>
 #include <xen/xen.h>
 #include <xen/io/blkif.h>

diff -Nur xen-unstable-0301.orig/tools/libxc/xc.h xen-unstable-0301-test/tools/libxc/xc.h
@@ -25,6 +25,21 @@
 #include <xen/event_channel.h>
 #include <xen/sched_ctl.h>
 
+
+/*\
+ *  DEFINITIONS FOR CPU BARRIERS
+\*/ 
+#if defined(__i386__)
+#define rmb() __asm__ __volatile__ ("lock; addl $0,0(%%esp)" : : : "memory")
+#define wmb() __asm__ __volatile__ ("" : : : "memory")
+#elif defined(__x86_64__)
+#define mb()     asm volatile("mfence":::"memory")
+#define rmb()    asm volatile("lfence":::"memory")
+#define wmb()    asm volatile( "" :::"memory")
+#else
+#error "Define barriers"
+#endif
+
 /*\
  *  INITIALIZATION FUNCTIONS
 \*/ 

diff -Nur xen-unstable-0301.orig/tools/python/xen/lowlevel/xu/xu.c xen-unstable-0301-test/tools/python/xen/lowlevel/xu/xu.c
@@ -49,14 +49,6 @@
 /* Size of a machine page frame. */
 #define PAGE_SIZE 4096
 
-#if defined(__i386__)
-#define rmb() __asm__ __volatile__ ( "lock; addl $0,0(%%esp)" : : : "memory" )
-#define wmb() __asm__ __volatile__ ( "" : : : "memory" )
-#else
-#error "Define barriers"
-#endif
-
-
 /* Set the close-on-exec flag on a file descriptor.  Doesn't currently bother
  * to check for errors. */
 /*

diff -Nur xen-unstable-0301.orig/tools/xcs/xcs.h xen-unstable-0301-test/tools/xcs/xcs.h
@@ -39,13 +39,6 @@
 /* Size of a machine page frame. */
 #define PAGE_SIZE 4096
 
-#if defined(__i386__)
-#define rmb() __asm__ __volatile__ ( "lock; addl $0,0(%%esp)" : : : "memory" )
-#define wmb() __asm__ __volatile__ ( "" : : : "memory" )
-#else
-#error "Define barriers"
-#endif
-
 #ifndef timersub /* XOPEN and __BSD don't cooperate well... */
 #define timersub(a, b, result)                                                \
   do {                                                                        \

Ok pay this email no mind. This was the first one I sent yesterday. Something must have happened with the IBM smtp server and caused my email to be held and outragously long time then sent. Sorry about this guys.

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

See volume 1, section 3.9.2 of the AMD64 architecture manual
(http://www.amd.com/us-en/Processors/DevelopWithAMD/0,,30_2252_875_7044,00.html),
quoted below for convenience.

This should be guaranteed to work:

#define rmb() __asm__ __volatile__ ( "mfence" : : : "memory" ) #define wmb() __asm__ __volatile__ ( "mfence" : : : "memory" )

but may be overkill. Are these macros used in any performance-critical situations? If not then the overkill wouldn't matter.

# 3.9.2 Forcing Memory Order
#
# Special instructions are provided for application software to force

# memory ordering in situations where such ordering is important. These
# instructions are:
#  * Load Fence -- The LFENCE instruction forces ordering of
#    memory loads (reads). All memory loads preceding the
#    LFENCE (in program order) are completed prior to
#    completing memory loads following the LFENCE. Memory
#    loads cannot be reordered around an LFENCE instruction,
#    but other non-serializing instructions (such as memory
#    writes) can be reordered around the LFENCE.
#  * Store Fence -- The SFENCE instruction forces ordering of
#    memory stores (writes). All memory stores preceding the
#    SFENCE (in program order) are completed prior to
#    completing memory stores following the SFENCE. Memory
#    stores cannot be reordered around an SFENCE instruction,
#    but other non-serializing instructions (such as memory
#    loads) can be reordered around the SFENCE.
#  * Memory Fence -- The MFENCE instruction forces ordering of
#    all memory accesses (reads and writes). All memory accesses
#    preceding the MFENCE (in program order) are completed
#    prior to completing any memory access following the
#    MFENCE. Memory accesses cannot be reordered around an
#    MFENCE instruction, but other non-serializing instructions
#    that do not access memory can be reordered around the
#    MFENCE.

David Hopwood <[email protected]>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

sfence and lfence would seem to be more appropriate. mfence would seem to be appropriate for an mb() macro (which is usually also defined--but in this case not necessary).

Not at all. They're currently used to prevent race conditions within the domain control channel ring queue. This queue is relatively low-traffic.

The confusion stemmed from the x86-32 code that doesn't actually implement a wmb() since no relevant architecture does out-of-order stores.

x86-64 is another story. The linux includes still have the same comments indicating that wmb() is not necessary (an obvious cut-and-paste job) even though wmb() is defined to sfence. To make matters worse, the wmb() is wrapped in the same CONFIG_UNORDERED_IO guarded (that's enabled by default in x86-64 but disabled by default in x86-32).

It's understandably confusing. I'm hoping we can do a much better job in Xen in supporting both architectures to avoid these sort of confusions.

Regards,
Anthony Liguori

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click