linux grundler
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 4 lines · Grant Grundler
Quoted 4 lines
Kudos to John David Anglin and Carlos O'Donnell for realizing PA 2.0 is not strongly ordered like PA1.x is.
Read appendix G or PA-RISC 2.0 Architecture (Gerry Kane) for details on "Memory Ordering Model".
I'm confused. This directly contradicts the comments in include/asm-parisc/system.h. If this needs updating, so does that.
Quoted 2 lines
Quoted 2 lines
I've uploaded palinux-20030208.tgz to dsl2 and will try to build a binutils.deb with this change as well.
Is all that's needed to take the latest binutils from debian unstable and rebuild it on woody?
Review CROSS_COMPILE := hppa-linux- · line 34
Quoted 5 lines
+++
arch/parisc/Makefile8 Feb 2003 06:21:32 -0000(quoted diff) +3
-0@@ -32,6 +32,10 @@ CROSS_COMPILE := hppa-linux- +ifdef CONFIG_PA20 +CFLAGS += -mpa-risc-2-0 +endif
I've reverted this one. We already have:
ifdef CONFIG_PA8X00
CFLAGS += -march=2.0 -mschedule=8000
endif
which covers the same cases.
Review CROSS_COMPILE := hppa-linux- · line 40
Quoted 6 lines
(quoted diff) +6
-0+++ include/asm-parisc/spinlock_t.h 8 Feb 2003 06:21:34 -0000 +> I've attached a summary of the change, but basically, for PA 2.0, as +> long as the ",CO" (coherent operation) completer is specified, then the +> 16-byte alignment requirement for ldcw and ldcd is relaxed, and instead +> they only require "natural" alignment (4-byte for ldcw, 8-byte for +> ldcd).
That's interesting from an architecture PoV. From my recollection when jsm was debugging problems on the 710, PCX-S is the only processor which actually enforces the 16-byte alignment restriction on ldcw. So _practically_, we don't need it unless we're supporting those old processors.
Review CROSS_COMPILE := hppa-linux-
Quoted 12 lines
(quoted diff) +11
-0+#ifdef CONFIG_PA20 +/* PA2.0 is not strongly ordered. ldcw enforces ordering + * and we need to make sure ordering is enforced on the unlock too. + */ +#define spin_unlock(x) \ + __asm__ __volatile__ ("stw,o %%sp,0(%0)" : : "r" (x) : "memory" ) +#else + +/* PA1.1 is strongly ordered. No issues here. */ #define spin_unlock(x) do { (x)->lock = 1; } while(0) + +#endif
Actually... this may be a long-standing bug in our spinlocks. There's nothing to prevent gcc reordering writes around this assignment. We need a barrier() before the assignment, or maybe it'd be as well to do the assignment in an asm() statement.
Quoted 1 line
#define
spin_unlock_wait(x)do {barrier(); }while(((volatile spinlock_t *)(x))->lock == 0)
i wonder if this one was working around some obscure compiler bug a while back. I don't see why we need to cast to a volatile spinlock_t * given that lock is already defined as volatile.
One final point.... up till now, we've been telling people it's OK to run kernels configured for PA1.1 on PA2.0 processors. This patch says to me that's not safe. Do we need our distros (yeah, I hear there'll soon be more than Debian supporting PA) to ship 5 flavours of kernel (PA1.1 UP & SMP, PA2.0 32-bit SMP, 64-bit UP and 64-bit SMP) rather than the current four?
Signature · 3 lines
"It's not Hollywood. War is real, war is primarily not about defeat or victory, it is about death. I've seen thousands and thousands of dead bodies. Do you think I want to have an academic debate on this subject?" -- Robert Fisk
Quoted 4 lines
That's interesting from an architecture PoV. From my recollection when jsm was debugging problems on the 710, PCX-S is the only processor which actually enforces the 16-byte alignment restriction on ldcw. So _practically_, we don't need it unless we're supporting those old processors.
I am pretty sure that any PA 1.x machine needs the alignment.
Quoted 6 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+#ifdef CONFIG_PA20 +/* PA2.0 is not strongly ordered. ldcw enforces ordering + * and we need to make sure ordering is enforced on the unlock too. + */ +#define spin_unlock(x) \ + __asm__ __volatile__ ("stw,o %%sp,0(%0)" : : "r" (x) : "memory" )
If you change the above to
__asm__ __volatile__ ("stw,ma %%sp,0(%0)" : : "r" (x) : "memory")
it should work on both PA11 and PA20. The ordered completer is only PA 2.
Quoted 6 lines
Actually... this may be a long-standing bug in our spinlocks. There's nothing to prevent gcc reordering writes around this assignment. We need a
barrier()before the assignment, or maybe it'd be as well to do the assignment in anasm()statement.Quoted 1 line
#define
spin_unlock_wait(x)do {barrier(); }while(((volatile spinlock_t *)(x))->lock == 0)
I think the volatile provides the barrier in my suggested version.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6605)
Quoted 1 line
I think the volatile provides the barrier in my suggested version.
volatile would prevent reordering accesses to a particular spinlock, but does it ensure that code that happens after the spinlock will not be reordered to happen before the spin-unlock?
for example, if we had:
volatile spinlock l1, l2;
lock(l1)
do something
unlock(l1)
lock(l2)
do something else
unlock(l2)
will gcc ever move the lock(l2) before the unlock(l1)?
(for reference, see this thread:
http://lists.debian.org/debian-gcc/2002/debian-gcc-200210/msg00058.html)
randolph
Signature · 3 lines
Randolph Chung Debian GNU/Linux Developer, hppa/ia64 ports http://www.tausq.org/
Quoted 9 lines
lock(l1)
do something
unlock(l1)lock(l2)
do something else
unlock(l2)will gcc ever move the lock(l2) before the unlock(l1)?
well, bad example, since l1 and l2 are both volatile, but in general, nothing will prevent part of "do something else" to happen inside the l1 lock, right?
randolph
Quoted 8 lines · Randolph Chung
Quoted 8 lines
lock(l1)
do something
unlock(l1)lock(l2)
do something else
unlock(l2)
Quoted 5 lines
Quoted 1 line
will gcc ever move the lock(l2) before the unlock(l1)?
well, bad example, since l1 and l2 are both volatile, but in general, nothing will prevent part of "do something else" to happen inside the l1 lock, right?
I think it's even worse than that. What stops gcc reordering:
typedef struct {
spinlock_t lock;
volatile int counter;
} rwlock_t;static __inline__ void _raw_read_lock(rwlock_t *rw)
{
while (__ldcw (&(x)->lock) == 0) \
while (((x)->lock) == 0) ; } while (0)
rw->counter++;
do { (x)->lock = 1; } while(0)
}to:
while (__ldcw (&(x)->lock) == 0) \
while (((x)->lock) == 0) ; } while (0)
do { (x)->lock = 1; } while(0)
rw->counter++;Signature · 3 lines
"It's not Hollywood. War is real, war is primarily not about defeat or victory, it is about death. I've seen thousands and thousands of dead bodies. Do you think I want to have an academic debate on this subject?" -- Robert Fisk
Quoted 15 lines
I think it's even worse than that. What stops gcc reordering:
typedef struct { spinlock_t lock; volatile int counter; } rwlock_t;static
__inline__void_raw_read_lock(rwlock_t *rw){ while (__ldcw (&(x)->lock) == 0) \ while (((x)->lock) == 0) ; } while (0) rw->counter++; do { (x)->lock = 1; } while(0) }
Quoted 6 lines
to:
while (__ldcw (&(x)->lock) == 0) \ while (((x)->lock) == 0) ; } while (0) do { (x)->lock = 1; } while(0) rw->counter++;
Nothing. The reset needs to be a volatile asm. This will stop GCC from doing the above. On PA 2.0, the processor can do similar reordering. So, the reset needs to be an ordered store. Well, I think the scheduling model tends to try to do things as early as possible, consistent with not over feeding the pipeline. However, I wouldn't rely on this to get the instrunction order that you want.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6605)
Quoted 15 lines · Matthew Wilcox
I think it's even worse than that. What stops gcc reordering:
typedef struct { spinlock_t lock; volatile int counter; } rwlock_t;static
__inline__void_raw_read_lock(rwlock_t *rw){ while (__ldcw (&(x)->lock) == 0) \ while (((x)->lock) == 0) ; } while (0) rw->counter++; do { (x)->lock = 1; } while(0) }
Quoted 6 lines
to:
while (__ldcw (&(x)->lock) == 0) \ while (((x)->lock) == 0) ; } while (0) do { (x)->lock = 1; } while(0) rw->counter++;
Compilers themselves have fairly strong reordering rules. For instance, do { } while() blocks cannot be reordered like your example (that's why the kernel uses do { } while(0); as a compiler reordering barrier.
You can if you prefer use the barrier() macro, which prevents the compiler from reordering statements around it.
Of course, the processor can still reorder what the compiler doesn't as part of its speculation and optimisation (that's what mb(), rmb() and wmb() are all about).
James
Quoted 8 lines
Quoted 8 lines
lock(l1)
do something
unlock(l1)lock(l2)
do something else
unlock(l2)
Quoted 5 lines
Quoted 1 line
will gcc ever move the lock(l2) before the unlock(l1)?
well, bad example, since l1 and l2 are both volatile, but in general, nothing will prevent part of "do something else" to happen inside the l1 lock, right?
Volatile would stop gcc from moving code across either the lock or the unlock. A blockage is in fact implemented as an UNSPEC_VOLATILE instruction. The PA 8000 has a large reorder buffer, so there are similar issues. It's guaranteed that instructions will appear to execute in order on a processor but the memory updates as viewed from another processor may occur out of order. To ensure that other processors have a consistent view of whats happening, one must use ordered instructions for the lock and unlock operations. This will force all stores and loads inside the lock to complete. Further, if cache flush operations occur inside a lock, you must also do a "sync" before unlocking to ensure that these operations complete. The ldcw insn is strongly ordered. A load and store on PA 2.0 can be made ordered with the correct completer. Cache flush operations are weakly ordered. On PA 1.x, all load and store instructions are strongly ordered, so ordering isn't an issue.
The current spinlock reset is done in high-level C. So, gcc can move code that that isn't dependent across the reset. PA 2.0 processors can also do the same.
Appendix G describes ordering in much more detail. Hopefully, I haven't butchered what's said there too badly.
The reordering that GCC does is conditional on instruction dependencies. I just reworked the handling of the PIC register restore in PIC code because all the PIC register were not apparent in the initial scheduling pass. This is in 3.3 and main, but not 3.2.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6605)
Quoted 9 lines · John David Anglin
Review
Quoted 2 lines
(quoted diff) +2
-0+#define spin_unlock(x) \ + __asm__ __volatile__ ("stw,o %%sp,0(%0)" : : "r" (x) : "memory" )If you change the above to
__asm____volatile__("stw,ma %%sp,0(%0)" : : "r" (x) : "memory")it should work on both PA11 and PA20. The ordered completer is only PA 2.
Excellent idea.
In case it's not obvious to others, "stw,o" is an alias for "stw,ma" with a Zero index value. But PA11 assembler will grok the stw,ma properly.
I'll look at getting this into the kernel this week if someone else doesn't beat me to it. I'd like to test what I've got a bit more now and get the binutils .deb availability issue resolved.
thanks,
grant
Quoted 8 lines
Quoted 6 lines
If you change the above to
__asm____volatile__("stw,ma %%sp,0(%0)" : : "r" (x) : "memory")it should work on both PA11 and PA20. The ordered completer is only PA 2.
Excellent idea.
The reason I am so up on this is that in testing gcc 3.4 on hppa1.1 this weekend I found a problem with the locking code in atomicity.h. This was the 16-byte alignment issue. In reviewing the assembly code being generated, I revisited how the lock reset was done and came up with the above.
This is also relevant to locking in glibc.
Dave
Signature · 2 lines
J. David Anglin [email protected] National Research Council of Canada (613) 990-0752 (FAX: 952-6605)
Quoted 2 lines · Matthew Wilcox
Is all that's needed to take the latest binutils from debian unstable and rebuild it on woody?
Almost. It needs a patch too:
ftp://ftp.parisc-linux.org/patches/900_order_hppa.diff
Just drop that in debian/patches before building the debs. That patch applies clean to the binutils for unstable. I don't even pretend to understand how to properly build debian packages, much less binutils or "cross release" builds. That's why I made the tarball for "testing".
Quoted 4 lines
That's interesting from an architecture PoV. From my recollection when jsm was debugging problems on the 710, PCX-S is the only processor which actually enforces the 16-byte alignment restriction on ldcw. So _practically_, we don't need it unless we're supporting those old processors.
I don't care to find out the hard way.
I'd rather just comply with the architecture and not worry about it. If someone can demonstrate a perf advantage or issue, I'll be more receptive.
Quoted 4 lines
Actually... this may be a long-standing bug in our spinlocks. There's nothing to prevent gcc reordering writes around this assignment. We need a
barrier()before the assignment, or maybe it'd be as well to do the assignment in anasm()statement.
I've read the followups to this and I gather our spinlocks are very broken. If someone tells me what the right fix is, I'll test on PA20 32/64 bit and commit.
Quoted 3 lines
One final point.... up till now, we've been telling people it's OK to run kernels configured for PA1.1 on PA2.0 processors. This patch says to me that's not safe.
Only for SMP. I think for UP the rule still holds.
Quoted 4 lines
Do we need our distros (yeah, I hear there'll
soon be more than Debian supporting PA) to ship 5 flavours of kernel (PA1.1 UP & SMP, PA2.0 32-bit SMP, 64-bit UP and 64-bit SMP) rather than the current four?
Unfortunately yes.
OTOH, PA20 SMP still hasn't proven stable so maybe it's not worth doing at the moment either. Once PA20 SMP is stable, we could drop the 64-bit UP kernels since most systems that *require* 64-bit are SMP.
thanks,
grant
Quoted 5 lines · Grant Grundler
Just drop that in debian/patches before building the debs. That patch applies clean to the binutils for unstable. I don't even pretend to understand how to properly build debian packages, much less binutils or "cross release" builds. That's why I made the tarball for "testing".
OK. It's not really hard, but I don't mind doing it. It's not like this is a regular occurrence. It's building now; when I've finished I'll upload it to unofficial-debs and people will be able to apt-get install it from there.
Quoted 4 lines
I don't care to find out the hard way.
I'd rather just comply with the architecture and not worry about it. If someone can demonstrate a perf advantage or issue, I'll be more receptive.
Sure. When someone's trying to implement futexes, this may prove critical..
Quoted 5 lines
Quoted 3 lines
One final point.... up till now, we've been telling people it's OK to run kernels configured for PA1.1 on PA2.0 processors. This patch says to me that's not safe.
Only for SMP. I think for UP the rule still holds.
Agree. That was implicit in the kernel list i gave later, but I should've stated it explicitly.
Quoted 5 lines
Unfortunately yes.
OTOH, PA20 SMP still hasn't proven stable so maybe it's not worth doing at the moment either. Once PA20 SMP is stable, we could drop the 64-bit UP kernels since most systems that *require* 64-bit are SMP.
Sure, but there's a measurable performance difference if you compile out spinlocks
Signature · 3 lines
"It's not Hollywood. War is real, war is primarily not about defeat or victory, it is about death. I've seen thousands and thousands of dead bodies. Do you think I want to have an academic debate on this subject?" -- Robert Fisk
Quoted 4 lines · Matthew Wilcox
OK. It's not really hard, but I don't mind doing it. It's not like this is a regular occurrence. It's building now; when I've finished I'll upload it to unofficial-debs and people will be able to apt-get install it from there.
done. http://ftp.parisc-linux.org/unofficial-debs/README
untested as i don't have root on any machine that's convenient.
Signature · 3 lines
"It's not Hollywood. War is real, war is primarily not about defeat or victory, it is about death. I've seen thousands and thousands of dead bodies. Do you think I want to have an academic debate on this subject?" -- Robert Fisk
Quoted 2 lines · Matthew Wilcox
I'll upload it to unofficial-debs and people will be able to apt-get install it from there.
cool - thanks.
Quoted 2 lines
Sure, but there's a measurable performance difference if you compile out spinlocks
if people care about the last 5% performance, they still have several options:
o build your own kernel
o run HPUX
I've not done or seen any lmbench perf results recently but before hpux was 10x faster on context switch/task related stuff. And X11 server takes advantage of HW acceleration.
(I'm associating UP systems with graphics which isn't always true)
grant
Quoted 4 lines · In another life Grant Grundler
Quoted 2 lines · Matthew Wilcox
I'll upload it to unofficial-debs and people will be able to apt-get install it from there.
Quoted 8 lines
cool - thanks.
Quoted 2 lines
Sure, but there's a measurable performance difference if you compile out spinlocks
if people care about the last 5% performance, they still have several options:
o build your own kernel
As far as you know am I the only person, that has reported not being able to boot a 2.4.20-pa?? kernel compiled either natively or cross ?
By the way, I tried pa24 today, though I haven't built it natively yet I cross compiled it to no avail...
Quoted 12 lines
o run HPUX
I've not done or seen any lmbench perf results recently but before hpux was 10x faster on context switch/task related stuff. And X11 server takes advantage of HW acceleration.
(I'm associating UP systems with graphics which isn't always true)grant
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
Signature · 5 lines
Aaron St. Pierre tel: 978.828.6177 [email protected] Either I'm dead or my watch has stopped. -- Groucho Marx's last words
Quoted 7 lines · Aaron St. Pierre
Quoted 3 lines · In another life Grant Grundler
if people care about the last 5% performance, they still have several options:
o build your own kernelAs far as you know am I the only person, that has reported not being able to boot a 2.4.20-pa?? kernel compiled either natively or cross ?
yes. So it's not an option for you.
Quoted 4 lines
Quoted 4 lines
o run HPUX
I've not done or seen any lmbench perf results recently but before hpux was 10x faster on context switch/task related stuff. And X11 server
Seems I remembered this partially wrong.
Linux was slightly faster in lots of areas, ~5x slower on fork/exec/shi, and ~10x slower mmap latency.
http://lists.parisc-linux.org/pipermail/parisc-linux/2002-March/015966.html
grant
Quoted 1 line
I've not done or seen any lmbench perf results recently but before hpux
well, we can fix that... :) Here are the numbers for a 2x440Mhz A500 running 2.4.20-pa24 SMP.
btw, if you look at
http://lists.parisc-linux.org/pipermail/parisc-linux/2002-April/015984.html
we are only 2-3x slower than hpux :-)
I'm a bit confused about the "mmap latency" numbers. why are they so high?
Related to this, thibaut and I were experimenting with running dbench on two A500s today running identical kernels, but one is 2x440MHz and one is 2x550MHz. The 550MHz is almost 10x faster. Is that expected? Memory/disk configurations (3GB RAM in 440MHz, 2.5GB in 550MHz, etc) are not exactly the same, but I wouldn't have expected them to differ by that much.
randolph
L M B E N C H 2 . 0 S U M M A R Y
------------------------------------
Basic system parameters
----------------------------------------------------
Host OS Description Mhz
--------- ------------- ----------------------- ----
ios Linux 2.4.20- hppa64-linux-gnu 440
Processor, Processes - times in microseconds - smaller is better
----------------------------------------------------------------
Host OS Mhz null null open selct sig sig fork exec sh
call I/O stat clos TCP inst hndl proc proc proc
--------- ------------- ---- ---- ---- ---- ---- ----- ---- ---- ---- ---- ----
ios Linux 2.4.20- 440 0.73 1.85 10.2 11.8 110.8 2.17 16.9 17.K 41.K 85.K
Context switching - times in microseconds - smaller is better
-------------------------------------------------------------
Host OS 2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw
--------- ------------- ----- ------ ------ ------ ------ ------- -------
ios Linux 2.4.20- 5.630 5.1500 4.9900 9.0800 23.4 52.1 175.8
*Local* Communication latencies in microseconds - smaller is better
-------------------------------------------------------------------
Host OS 2p/0K Pipe AF UDP RPC/ TCP RPC/ TCP
ctxsw UNIX UDP TCP conn
--------- ------------- ----- ----- ---- ----- ----- ----- ----- ----
ios Linux 2.4.20- 5.630 20.4 48.4 89.2 172.5 114.1 351.5 1000
File & VM system latencies in microseconds - smaller is better
--------------------------------------------------------------
Host OS 0K File 10K File Mmap Prot Page
Create Delete Create Delete Latency Fault Fault
--------- ------------- ------ ------ ------ ------ ------- ----- -----
ios Linux 2.4.20- 130.0 80.0 509.9 180.0 3120.0K 603.4 58.0*Local* Communication bandwidths in MB/s - bigger is better
-----------------------------------------------------------
Host OS Pipe AF TCP File Mmap Bcopy Bcopy Mem Mem
UNIX reread reread (libc) (hand) read write
--------- ------------- ---- ---- ---- ------ ------ ------ ------ ---- -----
ios Linux 2.4.20- 49.1 81.0 33.6 74.0 268.4 201.8 168.8 260. 250.9
Memory latencies in nanoseconds - smaller is better
(WARNING - may not be correct, check graphs)
---------------------------------------------------
Host OS Mhz L1 $ L2 $ Main mem Guesses --------- ------------- ---- ----- ------ -------- -------
ios Linux 2.4.20- 440 20.0 20.0 220.0 No L1 cache?
Quoted 2 lines · Matthew Wilcox
Is all that's needed to take the latest binutils from debian unstable and rebuild it on woody?
I've rebuilt "unstable" binutils + ldwa,o patch on "testing" and dropped them on
ftp://ftp.parisc-linux.org/unofficial-debs/
-rw-r--r-- 1 grundler ftpadmin 478700 Feb 8 23:51 binutils-dev_2.13.90.0.16-1_hppa.deb -rw-r--r-- 1 grundler ftpadmin 3311272 Feb 8 23:53 binutils-multiarch_2.13.90.0.16-1_hppa.deb -rw-r--r-- 1 grundler ftpadmin 2307548 Feb 8 23:54 binutils_2.13.90.0.16-1_hppa.deb
They install fine on my c3k though TBH, I haven't tried to use them. (I did use the tarballs before posting earlier).
If someone wants to repeat that exercise for woody, that would be fine with me.
/me checks off an item from his TODO list
Apperently, we still have problems with timers or timer code on PA20 SMP. Still haven't been able to isolate this to any peice of code. :^(
g'night,
grant
Review
Quoted 7 lines · Grant Grundler
I've rebuilt "unstable" binutils + ldwa,o patch on "testing" and dropped them on
ftp://ftp.parisc-linux.org/unofficial-debs/(quoted diff) +0
-3-rw-r--r-- 1 grundler ftpadmin 478700 Feb 8 23:51 binutils-dev_2.13.90.0.16-1_hppa.deb -rw-r--r-- 1 grundler ftpadmin 3311272 Feb 8 23:53 binutils-multiarch_2.13.90.0.16-1_hppa.deb -rw-r--r-- 1 grundler ftpadmin 2307548 Feb 8 23:54 binutils_2.13.90.0.16-1_hppa.deb
I've removed these. Few problems:
- Same version number as the unstable package, so an upgrade wouldn't replace them
- Not listed in the Packages file.
- debian/changelog not updated
I'm doing a fresh build now.
Signature · 3 lines
"It's not Hollywood. War is real, war is primarily not about defeat or victory, it is about death. I've seen thousands and thousands of dead bodies. Do you think I want to have an academic debate on this subject?" -- Robert Fisk