Skip to contentSkip to conversation

parisc: Rewrite light-weight syscall and futex code

PATCH Superseded John David Anglin 2 participants 10 messages Dec 23, 2021 PA-RISC architecture

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 parisc: Rewrite light-weight syscall and futex code

The parisc architecture lacks general hardware support for compare and swap. Particularly for userspace, it is difficult to implement software atomic support. Page faults in critical regions can cause processes to sleep and block the forward progress of other processes. Thus, it is essential that page faults be disabled in critical regions. For performance reasons, we also need to disable external interrupts in critical regions.

In order to do this, we need a mechanism to trigger COW breaks outside the critical region. Fortunately, parisc has the "stbys,e" instruction. When the leftmost byte of a word is addressed, this instruction triggers all the exceptions of a normal store but it does not write to memory. Thus, we can use it to trigger COW breaks outside the critical region without modifying the data that is to be updated atomically.

COW breaks occur randomly. So even if we have priviously executed a "stbys,e" instruction, we still need to disable pagefaults around the critical region. If a fault occurs in the critical region, we return -EAGAIN. I had to add a wrapper around _arch_futex_atomic_op_inuser() as I found in testing that returning -EAGAIN caused problems for some processes even though it is listed as a possible return value.

The patch implements the above. The code no longer attempts to sleep with interrupts disabled and I haven't seen any stalls with the change.

I have attempted to merge common code and streamline the fast path. In the futex code, we only compute the spinlock address once.

I eliminated some debug code in the original CAS routine that just made the flow more complicated.

I don't clip the arguments when called from wide mode. As a result, the LWS routines should work when called from 64-bit processes.

I defined TASK_PAGEFAULT_DISABLED offset for use in the lws_pagefault_disable and lws_pagefault_enable macros.

Since we now disable interrupts on the gateway page where necessary, it might be possible to allow processes to be scheduled when they are on the gateway page.

Change has been tested on c8000 and rp3440. It improves glibc build and test time by about 10%.

In v2, I removed the lws_atomic_xchg and and lws_atomic_store calls. I also removed the bug fixes that were not directly related to this patch.

---

arch/parisc/include/asm/futex.h +79 -24
index 9cd4dd6e63ad..8f97db995b04 100644
@@ -8,39 +8,74 @@
 #include <asm/errno.h>
 
 /* The following has to match the LWS code in syscall.S.  We have
-   sixteen four-word locks. */
+ * 256 four-word locks. We use bits 20-27 of the futex virtual
+ * address for the hash index.
+ */
+
+static inline unsigned long _futex_hash_index(unsigned long ua)
+{
+	return (ua >> 2) & 0x3fc;
+}
+
+/* Force store interruptions without writing anything. Return true
+ * if load and store fault.
+ */
+
+static inline bool _futex_force_interruptions(unsigned long ua)
+{
+	bool result;
+
+	__asm__ __volatile__(
+		"1:\tldw 0(%1), %0\n"
+		"2:\tstbys,e %%r0, 0(%1)\n"
+		"\tcomclr,= %%r0, %%r0, %0\n"
+		"3:\tldi 1, %0\n"
+		ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
+		ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
+		: "=&r" (result) : "r" (ua) : "memory"
+	);
+	return result;
+}
 
 static inline void
-_futex_spin_lock(u32 __user *uaddr)
+_futex_spin_lock_irqsave(arch_spinlock_t *s, unsigned long *flags)
 {
-	extern u32 lws_lock_start[];
-	long index = ((long)uaddr & 0x7f8) >> 1;
-	arch_spinlock_t *s = (arch_spinlock_t *)&lws_lock_start[index];
-	preempt_disable();
+	local_irq_save(*flags);
 	arch_spin_lock(s);
 }
 
 static inline void
-_futex_spin_unlock(u32 __user *uaddr)
+_futex_spin_unlock_irqrestore(arch_spinlock_t *s, unsigned long *flags)
 {
-	extern u32 lws_lock_start[];
-	long index = ((long)uaddr & 0x7f8) >> 1;
-	arch_spinlock_t *s = (arch_spinlock_t *)&lws_lock_start[index];
 	arch_spin_unlock(s);
-	preempt_enable();
+	local_irq_restore(*flags);
 }
 
 static inline int
-arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
+_arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
 {
+	extern u32 lws_lock_start[];
+	unsigned long ua = (unsigned long)uaddr;
+	arch_spinlock_t *s;
+	unsigned long flags;
 	int oldval, ret;
 	u32 tmp;
 
-	ret = -EFAULT;
+	/* Force interruptions and page faults */
+	if (_futex_force_interruptions(ua))
+		return -EFAULT;
 
-	_futex_spin_lock(uaddr);
-	if (unlikely(get_user(oldval, uaddr) != 0))
+	s = (arch_spinlock_t *)&lws_lock_start[_futex_hash_index(ua)];
+
+	/* Don't sleep */
+	pagefault_disable();
+	_futex_spin_lock_irqsave(s, &flags);
+
+	/* Return -EAGAIN if we encounter page fault contention */
+	if (unlikely(get_user(oldval, uaddr) != 0)) {
+		ret = -EAGAIN;
 		goto out_pagefault_enable;
+	}
 
 	ret = 0;
 	tmp = oldval;
@@ -63,13 +98,15 @@ arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
 		break;
 	default:
 		ret = -ENOSYS;
+		goto out_pagefault_enable;
 	}
 
-	if (ret == 0 && unlikely(put_user(tmp, uaddr) != 0))
-		ret = -EFAULT;
+	if (unlikely(put_user(tmp, uaddr) != 0))
+		ret = -EAGAIN;
 
 out_pagefault_enable:
-	_futex_spin_unlock(uaddr);
+	_futex_spin_unlock_irqrestore(s, &flags);
+	pagefault_enable();
 
 	if (!ret)
 		*oval = oldval;
@@ -77,11 +114,27 @@ arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
 	return ret;
 }
 
+static inline int
+arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
+{
+	int ret, cnt = 0;
+
+	/* Avoid returning -EAGAIN */
+	do {
+		ret = _arch_futex_atomic_op_inuser(op, oparg, oval, uaddr);
+	} while (ret == -EAGAIN && cnt++ < 4);
+	return ret == -EAGAIN ? -EFAULT : ret;
+}
+
 static inline int
 futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr,
 			      u32 oldval, u32 newval)
 {
+	extern u32 lws_lock_start[];
+	unsigned long ua = (unsigned long)uaddr;
+	arch_spinlock_t *s;
 	u32 val;
+	unsigned long flags;
 
 	/* futex.c wants to do a cmpxchg_inatomic on kernel NULL, which is
 	 * our gateway page, and causes no end of trouble...
@@ -94,23 +147,25 @@ futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr,
 
 	/* HPPA has no cmpxchg in hardware and therefore the
 	 * best we can do here is use an array of locks. The
-	 * lock selected is based on a hash of the userspace
-	 * address. This should scale to a couple of CPUs.
+	 * lock selected is based on a hash of the virtual
+	 * address of the futex. This should scale to a couple
+	 * of CPUs.
 	 */
 
-	_futex_spin_lock(uaddr);
+	s = (arch_spinlock_t *)&lws_lock_start[_futex_hash_index(ua)];
+	_futex_spin_lock_irqsave(s, &flags);
 	if (unlikely(get_user(val, uaddr) != 0)) {
-		_futex_spin_unlock(uaddr);
+		_futex_spin_unlock_irqrestore(s, &flags);
 		return -EFAULT;
 	}
 
 	if (val == oldval && unlikely(put_user(newval, uaddr) != 0)) {
-		_futex_spin_unlock(uaddr);
+		_futex_spin_unlock_irqrestore(s, &flags);
 		return -EFAULT;
 	}
 
 	*uval = val;
-	_futex_spin_unlock(uaddr);
+	_futex_spin_unlock_irqrestore(s, &flags);
 
 	return 0;
 }
arch/parisc/kernel/asm-offsets.c +1 -0
index 55c1c5189c6a..396aa3b47712 100644
@@ -37,6 +37,7 @@ int main(void)
 {
 	DEFINE(TASK_TI_FLAGS, offsetof(struct task_struct, thread_info.flags));
 	DEFINE(TASK_STACK, offsetof(struct task_struct, stack));
+	DEFINE(TASK_PAGEFAULT_DISABLED, offsetof(struct task_struct, pagefault_disabled));
 	BLANK();
 	DEFINE(TASK_REGS, offsetof(struct task_struct, thread.regs));
 	DEFINE(TASK_PT_PSW, offsetof(struct task_struct, thread.regs.gr[ 0]));
arch/parisc/kernel/syscall.S +192 -190

562 lines not on this page · raw message

maintainer
#1 lore raw

...

Quoted 6 lines

In order to do this, we need a mechanism to trigger COW breaks outside the critical region. Fortunately, parisc has the "stbys,e" instruction. When the leftmost byte of a word is addressed, this instruction triggers all the exceptions of a normal store but it does not write to memory. Thus, we can use it to trigger COW breaks outside the critical region without modifying the data that is to be updated atomically.

...

Quoted 4 lines
arch/parisc/include/asm/futex.h +0 -0
index 9cd4dd6e63ad..8f97db995b04 100644

...

Quoted 4 lines
arch/parisc/include/asm/futex.h +4 -0
+static inline bool _futex_force_interruptions(unsigned long ua)
+{
+	bool result;
+

arch/parisc/include/asm/futex.h

Quoted 10 lines
arch/parisc/include/asm/futex.h +10 -0
+	__asm__ __volatile__(
+		"1:\tldw 0(%1), %0\n"
+		"2:\tstbys,e %%r0, 0(%1)\n"
+		"\tcomclr,= %%r0, %%r0, %0\n"
+		"3:\tldi 1, %0\n"
+		ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
+		ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
+		: "=&r" (result) : "r" (ua) : "memory"
+	);
+	return result;

I wonder if we can get rid of the comclr,= instruction by using ASM_EXCEPTIONTABLE_ENTRY_EFAULT instead of ASM_EXCEPTIONTABLE_ENTRY, e.g.:

arch/parisc/include/asm/futex.h +11 -10
index 8f97db995b04..ea052f013865 100644
@@ -21,20 +21,21 @@ static inline unsigned long _futex_hash_index(unsigned long ua)
  * if load and store fault.
  */
 
-static inline bool _futex_force_interruptions(unsigned long ua)
+static inline unsigned long _futex_force_interruptions(unsigned long ua)
 {
-	bool result;
+	register unsigned long error __asm__ ("r8") = 0;
+	register unsigned long temp;
 
 	__asm__ __volatile__(
-		"1:\tldw 0(%1), %0\n"
-		"2:\tstbys,e %%r0, 0(%1)\n"
-		"\tcomclr,= %%r0, %%r0, %0\n"
-		"3:\tldi 1, %0\n"
-		ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
-		ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
-		: "=&r" (result) : "r" (ua) : "memory"
+		"1:\tldw 0(%2), %0\n"
+		"2:\tstbys,e %%r0, 0(%2)\n"
+		"3:\n"
+		ASM_EXCEPTIONTABLE_ENTRY_EFAULT(1b, 3b)
+		ASM_EXCEPTIONTABLE_ENTRY_EFAULT(2b, 3b)
+		: "=r" (temp), "=r" (error)
+		: "r" (ua), "1" (error) : "memory"
 	);
-	return result;
+	return error;
 }
 
 

Helge

#2 lore raw
Quoted 55 lines · Helge Deller

...

Quoted 6 lines

In order to do this, we need a mechanism to trigger COW breaks outside the critical region. Fortunately, parisc has the "stbys,e" instruction. When the leftmost byte of a word is addressed, this instruction triggers all the exceptions of a normal store but it does not write to memory. Thus, we can use it to trigger COW breaks outside the critical region without modifying the data that is to be updated atomically.

...

Quoted 4 lines
arch/parisc/include/asm/futex.h +0 -0
index 9cd4dd6e63ad..8f97db995b04 100644

...

Quoted 4 lines
arch/parisc/include/asm/futex.h +4 -0
+static inline bool _futex_force_interruptions(unsigned long ua)
+{
+	bool result;
+

arch/parisc/include/asm/futex.h

Quoted 10 lines
arch/parisc/include/asm/futex.h +10 -0
+	__asm__ __volatile__(
+		"1:\tldw 0(%1), %0\n"
+		"2:\tstbys,e %%r0, 0(%1)\n"
+		"\tcomclr,= %%r0, %%r0, %0\n"
+		"3:\tldi 1, %0\n"
+		ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
+		ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
+		: "=&r" (result) : "r" (ua) : "memory"
+	);
+	return result;

I wonder if we can get rid of the comclr,= instruction by using ASM_EXCEPTIONTABLE_ENTRY_EFAULT instead of ASM_EXCEPTIONTABLE_ENTRY, e.g.:

arch/parisc/include/asm/futex.h +4 -9
index 8f97db995b04..ea052f013865 100644
@@ -21,20 +21,21 @@ static inline unsigned long _futex_hash_index(unsigned long ua)
   * if load and store fault.
   */
 
-static inline bool _futex_force_interruptions(unsigned long ua)
+static inline unsigned long _futex_force_interruptions(unsigned long ua)
  {
-	bool result;
+	register unsigned long error __asm__ ("r8") = 0;
+	register unsigned long temp;
 
  	__asm__ __volatile__(
-		"1:\tldw 0(%1), %0\n"
-		"2:\tstbys,e %%r0, 0(%1)\n"
-		"\tcomclr,= %%r0, %%r0, %0\n"
-		"3:\tldi 1, %0\n"
-		ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
-		ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
-		: "=&r" (result) : "r" (ua) : "memory"
+		"1:\tldw 0(%2), %0\n"

arch/parisc/include/asm/futex.h static inline unsigned long _futex_hash_index(unsigned long ua) · line 39

Quoted 10 lines
arch/parisc/include/asm/futex.h +7 -1
+		"2:\tstbys,e %%r0, 0(%2)\n"
+		"3:\n"
+		ASM_EXCEPTIONTABLE_ENTRY_EFAULT(1b, 3b)
+		ASM_EXCEPTIONTABLE_ENTRY_EFAULT(2b, 3b)
+		: "=r" (temp), "=r" (error)
+		: "r" (ua), "1" (error) : "memory"
  	);
-	return result;
+	return error;
  }

I don't think this is a win.

1) Register %r8 needs to get loaded with 0. So, that's one instruction.
2) Register %r8 is a caller saves register. Using it will cause gcc to save and restore it from stack. This may cause a stack frame to be created when one isn't needed. The save and restore instructions are more expensive, particularly if they cause a TLB miss.

Note that the comclr both clears result and nullifies the following ldi instruction, so it is not executed in the fast path.

Dave

Signature · 1 line
John David Anglin  [email protected]
maintainer
#3 lore raw
Quoted 68 lines · John David Anglin
Quoted 57 lines · Helge Deller

...

Quoted 6 lines

In order to do this, we need a mechanism to trigger COW breaks outside the critical region. Fortunately, parisc has the "stbys,e" instruction. When the leftmost byte of a word is addressed, this instruction triggers all the exceptions of a normal store but it does not write to memory. Thus, we can use it to trigger COW breaks outside the critical region without modifying the data that is to be updated atomically.

...

Quoted 4 lines
arch/parisc/include/asm/futex.h +0 -0
index 9cd4dd6e63ad..8f97db995b04 100644

...

Quoted 4 lines
arch/parisc/include/asm/futex.h +4 -0
+static inline bool _futex_force_interruptions(unsigned long ua)
+{
+    bool result;
+

arch/parisc/include/asm/futex.h

Quoted 10 lines
arch/parisc/include/asm/futex.h +10 -0
+    __asm__ __volatile__(
+        "1:\tldw 0(%1), %0\n"
+        "2:\tstbys,e %%r0, 0(%1)\n"
+        "\tcomclr,= %%r0, %%r0, %0\n"
+        "3:\tldi 1, %0\n"
+        ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
+        ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
+        : "=&r" (result) : "r" (ua) : "memory"
+    );
+    return result;

I wonder if we can get rid of the comclr,= instruction by using ASM_EXCEPTIONTABLE_ENTRY_EFAULT instead of ASM_EXCEPTIONTABLE_ENTRY, e.g.:

arch/parisc/include/asm/futex.h +3 -2
index 8f97db995b04..ea052f013865 100644
@@ -21,20 +21,21 @@ static inline unsigned long _futex_hash_index(unsigned long ua)
    * if load and store fault.
    */
 
-static inline bool _futex_force_interruptions(unsigned long ua)
+static inline unsigned long _futex_force_interruptions(unsigned long ua)
   {
-    bool result;
+    register unsigned long error __asm__ ("r8") = 0;
+    register unsigned long temp;
 

      __asm__ __volatile__(
-        "1:\tldw 0(%1), %0\n"
-        "2:\tstbys,e %%r0, 0(%1)\n"
-        "\tcomclr,= %%r0, %%r0, %0\n"
-        "3:\tldi 1, %0\n"
-        ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
-        ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
-        : "=&r" (result) : "r" (ua) : "memory"
+        "1:\tldw 0(%2), %0\n"
+        "2:\tstbys,e %%r0, 0(%2)\n"
+        "3:\n"

Quoted 8 lines

+        ASM_EXCEPTIONTABLE_ENTRY_EFAULT(1b, 3b)
+        ASM_EXCEPTIONTABLE_ENTRY_EFAULT(2b, 3b)
+        : "=r" (temp), "=r" (error)
+        : "r" (ua), "1" (error) : "memory"
      );
-    return result;
+    return error;
  }

I don't think this is a win.

Quoted 1 line

1) Register %r8 needs to get loaded with 0. So, that's one instruction.

True. On the other hand you don't need the "ldi 1,%0" then, which brings parity.

Quoted 3 lines

2) Register %r8 is a caller saves register. Using it will cause gcc to save and restore it from stack. This may cause a stack frame to be created when one isn't needed. The save and restore instructions are more expensive, particularly if they cause a TLB miss.

Because of this reason, wouln't it make sense to switch the uaccess functions away from r8 too, and use another temp register in both?

Quoted 1 line

Note that the comclr both clears result and nullifies the following ldi instruction, so it is not executed in the fast path.

Yes, but when emulating with qemu, it generates comparism and jumps, while the loading of r8 (see 1)), is trivial.

If we change the uaccess functions away from r8, then we can drop comclr (and save one instruction).

Helge

maintainer
#4 lore raw
Quoted 77 lines · Helge Deller
Quoted 68 lines · John David Anglin
Quoted 57 lines · Helge Deller

...

Quoted 6 lines

In order to do this, we need a mechanism to trigger COW breaks outside the critical region. Fortunately, parisc has the "stbys,e" instruction. When the leftmost byte of a word is addressed, this instruction triggers all the exceptions of a normal store but it does not write to memory. Thus, we can use it to trigger COW breaks outside the critical region without modifying the data that is to be updated atomically.

...

Quoted 4 lines
arch/parisc/include/asm/futex.h +0 -0
index 9cd4dd6e63ad..8f97db995b04 100644

...

Quoted 4 lines
arch/parisc/include/asm/futex.h +4 -0
+static inline bool _futex_force_interruptions(unsigned long ua)
+{
+    bool result;
+

arch/parisc/include/asm/futex.h

Quoted 10 lines
arch/parisc/include/asm/futex.h +10 -0
+    __asm__ __volatile__(
+        "1:\tldw 0(%1), %0\n"
+        "2:\tstbys,e %%r0, 0(%1)\n"
+        "\tcomclr,= %%r0, %%r0, %0\n"
+        "3:\tldi 1, %0\n"
+        ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
+        ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
+        : "=&r" (result) : "r" (ua) : "memory"
+    );
+    return result;

I wonder if we can get rid of the comclr,= instruction by using ASM_EXCEPTIONTABLE_ENTRY_EFAULT instead of ASM_EXCEPTIONTABLE_ENTRY, e.g.:

arch/parisc/include/asm/futex.h +3 -2
index 8f97db995b04..ea052f013865 100644
@@ -21,20 +21,21 @@ static inline unsigned long _futex_hash_index(unsigned long ua)
    * if load and store fault.
    */
 
-static inline bool _futex_force_interruptions(unsigned long ua)
+static inline unsigned long _futex_force_interruptions(unsigned long ua)
   {
-    bool result;
+    register unsigned long error __asm__ ("r8") = 0;
+    register unsigned long temp;
 

      __asm__ __volatile__(
-        "1:\tldw 0(%1), %0\n"
-        "2:\tstbys,e %%r0, 0(%1)\n"
-        "\tcomclr,= %%r0, %%r0, %0\n"
-        "3:\tldi 1, %0\n"
-        ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
-        ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
-        : "=&r" (result) : "r" (ua) : "memory"
+        "1:\tldw 0(%2), %0\n"
+        "2:\tstbys,e %%r0, 0(%2)\n"
+        "3:\n"

Quoted 8 lines

+        ASM_EXCEPTIONTABLE_ENTRY_EFAULT(1b, 3b)
+        ASM_EXCEPTIONTABLE_ENTRY_EFAULT(2b, 3b)
+        : "=r" (temp), "=r" (error)
+        : "r" (ua), "1" (error) : "memory"
      );
-    return result;
+    return error;
  }

I don't think this is a win.

Quoted 1 line

1) Register %r8 needs to get loaded with 0. So, that's one instruction.

True. On the other hand you don't need the "ldi 1,%0" then, which brings parity.

Quoted 3 lines

2) Register %r8 is a caller saves register. Using it will cause gcc to save and restore it from stack. This may cause a stack frame to be created when one isn't needed. The save and restore instructions are more expensive, particularly if they cause a TLB miss.

Quoted 8 lines

Because of this reason, wouln't it make sense to switch the uaccess functions away from r8 too, and use another temp register in both?

Quoted 1 line

Note that the comclr both clears result and nullifies the following ldi instruction, so it is not executed in the fast path.

Yes, but when emulating with qemu, it generates comparism and jumps, while the loading of r8 (see 1)), is trivial.

If we change the uaccess functions away from r8, then we can drop comclr (and save one instruction).

Independend of your patch I think we should go away in uacces from r8. I did some testing (on 32bit kernel) and r29 seems good. Opinion?

When changing from r8 to r22:

# scripts/bloat-o-meter vmlinux1 vmlinux
add/remove: 0/0 grow/shrink: 23/85 up/down: 216/-964 (-748)
...

and when changing from r8 to r29:

# scripts/bloat-o-meter vmlinux1 vmlinux
add/remove: 0/0 grow/shrink: 23/86 up/down: 228/-980 (-752)
Function                                     old     new   delta
tty_mode_ioctl                              1468    1512     +44
fat_generic_ioctl                           1428    1464     +36
__receive_fd                                 260     276     +16
vt_do_kdsk_ioctl                             880     892     +12
futex_requeue                               2972    2984     +12
do_tcp_getsockopt.constprop                 4724    4736     +12
blkdev_ioctl                                2644    2656     +12
vt_do_diacrit                               1000    1008      +8
tty_jobctrl_ioctl                           1060    1068      +8
tioclinux                                    700     708      +8
sg_read                                     1256    1264      +8
scsi_ioctl                                  2004    2012      +8
uart_ioctl                                  1468    1472      +4
sys_sendfile64                               204     208      +4
sys_sendfile                                 188     192      +4
sys_rt_sigreturn                             288     292      +4
st_ioctl                                    5436    5440      +4
rtc_dev_read                                 412     416      +4
ipv6_get_msfilter                            252     256      +4
ip_get_mcast_msfilter                        196     200      +4
futex_unlock_pi                              756     760      +4
fillonedir                                   276     280      +4
dev_ifconf                                   256     260      +4
vt_ioctl                                    4084    4080      -4
sg_ioctl                                    2120    2116      -4
n_tty_ioctl                                  316     312      -4
handle_futex_death.part                      404     400      -4
do_ipv6_getsockopt.constprop                2068    2064      -4
count.constprop                              248     244      -4
autofs_root_ioctl                            576     572      -4
write_sysrq_trigger                           96      88      -8
write_port                                   220     212      -8
sys_utime32                                  132     124      -8
sys_time32                                    80      72      -8
sys_settimeofday                             208     200      -8
sys_pselect6_time32                          320     312      -8
sys_pselect6                                 320     312      -8
sys_name_to_handle_at                        488     480      -8
sys_msgsnd                                    80      72      -8
sys_ioctl                                   2396    2388      -8
sys_io_submit                               2944    2936      -8
sys_gettimeofday                             160     152      -8
sys_getresgid                                124     116      -8
sys_getgroups                                140     132      -8
sys_getdents64                               288     280      -8
sys_getdents                                 280     272      -8
sys_getcpu                                   112     104      -8
sock_getsockopt                             2476    2468      -8
serport_ldisc_ioctl                           88      80      -8
schedule_tail                                100      92      -8
raw_getsockopt                               196     188      -8
proc_bus_pci_write                           604     596      -8
proc_bus_pci_read                            624     616      -8
pipe_ioctl                                   184     176      -8
move_addr_to_user                            156     148      -8
mmc_ioctl_cdrom_last_written                  84      76      -8
mm_release                                   180     172      -8
load_elf_binary                             4336    4328      -8
ksys_msgsnd                                   80      72      -8
kpagecount_read                              552     544      -8
kpagecgroup_read                             424     416      -8
kernel_wait4                                 304     296      -8
ip_mc_msfget                                 472     464      -8
generic_ptrace_peekdata                      100      92      -8
futex_wait_setup                             248     240      -8
futex_get_value_locked                        92      84      -8
fat_dir_ioctl                                340     332      -8
do_signal.part                               332     324      -8
do_signal                                   1144    1136      -8
do_recvmmsg                                  632     624      -8
do_msg_fill                                  108     100      -8
do_compat_futimesat                          204     196      -8
check_syscallno_in_delay_branch.constprop     172     164      -8
autofs_expire_multi                           76      68      -8
__sys_socketpair                             560     552      -8
__sys_sendmmsg                               396     388      -8
__save_altstack                               92      84      -8
udp_ioctl                                    168     156     -12
tcp_ioctl                                    512     500     -12
sys_waitid                                   312     300     -12
sys_setgroups                                484     472     -12
sys_getresuid                                140     128     -12
sys_get_robust_list                          220     208     -12
sys_capset                                   460     448     -12
sys_capget                                   344     332     -12
pty_set_lock                                 224     212     -12
futex_cmpxchg_value_locked                   216     204     -12
fpr_set                                      240     228     -12
fault_in_writeable                           176     164     -12
fault_in_readable                            188     176     -12
copy_to_kernel_nofault                       352     340     -12
copy_from_kernel_nofault                     400     388     -12
strncpy_from_kernel_nofault                  284     268     -16
put_cmsg                                     340     324     -16
packet_ioctl                                 344     328     -16
check_zeroed_user                            220     204     -16
cap_validate_magic                           336     320     -16
strncpy_from_user                            484     464     -20
random_ioctl                                 532     512     -20
do_epoll_wait                               1704    1684     -20
vt_do_kdskled                                632     608     -24
unix_ioctl                                   484     460     -24
read_port                                    208     184     -24
ns_ioctl                                     260     236     -24
ata_sas_scsi_ioctl                           704     680     -24
pty_bsd_ioctl                                288     260     -28
arch_ptrace                                  524     496     -28
pty_unix98_ioctl                             312     280     -32
copy_process                                6364    6316     -48
Total: Before=8913722, After=8912970, chg -0.01%
#5 lore raw
Quoted 79 lines · Helge Deller
Quoted 78 lines · Helge Deller
Quoted 68 lines · John David Anglin
Quoted 55 lines · Helge Deller

...

Quoted 6 lines

In order to do this, we need a mechanism to trigger COW breaks outside the critical region. Fortunately, parisc has the "stbys,e" instruction. When the leftmost byte of a word is addressed, this instruction triggers all the exceptions of a normal store but it does not write to memory. Thus, we can use it to trigger COW breaks outside the critical region without modifying the data that is to be updated atomically.

...

Quoted 4 lines
arch/parisc/include/asm/futex.h +0 -0
index 9cd4dd6e63ad..8f97db995b04 100644

...

Quoted 4 lines
arch/parisc/include/asm/futex.h +4 -0
+static inline bool _futex_force_interruptions(unsigned long ua)
+{
+    bool result;
+

arch/parisc/include/asm/futex.h

Quoted 10 lines
arch/parisc/include/asm/futex.h +10 -0
+    __asm__ __volatile__(
+        "1:\tldw 0(%1), %0\n"
+        "2:\tstbys,e %%r0, 0(%1)\n"
+        "\tcomclr,= %%r0, %%r0, %0\n"
+        "3:\tldi 1, %0\n"
+        ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
+        ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
+        : "=&r" (result) : "r" (ua) : "memory"
+    );
+    return result;

I wonder if we can get rid of the comclr,= instruction by using ASM_EXCEPTIONTABLE_ENTRY_EFAULT instead of ASM_EXCEPTIONTABLE_ENTRY, e.g.:

arch/parisc/include/asm/futex.h +4 -9
index 8f97db995b04..ea052f013865 100644
@@ -21,20 +21,21 @@ static inline unsigned long _futex_hash_index(unsigned long ua)
    * if load and store fault.
    */
 
-static inline bool _futex_force_interruptions(unsigned long ua)
+static inline unsigned long _futex_force_interruptions(unsigned long ua)
   {
-    bool result;
+    register unsigned long error __asm__ ("r8") = 0;
+    register unsigned long temp;
 
       __asm__ __volatile__(
-        "1:\tldw 0(%1), %0\n"
-        "2:\tstbys,e %%r0, 0(%1)\n"
-        "\tcomclr,= %%r0, %%r0, %0\n"
-        "3:\tldi 1, %0\n"
-        ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
-        ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
-        : "=&r" (result) : "r" (ua) : "memory"
+        "1:\tldw 0(%2), %0\n"

arch/parisc/include/asm/futex.h static inline unsigned long _futex_hash_index(unsigned long ua) · line 39

Quoted 10 lines
arch/parisc/include/asm/futex.h +7 -1
+        "2:\tstbys,e %%r0, 0(%2)\n"
+        "3:\n"
+        ASM_EXCEPTIONTABLE_ENTRY_EFAULT(1b, 3b)
+        ASM_EXCEPTIONTABLE_ENTRY_EFAULT(2b, 3b)
+        : "=r" (temp), "=r" (error)
+        : "r" (ua), "1" (error) : "memory"
       );
-    return result;
+    return error;
   }

I don't think this is a win.

Quoted 1 line

1) Register %r8 needs to get loaded with 0. So, that's one instruction.

True. On the other hand you don't need the "ldi 1,%0" then, which brings parity.

Quoted 3 lines

2) Register %r8 is a caller saves register. Using it will cause gcc to save and restore it from stack. This may cause a stack frame to be created when one isn't needed. The save and restore instructions are more expensive, particularly if they cause a TLB miss.

Because of this reason, wouln't it make sense to switch the uaccess functions away from r8 too, and use another temp register in both?

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Note that the comclr both clears result and nullifies the following ldi instruction, so it is not executed in the fast path.

Yes, but when emulating with qemu, it generates comparism and jumps, while the loading of r8 (see 1)), is trivial.

If we change the uaccess functions away from r8, then we can drop comclr (and save one instruction).

Independend of your patch I think we should go away in uacces from r8. I did some testing (on 32bit kernel) and r29 seems good. Opinion?

I'm thinking r28 is better.  It's standard return value and would work better in _futex_force_interruptions().

Dave

Signature · 1 line
John David Anglin  [email protected]
#6 lore raw
In reply to Helge Deller
Quoted 69 lines · Helge Deller
Quoted 68 lines · John David Anglin
Quoted 55 lines · Helge Deller

...

Quoted 6 lines

In order to do this, we need a mechanism to trigger COW breaks outside the critical region. Fortunately, parisc has the "stbys,e" instruction. When the leftmost byte of a word is addressed, this instruction triggers all the exceptions of a normal store but it does not write to memory. Thus, we can use it to trigger COW breaks outside the critical region without modifying the data that is to be updated atomically.

...

Quoted 4 lines
arch/parisc/include/asm/futex.h +0 -0
index 9cd4dd6e63ad..8f97db995b04 100644

...

Quoted 4 lines
arch/parisc/include/asm/futex.h +4 -0
+static inline bool _futex_force_interruptions(unsigned long ua)
+{
+    bool result;
+

arch/parisc/include/asm/futex.h

Quoted 10 lines
arch/parisc/include/asm/futex.h +10 -0
+    __asm__ __volatile__(
+        "1:\tldw 0(%1), %0\n"
+        "2:\tstbys,e %%r0, 0(%1)\n"
+        "\tcomclr,= %%r0, %%r0, %0\n"
+        "3:\tldi 1, %0\n"
+        ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
+        ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
+        : "=&r" (result) : "r" (ua) : "memory"
+    );
+    return result;

I wonder if we can get rid of the comclr,= instruction by using ASM_EXCEPTIONTABLE_ENTRY_EFAULT instead of ASM_EXCEPTIONTABLE_ENTRY, e.g.:

arch/parisc/include/asm/futex.h +4 -9
index 8f97db995b04..ea052f013865 100644
@@ -21,20 +21,21 @@ static inline unsigned long _futex_hash_index(unsigned long ua)
    * if load and store fault.
    */
 
-static inline bool _futex_force_interruptions(unsigned long ua)
+static inline unsigned long _futex_force_interruptions(unsigned long ua)
   {
-    bool result;
+    register unsigned long error __asm__ ("r8") = 0;
+    register unsigned long temp;
 
       __asm__ __volatile__(
-        "1:\tldw 0(%1), %0\n"
-        "2:\tstbys,e %%r0, 0(%1)\n"
-        "\tcomclr,= %%r0, %%r0, %0\n"
-        "3:\tldi 1, %0\n"
-        ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
-        ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
-        : "=&r" (result) : "r" (ua) : "memory"
+        "1:\tldw 0(%2), %0\n"

arch/parisc/include/asm/futex.h static inline unsigned long _futex_hash_index(unsigned long ua) · line 39

Quoted 10 lines
arch/parisc/include/asm/futex.h +7 -1
+        "2:\tstbys,e %%r0, 0(%2)\n"
+        "3:\n"
+        ASM_EXCEPTIONTABLE_ENTRY_EFAULT(1b, 3b)
+        ASM_EXCEPTIONTABLE_ENTRY_EFAULT(2b, 3b)
+        : "=r" (temp), "=r" (error)
+        : "r" (ua), "1" (error) : "memory"
       );
-    return result;
+    return error;
   }

I don't think this is a win.

Quoted 8 lines
Quoted 1 line

1) Register %r8 needs to get loaded with 0. So, that's one instruction.

True. On the other hand you don't need the "ldi 1,%0" then, which brings parity.

Quoted 3 lines

2) Register %r8 is a caller saves register. Using it will cause gcc to save and restore it from stack. This may cause a stack frame to be created when one isn't needed. The save and restore instructions are more expensive, particularly if they cause a TLB miss.

Because of this reason, wouln't it make sense to switch the uaccess functions away from r8 too, and use another temp register in both?

Yes. I think it would be best to use %r28.  Then, error will be in correct register for return.

The variables temp and error can combined.

Quoted 3 lines
Quoted 2 lines

Note that the comclr both clears result and nullifies the following ldi instruction, so it is not executed in the fast path.

Yes, but when emulating with qemu, it generates comparism and jumps, while the loading of r8 (see 1)), is trivial.

In this case, the comparison is always true. Does qemu detect that?

The comclr is equivalent to setting target register to 0 and a branch to .+8.

Quoted 2 lines

If we change the uaccess functions away from r8, then we can drop comclr (and save one instruction).

If we can use %r28, we should be able to save one instruction.

It's slightly less work for me if you change the uaccess and update the asm at the same time.

Dave

Signature · 1 line
John David Anglin  [email protected]
#7 lore raw
Quoted 9 lines · John David Anglin
arch/parisc/include/asm/futex.h +1 -1
-static inline bool _futex_force_interruptions(unsigned long ua)
+static inline unsigned long _futex_force_interruptions(unsigned long ua)

   {
-    bool result;
+    register unsigned long error __asm__ ("r8") = 0; +    register unsigned long temp;

       __asm__ __volatile__(
-        "1:\tldw 0(%1), %0\n"

We can avoid two variables if we clear the return value after ldw.

Quoted 3 lines

-        "2:\tstbys,e %%r0, 0(%1)\n"
-        "\tcomclr,= %%r0, %%r0, %0\n"
-        "3:\tldi 1, %0\n"

Dave

Signature · 1 line
John David Anglin  [email protected]
maintainer
#8 lore raw
In reply to John David Anglin
Quoted 71 lines · John David Anglin
Quoted 70 lines · Helge Deller
Quoted 69 lines · John David Anglin
Quoted 57 lines · Helge Deller

...

Quoted 6 lines

In order to do this, we need a mechanism to trigger COW breaks outside the critical region. Fortunately, parisc has the "stbys,e" instruction. When the leftmost byte of a word is addressed, this instruction triggers all the exceptions of a normal store but it does not write to memory. Thus, we can use it to trigger COW breaks outside the critical region without modifying the data that is to be updated atomically.

...

Quoted 4 lines
arch/parisc/include/asm/futex.h +0 -0
index 9cd4dd6e63ad..8f97db995b04 100644

...

Quoted 4 lines
arch/parisc/include/asm/futex.h +4 -0
+static inline bool _futex_force_interruptions(unsigned long ua)
+{
+    bool result;
+

arch/parisc/include/asm/futex.h

Quoted 10 lines
arch/parisc/include/asm/futex.h +10 -0
+    __asm__ __volatile__(
+        "1:\tldw 0(%1), %0\n"
+        "2:\tstbys,e %%r0, 0(%1)\n"
+        "\tcomclr,= %%r0, %%r0, %0\n"
+        "3:\tldi 1, %0\n"
+        ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
+        ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
+        : "=&r" (result) : "r" (ua) : "memory"
+    );
+    return result;

I wonder if we can get rid of the comclr,= instruction by using ASM_EXCEPTIONTABLE_ENTRY_EFAULT instead of ASM_EXCEPTIONTABLE_ENTRY, e.g.:

arch/parisc/include/asm/futex.h +3 -2
index 8f97db995b04..ea052f013865 100644
@@ -21,20 +21,21 @@ static inline unsigned long _futex_hash_index(unsigned long ua)
     * if load and store fault.
     */
 
-static inline bool _futex_force_interruptions(unsigned long ua)
+static inline unsigned long _futex_force_interruptions(unsigned long ua)
    {
-    bool result;
+    register unsigned long error __asm__ ("r8") = 0;
+    register unsigned long temp;
 

       __asm__ __volatile__(
-        "1:\tldw 0(%1), %0\n"
-        "2:\tstbys,e %%r0, 0(%1)\n"
-        "\tcomclr,= %%r0, %%r0, %0\n"
-        "3:\tldi 1, %0\n"
-        ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
-        ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
-        : "=&r" (result) : "r" (ua) : "memory"
+        "1:\tldw 0(%2), %0\n"
+        "2:\tstbys,e %%r0, 0(%2)\n"
+        "3:\n"

Quoted 8 lines

+        ASM_EXCEPTIONTABLE_ENTRY_EFAULT(1b, 3b)
+        ASM_EXCEPTIONTABLE_ENTRY_EFAULT(2b, 3b)
+        : "=r" (temp), "=r" (error)
+        : "r" (ua), "1" (error) : "memory"
       );
-    return result;
+    return error;
   }

I don't think this is a win.

1) Register %r8 needs to get loaded with 0. So, that's one instruction.

Quoted 8 lines
Quoted 7 lines

True. On the other hand you don't need the "ldi 1,%0" then, which brings parity.

Quoted 3 lines

2) Register %r8 is a caller saves register. Using it will cause gcc to save and restore it from stack. This may cause a stack frame to be created when one isn't needed. The save and restore instructions are more expensive, particularly if they cause a TLB miss.

Because of this reason, wouln't it make sense to switch the uaccess functions away from r8 too, and use another temp register in both?

Yes. I think it would be best to use %r28.  Then, error will be in correct register for return.

r29 seems to generate smaller code than r28, so I choosed r29.

Quoted 1 line

The variables temp and error can combined.

Then I need the copy %r0,%error in asm code. It's no option to use ldw with target register %r0 ?

Quoted 3 lines
Quoted 2 lines
Quoted 1 line

Note that the comclr both clears result and nullifies the following ldi instruction, so it is not executed in the fast path.

Yes, but when emulating with qemu, it generates comparism and jumps, while the loading of r8 (see 1)), is trivial.

In this case, the comparison is always true. Does qemu detect that?

Doesn't seem so. I think it's quite uncommon to use it that way... :-)

Quoted 4 lines

The comclr is equivalent to setting target register to 0 and a branch to .+8.

Quoted 2 lines

If we change the uaccess functions away from r8, then we can drop comclr (and save one instruction).

If we can use %r28, we should be able to save one instruction.

Yes.

Quoted 1 line

It's slightly less work for me if you change the uaccess and update the asm at the same time.

Sure. I'll clean it up and commit to for-next.

Helge

#9 lore raw
Quoted 74 lines · Helge Deller
Quoted 73 lines · John David Anglin
Quoted 72 lines · Helge Deller
Quoted 68 lines · John David Anglin
Quoted 55 lines · Helge Deller

...

Quoted 6 lines

In order to do this, we need a mechanism to trigger COW breaks outside the critical region. Fortunately, parisc has the "stbys,e" instruction. When the leftmost byte of a word is addressed, this instruction triggers all the exceptions of a normal store but it does not write to memory. Thus, we can use it to trigger COW breaks outside the critical region without modifying the data that is to be updated atomically.

...

Quoted 4 lines
arch/parisc/include/asm/futex.h +0 -0
index 9cd4dd6e63ad..8f97db995b04 100644

...

Quoted 4 lines
arch/parisc/include/asm/futex.h +4 -0
+static inline bool _futex_force_interruptions(unsigned long ua)
+{
+    bool result;
+

arch/parisc/include/asm/futex.h

Quoted 10 lines
arch/parisc/include/asm/futex.h +10 -0
+    __asm__ __volatile__(
+        "1:\tldw 0(%1), %0\n"
+        "2:\tstbys,e %%r0, 0(%1)\n"
+        "\tcomclr,= %%r0, %%r0, %0\n"
+        "3:\tldi 1, %0\n"
+        ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
+        ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
+        : "=&r" (result) : "r" (ua) : "memory"
+    );
+    return result;

I wonder if we can get rid of the comclr,= instruction by using ASM_EXCEPTIONTABLE_ENTRY_EFAULT instead of ASM_EXCEPTIONTABLE_ENTRY, e.g.:

arch/parisc/include/asm/futex.h +4 -9
index 8f97db995b04..ea052f013865 100644
@@ -21,20 +21,21 @@ static inline unsigned long _futex_hash_index(unsigned long ua)
     * if load and store fault.
     */
 
-static inline bool _futex_force_interruptions(unsigned long ua)
+static inline unsigned long _futex_force_interruptions(unsigned long ua)
    {
-    bool result;
+    register unsigned long error __asm__ ("r8") = 0;
+    register unsigned long temp;
 
        __asm__ __volatile__(
-        "1:\tldw 0(%1), %0\n"
-        "2:\tstbys,e %%r0, 0(%1)\n"
-        "\tcomclr,= %%r0, %%r0, %0\n"
-        "3:\tldi 1, %0\n"
-        ASM_EXCEPTIONTABLE_ENTRY(1b, 3b)
-        ASM_EXCEPTIONTABLE_ENTRY(2b, 3b)
-        : "=&r" (result) : "r" (ua) : "memory"
+        "1:\tldw 0(%2), %0\n"

arch/parisc/include/asm/futex.h static inline unsigned long _futex_hash_index(unsigned long ua) · line 39

Quoted 10 lines
arch/parisc/include/asm/futex.h +7 -1
+        "2:\tstbys,e %%r0, 0(%2)\n"
+        "3:\n"
+        ASM_EXCEPTIONTABLE_ENTRY_EFAULT(1b, 3b)
+        ASM_EXCEPTIONTABLE_ENTRY_EFAULT(2b, 3b)
+        : "=r" (temp), "=r" (error)
+        : "r" (ua), "1" (error) : "memory"
        );
-    return result;
+    return error;
    }

I don't think this is a win.

Quoted 1 line

1) Register %r8 needs to get loaded with 0. So, that's one instruction.

True. On the other hand you don't need the "ldi 1,%0" then, which brings parity.

Quoted 7 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

2) Register %r8 is a caller saves register. Using it will cause gcc to save and restore it from stack. This may cause a stack frame to be created when one isn't needed. The save and restore instructions are more expensive, particularly if they cause a TLB miss.

Because of this reason, wouln't it make sense to switch the uaccess functions away from r8 too, and use another temp register in both?

Yes. I think it would be best to use %r28.  Then, error will be in correct register for return.

r29 seems to generate smaller code than r28, so I choosed r29.

Okay.  Maybe it won't matter in an inline function.

Quoted 4 lines
Quoted 2 lines

The variables temp and error can combined.

Then I need the copy %r0,%error in asm code. It's no option to use ldw with target register %r0 ?

Correct. It's a cache prefetch instruction and doesn't trigger exceptions.

I'm rewriting routine.

Dave

Signature · 1 line
John David Anglin  [email protected]