Skip to content

RFC Paravirtualized ticketlocks

Jeremy Fitzhardinge kvm lkml xen-devel 5 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v5 RFC Latest11 patches 18 repliesIn review
  2. v4 RFC11 patches 0 repliesSuperseded
  3. v2 RFC posting 2/210 patches 24 repliesSuperseded
Earlier versions · 2
VersionPostedPatchesRepliesStatus
v2 RFC posting 1/28 patches0 repliesSuperseded
v1 RFC13 patches54 repliesSuperseded

Patches in v5 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/11x86/spinlock: replace pv spinlocks with pv ticketlocksarch/x86/include/asm/paravirt.h arch/x86/include/asm/paravirt_types.h arch/x86/include/asm/spinlock.h +3 more+56 -600 replies
2/11x86/ticketlock: don't inline _spin_unlock when using paravirt spinlocksarch/x86/Kconfig kernel/Kconfig.locks+4 -10 replies
3/11x86/ticketlock: collapse a layer of functionsarch/x86/include/asm/spinlock.h+5 -300 replies
4/11xen: defer spinlock setup until boot CPU setuparch/x86/xen/smp.c+1 -10 replies
5/11xen/pvticketlock: Xen implementation for PV ticket locksarch/x86/xen/spinlock.c+43 -2440 replies
6/11xen/pvticketlocks: add xen_nopvspin parameter to disable xen pv ticketlocksarch/x86/xen/spinlock.c+14 -00 replies
7/11x86/pvticketlock: use callee-save for lock_spinningarch/x86/include/asm/paravirt.h arch/x86/include/asm/paravirt_types.h arch/x86/kernel/paravirt-spinlocks.c +1 more+5 -40 replies
8/11x86/pvticketlock: when paravirtualizing ticket locks, increment by 2arch/x86/include/asm/spinlock.h arch/x86/include/asm/spinlock_types.h+14 -60 replies
9/11x86/ticketlock: add slowpath logicarch/x86/include/asm/paravirt.h arch/x86/include/asm/spinlock.h arch/x86/include/asm/spinlock_types.h +2 more+71 -210 replies
10/11xen/pvticketlock: allow interrupts to be enabled while blockingarch/x86/xen/spinlock.c+41 -70 replies
11/11xen: enable PV ticketlocks on HVM Xenarch/x86/xen/smp.c+1 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v5 yet.

Read the v5 thread

Cover letter [PATCH RFC V5 00/11] Paravirtualized ticketlocks

From: Jeremy Fitzhardinge <[email protected]>

[ Changes since last posting:
- Use "lock add" for unlock operation rather than "lock xadd"; it is equivalent to "add; mfence", but more efficient than both "lock xadd" and "mfence".

I think this version is ready for submission.
]

NOTE: this series is available in:
git://github.com/jsgf/linux-xen.git upstream/pvticketlock-slowflag
and is based on the previously posted ticketlock cleanup series in
git://github.com/jsgf/linux-xen.git upstream/ticketlock-cleanup

This series replaces the existing paravirtualized spinlock mechanism with a paravirtualized ticketlock mechanism.

Ticket locks have an inherent problem in a virtualized case, because the vCPUs are scheduled rather than running concurrently (ignoring gang scheduled vCPUs). This can result in catastrophic performance collapses when the vCPU scheduler doesn't schedule the correct "next" vCPU, and ends up scheduling a vCPU which burns its entire timeslice spinning. (Note that this is not the same problem as lock-holder preemption, which this series also addresses; that's also a problem, but not catastrophic).

(See Thomas Friebel's talk "Prevent Guests from Spinning Around"
http://www.xen.org/files/xensummitboston08/LHP.pdf for more details.)

Currently we deal with this by having PV spinlocks, which adds a layer of indirection in front of all the spinlock functions, and defining a completely new implementation for Xen (and for other pvops users, but there are none at present).

PV ticketlocks keeps the existing ticketlock implemenentation (fastpath) as-is, but adds a couple of pvops for the slow paths:

- If a CPU has been waiting for a spinlock for SPIN_THRESHOLD iterations, then call out to the __ticket_lock_spinning() pvop, which allows a backend to block the vCPU rather than spinning. This pvop can set the lock into "slowpath state".

- When releasing a lock, if it is in "slowpath state", the call __ticket_unlock_kick() to kick the next vCPU in line awake. If the lock is no longer in contention, it also clears the slowpath flag.

The "slowpath state" is stored in the LSB of the within the lock tail ticket. This has the effect of reducing the max number of CPUs by half (so, a "small ticket" can deal with 128 CPUs, and "large ticket" 32768).

This series provides a Xen implementation, but it should be straightforward to add a KVM implementation as well.

Overall, it results in a large reduction in code, it makes the native and virtualized cases closer, and it removes a layer of indirection around all the spinlock functions.

The fast path (taking an uncontended lock which isn't in "slowpath" state) is optimal, identical to the non-paravirtualized case.

The inner part of ticket lock code becomes:

	inc = xadd(&lock->tickets, inc);
	inc.tail &= ~TICKET_SLOWPATH_FLAG;

	if (likely(inc.head == inc.tail))
		goto out;

	for (;;) {
		unsigned count = SPIN_THRESHOLD;

		do {
			if (ACCESS_ONCE(lock->tickets.head) == inc.tail)
				goto out;
			cpu_relax();
		} while (--count);
		__ticket_lock_spinning(lock, inc.tail);
	}

out: barrier();

which results in:
push %rbp
mov %rsp,%rbp

mov $0x200,%eax
lock xadd %ax,(%rdi)
movzbl %ah,%edx
cmp %al,%dl
jne 1f # Slowpath if lock in contention

pop %rbp
retq

### SLOWPATH START
1: and $-2,%edx
movzbl %dl,%esi

2: mov $0x800,%eax
jmp 4f

3: pause

	sub    $0x1,%eax
	je     5f

4: movzbl (%rdi),%ecx
cmp %cl,%dl
jne 3b

pop %rbp
retq

5: callq *__ticket_lock_spinning
jmp 2b
### SLOWPATH END

with CONFIG_PARAVIRT_SPINLOCKS=n, the code has changed slightly, where the fastpath case is straight through (taking the lock without contention), and the spin loop is out of line:

push %rbp
mov %rsp,%rbp

mov $0x100,%eax
lock xadd %ax,(%rdi)
movzbl %ah,%edx
cmp %al,%dl
jne 1f

pop %rbp
retq

### SLOWPATH START
1: pause

	movzbl (%rdi),%eax
	cmp    %dl,%al
	jne    1b

	pop    %rbp
	retq   
	### SLOWPATH END

The unlock code is complicated by the need to both add to the lock's "head" and fetch the slowpath flag from "tail". This version of the patch uses a locked add to do this, followed by a test to see if the slowflag is set. The lock prefix acts as a full memory barrier, so we can be sure that other CPUs will have seen the unlock before we read the flag (without the barrier the read could be fetched from the store queue before it hits memory, which could result in a deadlock).

This is is all unnecessary complication if you're not using PV ticket locks, it also uses the jump-label machinery to use the standard "add"-based unlock in the non-PV case.

	if (TICKET_SLOWPATH_FLAG &&
	    unlikely(static_branch(&paravirt_ticketlocks_enabled))) {
		arch_spinlock_t prev;

		prev = *lock;
		add_smp(&lock->tickets.head, TICKET_LOCK_INC);

		/* add_smp() is a full mb() */

		if (unlikely(lock->tickets.tail & TICKET_SLOWPATH_FLAG))
			__ticket_unlock_slowpath(lock, prev);
	} else
		__add(&lock->tickets.head, TICKET_LOCK_INC, UNLOCK_LOCK_PREFIX);

which generates:
push %rbp
mov %rsp,%rbp

nop5 # replaced by 5-byte jmp 2f when PV enabled

# non-PV unlock
addb $0x2,(%rdi)

1: pop %rbp
retq

### PV unlock ###
2: movzwl (%rdi),%esi # Fetch prev

	lock addb $0x2,(%rdi)	# Do unlock

	testb  $0x1,0x1(%rdi)	# Test flag
	je     1b		# Finished if not set

### Slow path ###

	add    $2,%sil		# Add "head" in old lock state
	mov    %esi,%edx
	and    $0xfe,%dh	# clear slowflag for comparison

movzbl %dh,%eax
cmp %dl,%al # If head == tail (uncontended)
je 4f # clear slowpath flag

# Kick next CPU waiting for lock
3: movzbl %sil,%esi
callq *pv_lock_ops.kick

pop %rbp
retq

# Lock no longer contended - clear slowflag
4: mov %esi,%eax
lock cmpxchg %dx,(%rdi) # cmpxchg to clear flag
cmp %si,%ax
jne 3b # If clear failed, then kick

pop %rbp
retq

So when not using PV ticketlocks, the unlock sequence just has a 5-byte nop added to it, and the PV case is reasonable straightforward aside from requiring a "lock add".

Thoughts? Comments? Suggestions?

Jeremy Fitzhardinge (10):
x86/spinlock: replace pv spinlocks with pv ticketlocks x86/ticketlock: don't inline _spin_unlock when using paravirt
spinlocks
x86/ticketlock: collapse a layer of functions
xen: defer spinlock setup until boot CPU setup xen/pvticketlock: Xen implementation for PV ticket locks xen/pvticketlocks: add xen_nopvspin parameter to disable xen pv
ticketlocks
x86/pvticketlock: use callee-save for lock_spinning x86/pvticketlock: when paravirtualizing ticket locks, increment by 2 x86/ticketlock: add slowpath logic
xen/pvticketlock: allow interrupts to be enabled while blocking

Stefano Stabellini (1):
xen: enable PV ticketlocks on HVM Xen

 arch/x86/Kconfig                      |    3 +
 arch/x86/include/asm/paravirt.h       |   30 +---
 arch/x86/include/asm/paravirt_types.h |   10 +-
 arch/x86/include/asm/spinlock.h       |  126 +++++++++----
 arch/x86/include/asm/spinlock_types.h |   16 +-
 arch/x86/kernel/paravirt-spinlocks.c  |   18 +--
 arch/x86/xen/smp.c                    |    3 +-
 arch/x86/xen/spinlock.c               |  331 ++++++++++-----------------------
 kernel/Kconfig.locks                  |    2 +-
 9 files changed, 210 insertions(+), 329 deletions(-)
Signature · 1 line
1.7.6.4