Received: from eggs.gnu.org ([2001:4830:134:3::10]:59891) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gDJGU-0005dJ-9K for qemu-devel@nongnu.org; Thu, 18 Oct 2018 21:07:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gDJGO-00049L-9I for qemu-devel@nongnu.org; Thu, 18 Oct 2018 21:07:46 -0400 From: "Emilio G. Cota" Date: Thu, 18 Oct 2018 21:05:29 -0400 Message-Id: <20181019010625.25294-1-cota@braap.org> Subject: [Qemu-devel] [RFC v3 0/56] per-CPU locks List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Cc: Paolo Bonzini , Aleksandar Markovic , Alexander Graf , Alistair Francis , Andrzej Zaborowski , Anthony Green , Artyom Tarasenko , Aurelien Jarno , Bastian Koppelmann , Christian Borntraeger , Chris Wulff , Cornelia Huck , David Gibson , David Hildenbrand , "Edgar E. Iglesias" , Eduardo Habkost , Fabien Chouteau , Guan Xuetao , James Hogan , Laurent Vivier , Marek Vasut , Mark Cave-Ayland , Max Filippov , Michael Clark , Michael Walle , Palmer Dabbelt , Pavel Dovgalyuk , Peter Crosthwaite , Peter Maydell , qemu-arm@nongnu.org, qemu-ppc@nongnu.org, qemu-s390x@nongnu.org, Richard Henderson , Sagar Karandikar , Stafford Horne Cc: Aleksandar Markovic Cc: Alexander Graf Cc: Alistair Francis Cc: Andrzej Zaborowski Cc: Anthony Green Cc: Artyom Tarasenko Cc: Aurelien Jarno Cc: Bastian Koppelmann Cc: Christian Borntraeger Cc: Chris Wulff Cc: Cornelia Huck Cc: David Gibson Cc: David Hildenbrand Cc: "Edgar E. Iglesias" Cc: Eduardo Habkost Cc: Fabien Chouteau Cc: Guan Xuetao Cc: James Hogan Cc: Laurent Vivier Cc: Marek Vasut Cc: Mark Cave-Ayland Cc: Max Filippov Cc: Michael Clark Cc: Michael Walle Cc: Palmer Dabbelt Cc: Pavel Dovgalyuk Cc: Peter Crosthwaite Cc: Peter Maydell Cc: qemu-arm@nongnu.org Cc: qemu-ppc@nongnu.org Cc: qemu-s390x@nongnu.org Cc: Richard Henderson Cc: Sagar Karandikar Cc: Stafford Horne I'm calling this series a v3 because it supersedes the two series I previously sent about using atomics for interrupt_request: https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg02013.html The approach in that series cannot work reliably; using (locked) atomics to set interrupt_request but not using (locked) atomics to read it can lead to missed updates. This series takes a different approach: it serializes access to many CPUState fields, including .interrupt_request, with a per-CPU lock. Protecting more fields of CPUState with the lock then allows us to substitute the BQL for the per-CPU lock in many places, notably the execution loop in cpus.c. This leads to better scalability for MTTCG, since CPUs don't have to acquire a contended lock (the BQL) every time they stop executing code. Some hurdles that remain: 1. I am not happy with the shutdown path via pause_all_vcpus. What happens if (a) A CPU is added while we're calling pause_all_vcpus? (b) Some CPUs are trying to run exclusive work while we call pause_all_vcpus? Am I being overly paranoid here? 2. I have done very light testing with x86_64 KVM, and no testing with other accels (hvf, hax, whpx). check-qtest works, except for an s390x test that to me is broken in master -- I reported the problem here: https://lists.gnu.org/archive/html/qemu-devel/2018-10/msg03728.html 3. This might break record-replay. Furthermore, a quick test with icount on aarch64 seems to work, but I haven't tested icount extensively. 4. Some architectures still need the BQL in cpu_has_work. This leads to some contortions to avoid deadlock, since in this series cpu_has_work is called with the CPU lock held. 5. The interrupt handling path remains with the BQL held, mostly because the ISAs I routinely work with need the BQL anyway when handling the interrupt. We can complete the pushdown of the BQL to .do_interrupt/.exec_interrupt later on; this series is already way too long. Points (1)-(3) makes this series an RFC and not a proper patch series. I'd appreciate feedback on this approach and/or testing. Note that this series fixes a bug by which cpu_has_work is called without the BQL (from cpu_handle_halt). After this series, cpu_has_work is called with the CPU lock, and only the targets that need the BQL in cpu_has_work acquire it. For some performance numbers, see the last patch. The series is checkpatch-clean; only one warning due to the use of __COVERITY__ in cpus.c. You can fetch this series from: https://github.com/cota/qemu/tree/cpu-lock-v3 Note that it applies on top of tcg-next + my dynamic TLB series, which I'm using in the faint hope that the ubuntu experiments might run a bit faster. Thanks! Emilio