Skip to content

NT synchronization primitive driver

Elizabeth Figura linux-api linux-doc linux-kselftest lkml 5 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest0 of 30 patches 21 repliesIn review
  2. v330 patches 6 repliesSuperseded
  3. v2 posting 2/231 patches 6 repliesSuperseded
Earlier versions · 2
VersionPostedPatchesRepliesStatus
v2 RFC posting 1/229 patches2 repliesSuperseded
v1 RFC9 patches29 repliesSuperseded

Changes in v4 from the cover letter

* Add .gitignore and use KHDR_INCLUDES in selftest build files, per Muhammad Usama Anjum.

* Try to explain why we are rolling our own primitives a little better, per Greg Kroah-Hartman.

* Link to v3: https://lore.kernel.org/lkml/[email protected]/
* Link to v2: https://lore.kernel.org/lkml/[email protected]/
* Link to v1: https://lore.kernel.org/lkml/[email protected]/
* Link to RFC v2: https://lore.kernel.org/lkml/[email protected]/
* Link to RFC v1: https://lore.kernel.org/lkml/[email protected]/

Patches in v4 No applies recorded yet

None of the 30 patches the cover letter announces are in this archive.

Reviews trailers per patch

PatchBagas SanjayaRandy Dunlap
SeriesR-b T-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v4 thread

Cover letter [PATCH v4 00/30] NT synchronization primitive driver

This patch series implements a new char misc driver, /dev/ntsync, which is used to implement Windows NT synchronization primitives.

NT synchronization primitives are unique in that the wait functions both are vectored, operate on multiple types of object with different behaviour (mutex, semaphore, event), and affect the state of the objects they wait on. This model is not compatible with existing kernel synchronization objects or interfaces, and therefore the ntsync driver implements its own wait queues and locking.

Hence I would like to request review from someone familiar with locking to make sure that the usage of low-level kernel primitives is correct and that the wait queues work as intended, and to that end I've CC'd the locking maintainers.

== Background ==

The Wine project emulates the Windows API in user space. One particular part of that API, namely the NT synchronization primitives, have historically been implemented via RPC to a dedicated "kernel" process. However, more recent applications use these APIs more strenuously, and the overhead of RPC has become a bottleneck.

The NT synchronization APIs are too complex to implement on top of existing primitives without sacrificing correctness. Certain operations, such as NtPulseEvent() or the "wait-for-all" mode of NtWaitForMultipleObjects(), require direct control over the underlying wait queue, and implementing a wait queue sufficiently robust for Wine in user space is not possible. This proposed driver, therefore, implements the problematic interfaces directly in the Linux kernel.

This driver was presented at Linux Plumbers Conference 2023. For those further interested in the history of synchronization in Wine and past attempts to solve this problem in user space, a recording of the presentation can be viewed here:

https://www.youtube.com/watch?v=NjU4nyWyhU8

== Performance ==

The gain in performance varies wildly depending on the application in question and the user's hardware. For some games NT synchronization is not a bottleneck and no change can be observed, but for others frame rate improvements of 50 to 150 percent are not atypical. The following table lists frame rate measurements from a variety of games on a variety of hardware, taken by users Dmitry Skvortsov, FuzzyQuils, OnMars, and myself:

Game Upstream ntsync improvement
===========================================================================
Anger Foot 69 99 43%

Call of Juarez                   99.8           224.1           125%
Dirt 3                          110.6           860.7           678%
Forza Horizon 5                 108             160              48%
Lara Croft: Temple of Osiris    141             326             131%
Metro 2033                      164.4           199.2            21%
Resident Evil 2                  26              77             196%
The Crew                         26              51              96%
Tiny Tina's Wonderlands         130             360             177%
Total War Saga: Troy            109             146              34%

===========================================================================

== Patches ==

The intended semantics of the patches are broadly intended to match those of the corresponding Windows functions. For those not already familiar with the Windows functions (or their undocumented behaviour), patch 27/27 provides a detailed specification, and individual patches also include a brief description of the API they are implementing.

The patches making use of this driver in Wine can be retrieved or browsed here:

https://repo.or.cz/wine/zf.git/shortlog/refs/heads/ntsync5

== Implementation ==

Some aspects of the implementation may deserve particular comment:

* In the interest of performance, each object is governed only by a single spinlock. However, NTSYNC_IOC_WAIT_ALL requires that the state of multiple objects be changed as a single atomic operation. In order to achieve this, we first take a device-wide lock ("wait_all_lock") any time we are going to lock more than one object at a time.

The maximum number of objects that can be used in a vectored wait, and therefore the maximum that can be locked simultaneously, is 64. This number is NT's own limit.

The acquisition of multiple spinlocks will degrade performance. This is a conscious choice, however. Wait-for-all is known to be a very rare operation in practice, especially with counts that approach the maximum, and it is the intent of the ntsync driver to optimize wait-for-any at the expense of wait-for-all as much as possible.

* NT mutexes are tied to their threads on an OS level, and the kernel includes builtin support for "robust" mutexes. In order to keep the ntsync driver self-contained and avoid touching more code than necessary, it does not hook into task exit nor use pids.

Instead, the user space emulator is expected to manage thread IDs and pass them as an argument to any relevant functions; this is the "owner" field of ntsync_wait_args and ntsync_mutex_args.

When the emulator detects that a thread dies, it should therefore call NTSYNC_IOC_MUTEX_KILL on any open mutexes.

* ntsync is module-capable mostly because there was nothing preventing it, and because it aided development. It is not a hard requirement, though.

== Previous versions ==

Changes from v3:

* Add .gitignore and use KHDR_INCLUDES in selftest build files, per Muhammad Usama Anjum.

* Try to explain why we are rolling our own primitives a little better, per Greg Kroah-Hartman.

* Link to v3: https://lore.kernel.org/lkml/[email protected]/
* Link to v2: https://lore.kernel.org/lkml/[email protected]/
* Link to v1: https://lore.kernel.org/lkml/[email protected]/
* Link to RFC v2: https://lore.kernel.org/lkml/[email protected]/
* Link to RFC v1: https://lore.kernel.org/lkml/[email protected]/

Elizabeth Figura (27):
ntsync: Introduce NTSYNC_IOC_WAIT_ANY.
ntsync: Introduce NTSYNC_IOC_WAIT_ALL.
ntsync: Introduce NTSYNC_IOC_CREATE_MUTEX.
ntsync: Introduce NTSYNC_IOC_MUTEX_UNLOCK.
ntsync: Introduce NTSYNC_IOC_MUTEX_KILL.
ntsync: Introduce NTSYNC_IOC_CREATE_EVENT.
ntsync: Introduce NTSYNC_IOC_EVENT_SET.
ntsync: Introduce NTSYNC_IOC_EVENT_RESET.
ntsync: Introduce NTSYNC_IOC_EVENT_PULSE.
ntsync: Introduce NTSYNC_IOC_SEM_READ.
ntsync: Introduce NTSYNC_IOC_MUTEX_READ.
ntsync: Introduce NTSYNC_IOC_EVENT_READ.
ntsync: Introduce alertable waits.
selftests: ntsync: Add some tests for semaphore state.
selftests: ntsync: Add some tests for mutex state.
selftests: ntsync: Add some tests for NTSYNC_IOC_WAIT_ANY.
selftests: ntsync: Add some tests for NTSYNC_IOC_WAIT_ALL.
selftests: ntsync: Add some tests for wakeup signaling with
WINESYNC_IOC_WAIT_ANY.
selftests: ntsync: Add some tests for wakeup signaling with
WINESYNC_IOC_WAIT_ALL.
selftests: ntsync: Add some tests for manual-reset event state.
selftests: ntsync: Add some tests for auto-reset event state.
selftests: ntsync: Add some tests for wakeup signaling with events.
selftests: ntsync: Add tests for alertable waits.
selftests: ntsync: Add some tests for wakeup signaling via alerts.
selftests: ntsync: Add a stress test for contended waits.
maintainers: Add an entry for ntsync.
docs: ntsync: Add documentation for the ntsync uAPI.

 Documentation/userspace-api/index.rst         |    1 +
 Documentation/userspace-api/ntsync.rst        |  399 +++++
 MAINTAINERS                                   |    9 +
 drivers/misc/ntsync.c                         |  925 ++++++++++-
 include/uapi/linux/ntsync.h                   |   39 +
 tools/testing/selftests/Makefile              |    1 +
 .../selftests/drivers/ntsync/.gitignore       |    1 +
 .../testing/selftests/drivers/ntsync/Makefile |    7 +
 tools/testing/selftests/drivers/ntsync/config |    1 +
 .../testing/selftests/drivers/ntsync/ntsync.c | 1407 +++++++++++++++++
 10 files changed, 2786 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/userspace-api/ntsync.rst
 create mode 100644 tools/testing/selftests/drivers/ntsync/.gitignore
 create mode 100644 tools/testing/selftests/drivers/ntsync/Makefile
 create mode 100644 tools/testing/selftests/drivers/ntsync/config
 create mode 100644 tools/testing/selftests/drivers/ntsync/ntsync.c

base-commit: ebbc1a4789c666846b9854ef845a37a64879e5f9

Signature · 1 line
2.43.0