Skip to contentSkip to conversation

srcutiny: Add atomic SRCU operation checks and state

PATCH In review Kunwu Chan 2 participants 11 messages Aug 31 – Sep 2

Subsystems (2)
lore
Share & tools
lore

Kunwu Chan

author
#0 lore raw

PATCH 1/1 srcutiny: Add atomic SRCU operation checks and state

From: Kunwu Chan <[email protected]>

Add atomic SRCU operation checks and the associated state to Tiny SRCU.

An atomic SRCU domain does not use the normal SRCU callback and grace-period machinery. In particular, a callback queued with call_srcu() would never be processed. Use WARN_ON_ONCE() to reject call_srcu() and srcu_barrier() on atomic SRCU domains.

For synchronize_srcu(), redirect atomic SRCU domains to synchronize_srcu_atomic().

Add srcu_reader_flavor to the Tiny SRCU state for these checks. Tiny SRCU does not currently set the flavor for atomic SRCU domains, but keeping the flavor in the common state allows the operation checks to enforce the restriction once atomic flavor tracking is enabled.

Also initialize srcu_atomic_gp_flag, which was previously left uninitialized.

---
 include/linux/srcutiny.h |  1 +
 kernel/rcu/srcutiny.c    | 12 ++++++++++++
 2 files changed, 13 insertions(+)
include/linux/srcutiny.h +1 -0
index 47a368f945e3..2b293336525a 100644
@@ -20,6 +20,7 @@ struct srcu_struct {
 	u8 srcu_gp_running;		/* GP workqueue running? */
 	u8 srcu_gp_waiting;		/* GP waiting for readers? */
 	u8 srcu_atomic_gp_flag;		/* Serialize atomic GP work.*/
+	u8 srcu_reader_flavor;		/* Values: SRCU_READ_FLAVOR_.*  */
 	unsigned long srcu_idx;		/* Current reader array element in bit 0x2. */
 	unsigned long srcu_idx_max;	/* Furthest future srcu_idx request. */
 	struct swait_queue_head srcu_wq;
kernel/rcu/srcutiny.c +12 -0
index 26ea4bfbeaf2..22f7716cbb0e 100644
@@ -41,6 +41,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp)
 	ssp->srcu_cb_tail = &ssp->srcu_cb_head;
 	ssp->srcu_gp_running = false;
 	ssp->srcu_gp_waiting = false;
+	ssp->srcu_atomic_gp_flag = 0;
 	ssp->srcu_idx = 0;
 	ssp->srcu_idx_max = 0;
 	INIT_WORK(&ssp->srcu_work, srcu_drive_gp);
@@ -289,6 +290,9 @@ EXPORT_SYMBOL_GPL(srcu_defer_drain);
 void call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp,
 	       rcu_callback_t func)
 {
+	if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+		return;
+
 	if (should_rcu_defer()) {
 		/* A re-entrant call_srcu() during the drain would livelock it. */
 		if (READ_ONCE(srcu_defer_draining) && !in_nmi()) {
@@ -319,6 +323,11 @@ void synchronize_srcu(struct srcu_struct *ssp)
 {
 	struct rcu_synchronize rs;
 
+	if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC)) {
+		synchronize_srcu_atomic(ssp);
+		return;
+	}
+
 	srcu_lock_sync(&ssp->dep_map);
 
 	RCU_LOCKDEP_WARN(lockdep_is_held(ssp) ||
@@ -415,6 +424,9 @@ EXPORT_SYMBOL_GPL(synchronize_srcu_atomic);
 /* Register any deferred callbacks, then wait for all in-flight ones. */
 void srcu_barrier(struct srcu_struct *ssp)
 {
+	if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+		return;
+
 	__srcu_defer_drain(ssp);
 	synchronize_srcu(ssp);
 }
Signature · 1 line
2.43.0
maintainer
#1 lore raw
Quoted 45 lines · Kunwu Chan

From: Kunwu Chan <[email protected]>

Add atomic SRCU operation checks and the associated state to Tiny SRCU.

An atomic SRCU domain does not use the normal SRCU callback and grace-period machinery. In particular, a callback queued with call_srcu() would never be processed. Use WARN_ON_ONCE() to reject call_srcu() and srcu_barrier() on atomic SRCU domains.

For synchronize_srcu(), redirect atomic SRCU domains to synchronize_srcu_atomic().

Add srcu_reader_flavor to the Tiny SRCU state for these checks. Tiny SRCU does not currently set the flavor for atomic SRCU domains, but keeping the flavor in the common state allows the operation checks to enforce the restriction once atomic flavor tracking is enabled.

Also initialize srcu_atomic_gp_flag, which was previously left uninitialized.

---
 include/linux/srcutiny.h |  1 +
 kernel/rcu/srcutiny.c    | 12 ++++++++++++
 2 files changed, 13 insertions(+)
include/linux/srcutiny.h +1 -0
index 47a368f945e3..2b293336525a 100644
@@ -20,6 +20,7 @@ struct srcu_struct {
 	u8 srcu_gp_running;		/* GP workqueue running? */
 	u8 srcu_gp_waiting;		/* GP waiting for readers? */
 	u8 srcu_atomic_gp_flag;		/* Serialize atomic GP work.*/
+	u8 srcu_reader_flavor;		/* Values: SRCU_READ_FLAVOR_.*  */
 	unsigned long srcu_idx;		/* Current reader array element in bit 0x2. */
 	unsigned long srcu_idx_max;	/* Furthest future srcu_idx request. */
 	struct swait_queue_head srcu_wq;
kernel/rcu/srcutiny.c +0 -0
index 26ea4bfbeaf2..22f7716cbb0e 100644

kernel/rcu/srcutiny.c static int init_srcu_struct_fields(struct srcu_struct *ssp) · line 44

Quoted 4 lines
kernel/rcu/srcutiny.c +1 -0
@@ -41,6 +41,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp)
 	ssp->srcu_cb_tail = &ssp->srcu_cb_head;
 	ssp->srcu_gp_running = false;
 	ssp->srcu_gp_waiting = false;
+	ssp->srcu_atomic_gp_flag = 0;

Good catch! I will be folding this into the base commit with attribution on my next rebase:

9a2e9996ccec ("srcutiny: Add an atomic Tiny SRCU")

Quoted 26 lines
kernel/rcu/srcutiny.c +8 -0
 	ssp->srcu_idx = 0;
 	ssp->srcu_idx_max = 0;
 	INIT_WORK(&ssp->srcu_work, srcu_drive_gp);
@@ -289,6 +290,9 @@ EXPORT_SYMBOL_GPL(srcu_defer_drain);
 void call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp,
 	       rcu_callback_t func)
 {
+	if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+		return;
+
 	if (should_rcu_defer()) {
 		/* A re-entrant call_srcu() during the drain would livelock it. */
 		if (READ_ONCE(srcu_defer_draining) && !in_nmi()) {
@@ -319,6 +323,11 @@ void synchronize_srcu(struct srcu_struct *ssp)
 {
 	struct rcu_synchronize rs;
 
+	if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC)) {
+		synchronize_srcu_atomic(ssp);
+		return;
+	}
+
 	srcu_lock_sync(&ssp->dep_map);
 
 	RCU_LOCKDEP_WARN(lockdep_is_held(ssp) ||

kernel/rcu/srcutiny.c EXPORT_SYMBOL_GPL(synchronize_srcu_atomic); · line 432

Quoted 9 lines
kernel/rcu/srcutiny.c +3 -0
@@ -415,6 +424,9 @@ EXPORT_SYMBOL_GPL(synchronize_srcu_atomic);
 /* Register any deferred callbacks, then wait for all in-flight ones. */
 void srcu_barrier(struct srcu_struct *ssp)
 {
+	if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+		return;
+
 	__srcu_defer_drain(ssp);
 	synchronize_srcu(ssp);
 }

The rest is good as far as it goes, but don't we need to set the value of ssp->srcu_reader_flavor somewhere for atomic srcu_struct structures?

Thanx, Paul

author
#2 lore raw
Quoted 81 lines · Paul E. McKenney
Quoted 45 lines · Kunwu Chan

From: Kunwu Chan <[email protected]>

Add atomic SRCU operation checks and the associated state to Tiny SRCU.

An atomic SRCU domain does not use the normal SRCU callback and grace-period machinery. In particular, a callback queued with call_srcu() would never be processed. Use WARN_ON_ONCE() to reject call_srcu() and srcu_barrier() on atomic SRCU domains.

For synchronize_srcu(), redirect atomic SRCU domains to synchronize_srcu_atomic().

Add srcu_reader_flavor to the Tiny SRCU state for these checks. Tiny SRCU does not currently set the flavor for atomic SRCU domains, but keeping the flavor in the common state allows the operation checks to enforce the restriction once atomic flavor tracking is enabled.

Also initialize srcu_atomic_gp_flag, which was previously left uninitialized.

---
 include/linux/srcutiny.h |  1 +
 kernel/rcu/srcutiny.c    | 12 ++++++++++++
 2 files changed, 13 insertions(+)
include/linux/srcutiny.h +1 -0
index 47a368f945e3..2b293336525a 100644
@@ -20,6 +20,7 @@ struct srcu_struct {
      u8 srcu_gp_running;             /* GP workqueue running? */
      u8 srcu_gp_waiting;             /* GP waiting for readers? */
      u8 srcu_atomic_gp_flag;         /* Serialize atomic GP work.*/
+     u8 srcu_reader_flavor;          /* Values: SRCU_READ_FLAVOR_.*  */
      unsigned long srcu_idx;         /* Current reader array element in bit 0x2. */
      unsigned long srcu_idx_max;     /* Furthest future srcu_idx request. */
      struct swait_queue_head srcu_wq;
kernel/rcu/srcutiny.c +0 -0
index 26ea4bfbeaf2..22f7716cbb0e 100644

kernel/rcu/srcutiny.c static int init_srcu_struct_fields(struct srcu_struct *ssp) · line 44

Quoted 4 lines
kernel/rcu/srcutiny.c +1 -0
@@ -41,6 +41,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp)
      ssp->srcu_cb_tail = &ssp->srcu_cb_head;
      ssp->srcu_gp_running = false;
      ssp->srcu_gp_waiting = false;
+     ssp->srcu_atomic_gp_flag = 0;

Good catch! I will be folding this into the base commit with attribution on my next rebase:

9a2e9996ccec ("srcutiny: Add an atomic Tiny SRCU")

Quoted 24 lines
kernel/rcu/srcutiny.c +8 -0
      ssp->srcu_idx = 0;
      ssp->srcu_idx_max = 0;
      INIT_WORK(&ssp->srcu_work, srcu_drive_gp);
@@ -289,6 +290,9 @@ EXPORT_SYMBOL_GPL(srcu_defer_drain);
 void call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp,
             rcu_callback_t func)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      if (should_rcu_defer()) {
              /* A re-entrant call_srcu() during the drain would livelock it. */
              if (READ_ONCE(srcu_defer_draining) && !in_nmi()) {
@@ -319,6 +323,11 @@ void synchronize_srcu(struct srcu_struct *ssp)
 {
      struct rcu_synchronize rs;
 
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC)) {
+             synchronize_srcu_atomic(ssp);
+             return;
+     }
+
      srcu_lock_sync(&ssp->dep_map);
 
Quoted 15 lines

kernel/rcu/srcutiny.c EXPORT_SYMBOL_GPL(synchronize_srcu_atomic); · line 432

Quoted 11 lines
kernel/rcu/srcutiny.c +3 -0
      RCU_LOCKDEP_WARN(lockdep_is_held(ssp) ||
@@ -415,6 +424,9 @@ EXPORT_SYMBOL_GPL(synchronize_srcu_atomic);
 /* Register any deferred callbacks, then wait for all in-flight ones. */
 void srcu_barrier(struct srcu_struct *ssp)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      __srcu_defer_drain(ssp);
      synchronize_srcu(ssp);
 }

The rest is good as far as it goes, but don't we need to set the value of ssp->srcu_reader_flavor somewhere for atomic srcu_struct structures?

Hi Paul,

Yes, agreed. I have the flavor tracking changes implemented locally and am testing them now.
I’ll send the updated patch shortly.

I also have draft patches for the documentation and the fast path, as well as rcutorture testing for
tiny atomic srcu , which I’ll send separately.

Thanks,
Kunwu

Quoted 1 line

Thanx, Paul

maintainer
#3 lore raw
Quoted 100 lines · KunWu Chan
Quoted 81 lines · Paul E. McKenney
Quoted 45 lines · Kunwu Chan

From: Kunwu Chan <[email protected]>

Add atomic SRCU operation checks and the associated state to Tiny SRCU.

An atomic SRCU domain does not use the normal SRCU callback and grace-period machinery. In particular, a callback queued with call_srcu() would never be processed. Use WARN_ON_ONCE() to reject call_srcu() and srcu_barrier() on atomic SRCU domains.

For synchronize_srcu(), redirect atomic SRCU domains to synchronize_srcu_atomic().

Add srcu_reader_flavor to the Tiny SRCU state for these checks. Tiny SRCU does not currently set the flavor for atomic SRCU domains, but keeping the flavor in the common state allows the operation checks to enforce the restriction once atomic flavor tracking is enabled.

Also initialize srcu_atomic_gp_flag, which was previously left uninitialized.

---
 include/linux/srcutiny.h |  1 +
 kernel/rcu/srcutiny.c    | 12 ++++++++++++
 2 files changed, 13 insertions(+)
include/linux/srcutiny.h +1 -0
index 47a368f945e3..2b293336525a 100644
@@ -20,6 +20,7 @@ struct srcu_struct {
      u8 srcu_gp_running;             /* GP workqueue running? */
      u8 srcu_gp_waiting;             /* GP waiting for readers? */
      u8 srcu_atomic_gp_flag;         /* Serialize atomic GP work.*/
+     u8 srcu_reader_flavor;          /* Values: SRCU_READ_FLAVOR_.*  */
      unsigned long srcu_idx;         /* Current reader array element in bit 0x2. */
      unsigned long srcu_idx_max;     /* Furthest future srcu_idx request. */
      struct swait_queue_head srcu_wq;
kernel/rcu/srcutiny.c +0 -0
index 26ea4bfbeaf2..22f7716cbb0e 100644

kernel/rcu/srcutiny.c static int init_srcu_struct_fields(struct srcu_struct *ssp) · line 44

Quoted 4 lines
kernel/rcu/srcutiny.c +1 -0
@@ -41,6 +41,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp)
      ssp->srcu_cb_tail = &ssp->srcu_cb_head;
      ssp->srcu_gp_running = false;
      ssp->srcu_gp_waiting = false;
+     ssp->srcu_atomic_gp_flag = 0;

Good catch! I will be folding this into the base commit with attribution on my next rebase:

9a2e9996ccec ("srcutiny: Add an atomic Tiny SRCU")

Quoted 24 lines
kernel/rcu/srcutiny.c +8 -0
      ssp->srcu_idx = 0;
      ssp->srcu_idx_max = 0;
      INIT_WORK(&ssp->srcu_work, srcu_drive_gp);
@@ -289,6 +290,9 @@ EXPORT_SYMBOL_GPL(srcu_defer_drain);
 void call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp,
             rcu_callback_t func)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      if (should_rcu_defer()) {
              /* A re-entrant call_srcu() during the drain would livelock it. */
              if (READ_ONCE(srcu_defer_draining) && !in_nmi()) {
@@ -319,6 +323,11 @@ void synchronize_srcu(struct srcu_struct *ssp)
 {
      struct rcu_synchronize rs;
 
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC)) {
+             synchronize_srcu_atomic(ssp);
+             return;
+     }
+
      srcu_lock_sync(&ssp->dep_map);
 
Quoted 15 lines

kernel/rcu/srcutiny.c EXPORT_SYMBOL_GPL(synchronize_srcu_atomic); · line 432

Quoted 11 lines
kernel/rcu/srcutiny.c +3 -0
      RCU_LOCKDEP_WARN(lockdep_is_held(ssp) ||
@@ -415,6 +424,9 @@ EXPORT_SYMBOL_GPL(synchronize_srcu_atomic);
 /* Register any deferred callbacks, then wait for all in-flight ones. */
 void srcu_barrier(struct srcu_struct *ssp)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      __srcu_defer_drain(ssp);
      synchronize_srcu(ssp);
 }

The rest is good as far as it goes, but don't we need to set the value of ssp->srcu_reader_flavor somewhere for atomic srcu_struct structures?

Hi Paul,

Quoted 7 lines

Yes, agreed. I have the flavor tracking changes implemented locally and am testing them now.
I’ll send the updated patch shortly.

I also have draft patches for the documentation and the fast path, as well as rcutorture testing for
tiny atomic srcu , which I’ll send separately.

Sounds good, and I am looking forward to seeing them.

I will admit that I am curious as to why Tiny Atomic SRCU needs different rcutorture testing than does Tree Atomic SRCU, but I will ask myself that question again when I see your patches. ;-)

Thanx, Paul

author
#4 lore raw
Quoted 106 lines · Paul E. McKenney
Quoted 104 lines · KunWu Chan
Quoted 81 lines · Paul E. McKenney
Quoted 45 lines · Kunwu Chan

From: Kunwu Chan <[email protected]>

Add atomic SRCU operation checks and the associated state to Tiny SRCU.

An atomic SRCU domain does not use the normal SRCU callback and grace-period machinery. In particular, a callback queued with call_srcu() would never be processed. Use WARN_ON_ONCE() to reject call_srcu() and srcu_barrier() on atomic SRCU domains.

For synchronize_srcu(), redirect atomic SRCU domains to synchronize_srcu_atomic().

Add srcu_reader_flavor to the Tiny SRCU state for these checks. Tiny SRCU does not currently set the flavor for atomic SRCU domains, but keeping the flavor in the common state allows the operation checks to enforce the restriction once atomic flavor tracking is enabled.

Also initialize srcu_atomic_gp_flag, which was previously left uninitialized.

---
 include/linux/srcutiny.h |  1 +
 kernel/rcu/srcutiny.c    | 12 ++++++++++++
 2 files changed, 13 insertions(+)
include/linux/srcutiny.h +1 -0
index 47a368f945e3..2b293336525a 100644
@@ -20,6 +20,7 @@ struct srcu_struct {
      u8 srcu_gp_running;             /* GP workqueue running? */
      u8 srcu_gp_waiting;             /* GP waiting for readers? */
      u8 srcu_atomic_gp_flag;         /* Serialize atomic GP work.*/
+     u8 srcu_reader_flavor;          /* Values: SRCU_READ_FLAVOR_.*  */
      unsigned long srcu_idx;         /* Current reader array element in bit 0x2. */
      unsigned long srcu_idx_max;     /* Furthest future srcu_idx request. */
      struct swait_queue_head srcu_wq;
kernel/rcu/srcutiny.c +0 -0
index 26ea4bfbeaf2..22f7716cbb0e 100644

kernel/rcu/srcutiny.c static int init_srcu_struct_fields(struct srcu_struct *ssp) · line 44

Quoted 4 lines
kernel/rcu/srcutiny.c +1 -0
@@ -41,6 +41,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp)
      ssp->srcu_cb_tail = &ssp->srcu_cb_head;
      ssp->srcu_gp_running = false;
      ssp->srcu_gp_waiting = false;
+     ssp->srcu_atomic_gp_flag = 0;

Good catch! I will be folding this into the base commit with attribution on my next rebase:

9a2e9996ccec ("srcutiny: Add an atomic Tiny SRCU")

Quoted 24 lines
kernel/rcu/srcutiny.c +8 -0
      ssp->srcu_idx = 0;
      ssp->srcu_idx_max = 0;
      INIT_WORK(&ssp->srcu_work, srcu_drive_gp);
@@ -289,6 +290,9 @@ EXPORT_SYMBOL_GPL(srcu_defer_drain);
 void call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp,
             rcu_callback_t func)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      if (should_rcu_defer()) {
              /* A re-entrant call_srcu() during the drain would livelock it. */
              if (READ_ONCE(srcu_defer_draining) && !in_nmi()) {
@@ -319,6 +323,11 @@ void synchronize_srcu(struct srcu_struct *ssp)
 {
      struct rcu_synchronize rs;
 
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC)) {
+             synchronize_srcu_atomic(ssp);
+             return;
+     }
+
      srcu_lock_sync(&ssp->dep_map);
 
Quoted 15 lines

kernel/rcu/srcutiny.c EXPORT_SYMBOL_GPL(synchronize_srcu_atomic); · line 432

Quoted 11 lines
kernel/rcu/srcutiny.c +3 -0
      RCU_LOCKDEP_WARN(lockdep_is_held(ssp) ||
@@ -415,6 +424,9 @@ EXPORT_SYMBOL_GPL(synchronize_srcu_atomic);
 /* Register any deferred callbacks, then wait for all in-flight ones. */
 void srcu_barrier(struct srcu_struct *ssp)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      __srcu_defer_drain(ssp);
      synchronize_srcu(ssp);
 }

The rest is good as far as it goes, but don't we need to set the value of ssp->srcu_reader_flavor somewhere for atomic srcu_struct structures?

Hi Paul,

Yes, agreed. I have the flavor tracking changes implemented locally and am testing them now.
I’ll send the updated patch shortly.

Quoted 5 lines
Quoted 3 lines

I also have draft patches for the documentation and the fast path, as well as rcutorture testing for
tiny atomic srcu , which I’ll send separately.

Sounds good, and I am looking forward to seeing them.

Hi Paul,
I’ve sent the flavor tracking changes as [1].
[1] https://lore.kernel.org/all/[email protected]/

Quoted 4 lines

I will admit that I am curious as to why Tiny Atomic SRCU needs different rcutorture testing than does Tree Atomic SRCU, but I will ask myself that question again when I see your patches. ;-)

You’re right that the existing SRCU torture configuration can exercise the atomic path with rcutorture.reader_flavor=0x10. I had prepared the Tiny atomic torture test as a separate draft, mainly adding local configurations for convenience while validating the Tiny and Tree atomic paths. There is no Tiny-specific torture model here.

I can adjust the final rcutorture configuration and naming based on your preference.

Thanks,
KunWu

Quoted 2 lines

Thanx, Paul

maintainer
#5 lore raw
Quoted 122 lines · KunWu Chan
Quoted 106 lines · Paul E. McKenney
Quoted 104 lines · KunWu Chan
Quoted 81 lines · Paul E. McKenney
Quoted 45 lines · Kunwu Chan

From: Kunwu Chan <[email protected]>

Add atomic SRCU operation checks and the associated state to Tiny SRCU.

An atomic SRCU domain does not use the normal SRCU callback and grace-period machinery. In particular, a callback queued with call_srcu() would never be processed. Use WARN_ON_ONCE() to reject call_srcu() and srcu_barrier() on atomic SRCU domains.

For synchronize_srcu(), redirect atomic SRCU domains to synchronize_srcu_atomic().

Add srcu_reader_flavor to the Tiny SRCU state for these checks. Tiny SRCU does not currently set the flavor for atomic SRCU domains, but keeping the flavor in the common state allows the operation checks to enforce the restriction once atomic flavor tracking is enabled.

Also initialize srcu_atomic_gp_flag, which was previously left uninitialized.

---
 include/linux/srcutiny.h |  1 +
 kernel/rcu/srcutiny.c    | 12 ++++++++++++
 2 files changed, 13 insertions(+)
include/linux/srcutiny.h +1 -0
index 47a368f945e3..2b293336525a 100644
@@ -20,6 +20,7 @@ struct srcu_struct {
      u8 srcu_gp_running;             /* GP workqueue running? */
      u8 srcu_gp_waiting;             /* GP waiting for readers? */
      u8 srcu_atomic_gp_flag;         /* Serialize atomic GP work.*/
+     u8 srcu_reader_flavor;          /* Values: SRCU_READ_FLAVOR_.*  */
      unsigned long srcu_idx;         /* Current reader array element in bit 0x2. */
      unsigned long srcu_idx_max;     /* Furthest future srcu_idx request. */
      struct swait_queue_head srcu_wq;
kernel/rcu/srcutiny.c +0 -0
index 26ea4bfbeaf2..22f7716cbb0e 100644

kernel/rcu/srcutiny.c static int init_srcu_struct_fields(struct srcu_struct *ssp) · line 44

Quoted 4 lines
kernel/rcu/srcutiny.c +1 -0
@@ -41,6 +41,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp)
      ssp->srcu_cb_tail = &ssp->srcu_cb_head;
      ssp->srcu_gp_running = false;
      ssp->srcu_gp_waiting = false;
+     ssp->srcu_atomic_gp_flag = 0;

Good catch! I will be folding this into the base commit with attribution on my next rebase:

9a2e9996ccec ("srcutiny: Add an atomic Tiny SRCU")

Quoted 24 lines
kernel/rcu/srcutiny.c +8 -0
      ssp->srcu_idx = 0;
      ssp->srcu_idx_max = 0;
      INIT_WORK(&ssp->srcu_work, srcu_drive_gp);
@@ -289,6 +290,9 @@ EXPORT_SYMBOL_GPL(srcu_defer_drain);
 void call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp,
             rcu_callback_t func)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      if (should_rcu_defer()) {
              /* A re-entrant call_srcu() during the drain would livelock it. */
              if (READ_ONCE(srcu_defer_draining) && !in_nmi()) {
@@ -319,6 +323,11 @@ void synchronize_srcu(struct srcu_struct *ssp)
 {
      struct rcu_synchronize rs;
 
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC)) {
+             synchronize_srcu_atomic(ssp);
+             return;
+     }
+
      srcu_lock_sync(&ssp->dep_map);
 
Quoted 15 lines

kernel/rcu/srcutiny.c EXPORT_SYMBOL_GPL(synchronize_srcu_atomic); · line 432

Quoted 11 lines
kernel/rcu/srcutiny.c +3 -0
      RCU_LOCKDEP_WARN(lockdep_is_held(ssp) ||
@@ -415,6 +424,9 @@ EXPORT_SYMBOL_GPL(synchronize_srcu_atomic);
 /* Register any deferred callbacks, then wait for all in-flight ones. */
 void srcu_barrier(struct srcu_struct *ssp)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      __srcu_defer_drain(ssp);
      synchronize_srcu(ssp);
 }

The rest is good as far as it goes, but don't we need to set the value of ssp->srcu_reader_flavor somewhere for atomic srcu_struct structures?

Hi Paul,

Yes, agreed. I have the flavor tracking changes implemented locally and am testing them now.
I’ll send the updated patch shortly.

Quoted 5 lines
Quoted 3 lines

I also have draft patches for the documentation and the fast path, as well as rcutorture testing for
tiny atomic srcu , which I’ll send separately.

Sounds good, and I am looking forward to seeing them.

Hi Paul,
I’ve sent the flavor tracking changes as [1].
[1] https://lore.kernel.org/all/[email protected]/

Quoted 4 lines

I will admit that I am curious as to why Tiny Atomic SRCU needs different rcutorture testing than does Tree Atomic SRCU, but I will ask myself that question again when I see your patches. ;-)

Quoted 8 lines

You’re right that the existing SRCU torture configuration can exercise the atomic path with rcutorture.reader_flavor=0x10. I had prepared the Tiny atomic torture test as a separate draft, mainly adding local configurations for convenience while validating the Tiny and Tree atomic paths. There is no Tiny-specific torture model here.

I can adjust the final rcutorture configuration and naming based on your preference.

The best approach is to add a section to this script:

tools/testing/selftests/rcutorture/bin/torture.sh

This would require adding a command-line parameter such as --do-atomic-srcu and friends. The effect would be that people like me would run short tests of atomic SRCU frequently.

Thanx, Paul

author
#6 lore raw
Quoted 133 lines · Paul E. McKenney
Quoted 122 lines · KunWu Chan
Quoted 106 lines · Paul E. McKenney
Quoted 104 lines · KunWu Chan
Quoted 81 lines · Paul E. McKenney
Quoted 45 lines · Kunwu Chan

From: Kunwu Chan <[email protected]>

Add atomic SRCU operation checks and the associated state to Tiny SRCU.

An atomic SRCU domain does not use the normal SRCU callback and grace-period machinery. In particular, a callback queued with call_srcu() would never be processed. Use WARN_ON_ONCE() to reject call_srcu() and srcu_barrier() on atomic SRCU domains.

For synchronize_srcu(), redirect atomic SRCU domains to synchronize_srcu_atomic().

Add srcu_reader_flavor to the Tiny SRCU state for these checks. Tiny SRCU does not currently set the flavor for atomic SRCU domains, but keeping the flavor in the common state allows the operation checks to enforce the restriction once atomic flavor tracking is enabled.

Also initialize srcu_atomic_gp_flag, which was previously left uninitialized.

---
 include/linux/srcutiny.h |  1 +
 kernel/rcu/srcutiny.c    | 12 ++++++++++++
 2 files changed, 13 insertions(+)
include/linux/srcutiny.h +1 -0
index 47a368f945e3..2b293336525a 100644
@@ -20,6 +20,7 @@ struct srcu_struct {
      u8 srcu_gp_running;             /* GP workqueue running? */
      u8 srcu_gp_waiting;             /* GP waiting for readers? */
      u8 srcu_atomic_gp_flag;         /* Serialize atomic GP work.*/
+     u8 srcu_reader_flavor;          /* Values: SRCU_READ_FLAVOR_.*  */
      unsigned long srcu_idx;         /* Current reader array element in bit 0x2. */
      unsigned long srcu_idx_max;     /* Furthest future srcu_idx request. */
      struct swait_queue_head srcu_wq;
kernel/rcu/srcutiny.c +0 -0
index 26ea4bfbeaf2..22f7716cbb0e 100644

kernel/rcu/srcutiny.c static int init_srcu_struct_fields(struct srcu_struct *ssp) · line 44

Quoted 4 lines
kernel/rcu/srcutiny.c +1 -0
@@ -41,6 +41,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp)
      ssp->srcu_cb_tail = &ssp->srcu_cb_head;
      ssp->srcu_gp_running = false;
      ssp->srcu_gp_waiting = false;
+     ssp->srcu_atomic_gp_flag = 0;

Good catch! I will be folding this into the base commit with attribution on my next rebase:

9a2e9996ccec ("srcutiny: Add an atomic Tiny SRCU")

Quoted 24 lines
kernel/rcu/srcutiny.c +8 -0
      ssp->srcu_idx = 0;
      ssp->srcu_idx_max = 0;
      INIT_WORK(&ssp->srcu_work, srcu_drive_gp);
@@ -289,6 +290,9 @@ EXPORT_SYMBOL_GPL(srcu_defer_drain);
 void call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp,
             rcu_callback_t func)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      if (should_rcu_defer()) {
              /* A re-entrant call_srcu() during the drain would livelock it. */
              if (READ_ONCE(srcu_defer_draining) && !in_nmi()) {
@@ -319,6 +323,11 @@ void synchronize_srcu(struct srcu_struct *ssp)
 {
      struct rcu_synchronize rs;
 
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC)) {
+             synchronize_srcu_atomic(ssp);
+             return;
+     }
+
      srcu_lock_sync(&ssp->dep_map);
 
Quoted 15 lines

kernel/rcu/srcutiny.c EXPORT_SYMBOL_GPL(synchronize_srcu_atomic); · line 432

Quoted 11 lines
kernel/rcu/srcutiny.c +3 -0
      RCU_LOCKDEP_WARN(lockdep_is_held(ssp) ||
@@ -415,6 +424,9 @@ EXPORT_SYMBOL_GPL(synchronize_srcu_atomic);
 /* Register any deferred callbacks, then wait for all in-flight ones. */
 void srcu_barrier(struct srcu_struct *ssp)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      __srcu_defer_drain(ssp);
      synchronize_srcu(ssp);
 }

The rest is good as far as it goes, but don't we need to set the value of ssp->srcu_reader_flavor somewhere for atomic srcu_struct structures?

Hi Paul,

Yes, agreed. I have the flavor tracking changes implemented locally and am testing them now.
I’ll send the updated patch shortly.

Quoted 5 lines
Quoted 3 lines

I also have draft patches for the documentation and the fast path, as well as rcutorture testing for
tiny atomic srcu , which I’ll send separately.

Sounds good, and I am looking forward to seeing them.

Hi Paul,
I’ve sent the flavor tracking changes as [1].
[1] https://lore.kernel.org/all/[email protected]/

Quoted 4 lines

I will admit that I am curious as to why Tiny Atomic SRCU needs different rcutorture testing than does Tree Atomic SRCU, but I will ask myself that question again when I see your patches. ;-)

Quoted 8 lines

You’re right that the existing SRCU torture configuration can exercise the atomic path with rcutorture.reader_flavor=0x10. I had prepared the Tiny atomic torture test as a separate draft, mainly adding local configurations for convenience while validating the Tiny and Tree atomic paths. There is no Tiny-specific torture model here.

I can adjust the final rcutorture configuration and naming based on your preference.

Quoted 7 lines

The best approach is to add a section to this script:

tools/testing/selftests/rcutorture/bin/torture.sh

This would require adding a command-line parameter such as --do-atomic-srcu and friends. The effect would be that people like me would run short tests of atomic SRCU frequently.

Thanks Paul. I’ll add an atomic SRCU option to torture.sh so that it can be run as a short, frequent test.

During my atomic SRCU testing, I noticed that kvm-recheck-rcu.sh can report “Reader Batch close calls” even though the kernel reports End of test: SUCCESS with no WARNING/BUG. The atomic GP rate is also much higher than normal SRCU.

I’m still checking whether this is inherent to atomic SRCU, related to my earlier fastpath change, or an interaction with the torture reader delay. I’ll continue investigating this as part of the testing work.

Thanks,
Kunwu

Quoted 2 lines

Thanx, Paul

maintainer
#7 lore raw
Quoted 136 lines · KunWu Chan
Quoted 135 lines · Paul E. McKenney
Quoted 122 lines · KunWu Chan
Quoted 106 lines · Paul E. McKenney
Quoted 104 lines · KunWu Chan
Quoted 81 lines · Paul E. McKenney
Quoted 45 lines · Kunwu Chan

From: Kunwu Chan <[email protected]>

Add atomic SRCU operation checks and the associated state to Tiny SRCU.

An atomic SRCU domain does not use the normal SRCU callback and grace-period machinery. In particular, a callback queued with call_srcu() would never be processed. Use WARN_ON_ONCE() to reject call_srcu() and srcu_barrier() on atomic SRCU domains.

For synchronize_srcu(), redirect atomic SRCU domains to synchronize_srcu_atomic().

Add srcu_reader_flavor to the Tiny SRCU state for these checks. Tiny SRCU does not currently set the flavor for atomic SRCU domains, but keeping the flavor in the common state allows the operation checks to enforce the restriction once atomic flavor tracking is enabled.

Also initialize srcu_atomic_gp_flag, which was previously left uninitialized.

---
 include/linux/srcutiny.h |  1 +
 kernel/rcu/srcutiny.c    | 12 ++++++++++++
 2 files changed, 13 insertions(+)
include/linux/srcutiny.h +1 -0
index 47a368f945e3..2b293336525a 100644
@@ -20,6 +20,7 @@ struct srcu_struct {
      u8 srcu_gp_running;             /* GP workqueue running? */
      u8 srcu_gp_waiting;             /* GP waiting for readers? */
      u8 srcu_atomic_gp_flag;         /* Serialize atomic GP work.*/
+     u8 srcu_reader_flavor;          /* Values: SRCU_READ_FLAVOR_.*  */
      unsigned long srcu_idx;         /* Current reader array element in bit 0x2. */
      unsigned long srcu_idx_max;     /* Furthest future srcu_idx request. */
      struct swait_queue_head srcu_wq;
kernel/rcu/srcutiny.c +0 -0
index 26ea4bfbeaf2..22f7716cbb0e 100644

kernel/rcu/srcutiny.c static int init_srcu_struct_fields(struct srcu_struct *ssp) · line 44

Quoted 4 lines
kernel/rcu/srcutiny.c +1 -0
@@ -41,6 +41,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp)
      ssp->srcu_cb_tail = &ssp->srcu_cb_head;
      ssp->srcu_gp_running = false;
      ssp->srcu_gp_waiting = false;
+     ssp->srcu_atomic_gp_flag = 0;

Good catch! I will be folding this into the base commit with attribution on my next rebase:

9a2e9996ccec ("srcutiny: Add an atomic Tiny SRCU")

Quoted 24 lines
kernel/rcu/srcutiny.c +8 -0
      ssp->srcu_idx = 0;
      ssp->srcu_idx_max = 0;
      INIT_WORK(&ssp->srcu_work, srcu_drive_gp);
@@ -289,6 +290,9 @@ EXPORT_SYMBOL_GPL(srcu_defer_drain);
 void call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp,
             rcu_callback_t func)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      if (should_rcu_defer()) {
              /* A re-entrant call_srcu() during the drain would livelock it. */
              if (READ_ONCE(srcu_defer_draining) && !in_nmi()) {
@@ -319,6 +323,11 @@ void synchronize_srcu(struct srcu_struct *ssp)
 {
      struct rcu_synchronize rs;
 
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC)) {
+             synchronize_srcu_atomic(ssp);
+             return;
+     }
+
      srcu_lock_sync(&ssp->dep_map);
 
Quoted 15 lines

kernel/rcu/srcutiny.c EXPORT_SYMBOL_GPL(synchronize_srcu_atomic); · line 432

Quoted 11 lines
kernel/rcu/srcutiny.c +3 -0
      RCU_LOCKDEP_WARN(lockdep_is_held(ssp) ||
@@ -415,6 +424,9 @@ EXPORT_SYMBOL_GPL(synchronize_srcu_atomic);
 /* Register any deferred callbacks, then wait for all in-flight ones. */
 void srcu_barrier(struct srcu_struct *ssp)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      __srcu_defer_drain(ssp);
      synchronize_srcu(ssp);
 }

The rest is good as far as it goes, but don't we need to set the value of ssp->srcu_reader_flavor somewhere for atomic srcu_struct structures?

Hi Paul,

Yes, agreed. I have the flavor tracking changes implemented locally and am testing them now.
I’ll send the updated patch shortly.

Quoted 5 lines
Quoted 3 lines

I also have draft patches for the documentation and the fast path, as well as rcutorture testing for
tiny atomic srcu , which I’ll send separately.

Sounds good, and I am looking forward to seeing them.

Hi Paul,
I’ve sent the flavor tracking changes as [1].
[1] https://lore.kernel.org/all/[email protected]/

Quoted 4 lines

I will admit that I am curious as to why Tiny Atomic SRCU needs different rcutorture testing than does Tree Atomic SRCU, but I will ask myself that question again when I see your patches. ;-)

Quoted 8 lines

You’re right that the existing SRCU torture configuration can exercise the atomic path with rcutorture.reader_flavor=0x10. I had prepared the Tiny atomic torture test as a separate draft, mainly adding local configurations for convenience while validating the Tiny and Tree atomic paths. There is no Tiny-specific torture model here.

I can adjust the final rcutorture configuration and naming based on your preference.

The best approach is to add a section to this script:

Quoted 8 lines
Quoted 5 lines

tools/testing/selftests/rcutorture/bin/torture.sh

This would require adding a command-line parameter such as --do-atomic-srcu and friends. The effect would be that people like me would run short tests of atomic SRCU frequently.

Thanks Paul. I’ll add an atomic SRCU option to torture.sh so that it can be run as a short, frequent test.

Very good, looking forward to seeing it!

Quoted 8 lines

During my atomic SRCU testing, I noticed that kvm-recheck-rcu.sh can report “Reader Batch close calls” even though the kernel reports End of test: SUCCESS with no WARNING/BUG. The atomic GP rate is also much higher than normal SRCU.

I’m still checking whether this is inherent to atomic SRCU, related to my earlier fastpath change, or an interaction with the torture reader delay. I’ll continue investigating this as part of the testing work.

The close calls can happen due to memory reordering. But this is new code, so please do check them carefully.

The high atomic grace-period rate is expected behavior. On my x86 laptop, I see about 75/second for Tree Atomic SRCU and about 300 for Tiny. But at your end, how high is high?

Thanx, Paul

author
#8 lore raw
Quoted 154 lines · Paul E. McKenney
Quoted 136 lines · KunWu Chan
Quoted 135 lines · Paul E. McKenney
Quoted 122 lines · KunWu Chan
Quoted 106 lines · Paul E. McKenney
Quoted 104 lines · KunWu Chan
Quoted 81 lines · Paul E. McKenney
Quoted 45 lines · Kunwu Chan

From: Kunwu Chan <[email protected]>

Add atomic SRCU operation checks and the associated state to Tiny SRCU.

An atomic SRCU domain does not use the normal SRCU callback and grace-period machinery. In particular, a callback queued with call_srcu() would never be processed. Use WARN_ON_ONCE() to reject call_srcu() and srcu_barrier() on atomic SRCU domains.

For synchronize_srcu(), redirect atomic SRCU domains to synchronize_srcu_atomic().

Add srcu_reader_flavor to the Tiny SRCU state for these checks. Tiny SRCU does not currently set the flavor for atomic SRCU domains, but keeping the flavor in the common state allows the operation checks to enforce the restriction once atomic flavor tracking is enabled.

Also initialize srcu_atomic_gp_flag, which was previously left uninitialized.

---
 include/linux/srcutiny.h |  1 +
 kernel/rcu/srcutiny.c    | 12 ++++++++++++
 2 files changed, 13 insertions(+)
include/linux/srcutiny.h +1 -0
index 47a368f945e3..2b293336525a 100644
@@ -20,6 +20,7 @@ struct srcu_struct {
      u8 srcu_gp_running;             /* GP workqueue running? */
      u8 srcu_gp_waiting;             /* GP waiting for readers? */
      u8 srcu_atomic_gp_flag;         /* Serialize atomic GP work.*/
+     u8 srcu_reader_flavor;          /* Values: SRCU_READ_FLAVOR_.*  */
      unsigned long srcu_idx;         /* Current reader array element in bit 0x2. */
      unsigned long srcu_idx_max;     /* Furthest future srcu_idx request. */
      struct swait_queue_head srcu_wq;
kernel/rcu/srcutiny.c +0 -0
index 26ea4bfbeaf2..22f7716cbb0e 100644

kernel/rcu/srcutiny.c static int init_srcu_struct_fields(struct srcu_struct *ssp) · line 44

Quoted 4 lines
kernel/rcu/srcutiny.c +1 -0
@@ -41,6 +41,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp)
      ssp->srcu_cb_tail = &ssp->srcu_cb_head;
      ssp->srcu_gp_running = false;
      ssp->srcu_gp_waiting = false;
+     ssp->srcu_atomic_gp_flag = 0;

Good catch! I will be folding this into the base commit with attribution on my next rebase:

9a2e9996ccec ("srcutiny: Add an atomic Tiny SRCU")

Quoted 24 lines
kernel/rcu/srcutiny.c +8 -0
      ssp->srcu_idx = 0;
      ssp->srcu_idx_max = 0;
      INIT_WORK(&ssp->srcu_work, srcu_drive_gp);
@@ -289,6 +290,9 @@ EXPORT_SYMBOL_GPL(srcu_defer_drain);
 void call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp,
             rcu_callback_t func)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      if (should_rcu_defer()) {
              /* A re-entrant call_srcu() during the drain would livelock it. */
              if (READ_ONCE(srcu_defer_draining) && !in_nmi()) {
@@ -319,6 +323,11 @@ void synchronize_srcu(struct srcu_struct *ssp)
 {
      struct rcu_synchronize rs;
 
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC)) {
+             synchronize_srcu_atomic(ssp);
+             return;
+     }
+
      srcu_lock_sync(&ssp->dep_map);
 
Quoted 15 lines

kernel/rcu/srcutiny.c EXPORT_SYMBOL_GPL(synchronize_srcu_atomic); · line 432

Quoted 11 lines
kernel/rcu/srcutiny.c +3 -0
      RCU_LOCKDEP_WARN(lockdep_is_held(ssp) ||
@@ -415,6 +424,9 @@ EXPORT_SYMBOL_GPL(synchronize_srcu_atomic);
 /* Register any deferred callbacks, then wait for all in-flight ones. */
 void srcu_barrier(struct srcu_struct *ssp)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      __srcu_defer_drain(ssp);
      synchronize_srcu(ssp);
 }

The rest is good as far as it goes, but don't we need to set the value of ssp->srcu_reader_flavor somewhere for atomic srcu_struct structures?

Hi Paul,

Yes, agreed. I have the flavor tracking changes implemented locally and am testing them now.
I’ll send the updated patch shortly.

Quoted 5 lines
Quoted 3 lines

I also have draft patches for the documentation and the fast path, as well as rcutorture testing for
tiny atomic srcu , which I’ll send separately.

Sounds good, and I am looking forward to seeing them.

Hi Paul,
I’ve sent the flavor tracking changes as [1].
[1] https://lore.kernel.org/all/[email protected]/

Quoted 4 lines

I will admit that I am curious as to why Tiny Atomic SRCU needs different rcutorture testing than does Tree Atomic SRCU, but I will ask myself that question again when I see your patches. ;-)

Quoted 8 lines

You’re right that the existing SRCU torture configuration can exercise the atomic path with rcutorture.reader_flavor=0x10. I had prepared the Tiny atomic torture test as a separate draft, mainly adding local configurations for convenience while validating the Tiny and Tree atomic paths. There is no Tiny-specific torture model here.

I can adjust the final rcutorture configuration and naming based on your preference.

The best approach is to add a section to this script:

Quoted 8 lines
Quoted 5 lines

tools/testing/selftests/rcutorture/bin/torture.sh

This would require adding a command-line parameter such as --do-atomic-srcu and friends. The effect would be that people like me would run short tests of atomic SRCU frequently.

Thanks Paul. I’ll add an atomic SRCU option to torture.sh so that it can be run as a short, frequent test.

Very good, looking forward to seeing it!

Quoted 5 lines

During my atomic SRCU testing, I noticed that kvm-recheck-rcu.sh can report “Reader Batch close calls” even though the kernel reports End of test: SUCCESS with no WARNING/BUG. The atomic GP rate is also much higher than normal SRCU.

Quoted 7 lines
Quoted 3 lines

I’m still checking whether this is inherent to atomic SRCU, related to my earlier fastpath change, or an interaction with the torture reader delay. I’ll continue investigating this as part of the testing work.

The close calls can happen due to memory reordering. But this is new code, so please do check them carefully.

Thanks, Paul. That gives me a useful reference point.

Quoted 3 lines

The high atomic grace-period rate is expected behavior. On my x86 laptop, I see about 75/second for Tree Atomic SRCU and about 300 for Tiny. But at your end, how high is high?

On my x86 PC, with the earlier fastpath change, I was seeing:
Tree Atomic SRCU: about 250–530 GPs/sec
Tiny Atomic SRCU: about 290 GPs/sec

After reverting the fastpath change, Tree Atomic SRCU dropped to about 129 GPs/sec, while Tiny remained around 291 GPs/sec. So the Tiny result is close to your ~300/sec reference, while the Tree result is somewhat higher than your ~75/sec observation.

The close-call behavior also changed with the fastpath. With the fastpath, batch[2] was non-zero (6–334 in my runs). After reverting it, batch[2] was consistently 0. batch[1] can still occur on Tree SRCU, but I have not seen readers spanning 3 or more GPs.

I'll continue checking the close-call behavior with repeated torture runs, with particular attention to the memory-ordering aspect you mentioned.

Thanks,
Kunwu

Quoted 2 lines

Thanx, Paul

maintainer
#9 lore raw
Quoted 172 lines · KunWu Chan
Quoted 149 lines · Paul E. McKenney
Quoted 132 lines · KunWu Chan
Quoted 131 lines · Paul E. McKenney
Quoted 119 lines · KunWu Chan
Quoted 103 lines · Paul E. McKenney
Quoted 102 lines · KunWu Chan
Quoted 79 lines · Paul E. McKenney
Quoted 44 lines · Kunwu Chan

From: Kunwu Chan <[email protected]>
Add atomic SRCU operation checks and the associated state to Tiny SRCU.

An atomic SRCU domain does not use the normal SRCU callback and grace-period machinery. In particular, a callback queued with call_srcu() would never be processed. Use WARN_ON_ONCE() to reject call_srcu() and srcu_barrier() on atomic SRCU domains.

For synchronize_srcu(), redirect atomic SRCU domains to synchronize_srcu_atomic().

Add srcu_reader_flavor to the Tiny SRCU state for these checks. Tiny SRCU does not currently set the flavor for atomic SRCU domains, but keeping the flavor in the common state allows the operation checks to enforce the restriction once atomic flavor tracking is enabled.

Also initialize srcu_atomic_gp_flag, which was previously left uninitialized.

---
 include/linux/srcutiny.h |  1 +
 kernel/rcu/srcutiny.c    | 12 ++++++++++++
 2 files changed, 13 insertions(+)
include/linux/srcutiny.h +1 -0
index 47a368f945e3..2b293336525a 100644
@@ -20,6 +20,7 @@ struct srcu_struct {
      u8 srcu_gp_running;             /* GP workqueue running? */
      u8 srcu_gp_waiting;             /* GP waiting for readers? */
      u8 srcu_atomic_gp_flag;         /* Serialize atomic GP work.*/
+     u8 srcu_reader_flavor;          /* Values: SRCU_READ_FLAVOR_.*  */
      unsigned long srcu_idx;         /* Current reader array element in bit 0x2. */
      unsigned long srcu_idx_max;     /* Furthest future srcu_idx request. */
      struct swait_queue_head srcu_wq;
kernel/rcu/srcutiny.c +0 -0
index 26ea4bfbeaf2..22f7716cbb0e 100644

kernel/rcu/srcutiny.c static int init_srcu_struct_fields(struct srcu_struct *ssp) · line 44

Quoted 4 lines
kernel/rcu/srcutiny.c +1 -0
@@ -41,6 +41,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp)
      ssp->srcu_cb_tail = &ssp->srcu_cb_head;
      ssp->srcu_gp_running = false;
      ssp->srcu_gp_waiting = false;
+     ssp->srcu_atomic_gp_flag = 0;

Good catch! I will be folding this into the base commit with attribution on my next rebase:

9a2e9996ccec ("srcutiny: Add an atomic Tiny SRCU")

Quoted 24 lines
kernel/rcu/srcutiny.c +8 -0
      ssp->srcu_idx = 0;
      ssp->srcu_idx_max = 0;
      INIT_WORK(&ssp->srcu_work, srcu_drive_gp);
@@ -289,6 +290,9 @@ EXPORT_SYMBOL_GPL(srcu_defer_drain);
 void call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp,
             rcu_callback_t func)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      if (should_rcu_defer()) {
              /* A re-entrant call_srcu() during the drain would livelock it. */
              if (READ_ONCE(srcu_defer_draining) && !in_nmi()) {
@@ -319,6 +323,11 @@ void synchronize_srcu(struct srcu_struct *ssp)
 {
      struct rcu_synchronize rs;
 
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC)) {
+             synchronize_srcu_atomic(ssp);
+             return;
+     }
+
      srcu_lock_sync(&ssp->dep_map);
 
Quoted 15 lines

kernel/rcu/srcutiny.c EXPORT_SYMBOL_GPL(synchronize_srcu_atomic); · line 432

Quoted 11 lines
kernel/rcu/srcutiny.c +3 -0
      RCU_LOCKDEP_WARN(lockdep_is_held(ssp) ||
@@ -415,6 +424,9 @@ EXPORT_SYMBOL_GPL(synchronize_srcu_atomic);
 /* Register any deferred callbacks, then wait for all in-flight ones. */
 void srcu_barrier(struct srcu_struct *ssp)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      __srcu_defer_drain(ssp);
      synchronize_srcu(ssp);
 }

The rest is good as far as it goes, but don't we need to set the value of ssp->srcu_reader_flavor somewhere for atomic srcu_struct structures?

Hi Paul,

Yes, agreed. I have the flavor tracking changes implemented locally and am testing them now.
I’ll send the updated patch shortly.

Quoted 5 lines
Quoted 3 lines

I also have draft patches for the documentation and the fast path, as well as rcutorture testing for
tiny atomic srcu , which I’ll send separately.

Sounds good, and I am looking forward to seeing them.

Hi Paul,
I’ve sent the flavor tracking changes as [1].
[1] https://lore.kernel.org/all/[email protected]/

Quoted 4 lines

I will admit that I am curious as to why Tiny Atomic SRCU needs different rcutorture testing than does Tree Atomic SRCU, but I will ask myself that question again when I see your patches. ;-)

Quoted 8 lines

You’re right that the existing SRCU torture configuration can exercise the atomic path with rcutorture.reader_flavor=0x10. I had prepared the Tiny atomic torture test as a separate draft, mainly adding local configurations for convenience while validating the Tiny and Tree atomic paths. There is no Tiny-specific torture model here.

I can adjust the final rcutorture configuration and naming based on your preference.

The best approach is to add a section to this script:

Quoted 8 lines
Quoted 5 lines

tools/testing/selftests/rcutorture/bin/torture.sh

This would require adding a command-line parameter such as --do-atomic-srcu and friends. The effect would be that people like me would run short tests of atomic SRCU frequently.

Thanks Paul. I’ll add an atomic SRCU option to torture.sh so that it can be run as a short, frequent test.

Very good, looking forward to seeing it!

Quoted 5 lines

During my atomic SRCU testing, I noticed that kvm-recheck-rcu.sh can report “Reader Batch close calls” even though the kernel reports End of test: SUCCESS with no WARNING/BUG. The atomic GP rate is also much higher than normal SRCU.

Quoted 6 lines
Quoted 3 lines

I’m still checking whether this is inherent to atomic SRCU, related to my earlier fastpath change, or an interaction with the torture reader delay. I’ll continue investigating this as part of the testing work.

The close calls can happen due to memory reordering. But this is new code, so please do check them carefully.

Thanks, Paul. That gives me a useful reference point.

Quoted 3 lines

The high atomic grace-period rate is expected behavior. On my x86 laptop, I see about 75/second for Tree Atomic SRCU and about 300 for Tiny. But at your end, how high is high?

On my x86 PC, with the earlier fastpath change, I was seeing:
Tree Atomic SRCU: about 250–530 GPs/sec
Tiny Atomic SRCU: about 290 GPs/sec

After reverting the fastpath change, Tree Atomic SRCU dropped to about 129 GPs/sec, while Tiny remained around 291 GPs/sec. So the Tiny result is close to your ~300/sec reference, while the Tree result is somewhat higher than your ~75/sec observation.

Quoted 4 lines

The close-call behavior also changed with the fastpath. With the fastpath, batch[2] was non-zero (6–334 in my runs). After reverting it, batch[2] was consistently 0. batch[1] can still occur on Tree SRCU, but I have not seen readers spanning 3 or more GPs.

These results suggest that you should very carefully check memory ordering in the fastpath code.

Quoted 2 lines

I'll continue checking the close-call behavior with repeated torture runs, with particular attention to the memory-ordering aspect you mentioned.

Sounds good, thank you!

Thanx, Paul

author
#10 lore raw
Quoted 174 lines · Paul E. McKenney
Quoted 172 lines · KunWu Chan
Quoted 149 lines · Paul E. McKenney
Quoted 132 lines · KunWu Chan
Quoted 131 lines · Paul E. McKenney
Quoted 119 lines · KunWu Chan
Quoted 103 lines · Paul E. McKenney
Quoted 102 lines · KunWu Chan
Quoted 79 lines · Paul E. McKenney
Quoted 44 lines · Kunwu Chan

From: Kunwu Chan <[email protected]>
Add atomic SRCU operation checks and the associated state to Tiny SRCU.

An atomic SRCU domain does not use the normal SRCU callback and grace-period machinery. In particular, a callback queued with call_srcu() would never be processed. Use WARN_ON_ONCE() to reject call_srcu() and srcu_barrier() on atomic SRCU domains.

For synchronize_srcu(), redirect atomic SRCU domains to synchronize_srcu_atomic().

Add srcu_reader_flavor to the Tiny SRCU state for these checks. Tiny SRCU does not currently set the flavor for atomic SRCU domains, but keeping the flavor in the common state allows the operation checks to enforce the restriction once atomic flavor tracking is enabled.

Also initialize srcu_atomic_gp_flag, which was previously left uninitialized.

---
 include/linux/srcutiny.h |  1 +
 kernel/rcu/srcutiny.c    | 12 ++++++++++++
 2 files changed, 13 insertions(+)
include/linux/srcutiny.h +1 -0
index 47a368f945e3..2b293336525a 100644
@@ -20,6 +20,7 @@ struct srcu_struct {
      u8 srcu_gp_running;             /* GP workqueue running? */
      u8 srcu_gp_waiting;             /* GP waiting for readers? */
      u8 srcu_atomic_gp_flag;         /* Serialize atomic GP work.*/
+     u8 srcu_reader_flavor;          /* Values: SRCU_READ_FLAVOR_.*  */
      unsigned long srcu_idx;         /* Current reader array element in bit 0x2. */
      unsigned long srcu_idx_max;     /* Furthest future srcu_idx request. */
      struct swait_queue_head srcu_wq;
kernel/rcu/srcutiny.c +0 -0
index 26ea4bfbeaf2..22f7716cbb0e 100644

kernel/rcu/srcutiny.c static int init_srcu_struct_fields(struct srcu_struct *ssp) · line 44

Quoted 4 lines
kernel/rcu/srcutiny.c +1 -0
@@ -41,6 +41,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp)
      ssp->srcu_cb_tail = &ssp->srcu_cb_head;
      ssp->srcu_gp_running = false;
      ssp->srcu_gp_waiting = false;
+     ssp->srcu_atomic_gp_flag = 0;

Good catch! I will be folding this into the base commit with attribution on my next rebase:

9a2e9996ccec ("srcutiny: Add an atomic Tiny SRCU")

Quoted 24 lines
kernel/rcu/srcutiny.c +8 -0
      ssp->srcu_idx = 0;
      ssp->srcu_idx_max = 0;
      INIT_WORK(&ssp->srcu_work, srcu_drive_gp);
@@ -289,6 +290,9 @@ EXPORT_SYMBOL_GPL(srcu_defer_drain);
 void call_srcu(struct srcu_struct *ssp, struct rcu_head *rhp,
             rcu_callback_t func)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      if (should_rcu_defer()) {
              /* A re-entrant call_srcu() during the drain would livelock it. */
              if (READ_ONCE(srcu_defer_draining) && !in_nmi()) {
@@ -319,6 +323,11 @@ void synchronize_srcu(struct srcu_struct *ssp)
 {
      struct rcu_synchronize rs;
 
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC)) {
+             synchronize_srcu_atomic(ssp);
+             return;
+     }
+
      srcu_lock_sync(&ssp->dep_map);
 
Quoted 15 lines

kernel/rcu/srcutiny.c EXPORT_SYMBOL_GPL(synchronize_srcu_atomic); · line 432

Quoted 11 lines
kernel/rcu/srcutiny.c +3 -0
      RCU_LOCKDEP_WARN(lockdep_is_held(ssp) ||
@@ -415,6 +424,9 @@ EXPORT_SYMBOL_GPL(synchronize_srcu_atomic);
 /* Register any deferred callbacks, then wait for all in-flight ones. */
 void srcu_barrier(struct srcu_struct *ssp)
 {
+     if (WARN_ON_ONCE(ssp->srcu_reader_flavor == SRCU_READ_FLAVOR_ATOMIC))
+             return;
+
      __srcu_defer_drain(ssp);
      synchronize_srcu(ssp);
 }

The rest is good as far as it goes, but don't we need to set the value of ssp->srcu_reader_flavor somewhere for atomic srcu_struct structures?

Hi Paul,

Yes, agreed. I have the flavor tracking changes implemented locally and am testing them now.
I’ll send the updated patch shortly.

Quoted 5 lines
Quoted 3 lines

I also have draft patches for the documentation and the fast path, as well as rcutorture testing for
tiny atomic srcu , which I’ll send separately.

Sounds good, and I am looking forward to seeing them.

Hi Paul,
I’ve sent the flavor tracking changes as [1].
[1] https://lore.kernel.org/all/[email protected]/

Quoted 4 lines

I will admit that I am curious as to why Tiny Atomic SRCU needs different rcutorture testing than does Tree Atomic SRCU, but I will ask myself that question again when I see your patches. ;-)

Quoted 8 lines

You’re right that the existing SRCU torture configuration can exercise the atomic path with rcutorture.reader_flavor=0x10. I had prepared the Tiny atomic torture test as a separate draft, mainly adding local configurations for convenience while validating the Tiny and Tree atomic paths. There is no Tiny-specific torture model here.

I can adjust the final rcutorture configuration and naming based on your preference.

The best approach is to add a section to this script:

Quoted 8 lines
Quoted 5 lines

tools/testing/selftests/rcutorture/bin/torture.sh

This would require adding a command-line parameter such as --do-atomic-srcu and friends. The effect would be that people like me would run short tests of atomic SRCU frequently.

Thanks Paul. I’ll add an atomic SRCU option to torture.sh so that it can be run as a short, frequent test.

Very good, looking forward to seeing it!

Quoted 5 lines

During my atomic SRCU testing, I noticed that kvm-recheck-rcu.sh can report “Reader Batch close calls” even though the kernel reports End of test: SUCCESS with no WARNING/BUG. The atomic GP rate is also much higher than normal SRCU.

Quoted 6 lines
Quoted 3 lines

I’m still checking whether this is inherent to atomic SRCU, related to my earlier fastpath change, or an interaction with the torture reader delay. I’ll continue investigating this as part of the testing work.

The close calls can happen due to memory reordering. But this is new code, so please do check them carefully.

Thanks, Paul. That gives me a useful reference point.

Quoted 3 lines

The high atomic grace-period rate is expected behavior. On my x86 laptop, I see about 75/second for Tree Atomic SRCU and about 300 for Tiny. But at your end, how high is high?

On my x86 PC, with the earlier fastpath change, I was seeing:
Tree Atomic SRCU: about 250–530 GPs/sec
Tiny Atomic SRCU: about 290 GPs/sec

After reverting the fastpath change, Tree Atomic SRCU dropped to about 129 GPs/sec, while Tiny remained around 291 GPs/sec. So the Tiny result is close to your ~300/sec reference, while the Tree result is somewhat higher than your ~75/sec observation.

Quoted 8 lines
Quoted 4 lines

The close-call behavior also changed with the fastpath. With the fastpath, batch[2] was non-zero (6–334 in my runs). After reverting it, batch[2] was consistently 0. batch[1] can still occur on Tree SRCU, but I have not seen readers spanning 3 or more GPs.

These results suggest that you should very carefully check memory ordering in the fastpath code.

Thanks, Paul. I appreciate the reminder.
I'll carefully verify the memory-ordering assumptions in the fastpath and run more torture tests with the relevant configurations/architectures.

Quoted 4 lines
Quoted 2 lines

I'll continue checking the close-call behavior with repeated torture runs, with particular attention to the memory-ordering aspect you mentioned.

Sounds good, thank you!

Thanks again!
Kunwu

Quoted 2 lines

Thanx, Paul