Skip to contentSkip to conversation

drbd: fix throttling on newly created DM backing devices

PATCH In review Imre Palik 2 participants 5 messages Sep 5, 2014 – Sep 9, 2014

Subsystems (2)
lore
Share & tools
lore

Imre Palik

author
#0 lore raw

PATCH v3 1/1 drbd: fix throttling on newly created DM backing devices

From: "Palik, Imre" <[email protected]>

If the drbd backing device is a new device mapper device (e.g., a dm-linear mapping of an existing block device that contains data), the counters are initially 0 even though the device contains useful data. This causes throttling until something accesses the drbd device or the backing device.

The patch disables throttling, as long as only resync is responsible for disk activity on a freshly created device.

---
 drivers/block/drbd/drbd_int.h      |    4 ++--
 drivers/block/drbd/drbd_receiver.c |   10 +++++-----
 2 files changed, 7 insertions(+), 7 deletions(-)
drivers/block/drbd/drbd_int.h +2 -2
index 1a00001..298b1dc 100644
@@ -960,8 +960,8 @@ struct drbd_device {
 	atomic_t rs_sect_in; /* for incoming resync data rate, SyncTarget */
 	atomic_t rs_sect_ev; /* for submitted resync data rate, both */
 	int rs_last_sect_ev; /* counter to compare with */
-	int rs_last_events;  /* counter of read or write "events" (unit sectors)
-			      * on the lower level device when we last looked. */
+	unsigned int rs_last_events;  /* counter of read or write "events" (unit sectors)
+				       * on the lower level device when we last looked. */
 	int c_sync_rate; /* current resync rate after syncer throttle magic */
 	struct fifo_buffer *rs_plan_s; /* correction values of resync planer (RCU, connection->conn_update) */
 	int rs_in_flight; /* resync sectors in flight (to proxy, in proxy and from proxy) */
drivers/block/drbd/drbd_receiver.c +5 -5
index 9342b8d..147c917 100644
@@ -2467,7 +2467,7 @@ bool drbd_rs_c_min_rate_throttle(struct drbd_device *device)
 	struct gendisk *disk = device->ldev->backing_bdev->bd_contains->bd_disk;
 	unsigned long db, dt, dbdt;
 	unsigned int c_min_rate;
-	int curr_events;
+	unsigned int curr_events;
 
 	rcu_read_lock();
 	c_min_rate = rcu_dereference(device->ldev->disk_conf)->c_min_rate;
@@ -2477,12 +2477,12 @@ bool drbd_rs_c_min_rate_throttle(struct drbd_device *device)
 	if (c_min_rate == 0)
 		return false;
 
-	curr_events = (int)part_stat_read(&disk->part0, sectors[0]) +
-		      (int)part_stat_read(&disk->part0, sectors[1]) -
-			atomic_read(&device->rs_sect_ev);
+	curr_events = (unsigned int)part_stat_read(&disk->part0, sectors[0]) +
+		      (unsigned int)part_stat_read(&disk->part0, sectors[1]) -
+		(unsigned int)atomic_read(&device->rs_sect_ev);
 
 	if (atomic_read(&device->ap_actlog_cnt)
-	    || !device->rs_last_events || curr_events - device->rs_last_events > 64) {
+		|| curr_events - device->rs_last_events > 64) {
 		unsigned long rs_left;
 		int i;
 
Signature · 1 line
1.7.9.5
maintainer
#1 lore raw
Quoted 7 lines · Imre Palik

From: "Palik, Imre" <[email protected]>

If the drbd backing device is a new device mapper device (e.g., a dm-linear mapping of an existing block device that contains data), the counters are initially 0 even though the device contains useful data. This causes throttling until something accesses the drbd device or the backing device.

What was wrong with my previous proposal?

How does changing the signedness help with
rs_last_events not being properly initialized?

Are you sure you have also considered all wrap-around cases?

Maybe you are too focused on your particular corner case (disk_stats starting with 0).
Maybe I'm just thick right now, so please explain.

Lars

Quoted 59 lines

The patch disables throttling, as long as only resync is responsible for disk activity on a freshly created device.

---
 drivers/block/drbd/drbd_int.h      |    4 ++--
 drivers/block/drbd/drbd_receiver.c |   10 +++++-----
 2 files changed, 7 insertions(+), 7 deletions(-)
drivers/block/drbd/drbd_int.h +2 -2
index 1a00001..298b1dc 100644
@@ -960,8 +960,8 @@ struct drbd_device {
 	atomic_t rs_sect_in; /* for incoming resync data rate, SyncTarget */
 	atomic_t rs_sect_ev; /* for submitted resync data rate, both */
 	int rs_last_sect_ev; /* counter to compare with */
-	int rs_last_events;  /* counter of read or write "events" (unit sectors)
-			      * on the lower level device when we last looked. */
+	unsigned int rs_last_events;  /* counter of read or write "events" (unit sectors)
+				       * on the lower level device when we last looked. */
 	int c_sync_rate; /* current resync rate after syncer throttle magic */
 	struct fifo_buffer *rs_plan_s; /* correction values of resync planer (RCU, connection->conn_update) */
 	int rs_in_flight; /* resync sectors in flight (to proxy, in proxy and from proxy) */
drivers/block/drbd/drbd_receiver.c +5 -5
index 9342b8d..147c917 100644
@@ -2467,7 +2467,7 @@ bool drbd_rs_c_min_rate_throttle(struct drbd_device *device)
 	struct gendisk *disk = device->ldev->backing_bdev->bd_contains->bd_disk;
 	unsigned long db, dt, dbdt;
 	unsigned int c_min_rate;
-	int curr_events;
+	unsigned int curr_events;
 
 	rcu_read_lock();
 	c_min_rate = rcu_dereference(device->ldev->disk_conf)->c_min_rate;
@@ -2477,12 +2477,12 @@ bool drbd_rs_c_min_rate_throttle(struct drbd_device *device)
 	if (c_min_rate == 0)
 		return false;
 
-	curr_events = (int)part_stat_read(&disk->part0, sectors[0]) +
-		      (int)part_stat_read(&disk->part0, sectors[1]) -
-			atomic_read(&device->rs_sect_ev);
+	curr_events = (unsigned int)part_stat_read(&disk->part0, sectors[0]) +
+		      (unsigned int)part_stat_read(&disk->part0, sectors[1]) -
+		(unsigned int)atomic_read(&device->rs_sect_ev);
 
 	if (atomic_read(&device->ap_actlog_cnt)
-	    || !device->rs_last_events || curr_events - device->rs_last_events > 64) {
+		|| curr_events - device->rs_last_events > 64) {
 		unsigned long rs_left;
 		int i;
 
Signature · 1 line
1.7.9.5
author Reviewed-by
#2 lore raw
Quoted 10 lines · Lars
Quoted 7 lines · Imre Palik

From: "Palik, Imre" <[email protected]>

If the drbd backing device is a new device mapper device (e.g., a dm-linear mapping of an existing block device that contains data), the counters are initially 0 even though the device contains useful data. This causes throttling until something accesses the drbd device or the backing device.

What was wrong with my previous proposal?

Sorry, I haven't realised you added a proposal to your reply. It seems, I really needed that extra sleep during the weekend ...

Your proposal is good. Of course, I like my last one a slightly better.
But as they say, beauty is in the eye of the beholder :-)

Quoted 2 lines

How does changing the signedness help with
rs_last_events not being properly initialized?

It only helps with reasoning. I reason with modular arithmetic way easier than with signed integer overflows. Accidentally, 0 is a good initialisation value in case of unsigned arithmetic.

Quoted 5 lines

Are you sure you have also considered all wrap-around cases?

Maybe you are too focused on your particular corner case (disk_stats starting with 0).
Maybe I'm just thick right now, so please explain.

The idea is that 0 is the smallest possible value for an unsigned, and curr_events is monotonically increasing (mod 2^32) . This means, initially either curr_events > 64, that is, we enter the loop, and do the initialisation, or it will be bigger than 64 at most when we want to start throttle in an ideal world (after no more than 64 sectors of activity).

Basically, while you initialise rs_last_events to an ideal value with some calculation, I choose a safe static value. I am content with both approaches. I think, as a subsystem maintainer, you should choose the one you like better. If you choose yours, then you can add

Imre

Quoted 62 lines

Lars

Quoted 59 lines

The patch disables throttling, as long as only resync is responsible for disk activity on a freshly created device.

---
  drivers/block/drbd/drbd_int.h      |    4 ++--
  drivers/block/drbd/drbd_receiver.c |   10 +++++-----
  2 files changed, 7 insertions(+), 7 deletions(-)
drivers/block/drbd/drbd_int.h +2 -2
index 1a00001..298b1dc 100644
@@ -960,8 +960,8 @@ struct drbd_device {
  	atomic_t rs_sect_in; /* for incoming resync data rate, SyncTarget */
  	atomic_t rs_sect_ev; /* for submitted resync data rate, both */
  	int rs_last_sect_ev; /* counter to compare with */
-	int rs_last_events;  /* counter of read or write "events" (unit sectors)
-			      * on the lower level device when we last looked. */
+	unsigned int rs_last_events;  /* counter of read or write "events" (unit sectors)
+				       * on the lower level device when we last looked. */
  	int c_sync_rate; /* current resync rate after syncer throttle magic */
  	struct fifo_buffer *rs_plan_s; /* correction values of resync planer (RCU, connection->conn_update) */
  	int rs_in_flight; /* resync sectors in flight (to proxy, in proxy and from proxy) */
drivers/block/drbd/drbd_receiver.c +5 -6
index 9342b8d..147c917 100644
@@ -2467,7 +2467,7 @@ bool drbd_rs_c_min_rate_throttle(struct drbd_device *device)
  	struct gendisk *disk = device->ldev->backing_bdev->bd_contains->bd_disk;
  	unsigned long db, dt, dbdt;
  	unsigned int c_min_rate;
-	int curr_events;
+	unsigned int curr_events;
 
  	rcu_read_lock();
  	c_min_rate = rcu_dereference(device->ldev->disk_conf)->c_min_rate;
@@ -2477,12 +2477,12 @@ bool drbd_rs_c_min_rate_throttle(struct drbd_device *device)
  	if (c_min_rate == 0)
  		return false;
 
-	curr_events = (int)part_stat_read(&disk->part0, sectors[0]) +
-		      (int)part_stat_read(&disk->part0, sectors[1]) -
-			atomic_read(&device->rs_sect_ev);
+	curr_events = (unsigned int)part_stat_read(&disk->part0, sectors[0]) +
+		      (unsigned int)part_stat_read(&disk->part0, sectors[1]) -
+		(unsigned int)atomic_read(&device->rs_sect_ev);
 
  	if (atomic_read(&device->ap_actlog_cnt)
-	    || !device->rs_last_events || curr_events - device->rs_last_events > 64) {
+		|| curr_events - device->rs_last_events > 64) {
  		unsigned long rs_left;
  		int i;
 
--

1.7.9.5

maintainer
#3 lore raw
Quoted 25 lines · Imre Palik
Quoted 10 lines · Lars
Quoted 7 lines · Imre Palik

From: "Palik, Imre" <[email protected]>

If the drbd backing device is a new device mapper device (e.g., a dm-linear mapping of an existing block device that contains data), the counters are initially 0 even though the device contains useful data. This causes throttling until something accesses the drbd device or the backing device.

What was wrong with my previous proposal?

Sorry, I haven't realised you added a proposal to your reply. It seems, I really needed that extra sleep during the weekend ...

Your proposal is good. Of course, I like my last one a slightly better. But as they say, beauty is in the eye of the beholder :-)

Quoted 2 lines

How does changing the signedness help with
rs_last_events not being properly initialized?

It only helps with reasoning. I reason with modular arithmetic way easier than with signed integer overflows. Accidentally, 0 is a good initialisation value in case of unsigned arithmetic.

Quoted 8 lines
Quoted 5 lines

Are you sure you have also considered all wrap-around cases?

Maybe you are too focused on your particular corner case (disk_stats starting with 0).
Maybe I'm just thick right now, so please explain.

The idea is that 0 is the smallest possible value for an unsigned, and curr_events is monotonically increasing (mod 2^32) .

The problem is: it is not :-(

It's a difference between stats that are increased by the block core at (usually) completion time, and an atomic_t that is increased by DRBD at just before (or just after) submittion.

Depending very much on stress in the IO subsystem, and overall timing of events, a later call may see a smaller "curr_events" (because rs_last_sect_ev has already increased, but the disk stats have not yet noticed).

With unsigned, that may wrap around to UINT_MAX, which we don't want.

Quoted 6 lines

This
means, initially either curr_events > 64, that is, we enter the loop, and do the initialisation, or it will be bigger than 64 at most when we want to start throttle in an ideal world (after no more than 64 sectors of activity).

Quoted 6 lines

Basically, while you initialise rs_last_events to an ideal value with some calculation, I choose a safe static value. I am content with both approaches. I think, as a subsystem maintainer, you should choose the one you like better. If you choose yours, then you can add

Thanks,

Lars

author
#4 lore raw
Quoted 41 lines · Lars
Quoted 25 lines · Imre Palik
Quoted 10 lines · Lars
Quoted 7 lines · Imre Palik

From: "Palik, Imre" <[email protected]>

If the drbd backing device is a new device mapper device (e.g., a dm-linear mapping of an existing block device that contains data), the counters are initially 0 even though the device contains useful data. This causes throttling until something accesses the drbd device or the backing device.

What was wrong with my previous proposal?

Sorry, I haven't realised you added a proposal to your reply. It seems, I really needed that extra sleep during the weekend ...

Your proposal is good. Of course, I like my last one a slightly better. But as they say, beauty is in the eye of the beholder :-)

Quoted 2 lines

How does changing the signedness help with
rs_last_events not being properly initialized?

It only helps with reasoning. I reason with modular arithmetic way easier than with signed integer overflows. Accidentally, 0 is a good initialisation value in case of unsigned arithmetic.

Quoted 8 lines
Quoted 5 lines

Are you sure you have also considered all wrap-around cases?

Maybe you are too focused on your particular corner case (disk_stats starting with 0).
Maybe I'm just thick right now, so please explain.

The idea is that 0 is the smallest possible value for an unsigned, and curr_events is monotonically increasing (mod 2^32) .

The problem is: it is not :-(

It's a difference between stats that are increased by the block core at (usually) completion time, and an atomic_t that is increased by DRBD at just before (or just after) submittion.

Quoted 6 lines

Depending very much on stress in the IO subsystem, and overall timing of events, a later call may see a smaller "curr_events" (because rs_last_sect_ev has already increased, but the disk stats have not yet noticed).

With unsigned, that may wrap around to UINT_MAX, which we don't want.

I see. You hide the jitter behind the signedness. Thanks for the explanation.

Quoted 17 lines
Quoted 6 lines

This
means, initially either curr_events > 64, that is, we enter the loop, and do the initialisation, or it will be bigger than 64 at most when we want to start throttle in an ideal world (after no more than 64 sectors of activity).

Quoted 6 lines

Basically, while you initialise rs_last_events to an ideal value with some calculation, I choose a safe static value. I am content with both approaches. I think, as a subsystem maintainer, you should choose the one you like better. If you choose yours, then you can add

Thanks,

Lars