drbd: fix throttling on newly created DM backing devices
Subsystems (2)
PATCH v3 1/1 drbd: fix throttling on newly created DM backing devices
From: "Palik, Imre" <[email protected]>
If the drbd backing device is a new device mapper device (e.g., a dm-linear mapping of an existing block device that contains data), the counters are initially 0 even though the device contains useful data. This causes throttling until something accesses the drbd device or the backing device.
The patch disables throttling, as long as only resync is responsible for disk activity on a freshly created device.
--- drivers/block/drbd/drbd_int.h | 4 ++-- drivers/block/drbd/drbd_receiver.c | 10 +++++----- 2 files changed, 7 insertions(+), 7 deletions(-)
drivers/block/drbd/drbd_int.h +2 -2
@@ -960,8 +960,8 @@ struct drbd_device {atomic_t rs_sect_in;/* for incoming resync data rate, SyncTarget */ atomic_t rs_sect_ev;/* for submitted resync data rate, both */ int rs_last_sect_ev;/* counter to compare with */ -int rs_last_events;/* counter of read or write "events" (unit sectors) -* on the lower level device when we last looked. */ +unsigned int rs_last_events;/* counter of read or write "events" (unit sectors) +* on the lower level device when we last looked. */ int c_sync_rate;/* current resync rate after syncer throttle magic */ struct fifo_buffer *rs_plan_s;/* correction values of resync planer (RCU, connection->conn_update) */ int rs_in_flight;/* resync sectors in flight (to proxy, in proxy and from proxy) */
drivers/block/drbd/drbd_receiver.c +5 -5
@@ -2467,7 +2467,7 @@ bool drbd_rs_c_min_rate_throttle(struct drbd_device *device)struct gendisk *disk = device->ldev->backing_bdev->bd_contains->bd_disk;unsigned long db, dt, dbdt;unsigned int c_min_rate; -int curr_events; +unsigned int curr_events;rcu_read_lock (); c_min_rate =rcu_dereference (device->ldev->disk_conf)->c_min_rate;
@@ -2477,12 +2477,12 @@ bool drbd_rs_c_min_rate_throttle(struct drbd_device *device)if (c_min_rate ==0 )return false; - curr_events = (int )part_stat_read (&disk->part0, sectors[0 ]) + - (int )part_stat_read (&disk->part0, sectors[1 ]) - -atomic_read (&device->rs_sect_ev); + curr_events = (unsigned int )part_stat_read (&disk->part0, sectors[0 ]) + + (unsigned int )part_stat_read (&disk->part0, sectors[1 ]) - + (unsigned int )atomic_read (&device->rs_sect_ev);if (atomic_read (&device->ap_actlog_cnt) - || !device->rs_last_events || curr_events - device->rs_last_events >64 ) { + || curr_events - device->rs_last_events >64 ) {unsigned long rs_left;int i;
Signature · 1 line
1.7.9.5
Lars
maintainerQuoted 7 lines · Imre Palik
From: "Palik, Imre" <[email protected]>
If the drbd backing device is a new device mapper device (e.g., a dm-linear mapping of an existing block device that contains data), the counters are initially 0 even though the device contains useful data. This causes throttling until something accesses the drbd device or the backing device.
What was wrong with my previous proposal?
How does changing the signedness help withrs_last_events not being properly initialized?
Are you sure you have also considered all wrap-around cases?
Maybe you are too focused on your particular corner case (disk_stats starting with 0).
Maybe I'm just thick right now, so please explain.
Lars
Quoted 59 lines
The patch disables throttling, as long as only resync is responsible for disk activity on a freshly created device.
Reported-by Mikhail Sugakov <[email protected]>Cc Matt Wilson <[email protected]>Signed-off-by Imre Palik <[email protected]>--- drivers/block/drbd/drbd_int.h | 4 ++-- drivers/block/drbd/drbd_receiver.c | 10 +++++----- 2 files changed, 7 insertions(+), 7 deletions(-)drivers/block/drbd/drbd_int.h +2
-2@@ -960,8 +960,8 @@ struct drbd_device {atomic_t rs_sect_in;/* for incoming resync data rate, SyncTarget */ atomic_t rs_sect_ev;/* for submitted resync data rate, both */ int rs_last_sect_ev;/* counter to compare with */ -int rs_last_events;/* counter of read or write "events" (unit sectors) -* on the lower level device when we last looked. */ +unsigned int rs_last_events;/* counter of read or write "events" (unit sectors) +* on the lower level device when we last looked. */ int c_sync_rate;/* current resync rate after syncer throttle magic */ struct fifo_buffer *rs_plan_s;/* correction values of resync planer (RCU, connection->conn_update) */ int rs_in_flight;/* resync sectors in flight (to proxy, in proxy and from proxy) */ drivers/block/drbd/drbd_receiver.c +5
-5@@ -2467,7 +2467,7 @@ bool drbd_rs_c_min_rate_throttle(struct drbd_device *device)struct gendisk *disk = device->ldev->backing_bdev->bd_contains->bd_disk;unsigned long db, dt, dbdt;unsigned int c_min_rate; -int curr_events; +unsigned int curr_events;rcu_read_lock (); c_min_rate =rcu_dereference (device->ldev->disk_conf)->c_min_rate;@@ -2477,12 +2477,12 @@ bool drbd_rs_c_min_rate_throttle(struct drbd_device *device)if (c_min_rate ==0 )return false; - curr_events = (int )part_stat_read (&disk->part0, sectors[0 ]) + - (int )part_stat_read (&disk->part0, sectors[1 ]) - -atomic_read (&device->rs_sect_ev); + curr_events = (unsigned int )part_stat_read (&disk->part0, sectors[0 ]) + + (unsigned int )part_stat_read (&disk->part0, sectors[1 ]) - + (unsigned int )atomic_read (&device->rs_sect_ev);if (atomic_read (&device->ap_actlog_cnt) - || !device->rs_last_events || curr_events - device->rs_last_events >64 ) { + || curr_events - device->rs_last_events >64 ) {unsigned long rs_left;int i;Signature · 1 line
1.7.9.5
Imre Palik
Reviewed-byQuoted 10 lines · Lars
Quoted 7 lines · Imre Palik
From: "Palik, Imre" <[email protected]>
If the drbd backing device is a new device mapper device (e.g., a dm-linear mapping of an existing block device that contains data), the counters are initially 0 even though the device contains useful data. This causes throttling until something accesses the drbd device or the backing device.
What was wrong with my previous proposal?
Sorry, I haven't realised you added a proposal to your reply. It seems, I really needed that extra sleep during the weekend ...
Your proposal is good. Of course, I like my last one a slightly better.
But as they say, beauty is in the eye of the beholder :-)
Quoted 2 lines
How does changing the signedness help with
rs_last_eventsnot being properly initialized?
It only helps with reasoning. I reason with modular arithmetic way easier than with signed integer overflows. Accidentally, 0 is a good initialisation value in case of unsigned arithmetic.
Quoted 5 lines
Are you sure you have also considered all wrap-around cases?
Maybe you are too focused on your particular corner case (
disk_statsstarting with 0).
Maybe I'm just thick right now, so please explain.
The idea is that 0 is the smallest possible value for an unsigned, and curr_events is monotonically increasing (mod 2^32) . This means, initially either curr_events > 64, that is, we enter the loop, and do the initialisation, or it will be bigger than 64 at most when we want to start throttle in an ideal world (after no more than 64 sectors of activity).
Basically, while you initialise rs_last_events to an ideal value with some calculation, I choose a safe static value. I am content with both approaches. I think, as a subsystem maintainer, you should choose the one you like better. If you choose yours, then you can add
Imre
Quoted 62 lines
Lars
Quoted 59 lines
The patch disables throttling, as long as only resync is responsible for disk activity on a freshly created device.
Reported-by Mikhail Sugakov <[email protected]>Cc Matt Wilson <[email protected]>Signed-off-by Imre Palik <[email protected]>--- drivers/block/drbd/drbd_int.h | 4 ++-- drivers/block/drbd/drbd_receiver.c | 10 +++++----- 2 files changed, 7 insertions(+), 7 deletions(-)drivers/block/drbd/drbd_int.h +2
-2@@ -960,8 +960,8 @@ struct drbd_device {atomic_t rs_sect_in;/* for incoming resync data rate, SyncTarget */ atomic_t rs_sect_ev;/* for submitted resync data rate, both */ int rs_last_sect_ev;/* counter to compare with */ -int rs_last_events;/* counter of read or write "events" (unit sectors) -* on the lower level device when we last looked. */ +unsigned int rs_last_events;/* counter of read or write "events" (unit sectors) +* on the lower level device when we last looked. */ int c_sync_rate;/* current resync rate after syncer throttle magic */ struct fifo_buffer *rs_plan_s;/* correction values of resync planer (RCU, connection->conn_update) */ int rs_in_flight;/* resync sectors in flight (to proxy, in proxy and from proxy) */ drivers/block/drbd/drbd_receiver.c +5
-6@@ -2467,7 +2467,7 @@ bool drbd_rs_c_min_rate_throttle(struct drbd_device *device)struct gendisk *disk = device->ldev->backing_bdev->bd_contains->bd_disk;unsigned long db, dt, dbdt;unsigned int c_min_rate; -int curr_events; +unsigned int curr_events;rcu_read_lock (); c_min_rate =rcu_dereference (device->ldev->disk_conf)->c_min_rate;@@ -2477,12 +2477,12 @@ bool drbd_rs_c_min_rate_throttle(struct drbd_device *device)if (c_min_rate ==0 )return false; - curr_events = (int )part_stat_read (&disk->part0, sectors[0 ]) + - (int )part_stat_read (&disk->part0, sectors[1 ]) - -atomic_read (&device->rs_sect_ev); + curr_events = (unsigned int )part_stat_read (&disk->part0, sectors[0 ]) + + (unsigned int )part_stat_read (&disk->part0, sectors[1 ]) - + (unsigned int )atomic_read (&device->rs_sect_ev);if (atomic_read (&device->ap_actlog_cnt) - || !device->rs_last_events || curr_events - device->rs_last_events >64 ) { + || curr_events - device->rs_last_events >64 ) {unsigned long rs_left;int i; --1.7.9.5
Lars
maintainerQuoted 25 lines · Imre Palik
Quoted 10 lines · Lars
Quoted 7 lines · Imre Palik
From: "Palik, Imre" <[email protected]>
If the drbd backing device is a new device mapper device (e.g., a dm-linear mapping of an existing block device that contains data), the counters are initially 0 even though the device contains useful data. This causes throttling until something accesses the drbd device or the backing device.
What was wrong with my previous proposal?
Sorry, I haven't realised you added a proposal to your reply. It seems, I really needed that extra sleep during the weekend ...
Your proposal is good. Of course, I like my last one a slightly better. But as they say, beauty is in the eye of the beholder :-)
Quoted 2 lines
How does changing the signedness help with
rs_last_eventsnot being properly initialized?It only helps with reasoning. I reason with modular arithmetic way easier than with signed integer overflows. Accidentally, 0 is a good initialisation value in case of unsigned arithmetic.
Quoted 8 lines
Quoted 5 lines
Are you sure you have also considered all wrap-around cases?
Maybe you are too focused on your particular corner case (
disk_statsstarting with 0).
Maybe I'm just thick right now, so please explain.The idea is that 0 is the smallest possible value for an unsigned, and
curr_eventsis monotonically increasing (mod 2^32) .
The problem is: it is not :-(
It's a difference between stats that are increased by the block core at (usually) completion time, and an atomic_t that is increased by DRBD at just before (or just after) submittion.
Depending very much on stress in the IO subsystem, and overall timing of events, a later call may see a smaller "curr_events" (because rs_last_sect_ev has already increased, but the disk stats have not yet noticed).
With unsigned, that may wrap around to UINT_MAX, which we don't want.
Quoted 6 lines
This
means, initially eithercurr_events> 64, that is, we enter the loop, and do the initialisation, or it will be bigger than 64 at most when we want to start throttle in an ideal world (after no more than 64 sectors of activity).
Quoted 6 lines
Basically, while you initialise
rs_last_eventsto an ideal value with some calculation, I choose a safe static value. I am content with both approaches. I think, as a subsystem maintainer, you should choose the one you like better. If you choose yours, then you can addReviewed-by Imre Palik <[email protected]>
Thanks,
Lars
Quoted 41 lines · Lars
Quoted 25 lines · Imre Palik
Quoted 10 lines · Lars
Quoted 7 lines · Imre Palik
From: "Palik, Imre" <[email protected]>
If the drbd backing device is a new device mapper device (e.g., a dm-linear mapping of an existing block device that contains data), the counters are initially 0 even though the device contains useful data. This causes throttling until something accesses the drbd device or the backing device.
What was wrong with my previous proposal?
Sorry, I haven't realised you added a proposal to your reply. It seems, I really needed that extra sleep during the weekend ...
Your proposal is good. Of course, I like my last one a slightly better. But as they say, beauty is in the eye of the beholder :-)
Quoted 2 lines
How does changing the signedness help with
rs_last_eventsnot being properly initialized?It only helps with reasoning. I reason with modular arithmetic way easier than with signed integer overflows. Accidentally, 0 is a good initialisation value in case of unsigned arithmetic.
Quoted 8 lines
Quoted 5 lines
Are you sure you have also considered all wrap-around cases?
Maybe you are too focused on your particular corner case (
disk_statsstarting with 0).
Maybe I'm just thick right now, so please explain.The idea is that 0 is the smallest possible value for an unsigned, and
curr_eventsis monotonically increasing (mod 2^32) .The problem is: it is not :-(
It's a difference between stats that are increased by the block core at (usually) completion time, and an
atomic_tthat is increased by DRBD at just before (or just after) submittion.
Quoted 6 lines
Depending very much on stress in the IO subsystem, and overall timing of events, a later call may see a smaller "
curr_events" (becausers_last_sect_evhas already increased, but the disk stats have not yet noticed).With unsigned, that may wrap around to
UINT_MAX, which we don't want.
I see. You hide the jitter behind the signedness. Thanks for the explanation.
Quoted 17 lines
Quoted 6 lines
This
means, initially eithercurr_events> 64, that is, we enter the loop, and do the initialisation, or it will be bigger than 64 at most when we want to start throttle in an ideal world (after no more than 64 sectors of activity).Quoted 6 lines
Basically, while you initialise
rs_last_eventsto an ideal value with some calculation, I choose a safe static value. I am content with both approaches. I think, as a subsystem maintainer, you should choose the one you like better. If you choose yours, then you can addReviewed-by Imre Palik <[email protected]>Thanks,
Lars