Skip to content

RFC: Non blocking submit for activity log misses

Philipp Reisner drbd-dev lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest18 patches 4 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/18drbd: cleanup bogus assert messagedrivers/block/drbd/drbd_int.h drivers/block/drbd/drbd_main.c+4 -100 replies
2/18drbd: cleanup ondisk meta data layout calculations and definesdrivers/block/drbd/drbd_actlog.c drivers/block/drbd/drbd_bitmap.c drivers/block/drbd/drbd_int.h +2 more+123 -570 replies
3/18drbd: prepare for new striped layout of activity logdrivers/block/drbd/drbd_actlog.c drivers/block/drbd/drbd_int.h drivers/block/drbd/drbd_main.c +1 more+94 -400 replies
4/18drbd: use the cached meta_dev_idxdrivers/block/drbd/drbd_int.h drivers/block/drbd/drbd_nl.c+6 -330 replies
5/18drbd: mechanically rename la_size to la_size_sectdrivers/block/drbd/drbd_main.c drivers/block/drbd/drbd_nl.c drivers/block/drbd/drbd_receiver.c+12 -120 replies
6/18drbd: read meta data early, base on-disk offsets on super blockdrivers/block/drbd/drbd_actlog.c drivers/block/drbd/drbd_main.c drivers/block/drbd/drbd_nl.c +1 more+123 -370 replies
7/18drbd: Clarify when activity log I/O is delegated to the worker threaddrivers/block/drbd/drbd_actlog.c drivers/block/drbd/drbd_int.h drivers/block/drbd/drbd_receiver.c +2 more+31 -260 replies
8/18drbd: drbd_al_being_io: short circuit to reduce latencydrivers/block/drbd/drbd_actlog.c+12 -20 replies
9/18drbd: split __drbd_make_request in before and after drbd_al_begin_iodrivers/block/drbd/drbd_req.c+30 -100 replies
10/18drbd: prepare to queue write requests on a submit workerdrivers/block/drbd/drbd_int.h drivers/block/drbd/drbd_main.c drivers/block/drbd/drbd_nl.c +1 more+67 -12 replies
11/18drbd: split drbd_al_begin_io into fastpath, prepare, and commitdrivers/block/drbd/drbd_actlog.c drivers/block/drbd/drbd_int.h+72 -330 replies
12/18drbd: split out some helper functions to drbd_al_begin_iodrivers/block/drbd/drbd_actlog.c+28 -210 replies
13/18drbd: queue writes on submitter thread, unless they pass the activity log fastpathdrivers/block/drbd/drbd_req.c+12 -80 replies
14/18lru_cache: introduce lc_get_cumulative()include/linux/lru_cache.h lib/lru_cache.c+46 -100 replies
15/18drbd: consolidate as many updates as possible into one AL transactiondrivers/block/drbd/drbd_actlog.c drivers/block/drbd/drbd_int.h drivers/block/drbd/drbd_req.c+107 -140 replies
16/18drbd: move start io accounting before activity log transactiondrivers/block/drbd/drbd_req.c+3 -30 replies
17/18drbd: try hard to max out the updates per AL transactiondrivers/block/drbd/drbd_req.c+31 -00 replies
18/18drbd: adjust upper limit for activity log extentsdrivers/block/drbd/drbd_nl.c include/linux/drbd_limits.h+43 -150 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 00/18] RFC: Non blocking submit for activity log misses

The Issues

Since the beginning DRBD was written with the assumption that the write pattern has spacial locality. (This assumption was driven from the fact, that rotating media performs better if you do not send the head too far too often)

Backed by this assumption a caller that submits a request that is outside of the current active set, was blocked until the active set was changed. (Changing the active set is a synchronous write operation to the meta-data area on the backing storage = "an AL-update" in DRBD-speak)

A second effect was that DRBD's meta-data was located in a very narrow area. When DRBD is used on top of a RAID0 stripe set, this causes all AL-updates to got to the same disk.

The Proposed Solution

This patch series improves DRBD's behavior. A submitter is no longer blocked in the case of a AL-miss. For this a dedicated submitter worker is introduced (patch 13).

In order to better distribute the AL-updates to more disks in a stripe set this patch series also introduces an optional striped layout of the part of the meta-data that holds the AL-updates (patch 4).

The Results

This of course drastically improves DRBD's performance if the write pattern does not have any spacial locality. E.g. random writes spread out over the whole device.

In the test systems we have SSDs with are able to do up to 50000 writes per second. The test does random distributed writes over a work set size of 128GiB with IO depths from 1 to 1024.

At an IO depth of 64:
without this patch we observed ~100 IOPs.
With this patches we observed about 20000 IOPs.

Please find charts of the results here:
http://blogs.linbit.com/p/469/843-random-writes-faster/

Lars Ellenberg (18):
drbd: cleanup bogus assert message
drbd: cleanup ondisk meta data layout calculations and defines
drbd: prepare for new striped layout of activity log
drbd: use the cached meta_dev_idx
drbd: mechanically rename la_size to la_size_sect
drbd: read meta data early, base on-disk offsets on super block
drbd: Clarify when activity log I/O is delegated to the worker thread
drbd: drbd_al_being_io: short circuit to reduce latency
drbd: split __drbd_make_request in before and after drbd_al_begin_io
drbd: prepare to queue write requests on a submit worker
drbd: split drbd_al_begin_io into fastpath, prepare, and commit
drbd: split out some helper functions to drbd_al_begin_io
drbd: queue writes on submitter thread, unless they pass the activity
log fastpath
lru_cache: introduce lc_get_cumulative()
drbd: consolidate as many updates as possible into one AL transaction
drbd: move start io accounting before activity log transaction
drbd: try hard to max out the updates per AL transaction
drbd: adjust upper limit for activity log extents

 drivers/block/drbd/drbd_actlog.c   |  246 +++++++++++++++++++++++++++---------
 drivers/block/drbd/drbd_bitmap.c   |   13 +-
 drivers/block/drbd/drbd_int.h      |  179 +++++++++++++-------------
 drivers/block/drbd/drbd_main.c     |  243 +++++++++++++++++++++++++++++------
 drivers/block/drbd/drbd_nl.c       |  129 ++++++++++++-------
 drivers/block/drbd/drbd_receiver.c |    4 +-
 drivers/block/drbd/drbd_req.c      |  166 +++++++++++++++++++++---
 drivers/block/drbd/drbd_worker.c   |    5 +-
 include/linux/drbd_limits.h        |   11 +-
 include/linux/lru_cache.h          |    1 +
 lib/lru_cache.c                    |   55 ++++++--
 11 files changed, 782 insertions(+), 270 deletions(-)
Signature · 1 line
1.7.9.5