Skip to content

Fix some scaling problems for large VGs (many LVs)

Milan Broz lvm-devel 1 version New

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest5 patches 0 repliesNew

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/5Use hash table for quick lv reference when reading metadata.lib/format_text/import_vsn1.c+31 -110 replies
2/5Remove vg_validate call when parsing cached metadata.lib/cache/lvmcache.c+1 -21 reply
3/5Optimise PV segments search.lib/metadata/metadata.c lib/metadata/metadata.h lib/metadata/pv_manip.c+14 -150 replies
4/5Do not traverse PV segment list twice.lib/metadata/pv_alloc.h lib/metadata/pv_manip.c+25 -170 replies
5/5Fix all segments memory is allocated from vg private mempool.lib/format_text/import.c lib/metadata/lv_manip.c lib/metadata/merge.c +2 more+18 -140 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 0/5] Fix some scaling problems for large VGs (many LVs)

This patches fixes some problems in code when processing large VGs.

An example: VG with one PVs and 2000LVs
(I was not able to test more LVs without patches, swapped to death...)

# vgs vg_test
  VG      #PV #LV  #SN Attr   VSize  VFree
  vg_test   1 2000   0 wz--n- 18.53g 10.71g

Before (intentionally do not run any in-kernel operations -t test mode):

# time vgchange -t -a n vg_test
  Test mode: Metadata will NOT be updated.
    0 logical volume(s) in volume group "vg_test" now active
    real    9m31.463s
    user    9m29.841s
    sys     0m1.251s

After (patches applied):

# time vgchange -t -a n vg_test
  Test mode: Metadata will NOT be updated.
   0 logical volume(s) in volume group "vg_test" now active
    real    1m22.852s
    user    1m14.805s
    sys     0m7.961s

Memory use (cca, it oscillates according to vg_release) before:
root 24664 99.9 17.9 355588 350340 pts/0 R+ 14:43 9:34 vgchange -t -a n vg_test
after:
root 31022 98.5 0.2 7736 5084 pts/0 R+ 14:57 1:21 vgchange -t -a n vg_test

...

Milan Broz (5):
Use hash table for quick lv reference when reading metadata. Remove vg_validate call when parsing cached metadata. Optimise PV segments search.
Do not traverse PV segment list twice.
Fix all segments memory is allocated from vg private mempool.

 lib/cache/lvmcache.c          |    3 +-
 lib/format_text/import.c      |    2 +-
 lib/format_text/import_vsn1.c |   42 +++++++++++++++++++++-------
 lib/metadata/lv_manip.c       |    2 +-
 lib/metadata/merge.c          |    4 +-
 lib/metadata/metadata.c       |   12 --------
 lib/metadata/metadata.h       |    3 --
 lib/metadata/pv_alloc.h       |    4 ++-
 lib/metadata/pv_manip.c       |   60 ++++++++++++++++++++++++++++------------
 9 files changed, 81 insertions(+), 51 deletions(-)