fsck.gfs2: memory reduction patches
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v2 Latest40 patches 0 repliesNew
- v130 patches 0 repliesSuperseded
Patches in v2 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/40 | fsck.gfs2: Move pass5 to immediately follow pass1gfs2/ | +1 | 0 replies | |
| 2/40 | fsck.gfs2: Convert block_type to bitmap_type after pass1 and 5gfs2/ gfs2/ gfs2/ +5 more | +28 | 0 replies | |
| 3/40 | fsck.gfs2: Change bitmap_type variables to intgfs2/ gfs2/ gfs2/ +8 more | +38 | 0 replies | |
| 4/40 | fsck.gfs2: Use di_entries to determine if lost+found was createdgfs2/ | +3 | 0 replies | |
| 5/40 | fsck.gfs2: pass1b shouldn't complain about non-bitmap blocksgfs2/ | +1 | 0 replies | |
| 6/40 | fsck.gfs2: Change all fsck_blockmap_set to fsck_bitmap_setgfs2/ gfs2/ gfs2/ +4 more | +122 | 0 replies | |
| 7/40 | fsck.gfs2: Move set_ip_blockmap to pass1gfs2/ gfs2/ gfs2/ | +49 | 0 replies | |
| 8/40 | fsck.gfs2: Remove unneeded parameter instree from set_ip_blockmapgfs2/ | +3 | 0 replies | |
| 9/40 | fsck.gfs2: Move leaf repair to pass2gfs2/ gfs2/ gfs2/ +1 more | +146 | 0 replies | |
| 10/40 | fsck.gfs2: Eliminate astate codegfs2/ gfs2/ gfs2/ +2 more | +0 | 0 replies | |
| 11/40 | fsck.gfs2: Move reprocess code to pass1gfs2/ gfs2/ gfs2/ | +107 | 0 replies | |
| 12/40 | fsck.gfs2: Separate out functions that may only be done after pass1gfs2/ gfs2/ gfs2/ +7 more | +335 | 1 reply | |
| 12/40 | fsck.gfs2: Separate out functions that may only be done after pass1 | +0 | 0 replies | |
| 13/40 | fsck.gfs2: Divest check_metatree from fsck_blockmap_setgfs2/ gfs2/ | +8 | 0 replies | |
| 14/40 | fsck.gfs2: eliminate fsck_blockmap_set from check_eattr_entriesgfs2/ gfs2/ gfs2/ +3 more | +82 | 0 replies | |
| 15/40 | fsck.gfs2: Move blockmap stuff to pass1.cgfs2/ gfs2/ gfs2/ | +21 | 0 replies | |
| 16/40 | fsck: make pass1 call bitmap reconciliation AKA pass5gfs2/ gfs2/ gfs2/ +1 more | +40 | 0 replies | |
| 17/40 | fsck.gfs2: make blockmap global variable only to pass1gfs2/ gfs2/ gfs2/ +4 more | +112 | 0 replies | |
| 18/40 | fsck.gfs2: Add wrapper function pass1_check_metatreegfs2/ | +25 | 0 replies | |
| 19/40 | fsck.gfs2: pass counted_links into fix_link_count in pass4gfs2/ | +8 | 0 replies | |
| 20/40 | fsck.gfs2: refactor pass4 function scan_inode_listgfs2/ | +85 | 0 replies | |
| 21/40 | fsck.gfs2: More refactoring of pass4 function scan_inode_listgfs2/ | +32 | 0 replies | |
| 22/40 | fsck.gfs2: Fix white space problemsgfs2/ | +9 | 0 replies | |
| 23/40 | fsck.gfs2: move link count info for directories to directory treegfs2/ gfs2/ gfs2/ +6 more | +136 | 0 replies | |
| 24/40 | fsck.gfs2: Use bitmaps instead of linked list for inodes w/nlink == 1gfs2/ gfs2/ gfs2/ +6 more | +214 | 0 replies | |
| 25/40 | fsck.gfs2: Refactor check_n_fix_bitmap to make it more readablegfs2/ | +60 | 0 replies | |
| 26/40 | fsck.gfs2: adjust rgrp inode count when fixing bitmapgfs2/ | +43 | 0 replies | |
| 27/40 | fsck.gfs2: blocks cannot be UNLINKED in pass1b or after thatgfs2/ gfs2/ gfs2/ +1 more | +12 | 0 replies | |
| 28/40 | fsck.gfs2: Add error checks to get_next_leafgfs2/ | +11 | 0 replies | |
| 29/40 | fsck.gfs2: re-add a non-allocating repair_leaf to pass1gfs2/ | +47 | 0 replies | |
| 30/40 | libgfs2: Allocate new GFS1 metadata as type 3, not type 1gfs2/ | +15 | 0 replies | |
| 31/40 | fsck.gfs2: Undo partially done metadata recordsgfs2/ gfs2/ gfs2/ +2 more | +85 | 0 replies | |
| 32/40 | fsck.gfs2: Eliminate redundant code in _fsck_bitmap_setgfs2/ | +0 | 0 replies | |
| 33/40 | fsck.gfs2: Fix inode counting buggfs2/ | +27 | 0 replies | |
| 34/40 | fsck.gfs2: Adjust bitmap for lost+found after adding to dirtreegfs2/ | +4 | 0 replies | |
| 35/40 | GFS2: Add initialization checks for GFS1 used metadatagfs2/ | +49 | 0 replies | |
| 36/40 | fsck.gfs2: Use BLKST constants to make pass5 more cleargfs2/ | +19 | 0 replies | |
| 37/40 | fsck.gfs2: Fix GFS1 "used meta" accounting buggfs2/ | +7 | 0 replies | |
| 38/40 | fsck.gfs2: pass1b is too noisy wrt gfs1 non-dinode metadatagfs2/ | +4 | 0 replies | |
| 39/40 | fsck.gfs2: Fix rgrp dinode accounting buggfs2/ | +7 | 0 replies | |
| 40/40 | fsck.gfs2: Fix rgrp accounting in check_n_fix_bitmapgfs2/ | +59 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.
Cover letter [Cluster-devel] [fsck.gfs2 v2 PATCH 00/40] fsck.gfs2: memory reduction patches
On 15 April, I posted a set of 30 patches for fsck.gfs2 to reduce its memory requirements. Since that time, I've been testing and fixing problems. Here is version 2.
---
Right now, fsck.gfs2 is a memory hog with regard to very large file systems. A file system of 250TB would require an absurd about of memory. That's because it uses memory for (1) rgrp bitmaps, (2) its own internal blockmap, (3) a directory tree, and (4) a tree of ALL inodes in the file system, for reconciling link counts.
This series of 40 patches is designed to reduce the memory footprint of fsck.gfs2. It does so by almost entirely eliminating the inode tree in favor of two much smaller bitmaps. In order to do that, I rearranged the order in which things are done.
In order to save on memory, some things will be slower, so this will cause a performance hit to fsck.gfs2. However, I'm confident that we can fix them and make it fast again. The slowness is caused because most bitmap manipulation is now done on the bitmaps in the resource groups, which means the code needs to search out the rgrp for these operations. This can be optimized to pass in the rgrp, since it will be known in most cases (for example, pass1 operates on one rgrp at a time, and chances are quite good that the rgrp needed will be that one.) These further optimizations will be done in later patches.
I'm still testing these changes against my collection of gfs2 and gfs1 metadata, but the results are good so far. So there may be changes.
---
Bob Peterson (40):
fsck.gfs2: Move pass5 to immediately follow pass1 fsck.gfs2: Convert block_type to bitmap_type after pass1 and 5 fsck.gfs2: Change bitmap_type variables to int
fsck.gfs2: Use di_entries to determine if lost+found was created fsck.gfs2: pass1b shouldn't complain about non-bitmap blocks fsck.gfs2: Change all fsck_blockmap_set to fsck_bitmap_set fsck.gfs2: Move set_ip_blockmap to pass1
fsck.gfs2: Remove unneeded parameter instree from set_ip_blockmap fsck.gfs2: Move leaf repair to pass2
fsck.gfs2: Eliminate astate code
fsck.gfs2: Move reprocess code to pass1
fsck.gfs2: Separate out functions that may only be done after pass1 fsck.gfs2: Divest check_metatree from fsck_blockmap_set fsck.gfs2: eliminate fsck_blockmap_set from check_eattr_entries fsck.gfs2: Move blockmap stuff to pass1.c
fsck: make pass1 call bitmap reconciliation AKA pass5 fsck.gfs2: make blockmap global variable only to pass1 fsck.gfs2: Add wrapper function pass1_check_metatree fsck.gfs2: pass counted_links into fix_link_count in pass4 fsck.gfs2: refactor pass4 function scan_inode_list fsck.gfs2: More refactoring of pass4 function scan_inode_list fsck.gfs2: Fix white space problems
fsck.gfs2: move link count info for directories to directory tree fsck.gfs2: Use bitmaps instead of linked list for inodes w/nlink == 1 fsck.gfs2: Refactor check_n_fix_bitmap to make it more readable fsck.gfs2: adjust rgrp inode count when fixing bitmap fsck.gfs2: blocks cannot be UNLINKED in pass1b or after that fsck.gfs2: Add error checks to get_next_leaf
fsck.gfs2: re-add a non-allocating repair_leaf to pass1
libgfs2: Allocate new GFS1 metadata as type 3, not type 1 fsck.gfs2: Undo partially done metadata records
fsck.gfs2: Eliminate redundant code in _fsck_bitmap_set fsck.gfs2: Fix inode counting bug
fsck.gfs2: Adjust bitmap for lost+found after adding to dirtree
GFS2: Add initialization checks for GFS1 used metadata fsck.gfs2: Use BLKST constants to make pass5 more clear fsck.gfs2: Fix GFS1 "used meta" accounting bug
fsck.gfs2: pass1b is too noisy wrt gfs1 non-dinode metadata fsck.gfs2: Fix rgrp dinode accounting bug
fsck.gfs2: Fix rgrp accounting in check_n_fix_bitmap
gfs2/fsck/Makefile.am | 1 + gfs2/fsck/afterpass1_common.c | 319 ++++++++++++++++ gfs2/fsck/afterpass1_common.h | 31 ++ gfs2/fsck/fsck.h | 26 +- gfs2/fsck/initialize.c | 63 +++- gfs2/fsck/inode_hash.c | 3 +- gfs2/fsck/link.c | 139 ++++++- gfs2/fsck/link.h | 6 +- gfs2/fsck/lost_n_found.c | 17 +- gfs2/fsck/main.c | 38 +- gfs2/fsck/metawalk.c | 847 ++++++++---------------------------------- gfs2/fsck/metawalk.h | 59 +-- gfs2/fsck/pass1.c | 479 ++++++++++++++++++++++-- gfs2/fsck/pass1b.c | 148 +++++--- gfs2/fsck/pass2.c | 276 ++++++++++---- gfs2/fsck/pass3.c | 52 +-- gfs2/fsck/pass4.c | 339 ++++++++++------- gfs2/fsck/pass5.c | 60 +-- gfs2/fsck/util.c | 171 +++------ gfs2/fsck/util.h | 70 ++-- gfs2/libgfs2/fs_ops.c | 33 +- 21 files changed, 1849 insertions(+), 1328 deletions(-) create mode 100644 gfs2/fsck/afterpass1_common.c create mode 100644 gfs2/fsck/afterpass1_common.h
Signature · 1 line
2.5.5