fsck.gfs2: memory reduction patches

On 15 April, I posted a set of 30 patches for fsck.gfs2 to reduce its memory requirements. Since that time, I've been testing and fixing problems. Here is version 2.
---
Right now, fsck.gfs2 is a memory hog with regard to very large file systems. A file system of 250TB would require an absurd about of memory. That's because it uses memory for (1) rgrp bitmaps, (2) its own internal blockmap, (3) a directory tree, and (4) a tree of ALL inodes in the file system, for reconciling link counts.

This series of 40 patches is designed to reduce the memory footprint of fsck.gfs2. It does so by almost entirely eliminating the inode tree in favor of two much smaller bitmaps. In order to do that, I rearranged the order in which things are done.

In order to save on memory, some things will be slower, so this will cause a performance hit to fsck.gfs2. However, I'm confident that we can fix them and make it fast again. The slowness is caused because most bitmap manipulation is now done on the bitmaps in the resource groups, which means the code needs to search out the rgrp for these operations. This can be optimized to pass in the rgrp, since it will be known in most cases (for example, pass1 operates on one rgrp at a time, and chances are quite good that the rgrp needed will be that one.) These further optimizations will be done in later patches.

I'm still testing these changes against my collection of gfs2 and gfs1 metadata, but the results are good so far. So there may be changes.

Signed-off-by Bob Peterson <[email protected]>

---
Bob Peterson (40):
fsck.gfs2: Move pass5 to immediately follow pass1 fsck.gfs2: Convert block_type to bitmap_type after pass1 and 5 fsck.gfs2: Change bitmap_type variables to int
fsck.gfs2: Use di_entries to determine if lost+found was created fsck.gfs2: pass1b shouldn't complain about non-bitmap blocks fsck.gfs2: Change all fsck_blockmap_set to fsck_bitmap_set fsck.gfs2: Move set_ip_blockmap to pass1
fsck.gfs2: Remove unneeded parameter instree from set_ip_blockmap fsck.gfs2: Move leaf repair to pass2
fsck.gfs2: Eliminate astate code
fsck.gfs2: Move reprocess code to pass1
fsck.gfs2: Separate out functions that may only be done after pass1 fsck.gfs2: Divest check_metatree from fsck_blockmap_set fsck.gfs2: eliminate fsck_blockmap_set from check_eattr_entries fsck.gfs2: Move blockmap stuff to pass1.c
fsck: make pass1 call bitmap reconciliation AKA pass5 fsck.gfs2: make blockmap global variable only to pass1 fsck.gfs2: Add wrapper function pass1_check_metatree fsck.gfs2: pass counted_links into fix_link_count in pass4 fsck.gfs2: refactor pass4 function scan_inode_list fsck.gfs2: More refactoring of pass4 function scan_inode_list fsck.gfs2: Fix white space problems
fsck.gfs2: move link count info for directories to directory tree fsck.gfs2: Use bitmaps instead of linked list for inodes w/nlink == 1 fsck.gfs2: Refactor check_n_fix_bitmap to make it more readable fsck.gfs2: adjust rgrp inode count when fixing bitmap fsck.gfs2: blocks cannot be UNLINKED in pass1b or after that

fsck.gfs2: Add error checks to get_next_leaf fsck.gfs2: re-add a non-allocating repair_leaf to pass1
libgfs2: Allocate new GFS1 metadata as type 3, not type 1 fsck.gfs2: Undo partially done metadata records fsck.gfs2: Eliminate redundant code in _fsck_bitmap_set fsck.gfs2: Fix inode counting bug
fsck.gfs2: Adjust bitmap for lost+found after adding to dirtree
GFS2: Add initialization checks for GFS1 used metadata

fsck.gfs2: Use BLKST constants to make pass5 more clear fsck.gfs2: Fix GFS1 "used meta" accounting bug fsck.gfs2: pass1b is too noisy wrt gfs1 non-dinode metadata fsck.gfs2: Fix rgrp dinode accounting bug fsck.gfs2: Fix rgrp accounting in check_n_fix_bitmap

  gfs2/fsck/Makefile.am         |   1 +
  gfs2/fsck/afterpass1_common.c | 319 ++++++++++++++++
  gfs2/fsck/afterpass1_common.h |  31 ++
  gfs2/fsck/fsck.h              |  26 +-
  gfs2/fsck/initialize.c        |  63 +++-
  gfs2/fsck/inode_hash.c        |   3 +-
  gfs2/fsck/link.c              | 139 ++++++-
  gfs2/fsck/link.h              |   6 +-
  gfs2/fsck/lost_n_found.c      |  17 +-
  gfs2/fsck/main.c              |  38 +-
  gfs2/fsck/metawalk.c          | 847 ++++++++----------------------------------
  gfs2/fsck/metawalk.h          |  59 +--
  gfs2/fsck/pass1.c             | 479 ++++++++++++++++++++++--
  gfs2/fsck/pass1b.c            | 148 +++++---
  gfs2/fsck/pass2.c             | 276 ++++++++++----
  gfs2/fsck/pass3.c             |  52 +--
  gfs2/fsck/pass4.c             | 339 ++++++++++-------
  gfs2/fsck/pass5.c             |  60 +--
  gfs2/fsck/util.c              | 171 +++------
  gfs2/fsck/util.h              |  70 ++--
  gfs2/libgfs2/fs_ops.c         |  33 +-
  21 files changed, 1849 insertions(+), 1328 deletions(-)

create mode 100644 gfs2/fsck/afterpass1_common.c create mode 100644 gfs2/fsck/afterpass1_common.h