Skip to content

RFC Introduce parallel fsck to e2fsck pass1

Saranya Muruganandam linux-ext4 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v3 RFC Latest60 of 61 patches 23 repliesIn review

Changes in v3 from the cover letter

1) Parallelize rw_bitmaps
2) Configuration to turn off parallel fsck.(default=on)
3) Add f_multithread_ok
4) Add annotations to e2fsck_struct instead of reshuffle
5) Fix memory leaks
6) Some more extra fixes

60 of 61 patches are in this archive. The cover letter announces 61.

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/61e2fsck: add -m option for multithreade2fsck/e2fsck.h e2fsck/unix.c tests/f_multithread/expect.1 +24 more+217 -62 replies
2/61e2fsck: copy context when using multi-thread fscke2fsck/pass1.c e2fsck/unix.c+107 -85 replies
3/61e2fsck: copy fs when using multi-thread fscke2fsck/pass1.c+37 -41 reply
4/61e2fsck: clear icache when using multi-thread fscke2fsck/pass1.c+34 -31 reply
5/61e2fsck: add assert when copying contexte2fsck/pass1.c+14 -00 replies
6/61e2fsck: copy bitmaps when copying contexte2fsck/pass1.c+139 -120 replies
7/61e2fsck: copy badblocks when copying fse2fsck/pass1.c+13 -00 replies
8/61e2fsck: open io-channel when copying fse2fsck/e2fsck.h e2fsck/pass1.c e2fsck/unix.c +5 more+125 -212 replies
9/61e2fsck: create logs for mult-threadse2fsck/e2fsck.h e2fsck/logfile.c e2fsck/pass1.c +4 more+88 -81 reply
10/61e2fsck: optionally configure one pfsck threadconfigure configure.ac e2fsck/e2fsck.h +5 more+256 -331 reply
11/61e2fsck: add start/end group for threade2fsck/e2fsck.h e2fsck/logfile.c e2fsck/pass1.c +3 more+78 -100 replies
12/61e2fsck: split groups to different threadse2fsck/pass1.c+17 -60 replies
13/61e2fsck: print thread log properlye2fsck/e2fsck.h e2fsck/pass1.c e2fsck/problem.c +4 more+98 -71 reply
14/61e2fsck: merge bitmaps after thread completese2fsck/pass1.c lib/ext2fs/bitmaps.c lib/ext2fs/blkmap64_rb.c +3 more+264 -911 reply
15/61e2fsck: do not change global variablese2fsck/pass1.c+47 -280 replies
16/61e2fsck: optimize the inserting of dir_info_dbe2fsck/dirinfo.c+112 -600 replies
17/61e2fsck: merge dir_info after thread finishese2fsck/dirinfo.c e2fsck/e2fsck.h e2fsck/pass1.c+123 -30 replies
18/61e2fsck: rbtree bitmap for dire2fsck/pass1.c+1 -00 replies
19/61e2fsck: merge badblocks after thread finishese2fsck/pass1.c lib/ext2fs/badblocks.c lib/ext2fs/ext2fs.h +1 more+94 -130 replies
20/61e2fsck: merge icounts after thread finishese2fsck/pass1.c lib/ext2fs/ext2fs.h lib/ext2fs/icount.c+151 -00 replies
21/61e2fsck: merge dblist after thread finishese2fsck/pass1.c lib/ext2fs/dblist.c lib/ext2fs/ext2fs.h+56 -00 replies
22/61e2fsck: add debug codes for multiple threadse2fsck/e2fsck.h e2fsck/pass1.c+45 -00 replies
23/61e2fsck: merge counts after threads finishe2fsck/pass1.c+34 -00 replies
24/61e2fsck: merge fs flags when threads finishe2fsck/pass1.c+5 -00 replies
25/61e2fsck: merge dx_dir_info after threads finishe2fsck/dx_dirinfo.c e2fsck/e2fsck.h e2fsck/pass1.c+88 -00 replies
26/61e2fsck: merge dirs_to_hash when threads finishe2fsck/pass1.c+30 -00 replies
27/61e2fsck: merge context flags properlye2fsck/pass1.c+1 -30 replies
28/61e2fsck: merge quota context after threads finishe2fsck/pass1.c lib/support/mkquota.c lib/support/quotaio.h+56 -10 replies
29/61e2fsck: serialize fix operationse2fsck/e2fsck.h e2fsck/pass1.c e2fsck/util.c+100 -70 replies
30/61e2fsck: move some fixes out of parallel pthreadse2fsck/e2fsck.h e2fsck/pass1.c e2fsck/util.c +3 more+216 -1360 replies
31/61e2fsck: split and merge invalid bitmapse2fsck/pass1.c+71 -02 replies
32/61e2fsck: merge EA blocks properlye2fsck/e2fsck.h e2fsck/pass1.c+213 -330 replies
33/61e2fsck: kickoff mutex lock for block found mape2fsck/e2fsck.h e2fsck/pass1.c e2fsck/util.c +3 more+172 -810 replies
34/61e2fsck: allow admin specify number of threadse2fsck/e2fsck.h e2fsck/pass1.c e2fsck/unix.c +6 more+128 -680 replies
35/61e2fsck: adjust number of threadse2fsck/pass1.c lib/ext2fs/ext2fs.h+46 -180 replies
36/61e2fsck: fix readahead for pfsck of pass1e2fsck/pass1.c+19 -90 replies
37/61e2fsck: merge options after threads finishe2fsck/pass1.c+4 -51 reply
38/61e2fsck: reset lost_and_found after threads finishe2fsck/pass1.c+5 -00 replies
39/61e2fsck: merge extent depth count after threads finishe2fsck/pass1.c+10 -10 replies
40/61e2fsck: simplify e2fsck context merging codese2fsck/pass1.c+31 -1200 replies
41/61e2fsck: set E2F_FLAG_ALLOC_OK after threadse2fsck/pass1.c+8 -20 replies
42/61e2fsck: wait fix thread finish before checkinge2fsck/e2fsck.c e2fsck/e2fsck.h e2fsck/pass1.c +1 more+117 -120 replies
43/61e2fsck: cleanup e2fsck_pass1_thread_join()e2fsck/pass1.c+8 -260 replies
44/61e2fsck: avoid too much memory allocation for pfscklib/ext2fs/dblist.c lib/ext2fs/icount.c+6 -00 replies
45/61e2fsck: make default smallest RA size to 1Me2fsck/readahead.c+4 -00 replies
46/61ext2fs: parallel bitmap loadinglib/ext2fs/rw_bitmaps.c+250 -391 reply
47/61e2fsck: update mmp block in one threade2fsck/e2fsck.h e2fsck/pass1.c+35 -20 replies
48/61e2fsck: reset @inodes_to_rebuild if restarte2fsck/e2fsck.c+4 -00 replies
49/61e2fsck: fix build for make rpmMCONFIG.in e2fsck/Makefile.in+6 -40 replies
50/61e2fsck: move ext2fs_get_avg_group to rw_bitmaps.clib/ext2fs/ext2fs.h lib/ext2fs/rw_bitmaps.c+26 -281 reply
51/61configure: enable pfsck by defaultconfigure configure.ac+21 -50 replies
53/61e2fsck: fix f_multithread_ok testtests/f_multithread_ok/script+4 -40 replies
54/61e2fsck: fix race in ext2fs_read_bitmaps()e2fsck/pass1.c lib/ext2fs/rw_bitmaps.c+9 -180 replies
55/61e2fsck: fix readahead for pass1 without pfscke2fsck/pass1.c+2 -10 replies
56/61e2fsck: fix memory leaks with pfsck enablede2fsck/e2fsck.c e2fsck/pass1.c lib/ext2fs/rw_bitmaps.c+9 -100 replies
57/61ext2fs: fix to set tail flags with pfsck enabledlib/ext2fs/rw_bitmaps.c+19 -270 replies
58/61e2fsck: misc cleanups for pfscke2fsck/e2fsck.8.in e2fsck/e2fsck.h e2fsck/pass1.c +2 more+31 -280 replies
59/61e2fsck: update mmp block racee2fsck/pass1.c+9 -110 replies
60/61e2fsck: propagate number of threadse2fsck/unix.c+3 -01 reply
61/61e2fsck: Annotating fields in e2fsck_structe2fsck/e2fsck.h+14 -60 replies

Reviews trailers per patch

2 of 60 patches have a Reviewed-by or Acked-by.

PatchjenkinsMalooAndreas DilgerHarshad Shirwadkar
1/61R-b
2/61
3/61
4/61
5/61
6/61
7/61
8/61
9/61
10/61
11/61
12/61
13/61
14/61
15/61
16/61
17/61
18/61
19/61
20/61
21/61
22/61
23/61
24/61
25/61
26/61
27/61
28/61
29/61
30/61
31/61
32/61
33/61
34/61
35/61
36/61
37/61
38/61
39/61
40/61
41/61
42/61
43/61
44/61
45/61
46/61
47/61
48/61
49/61
50/61T-b T-b R-b
51/61
53/61
54/61
55/61
56/61
57/61
58/61
59/61
60/61
61/61

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.

Read the v3 thread

Cover letter [RFC PATCH v3 00/61] Introduce parallel fsck to e2fsck pass1

Currently it has been popular that single disk could be more than TiB, etc 16Tib with only one single disk, with this trend, one single filesystem could be larger and larger and easily reach PiB with LUN system.

The journal filesystem like ext4 need be offline to do regular check and repair from time to time, however the problem is e2fsck still do this using single thread, this could be challenging at scale for two reasons:

1) even with readahead, IO speed still limits several tens MiB per second.
2) could not utilize CPU cores.

It could be challenging to try multh-threads for all phase of e2fsck, but as first step, we might try this for most time-consuming pass1, according to our benchmarking it cost of 80% time for whole e2fck phase.

Pass1 is trying to scanning all valid inode of filesystem and check it one by one, and the patchset idea is trying to split these to different threads and trying to do this at the same time, we try to merge these inodes and corresponding inode's extent information after threads finish.

To simplify complexity and make it less error-prone, the fix is still serialized, since most of time there will be only minor errors for filesystem, what's important for us is parallel reading and checking.

Here is a benchmarking on our Lustre filesystem with 1.2 PiB OSD ext4 based filesystem:

DDN SFA18KE StorageServer
DCR(DeClustering RAID) with 162 x HGST 10TB NL-SAS Tested Server
A Virtual Machine running on SFA18KE
8 x CPU cores (Xeon(R) Gold 6140)
150GB memory
CentoOS7.7 (Lustre patched kernel)

Created 600 Million x 32K byte files.

Without Patch With Patch thr=64
pass1: 13079.66 488.57 seconds
Total: 15673.33 3188.42

We have 5x total time reduction of total time which is very inspiring.

I've tested the whole patch series using 'make test' of e2fsck itself, and i manually set default threads to 4 which still pass almost of test suite, failure cases are below:

f_h_badroot f_multithread f_multithread_logfile f_multithread_no f_multithread_ok

h_h_badroot failed because out of order checking output, and others are because of extra multiple threads log output.

The other test i've done to verify is:
1) filled a filesystem with different features enabled(encryption, shared xattr)
2) copied the above image out using e2image.
3) using e2fuzz to randomly corrupt above image, and then have another copy of corrupted image.
4) running e2fsck with single thread(without patches) and multiple threads with corrupted image respectively.
5) using valgrind to check any memory leak.
6) compare single thread and multiple threads fix results by mounting two images back and compare the directores and files.(size and name match)

Any review or comments are welcomed!

Thanks you very much!

changelog v2->v3:
1) Parallelize rw_bitmaps
2) Configuration to turn off parallel fsck.(default=on)
3) Add f_multithread_ok
4) Add annotations to e2fsck_struct instead of reshuffle
5) Fix memory leaks
6) Some more extra fixes

Andreas Dilger (2):
e2fsck: fix f_multithread_ok test
e2fsck: misc cleanups for pfsck

Li Xi (18):
e2fsck: add -m option for multithread
e2fsck: copy context when using multi-thread fsck
e2fsck: copy fs when using multi-thread fsck
e2fsck: add assert when copying context
e2fsck: copy bitmaps when copying context
e2fsck: open io-channel when copying fs
e2fsck: create logs for mult-threads
e2fsck: optionally configure one pfsck thread
e2fsck: add start/end group for thread
e2fsck: split groups to different threads
e2fsck: print thread log properly
e2fsck: do not change global variables
e2fsck: optimize the inserting of dir_info_db
e2fsck: merge dir_info after thread finishes
e2fsck: merge icounts after thread finishes
e2fsck: merge dblist after thread finishes
e2fsck: add debug codes for multiple threads
e2fsck: merge fs flags when threads finish

Saranya Muruganandam (2):
e2fsck: propagate number of threads
e2fsck: Annotating fields in e2fsck_struct

Wang Shilong (39):
e2fsck: clear icache when using multi-thread fsck
e2fsck: copy badblocks when copying fs
e2fsck: merge bitmaps after thread completes
e2fsck: rbtree bitmap for dir
e2fsck: merge badblocks after thread finishes
e2fsck: merge counts after threads finish
e2fsck: merge dx_dir_info after threads finish
e2fsck: merge dirs_to_hash when threads finish
e2fsck: merge context flags properly
e2fsck: merge quota context after threads finish
e2fsck: serialize fix operations
e2fsck: move some fixes out of parallel pthreads
e2fsck: split and merge invalid bitmaps
e2fsck: merge EA blocks properly
e2fsck: kickoff mutex lock for block found map
e2fsck: allow admin specify number of threads
e2fsck: adjust number of threads
e2fsck: fix readahead for pfsck of pass1
e2fsck: merge options after threads finish
e2fsck: reset lost_and_found after threads finish
e2fsck: merge extent depth count after threads finish
e2fsck: simplify e2fsck context merging codes
e2fsck: set E2F_FLAG_ALLOC_OK after threads
e2fsck: wait fix thread finish before checking
e2fsck: cleanup e2fsck_pass1_thread_join()
e2fsck: avoid too much memory allocation for pfsck
e2fsck: make default smallest RA size to 1M
ext2fs: parallel bitmap loading
e2fsck: update mmp block in one thread
e2fsck: reset @inodes_to_rebuild if restart
e2fsck: fix build for make rpm
e2fsck: move ext2fs_get_avg_group to rw_bitmaps.c
configure: enable pfsck by default
test: add pfsck test
e2fsck: fix race in ext2fs_read_bitmaps()
e2fsck: fix readahead for pass1 without pfsck
e2fsck: fix memory leaks with pfsck enabled
ext2fs: fix to set tail flags with pfsck enabled
e2fsck: update mmp block race

 MCONFIG.in                              |    1 +
 configure                               |   90 +-
 configure.ac                            |   26 +
 e2fsck/Makefile.in                      |    9 +-
 e2fsck/dirinfo.c                        |  238 ++-
 e2fsck/dx_dirinfo.c                     |   64 +
 e2fsck/e2fsck.8.in                      |    8 +-
 e2fsck/e2fsck.c                         |   11 +
 e2fsck/e2fsck.h                         |  102 +-
 e2fsck/logfile.c                        |   13 +-
 e2fsck/pass1.c                          | 1766 ++++++++++++++++++++---
 e2fsck/problem.c                        |   11 +
 e2fsck/problem.h                        |    3 +
 e2fsck/readahead.c                      |    4 +
 e2fsck/unix.c                           |   59 +-
 e2fsck/util.c                           |  193 ++-
 lib/config.h.in                         |    3 +
 lib/ext2fs/badblocks.c                  |   85 +-
 lib/ext2fs/bitmaps.c                    |   10 +
 lib/ext2fs/bitops.h                     |    2 +
 lib/ext2fs/blkmap64_rb.c                |   65 +
 lib/ext2fs/bmap64.h                     |    4 +
 lib/ext2fs/dblist.c                     |   38 +
 lib/ext2fs/ext2_err.et.in               |    3 +
 lib/ext2fs/ext2_io.h                    |    2 +
 lib/ext2fs/ext2fs.h                     |   19 +-
 lib/ext2fs/ext2fsP.h                    |    1 -
 lib/ext2fs/gen_bitmap64.c               |   62 +
 lib/ext2fs/icount.c                     |  107 ++
 lib/ext2fs/openfs.c                     |   48 +-
 lib/ext2fs/rw_bitmaps.c                 |  301 +++-
 lib/ext2fs/undo_io.c                    |   19 +
 lib/ext2fs/unix_io.c                    |   24 +-
 lib/support/mkquota.c                   |   39 +
 lib/support/quotaio.h                   |    3 +
 tests/f_itable_collision/expect.1       |    3 -
 tests/f_multithread/expect.1            |   25 +
 tests/f_multithread/expect.2            |    7 +
 tests/f_multithread/image.gz            |    1 +
 tests/f_multithread/name                |    1 +
 tests/f_multithread/script              |    4 +
 tests/f_multithread_completion/expect.1 |    2 +
 tests/f_multithread_completion/expect.2 |   23 +
 tests/f_multithread_completion/image.gz |    1 +
 tests/f_multithread_completion/name     |    1 +
 tests/f_multithread_completion/script   |    4 +
 tests/f_multithread_logfile/expect.1    |   25 +
 tests/f_multithread_logfile/image.gz    |    1 +
 tests/f_multithread_logfile/name        |    1 +
 tests/f_multithread_logfile/script      |   32 +
 tests/f_multithread_no/expect.1         |   26 +
 tests/f_multithread_no/expect.2         |   23 +
 tests/f_multithread_no/image.gz         |    1 +
 tests/f_multithread_no/name             |    1 +
 tests/f_multithread_no/script           |    4 +
 tests/f_multithread_ok/expect.1         |    8 +
 tests/f_multithread_ok/image.gz         |  Bin 0 -> 796311 bytes
 tests/f_multithread_ok/name             |    1 +
 tests/f_multithread_ok/script           |   21 +
 tests/f_multithread_preen/expect.1      |   11 +
 tests/f_multithread_preen/expect.2      |   23 +
 tests/f_multithread_preen/image.gz      |    1 +
 tests/f_multithread_preen/name          |    1 +
 tests/f_multithread_preen/script        |    4 +
 tests/f_multithread_yes/expect.1        |    2 +
 tests/f_multithread_yes/expect.2        |   23 +
 tests/f_multithread_yes/image.gz        |    1 +
 tests/f_multithread_yes/name            |    1 +
 tests/f_multithread_yes/script          |    4 +
 tests/test_one.in                       |    8 +
 70 files changed, 3335 insertions(+), 393 deletions(-)
 create mode 100644 tests/f_multithread/expect.1
 create mode 100644 tests/f_multithread/expect.2
 create mode 120000 tests/f_multithread/image.gz
 create mode 100644 tests/f_multithread/name
 create mode 100644 tests/f_multithread/script
 create mode 100644 tests/f_multithread_completion/expect.1
 create mode 100644 tests/f_multithread_completion/expect.2
 create mode 120000 tests/f_multithread_completion/image.gz
 create mode 100644 tests/f_multithread_completion/name
 create mode 100644 tests/f_multithread_completion/script
 create mode 100644 tests/f_multithread_logfile/expect.1
 create mode 120000 tests/f_multithread_logfile/image.gz
 create mode 100644 tests/f_multithread_logfile/name
 create mode 100644 tests/f_multithread_logfile/script
 create mode 100644 tests/f_multithread_no/expect.1
 create mode 100644 tests/f_multithread_no/expect.2
 create mode 120000 tests/f_multithread_no/image.gz
 create mode 100644 tests/f_multithread_no/name
 create mode 100644 tests/f_multithread_no/script
 create mode 100644 tests/f_multithread_ok/expect.1
 create mode 100644 tests/f_multithread_ok/image.gz
 create mode 100644 tests/f_multithread_ok/name
 create mode 100644 tests/f_multithread_ok/script
 create mode 100644 tests/f_multithread_preen/expect.1
 create mode 100644 tests/f_multithread_preen/expect.2
 create mode 120000 tests/f_multithread_preen/image.gz
 create mode 100644 tests/f_multithread_preen/name
 create mode 100644 tests/f_multithread_preen/script
 create mode 100644 tests/f_multithread_yes/expect.1
 create mode 100644 tests/f_multithread_yes/expect.2
 create mode 120000 tests/f_multithread_yes/image.gz
 create mode 100644 tests/f_multithread_yes/name
 create mode 100644 tests/f_multithread_yes/script
Signature · 1 line
2.29.2.299.gdc1121823c-goog