e2fsprogs: Parallel fsck support

f_crashdisk test failed with UNIX_IO_FORCE_BOUNCE=yes due to unbalanced mutex unlock in below path.

This patch fixes it.

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

Quoted 6 lines · Ritesh Harjani (IBM)

f_crashdisk test failed with UNIX_IO_FORCE_BOUNCE=yes due to unbalanced mutex unlock in below path.

This patch fixes it.

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

This has been fixed in upstream (for a while, actually);

what commit is your patches based upon?

f_crashdisk test failed with UNIX_IO_FORCE_BOUNCE=yes due to unbalanced mutex unlock in below path.

This patch fixes it.

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
lib/ext2fs/unix_io.c | 1 -

1 file changed, 1 deletion(-)

lib/ext2fs/unix_io.c +0 -0

index e53db333..5b894826 100644

lib/ext2fs/unix_io.c +0 -0

           actual = really_read;
           buf -= really_read;
           size += really_read;

Signature · 1 line

2.37.3

Quoted 11 lines · Ritesh Harjani (IBM)

f_crashdisk test failed with UNIX_IO_FORCE_BOUNCE=yes due to unbalanced mutex unlock in below path.

This patch fixes it.

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
lib/ext2fs/unix_io.c | 1 -

1 file changed, 1 deletion(-)

lib/ext2fs/unix_io.c bounce_read: · line 308

Quoted 9 lines

lib/ext2fs/unix_io.c +0 -1

index e53db333..5b894826 100644
@@ -305,7 +305,6 @@ bounce_read:
   while (size > 0) {
       actual = read(data->dev, data->bounce, align_size);
       if (actual != align_size) {
-            mutex_unlock(data, BOUNCE_MTX);

This patch doesn't show enough context, but AFAIK this is jumping before mutex_down() is called, so this *should* be correct as is?

Quoted 12 lines · Andreas Dilger

Quoted 12 lines · Ritesh Harjani (IBM)

f_crashdisk test failed with UNIX_IO_FORCE_BOUNCE=yes due to unbalanced mutex unlock in below path.

This patch fixes it.

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
lib/ext2fs/unix_io.c | 1 -

1 file changed, 1 deletion(-)

Quoted 12 lines

lib/ext2fs/unix_io.c bounce_read: · line 308

Quoted 9 lines

lib/ext2fs/unix_io.c +0 -1

index e53db333..5b894826 100644
@@ -305,7 +305,6 @@ bounce_read:
  while (size > 0) {
      actual = read(data->dev, data->bounce, align_size);
      if (actual != align_size) {
-            mutex_unlock(data, BOUNCE_MTX);

This patch doesn't show enough context, but AFAIK this is jumping before mutex_down() is called, so this *should* be correct as is?

Thanks for the review, Andreas.

Yeah, the patch diff above is not sufficient since it doesn't share enuf context.
But essentially when "actual" is not equal to "align_size", then in this if condition it goes to label "short_read:", which always goto error_unlock, where we anyways call mutex_unlock()

Looking at a lot of labels in this function, this definitely looks like something which can be cleaned up ("raw_read_blk()").
I will add that to my list of todos.

Quoted 35 lines · Ritesh Harjani (IBM)

Quoted 12 lines · Andreas Dilger

Quoted 12 lines · Ritesh Harjani (IBM)

f_crashdisk test failed with UNIX_IO_FORCE_BOUNCE=yes due to unbalanced mutex unlock in below path.

This patch fixes it.

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
lib/ext2fs/unix_io.c | 1 -

1 file changed, 1 deletion(-)

Quoted 12 lines

lib/ext2fs/unix_io.c bounce_read: · line 308

Quoted 9 lines

lib/ext2fs/unix_io.c +0 -1

index e53db333..5b894826 100644
@@ -305,7 +305,6 @@ bounce_read:
  while (size > 0) {
      actual = read(data->dev, data->bounce, align_size);
      if (actual != align_size) {
-            mutex_unlock(data, BOUNCE_MTX);

This patch doesn't show enough context, but AFAIK this is jumping before mutex_down() is called, so this *should* be correct as is?

Thanks for the review, Andreas.

Yeah, the patch diff above is not sufficient since it doesn't share enuf context.
But essentially when "actual" is not equal to "align_size", then in this if condition it goes to label "short_read:", which always goto error_unlock, where we anyways call mutex_unlock()

Quoted 3 lines

Looking at a lot of labels in this function, this definitely looks like something which can be cleaned up ("raw_read_blk()").
I will add that to my list of todos.

You are correct, and it means this code is just not very clear to the reader. I think it would make more sense to move the "short_read:" label to the end of the code:

                  actual = read(...);
                  if (actual != size)
                          goto error_short_read;
                  goto success_unlock;
        :
                 actual = read(...);
                 if (actual != align_size) {
                           actual = really_read;
                           buf -= really_read;
                           size += really_read;
                           goto error_short_read;
                 }
        :
success_unlock:
        mutex_unlock(...);
        return 0;

error_short_read:

        if (actual < 0) {
                 retval = errno;
                 actual = 0;
        } else {
                 retval = EXT2_ET_SHORT_READ;
        }
error_unlock:
        mutex_unlock(...);

That way the code follows the normal error handling convention and is less likely to be surprising to the reader.

f_crashdisk test failed with UNIX_IO_FORCE_BOUNCE=yes due to unbalanced mutex unlock in below path.

This patch fixes it.

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

Quoted 9 lines · Ritesh Harjani (IBM)

Currently this function was not correctly comparing against the right length of the bitmap. Also when we compare bitarray v/s rbtree bitmap the value returned by ext2fs_test_generic_bmap() could be different in these two implementations. Hence only check against boolean value.

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
lib/ext2fs/gen_bitmap.c | 9 ++++++---

lib/ext2fs/gen_bitmap64.c errcode_t ext2fs_compare_generic_bmap(errcode_t neq, · line 629

Quoted 6 lines

lib/ext2fs/gen_bitmap64.c +0 -0

index c860c10e..f7710afd 100644
@@ -629,10 +629,14 @@ errcode_t ext2fs_compare_generic_bmap(errcode_t neq,
 	    (bm1->end != bm2->end))

Conversely, *this* version of the function is *not* doing the memcmp() of the bulk of the bitmap contents, so it would appear to have a bug that the patch fixes, but in a very slow manner. It would be better to use memcmp().

Just a quick common helper for bits size calculation.

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
lib/ext2fs/blkmap64_ba.c | 20 +++++++++++++-------

1 file changed, 13 insertions(+), 7 deletions(-)

lib/ext2fs/blkmap64_ba.c +13 -7

index 5d8f1548..4e7007f0 100644
@@ -40,6 +40,13 @@ struct ext2fs_ba_private_struct {
 
 typedef struct ext2fs_ba_private_struct *ext2fs_ba_private;
 
+#define ba_bits_size(start, end) ((((end) - (start)) / 8 + 1))
+
+static size_t ba_bitmap_size(ext2fs_generic_bitmap_64 bitmap)
+{
+    return (size_t) ba_bits_size(bitmap->start, bitmap->real_end);
+}
+
 static errcode_t ba_alloc_private_data (ext2fs_generic_bitmap_64 bitmap)
 {
  ext2fs_ba_private bp;

@@ -56,7 +63,7 @@ static errcode_t ba_alloc_private_data (ext2fs_generic_bitmap_64 bitmap)
  if (retval)
      return retval;
 
-    size = (size_t) (((bitmap->real_end - bitmap->start) / 8) + 1);
+    size = ba_bitmap_size(bitmap);
 
  retval = ext2fs_get_mem(size, &bp->bitarray);
  if (retval) {

@@ -80,7 +87,7 @@ static errcode_t ba_new_bmap(ext2_filsys fs EXT2FS_ATTR((unused)),
      return retval;
 
  bp = (ext2fs_ba_private) bitmap->private;
-    size = (size_t) (((bitmap->real_end - bitmap->start) / 8) + 1);
+    size = ba_bitmap_size(bitmap);
  memset(bp->bitarray, 0, size);
 
  return 0;

@@ -115,7 +122,7 @@ static errcode_t ba_copy_bmap(ext2fs_generic_bitmap_64 src,
 
  dest_bp = (ext2fs_ba_private) dest->private;
 
-    size = (size_t) (((src->real_end - src->start) / 8) + 1);
+    size = ba_bitmap_size(src);
  memcpy (dest_bp->bitarray, src_bp->bitarray, size);
 
  return 0;

@@ -145,8 +152,8 @@ static errcode_t ba_resize_bmap(ext2fs_generic_bitmap_64 bmap,
      return 0;
  }
 
-    size = ((bmap->real_end - bmap->start) / 8) + 1;
-    new_size = ((new_real_end - bmap->start) / 8) + 1;
+    size = ba_bitmap_size(bmap);
+    new_size = ba_bits_size(new_real_end, bmap->start);
 
  if (size != new_size) {
      retval = ext2fs_resize_mem(size, new_size, &bp->bitarray);

@@ -306,8 +313,7 @@ static void ba_clear_bmap(ext2fs_generic_bitmap_64 bitmap)
 {
  ext2fs_ba_private bp = (ext2fs_ba_private) bitmap->private;
 
-    memset(bp->bitarray, 0,
-           (size_t) (((bitmap->real_end - bitmap->start) / 8) + 1));
+    memset(bp->bitarray, 0, ba_bitmap_size(bitmap));
 }
 
 #ifdef ENABLE_BMAP_STATS

Signature · 1 line

2.37.3

badblocks_flags is not used anywhere. So just remove it.

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
lib/ext2fs/badblocks.c | 6 +-----
lib/ext2fs/ext2fsP.h | 1 -

2 files changed, 1 insertion(+), 6 deletions(-)

lib/ext2fs/badblocks.c +1 -5

index a306bc06..345168e0 100644
@@ -81,11 +81,7 @@ errcode_t ext2fs_u32_copy(ext2_u32_list src, ext2_u32_list *dest)
 {
   errcode_t    retval;
 
-    retval = make_u32_list(src->size, src->num, src->list, dest);
-    if (retval)
-        return retval;
-    (*dest)->badblocks_flags = src->badblocks_flags;
-    return 0;
+    return make_u32_list(src->size, src->num, src->list, dest);
 }
 
 errcode_t ext2fs_badblocks_copy(ext2_badblocks_list src,

lib/ext2fs/ext2fsP.h +0 -1

index a20a0502..d2045af8 100644
@@ -34,7 +34,6 @@ struct ext2_struct_u32_list {
   int    num;
   int    size;
   __u32    *list;
-    int    badblocks_flags;
 };
 
 struct ext2_struct_u32_iterate {

Signature · 1 line

2.37.3

From: Wang Shilong <[email protected]>

Add badblocks merge logic

Signed-off-by Wang Shilong <[email protected]>

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
lib/ext2fs/badblocks.c | 75 ++++++++++++++++++++++++++++++++++++++++++
lib/ext2fs/ext2fs.h | 2 ++

2 files changed, 77 insertions(+)

lib/ext2fs/badblocks.c +75 -0

index 345168e0..36794e69 100644
@@ -56,6 +56,74 @@ static errcode_t make_u32_list(int size, int num, __u32 *list,
   return 0;
 }
 
+static inline int insert_ok(blk_t *array, int cnt, blk_t new)
+{
+   return (cnt == 0 || array[cnt - 1] != new);
+}
+
+/*
+ * Merge list from src to dest
+ */
+static errcode_t merge_u32_list(ext2_u32_list src, ext2_u32_list dest)
+{
+   errcode_t    retval;
+   int      src_count = src->num;
+   int      dest_count = dest->num;
+   int      size = src_count + dest_count;
+   int      size_entry = sizeof(blk_t);
+   blk_t       *array;
+   blk_t       *src_array = src->list;
+   blk_t       *dest_array = dest->list;
+   int      src_index = 0;
+   int      dest_index = 0;
+   int      uniq_cnt = 0;
+
+   if (src->num == 0)
+       return 0;
+
+   retval = ext2fs_get_array(size, size_entry, &array);
+   if (retval)
+       return retval;
+
+   /*
+    * It is possible that src list and dest list could be
+    * duplicated when merging badblocks.
+    */
+   while (src_index < src_count || dest_index < dest_count) {
+       if (src_index >= src_count) {
+           for (; dest_index < dest_count; dest_index++)
+               if (insert_ok(array, uniq_cnt, dest_array[dest_index]))
+                   array[uniq_cnt++] = dest_array[dest_index];
+           break;
+       }
+       if (dest_index >= dest_count) {
+           for (; src_index < src_count; src_index++)
+               if (insert_ok(array, uniq_cnt, src_array[src_index]))
+                   array[uniq_cnt++] = src_array[src_index];
+           break;
+       }
+       if (src_array[src_index] < dest_array[dest_index]) {
+           if (insert_ok(array, uniq_cnt, src_array[src_index]))
+               array[uniq_cnt++] = src_array[src_index];
+           src_index++;
+       } else if (src_array[src_index] > dest_array[dest_index]) {
+           if (insert_ok(array, uniq_cnt, dest_array[dest_index]))
+               array[uniq_cnt++] = dest_array[dest_index];
+           dest_index++;
+       } else {
+           if (insert_ok(array, uniq_cnt, dest_array[dest_index]))
+               array[uniq_cnt++] = dest_array[dest_index];
+           src_index++;
+           dest_index++;
+       }
+   }
+
+   ext2fs_free_mem(&dest->list);
+   dest->list = array;
+   dest->num = uniq_cnt;
+   dest->size = size;
+   return 0;
+}
 
 /*
 * This procedure creates an empty u32 list.

@@ -91,6 +159,13 @@ errcode_t ext2fs_badblocks_copy(ext2_badblocks_list src,
                  (ext2_u32_list *) dest);
 }
 
+errcode_t ext2fs_badblocks_merge(ext2_badblocks_list src,
+                ext2_badblocks_list dest)
+{
+   return merge_u32_list((ext2_u32_list) src,
+                 (ext2_u32_list) dest);
+}
+
 /*
 * This procedure frees a badblocks list.
 *

lib/ext2fs/ext2fs.h +2 -0

index 9cc994b1..18dddc2c 100644
@@ -845,6 +845,8 @@ extern int ext2fs_badblocks_list_iterate(ext2_badblocks_iterate iter,
 extern void ext2fs_badblocks_list_iterate_end(ext2_badblocks_iterate iter);
 extern errcode_t ext2fs_badblocks_copy(ext2_badblocks_list src,
                      ext2_badblocks_list *dest);
+extern errcode_t ext2fs_badblocks_merge(ext2_badblocks_list src,
+                   ext2_badblocks_list dest);
 extern int ext2fs_badblocks_equal(ext2_badblocks_list bb1,
                 ext2_badblocks_list bb2);
 extern int ext2fs_u32_list_count(ext2_u32_list bb);

Signature · 1 line

2.37.3

From: Li Xi <[email protected]>

This adds dblist merge logic.

TODO: Add a unit test for core operations of dblist. Currently there is no such test for this.

Signed-off-by Li Xi <[email protected]>

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
lib/ext2fs/dblist.c | 36 ++++++++++++++++++++++++++++++++++++ lib/ext2fs/ext2fs.h | 1 +

2 files changed, 37 insertions(+)

lib/ext2fs/dblist.c +36 -0

index bbdb221d..5568b8ec 100644
@@ -119,6 +119,42 @@ errcode_t ext2fs_copy_dblist(ext2_dblist src, ext2_dblist *dest)
   return 0;
 }
 
+/*
+ * Merge a directory block list @src to @dest
+ */
+errcode_t ext2fs_merge_dblist(ext2_dblist src, ext2_dblist dest)
+{
+   unsigned long long src_count = src->count;
+   unsigned long long dest_count = dest->count;
+   unsigned long long size = src_count + dest_count;
+   size_t size_entry = sizeof(struct ext2_db_entry2);
+   struct ext2_db_entry2 *array, *array2;
+   errcode_t retval;
+
+   if (src_count == 0)
+       return 0;
+
+   if (src->sorted || (dest->sorted && dest_count != 0))
+       return EINVAL;
+
+   retval = ext2fs_get_array(size, size_entry, &array);
+   if (retval)
+       return retval;
+
+   array2 = array;
+   memcpy(array, src->list, src_count * size_entry);
+   array += src_count;
+   memcpy(array, dest->list, dest_count * size_entry);
+   ext2fs_free_mem(&dest->list);
+
+   dest->list = array2;
+   dest->count = src_count + dest_count;
+   dest->size = size;
+   dest->sorted = 0;
+
+   return 0;
+}
+
 /*
 * Close a directory block list
 *

lib/ext2fs/ext2fs.h +1 -0

index 18dddc2c..443f93d2 100644
@@ -1143,6 +1143,7 @@ extern errcode_t ext2fs_add_dir_block(ext2_dblist dblist, ext2_ino_t ino,
                     blk_t blk, int blockcnt);
 extern errcode_t ext2fs_add_dir_block2(ext2_dblist dblist, ext2_ino_t ino,
                      blk64_t blk, e2_blkcnt_t blockcnt);
+extern errcode_t ext2fs_merge_dblist(ext2_dblist src, ext2_dblist dest);
 extern void ext2fs_dblist_sort(ext2_dblist dblist,
                  EXT2_QSORT_TYPE (*sortfunc)(const void *,
                              const void *));

Signature · 1 line

2.37.3

From: Wang Shilong <[email protected]>

Signed-off-by Li Xi <[email protected]>

Signed-off-by Wang Shilong <[email protected]>

[Note: splitted rb merge tree logic patch such that we seperate out libext2fs changes from e2fsck specific changes]

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---

lib/ext2fs/bitmaps.c      | 10 ++++++++++
lib/ext2fs/bmap64.h       |  4 ++++
lib/ext2fs/ext2fs.h       |  8 ++++++++
lib/ext2fs/gen_bitmap64.c | 29 +++++++++++++++++++++++++++++
4 files changed, 51 insertions(+)

lib/ext2fs/bitmaps.c +10 -0

index 8bfa24b1..9437331e 100644
@@ -45,6 +45,16 @@ errcode_t ext2fs_copy_bitmap(ext2fs_generic_bitmap src,
 {
   return (ext2fs_copy_generic_bmap(src, dest));
 }
+
+errcode_t ext2fs_merge_bitmap(ext2fs_generic_bitmap src,
+                  ext2fs_generic_bitmap dest,
+                  ext2fs_generic_bitmap dup,
+                  ext2fs_generic_bitmap dup_allowed)
+{
+    return ext2fs_merge_generic_bmap(src, dest, dup,
+                     dup_allowed);
+}
+
 void ext2fs_set_bitmap_padding(ext2fs_generic_bitmap map)
 {
   ext2fs_set_generic_bmap_padding(map);

lib/ext2fs/bmap64.h +4 -0

index de334548..555193ee 100644
@@ -72,6 +72,10 @@ struct ext2_bitmap_ops {
   void    (*free_bmap)(ext2fs_generic_bitmap_64 bitmap);
   errcode_t (*copy_bmap)(ext2fs_generic_bitmap_64 src,
                ext2fs_generic_bitmap_64 dest);
+    errcode_t (*merge_bmap)(ext2fs_generic_bitmap_64 src,
+                ext2fs_generic_bitmap_64 dest,
+                ext2fs_generic_bitmap_64 dup,
+                ext2fs_generic_bitmap_64 dup_allowed);
   errcode_t (*resize_bmap)(ext2fs_generic_bitmap_64 bitmap,
                  __u64 new_end,
                  __u64 new_real_end);

lib/ext2fs/ext2fs.h +8 -0

index 443f93d2..54aed5d1 100644
@@ -870,6 +870,10 @@ extern void ext2fs_free_block_bitmap(ext2fs_block_bitmap bitmap);
 extern void ext2fs_free_inode_bitmap(ext2fs_inode_bitmap bitmap);
 extern errcode_t ext2fs_copy_bitmap(ext2fs_generic_bitmap src,
                   ext2fs_generic_bitmap *dest);
+extern errcode_t ext2fs_merge_bitmap(ext2fs_generic_bitmap src,
+                  ext2fs_generic_bitmap dest,
+                  ext2fs_generic_bitmap dup,
+                  ext2fs_generic_bitmap dup_allowed);
 extern errcode_t ext2fs_allocate_block_bitmap(ext2_filsys fs,
                         const char *descr,
                         ext2fs_block_bitmap *ret);

@@ -1467,6 +1471,10 @@ void ext2fs_set_generic_bmap_padding(ext2fs_generic_bitmap bmap);
 errcode_t ext2fs_resize_generic_bmap(ext2fs_generic_bitmap bmap,
                    __u64 new_end,
                    __u64 new_real_end);
+errcode_t ext2fs_merge_generic_bmap(ext2fs_generic_bitmap gen_src,
+                                    ext2fs_generic_bitmap gen_dest,
+                    ext2fs_generic_bitmap gen_dup,
+                    ext2fs_generic_bitmap dup_allowed);
 errcode_t ext2fs_compare_generic_bmap(errcode_t neq,
                     ext2fs_generic_bitmap bm1,
                     ext2fs_generic_bitmap bm2);

lib/ext2fs/gen_bitmap64.c +29 -0

index f7710afd..c31f942f 100644
@@ -346,6 +346,35 @@ errcode_t ext2fs_copy_generic_bmap(ext2fs_generic_bitmap gen_src,
   return 0;
 }
 
+errcode_t ext2fs_merge_generic_bmap(ext2fs_generic_bitmap gen_src,
+                    ext2fs_generic_bitmap gen_dest,
+                    ext2fs_generic_bitmap gen_dup,
+                    ext2fs_generic_bitmap gen_dup_allowed)
+{
+    ext2fs_generic_bitmap_64 src = (ext2fs_generic_bitmap_64)gen_src;
+    ext2fs_generic_bitmap_64 dest = (ext2fs_generic_bitmap_64)gen_dest;
+    ext2fs_generic_bitmap_64 dup = (ext2fs_generic_bitmap_64)gen_dup;
+    ext2fs_generic_bitmap_64 dup_allowed = (ext2fs_generic_bitmap_64)gen_dup_allowed;
+
+    if (!src || !dest)
+        return EINVAL;
+
+    if (!EXT2FS_IS_64_BITMAP(src) || !EXT2FS_IS_64_BITMAP(dest) ||
+        (dup && !EXT2FS_IS_64_BITMAP(dup)) ||
+        (dup_allowed && !EXT2FS_IS_64_BITMAP(dup_allowed)))
+        return EINVAL;
+
+    if (src->bitmap_ops != dest->bitmap_ops ||
+        (dup && src->bitmap_ops != dup->bitmap_ops) ||
+        (dup_allowed && src->bitmap_ops != dup_allowed->bitmap_ops))
+        return EINVAL;
+
+    if (src->bitmap_ops->merge_bmap == NULL)
+        return EOPNOTSUPP;
+
+    return src->bitmap_ops->merge_bmap(src, dest, dup, dup_allowed);
+}
+
 errcode_t ext2fs_resize_generic_bmap(ext2fs_generic_bitmap gen_bmap,
                    __u64 new_end,
                    __u64 new_real_end)

Signature · 1 line

2.37.3

From: Li Xi <[email protected]>

Signed-off-by Li Xi <[email protected]>

Signed-off-by Wang Shilong <[email protected]>

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---

lib/ext2fs/ext2_io.h |  2 ++
lib/ext2fs/undo_io.c | 19 +++++++++++++++++++
lib/ext2fs/unix_io.c | 24 +++++++++++++++++++++---
3 files changed, 42 insertions(+), 3 deletions(-)

lib/ext2fs/ext2_io.h +2 -0

index 8fe5b323..8cc355be 100644
@@ -82,6 +82,7 @@ struct struct_io_manager {
   errcode_t (*write_blk)(io_channel channel, unsigned long block,
                  int count, const void *data);
   errcode_t (*flush)(io_channel channel);
+    errcode_t (*flush_cleanup)(io_channel channel);
   errcode_t (*write_byte)(io_channel channel, unsigned long offset,
               int count, const void *data);
   errcode_t (*set_option)(io_channel channel, const char *option,

@@ -116,6 +117,7 @@ struct struct_io_manager {
 #define io_channel_read_blk(c,b,n,d)    ((c)->manager->read_blk((c),b,n,d))
 #define io_channel_write_blk(c,b,n,d)    ((c)->manager->write_blk((c),b,n,d))
 #define io_channel_flush(c)        ((c)->manager->flush((c)))
+#define io_channel_flush_cleanup(c)    ((c)->manager->flush_cleanup((c)))
 #define io_channel_bumpcount(c)        ((c)->refcount++)
 
 /* io_manager.c */

lib/ext2fs/undo_io.c +19 -0

index f4a6d526..678ff421 100644
@@ -1024,6 +1024,24 @@ static errcode_t undo_flush(io_channel channel)
   return retval;
 }
 
+/*
+ * Flush data buffers to disk and cleanup the cache.
+ */
+static errcode_t undo_flush_cleanup(io_channel channel)
+{
+    errcode_t    retval = 0;
+    struct undo_private_data *data;
+
+    EXT2_CHECK_MAGIC(channel, EXT2_ET_MAGIC_IO_CHANNEL);
+    data = (struct undo_private_data *) channel->private_data;
+    EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_UNIX_IO_CHANNEL);
+
+    if (data->real)
+        retval = io_channel_flush_cleanup(data->real);
+
+    return retval;
+}
+
 static errcode_t undo_set_option(io_channel channel, const char *option,
                const char *arg)
 {

@@ -1095,6 +1113,7 @@ static struct struct_io_manager struct_undo_manager = {
   .read_blk    = undo_read_blk,
   .write_blk    = undo_write_blk,
   .flush        = undo_flush,
+    .flush_cleanup    = undo_flush_cleanup,
   .write_byte    = undo_write_byte,
   .set_option    = undo_set_option,
   .get_stats    = undo_get_stats,

lib/ext2fs/unix_io.c +21 -3

index 5b894826..8f8118a3 100644
@@ -1173,9 +1173,9 @@ static errcode_t unix_write_byte(io_channel channel, unsigned long offset,
 }
 
 /*
- * Flush data buffers to disk.
+ * Flush data buffers to disk and invalidate cache if needed
 */
-static errcode_t unix_flush(io_channel channel)
+static errcode_t _unix_flush(io_channel channel, int invalidate)
 {
   struct unix_private_data *data;
   errcode_t retval = 0;

@@ -1185,7 +1185,7 @@ static errcode_t unix_flush(io_channel channel)
   EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_UNIX_IO_CHANNEL);
 
 #ifndef NO_IO_CACHE
-    retval = flush_cached_blocks(channel, data, 0);
+    retval = flush_cached_blocks(channel, data, invalidate);
 #endif
 #ifdef HAVE_FSYNC
   if (!retval && fsync(data->dev) != 0)

@@ -1194,6 +1194,22 @@ static errcode_t unix_flush(io_channel channel)
   return retval;
 }
 
+/*
+ * Flush data buffers to disk.
+ */
+static errcode_t unix_flush(io_channel channel)
+{
+    return _unix_flush(channel, 0);
+}
+
+/*
+ * Flush data buffers to disk and invalidate cache.
+ */
+static errcode_t unix_flush_cleanup(io_channel channel)
+{
+    return _unix_flush(channel, 1);
+}
+
 static errcode_t unix_set_option(io_channel channel, const char *option,
                const char *arg)
 {

@@ -1383,6 +1399,7 @@ static struct struct_io_manager struct_unix_manager = {
   .discard    = unix_discard,
   .cache_readahead    = unix_cache_readahead,
   .zeroout    = unix_zeroout,
+    .flush_cleanup    = unix_flush_cleanup,
 };
 
 io_manager unix_io_manager = &struct_unix_manager;

@@ -1404,6 +1421,7 @@ static struct struct_io_manager struct_unixfd_manager = {
   .discard    = unix_discard,
   .cache_readahead    = unix_cache_readahead,
   .zeroout    = unix_zeroout,
+    .flush_cleanup    = unix_flush_cleanup,
 };
 
 io_manager unixfd_io_manager = &struct_unixfd_manager;

Signature · 1 line

2.37.3

From: Li Xi <[email protected]>

Merge inode_count and inode_link_info properly after threads finish.

Signed-off-by Li Xi <[email protected]>

Signed-off-by Wang Shilong <[email protected]>

[Note: splitted the patch to seperate libext2fs changes from e2fsck]

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
lib/ext2fs/ext2fs.h | 1 +
lib/ext2fs/icount.c | 103 ++++++++++++++++++++++++++++++++++++++++++++

2 files changed, 104 insertions(+)

lib/ext2fs/ext2fs.h +1 -0

index 54aed5d1..139a25fc 100644
@@ -1546,6 +1546,7 @@ extern errcode_t ext2fs_icount_decrement(ext2_icount_t icount, ext2_ino_t ino,
                    __u16 *ret);
 extern errcode_t ext2fs_icount_store(ext2_icount_t icount, ext2_ino_t ino,
                    __u16 count);
+extern errcode_t ext2fs_icount_merge(ext2_icount_t src, ext2_icount_t dest);
 extern ext2_ino_t ext2fs_get_icount_size(ext2_icount_t icount);
 errcode_t ext2fs_icount_validate(ext2_icount_t icount, FILE *);
 

lib/ext2fs/icount.c +103 -0

index 888a90b2..766eccca 100644
@@ -13,6 +13,7 @@
 #if HAVE_UNISTD_H
 #include <unistd.h>
 #endif
+#include <assert.h>
 #include <string.h>
 #include <stdio.h>
 #include <sys/stat.h>

@@ -701,6 +702,108 @@ errcode_t ext2fs_icount_store(ext2_icount_t icount, ext2_ino_t ino,
   return 0;
 }
 
+errcode_t ext2fs_icount_merge_full_map(ext2_icount_t src, ext2_icount_t dest)
+{
+    /* TODO: add the support for full map */
+    return EOPNOTSUPP;
+}
+
+errcode_t ext2fs_icount_merge_el(ext2_icount_t src, ext2_icount_t dest)
+{
+    int             src_count = src->count;
+    int             dest_count = dest->count;
+    int             size = src_count + dest_count;
+    int             size_entry = sizeof(struct ext2_icount_el);
+    struct ext2_icount_el    *array;
+    struct ext2_icount_el    *array_ptr;
+    struct ext2_icount_el    *src_array = src->list;
+    struct ext2_icount_el    *dest_array = dest->list;
+    int             src_index = 0;
+    int             dest_index = 0;
+    errcode_t         retval;
+
+    if (src_count == 0)
+        return 0;
+
+    retval = ext2fs_get_array(size, size_entry, &array);
+    if (retval)
+        return retval;
+
+    array_ptr = array;
+    /*
+     * This can be improved by binary search and memcpy, but codes
+     * would be more complex. And if number of bad blocks is small,
+     * the optimization won't improve performance a lot.
+     */
+    while (src_index < src_count || dest_index < dest_count) {
+        if (src_index >= src_count) {
+            memcpy(array_ptr, &dest_array[dest_index],
+                   (dest_count - dest_index) * size_entry);
+            break;
+        }
+        if (dest_index >= dest_count) {
+            memcpy(array_ptr, &src_array[src_index],
+                   (src_count - src_index) * size_entry);
+            break;
+        }
+        if (src_array[src_index].ino < dest_array[dest_index].ino) {
+            *array_ptr = src_array[src_index];
+            src_index++;
+        } else {
+            assert(src_array[src_index].ino >
+                   dest_array[dest_index].ino);
+            *array_ptr = dest_array[dest_index];
+            dest_index++;
+        }
+        array_ptr++;
+    }
+
+    ext2fs_free_mem(&dest->list);
+    dest->list = array;
+    dest->count = src_count + dest_count;
+    dest->size = size;
+    dest->last_lookup = NULL;
+    return 0;
+}
+
+errcode_t ext2fs_icount_merge(ext2_icount_t src, ext2_icount_t dest)
+{
+    errcode_t    retval;
+
+    if (src->fullmap && !dest->fullmap)
+        return EINVAL;
+
+    if (!src->fullmap && dest->fullmap)
+        return EINVAL;
+
+    if (src->multiple && !dest->multiple)
+        return EINVAL;
+
+    if (!src->multiple && dest->multiple)
+        return EINVAL;
+
+    if (src->fullmap)
+        return ext2fs_icount_merge_full_map(src, dest);
+
+    retval = ext2fs_merge_bitmap(src->single, dest->single, NULL,
+                     NULL);
+    if (retval)
+        return retval;
+
+    if (src->multiple) {
+        retval = ext2fs_merge_bitmap(src->multiple, dest->multiple,
+                         NULL, NULL);
+        if (retval)
+            return retval;
+    }
+
+    retval = ext2fs_icount_merge_el(src, dest);
+    if (retval)
+        return retval;
+
+    return 0;
+}
+
 ext2_ino_t ext2fs_get_icount_size(ext2_icount_t icount)
 {
   if (!icount || icount->magic != EXT2_ET_MAGIC_ICOUNT)

Signature · 1 line

2.37.3

From: Wang Shilong <[email protected]>

Every threads calculate its own quota accounting,
merge them after threads finish.

Signed-off-by Wang Shilong <[email protected]>

[Note: splitted the patch to seperate libext2fs changes from e2fsck]

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
lib/support/mkquota.c | 39 +++++++++++++++++++++++++++++++++++++++ lib/support/quotaio.h | 3 +++

2 files changed, 42 insertions(+)

lib/support/mkquota.c +39 -0

index 9339c994..ce1ab4cd 100644
@@ -618,6 +618,45 @@ out:
   return err;
 }
 
+static errcode_t merge_usage(dict_t *dest, dict_t *src)
+{
+    dnode_t *n;
+    struct dquot *src_dq, *dest_dq;
+
+    for (n = dict_first(src); n; n = dict_next(src, n)) {
+        src_dq = dnode_get(n);
+        if (!src_dq)
+            continue;
+        dest_dq = get_dq(dest, src_dq->dq_id);
+        if (dest_dq == NULL)
+            return -ENOMEM;
+        dest_dq->dq_dqb.dqb_curspace += src_dq->dq_dqb.dqb_curspace;
+        dest_dq->dq_dqb.dqb_curinodes += src_dq->dq_dqb.dqb_curinodes;
+    }
+
+    return 0;
+}
+
+
+errcode_t quota_merge_and_update_usage(quota_ctx_t dest_qctx,
+                    quota_ctx_t src_qctx)
+{
+    dict_t *dict;
+    enum quota_type    qtype;
+    errcode_t retval = 0;
+
+    for (qtype = 0; qtype < MAXQUOTAS; qtype++) {
+        dict = src_qctx->quota_dict[qtype];
+        if (!dict)
+            continue;
+        retval = merge_usage(dest_qctx->quota_dict[qtype], dict);
+        if (retval)
+            break;
+    }
+
+    return retval;
+}
+
 /*
 * Compares the measured quota in qctx->quota_dict with that in the quota inode
 * on disk and updates the limits in qctx->quota_dict. 'usage_inconsistent' is

lib/support/quotaio.h +1 -0

index 84fac35d..240a0762 100644
@@ -40,6 +40,7 @@
 #include "ext2fs/ext2_fs.h"
 #include "ext2fs/ext2fs.h"
 #include "dqblk_v2.h"
+#include "support/dict.h"
 

typedef int64_t qsize_t; /* Type in which we store size limitations */

lib/support/quotaio.h +2 -0

@@ -236,6 +237,8 @@ int quota_file_exists(ext2_filsys fs, enum quota_type qtype);
 void quota_set_sb_inum(ext2_filsys fs, ext2_ino_t ino, enum quota_type qtype);
 errcode_t quota_compare_and_update(quota_ctx_t qctx, enum quota_type qtype,
                  int *usage_inconsistent);
+errcode_t quota_merge_and_update_usage(quota_ctx_t dest_qctx,
+                    quota_ctx_t src_qctx);
 int parse_quota_opts(const char *opts, int (*func)(char *));
 
 /* parse_qtype.c */

Signature · 1 line

2.37.3

From: Saranya Muruganandam <[email protected]>

This patch mainly adds "parent" & "clone_flags" member in ext2_filsys struct for enabling multi-threading. Based on what CLONE flags will be passed from the client of libext2fs down to ext2fs_clone_fs(), those structures/bitmaps will be cloned (thread-aware child copy) and rest will be shared with the parent fs.

The same flags will also help to merge those cloned bitmap structures back into the parent bitmaps when ext2fs_merge_fs() will be called with childfs struct.

Signed-off-by Saranya Muruganandam <[email protected]>

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
lib/ext2fs/dupfs.c | 183 ++++++++++++++++++++++++++++++++++++++++++++ lib/ext2fs/ext2fs.h | 23 ++++++

2 files changed, 206 insertions(+)

lib/ext2fs/dupfs.c +16 -0

index 02721e1a..ecc57cf7 100644
@@ -14,8 +14,12 @@
 #if HAVE_UNISTD_H
 #include <unistd.h>
 #endif
+#if HAVE_PTHREAD_H
+#include <pthread.h>
+#endif
 #include <time.h>
 #include <string.h>
+#include <assert.h>
 
 #include "ext2_fs.h"
 #include "ext2fsP.h"

@@ -120,3 +124,182 @@ errout:
 
 }
 
+#ifdef HAVE_PTHREAD
+errcode_t ext2fs_clone_fs(ext2_filsys fs, ext2_filsys *dest, unsigned int flags)
+{
+    errcode_t retval;
+    ext2_filsys childfs;
+
+    EXT2_CHECK_MAGIC(fs, EXT2_ET_MAGIC_EXT2FS_FILSYS);
+
+    retval = ext2fs_get_mem(sizeof(struct struct_ext2_filsys), &childfs);
+    if (retval)
+        return retval;
+

lib/ext2fs/dupfs.c +19 -0

+    if ((flags & EXT2FS_CLONE_INODE) && fs->inode_map) {
+        retval = ext2fs_copy_bitmap(fs->inode_map, &childfs->inode_map);
+        if (retval)
+            return retval;
+        childfs->inode_map->fs = childfs;
+    }
+
+    if ((flags & EXT2FS_CLONE_BLOCK) && fs->block_map) {
+        retval = ext2fs_copy_bitmap(fs->block_map, &childfs->block_map);
+        if (retval)
+            return retval;
+        childfs->block_map->fs = childfs;
+    }
+
+    if ((flags & EXT2FS_CLONE_BADBLOCKS) && fs->badblocks) {
+        retval = ext2fs_badblocks_copy(fs->badblocks, &childfs->badblocks);
+        if (retval)
+            return retval;
+    }

lib/ext2fs/dupfs.c +19 -0

+
+    childfs->clone_flags = flags;
+    childfs->parent = fs;
+    *dest = childfs;
+
+    return 0;
+}
+
+errcode_t ext2fs_merge_fs(ext2_filsys *thread_fs)
+{
+    ext2_filsys fs = *thread_fs;
+    errcode_t retval = 0;
+    ext2_filsys dest = fs->parent;
+    ext2_filsys src = fs;
+    unsigned int flags = fs->clone_flags;
+    struct ext2_inode_cache *icache;
+    io_channel dest_io;
+    io_channel dest_image_io;
+    ext2fs_inode_bitmap inode_map;

lib/ext2fs/dupfs.c +97 -0

+    icache = dest->icache;
+    dest_io = dest->io;
+    dest_image_io = dest->image_io;
+    inode_map = dest->inode_map;
+    block_map = dest->block_map;
+    badblocks = dest->badblocks;
+    dblist = dest->dblist;
+    priv_data = dest->priv_data;
+    fsflags = dest->flags;
+
+    memcpy(dest, src, sizeof(struct struct_ext2_filsys));
+
+    dest->io = dest_io;
+    dest->image_io = dest_image_io;
+    dest->icache = icache;
+    dest->inode_map = inode_map;
+    dest->block_map = block_map;
+    dest->badblocks = badblocks;
+    dest->dblist = dblist;
+    dest->priv_data = priv_data;
+    if (dest->dblist)
+        dest->dblist->fs = dest;
+    dest->flags = src->flags | fsflags;
+    if (!(src->flags & EXT2_FLAG_VALID) || !(dest->flags & EXT2_FLAG_VALID))
+        ext2fs_unmark_valid(dest);
+
+    if ((flags & EXT2FS_CLONE_INODE) && src->inode_map) {
+        if (dest->inode_map == NULL) {
+            dest->inode_map = src->inode_map;
+            src->inode_map = NULL;
+        } else {
+            retval = ext2fs_merge_bitmap(src->inode_map, dest->inode_map, NULL, NULL);
+            if (retval)
+                goto out;
+        }
+        dest->inode_map->fs = dest;
+    }
+
+    if ((flags & EXT2FS_CLONE_BLOCK) && src->block_map) {
+        if (dest->block_map == NULL) {
+            dest->block_map = src->block_map;
+            src->block_map = NULL;
+        } else {
+            retval = ext2fs_merge_bitmap(src->block_map, dest->block_map, NULL, NULL);
+            if (retval)
+                goto out;
+        }
+        dest->block_map->fs = dest;
+    }
+
+    if ((flags & EXT2FS_CLONE_BADBLOCKS) && src->badblocks) {
+        if (dest->badblocks == NULL)
+            retval = ext2fs_badblocks_copy(src->badblocks, &dest->badblocks);
+        else
+            retval = ext2fs_badblocks_merge(src->badblocks, dest->badblocks);
+        if (retval)
+            goto out;
+    }
+
+    if ((flags & EXT2FS_CLONE_DBLIST) && src->dblist) {
+        if (dest->dblist == NULL) {
+            dest->dblist = src->dblist;
+            src->dblist = NULL;
+        } else {
+            retval = ext2fs_merge_dblist(src->dblist, dest->dblist);
+            if (retval)
+                goto out;
+        }
+        dest->dblist->fs = dest;
+    }
+
+    if (src->icache) {
+        ext2fs_free_inode_cache(src->icache);
+        src->icache = NULL;
+    }
+
+out:
+    if (src->io)
+        io_channel_close(src->io);
+
+    if ((flags & EXT2FS_CLONE_INODE) && src->inode_map)
+        ext2fs_free_generic_bmap(src->inode_map);
+    if ((flags & EXT2FS_CLONE_BLOCK) && src->block_map)
+        ext2fs_free_generic_bmap(src->block_map);
+    if ((flags & EXT2FS_CLONE_BADBLOCKS) && src->badblocks)
+        ext2fs_badblocks_list_free(src->badblocks);
+    if ((flags & EXT2FS_CLONE_DBLIST) && src->dblist) {
+        ext2fs_free_dblist(src->dblist);
+        src->dblist = NULL;
+    }
+
+    ext2fs_free_mem(&src);
+    *thread_fs = NULL;
+
+    return retval;
+}
+#endif

lib/ext2fs/ext2fs.h +23 -0

index 139a25fc..b1505f95 100644
@@ -12,6 +12,10 @@
 #ifndef _EXT2FS_EXT2FS_H
 #define _EXT2FS_EXT2FS_H
 
+#ifdef HAVE_PTHREAD_H
+#include <pthread.h>
+#endif
+
 #ifdef __GNUC__
 #define EXT2FS_ATTR(x) __attribute__(x)
 #else

@@ -331,6 +335,13 @@ struct struct_ext2_filsys {
   struct ext2fs_hashmap* block_sha_map;
 
   const struct ext2fs_nls_table *encoding;
+
+#ifdef HAVE_PTHREAD
+    struct struct_ext2_filsys *parent;
+    size_t refcount;
+    pthread_mutex_t refcount_mutex;
+    unsigned int clone_flags;
+#endif
 };
 
 #if EXT2_FLAT_INCLUDES

@@ -1057,6 +1068,18 @@ extern errcode_t ext2fs_move_blocks(ext2_filsys fs,
 /* check_desc.c */
 extern errcode_t ext2fs_check_desc(ext2_filsys fs);
 
+#ifdef HAVE_PTHREAD
+/* flags for ext2fs_clone_fs */
+#define EXT2FS_CLONE_BLOCK            0x0001
+#define EXT2FS_CLONE_INODE            0x0002
+#define EXT2FS_CLONE_BADBLOCKS        0x0004
+#define EXT2FS_CLONE_DBLIST            0x0008
+
+extern errcode_t ext2fs_clone_fs(ext2_filsys fs, ext2_filsys *dest,
+                                 unsigned int flags);
+extern errcode_t ext2fs_merge_fs(ext2_filsys *fs);
+#endif
+
 /* closefs.c */
 extern errcode_t ext2fs_close(ext2_filsys fs);
 extern errcode_t ext2fs_close2(ext2_filsys fs, int flags);

Signature · 1 line

2.37.3

Quoted 51 lines · Ritesh Harjani (IBM)

From: Saranya Muruganandam <[email protected]>

This patch mainly adds "parent" & "clone_flags" member in ext2_filsys struct for enabling multi-threading. Based on what CLONE flags will be passed from the client of libext2fs down to ext2fs_clone_fs(), those structures/bitmaps will be cloned (thread-aware child copy) and rest will be shared with the parent fs.

The same flags will also help to merge those cloned bitmap structures back into the parent bitmaps when ext2fs_merge_fs() will be called with childfs struct.

Signed-off-by Saranya Muruganandam <[email protected]>

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
lib/ext2fs/dupfs.c | 183 ++++++++++++++++++++++++++++++++++++++++++++ lib/ext2fs/ext2fs.h | 23 ++++++

2 files changed, 206 insertions(+)

lib/ext2fs/dupfs.c +16 -0

index 02721e1a..ecc57cf7 100644
@@ -14,8 +14,12 @@
 #if HAVE_UNISTD_H
 #include <unistd.h>
 #endif
+#if HAVE_PTHREAD_H
+#include <pthread.h>
+#endif
 #include <time.h>
 #include <string.h>
+#include <assert.h>
 
 #include "ext2_fs.h"
 #include "ext2fsP.h"

@@ -120,3 +124,182 @@ errout:
 
 }
 
+#ifdef HAVE_PTHREAD
+errcode_t ext2fs_clone_fs(ext2_filsys fs, ext2_filsys *dest, unsigned int flags)
+{
+    errcode_t retval;
+    ext2_filsys childfs;
+
+    EXT2_CHECK_MAGIC(fs, EXT2_ET_MAGIC_EXT2FS_FILSYS);
+
+    retval = ext2fs_get_mem(sizeof(struct struct_ext2_filsys), &childfs);
+    if (retval)
+        return retval;
+

lib/ext2fs/dupfs.c errout: · line 148

Quoted 10 lines

lib/ext2fs/dupfs.c +10 -0

+    /* make an exact copy implying lists and memory structures are shared */
+    memcpy(childfs, fs, sizeof(struct struct_ext2_filsys));
+    childfs->inode_map = NULL;
+    childfs->block_map = NULL;
+    childfs->badblocks = NULL;
+    childfs->dblist = NULL;
+
+    pthread_mutex_lock(&fs->refcount_mutex);
+    fs->refcount++;
+    pthread_mutex_unlock(&fs->refcount_mutex);

The locking her doesn't make sense. Why is the mutex only protecting the "refcount", and not the rest of the fields in fs?

It would make more sense to hold the mutex over the whole copy process, from up before the memcpy(), until down after the last structure is cloned. Otherwise, once fs has been cloned once and given to another thread it would be possible for them to change these structures.

Quoted 24 lines

lib/ext2fs/dupfs.c +23 -0

+    if ((flags & EXT2FS_CLONE_INODE) && fs->inode_map) {
+        retval = ext2fs_copy_bitmap(fs->inode_map, &childfs->inode_map);
+        if (retval)
+            return retval;
+        childfs->inode_map->fs = childfs;
+    }
+
+    if ((flags & EXT2FS_CLONE_BLOCK) && fs->block_map) {
+        retval = ext2fs_copy_bitmap(fs->block_map, &childfs->block_map);
+        if (retval)
+            return retval;
+        childfs->block_map->fs = childfs;
+    }
+
+    if ((flags & EXT2FS_CLONE_BADBLOCKS) && fs->badblocks) {
+        retval = ext2fs_badblocks_copy(fs->badblocks, &childfs->badblocks);
+        if (retval)
+            return retval;
+    }
+
+    if ((flags & EXT2FS_CLONE_DBLIST) && fs->dblist) {
+        retval = ext2fs_copy_dblist(fs->dblist, &childfs->dblist);
+        if (retval)

lib/ext2fs/dupfs.c errout: · line 178

Quoted 6 lines

lib/ext2fs/dupfs.c +6 -0

+            return retval;
+        childfs->dblist->fs = childfs;
+    }
+
+    /* icache when NULL will be rebuilt if needed */
+    childfs->icache = NULL;

This should be up where the other NULL values are set.

Quoted 24 lines

lib/ext2fs/dupfs.c +23 -0

+
+    childfs->clone_flags = flags;
+    childfs->parent = fs;
+    *dest = childfs;
+
+    return 0;
+}
+
+errcode_t ext2fs_merge_fs(ext2_filsys *thread_fs)
+{
+    ext2_filsys fs = *thread_fs;
+    errcode_t retval = 0;
+    ext2_filsys dest = fs->parent;
+    ext2_filsys src = fs;
+    unsigned int flags = fs->clone_flags;
+    struct ext2_inode_cache *icache;
+    io_channel dest_io;
+    io_channel dest_image_io;
+    ext2fs_inode_bitmap inode_map;
+    ext2fs_block_bitmap block_map;
+    ext2_badblocks_list badblocks;
+    ext2_dblist dblist;
+    void *priv_data;

lib/ext2fs/dupfs.c errout: · line 207

Quoted 6 lines

lib/ext2fs/dupfs.c +6 -0

+    int fsflags;
+
+    pthread_mutex_lock(&fs->refcount_mutex);
+    fs->refcount--;
+    assert(fs->refcount >= 0);
+    pthread_mutex_unlock(&fs->refcount_mutex);

Same here. The mutex should be held over the whole copy process.

Cheers, Andreas

Quoted 148 lines

lib/ext2fs/dupfs.c +97 -0

+    icache = dest->icache;
+    dest_io = dest->io;
+    dest_image_io = dest->image_io;
+    inode_map = dest->inode_map;
+    block_map = dest->block_map;
+    badblocks = dest->badblocks;
+    dblist = dest->dblist;
+    priv_data = dest->priv_data;
+    fsflags = dest->flags;
+
+    memcpy(dest, src, sizeof(struct struct_ext2_filsys));
+
+    dest->io = dest_io;
+    dest->image_io = dest_image_io;
+    dest->icache = icache;
+    dest->inode_map = inode_map;
+    dest->block_map = block_map;
+    dest->badblocks = badblocks;
+    dest->dblist = dblist;
+    dest->priv_data = priv_data;
+    if (dest->dblist)
+        dest->dblist->fs = dest;
+    dest->flags = src->flags | fsflags;
+    if (!(src->flags & EXT2_FLAG_VALID) || !(dest->flags & EXT2_FLAG_VALID))
+        ext2fs_unmark_valid(dest);
+
+    if ((flags & EXT2FS_CLONE_INODE) && src->inode_map) {
+        if (dest->inode_map == NULL) {
+            dest->inode_map = src->inode_map;
+            src->inode_map = NULL;
+        } else {
+            retval = ext2fs_merge_bitmap(src->inode_map, dest->inode_map, NULL, NULL);
+            if (retval)
+                goto out;
+        }
+        dest->inode_map->fs = dest;
+    }
+
+    if ((flags & EXT2FS_CLONE_BLOCK) && src->block_map) {
+        if (dest->block_map == NULL) {
+            dest->block_map = src->block_map;
+            src->block_map = NULL;
+        } else {
+            retval = ext2fs_merge_bitmap(src->block_map, dest->block_map, NULL, NULL);
+            if (retval)
+                goto out;
+        }
+        dest->block_map->fs = dest;
+    }
+
+    if ((flags & EXT2FS_CLONE_BADBLOCKS) && src->badblocks) {
+        if (dest->badblocks == NULL)
+            retval = ext2fs_badblocks_copy(src->badblocks, &dest->badblocks);
+        else
+            retval = ext2fs_badblocks_merge(src->badblocks, dest->badblocks);
+        if (retval)
+            goto out;
+    }
+
+    if ((flags & EXT2FS_CLONE_DBLIST) && src->dblist) {
+        if (dest->dblist == NULL) {
+            dest->dblist = src->dblist;
+            src->dblist = NULL;
+        } else {
+            retval = ext2fs_merge_dblist(src->dblist, dest->dblist);
+            if (retval)
+                goto out;
+        }
+        dest->dblist->fs = dest;
+    }
+
+    if (src->icache) {
+        ext2fs_free_inode_cache(src->icache);
+        src->icache = NULL;
+    }
+
+out:
+    if (src->io)
+        io_channel_close(src->io);
+
+    if ((flags & EXT2FS_CLONE_INODE) && src->inode_map)
+        ext2fs_free_generic_bmap(src->inode_map);
+    if ((flags & EXT2FS_CLONE_BLOCK) && src->block_map)
+        ext2fs_free_generic_bmap(src->block_map);
+    if ((flags & EXT2FS_CLONE_BADBLOCKS) && src->badblocks)
+        ext2fs_badblocks_list_free(src->badblocks);
+    if ((flags & EXT2FS_CLONE_DBLIST) && src->dblist) {
+        ext2fs_free_dblist(src->dblist);
+        src->dblist = NULL;
+    }
+
+    ext2fs_free_mem(&src);
+    *thread_fs = NULL;
+
+    return retval;
+}
+#endif

lib/ext2fs/ext2fs.h +23 -0

index 139a25fc..b1505f95 100644
@@ -12,6 +12,10 @@
 #ifndef _EXT2FS_EXT2FS_H
 #define _EXT2FS_EXT2FS_H
 
+#ifdef HAVE_PTHREAD_H
+#include <pthread.h>
+#endif
+
 #ifdef __GNUC__
 #define EXT2FS_ATTR(x) __attribute__(x)
 #else

@@ -331,6 +335,13 @@ struct struct_ext2_filsys {
   struct ext2fs_hashmap* block_sha_map;
 
   const struct ext2fs_nls_table *encoding;
+
+#ifdef HAVE_PTHREAD
+    struct struct_ext2_filsys *parent;
+    size_t refcount;
+    pthread_mutex_t refcount_mutex;
+    unsigned int clone_flags;
+#endif
 };
 
 #if EXT2_FLAT_INCLUDES

@@ -1057,6 +1068,18 @@ extern errcode_t ext2fs_move_blocks(ext2_filsys fs,
 /* check_desc.c */
 extern errcode_t ext2fs_check_desc(ext2_filsys fs);
 
+#ifdef HAVE_PTHREAD
+/* flags for ext2fs_clone_fs */
+#define EXT2FS_CLONE_BLOCK            0x0001
+#define EXT2FS_CLONE_INODE            0x0002
+#define EXT2FS_CLONE_BADBLOCKS        0x0004
+#define EXT2FS_CLONE_DBLIST            0x0008
+
+extern errcode_t ext2fs_clone_fs(ext2_filsys fs, ext2_filsys *dest,
+                                 unsigned int flags);
+extern errcode_t ext2fs_merge_fs(ext2_filsys *fs);
+#endif
+
 /* closefs.c */
 extern errcode_t ext2fs_close(ext2_filsys fs);
 extern errcode_t ext2fs_close2(ext2_filsys fs, int flags);

Signature · 1 line

2.37.3

Add unit test to verify badblocks list merge api i.e. ext2fs_badblocks_merge()

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
lib/ext2fs/tst_badblocks.c | 61 ++++++++++++++++++++++++++++++++++++--

1 file changed, 59 insertions(+), 2 deletions(-)

lib/ext2fs/tst_badblocks.c +59 -3

index b6e766ab..946de0ae 100644
@@ -119,6 +119,40 @@ static void print_list(badblocks_list bb, int verify)
 	}
 }
 
+static void do_list_merge_verify(badblocks_list bb, badblocks_list bbm, int verify)
+{
+	errcode_t retval;
+	badblocks_iterate iter;
+	blk_t blk;
+	int i, ok;
+
+	retval = ext2fs_badblocks_merge(bb, bbm);
+	if (retval) {
+		com_err("do_list_merge_verify", retval, "while doing list merge");
+		return;
+	}
+
+	if (!verify)
+		return;
+
+	retval = ext2fs_badblocks_list_iterate_begin(bb, &iter);
+	if (retval) {
+		com_err("do_list_merge_verify", retval, "while setting up iterator");
+		return;
+	}
+
+	while (ext2fs_badblocks_list_iterate(iter, &blk)) {
+		retval = ext2fs_badblocks_list_test(bbm, blk);
+		if (retval == 0) {
+			printf(" --- NOT OK\n");
+			test_fail++;
+			return;
+		}
+	}
+	ext2fs_badblocks_list_iterate_end(iter);
+	printf(" --- OK\n");
+}
+
 static void validate_test_seq(badblocks_list bb, blk_t *vec)
 {
 	int	i, match, ok;

@@ -275,13 +309,13 @@ out:
 
 int main(int argc, char **argv)
 {
-	badblocks_list bb1, bb2, bb3, bb4, bb5;
+	badblocks_list bb1, bb2, bb3, bb4, bb5, bbm;
 	int	equal;
 	errcode_t	retval;
 
 	add_error_table(&et_ext2_error_table);
 
-	bb1 = bb2 = bb3 = bb4 = bb5 = 0;
+	bb1 = bb2 = bb3 = bb4 = bb5 = bbm = 0;
 
 	printf("test1: ");
 	retval = create_test_list(test1, &bb1);

@@ -346,6 +380,27 @@ int main(int argc, char **argv)
 		printf("\n");
 	}
 
+	printf("Create merge bb list\n");
+	retval = ext2fs_badblocks_list_create(&bbm, 5);
+	if (retval) {
+		com_err("ext2fs_badblocks_list_create", retval, "while creating list");
+		test_fail++;
+	}
+
+	printf("Merge & Verify all bb{1..5} into bbm\n");
+	if (bb1 && bb2 && bb3 && bb4 && bb5 && bbm) {
+		printf("Merge bb1 into bbm");
+		do_list_merge_verify(bb1, bbm, 1);
+		printf("Merge bb2 into bbm");
+		do_list_merge_verify(bb2, bbm, 1);
+		printf("Merge bb3 into bbm");
+		do_list_merge_verify(bb3, bbm, 1);
+		printf("Merge bb4 into bbm");
+		do_list_merge_verify(bb4, bbm, 1);
+		printf("Merge bb5 into bbm");
+		do_list_merge_verify(bb5, bbm, 1);
+	}
+
 	file_test(bb4);
 
 	file_test_invalid(bb4);

@@ -363,6 +418,8 @@ int main(int argc, char **argv)
 		ext2fs_badblocks_list_free(bb4);
 	if (bb5)
 		ext2fs_badblocks_list_free(bb5);
+	if (bbm)
+		ext2fs_badblocks_list_free(bbm);
 
 	return test_fail;
 
--

2.37.3

This adds a basic copy and merge api test for rbtree bitmap type.

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
lib/ext2fs/Makefile.in | 25 +++-
lib/ext2fs/tst_bitmaps_standalone.c | 170 ++++++++++++++++++++++++++++

2 files changed, 189 insertions(+), 6 deletions(-)

create mode 100644 lib/ext2fs/tst_bitmaps_standalone.c

lib/ext2fs/Makefile.in +8 -2

index f6a050a2..1692500e 100644
@@ -227,6 +227,7 @@ SRCS= ext2_err.c \
 	$(srcdir)/write_bb_file.c \
 	$(srcdir)/rbtree.c \
 	$(srcdir)/tst_libext2fs.c \
+	$(srcdir)/tst_bitmaps_standalone.c \
 	$(DEBUG_SRCS)
 
 HFILES= bitops.h ext2fs.h ext2_io.h ext2_fs.h ext2_ext_attr.h ext3_extents.h \

@@ -328,9 +329,9 @@ tst_getsectsize: tst_getsectsize.o getsectsize.o $(STATIC_LIBEXT2FS) \
 		$(ALL_LDFLAGS) $(STATIC_LIBEXT2FS) $(STATIC_LIBCOM_ERR) \
 		$(SYSLIBS)
 
-tst_types.o: $(srcdir)/tst_types.c ext2_types.h
+tst_types.o: $(srcdir)/tst_types.c ext2_types.h
 
-tst_types: tst_types.o ext2_types.h
+tst_types: tst_types.o ext2_types.h
 	$(E) "	LD $@"
 	$(Q) $(CC) -o tst_types tst_types.o $(ALL_LDFLAGS) $(SYSLIBS)
 

@@ -362,6 +363,11 @@ tst_sha512: $(srcdir)/sha512.c $(srcdir)/ext2_fs.h
 	$(Q) $(CC) $(ALL_LDFLAGS) $(ALL_CFLAGS) -o tst_sha512 \
 		$(srcdir)/sha512.c -DUNITTEST $(SYSLIBS)
 
+tst_bitmaps_standalone: tst_bitmaps_standalone.o $(STATIC_LIBEXT2FS) $(DEPSTATIC_LIBCOM_ERR)
+	$(E) "	LD $@"
+	$(Q) $(CC) -o tst_bitmaps_standalone tst_bitmaps_standalone.o $(ALL_LDFLAGS) \
+		$(STATIC_LIBEXT2FS) $(STATIC_LIBCOM_ERR) $(SYSLIBS)
+

ext2_tdbtool: tdbtool.o

	$(E) "	LD $@"
	$(Q) $(CC) -o ext2_tdbtool tdbtool.o tdb.o $(ALL_LDFLAGS) $(SYSLIBS)

lib/ext2fs/Makefile.in +2 -1

@@ -533,7 +539,7 @@ mkjournal: mkjournal.c $(STATIC_LIBEXT2FS) $(DEPLIBCOM_ERR)
 fullcheck check:: tst_bitops tst_badblocks tst_iscan tst_types tst_icount \
    tst_super_size tst_types tst_inode_size tst_csum tst_crc32c tst_bitmaps \
    tst_inline tst_inline_data tst_libext2fs tst_sha256 tst_sha512 \
-    tst_digest_encode tst_getsize tst_getsectsize
+    tst_digest_encode tst_getsize tst_getsectsize tst_bitmaps_standalone
 	$(TESTENV) ./tst_bitops
 	$(TESTENV) ./tst_badblocks
 	$(TESTENV) ./tst_iscan

@@ -556,6 +562,7 @@ fullcheck check:: tst_bitops tst_badblocks tst_iscan tst_types tst_icount \
 	$(TESTENV) ./tst_bitmaps -l -f $(srcdir)/tst_bitmaps_cmds > tst_bitmaps_out
 	diff $(srcdir)/tst_bitmaps_exp tst_bitmaps_out
 	$(TESTENV) ./tst_digest_encode
+	$(TESTENV) ./tst_bitmaps_standalone
 

installdirs::

	$(E) "	MKDIR_P $(libdir) $(includedir)/ext2fs"

lib/ext2fs/Makefile.in +0 -0

@@ -581,7 +588,7 @@ install:: all $(HFILES) $(HFILES_IN) installdirs ext2fs.pc

uninstall::

	$(RM) -f $(DESTDIR)$(libdir)/libext2fs.a \
		$(DESTDIR)$(pkgconfigdir)/ext2fs.pc

lib/ext2fs/Makefile.in +1 -1

-	$(RM) -rf $(DESTDIR)$(includedir)/ext2fs
+	$(RM) -rf $(DESTDIR)$(includedir)/ext2fs

clean::

	$(RM) -f \#* *.s *.o *.a *~ *.bak core profiled/* \

lib/ext2fs/Makefile.in +8 -2

@@ -590,7 +597,7 @@ clean::
 		tst_bitops tst_types tst_icount tst_super_size tst_csum \
 		tst_bitmaps tst_bitmaps_out tst_extents tst_inline \
 		tst_inline_data tst_inode_size tst_bitmaps_cmd.c \
-		tst_digest_encode tst_sha256 tst_sha512 \
+		tst_digest_encode tst_sha256 tst_sha512  tst_bitmaps_standalone \
 		ext2_tdbtool mkjournal debug_cmds.c tst_cmds.c extent_cmds.c \
 		../libext2fs.a ../libext2fs_p.a ../libext2fs_chk.a \
 		crc32c_table.h gen_crc32ctable tst_crc32c tst_libext2fs \

@@ -646,7 +653,7 @@ windows_io.o: $(srcdir)/windows_io.c $(top_builddir)/lib/config.h \
 $(srcdir)/ext2_ext_attr.h $(srcdir)/bitops.h $(srcdir)/ext2fsP.h
 
 # +++ Dependency line eater +++
-#
+#
 # Makefile dependencies follow.  This must be the last section in
 # the Makefile.in file
 #

@@ -1156,6 +1163,12 @@ tst_iscan.o: $(srcdir)/tst_iscan.c $(top_builddir)/lib/config.h \
 $(srcdir)/ext2_fs.h $(srcdir)/ext3_extents.h $(top_srcdir)/lib/et/com_err.h \
 $(srcdir)/ext2_io.h $(top_builddir)/lib/ext2fs/ext2_err.h \
 $(srcdir)/ext2_ext_attr.h $(srcdir)/hashmap.h $(srcdir)/bitops.h
+tst_bitmaps_standalone.o: $(srcdir)/tst_bitmaps_standalone.c $(top_builddir)/lib/config.h \
+ $(top_builddir)/lib/dirpaths.h $(srcdir)/ext2_fs.h \
+ $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fs.h \
+ $(srcdir)/ext2_fs.h $(srcdir)/ext3_extents.h $(top_srcdir)/lib/et/com_err.h \
+ $(srcdir)/ext2_io.h $(top_builddir)/lib/ext2fs/ext2_err.h \
+ $(srcdir)/ext2_ext_attr.h $(srcdir)/hashmap.h $(srcdir)/bitops.h
 undo_io.o: $(srcdir)/undo_io.c $(top_builddir)/lib/config.h \
 $(top_builddir)/lib/dirpaths.h $(srcdir)/ext2_fs.h \
 $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fs.h \

lib/ext2fs/tst_bitmaps_standalone.c new file +170 -1

new file mode 100644
index 00000000..68b598a8
@@ -0,0 +1,170 @@
+#include "config.h"
+#include <stdio.h>
+#include <string.h>
+#include <assert.h>
+#if HAVE_UNISTD_H
+#include <unistd.h>
+#endif
+#include <fcntl.h>
+#include <time.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#if HAVE_ERRNO_H
+#include <errno.h>
+#endif
+
+#include "ext2_fs.h"
+#include "ext2fs.h"
+#include "bmap64.h"
+
+ext2_filsys test_fs;
+ext2fs_block_bitmap block_map_1;
+ext2fs_block_bitmap block_map_2;
+ext2fs_block_bitmap block_map;
+
+static int test_fail = 0;
+
+void dump_bitmap(ext2fs_generic_bitmap bmap, unsigned int start, unsigned num)
+{
+	unsigned char	*buf;
+	errcode_t	retval;
+	int	i, len = (num - start + 7) / 8;
+
+	buf = malloc(len);
+	if (!buf) {
+		com_err("dump_bitmap", 0, "couldn't allocate buffer");
+		return;
+	}
+	memset(buf, 0, len);
+	retval = ext2fs_get_generic_bmap_range(bmap, (__u64) start, num, buf);
+	if (retval) {
+		com_err("dump_bitmap", retval,
+			"while calling ext2fs_generic_bmap_range");
+		free(buf);
+		return;
+	}
+	for (i=len-1; i >= 0; i--)
+		printf("%02x ", buf[i]);
+	printf("\n");
+	printf("bits set: %u\n", ext2fs_bitcount(buf, len));
+	free(buf);
+}
+
+static void test_copy_run()
+{
+	int blocks[] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 21, 23, 26, 29, 33, 37, 38};
+	errcode_t ret;
+	char *buf_map = NULL;
+	char *buf_copy_map = NULL;
+
+	assert(ext2fs_allocate_block_bitmap(test_fs, "block bitmap", &block_map_1) == 0);
+
+	for (int i = 0; i < sizeof(blocks)/sizeof(blocks[0]); i++) {
+		ext2fs_mark_block_bitmap2(block_map_1, blocks[i]);
+	}
+
+	assert(ext2fs_copy_bitmap(block_map_1, &block_map) == 0);
+
+	if (ext2fs_compare_block_bitmap(block_map_1, block_map) != 0) {
+		printf("block bitmap copy test failed\n");
+		test_fail++;
+
+		dump_bitmap(block_map_1, test_fs->super->s_first_data_block,
+				test_fs->super->s_blocks_count);
+
+		dump_bitmap(block_map, test_fs->super->s_first_data_block,
+				test_fs->super->s_blocks_count);
+	}
+
+	ext2fs_free_block_bitmap(block_map_1);
+	ext2fs_free_block_bitmap(block_map);
+}
+
+void test_merge_run()
+{
+	int blocks_odd[] = {1, 3, 5, 7, 9, 21, 23, 29, 33, 37};
+	int blocks_even[] = {2, 4, 6, 8, 10, 26, 38};
+	ext2fs_generic_bitmap_64 tmp_map;
+
+	assert(ext2fs_allocate_block_bitmap(test_fs, "block bitmap 1", &block_map_1) == 0);
+	assert(ext2fs_allocate_block_bitmap(test_fs, "block bitmap 2", &block_map_2) == 0);
+	assert(ext2fs_allocate_block_bitmap(test_fs, "block bitmap 2", &block_map) == 0);
+
+	for (int i = 0; i < sizeof(blocks_odd) / sizeof(blocks_odd[0]); i++) {
+		ext2fs_mark_block_bitmap2(block_map_1, blocks_odd[i]);
+		ext2fs_mark_block_bitmap2(block_map, blocks_odd[i]);
+	}
+
+	for (int i = 0; i < sizeof(blocks_even) / sizeof(blocks_even[0]); i++) {
+		ext2fs_mark_block_bitmap2(block_map_2, blocks_even[i]);
+		ext2fs_mark_block_bitmap2(block_map, blocks_even[i]);
+	}
+
+	assert(ext2fs_merge_bitmap(block_map_2, block_map_1, NULL, NULL) == 0);
+	if (ext2fs_compare_block_bitmap(block_map_1, block_map) != 0) {
+		printf("block bitmap merge test failed\n");
+		test_fail++;
+
+		dump_bitmap(block_map_1, test_fs->super->s_first_data_block,
+				test_fs->super->s_blocks_count);
+
+		dump_bitmap(block_map, test_fs->super->s_first_data_block,
+				test_fs->super->s_blocks_count);
+	}
+
+	ext2fs_free_block_bitmap(block_map_1);
+	ext2fs_free_block_bitmap(block_map_2);
+	ext2fs_free_block_bitmap(block_map);
+}
+
+static void setup_filesystem(const char *name, unsigned int blocks,
+							 unsigned int inodes, unsigned int type,
+							 unsigned int flags)
+{
+	struct ext2_super_block param;
+	errcode_t ret;
+
+	memset(&param, 0, sizeof(param));
+	ext2fs_blocks_count_set(&param, blocks);
+	param.s_inodes_count = inodes;
+
+	ret = ext2fs_initialize(name, flags, &param, test_io_manager,
+							&test_fs);
+	if (ret) {
+		com_err(name, ret, "while initializing filesystem");
+		return;
+	}
+
+	test_fs->default_bitmap_type = type;
+
+	ext2fs_free_block_bitmap(test_fs->block_map);
+	ext2fs_free_block_bitmap(test_fs->inode_map);
+
+	return;
+errout:
+	ext2fs_close_free(&test_fs);
+}
+
+int main(int argc, char **argv)
+{
+	unsigned int blocks = 127;
+	unsigned int inodes = 0;
+	unsigned int type = EXT2FS_BMAP64_RBTREE;
+	unsigned int flags = EXT2_FLAG_64BITS;
+	char *buf = NULL;
+
+	setup_filesystem(argv[0], blocks, inodes, type, flags);
+
+	/* test for EXT2FS_BMAP64_RBTREE */
+	test_copy_run();
+	test_merge_run();
+
+	/* TODO: test for EXT2FS_BMAP64_BITARRAY */
+
+	if (test_fail)
+		printf("%s: Test copy & merge bitmaps -- NOT OK\n", argv[0]);
+	else
+		printf("%s: Test copy & merge bitmaps -- OK\n", argv[0]);
+
+	return test_fail;
+}
--

2.37.3

Quoted 4 lines · Ritesh Harjani (IBM)

This adds a basic copy and merge api test for rbtree bitmap type.

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

Thanks for adding this test. It looks like it is only testing the default rbtree bitmap type. It would be good to also add a test for regular bitmaps.

Reviewed-by Andreas Dilger <[email protected]>

Quoted 277 lines

---
lib/ext2fs/Makefile.in | 25 +++-
lib/ext2fs/tst_bitmaps_standalone.c | 170 ++++++++++++++++++++++++++++

2 files changed, 189 insertions(+), 6 deletions(-)

create mode 100644 lib/ext2fs/tst_bitmaps_standalone.c

lib/ext2fs/Makefile.in +8 -2

index f6a050a2..1692500e 100644
@@ -227,6 +227,7 @@ SRCS= ext2_err.c \
 	$(srcdir)/write_bb_file.c \
 	$(srcdir)/rbtree.c \
 	$(srcdir)/tst_libext2fs.c \
+	$(srcdir)/tst_bitmaps_standalone.c \
 	$(DEBUG_SRCS)
 
 HFILES= bitops.h ext2fs.h ext2_io.h ext2_fs.h ext2_ext_attr.h ext3_extents.h \

@@ -328,9 +329,9 @@ tst_getsectsize: tst_getsectsize.o getsectsize.o $(STATIC_LIBEXT2FS) \
 		$(ALL_LDFLAGS) $(STATIC_LIBEXT2FS) $(STATIC_LIBCOM_ERR) \
 		$(SYSLIBS)
 
-tst_types.o: $(srcdir)/tst_types.c ext2_types.h
+tst_types.o: $(srcdir)/tst_types.c ext2_types.h
 
-tst_types: tst_types.o ext2_types.h
+tst_types: tst_types.o ext2_types.h
 	$(E) "	LD $@"
 	$(Q) $(CC) -o tst_types tst_types.o $(ALL_LDFLAGS) $(SYSLIBS)
 

@@ -362,6 +363,11 @@ tst_sha512: $(srcdir)/sha512.c $(srcdir)/ext2_fs.h
 	$(Q) $(CC) $(ALL_LDFLAGS) $(ALL_CFLAGS) -o tst_sha512 \
 		$(srcdir)/sha512.c -DUNITTEST $(SYSLIBS)
 
+tst_bitmaps_standalone: tst_bitmaps_standalone.o $(STATIC_LIBEXT2FS) $(DEPSTATIC_LIBCOM_ERR)
+	$(E) "	LD $@"
+	$(Q) $(CC) -o tst_bitmaps_standalone tst_bitmaps_standalone.o $(ALL_LDFLAGS) \
+		$(STATIC_LIBEXT2FS) $(STATIC_LIBCOM_ERR) $(SYSLIBS)
+

ext2_tdbtool: tdbtool.o

	$(E) "	LD $@"
	$(Q) $(CC) -o ext2_tdbtool tdbtool.o tdb.o $(ALL_LDFLAGS) $(SYSLIBS)

lib/ext2fs/Makefile.in +2 -1

@@ -533,7 +539,7 @@ mkjournal: mkjournal.c $(STATIC_LIBEXT2FS) $(DEPLIBCOM_ERR)
 fullcheck check:: tst_bitops tst_badblocks tst_iscan tst_types tst_icount \
    tst_super_size tst_types tst_inode_size tst_csum tst_crc32c tst_bitmaps \
    tst_inline tst_inline_data tst_libext2fs tst_sha256 tst_sha512 \
-    tst_digest_encode tst_getsize tst_getsectsize
+    tst_digest_encode tst_getsize tst_getsectsize tst_bitmaps_standalone
 	$(TESTENV) ./tst_bitops
 	$(TESTENV) ./tst_badblocks
 	$(TESTENV) ./tst_iscan

@@ -556,6 +562,7 @@ fullcheck check:: tst_bitops tst_badblocks tst_iscan tst_types tst_icount \
 	$(TESTENV) ./tst_bitmaps -l -f $(srcdir)/tst_bitmaps_cmds > tst_bitmaps_out
 	diff $(srcdir)/tst_bitmaps_exp tst_bitmaps_out
 	$(TESTENV) ./tst_digest_encode
+	$(TESTENV) ./tst_bitmaps_standalone
 

installdirs::

	$(E) "	MKDIR_P $(libdir) $(includedir)/ext2fs"

lib/ext2fs/Makefile.in +0 -0

@@ -581,7 +588,7 @@ install:: all $(HFILES) $(HFILES_IN) installdirs ext2fs.pc

uninstall::

	$(RM) -f $(DESTDIR)$(libdir)/libext2fs.a \
		$(DESTDIR)$(pkgconfigdir)/ext2fs.pc

lib/ext2fs/Makefile.in +1 -1

-	$(RM) -rf $(DESTDIR)$(includedir)/ext2fs
+	$(RM) -rf $(DESTDIR)$(includedir)/ext2fs

clean::

	$(RM) -f \#* *.s *.o *.a *~ *.bak core profiled/* \

lib/ext2fs/Makefile.in +8 -2

@@ -590,7 +597,7 @@ clean::
 		tst_bitops tst_types tst_icount tst_super_size tst_csum \
 		tst_bitmaps tst_bitmaps_out tst_extents tst_inline \
 		tst_inline_data tst_inode_size tst_bitmaps_cmd.c \
-		tst_digest_encode tst_sha256 tst_sha512 \
+		tst_digest_encode tst_sha256 tst_sha512  tst_bitmaps_standalone \
 		ext2_tdbtool mkjournal debug_cmds.c tst_cmds.c extent_cmds.c \
 		../libext2fs.a ../libext2fs_p.a ../libext2fs_chk.a \
 		crc32c_table.h gen_crc32ctable tst_crc32c tst_libext2fs \

@@ -646,7 +653,7 @@ windows_io.o: $(srcdir)/windows_io.c $(top_builddir)/lib/config.h \
 $(srcdir)/ext2_ext_attr.h $(srcdir)/bitops.h $(srcdir)/ext2fsP.h
 
 # +++ Dependency line eater +++
-#
+#
 # Makefile dependencies follow.  This must be the last section in
 # the Makefile.in file
 #

@@ -1156,6 +1163,12 @@ tst_iscan.o: $(srcdir)/tst_iscan.c $(top_builddir)/lib/config.h \
 $(srcdir)/ext2_fs.h $(srcdir)/ext3_extents.h $(top_srcdir)/lib/et/com_err.h \
 $(srcdir)/ext2_io.h $(top_builddir)/lib/ext2fs/ext2_err.h \
 $(srcdir)/ext2_ext_attr.h $(srcdir)/hashmap.h $(srcdir)/bitops.h
+tst_bitmaps_standalone.o: $(srcdir)/tst_bitmaps_standalone.c $(top_builddir)/lib/config.h \
+ $(top_builddir)/lib/dirpaths.h $(srcdir)/ext2_fs.h \
+ $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fs.h \
+ $(srcdir)/ext2_fs.h $(srcdir)/ext3_extents.h $(top_srcdir)/lib/et/com_err.h \
+ $(srcdir)/ext2_io.h $(top_builddir)/lib/ext2fs/ext2_err.h \
+ $(srcdir)/ext2_ext_attr.h $(srcdir)/hashmap.h $(srcdir)/bitops.h
 undo_io.o: $(srcdir)/undo_io.c $(top_builddir)/lib/config.h \
 $(top_builddir)/lib/dirpaths.h $(srcdir)/ext2_fs.h \
 $(top_builddir)/lib/ext2fs/ext2_types.h $(srcdir)/ext2fs.h \

lib/ext2fs/tst_bitmaps_standalone.c new file +170 -1

new file mode 100644
index 00000000..68b598a8
@@ -0,0 +1,170 @@
+#include "config.h"
+#include <stdio.h>
+#include <string.h>
+#include <assert.h>
+#if HAVE_UNISTD_H
+#include <unistd.h>
+#endif
+#include <fcntl.h>
+#include <time.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#if HAVE_ERRNO_H
+#include <errno.h>
+#endif
+
+#include "ext2_fs.h"
+#include "ext2fs.h"
+#include "bmap64.h"
+
+ext2_filsys test_fs;
+ext2fs_block_bitmap block_map_1;
+ext2fs_block_bitmap block_map_2;
+ext2fs_block_bitmap block_map;
+
+static int test_fail = 0;
+
+void dump_bitmap(ext2fs_generic_bitmap bmap, unsigned int start, unsigned num)
+{
+	unsigned char	*buf;
+	errcode_t	retval;
+	int	i, len = (num - start + 7) / 8;
+
+	buf = malloc(len);
+	if (!buf) {
+		com_err("dump_bitmap", 0, "couldn't allocate buffer");
+		return;
+	}
+	memset(buf, 0, len);
+	retval = ext2fs_get_generic_bmap_range(bmap, (__u64) start, num, buf);
+	if (retval) {
+		com_err("dump_bitmap", retval,
+			"while calling ext2fs_generic_bmap_range");
+		free(buf);
+		return;
+	}
+	for (i=len-1; i >= 0; i--)
+		printf("%02x ", buf[i]);
+	printf("\n");
+	printf("bits set: %u\n", ext2fs_bitcount(buf, len));
+	free(buf);
+}
+
+static void test_copy_run()
+{
+	int blocks[] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 21, 23, 26, 29, 33, 37, 38};
+	errcode_t ret;
+	char *buf_map = NULL;
+	char *buf_copy_map = NULL;
+
+	assert(ext2fs_allocate_block_bitmap(test_fs, "block bitmap", &block_map_1) == 0);
+
+	for (int i = 0; i < sizeof(blocks)/sizeof(blocks[0]); i++) {
+		ext2fs_mark_block_bitmap2(block_map_1, blocks[i]);
+	}
+
+	assert(ext2fs_copy_bitmap(block_map_1, &block_map) == 0);
+
+	if (ext2fs_compare_block_bitmap(block_map_1, block_map) != 0) {
+		printf("block bitmap copy test failed\n");
+		test_fail++;
+
+		dump_bitmap(block_map_1, test_fs->super->s_first_data_block,
+				test_fs->super->s_blocks_count);
+
+		dump_bitmap(block_map, test_fs->super->s_first_data_block,
+				test_fs->super->s_blocks_count);
+	}
+
+	ext2fs_free_block_bitmap(block_map_1);
+	ext2fs_free_block_bitmap(block_map);
+}
+
+void test_merge_run()
+{
+	int blocks_odd[] = {1, 3, 5, 7, 9, 21, 23, 29, 33, 37};
+	int blocks_even[] = {2, 4, 6, 8, 10, 26, 38};
+	ext2fs_generic_bitmap_64 tmp_map;
+
+	assert(ext2fs_allocate_block_bitmap(test_fs, "block bitmap 1", &block_map_1) == 0);
+	assert(ext2fs_allocate_block_bitmap(test_fs, "block bitmap 2", &block_map_2) == 0);
+	assert(ext2fs_allocate_block_bitmap(test_fs, "block bitmap 2", &block_map) == 0);
+
+	for (int i = 0; i < sizeof(blocks_odd) / sizeof(blocks_odd[0]); i++) {
+		ext2fs_mark_block_bitmap2(block_map_1, blocks_odd[i]);
+		ext2fs_mark_block_bitmap2(block_map, blocks_odd[i]);
+	}
+
+	for (int i = 0; i < sizeof(blocks_even) / sizeof(blocks_even[0]); i++) {
+		ext2fs_mark_block_bitmap2(block_map_2, blocks_even[i]);
+		ext2fs_mark_block_bitmap2(block_map, blocks_even[i]);
+	}
+
+	assert(ext2fs_merge_bitmap(block_map_2, block_map_1, NULL, NULL) == 0);
+	if (ext2fs_compare_block_bitmap(block_map_1, block_map) != 0) {
+		printf("block bitmap merge test failed\n");
+		test_fail++;
+
+		dump_bitmap(block_map_1, test_fs->super->s_first_data_block,
+				test_fs->super->s_blocks_count);
+
+		dump_bitmap(block_map, test_fs->super->s_first_data_block,
+				test_fs->super->s_blocks_count);
+	}
+
+	ext2fs_free_block_bitmap(block_map_1);
+	ext2fs_free_block_bitmap(block_map_2);
+	ext2fs_free_block_bitmap(block_map);
+}
+
+static void setup_filesystem(const char *name, unsigned int blocks,
+							 unsigned int inodes, unsigned int type,
+							 unsigned int flags)
+{
+	struct ext2_super_block param;
+	errcode_t ret;
+
+	memset(&param, 0, sizeof(param));
+	ext2fs_blocks_count_set(&param, blocks);
+	param.s_inodes_count = inodes;
+
+	ret = ext2fs_initialize(name, flags, &param, test_io_manager,
+							&test_fs);
+	if (ret) {
+		com_err(name, ret, "while initializing filesystem");
+		return;
+	}
+
+	test_fs->default_bitmap_type = type;
+
+	ext2fs_free_block_bitmap(test_fs->block_map);
+	ext2fs_free_block_bitmap(test_fs->inode_map);
+
+	return;
+errout:
+	ext2fs_close_free(&test_fs);
+}
+
+int main(int argc, char **argv)
+{
+	unsigned int blocks = 127;
+	unsigned int inodes = 0;
+	unsigned int type = EXT2FS_BMAP64_RBTREE;
+	unsigned int flags = EXT2_FLAG_64BITS;
+	char *buf = NULL;
+
+	setup_filesystem(argv[0], blocks, inodes, type, flags);
+
+	/* test for EXT2FS_BMAP64_RBTREE */
+	test_copy_run();
+	test_merge_run();
+
+	/* TODO: test for EXT2FS_BMAP64_BITARRAY */
+
+	if (test_fail)
+		printf("%s: Test copy & merge bitmaps -- NOT OK\n", argv[0]);
+	else
+		printf("%s: Test copy & merge bitmaps -- OK\n", argv[0]);
+
+	return test_fail;
+}
--

Quoted 2 lines

2.37.3

Cheers, Andreas