Skip to contentSkip to conversation

xfs_db: Scan entire file system when using 'frag'

PATCH In review Jorge Guerra 3 participants 6 messages Apr 26, 2019 – Apr 27, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xfs_db: Scan entire file system when using 'frag'

From: Jorge Guerra <[email protected]>

While running the 'frag' command of 'xfs_db' we noticed that the tool is not scanning all the files in the file system. We noticed this when we modified the tool to print the inodes of all the files scanned. For example:

 $ find /mnt/xfsdisk -type f | wc -l
 1782674
 $ xfs_db -r -c frag /dev/sdXX  | grep MB | awk '{print $5}' | paste -s -d+ | bc
 656818

Upon inspecting the code we noticed that the scanfunc_ino function stops processing a given inode block once it encounters a free leaf. However, in practice we see that inodes are necessarily always layed out contiguously on the leaf node. This resulted in the 'frag' command skipping some valid inodes.

In this change we modify the scanfunc_ino function to skip freed inodes. With the change in place we ran the same experiment again and noticed a more accurate file count:

 $ find /mnt/d0 -type f | wc -l
 1810442
 $ xfs_db -r -c frag /dev/sdXX  | grep MB | awk '{print $5}' | paste -s -d+ | bc
 1810442
---
 db/frag.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
db/frag.c +1 -1
index 5f33cb73..91395234 100644
@@ -507,7 +507,7 @@ scanfunc_ino(
 
 				for (j = 0; j < inodes_per_buf; j++) {
 					if (XFS_INOBT_IS_FREE_DISK(&rp[i], ioff + j))
-						goto next_buf;
+						continue;
 					dip = (xfs_dinode_t *)((char *)iocur_top->data +
 						((off + j) << mp->m_sb.sb_inodelog));
 					process_inode(agf, agino + ioff + j, dip);
Signature · 1 line
2.13.5
Reviewed-by on v1 1/1
#1 lore raw
Quoted 22 lines · Jorge Guerra

From: Jorge Guerra <[email protected]>

While running the 'frag' command of 'xfs_db' we noticed that the tool is not scanning all the files in the file system. We noticed this when we modified the tool to print the inodes of all the files scanned. For example:

 $ find /mnt/xfsdisk -type f | wc -l
 1782674
 $ xfs_db -r -c frag /dev/sdXX  | grep MB | awk '{print $5}' | paste -s -d+ | bc
 656818

Upon inspecting the code we noticed that the scanfunc_ino function stops processing a given inode block once it encounters a free leaf. However, in practice we see that inodes are necessarily always layed out contiguously on the leaf node. This resulted in the 'frag' command skipping some valid inodes.

In this change we modify the scanfunc_ino function to skip freed inodes. With the change in place we ran the same experiment again and noticed a more accurate file count:

Quoted 6 lines
 $ find /mnt/d0 -type f | wc -l
 1810442
 $ xfs_db -r -c frag /dev/sdXX  | grep MB | awk '{print $5}' | paste -s -d+ | bc
 1810442

This looks right, but I'll warn you that xfs_db's frag command is largely useless in the first place. ;)

Also, I think:

Fixes 2a5eb70c ("xfs_db: teach the frag command about sparse inode chunks")

Thanks!

Quoted 18 lines
---
 db/frag.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
db/frag.c +1 -1
index 5f33cb73..91395234 100644
@@ -507,7 +507,7 @@ scanfunc_ino(
 
 				for (j = 0; j < inodes_per_buf; j++) {
 					if (XFS_INOBT_IS_FREE_DISK(&rp[i], ioff + j))
-						goto next_buf;
+						continue;
 					dip = (xfs_dinode_t *)((char *)iocur_top->data +
 						((off + j) << mp->m_sb.sb_inodelog));
 					process_inode(agf, agino + ioff + j, dip);
Reviewed-by
#2 lore raw
Quoted 28 lines · Eric Sandeen
Quoted 27 lines · Jorge Guerra

From: Jorge Guerra <[email protected]>

While running the 'frag' command of 'xfs_db' we noticed that the tool is not scanning all the files in the file system. We noticed this when we modified the tool to print the inodes of all the files scanned. For example:

 $ find /mnt/xfsdisk -type f | wc -l
 1782674
 $ xfs_db -r -c frag /dev/sdXX  | grep MB | awk '{print $5}' | paste -s -d+ | bc
 656818

Upon inspecting the code we noticed that the scanfunc_ino function stops processing a given inode block once it encounters a free leaf. However, in practice we see that inodes are necessarily always layed out contiguously on the leaf node. This resulted in the 'frag' command skipping some valid inodes.

In this change we modify the scanfunc_ino function to skip freed inodes. With the change in place we ran the same experiment again and noticed a more accurate file count:

 $ find /mnt/d0 -type f | wc -l
 1810442
 $ xfs_db -r -c frag /dev/sdXX  | grep MB | awk '{print $5}' | paste -s -d+ | bc
 1810442
Quoted 4 lines
Quoted 1 line

This looks right, but I'll warn you that xfs_db's frag command is largely useless in the first place. ;)

I was looking through the manpages for the part where it says frag is useless but couldn't find it. Oh well. :(

--D

Quoted 27 lines

Also, I think:

Fixes 2a5eb70c ("xfs_db: teach the frag command about sparse inode chunks")

Thanks!

Quoted 18 lines
---
 db/frag.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
db/frag.c +1 -1
index 5f33cb73..91395234 100644
@@ -507,7 +507,7 @@ scanfunc_ino(
 
 				for (j = 0; j < inodes_per_buf; j++) {
 					if (XFS_INOBT_IS_FREE_DISK(&rp[i], ioff + j))
-						goto next_buf;
+						continue;
 					dip = (xfs_dinode_t *)((char *)iocur_top->data +
 						((off + j) << mp->m_sb.sb_inodelog));
 					process_inode(agf, agino + ioff + j, dip);
#3 lore raw

Lol!

Thanks Darrick!

Quoted 69 lines · Darrick J. Wong
Quoted 28 lines · Eric Sandeen
Quoted 27 lines · Jorge Guerra

From: Jorge Guerra <[email protected]>

While running the 'frag' command of 'xfs_db' we noticed that the tool is not scanning all the files in the file system. We noticed this when we modified the tool to print the inodes of all the files scanned. For example:

 $ find /mnt/xfsdisk -type f | wc -l
 1782674
 $ xfs_db -r -c frag /dev/sdXX  | grep MB | awk '{print $5}' | paste -s -d+ | bc
 656818

Upon inspecting the code we noticed that the scanfunc_ino function stops processing a given inode block once it encounters a free leaf. However, in practice we see that inodes are necessarily always layed out contiguously on the leaf node. This resulted in the 'frag' command skipping some valid inodes.

In this change we modify the scanfunc_ino function to skip freed inodes. With the change in place we ran the same experiment again and noticed a more accurate file count:

 $ find /mnt/d0 -type f | wc -l
 1810442
 $ xfs_db -r -c frag /dev/sdXX  | grep MB | awk '{print $5}' | paste -s -d+ | bc
 1810442
Quoted 4 lines
Quoted 1 line

This looks right, but I'll warn you that xfs_db's frag command is largely useless in the first place. ;)

I was looking through the manpages for the part where it says frag is useless but couldn't find it. Oh well. :(

--D

Quoted 27 lines

Also, I think:

Fixes 2a5eb70c ("xfs_db: teach the frag command about sparse inode chunks")

Thanks!

Quoted 18 lines
---
 db/frag.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
db/frag.c +1 -1
index 5f33cb73..91395234 100644
@@ -507,7 +507,7 @@ scanfunc_ino(
 
                            for (j = 0; j < inodes_per_buf; j++) {
                                    if (XFS_INOBT_IS_FREE_DISK(&rp[i], ioff + j))
-                                           goto next_buf;
+                                           continue;
                                    dip = (xfs_dinode_t *)((char *)iocur_top->data +
                                            ((off + j) << mp->m_sb.sb_inodelog));
                                    process_inode(agf, agino + ioff + j, dip);
Signature · 1 line
Jorge E Guerra D
#4 lore raw
In reply to Eric Sandeen

Thanks Eric,

I'll update the title.

We are using the frag command to quickly scan the file system and obtain info such as file size distribution and overheads. I'll send that change out for review soon :)

Hopefully that will make the frag command great again! :D

Quoted 60 lines · Eric Sandeen
Quoted 22 lines · Jorge Guerra

From: Jorge Guerra <[email protected]>

While running the 'frag' command of 'xfs_db' we noticed that the tool is not scanning all the files in the file system. We noticed this when we modified the tool to print the inodes of all the files scanned. For example:

 $ find /mnt/xfsdisk -type f | wc -l
 1782674
 $ xfs_db -r -c frag /dev/sdXX  | grep MB | awk '{print $5}' | paste -s -d+ | bc
 656818

Upon inspecting the code we noticed that the scanfunc_ino function stops processing a given inode block once it encounters a free leaf. However, in practice we see that inodes are necessarily always layed out contiguously on the leaf node. This resulted in the 'frag' command skipping some valid inodes.

In this change we modify the scanfunc_ino function to skip freed inodes. With the change in place we ran the same experiment again and noticed a more accurate file count:

Quoted 6 lines
 $ find /mnt/d0 -type f | wc -l
 1810442
 $ xfs_db -r -c frag /dev/sdXX  | grep MB | awk '{print $5}' | paste -s -d+ | bc
 1810442

This looks right, but I'll warn you that xfs_db's frag command is largely useless in the first place. ;)

Also, I think:

Fixes 2a5eb70c ("xfs_db: teach the frag command about sparse inode chunks")

Thanks!

Quoted 18 lines
---
 db/frag.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
db/frag.c +1 -1
index 5f33cb73..91395234 100644
@@ -507,7 +507,7 @@ scanfunc_ino(
 
                              for (j = 0; j < inodes_per_buf; j++) {
                                      if (XFS_INOBT_IS_FREE_DISK(&rp[i], ioff + j))
-                                             goto next_buf;
+                                             continue;
                                      dip = (xfs_dinode_t *)((char *)iocur_top->data +
                                              ((off + j) << mp->m_sb.sb_inodelog));
                                      process_inode(agf, agino + ioff + j, dip);
Signature · 1 line
Jorge E Guerra D
#5 lore raw
Quoted 3 lines · Jorge Guerra

Thanks Eric,

I'll update the title.

No need, I can add the fixes tag when I queue it up.

Quoted 5 lines

We are using the frag command to quickly scan the file system and obtain info such as file size distribution and overheads. I'll send that change out for review soon :)

Hopefully that will make the frag command great again! :D

;) ok, if it becomes less meaningless, you may want to remove the printf which claims that it is meaningless. :)

-Eric

Quoted 64 lines · Eric Sandeen
Quoted 60 lines · Jorge Guerra
Quoted 22 lines

From: Jorge Guerra <[email protected]>

While running the 'frag' command of 'xfs_db' we noticed that the tool is not scanning all the files in the file system. We noticed this when we modified the tool to print the inodes of all the files scanned. For example:

 $ find /mnt/xfsdisk -type f | wc -l
 1782674
 $ xfs_db -r -c frag /dev/sdXX  | grep MB | awk '{print $5}' | paste -s -d+ | bc
 656818

Upon inspecting the code we noticed that the scanfunc_ino function stops processing a given inode block once it encounters a free leaf. However, in practice we see that inodes are necessarily always layed out contiguously on the leaf node. This resulted in the 'frag' command skipping some valid inodes.

In this change we modify the scanfunc_ino function to skip freed inodes. With the change in place we ran the same experiment again and noticed a more accurate file count:

Quoted 6 lines
 $ find /mnt/d0 -type f | wc -l
 1810442
 $ xfs_db -r -c frag /dev/sdXX  | grep MB | awk '{print $5}' | paste -s -d+ | bc
 1810442

This looks right, but I'll warn you that xfs_db's frag command is largely useless in the first place. ;)

Also, I think:

Fixes 2a5eb70c ("xfs_db: teach the frag command about sparse inode chunks")

Thanks!

Quoted 18 lines
---
 db/frag.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
db/frag.c +1 -1
index 5f33cb73..91395234 100644
@@ -507,7 +507,7 @@ scanfunc_ino(
 
                              for (j = 0; j < inodes_per_buf; j++) {
                                      if (XFS_INOBT_IS_FREE_DISK(&rp[i], ioff + j))
-                                             goto next_buf;
+                                             continue;
                                      dip = (xfs_dinode_t *)((char *)iocur_top->data +
                                              ((off + j) << mp->m_sb.sb_inodelog));
                                      process_inode(agf, agino + ioff + j, dip);