Skip to contentSkip to conversation

New: Internal error !ino_ok at line 200 of file fs/xfs/libxfs/xfs_dir2.c. Caller xfs_dir_ino_validate+0x5d/0xd0 [xfs]

Bug 215506 [email protected] 2 participants 4 messages Jan 19, 2022

lore
Share & tools
lore
#0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=215506

            Bug ID: 215506
           Summary: Internal error !ino_ok at line 200 of file
                    fs/xfs/libxfs/xfs_dir2.c.  Caller
                    xfs_dir_ino_validate+0x5d/0xd0 [xfs]
           Product: File System
           Version: 2.5
    Kernel Version: 5.15.4
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: XFS
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 300288
--> https://bugzilla.kernel.org/attachment.cgi?id=300288&action=edit
tmp.c

I have encountered a bug in xfs file system.

I created a disk image and modified some properties. After that I mount the image and run some commands related to file operations, and the bug occured.

The file operations are in the "tmp.c" file, and a modified image named "tmp.img" can be found in
(https://drive.google.com/file/d/1SujibjuGYcBA-jjZ5FtR-rSi7koVt0_d/view?usp=sharing).
You can simply reproduce the bug by running the following commands:

gcc -o tmp tmp.c
losetup /dev/loop7 tmp.img
mount -o "attr2,discard,grpid,filestreams,noikeep,inode32,largeio,logbufs=5,noalign,nouuid,noquota,loop" -t xfs /dev/loop7 /root/mnt
./tmp

The kernel message is shown below:

6,2489,54115223218,-;loop7: detected capacity change from 0 to 131072 6,2490,54115436921,-;loop8: detected capacity change from 0 to 131072 5,2491,54115497587,-;XFS (loop8): Mounting V5 Filesystem 1,2492,54115636142,-;XFS (loop8): Internal error !ino_ok at line 200 of file fs/xfs/libxfs/xfs_dir2.c. Caller xfs_dir_ino_validate+0x5d/0xd0 [xfs]
4,2493,54115637100,-;CPU: 0 PID: 17928 Comm: mount Tainted: G W L
5.15.4 #3
4,2494,54115637493,-;Hardware name: LENOVO 20J6A00NHH/20J6A00NHH, BIOS R0FET24W (1.04 ) 12/21/2016
4,2495,54115637742,-;Call Trace:
4,2496,54115637857,-; <TASK>
4,2497,54115638019,-; dump_stack_lvl+0xea/0x130 4,2498,54115638586,-; dump_stack+0x1c/0x25
4,2499,54115639082,-; xfs_error_report+0xd3/0xe0 [xfs] 4,2500,54115639769,-; ? xfs_dir_ino_validate+0x5d/0xd0 [xfs] 4,2501,54115640025,-; ? xfs_dir_ino_validate+0x5d/0xd0 [xfs] 4,2502,54115640025,-; xfs_corruption_error+0xab/0x120 [xfs] 4,2503,54115640025,-; ? write_comp_data+0x37/0xc0 4,2504,54115640025,-; xfs_dir_ino_validate+0xa2/0xd0 [xfs] 4,2505,54115640025,-; ? xfs_dir_ino_validate+0x5d/0xd0 [xfs] 4,2506,54115640025,-; xfs_dir2_sf_verify+0x5d2/0xb50 [xfs] 4,2507,54115640025,-; xfs_ifork_verify_local_data+0xd6/0x180 [xfs] 4,2508,54115640025,-; ? __sanitizer_cov_trace_pc+0x31/0x80 4,2509,54115640025,-; xfs_iformat_data_fork+0x3ff/0x4c0 [xfs] 4,2510,54115640025,-; xfs_inode_from_disk+0xb5a/0x1460 [xfs] 4,2511,54115640025,-; xfs_iget+0x1281/0x2850 [xfs] 4,2512,54115640025,-; ? _raw_write_lock_bh+0x130/0x130 4,2513,54115640025,-; ? xfs_verify_icount+0x31a/0x3f0 [xfs] 4,2514,54115640025,-; ? write_comp_data+0x37/0xc0 4,2515,54115640025,-; ? write_comp_data+0x37/0xc0 4,2516,54115640025,-; ? xfs_perag_get+0x260/0x260 [xfs] 4,2517,54115640025,-; ? xfs_inode_free+0xe0/0xe0 [xfs] 4,2518,54115640025,-; ? xfs_mountfs+0x1227/0x1ff0 [xfs] 4,2519,54115640025,-; ? xfs_blockgc_start+0x76/0x490 [xfs] 4,2520,54115640025,-; ? write_comp_data+0x37/0xc0 4,2521,54115640025,-; xfs_mountfs+0x12f5/0x1ff0 [xfs] 4,2522,54115640025,-; ? xfs_mount_reset_sbqflags+0x1a0/0x1a0 [xfs] 4,2523,54115640025,-; ? __sanitizer_cov_trace_pc+0x31/0x80 4,2524,54115640025,-; ? xfs_mru_cache_create+0x4d2/0x690 [xfs] 4,2525,54115640025,-; ? xfs_filestream_get_ag+0x90/0x90 [xfs] 4,2526,54115640025,-; ? write_comp_data+0x37/0xc0 4,2527,54115640025,-; xfs_fs_fill_super+0x1198/0x2030 [xfs] 4,2528,54115640025,-; get_tree_bdev+0x494/0x850 4,2529,54115640025,-; ? xfs_fs_parse_param+0x1920/0x1920 [xfs] 4,2530,54115640025,-; xfs_fs_get_tree+0x2a/0x40 [xfs] 4,2531,54115640025,-; vfs_get_tree+0x9a/0x380 4,2532,54115640025,-; path_mount+0x7e3/0x24c0 4,2533,54115640025,-; ? __kasan_slab_free+0x147/0x1f0 4,2534,54115640025,-; ? finish_automount+0x860/0x860 4,2535,54115640025,-; ? __sanitizer_cov_trace_pc+0x31/0x80 4,2536,54115640025,-; ? putname+0x165/0x1e0
4,2537,54115640025,-; ? write_comp_data+0x37/0xc0 4,2538,54115640025,-; do_mount+0x11b/0x140
4,2539,54115640025,-; ? path_mount+0x24c0/0x24c0 4,2540,54115640025,-; ? write_comp_data+0x37/0xc0 4,2541,54115640025,-; ? __sanitizer_cov_trace_pc+0x31/0x80 4,2542,54115640025,-; ? write_comp_data+0x37/0xc0 4,2543,54115640025,-; __x64_sys_mount+0x1c3/0x2c0 4,2544,54115640025,-; do_syscall_64+0x3b/0xc0 4,2545,54115640025,-; entry_SYSCALL_64_after_hwframe+0x44/0xae 4,2546,54115640025,-;RIP: 0033:0x7fa63cbb0dde 4,2547,54115640025,-;Code: 48 8b 0d b5 80 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 82 80 0c 00 f7 d8 64 89 01 48 4,2548,54115640025,-;RSP: 002b:00007ffcd394f958 EFLAGS: 00000246 ORIG_RAX:
00000000000000a5
4,2549,54115640025,-;RAX: ffffffffffffffda RBX: 00007fa63ccdf204 RCX:
00007fa63cbb0dde
4,2550,54115640025,-;RDX: 000056155b8a6d10 RSI: 000056155b8a6d90 RDI:
000056155b8af870
4,2551,54115640025,-;RBP: 000056155b8a6b00 R08: 0000000000000000 R09:
000056155b8af980
4,2552,54115640025,-;R10: 0000000000000000 R11: 0000000000000246 R12:
0000000000000000
4,2553,54115640025,-;R13: 000056155b8af870 R14: 000056155b8a6d10 R15:
000056155b8a6b00
4,2554,54115640025,-; </TASK>
1,2555,54115662742,-;XFS (loop8): Corruption detected. Unmount and run xfs_repair
4,2556,54115663126,-;XFS (loop8): Invalid inode number 0x2000000 1,2557,54115663448,-;XFS (loop8): Metadata corruption detected at xfs_dir2_sf_verify+0x906/0xb50 [xfs], inode 0x60 data fork 1,2558,54115664625,-;XFS (loop8): Unmount and run xfs_repair 1,2559,54115665007,-;XFS (loop8): First 17 bytes of corrupted metadata buffer:
1,2560,54115665553,-;00000000: 01 00 00 00 00 60 03 00 60 66 6f 6f 02 00 00 00 .......foo....
1,2561,54115666121,-;00000010: 63
c
4,2562,54115666649,-;XFS (loop8): Failed to read root inode 0x60, error 117

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#1 lore raw
Quoted 1 line · On Wed, Jan 19, 2022 at 03:08:33AM +0000, [email protected]…

I have encountered a bug in xfs file system.

I don't think so.

Quoted 1 line

I created a disk image and modified some properties.

Let's call it what it is: you -corrupted- the disk image.

Quoted 1 line

After that I mount the image

According to the trace, the mount failed because XFS detected one of the many corruptions you created in the filesystem.

Quoted 1 line

and run some commands related to file operations, and the bug occured.

The filesystem mount was aborted due to the detected corruption, so I don't actually beleive that this corruption was produced by the test program....

Quoted 7 lines

The kernel message is shown below:

loop7: detected capacity change from 0 to 131072
loop8: detected capacity change from 0 to 131072
XFS (loop8): Mounting V5 Filesystem
XFS (loop8): Internal error !ino_ok at line 200 of file fs/xfs/libxfs/xfs_dir2.c. Caller xfs_dir_ino_validate+0x5d/0xd0 [xfs]

XFS found an invalid inode number in a directory entry.

Quoted 11 lines
Call Trace:
 <TASK>
 dump_stack_lvl+0xea/0x130
 dump_stack+0x1c/0x25
 xfs_error_report+0xd3/0xe0 [xfs]
 xfs_corruption_error+0xab/0x120 [xfs]
 xfs_dir_ino_validate+0xa2/0xd0 [xfs]
 xfs_dir2_sf_verify+0x5d2/0xb50 [xfs]
 xfs_ifork_verify_local_data+0xd6/0x180 [xfs]
 xfs_iformat_data_fork+0x3ff/0x4c0 [xfs]
 xfs_inode_from_disk+0xb5a/0x1460 [xfs]
Quoted 8 lines
 xfs_iget+0x1281/0x2850 [xfs]
 xfs_mountfs+0x12f5/0x1ff0 [xfs]
 xfs_fs_fill_super+0x1198/0x2030 [xfs]
 get_tree_bdev+0x494/0x850
 xfs_fs_get_tree+0x2a/0x40 [xfs]
 vfs_get_tree+0x9a/0x380
 path_mount+0x7e3/0x24c0
 do_mount+0x11b/0x140

Whilst reading and validating the root inode during mount. The error messages emitted tell us that this isn't a bug but an on-disk corruption that was encountered:

Quoted 8 lines

XFS (loop8): Corruption detected. Unmount and run xfs_repair
XFS (loop8): Invalid inode number 0x2000000
XFS (loop8): Metadata corruption detected at xfs_dir2_sf_verify+0x906/0xb50 [xfs], inode 0x60 data fork XFS (loop8): Unmount and run xfs_repair
XFS (loop8): First 17 bytes of corrupted metadata buffer:
00000000: 01 00 00 00 00 60 03 00 60 66 6f 6f 02 00 00 00 .......foo....
00000010: 63 c
XFS (loop8): Failed to read root inode 0x60, error 117

Using xfs_db, the root inode contains:

u3.sfdir2.hdr.count = 1
u3.sfdir2.hdr.i8count = 0
u3.sfdir2.hdr.parent.i4 = 96
u3.sfdir2.list[0].namelen = 3
u3.sfdir2.list[0].offset = 0x60
u3.sfdir2.list[0].name = "foo"
u3.sfdir2.list[0].inumber.i4 = 33554432

That the inode number for foo is, indeed, 0x200000. Given that this is a 64MB filesystem, the largest inode number that is valid is around 0x20000 (131072). Hence the mount failed immediately as the root inode has clearly been corrupted.

Looking at the AGI btree for AG 0, I see that inode 99 is also allocated, which is where that should point. That contains one entry:

u3.sfdir2.hdr.count = 1
u3.sfdir2.hdr.i8count = 0
u3.sfdir2.hdr.parent.i4 = 96
u3.sfdir2.list[0].namelen = 3
u3.sfdir2.list[0].offset = 0x160
u3.sfdir2.list[0].name = "bar"
u3.sfdir2.list[0].inumber.i4 = 33554560

Which points to inode 0x200080, also beyond EOFS.

So that's two corrupt inode numbers out of two shortform directory entries, both with a very similar corruption.

Then looking at your reproducer, it assumes that the directory structure /foo/bar/ needs to pre-exist before the test file is run. I can also see ifrom the LSN values in various metadata that the filesystem was mounted and foo/bar was created. e.g. those two inodes have a LSN of 0x100000002 (cycle 1, block 2), and so would have been the very first modifications made after the filesystem was created.

I can see other inodes that are marked allocated in AGI 1, and I found that inode 0x8060 should have been the /foo/bar/ directory inode. I can see the traces of it in the literal area when I change the inode to local format:

u3.sfdir2.hdr.count = 7
u3.sfdir2.hdr.i8count = 0
u3.sfdir2.hdr.parent.i4 = 99
u3.sfdir2.list[0].namelen = 255
u3.sfdir2.list[0].offset = 0xffff
u3.sfdir2.list[0].name = "\177az\001\000\000\200a\003\000phln\001\000\000\200b\005\000\200xattr\001\000\000\200c\003\000\230acl\001
....

as it's parent directory points back to inode 99 that I identified above. But it's been corrupted - not only have the format been changed, the first few bytes of the shortform entry have been overwritten and so the directory entry data is corrupt, too.

IOWs, it's pretty clear that this filesystem image was corrupted -after- the test binary was run to create all these files.

So I thought "lets run repair just to see how many structures have been maliciously corrupted":

# xfs_repair -n tmp.img
Phase 1 - find and verify superblock...
bad primary superblock - inconsistent inode alignment value !!!
attempting to find secondary superblock...
.found candidate secondary superblock...
verified secondary superblock...
would write modified primary superblock
Primary superblock would have been modified.
Cannot proceed further in no_modify mode.
Exiting now.
#

Yeah, didn't get far. There's significant modification to all the superblocks - the inode alignment, and all the features fields (f2, bad f2, ro feat, incompat feat) have been zeroed too. SO there are serious inconsistencies between what the filesystem says is the format on disk and what is actually on disk.

Other superblocks fail simple checks, too:

xfs_db> sb 1
Superblock has unknown read-only compatible features (0xff000000) enabled. Attempted to mount read-only compatible filesystem read-write. Filesystem can only be safely mounted read only.
xfs_db> sb 2
SB sanity check failed
Metadata corruption detected at 0x561fdae339f5, xfs_sb block 0x10000/0x200

and so on. There's random metadata field corruption all over the place, and it clearly has been done by a third party as there are corruptions in metadata that clearly is not written by the kernel, yet the CRCs for those objects are correct.

IOWs, you've maliciously modified random fields in the filesystem to try to induce a failure, but the kernel has detected those malicious corruptions and correctly aborted mounting the filesystem. I'd say the kernel code is working exactly as intended at this point in time.

Please close this bug, and for next time, please learn the difference between XFS reporting on-disk corruption and an actual kernel bug.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#2 lore raw

[Bug 215506] Internal error !ino_ok at line 200 of file fs/xfs/libxfs/xfs_dir2.c. Caller xfs_dir_ino_validate+0x5d/0xd0 [xfs]

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=215506

--- Comment #1 from Dave Chinner ([email protected]) ---

Quoted 1 line · On Wed, Jan 19, 2022 at 03:08:33AM +0000, [email protected]…

I have encountered a bug in xfs file system.

I don't think so.

Quoted 1 line

I created a disk image and modified some properties.

Let's call it what it is: you -corrupted- the disk image.

Quoted 1 line

After that I mount the image

According to the trace, the mount failed because XFS detected one of the many corruptions you created in the filesystem.

Quoted 1 line

and run some commands related to file operations, and the bug occured.

The filesystem mount was aborted due to the detected corruption, so I don't actually beleive that this corruption was produced by the test program....

Quoted 7 lines

The kernel message is shown below:

loop7: detected capacity change from 0 to 131072
loop8: detected capacity change from 0 to 131072
XFS (loop8): Mounting V5 Filesystem
XFS (loop8): Internal error !ino_ok at line 200 of file fs/xfs/libxfs/xfs_dir2.c. Caller xfs_dir_ino_validate+0x5d/0xd0 [xfs]

XFS found an invalid inode number in a directory entry.

Quoted 11 lines
Call Trace:
 <TASK>
 dump_stack_lvl+0xea/0x130
 dump_stack+0x1c/0x25
 xfs_error_report+0xd3/0xe0 [xfs]
 xfs_corruption_error+0xab/0x120 [xfs]
 xfs_dir_ino_validate+0xa2/0xd0 [xfs]
 xfs_dir2_sf_verify+0x5d2/0xb50 [xfs]
 xfs_ifork_verify_local_data+0xd6/0x180 [xfs]
 xfs_iformat_data_fork+0x3ff/0x4c0 [xfs]
 xfs_inode_from_disk+0xb5a/0x1460 [xfs]
Quoted 8 lines
 xfs_iget+0x1281/0x2850 [xfs]
 xfs_mountfs+0x12f5/0x1ff0 [xfs]
 xfs_fs_fill_super+0x1198/0x2030 [xfs]
 get_tree_bdev+0x494/0x850
 xfs_fs_get_tree+0x2a/0x40 [xfs]
 vfs_get_tree+0x9a/0x380
 path_mount+0x7e3/0x24c0
 do_mount+0x11b/0x140

Whilst reading and validating the root inode during mount. The error messages emitted tell us that this isn't a bug but an on-disk corruption that was encountered:

Quoted 9 lines

XFS (loop8): Corruption detected. Unmount and run xfs_repair XFS (loop8): Invalid inode number 0x2000000
XFS (loop8): Metadata corruption detected at xfs_dir2_sf_verify+0x906/0xb50 [xfs], inode 0x60 data fork
XFS (loop8): Unmount and run xfs_repair
XFS (loop8): First 17 bytes of corrupted metadata buffer:
00000000: 01 00 00 00 00 60 03 00 60 66 6f 6f 02 00 00 00 .......foo....
00000010: 63 c
XFS (loop8): Failed to read root inode 0x60, error 117

Using xfs_db, the root inode contains:

u3.sfdir2.hdr.count = 1
u3.sfdir2.hdr.i8count = 0
u3.sfdir2.hdr.parent.i4 = 96
u3.sfdir2.list[0].namelen = 3
u3.sfdir2.list[0].offset = 0x60
u3.sfdir2.list[0].name = "foo"
u3.sfdir2.list[0].inumber.i4 = 33554432

That the inode number for foo is, indeed, 0x200000. Given that this is a 64MB filesystem, the largest inode number that is valid is around 0x20000 (131072). Hence the mount failed immediately as the root inode has clearly been corrupted.

Looking at the AGI btree for AG 0, I see that inode 99 is also allocated, which is where that should point. That contains one entry:

u3.sfdir2.hdr.count = 1
u3.sfdir2.hdr.i8count = 0
u3.sfdir2.hdr.parent.i4 = 96
u3.sfdir2.list[0].namelen = 3
u3.sfdir2.list[0].offset = 0x160
u3.sfdir2.list[0].name = "bar"
u3.sfdir2.list[0].inumber.i4 = 33554560

Which points to inode 0x200080, also beyond EOFS.

So that's two corrupt inode numbers out of two shortform directory entries, both with a very similar corruption.

Then looking at your reproducer, it assumes that the directory structure /foo/bar/ needs to pre-exist before the test file is run. I can also see ifrom the LSN values in various metadata that the filesystem was mounted and foo/bar was created. e.g. those two inodes have a LSN of 0x100000002 (cycle 1, block 2), and so would have been the very first modifications made after the filesystem was created.

I can see other inodes that are marked allocated in AGI 1, and I found that inode 0x8060 should have been the /foo/bar/ directory inode. I can see the traces of it in the literal area when I change the inode to local format:

u3.sfdir2.hdr.count = 7
u3.sfdir2.hdr.i8count = 0
u3.sfdir2.hdr.parent.i4 = 99
u3.sfdir2.list[0].namelen = 255
u3.sfdir2.list[0].offset = 0xffff
u3.sfdir2.list[0].name = "\177az\001\000\000\200a\003\000phln\001\000\000\200b\005\000\200xattr\001\000\000\200c\003\000\230acl\001
....

as it's parent directory points back to inode 99 that I identified above. But it's been corrupted - not only have the format been changed, the first few bytes of the shortform entry have been overwritten and so the directory entry data is corrupt, too.

IOWs, it's pretty clear that this filesystem image was corrupted -after- the test binary was run to create all these files.

So I thought "lets run repair just to see how many structures have been maliciously corrupted":

# xfs_repair -n tmp.img
Phase 1 - find and verify superblock...
bad primary superblock - inconsistent inode alignment value !!!
attempting to find secondary superblock...
.found candidate secondary superblock...
verified secondary superblock...
would write modified primary superblock
Primary superblock would have been modified.
Cannot proceed further in no_modify mode.
Exiting now.
#

Yeah, didn't get far. There's significant modification to all the superblocks - the inode alignment, and all the features fields (f2, bad f2, ro feat, incompat feat) have been zeroed too. SO there are serious inconsistencies between what the filesystem says is the format on disk and what is actually on disk.

Other superblocks fail simple checks, too:

xfs_db> sb 1
Superblock has unknown read-only compatible features (0xff000000) enabled. Attempted to mount read-only compatible filesystem read-write. Filesystem can only be safely mounted read only.
xfs_db> sb 2
SB sanity check failed
Metadata corruption detected at 0x561fdae339f5, xfs_sb block 0x10000/0x200

and so on. There's random metadata field corruption all over the place, and it clearly has been done by a third party as there are corruptions in metadata that clearly is not written by the kernel, yet the CRCs for those objects are correct.

IOWs, you've maliciously modified random fields in the filesystem to try to induce a failure, but the kernel has detected those malicious corruptions and correctly aborted mounting the filesystem. I'd say the kernel code is working exactly as intended at this point in time.

Please close this bug, and for next time, please learn the difference between XFS reporting on-disk corruption and an actual kernel bug.

-Dave.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#3 lore raw

[Bug 215506] Internal error !ino_ok at line 200 of file fs/xfs/libxfs/xfs_dir2.c. Caller xfs_dir_ino_validate+0x5d/0xd0 [xfs]

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=215506

bughunter ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|NEW                         |RESOLVED
         Resolution|---                         |ANSWERED

--- Comment #2 from bughunter ([email protected]) --- Thank you for reply! I will learn what you have suggested.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.