New: dmesg flooded with ext4 backtraces when underlying USB device chokes

https://bugzilla.kernel.org/show_bug.cgi?id=220842

            Bug ID: 220842
           Summary: dmesg flooded with ext4 backtraces when underlying USB
                    device chokes
           Product: File System
           Version: 2.5
          Hardware: i386
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P3
         Component: ext4
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 309006
--> https://bugzilla.kernel.org/attachment.cgi?id=309006&action=edit
dmesg

There's an ext4 filesystem on an µSD card inside an USB modem's slot. It looks like ext4 fs driver can't cope with some possibly USB-related problem which stops the underlying device from working. Instead, it floods the kernel log with various call traces like this one:

Call Trace:
 ? dump_stack_lvl+0x42/0x5c
 ? dump_stack+0xd/0x10
 ? __ext4_std_error+0x2ee/0xd38 [ext4]
 ? destroy_inode+0x4b/0x58
 ? evict+0x16a/0x174
 ? simple_inode_init_ts+0xe/0x30
 ? _raw_spin_lock+0x8/0xc
 ? simple_inode_init_ts+0xe/0x30
 ? _raw_spin_lock+0x8/0xc
 ? _atomic_dec_and_lock+0x27/0x38
 ? iput+0x10a/0x110
 ? iget_failed+0x19/0x1c
 ? __ext4_iget+0xbfd/0xc18 [ext4]
 ? ext4_search_dir+0x244/0x69c [ext4]
 ? kmem_cache_alloc_lru_noprof+0x5e/0xb4
 ? ext4_search_dir+0x602/0x69c [ext4]
 ? ext4_search_dir+0x602/0x69c [ext4]
 ? path_openat+0x443/0x81c
 ? do_filp_open+0x80/0xc0
 ? __set_open_fd+0x16/0x2c
 ? alloc_fd+0xf7/0x104
 ? do_sys_openat2+0x4e/0x7c
 ? do_sys_open+0x26/0x30
 ? __ia32_sys_open+0x17/0x1c
 ? ia32_sys_call+0x48/0x10fc
 ? do_int80_syscall_32+0x61/0xb0
 ? do_int80_syscall_32+0x9c/0xb0
 ? __schedule+0x3cf/0x3f8
 ? switch_fpu_return+0x8/0xc
 ? irqentry_exit_to_user_mode+0x65/0xec
 ? sysvec_call_function_single+0x2c/0x2c
 ? irqentry_exit+0x14/0x24
 ? sysvec_apic_timer_interrupt+0x28/0x2c
 ? entry_INT80_32+0xf0/0xf0

It happens when the filesystem is already remounted read-only. Unfortunately I can't tell what happened before because of exhausted kernel log ring buffer.

$ mount
/dev/sdb1 on /media/sdb1 type ext4 (rw,noatime,nodiratime,resuid=82,commit=60)

Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype
needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink
extra_isize bigalloc metadata_csum
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              975360
Block count:              15601664
Reserved block count:     0
Overhead clusters:        16124
Free blocks:              33400
Free inodes:              916292
First block:              0
Block size:               4096
Cluster size:             32768
Group descriptor size:    64
Reserved GDT blocks:      255
Blocks per group:         262144
Clusters per group:       32768
Inodes per group:         16256
Inode blocks per group:   1016
Flex block group size:    16
Filesystem created:       Thu Apr 28 11:12:46 2022
Mount count:              18
Maximum mount count:      -1
Last checked:             Sun Nov 30 00:28:18 2025
Check interval:           0 (<none>)
Lifetime writes:          4604 GB

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220842

Deweloper ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
Kernel Version| |6.17.10 SMP

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220842

--- Comment #1 from Deweloper ([email protected]) ---
Created attachment 309008
--> https://bugzilla.kernel.org/attachment.cgi?id=309008&action=edit
Messages from before 1st call trace

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220842

Theodore Tso ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #2 from Theodore Tso ([email protected]) ---
If the USB block device disappears out from under the file system, there's not
much ext4 can do, in terms of "coping". We can ratelimit some the debugging
messages printed by ext4_destroy_inode(), but no magic that will allow us to recover from a failed storage device.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220842

--- Comment #3 from Deweloper ([email protected]) ---
This message and tons of call traces overflowing dmesg buffer make me think there is some software bug in ext4 fs driver:

<2>[ 5451.308209] EXT4-fs (sdb1): This should not happen!! Data will be lost

What should not happen?
Some invalid internal state?
Or an external event out of control of the software, like disappearing block device, what is normal/expected?
In other words, is all of this noise just a way to say that a mounted device is lost?

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220842

--- Comment #4 from Theodore Tso ([email protected]) ---
Linux doesn't have a way for the block layer to let the system know that
something terrible has happened to a storage device. So whether the cause is
an explosion in the data center, or hard drive head crashing into the spinning magnetic platter at 10,000 RPM, or the user just simply carelessly pulling out
a USB drive, all any file system will see is a whole set of I/O errors. It's
not always this noisy, but in this particular case, you got unlock and it resulted in a whole series of messages to the log.

We can try to reduce the noise, but in general, if there are problems with the
underlying storage device --- fix that first. It should be self-evident that
if there is a problem with the underlying device, (a) it shouldn't happen, and
(b) data will be lost.

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220842

Theodore Tso ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|NEW                         |RESOLVED
         Resolution|---                         |INVALID

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=220842

--- Comment #5 from Artem S. Tashkinov ([email protected]) ---
Ted, maybe it's actually worth implementing this feature in the block layer? To notify filesystems that the device has gone for good?

You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.