New: Drive issues cause system and coredump and reboot
https://bugzilla.kernel.org/show_bug.cgi?id=220669
Bug ID: 220669
Summary: Drive issues cause system and coredump and reboot
Product: File System
Version: 2.5
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: XFS
Assignee: [email protected]
Reporter: [email protected]
Regression: No
We have a number of drives across our clusters that don't present as failing, but seem to have read errors that cause the system to coredump and reboot. Failing drives is obviously not the fault of XFS, but my expectation would be that it doesn't completely cause the system to hang and need to reboot.
Kernel Version:
$ uname -r 4.18.0-553.69.1.el8_10.x86_64
Ideally, I think XFS should be able to more gracefully handle this situation.
We can see the drive at /dev/sdv have issues here before the Call Trace and coredump:
[ 998.356008] sd 0:0:21:0: [sdv] tag#415 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=3s [ 998.365528] sd 0:0:21:0: [sdv] tag#415 Sense Key : Medium Error [current] [descriptor] [ 998.373544] sd 0:0:21:0: [sdv] tag#415 Add. Sense: Unrecovered read error [ 998.380347] sd 0:0:21:0: [sdv] tag#415 CDB: Read(16) 88 00 00 00 00 05 01 1f 6d 90 00 00 00 08 00 00 [ 998.389524] blk_update_request: critical medium error, dev sdv, sector 21493673360 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0 [ 998.404141] XFS (sdv): metadata I/O error in "xfs_da_read_buf+0xd3/0x120 [xfs]" at daddr 0x5011f6d90 len 8 error 61 [ 998.414706] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020 [ 998.422554] PGD 1218fc067 P4D 1218fc067 PUD 1d0139067 PMD 0 [ 998.428247] Oops: 0002 [#1] SMP NOPTI [ 998.431931] CPU: 72 PID: 4802 Comm: swift-object-au Kdump: loaded Not tainted 4.18.0-553.69.1.el8_10.x86_64 #1 [ 998.441958] Hardware name: Dell Inc. PowerEdge R740xd2/0VNGN1, BIOS 2.14.2 03/23/2022 [ 998.449818] RIP: 0010:_raw_spin_lock_irqsave+0x1e/0x40 [ 998.454975] Code: cf 80 0b 08 eb 88 90 90 90 90 90 90 0f 1f 44 00 00 53 9c 58 0f 1f 44 00 00 48 89 c3 fa 66 0f 1f 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 75 09 48 89 d8 5b c3 cc cc cc cc 89 c6 e8 1c 7b 73 ff [ 998.473799] RSP: 0018:ffff99455ccafbe0 EFLAGS: 00010046 [ 998.479344] RAX: 0000000000000000 RBX: 0000000000000286 RCX: 0000000000000000 [ 998.486695] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000020 [ 998.494044] RBP: 0000000000000000 R08: 0000000080000000 R09: ffff8e3abffd5000 [ 998.501360] R10: 0000000000000001 R11: ffff99455ccaf848 R12: 0000000000000000 [ 998.508690] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 998.516004] FS: 00007f74aa03e740(0000) GS:ffff8e3941100000(0000) knlGS:0000000000000000 [ 998.524333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 998.530270] CR2: 0000000000000020 CR3: 00000001cfbb6003 CR4: 00000000007706e0 [ 998.537585] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 998.544901] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 998.552224] PKRU: 55555554 [ 998.555136] Call Trace: [ 998.557765] ? __die_body+0x1a/0x60 [ 998.561435] ? no_context+0x1ba/0x3f0 [ 998.565276] ? __bad_area_nosemaphore+0x157/0x180 [ 998.570202] ? xfs_trans_read_buf_map+0x128/0x360 [xfs] [ 998.575693] ? do_page_fault+0x37/0x12d [ 998.579713] ? page_fault+0x1e/0x30 [ 998.583390] ? _raw_spin_lock_irqsave+0x1e/0x40 [ 998.588113] up+0x12/0x50 [ 998.590913] xfs_buf_unlock+0x15/0x70 [xfs] [ 998.595350] xfs_trans_brelse+0xc6/0xe0 [xfs] [ 998.599960] xfs_attr_leaf_get+0xb0/0xc0 [xfs] [ 998.604639] xfs_attr_get+0x9e/0xc0 [xfs] [ 998.608888] xfs_xattr_get+0x75/0xb0 [xfs] [ 998.613238] __vfs_getxattr+0x54/0x70 [ 998.617104] vfs_getxattr+0x118/0x140 [ 998.620959] getxattr+0x187/0x1b0 [ 998.624470] ? cp_new_stat+0x150/0x190 [ 998.628411] ? do_vfs_ioctl+0xa4/0x690 [ 998.632356] ? __do_sys_newfstat+0x5e/0x70 [ 998.636644] ? syscall_trace_enter+0x1ff/0x2d0 [ 998.641283] __x64_sys_fgetxattr+0x5f/0xb0 [ 998.645573] do_syscall_64+0x5b/0x1a0 [ 998.649429] entry_SYSCALL_64_after_hwframe+0x66/0xcb [ 998.654672] RIP: 0033:0x7f74a88dc5fe [ 998.658434] Code: 48 8b 0d 8d 78 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 c1 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5a 78 39 00 f7 d8 64 89 01 48 [ 998.677570] RSP: 002b:00007ffe2289a9f8 EFLAGS: 00000246 ORIG_RAX: 00000000000000c1 [ 998.685351] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f74a88dc5fe [ 998.692683] RDX: 0000000000000000 RSI: 00007f749538e7d0 RDI: 0000000000000003 [ 998.700024] RBP: 00007ffe2289aa80 R08: 0000000000000000 R09: 00007f74a9f400d0 [ 998.707357] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 998.714680] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000000 [ 998.722013] Modules linked in: dm_mod tun xfs ipt_MASQUERADE
nf_conntrack_netlink xt_addrtype br_netfilter bridge stp llc tcp_diag inet_diag overlay xt_multiport ip6t_REJECT nf_reject_ipv6 ip6t_rpfilter ipt_REJECT nf_reject_ipv4 xt_conntrack nft_counter nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6_tables ip_tables nft_compat ip_set nf_tables libcrc32c nfnetlink dell_rbu vfat fat ipmi_ssif intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel irdma kvm i40e irqbypass crc32_pclmul ib_uverbs rapl dell_smbios iTCO_wdt intel_cstate iTCO_vendor_support acpi_ipmi dell_wmi_descriptor wmi_bmof mei_me dcdbas ib_core intel_uncore pcspkr mei i2c_i801 ipmi_si lpc_ich wmi ipmi_devintf ipmi_msghandler ext4 mbcache jbd2 sd_mod sg mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt i2c_algo_bit crct10dif_pclmul drm_shmem_helper ahci ice nvme
[ 998.722079] crc32c_intel libahci drm nvme_core ghash_clmulni_intel libata
megaraid_sas tg3 t10_pi fuse
crash> sys | grep -e CPUS -e LOAD -e RELEASE -e PANIC
CPUS: 80
LOAD AVERAGE: 79.36, 197.55, 179.00
RELEASE: 4.18.0-553.69.1.el8_10.x86_64
PANIC: "BUG: unable to handle kernel NULL pointer dereference at
0000000000000020"
crash> set -p
PID: 4802
COMMAND: "swift-object-au"
TASK: ffff8e3ad2990000 [THREAD_INFO: ffff8e3ad2990000]
CPU: 72
STATE: TASK_RUNNING (PANIC)crash> bt
PID: 4802 TASK: ffff8e3ad2990000 CPU: 72 COMMAND: "swift-object-au"
#0 [ffff99455ccaf910] machine_kexec at ffffffff9b86de53 #1 [ffff99455ccaf968] __crash_kexec at ffffffff9b9b9e1a #2 [ffff99455ccafa28] crash_kexec at ffffffff9b9bad51 #3 [ffff99455ccafa40] oops_end at ffffffff9b82c131
#4 [ffff99455ccafa60] no_context at ffffffff9b880da3 #5 [ffff99455ccafab8] __bad_area_nosemaphore at ffffffff9b881107 #6 [ffff99455ccafb00] do_page_fault at ffffffff9b881dc7 #7 [ffff99455ccafb30] page_fault at ffffffff9c4011fe
[exception RIP: _raw_spin_lock_irqsave+30]
RIP: ffffffff9c2270be RSP: ffff99455ccafbe0 RFLAGS: 00010046
RAX: 0000000000000000 RBX: 0000000000000286 RCX: 0000000000000000
RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000020
RBP: 0000000000000000 R8: 0000000080000000 R9: ffff8e3abffd5000
R10: 0000000000000001 R11: ffff99455ccaf848 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #8 [ffff99455ccafbe8] up at ffffffff9c2230e2
#9 [ffff99455ccafc00] xfs_buf_unlock at ffffffffc0d95455 [xfs]
#10 [ffff99455ccafc20] xfs_trans_brelse at ffffffffc0dcf3b6 [xfs] #11 [ffff99455ccafc40] xfs_attr_leaf_get at ffffffffc0d49e80 [xfs] #12 [ffff99455ccafc70] xfs_attr_get at ffffffffc0d4afae [xfs] #13 [ffff99455ccafc90] xfs_xattr_get at ffffffffc0dbaa05 [xfs] #14 [ffff99455ccafd40] __vfs_getxattr at ffffffff9bb9a064
#15 [ffff99455ccafd70] vfs_getxattr at ffffffff9bb9a5a8
#16 [ffff99455ccafdb8] getxattr at ffffffff9bb9aa07
#17 [ffff99455ccaff00] __x64_sys_fgetxattr at ffffffff9bb9b17f #18 [ffff99455ccaff38] do_syscall_64 at ffffffff9b803cab
#19 [ffff99455ccaff50] entry_SYSCALL_64_after_hwframe at ffffffff9c40012e
RIP: 00007f74a88dc5fe RSP: 00007ffe2289a9f8 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f74a88dc5fe
RDX: 0000000000000000 RSI: 00007f749538e7d0 RDI: 0000000000000003
RBP: 00007ffe2289aa80 R8: 0000000000000000 R9: 00007f74a9f400d0
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: 00000000000000c1 CS: 0033 SS: 002bcrash> dis _raw_spin_lock_irqsave+300xffffffff9c2270be <_raw_spin_lock_irqsave+30>: lock cmpxchg %edx,(%rdi)
RDI From the above
[exception RIP: _raw_spin_lock_irqsave+30]
RIP: ffffffff9c2270be RSP: ffff99455ccafbe0 RFLAGS: 00010046
RAX: 0000000000000000 RBX: 0000000000000286 RCX: 0000000000000000
RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000020
RBP: 0000000000000000 R8: 0000000080000000 R9: ffff8e3abffd5000
R10: 0000000000000001 R11: ffff99455ccaf848 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000Seems like that memory address would be incorrect for RDI?
https://www.kernel.org/doc/Documentation/x86/x86_64/mm.txt
The vmcore is quite large. But I can probably get any information that you need to investigate further if required.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 220669] Drive issues cause system and coredump and reboot
https://bugzilla.kernel.org/show_bug.cgi?id=220669
--- Comment #1 from Brendan Shephard ([email protected]) --- Created attachment 308812
--> https://bugzilla.kernel.org/attachment.cgi?id=308812&action=edit
vmcore-dmesg.txt
The generated vmcore-dmesg.txt file
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Quoted 16 lines · On Thu, Oct 16, 2025 at 12:10:43AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=220669
Bug ID: 220669
Summary: Drive issues cause system and coredump and reboot
Product: File System
Version: 2.5
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: XFS
Assignee: [email protected]
Reporter: [email protected]
Regression: No
Quoted 4 lines
We have a number of drives across our clusters that don't present as failing, but seem to have read errors that cause the system to coredump and reboot. Failing drives is obviously not the fault of XFS, but my expectation would be that it doesn't completely cause the system to hang and need to reboot.
Fixed upstream a couple of months back by commit ae668cd567a6 ("xfs:
do not propagate ENODATA disk errors into xattr code"). You'll need to raise a distro bug (Rocky Linux) to get them to back port it for you.
Please close this bz.
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
[Bug 220669] Drive issues cause system and coredump and reboot
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=220669
--- Comment #2 from Dave Chinner ([email protected]) ---
Quoted 16 lines · On Thu, Oct 16, 2025 at 12:10:43AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=220669
Bug ID: 220669
Summary: Drive issues cause system and coredump and reboot
Product: File System
Version: 2.5
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: XFS
Assignee: [email protected]
Reporter: [email protected]
Regression: No
Quoted 4 lines
We have a number of drives across our clusters that don't present as failing, but seem to have read errors that cause the system to coredump and reboot. Failing drives is obviously not the fault of XFS, but my expectation would be that it doesn't completely cause the system to hang and need to reboot.
Fixed upstream a couple of months back by commit ae668cd567a6 ("xfs:
do not propagate ENODATA disk errors into xattr code"). You'll need to raise a distro bug (Rocky Linux) to get them to back port it for you.
Please close this bz.
-Dave.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 220669] Drive issues cause system and coredump and reboot
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=220669
Artem S. Tashkinov ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |WILL_NOT_FIX--- Comment #3 from Artem S. Tashkinov ([email protected]) --- Please request support from RedHat.
Vendor kernels are not supported here.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 220669] Drive issues cause system and coredump and reboot
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=220669
--- Comment #4 from Artem S. Tashkinov ([email protected]) ---
Quoted 9 lines
[ 998.356008] sd 0:0:21:0: [sdv] tag#415 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=3s [ 998.365528] sd 0:0:21:0: [sdv] tag#415 Sense Key : Medium Error [current] [descriptor] [ 998.373544] sd 0:0:21:0: [sdv] tag#415 Add. Sense: Unrecovered read error [ 998.380347] sd 0:0:21:0: [sdv] tag#415 CDB: Read(16) 88 00 00 00 00 05 01 1f 6d 90 00 00 00 08 00 00 [ 998.389524] blk_update_request: critical medium error, dev sdv, sector21493673360 op 0x0:(READ) flags
0x1000phys_seg1 prio class 0
You may need to replace your failing hardware first.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 220669] Drive issues cause system and coredump and reboot
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=220669
--- Comment #5 from Brendan Shephard ([email protected]) --- (In reply to Dave Chinner from comment #2)
Quoted 24 lines · On Thu, Oct 16, 2025 at 12:10:43AM +0000, [email protected] wrote:
Quoted 16 lines
https://bugzilla.kernel.org/show_bug.cgi?id=220669
Bug ID: 220669
Summary: Drive issues cause system and coredump and reboot
Product: File System
Version: 2.5
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P3
Component: XFS
Assignee: [email protected]
Reporter: [email protected]
Regression: NoQuoted 1 line
We have a number of drives across our clusters that don't present as
failing,
Quoted 2 lines
but seem to have read errors that cause the system to coredump and reboot. Failing drives is obviously not the fault of XFS, but my expectation would
be
Quoted 1 line
that it doesn't completely cause the system to hang and need to reboot.
Quoted 8 lines
Fixed upstream a couple of months back by commit
ae668cd567a6("xfs:
do not propagate ENODATA disk errors into xattr code"). You'll need to raise a distro bug (Rocky Linux) to get them to back port it for you.Please close this bz.
-Dave.
Legend! Thanks for the pointer to the commit. I really appreciate you taking a look so quickly!
All the best
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.