Skip to contentSkip to conversation

scsi bus disconnect on high load in qemu kvm anno 2024

DISCUSSION Marius Schwarz 3 participants 5 messages Nov 19, 2024 SCSI

lore
Share & tools
lore
#0 lore raw

Hi,

a bug in the scsi subsystem has been discovered in conjunction with Fedora 40, that can&will destroy data:

https://bugzilla.redhat.com/show_bug.cgi?id=2326393

Every  available information has been added to that ticket, here a short summery:

Host OS: Proxmox / Debian
Guest-OS: Fedora 40
When: right after upgrade to Fedora 40 from 39.
Type: QEMU
Storage IO Cap: 7 GB/s read ( GigaByte, not Gb/s ;) )

Issue:  High Performance SCSI Bus crash in SCSI Subsystem,

BUT: it's not only the kernel, the trigger MUST be a difference in the distribution between F39 and F40,

BECAUSE: after the upgrade to F40, the F39 kernel, which worked for 6 months, had the same issues.

The FIX was, to switch the VM to use SATA connections to the drives.(dd copy of entire disks required)

...[REPEATING over and over again]...
Nov14 23:49] sd 2:0:0:0: [sda] tag#76 ABORT operation started

[  +4,898867] sd 2:0:0:0: ABORT operation timed-out.
[  +0,000008] sd 2:0:1:0: [sdb] tag#383 ABORT operation started
[  +4,914194] sd 2:0:1:0: ABORT operation timed-out.
[  +0,000014] sd 2:0:1:0: [sdb] tag#322 ABORT operation started
[  +4,916147] sd 2:0:1:0: ABORT operation timed-out.
[  +0,000007] sd 2:0:1:0: [sdb] tag#321 ABORT operation started
[  +4,914256] sd 2:0:1:0: ABORT operation timed-out.
[  +0,000009] sd 2:0:1:0: [sdb] tag#320 ABORT operation started
[  +4,915142] sd 2:0:1:0: ABORT operation timed-out.
[  +0,000006] sd 2:0:1:0: [sdb] tag#323 ABORT operation started
[  +4,915177] sd 2:0:1:0: ABORT operation timed-out.
[  +0,000007] sd 2:0:1:0: [sdb] tag#324 ABORT operation started
[  +4,915221] sd 2:0:1:0: ABORT operation timed-out.
[  +0,000007] sd 2:0:1:0: [sdb] tag#325 ABORT operation started
[Nov14 23:50] sd 2:0:1:0: ABORT operation timed-out.
[  +0,009594] scsi target2:0:0: TARGET RESET operation started
[  +4,905550] scsi target2:0:0: TARGET RESET operation timed-out.
[  +0,000005] scsi target2:0:1: TARGET RESET operation started
[ +34,407328] scsi target2:0:1: TARGET RESET operation timed-out.
[  +9,829432] sd 2:0:0:0: [sda] tag#76 ABORT operation started
[  +0,002336] sym0: SCSI BUS reset detected.
[  +0,002062] sd 2:0:0:0: ABORT operation complete.
[  +0,005062] sym0: SCSI BUS has been reset.
[  +3,193215] sd 2:0:1:0: Power-on or device reset occurred
[  +0,000057] sd 2:0:0:0: [sda] tag#76 BUS RESET operation started
[  +0,002319] sd 2:0:0:0: BUS RESET operation complete.
[  +0,000008] sym0: SCSI BUS reset detected.
[  +0,006945] sym0: SCSI BUS has been reset.

...[REPEATING over and over again]...

There is a longer list in the ticket and in the fedora dev mailinglist, including write errors:

[  +0,000000] I/O error, dev sdb, sector 41792 op 0x1:(WRITE) flags 0xc800 phys_seg 96 prio class 2
[  +0,000001] I/O error, dev sdb, sector 41792 op 0x1:(WRITE) flags 0xc800 phys_seg 96 prio class 2
[  +0,000002] sd 2:0:1:0: [sdb] tag#395 timing out command, waited 180s

The disconnects happend as soon as the io system got under high pressure:

- Tar Backups
- database upgrades on the low level structure
- database access that generated a lot of disk io.

The drives returned after 5-10 minutes of reseting the drives.

O== Solution for VM users:

Copy data to virtual SATA controller bases harddrives and delete the old scsi ones. Solved the issue on the spot.

O== Informed

Fedora Dev ML
Fedora Kernel Team
Proxmox Forum
You

#1 lore raw

Marius,

Quoted 15 lines

[ +0,000007] sd 2:0:1:0: [sdb] tag#325 ABORT operation started [Nov14 23:50] sd 2:0:1:0: ABORT operation timed-out.

[  +0,009594] scsi target2:0:0: TARGET RESET operation started
[  +4,905550] scsi target2:0:0: TARGET RESET operation timed-out.
[  +0,000005] scsi target2:0:1: TARGET RESET operation started
[ +34,407328] scsi target2:0:1: TARGET RESET operation timed-out.
[  +9,829432] sd 2:0:0:0: [sda] tag#76 ABORT operation started
[  +0,002336] sym0: SCSI BUS reset detected.
[  +0,002062] sd 2:0:0:0: ABORT operation complete.
[  +0,005062] sym0: SCSI BUS has been reset.
[  +3,193215] sd 2:0:1:0: Power-on or device reset occurred
[  +0,000057] sd 2:0:0:0: [sda] tag#76 BUS RESET operation started
[  +0,002319] sd 2:0:0:0: BUS RESET operation complete.
[  +0,000008] sym0: SCSI BUS reset detected.
[  +0,006945] sym0: SCSI BUS has been reset.

Hannes made some tweaks to the sym53c8xx_2 handling about a year ago which may explain the difference in behavior when I/Os time out due to load.

But why rely on emulating 20-year old hardware in a performance-conscious environment? I thought Proxmox defaulted to virtio?

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#2 lore raw

Am 19.11.24 um 18:05 schrieb Martin K. Petersen:

Quoted 9 lines

Hannes made some tweaks to the sym53c8xx_2 handling about a year ago which may explain the difference in behavior when I/Os time out due to load.

But why rely on emulating 20-year old hardware in a performance-conscious environment? I thought Proxmox defaulted to virtio?

The vms have been setuped per default Proxmox settings. The help page also states, that scsi is the fastest..

The important part is, it worked until the F40 system was wake, than it started to fail apart with the same kernels that worked for months.

best regards,
Marius Schwarz

#3 lore raw
In reply to Marius Schwarz
Quoted 1 line · Marius Schwarz

a bug in the scsi subsystem has been discovered

Why would the observed behavior indicate a bug in the SCSI subsystem? If the queue depth is high enough and the I/O timeout is short enough, any storage controller will get overloaded and will start aborting I/O requests, isn't it?

Bart.

#4 lore raw

Am 19.11.24 um 18:42 schrieb Bart Van Assche:

Quoted 9 lines · Marius Schwarz
Quoted 1 line

a bug in the scsi subsystem has been discovered

Why would the observed behavior indicate a bug in the SCSI subsystem? If the queue depth is high enough and the I/O timeout is short enough, any storage controller will get overloaded and will start aborting I/O requests, isn't it?

Bart.

Because it was working rock solid under any pressure before os upgrade,
and now, with something changed on the os side, not the kernel side, it was producing these resets.

After switching to SATA in proxmox, the same tasks have run flawless again.

Conclusion: there is an unkown trigger in the os for a unseen bug in the scsi subsystem.

best regards,
Marius Schwarz